LMArena Co-Founders on the Future of AI Rankings

Sure. So LMArena is an open community platform for testing and evaluating all the top AI models. So you can go to lmarena.ai, and what you'll see is the ability to interact with not just ChatGPT, but also Google's Gemini or Anthropic's Claude, and not just interact with them, but also to battle them against one another. So you can chat and then you'll see two anonymous models and say, compare which one is better, vote for it. And we use those votes to construct a leaderboard of which models are best on the distribution of real-world use, also on different tasks like math, coding, instruction following. And it's not just us using this, but we have a community of millions of people that come to the platform and vote. They come for all sorts of reasons. It's pretty diverse: 60% tech, but 40% all sorts of other things, just to try different things and to use them all in one platform. So that's what LMArena is. And if you have anything to add, Wei-Lin.

So yeah, maybe it's also worth saying, you know, part of the reason why people come is they like seeing multiple opinions. They like seeing multiple AIs in one place and using them all in one interface and comparing them, being able to do so on one platform. And they're also really excited by the opportunity to look at some new models from the top providers pre-release. So they all sort of appreciate that, and that's what's led to some growth.

Sure. So I think it's worth maybe zooming out and thinking a little bit about what is a standard benchmark in AI versus what is LMArena. So in a standard benchmark, what happens is you are looking at a narrow vertical of tasks. Think about something like an ImageNet or an MMLU, which is a multiple-choice question-answering dataset. Let's say multiple-choice question answering. I want to understand how good my models are, so I'm going to collect a bunch of multiple-choice questions. I'm going to have experts label them. I'm going to collect that into a dataset. I release it online or I keep it private. And then I have models answer those questions and then I see how they perform. So that's the standard paradigm of so-called benchmarking. Now, what's the issue with this? The issue with this is in the current era of AI, things are moving so quickly and AI models are doing all these subjective tasks that are so broad that you really could never collect a single dataset or even 10 or 20 datasets that would parameterize all the possible real use cases. We work a little bit differently, though, in the sense that we have this community of millions of people that come to the website and they're asking basically whatever they want. And so it encompasses a huge diversity of tasks that we then collect human preference about. So we ask them which ones they like better and then use that massive data to extract analytics. So we're basically inverting the problem from, hey, I'm going to look at this particular narrow task, and instead saying, let's take all the data we have and extract analytics from that in terms of which models are best.

So we haven't commented at all on the valuation, first of all. But in terms of where we're going to put the money, the platform is actually quite expensive to run, and the reason is because we have a lot of users that are coming in using the AIs, and we're hoping to scale the community. I think Wei-Lin should talk more about that. But as the community scales, our costs of running it are also going to scale. So we're just planning for success here in terms of growing the platform and serving our user base.