Can We Trust AI Benchmarks?

Arena is a company dedicated towards measuring intelligence. So what we do is we run a benchmarking platform. It's a live evaluation, unlike any other evaluations, where real users, tens of millions of them, are coming to our site, arena.ai, using AI for their daily work tasks, whether they're doing software engineering or legal tasks, medical tasks, writing, and so on. And in so doing, they're giving us feedback that we use to construct a public leaderboard that's shaping the future of the AI industry.

The way we make money is by helping labs and enterprises evaluate models pre-release or AI systems pre-release. So what happens is that during the course of developing a model, before let's say labs want to release it to the public, they might want to try it out on some users, you know, to get the distribution of real world use and see how well they're doing and try to understand its strengths and weaknesses, know how to improve, whether that be, hey, this model's not so good at math, and then they maybe go and collect more math data to try to improve it, or whether that's coding data, trying to understand where maybe it's making errors. Maybe it's making code that doesn't compile. Maybe not solving users' tasks. We help them understand this on the distribution of, you know, our real users through these cloaked models with these code names. And that's a service that we offer to providers to help them.

Major AI labs and also enterprises. Yes.

Okay. You should know though that the public leaderboard, the one that you see on arena.ai/leaderboard, never touches money. Model providers cannot pay to be on the leaderboard. They can't pay to be off the leaderboard. They can't pay to change their score. And all public models are evaluated according to the same open source methodology that you can see on our GitHub. You can go to our Arena Rank repo and it shows you exactly how we take the public's votes and we turn them into scores. This procedure only applies to pre-release or, you know, cloaked models where providers are trying to sort of predict how well they would do if they were to release their model.

Well, you know, I think the neutrality is driven by people's faith in our scientific rigor. We're all former academics. You know, myself, my co-founders Wei-Lin and Ion come from Berkeley where we were running this project as PhD students. Well, Ion was our professor, but Wei-Lin and I were PhD students at the time and just thinking about, hey, how do we do this? Like it was at the time of ChatGPT and nobody really knew how to compare two language models. You know, you would put them on static benchmarks like, you know, an MMLU or something and it'd be like, oh, well, model A is better at taking the test than model B, but then you put them in users' hands, like you try to chat with them and it all, you know, basically fails. It's like you can't talk to this model and it's clearly so dumb. So what we did in order to fix that, and it wasn't just us, it was a large group of students at Berkeley, is create this arena-type system that kind of became, you know, the gold standard way that such chat systems are evaluated and has sort of propelled to this point. We remain very scientifically oriented. We continue to publish papers on our methodology. Again, like with a very open-source, you know, system. And by the way, if you think about our incentives commercially,

trust is at the center of it because if any model provider starts feeling like, hey, these guys are not in the game to be neutral and it's not a neutral playing field, then the whole platform loses everything. So I would say that's the real, that's one of the incentives also keeping us neutral.

Well, let's distinguish between two things. There's the models becoming commoditized and then there's the space fragmenting. So models can become commoditized while the space also fragments, and by that I mean the cost of sort of the margin of inference can be driven to zero. But at the same time what can happen is that different providers and different, you know, models are good at different things and that's kind of the world that we see ourselves going towards. Cost of inference certainly going down. You know, a lot of these companies are talking about driving the cost of intelligence to zero, making sure that you can have like intelligence in your pocket at essentially free cost, too cheap to meter, blah, blah, blah. But also we're seeing that different providers are starting to specialize in different things. So as an example, like the consumer war is kind of being fought right now between ChatGPT and Gemini, but the enterprise war is certainly being, you know, it's not necessarily being led by Anthropic in terms of dollars, but Anthropic is kind of positioning themselves there in the market and growing quite quickly. And so if you believe in a world where many flowers will bloom and there's a huge amount of space for AI to, you know, capture dollars everywhere in the economy, then the natural incentive for these companies will be to sort of tile up the space and fragment. And in that kind of world our business is actually quite valuable because we can solve that fragmentation problem for users. We can help you.