EP26: Measuring Intelligence in the Wild - Arena and the Future of AI Evaluation

Hello. It's great to see you.

First of all, thank you so much for having me. It's great to be here with you guys. Excited to be on your podcast. I love the academic tone that you guys take and the deep dives you do on all the technical matters. So excited about that. Arena is a company that's dedicated to measuring intelligence. And the core thesis of the company is that if you want to understand real-world performance of AI, then you need to subject it to real users and try to understand whether AI is working for them. It's not enough to collect a static benchmark and to annotate that offline, but rather you need to understand the in-the-wild behavior when you put it in the hands of millions and millions of people worldwide and what happens. As anybody that has built software knows, users are unpredictable and their needs and preferences are very complex. They're too complex to model without actually giving them a voice, which is what we try to do at Arena. Our platform now, if you go to arena.ai, serves tens of millions of users every month. We are getting hundreds of millions of conversations that we then use to create analytics and leaderboards that drive the north star of the industry. We're the gold standard AI benchmark on real-world usage. Our leaderboards are referenced with essentially every model release. And we take that responsibility very seriously as the guardians of the field.

So your question, to be clear, is about sort of the coverage of use cases?

In other words, are there enough users in every single different pocket of in-the-wild human usage to reflect that? And I would say there's never enough. We always can work harder to both attract different types of users and also report results for them. If you think about it, every single person on Earth is a little bit different. And we all use generative AI slightly differently. And so if you really want to provide the most actionable insights to everyone on Earth, ultimately what you need is a personalized leaderboard that's just for them. Now, there's challenges in doing that, methodological ones, which I'm excited by. You know, my background is in statistics. I did my PhD in theoretical machine learning and statistics. So I care about that methodology and in fact it's one of the things that attracts me to this problem. But it is an open problem how to do that. That said, we work on it every day at Arena, both in terms of attracting users from all different types of use cases and in terms of analyzing the data. These days our users, 28% of them are software engineers. Another 20-plus percent of them are creative professionals like marketers and designers. And another 15-20% of them are researchers and students. So it's a pretty high-quality subset of the human usage patterns, but of course doesn't reflect everything.

Yeah. So maybe I should just talk a little bit about style control. Style control just — okay, zooming out. Human preference is very complicated. The hardest part about human preferences is trying to understand why people vote the way they vote. And ultimately what we do with those human preferences is we use them in a downstream leaderboard calculation that computes so-called Bradley-Terry coefficients that tell you about the strength of each model. So what do I mean by strength? We use in Arena a tried-and-true statistical methodology, this Bradley-Terry methodology, which presumes that every model has a strength which is a real number that tells you how likely it is to win in a battle against another model which has its own strength, and the difference in those real numbers, appropriately scaled, is the win rate between those two models. So it assumes there exists these strengths and then it tries to estimate them with a particular form which looks like a logistic regression, and that's how you get Elo coefficients, which we call the Arena Score on Arena. Now, because of the complexities of human preference, when you calculate that score, you are calculating an average. You're calculating an average over all the reasons why people vote and all the different people on the leaderboard and all the different prompts, the whole prompt distribution, so on and so forth. And one of the questions that we've been getting is, are people voting because of superficial factors — factors that people consider, quote-unquote, superficial, like style, communication ability, response length — or are they voting because the answer is better on a substantive basis or more factual and so on? And of course, human preference is some mix of the two. It's neither A nor B. It's some averaging. And so what we try to do is we try to fit a statistical model. We try to say, hey, let's extract all the components of style, whether that's the difference in response length between the models, the number of markdown headers in the responses, the sentiment and the fanciness of the models, and we turn those into features that we can then regress away to say what's the predictive power of those features. Well, okay, if those predict the vote, let's subtract that part out and regress on the residual essentially. And after you've done that, you're getting something that, you know, we believe more faithfully reflects the non-style strength of the models. So, when you turn the toggle, that's what it does. It just runs a slightly different regression. Now there's a little bit of a value judgment about whether style is good or bad that I think we should not be making as a platform because, you know, I know people complain about style, but the reality is the way models communicate with humans is very important and the way humans understand responses and how useful they are to people is highly dependent on the communication ability of the models. I think it's a real benefit of Arena that we're able to measure this signal.

That's correct.

Yeah, I completely agree. It's a methodological challenge how to do this optimally. I certainly don't think the method that we've implemented is perfect or even near perfect, and we think about how to improve it all the time. In fact, we have an active work stream on it. Not to say that we'll be able to get it right the next time, but you know, I think incremental improvements go a long way. The two major challenges, as I think you brought up, are number one, how do you represent style as a feature, and two, what the right paradigm is for doing the causal inference that's necessary in order to actually disentangle these components, which of course we're currently doing a regression analysis, which is essentially, except under strong modeling assumptions, observational regression. And so those are two questions that we think about a lot, and it'd be great for the — I think it's also a fruitful question for the research community actually to try to contribute to these.

That's a great question. I've wanted to create a slop leaderboard for a while. I've been blocked. I think...

Yeah. No, the team has opinions on it, but I think it'd be at least very, very funny. Maybe for April Fools. I would like to measure those factors. It's hard to define what slop is. If you give me a great definition, it might actually, you know, make its way into the leaderboard. So...

That's great, you guys. That would be awesome. An Ig Nobel Prize is like such a career goal. That's incredible.

Well, what we found is that a lot of our users actually do have quite a bit of knowledge, and because what's happening is that they're asking questions about things that they care about. For example, I go on Arena, I ask a lot of math questions. If I showed you my history, it's a ton of proofs. And when I get responses from models, I look through them actually quite carefully, and they're useful to me, but also I'm competent to be able to judge the answer. And there's a ton of users like that on Arena. There's also sometimes noise votes for sure. And I think about our challenge as being how do we separate out the signal from the noise? How do we identify the highest-quality users that are giving us high-quality votes? How do we incentivize people to give us high-quality votes? For example, through private or personal leaderboards that sort of curate so that they are only voting on things where, you know, they actually have an opinion and they want to shape their algorithm. So those are the types of incentives that I think could make the system work.

Yes, today we have a few tools on the platform. Of course, there's search, but there's also Code Arena, which you can go to, and it's essentially a vibe coding arena. And models there can make plans, they can execute multi-step tool calls, and it's very much an agentic evaluation.

I think it'd be great to define intermediate types of feedback signals and more types of feedback signals. That's something that we're thinking about a lot going into this year, is how do we extract, you know, more than preference data — final preference data, which is of course I think one of the most valuable signals is final preference data. But to your point, it is definitely not the only signal. I mean, even something as simple as are the models failing to call tools correctly, and are they generating code that compiles? Like we have a lot of information like that that we should probably be releasing as part of our leaderboards.

Well, the accuracy of tool calling has definitely gone up over time. Yeah, I mean none of these top models are doing 50% tool-calling accuracy anymore. They're all way above that, like, you know, Gemini and Opus and so on. So for sure there's been big improvements on that front. And what we see from Code Arena when tool calls fail, of course, is that the final websites don't compile, and then the user gets mad and they vote for the other one because they're like, hey, I hate this, the model just didn't do anything, or like it made a plan, it was a good plan, and it wasn't able to execute. And so yeah, we see that all the time in our data. It's definitely reflected. That's one of the main signals that goes into, of course, the Code Arena leaderboard because it affects the final compilation so egregiously. There's also retry mechanisms that models have built in, which is quite interesting to see actually, because they'll make the plan and then you'll see along the way that they fail, and then they'll be like, hey, okay, I'm going to like roll back a step and then I'm going to retry it with something different, and then they can like move their way around obstacles. It's quite an interesting behavior. It's very much agentic.

4%?

So I'm embarrassed. I'm too proud to admit that I've ceded control of my coding to Claude, but it's true. I've done it for a lot of things. Although, you know what I find ends up happening is that it creates a ton of slop. I'm sorry. It does. Like I ask it, 'Hey, make me a repo that does XYZ blah blah blah.' And then I go through a few steps and suddenly there's like, you know, 60% of the files in the repo are unused. I'm like, what the hell is all this crap that's sitting on my computer?

It's true, but at the same time when I'm doing things that I need to verify, because what I'm trying to do oftentimes is make scientific figures, you know, I'm trying to like plot my stupid experiments where I'm like resampling noise a thousand times so that I can, you know, check whether my coverage is correct or whether my, you know, statistical method has the right type one error, or I can do my bootstrap replicates or whatever. And what'll happen is that there's like 10 different files that I need to sift through that have different variations of methods which all kind of look right, and it's just like — it's not efficient enough for me to be like, okay, let's check it. So I end up having to tear it apart and spend a long time verifying. I do think it accelerates me, but it is its own problem.

Totally.