Anastasios N Angelopoulos - Agent Arena: Causal Evaluations of Agents in the Real World

All right, everybody, thank you. I'm honored to be speaking here today. I was a student at Berkeley not too long ago. Actually, Arena started as a student project. It was just—I was a PhD student here. I was working with Mike Jordan and Jitender Malik, and we had this collaboration with Ion Stoica, who, as you guys know, was the first CEO of Databricks and also a professor here, that turned into Arena. All of the model labs started competing on our platform, and now it's become a company, which has grown gangbusters from there. And so some of you might have seen it online. But here, I'm going to tell you guys a little bit about how we evaluate agents. Of course, this is the conference for agents, so it won't be any surprise to you. But we recently released our new agent, Arena, about seven weeks ago, that has led to millions and millions of users online using agents on arena.ai, and us being able to extract organic signals of utility to people for agentic tasks. So with that, a little bit about Arena.

Arena is the evaluation company. The purpose of Arena is to say that static benchmarks and so on—they have a lot of flaws. You can overfit them. And then there's also, often, a distribution shift between what you can collect a data set for and what actually happens in production. Arena works a little bit differently. Our philosophy on evaluations is that there's only one judge that matters, and that's reality. Reality is actually the source of objective truth. It's our way of measuring the post-deployment impacts of models. And it can't be gamed. Because it's actually just—in order to do well in reality, you needed to have real people's jobs, and then be able to complete them successfully. And so we measure intelligence in the real world. And I'll tell you a little bit about the way that we do that.

Long story short, Arena has a consumer application, which is arena.ai, that you guys can go visit today. I'll tell you a little bit about it in a second. But we collect two types of data on Arena that we can turn into benchmarks. I've shown the two up here on this slide. The first type of data, which you might be familiar with Arena for, is pairwise preference data. So that's one prompt, and then two responses, and then the user votes for the better response. Now this is not on the talk track, but I'll tell you, guys, there's something that I call the fundamental principle of Arena, which drives all of this. The fundamental principle of Arena is that the act of providing feedback should give value back to the user. And that's what allows us to collect millions and millions of data points without having to pay people. People are coming to the website and using it for free. And then in doing so, they're giving us valuable feedback that can turn into evaluations. And the way that we give value back to the user, through the voting mechanism, is that when they vote for the response that they like better, they continue with it in the chat. So anybody that doesn't want to screw up their model's context is incentivized to vote their true preference. And so this has led us to have quite a few high-quality votes on the platform.

The second type of data, which you might be unfamiliar with Arena for collecting, is called trace mining data. So here, it's away from the battle mechanism, so it's one prompt, one response. And then there's these long multi-turn conversations happening over days and weeks on the platform, that we can mine for signals. So here, I've put an example of a user that architected a RAG financial brain. You can read this text—architected a RAG financial brain that ingests, cleans, chunks, and embeds finance articles, blah, blah, blah, a full architecture flow chart, and a passing 160-test suite. And this was all done in Persian. So it was an 84-turn conversation over many days, in Persian, with 600-plus tool calls. It's just very, very rich data. And from this, we can mine things like task completion signals, users telling us explicitly through button clicks whether or not they got their job done or not, hallucination rates, steerability—these are all done through automated pipelines—error rates, user frustration, user praise, user negativity, and then report these back to you guys, which is what we do in our leaderboards. So I'm going to try to do this.

So just to give you a sense of scale, by visits, if you look at startups founded since 2020, Arena is actually just top 10 in terms of raw users. A lot of people don't know this. We're bigger than xAI. We're bigger than Hugging Face. We're bigger than Manus. We're bigger than GenSpark. It's 35 million monthly visitors. It's a freaking massive flywheel of all these users. And with that—and 28% of them, by the way, are doing software. And so what this means is that we have the equivalent of a thousand SWE-benches every week of real-world coding data that we can use for evaluation on this platform through our community evaluation model. It's really amazing. This is a little bit more on the data collection, the user base that feeds this data flywheel. You can see this 28% software. It's like 17% sciences, 12% finance, 10% math, 6% legal, 6% medicine, all using these agentic cross-provider workflows on Arena.

I'm going to show you guys a demo of the Agent Arena here. So this is what you might not have seen. This might be new to you, but you can go use it on arena.ai today. And so what you'll see is, here, the user says, "Download Google's Q1 '26 earnings call transcript and create a PowerPoint with key results." And what happens is that we spawn off a computer. The agent can use a computer, and then it'll go search the internet. Oh, let me find the earnings call report. I'm going to download it into my workspace. Then I'm going to use bash to create a PowerPoint. The user can view that. They can scroll through it, blah, blah, blah. Platform's actually gotten much better since we took this video. Then they said, let's keep working with it, and let's turn that into a website and make it more Google-branded. And so, they did that. And now, the model is, again, going to use bash in order to create a website. It makes the earnings call website. And then you can see it here. And then at the end of this whole experience, if they're happy, again, they're incentivized to do this only insofar as they're getting better responses. They can then click at the bottom there, "Yes, my task was successful. I'm going to move on to my next one." And this is the type of feedback that we can collect. Every message the user sends, that's an opportunity for feedback. If they say—and many of us in this room have done this—we've gone to a model and said, "Screw you. I hate you. I hate that response. That sucked." That is a great piece of feedback. It's very high intent. We all know what we mean when we say F-U. And then, sometimes, you'll say, "Good job." Now, a little bit of an aside here, again, like a tangent to the main thread. So you can't believe how much money everyone in this room is spending just saying thank you to Claude, OK? Because what happens is that you'll have a 20-turn conversation, and then someone will be like, "Thank you, good job." Enter. And that costs you $10. Because all that context was in that. So, be careful how you spend money, OK?

OK. Now, what do we actually do with all this data? How do we report these leaderboards? Now, again, some of you who have known the historical Arena might be familiar with the Bradley-Terry model or the Elo mechanism that we introduced to this space. Of course, it's an old statistical methodology. And so this is another example. Now that we don't have pairwise preference battles to do Elos of, how we have brought some old tried-and-true statistics ideas into the foray of AI evaluation. And here, we're doing causal inference. So the idea is very simple. In these agentic threads that you see in the previous video that you saw, you didn't see the model. And so what that allowed us to do as Arena is intervene on the system by randomizing the model, which allows us to then do causal inference. And by causal inference, what I mean is to say that you can think about, let's say, the orchestrator in the agent or the tools of the agent as a treatment. And then try to measure the treatment effect of what would happen if you used one orchestrator versus another. What happens if I use Fable as my orchestrator model, as opposed to using the baseline model, which might be a randomized average of the rest of them. And so that's what our leaderboards actually show. If you go to arena.ai/leaderboard, the default leaderboard that you'll see will be the leaderboard, and it will be a treatment-effect-based leaderboard. That runs basically a large multi-factor A/B test experiment on all the subcomponents of the agent. And what we do is we compile that leaderboard out of a bunch of different signals. One of the signals is that task completion, yes/no, "my task was done or not," that you saw on the previous video. But we also have all these other signals, like praise versus complaint, which is the fraction of positive versus negative feedback that users are giving to the agent. We also have hallucination rates. We're stripping every factual claim, and then using a bunch of search models to go look along the internet to try to see if we can verify that claim or verify that it was incorrect. Steerability, which is a user will say something, the model will respond, and then is the user struggling to do the same thing over and over again, or the model like correctly land the fix after they initially complained. Bash recovery. Is the model able to recover from errors that it makes in bash? Does it make errors in the first place? All of these are very rich signals that you can mine from organic data, and would be surprising. Because, of course, if you go look at any benchmark, models are not making errors at all because they've all overfit to them. But here, in reality, you go to somebody's actual real use case, like many of us have, and it's a very different story. So you can do that, and then you can break these causal inference, these treatment effects out in terms of things like cost per task or latency. You get very rich analysis of performance. So you should take a look at that. Yeah.

So this is a little bit more of an explanation. Every week, we're getting like millions of agentic traces, implicit signals, explicit signals, environment signals. We're turning these into signal-by-signal leaderboards. So this is the causal effect leaderboard as of a few weeks ago for confirmed successes, which is that explicit button click. And you can see there's confidence intervals on these that are determined through limit theorems in a standard way. And it's all a treatment effect that's based on that gray line in the center. And then what we do is then we aggregate these signals—we have a bunch of them, we have 25 now, actually—into a leaderboard via weighted average, which we can do. That's another benefit of the causal methodology, by the way, is that it allows you to combine many, many signals into one coherent leaderboard. And that's what we do, allowing for these—extracting many signals from deep workflows, and then combining them into a useful summary signal of the model's performance. This is the leaderboard as of a little while ago, a few weeks ago, and it's obviously changed a bunch since then. Fable, I think, remains the top on the Agent Arena, but has given it a run for its money. So, yeah, that's a little bit—that's the majority of the presentation here. I'm not sure if we have time for questions, or if we're doing questions from a technical perspective. But if you guys are ever interested in working in this kind of stuff, the intersection between deep mathematical statistics, causal inference, estimation theory, and AI evaluation, trying to push the frontier of how we measure these models' performance, you should absolutely contact us. You can find me on X or Twitter, or whatever you call it, or email our team, [email protected], [email protected]. We'd love to talk more with you. So thanks so much.