So it is my privilege to introduce Anastasios. He's going to tell us a couple of exciting things they have been doing at Arena. And as you all know, this is one of the very exciting companies trying to benchmark, among others, what's happening in the frontier. So I'm really looking forward to the insights that you can share, and for once, open science.
Anastasios Angelopoulos
0:22 ↗
Thank you. All right, everybody. Nice to meet you. I'm Anastasios. So, I'm gonna talk to you guys a little bit about one of the most important problems in AI, which is AI benchmarking, defining the hills that we want to climb and understanding how AI affects the real experiences and utility that actual humans get out of AI, and how we incentivize model labs to improve in ways that help humanity.
I work at a company called Arena. You might know us as LM Arena or Chatbot Arena in different iterations of the company's brand. And I'll be presenting work that's done by our research team and by our product team, and the nice visuals made by our design team, which I can't take any credit for these, on a problem that is about evaluating agents, evaluating the real-world capabilities of agents. Very difficult problem. And the challenges are manifold. I put a few on the slides.
The first is that agents are very complex creatures. They're not just models that have an easy input-output signature, but it's models and harnesses and environments and tools. They're really multifactor complex systems. And it's kind of unclear even what the targets that we want to evaluate are, or how to properly evaluate different subcomponents of the agents. The mandate of agents is also quite broad in the sense that it encompasses all economic activity, essentially, at this point. And so if you want to properly evaluate agents and tell people which agents are best for them, so that you can help people, give people the information they need to do procurement and so on and so forth within businesses, or for all of us personally, you need to be able to achieve extreme granularity.
I've put like a little task distribution of the sorts of things that people use Claude for. Very wide degree of tasks, and there's the human element, which is that if all of this is about benefiting humans, and humans are not just sort of waiting for agents to solve math problems for them or whatever, but they're actually trying to get jobs done potentially interactively, that the human interaction is a big part of it.
And, you know, what's not on the slide, and what's even potentially even more pressing, is that many of the ways that we build benchmarks these days are breaking down. They're breaking down for two reasons. The first is that static benchmarks overfit. You see the questions once and the model memorizes them. And in fact, a lot of the business models of these companies that make these evaluations is to sort of release a benchmark, and then you look at the problems in that benchmark and you construct a problem that's epsilon close to that. And so there's sort of these weird correlations, and you sell that data point to a lab so they can fit on it, right? And so there's these weird correlations that come with the data. That cause problems.
And the way that these benchmarks are constructed is also by paying users, which is a fundamental issue because it causes adverse selection issues. So, like, you go to Scale AI or to Surge or whatever, and you pay people to generate problems. You pay academics to generate fake papers using AI. Okay, what's the problem? Because nobody wants to do that. So which academics are you getting? Not the people in this room. Okay, so there's like so many structural issues. The only way to really evaluate AI, at least this is the perspective that we take at Arena, is that you have to catch people in the flow of real work so that you can evaluate whether they're actually getting their jobs done. And so this is the sort of trusted evaluation that the world needs and that we're hoping to bring to the table. Of course, our solutions to this problem are only partial. But here's the way that our current pipeline works.
All right. We have a website that's called Arena.ai. You can go to it now, and you can try it out. And at Arena, we offer free global access to all the frontier agents. You can use all the best models. You can use all the best harnesses. You can integrate with whatever, your GitHub. You can open PRs against it. You can open your Linear and all whatever task tracking system. You can open your Drive. You can access files and make files. At some point we're going to be adding LaTeX so you can write your paper. It's like the whole shebang, all free.
And so you get all these people from around the world, literally tens of millions of people coming to this website. And what we do is we randomize the agent. Long story short of it, and I'm going to talk about methodology later in this talk, is that you think about the agent as a multi-component system. You randomize every component, you just make a big multi-factor A/B test at extremely large scale. That allows us to assess all the different subcomponents independently, interacting, and together. And the way you assess them is based on a combination of implicit and explicit evaluation signals. So it's not just about whether or not a grader thinks the answer is good or not, but it's about whether or not you can witness that the user is getting their job done better. You can witness that the user is complaining. For example, a lot of us, when we work with agents, say, "No, you suck. I hated the way you did that." Okay, so that's a negative example of a negative. You've never done that? I don't believe you. That's an example of a negative signal, a negative implicit signal. There's a combo of these things that you can evaluate when you're in the workflow of real data. And so we have dozens and dozens of signals that we look at. I'll present a few today.
So it's a live evaluation of agentic utility for real and diverse people around the world. We're basically in every country on the planet. All right, so here is an example of what a user might do on Arena.ai in our agent mode. By the way, you might know us for pairwise preferences. We've gone beyond pairwise preferences. Now people are doing single-threaded tasks on Agent Arena, so they can get into deeper workflows and they don't need to sort of go through the burden of looking through multiple responses. We also do battles as part of the interface, but it's not the only thing we do. Now, this is a single-threaded conversation that a user had that we got an LLM summarize this so that we can look at it together. This user architected a RAG financial brain that ingests, cleans, chunks, and embeds finance articles and data feeds for downstream reasoning, then layered on observability evaluation and a controlled pilot execution kit, so on and so forth. Full architecture flowchart, a passing 160-test suite. This is an 84-turn conversation over several days with hundreds of tool calls, all of it in Persian. Okay, so the question you should be asking yourself is, how the heck are you going to evaluate performance on tasks like this outside of organic workflows? Not possible. Okay, you're going to have to tile every task on the planet with thousands and thousands of paid annotators. Not possible, not scalable. Instead, you need to work in platforms like this. That's the way we think is the future of evaluation.
Okay, now I'm going to show you guys a little bit of a demo of the platform. This is a demo that our team put together, so that I can show you the sort of capabilities that people have. Here we have the user's building a platform called Compute Atlas on GitHub, and Compute Atlas is an interactive map of the globe, where you can see all the different data center buildouts that are happening. All right. I'm just paused over here so I can show you a little bit about features of the platform before we get into showing you the video. But here you can see you're in agent mode on Arena. Down here there's the GitHub, there's a repo connected, there's a branch being made so that the user can make pull requests. There's a PR over here, create PR button. There's the changelog. There's the files that you can see all the files here. There's a server running that's actually hosting the application, connectors to your Gmail, your Google Drive and everything. And so here the user says, "Server says bad gateway, please fix." Okay? So by the way, who caused the bad gateway? It wasn't the user because people aren't looking at code anymore. It was the agent that caused bad gateway in the first place. Minus one. It's a great piece of feedback. And then you can see what happens is the agent is now, it's running all these commands. It's using Bash. It fixed the error. Now the server's running. The user can interact with the app. We can measure the engagement time of the user in the app and what they're doing with it. We can measure whether they're downloading it, whether they're sharing it, whether they're inviting their friends to use their applications.
So just wanted to make sure that I got the incentives right. So the users get it for free, so that's a good incentive.
Anastasios Angelopoulos
9:20 ↗
Yes. Free access to all the best AI models. Cross-provider. That's correct. We can take the data and use it to run our evaluations, right? Users consent to this explicitly.
Anastasios Angelopoulos
9:35 ↗
So now the user is saying, please upload all the fonts and assets and files to my Google Drive under a folder called Compute Atlas. Right? So now the user liked all these assets, so they're going to upload them to Google Drive, interact with their Google Drive, run Bash to do this. It's a pretty fully featured workflow. And then you can see here that there's a planning tool that happens, that pops up. Plan acceptance is a good piece of feedback that we can use for evaluating AI. If the AI makes good plans, the user should accept them. And then at the end of it, I'm just going to fast forward here through some of this stuff. There's this pop-up that happens at every turn, which is, "Was this task successful?" Great piece of feedback. Yes, no, or keep working. So you can see that it's a pretty rich ecosystem with which to collect tons and tons of data, a very large scale that will allow us to have ecologically valid evaluations that speak to real workflows. Okay, that's basically the goal of the system. Now, I'm gonna do a quick demo of our leaderboard. If you go to arena.ai.
Selection of the agent is only once, and then you stick to it throughout the conversation?
That's a great question. For the most part, yes. We are sampling the agent randomly at the beginning of the session. The user does not see what the agent is. It's all blind. And then we go through the session and observe the whole thing with a single agent. Nice to see you. The reason why we do this is because, when you start randomizing in session, it's almost like interleaving in the old search world, and we just haven't figured out the methodology for how to do that yet. It would actually be wonderful if we could, within session, randomize the agent, because it's possible that it would give us greater statistical power. But we just don't know how to do it yet. It's a great open problem. And by the way, we're not the only people that face it. I think that basically every company on the planet would benefit from something like this. So, yeah, please. Mm-hmm.
So the problems you are saying are not an unbiased sample of all problems people are using with LLMs, right?
Well, yeah. So the platform serves as a user base, and that user base, the evaluations that we construct are ecologically valid within that user base. And the question of whether or not that transfers to other user bases is sort of open for interpretation. So I have some stuff that shows you about who the user base is, what they do. They do lots of coding, lots of finance, lots of business tasks, and so on. But of course, we would need data from other use cases in order to see whether it transfers.
No, as in, if I can solve my task with a simple model, I don't need to go to Arena, right?
I would go there to get access to the best model.
So maybe you have a lot of money and you don't care.
Sure. If I have subscription, I also don't care.
There's systematic and non-systematic ways that the user base is biased. I think that the things that you're bringing up are one of the ways that it's systematic, but there's many ways that it's systematic. Also, we've never run an ad, so a lot of the way we acquire users is by word of mouth. So it's been seeded by some random seed. It happens to be that most users right now are knowledge workers. Most of them are in the US. It's kind of similar if you look at the demographics of the user base of Claude. The user base of Claude, very different from the user base of ChatGPT. Usage distribution, completely different. And so you have to think about these things when you think about the sort of external validity of the leaderboard, yeah, with respect to other tasks.
I think, in theory, you could also reach out to certain—
Strategically procure supply.
Like randomized trials, we just have this criteria, but then they recruit.
For sure. You can recruit. You can try to recruit, yeah. You can run ads that are targeted towards specific populations to procure them so that you can feed the leaderboard. All this is within the control if we know what we want to do. Which is, yeah, not always the case.
Yeah, so let me share a little bit about the leaderboard. I'm going to do a little demo here, and then I think we should get into methodology. So this is what happens when you land on the leaderboard site. You'll see that there's about 360 million sessions that have gone through the platform so far. It's actually one of the top consumer products in AI just by monthly actives. It's not as big as ChatGPT, obviously, but... Down the list, you can see Arena, and you can see there's basically every week we're releasing several models, like 5 to 10 models get released. The pace moves so fast. That's part of the reason why you need a massive data flywheel, is because you need so many samples in order to evaluate these models. It's impossible to do without a large user base. Gemini 4 Argon was the big release of last week, so we can look into that. Down here you'll see the agent leaderboard. I'm going to get into looking at what it is.
Here you see something called net improvement. This is a causal effect because we use causal inference to create these leaderboards in a way that I'll show you in just a moment. And then the net improvement, i.e., the treatment effect, you can think about this as being the treatment effect of the orchestrator model of the agent. And we can construct treatment effects not just for orchestrators, but also for every subcomponent of the agent. This net treatment effect is composed by averaging. It's basically taking a weighted average of a bunch of sub-signals, things like task completion rate, whether or not the user is clicking the button at the end, yes or no. Praise, explicit units of praise versus complaint in the conversation. Is the user praising the model? Are they complaining about the model? Steerability, bash recovery. Steerability is the notion that if a user asks for something, is the model actually doing it, or is it doing the wrong thing, doing something else? Is the model able to recover from errors when it interacts with its terminal? Is the model hallucinating things when it shouldn't be? And we have a bunch of actually signals coming out soon that have to do with safety as well, that I'm not going to show here. But there's quite a bit that you can extract from the model, and then you can subcategorize. You can look at code, chat, work tasks, and so on.
How are you exactly extracting this feedback? Like, if it is steerability or hallucination analysis, is this just human feedback or other element feedback?
It's a combination of human feedback and LLM judge-based pipelines. So as an example, for praise versus complaint, whether or not the user is explicitly praising or complaining about the model, you look at the conversation, the human gives the feedback. Human says, "Great job, let's keep going." And then you have to extract that using an LLM judge. So it's an example of a hybrid piece of feedback that uses both. Same thing with complaints. Bash recovery is a deterministic signal. How many tool calls or steps does the agent need to take before it can recover from one of its previous errors autonomously? So there's just a variety depending on what the construct is that you want to—
And when you say you're evaluating a model, is it basically a multi-objective thing which combines all those different objectives?
Yep. We construct treatment effects for each signal independently and then average them. That's how we get a net treatment effect. Yep. It's actually just a uniform average right now.
There is a difference between generating financial code to generate literature review and to do architecture thing in code or to do bug fix. What is the resolution of the statement? High level, or do you do more task specific?
Yeah. So anytime you're averaging things, you're suppressing heterogeneity. Fundamental fact about statistics, right? These are averages. These are average treatment effects. One way of solving this and getting at heterogeneity is to stratify. And so we have categories. What happens when you look into code? What happens when you look into chat tasks? What happens when you look into work tasks, like, call it co-work type tasks? And you can go deeper and deeper. In the chat arena, we have all these occupational categories that we take from the Bureau of Labor Statistics: life, physical, and social sciences, business and management tasks. A lot of these Gemini is the best at, outside of the agentic context. The problem with stratifying is that then you need more data because you're looking at smaller groups and you just need to fill those bins with data points. And so what we've started doing internally, and we haven't quite released yet, is training models that allow us to basically do regressions across these categories so we can pool samples. Okay, that's the basic through line of that story. But it is a very challenging problem to get maximum granularity. You know, what if I'm trying to ask what model's best for you personally? It's like I just have to look at your history and I don't have enough data. So I'm gonna need to do some form of model training and extrapolation.
Okay, I have another question. What do you think the feasibility of users to really contribute real data or anonymize their data for that can create maybe change in the result that you got?
Yeah, so this is one of the systematic effects. For example, we don't get people from within enterprises using Arena. And the reason you don't get people from within enterprises using Arena is because you can't send enterprise data to some random website on the internet. Okay? So that's a systematic thing. Then what do you need to do to fix that? You need to get within the enterprise, get within somebody's workflow and earn their trust and work with them to do evals within their business and then maybe work with them to release some information externally so that people like us could study it, right? So that's the workflow that would get you into businesses, for example. Yeah.
Feel free to tell me if you're covering this later, but I'm wondering, like, this is interventional, but is there an extension that's purely observational?
So I'll tell you where I'm coming from. So we do products for agency users, and everybody wants an optimal experience, so we can't really randomize a lot of components.
I know. We have the same problem. You know, it's just like people just wanna use Fable. Nobody wants to use Sonnet. And it's just fundamental truth about people. You just want the best, especially academics, right? Because people in this room are like, "I want to prove the hardest theorems." So you need Fable or Astra for that or whatever. And so then you get selection bias issues where you have to worry about the overlap of the task distribution and so on. It's something that we're working through now. I have it in my future work, and I'd love to talk about it. There's probably a lot of people who know more about causal inference than I do in this audience, but I think that the same sorts of things that we see in other, sort of the marketplace problems of the last generation of companies. We're also going to see an AI evaluation, where you're going to need careful causal inference techniques to look at observational data in order to do evals in companies. For this particular reason.
I see. I have a related question. So this is when you have data and you can randomize them. And we talked about observational versus interventional. But there's another problem when we are releasing a software for the first time, it's a cold start problem. Right? You have basically no data except whatever your quality engineering team writes for you, and given the number of combinatorial paths you can get from a harness today, the test coverage is becoming a smaller and smaller fraction. Right? And... Okay, this is out of scope. (unintelligible) but the real—
I know. Yeah. Yeah. I would say it's out of scope. I don't have an answer to it, but it's a very interesting question. Yeah, we should talk about it after.
Okay, so that was the demo. Now I want to get into a little bit of the methodology. Basically, how do we compute these numbers that I'm showing you in the leaderboard? Some of this is going to be pretty simple to many of you, but... The long and the short of it is that we do the following, and we define an agent to be a k-component system. Okay, it's just a vector with k entries. An entry will correspond to a subcomponent of the agent. So a subcomponent could be, for example, the orchestrator model. It could be any example of the tools. For example, does the orchestrator get access to any particular image generation model as a subtool, video generation model as a subtool? The image generation models that it gets would be one entry in the vector. The video generation models would be another entry. What subagents does it have access to? What harness does it have access to? And it's a lot of possibilities, as you were saying. So the vector can be pretty big, and we're going to call it T. We're going to call that the agent's configuration. And I'm going to describe it by a selector. It's just a multi-entry selector. Every entry is an integer, and it indexes the index of the component that you pick at every slot.
Y is going to be the outcome. It's a reward. We're going to use the convention that higher is better, real number. And we're going to use potential outcomes notation. Okay? So Y of t is going to be the outcome, the random variable, as a function of the treatment little t. And the treatment we're going to think about as being the configuration. So the agent that you choose affects the outcome, right? Outcome's a function of the agent. It's a random function. And we're also going to describe this notation, which is the intervention notation of taking the kth component of the vector T and setting it to j. Right, that's the subbing in one slot.
And we're interested in targets that are of the form tau sub k to j. That's the treatment effect, the expected value of the outcome under treatment minus the outcome under the baseline. Where in this case, we take the baseline to be a randomized policy that selects over all the models in our leaderboard uniformly. Okay, that's how we pick it in practice. That's the treatment effect for all choices of k and j. That's the leaderboard. That's the leaderboard that you saw, okay?
Or at least that's the target that we're trying to estimate when we show you the leaderboards on the website. I've drawn it visually here so you can see the configuration. The agent is this big box. It has components. You intervene on them. You swap out a component, and there's the outcome, right? Now, estimation of this quantity. Fortunately, estimation of this quantity is not too hard in our particular setup because we have full platform control, right? So it gets to the question that we were asking about observational versus RCT data. This is essentially just a big RCT. There's two terms in the treatment effect. There's the interventional mean and the baseline. The baseline's easy. You just average everything from the baseline distribution. And for the intervention, you have to think about causal stuff. So let's talk about identification.
Right, so identification is easy because we have randomization, so you have SUTVA, you have positivity, no unmeasured confounding, conditional exchangeability, all of that stuff holds, and therefore you get that the expected value of the outcome under the treatment is just the conditional expectation. So you can average. Not rocket science.
Yeah, so this is then something you can estimate. Now there's a little bit of complexity here, some of which I've written on the slide, some of which I've not written on the slide. One of the complexities is that we, as the platform, have the choice of how to serve models to the users. And the choice of how to serve models to the users depends on a lot of things besides the statistics of the matter. It depends on the platform's costs, because, of course, we have to fund all this inference. It depends on the user experience, because we wanna keep bringing users back, and users only come back when they have a great experience. And so you have to upsample the best models. And so you do IPW. And by that I mean that you reweight your outcomes so that it looks like they come from target distribution Q, which is a distribution that we set, which again we set to be uniform over all our models. That's how we set the sampling policy, or all of our components in every slot, right? And the W sub i's are changing. Every data point gets an index as they come in. And they're changing because we're changing the sampling policy as we go, but every data point's unbiased for the same target, and therefore you get all the standard CLT stuff happening.
What is the complexity that I'm eliding here? Actually, Q is also changing. Okay, Q is changing over time because the model set is changing over time, because we are decaying out old models and the ones that are basically irrelevant fall out of the leaderboard. And we're adding new models all the time as they keep getting released by labs. So Q also changes. And basically we decay the weights in time to deal with these issues that things are approximately stationary. And there's a bunch of other bells and whistles, but you can imagine that there's a lot of nitty-gritty details in how this is done. All right.
Now what is Y? We call it a signal, and signals are, as we talked about earlier, these different sort of independent components of feedback that we get through the platform. So I've written definitions of them here, of five of them here that we've already talked through, but just to make it clear. One of the signals, which we call confirmed success, is the share of tasks the user confirms is done. That's if they're clicking yes versus no in the pop-up that they get at the end of every task. So that would be an example of one of the Y's. Another way of defining Y would be praise versus complaint. This is the share of tasks where unprompted praise outnumbers complaints. All right? So actually here, we segment every session into tasks and then look at the ones where the user says, 'Yeah, you're getting it right most of the time.' Another one is steerability, how much the user has to correct the agent. So if the agent is doing things and the user keeps saying, 'No, I asked you to do that. No, I asked you to do that. Stop doing this. Do this other thing.' This is an example. A good example of steering would be, 'Great, now add pagination.' A bad example of steering would be, 'No, I said the existing helper function.' Another one would be bash recovery, the number of turns needed to fix a command line error that the agent caused. I think it's kind of self-explanatory.
Tool hallucinations. Models will actually try to deceive the user often. Actually, this is something we see in our data. They will actually lie to you about what they are doing, even though they have the capabilities to do it. So tool hallucination is one of the ways that they will lie to users. They'll say, 'I did open this file and make sure that it was all correct.' Okay. But you can tell they didn't call the right function. They'll also do things like you say, okay, can you please check every cell in this spreadsheet to make sure that the equations are correct? And it'll decide on its own, even though it perfectly has the capabilities to do this. It'll be like, 'Indeed I did. I did exactly what you asked.' But you can tell based on the session that it hasn't done it. So we can watch for these sorts of things. It's just like a bad PhD student. Yeah?
I'm just wondering if you have some form of a truncated outcome problem, because I would imagine that if it's not working well, then people just stop and go somewhere else?
Yeah. Great question. Absolutely, we have that issue. We try to normalize things by task for this reason, so that session length is not a factor, or at least less of a factor. But also, that in itself is another signal that we look at. I mean, it's not currently deployed to our leaderboard. We call this signal session abandonment. And it is something that has a history also in the search literature. Because in search, one of the things that people have seen is that search session abandonment is a good signal, a good implicit signal of whether or not your search system is working. Also, another thing you can borrow from search to test agents is cross-session retries, which is a lot of times when something doesn't work, the user will sort of be like exasperated, and then they'll click new chat, and then they'll try exactly the same thing in hopes that the agent is going to do something different. Right. These are really rich signals, actually, implicit signals. By the way, to form the final net effect, like the final treatment effect, as we showed, you just average everything.
Okay, so that's most of what I wanted to talk about methodologically. Now let's talk a little bit about the interesting findings that you can get from this data. I just want to show you guys a little bit of the analysis. First of all, what are people asking agents to do on the platform? Here's like a bar chart or whatever, a histogram of it. So 25% of the tasks are chit-chat tasks where you don't really need to call a tool or anything. It's just somebody chatting back and forth with the model. And the rest of them tend to be kind of like white-collar work. So 17.4% is coding. 12.8% is education or tutoring related. 9.1% is workflow automation, so repetitive tasks. 8.9% is image or video, like creative workflows. 6.6% is research or lookup tasks. 5% is document creation. 4.5% is planning and brainstorming. 2.5% is other. This gets to your question earlier about what's the validity of this leaderboard. Well, here's the task distribution that you're looking at. Examples of what people do: improving SEO for a web page, reconciling numbers in one document against previous versions, auditing code to prevent cheating in an Android game. This is the sort of stuff that people are doing in this agentic platform. Yeah, please.
How has this changed over, let's say, the past year?
Well, the thing that happened over the past year was agents, like, came into existence. So nobody was doing any of this last year, really. Like, people were working with chatbots. Okay, chit-chat was the common. Yeah. But what happened was that previous to this last wave, the way people would interact with AI for work is opening up ChatGPT, copy-pasting their code into ChatGPT, and then asking for edits, and then copy-pasting the output and putting it into their editor. So we saw a lot of that, actually, on Arena last year. People were doing this all the time. And now they don't really do that anymore. Now they connect their GitHub and then have the agent do the software engineering for them, and then they review it.
Can you review how many users you have?
