How to evaluate LLMs | the statistics behind Arena's rankings

Hey everybody, I'm Anastasios, co-founder and CEO of Arena. I'm here to tell you guys a little bit about the platform, about the methodology behind it, how we calculate the leaderboard, the kind of data and data science that we do here, just to share with you a little bit more about the science of how we do things.

Just to tell you a little about the origins of the platform, Arena started as an academic project at UC Berkeley. None of us ever thought of it really ever becoming a company. It was just supposed to be a paper, and it was a paper being driven not just by my co-founders Wayin and Yan and myself, but also by an awesome group of people at Berkeley, you know, a lot of undergrads, PhD students, professors, you know, Joey, Lisa, you know, Arian and Evan who still work with us, Tim, you know, and a whole group here. I won't name everyone, but you know who you are, as well as at CMU. And it's been an incredible journey starting out at Berkeley where if you look, these are some messages that are in the Berkeley Slack, you know, same day UC Berkeley wins five Nobel prizes and we have a dead rat in the ceiling, okay, so things are a little bit, you know, earthy crunchy. And I'm going to tell you about the work that we did as students there as well as some of the work that we've continued to do now that Arena's become a company.

So let's first talk motivations of Arena. Why do we even have this platform? Well, you know, in the past three years, it's been kind of a crazy ascent of AI. And with systems like ChatGPT, we have humans that are interacting with AI every day. Billions of weekly users of AI out there in the world. And because of that, it has become ever more urgent to build human evaluations that go beyond benchmarks, benchmarks being sort of these static data sets that you go out and collect in sort of relatively narrow regimes. How do we try to understand how to assess people's subjective experiences when using AI and their utility to humans?

So, at the time when Arena was first developed at SkyLab at Berkeley, there was a question as to how do we even compare models beyond the benchmarks? You know, you go and you test them on a multiple choice exam. And what you'll see is that, you know, one will perform better on the exam, but then you talk to them and one is dumb and the other's smart and those two might not correspond. So the actual in-the-wild deployment performance is really what you seek to measure. As an example, you know, no benchmark can tell you how to write the perfect love poem, for example. It's just about being able to look at it as a human and as a person and assess how you feel when you look at it. So that's just one example of how human evaluations are so critical. If we want models to be good at things like creative writing, we really need humans in the loop to do that. And it goes far beyond that, too.

So here I'm going to show you one feature that you might not have seen even if you're a user of the Arena platform, which is our coding arena. So this is Code Arena. You can look at it and it'll say build a working clone of Notion. Okay. And I'll press enter. What you'll see is that two agents, two coding agents start generating and you can see them in two tabs, option A and option B, and they start calling tools, editing files, basically they're in the back end building a whole file system and creating the website, a React website from scratch. And then you'll see option A, it compiles and I say I'm typing in Notion blah blah blah. I can create headers, to-do lists, and I'll press enter. It's not perfect, so it's not actually adding items when I press return on the to-do list. Option B actually looks a lot more like Notion. I'm playing with it, but it's typing backwards. Okay, big issue. But what I'm going to do is I'm going to vote for option B because I think it's closer to the actual final prototype that I want. And so you'll see in a second I'm going to vote in this video for B is better. And that was Claude Opus. It beat Qwen Coder 3. And then I'm going to say it's typing backwards. Can you fix that? We're going to enter another turn. Two models are going to be sampled and then, you know, option A didn't quite work and option B looks like it's at least almost functional and so I'm going to pick it. And this way we're iterating with human in the loop on the Notion experience that I like better. And in that one Gemini 3 Flash beat GPT-5.1. This is the way the platform works. That's the way Code Arena works. And we've got arenas across basically all different modalities. That means like text, image generation, image editing, video editing, video generation, different categories, professional categories like law, medicine, you know, definitely coding is one of them, and so on.

Okay, so that's a little bit about the platform, that's Code Arena. And so since launch, to just get a sense of the platform, this platform now has millions of users every month, tens of millions even, depending on how you calculate the number. Hundreds of millions of prompts. We probably have half a billion conversations that have happened over the platform at least over its lifetime. Millions of monthly votes, 700-plus models that have been tested, over 150 countries, 125 languages. And the data that we get from the organic usage of the arena.ai platform allows us to have a constant fresh stream of data with which to calculate leaderboards. So here's for example our coding leaderboard. This is a little bit old so don't look at exactly, you know, what's happening here and who knows when we'll actually post the video. But what's happening here is that, you know, the number one model is the model that basically is most likely to win in one of these pairwise comparison battles. And I'll tell you a little bit more about how we calculate that score from a methodological perspective in a second. But this is kind of the critical column score and it's the result basically of a regression that's run on the vote data that we collect. And we collect new vote data every day. So they constantly update and we construct confidence intervals on them and so on. That's just one category, but there's many, many categories on Arena that allow you to assess the real-world performance on our user base. Of course, it's reflective of our user base and, you know, there's limitations to that in the sense that our user base is not necessarily reflective of every possible user base or use case, but that's why the scale of the user base matters because it allows us to kind of cover a bunch of different domains and then give granular insights to the world.

We also do pre-release testing. We work with the major model labs, in fact with every model provider, to give them the option of testing their model under a code name on Arena. So many of you already know this, but they can submit their models. We'll evaluate them for the provider to give them sort of information about how well their model is doing on the distribution of real-world use. And, you know, you guys on the community probably see the code names and have a lot of fun voting for them or against them. And then the model provider might at some point say, "Hey, you know, we're ready to release a model." And then they'll uncloak it. And the world gets to see what model that is. We've done this with many, many of the top models, Gemini, Grok, you know, o1, GPT-4o. Many of the models that you guys know and love sort of were first released to the public actually on Arena.

A little bit about the demographics of the Arena population. This is the prompt distribution that goes into our leaderboard. It has a little bit of filtering to make sure there's no spam or attackers and so on. But it represents the votes that actually make it all the way to the leaderboard. And what you'll see is that almost 28% of the prompts are in software and IT. Almost 25% of them are writing, literature, and language. 17% in the sciences, life or physical sciences. 16% entertainment, sports, and media. 16% business, management, finance. 10% math. 6% legal. 5% medicine. Only 6% of them are not in an economically valuable industry as far as we can tell. We also use like the Bureau of Labor Statistics occupational tags. I can link you guys to those in order to sort of define these categories. And so what does this mean? You know, obviously this is not a typical population, but the Arena population tends to skew towards pretty smart people, experts, pro users, and that's what you'll see in this categorization.

Okay, that's not to say, you know, everything they're asking is necessarily an expert prompt. We do have like some subset of prompts that we've identified as expert prompts, but in terms of the demographics, as evidenced by the prompts, it's pretty much pro users.

Now that I've told you a little bit about the platform, I want to transition into a section of our conversation that has to do with methodology. So what is the purpose of building such a live platform? What are the statistical and mathematical considerations that come into play when we're constructing the leaderboard? Basically, what are the benefits and drawbacks? How do we accentuate the benefits and take advantage of them? How do we mitigate the drawbacks from a methodological perspective? Okay. So I'm going to start with a discussion of the difference between static benchmarks and live benchmarks.

So first distinction that I'm going to make: what is a static benchmark? A static benchmark is a test that you, you know, as I'm defining it now, it's a test that you go and collect by let's say compiling a fixed set of questions in a domain and then assessing a model's performance by making them take that test, basically making them answer that fixed set of questions and grading them against a rubric. That is what a static benchmark is. It's static because the questions don't change. And so one thing that you'll see with static benchmarks, just like you would for example with a student, is that because the questions are fixed, if a model sees those questions once, it's more likely to do well on them the second time that it sees them. And that's a phenomenon that for various reasons we call overfitting. So it might be that it saw the questions and it saw then the answers and then by studying the answers it can go back and do well on the questions. So really the performance on that test the second time around is not necessarily reflective of the model's performance on the domain from which the test is sort of drawn from. That's called overfitting. There's various ways of... overfitting is quite a subtle concept, kind of a confusing term because there's many ways of slicing the pie. Maybe the model hasn't even seen the answers, but it's by seeing the questions that it can go out and sort of find similar question-answer pairs and then it overfits again to those specific questions because it's trained on similar question-answer pairs. That's another phenomenon. You know, that's another form that overfitting can take.

I'm going to contrast that with a live benchmark. A live benchmark is one in which questions are continuously randomly sampled from the testing domain. That means that you can sort of conceptualize those questions as being IID drawn from a distribution of questions, in which case overfitting is impossible. And the reason is because the questions are always new. So in order to do well on a test that's continuously generated, you actually need to be good at the domain. Now you can study for the domain. You can be good at that domain but not other domains. But in any case the purpose of that test, if you think about it as assessing the domain-specific performance, then there is no overfitting capability there. So that's one benefit of live benchmarks is that they are immune to overfitting.

Another distinction between static and live benchmarks is the issue of contamination. So what happens oftentimes, you'll see this and there's arguments about it online, is that with static benchmarks you take the benchmark, you release it, and then even if the model trainer may not have intended it, those questions will somehow find their way on the internet and then make their way into the training data of the model, which again leads to the overfitting issue because the model has seen the data, basically the questions and/or the answers online, and so it's prepared for them for the test, and then the test performance is not reflective of reality.

Live benchmarks, because the questions are again IID new samples to the extent they are, so there's no contamination possible because there's no overfitting issue.

Now on the other hand, live benchmarks have certain drawbacks that we need to understand and mitigate. Static benchmarks are particularly good at targeting. If you want to look at a very, very narrow subpopulation, what you can do is for example pay people to go out there and collect, you know, I'm going to find radiologists that are specialists in heart blah blah blah. I'm going to find a thousand of them. I'm going to pay them 200 bucks each to generate a question for me and then I get a really targeted evaluation in that domain. Okay, it's kind of clear how to do that with a static evaluation. With a live benchmark, especially one that's based on organic usage like Arena, what's happening is that you're getting an organic stream of prompts from who knows, like all sorts of people. I mean I showed you our demographics, a lot of them are experts, but nonetheless we need to kind of mine that data to understand the distribution of the prompts because it's organic, it's unstructured, okay.

Another sort of convenient aspect of static benchmarks is that you can create rubrics. Oftentimes, because you're paying people to generate the static benchmarks, you can ask them like a really detailed sequence of questions that's like, okay, you know, how do you feel about the creativity of the response and the originality of the response, or is it correct? Is it factual? You know, can you cross-reference this against your textbooks? And you know, a grader can spend hours and hours, even on a specific question, because they've been paid to do that.

Live benchmarks that are based on organic usage are a little different because it's all natural data. So users, you kind of have to meet the users where they are and gather usage data in a way that, you know, they're actually going to basically convert on that. You know, in the case of Arena, for example, they have to vote. If they don't vote, then we don't have signals, or you have to be able to draw implicit signals that allow you to do evaluations without necessarily the voting mechanism. And because you're not giving them a rubric, their responses are going to reflect their sort of personal opinions and their personal perspective, which is a blessing and a curse. On one hand, you get to measure people's personal perspectives and opinions, which is super valuable. On the other hand, you need to probably understand how to control that and slice different subpopulations of users that might have different preferences. For example, people that care about super concise responses versus people that want lengthier or more detailed responses.

And then another distinction that I'll make is paid annotators versus organic, you know, anonymous users. In our case, they have login now, so it's not exactly anonymous, but you know, paid annotators are a particular subset of the population, the people that you can pay in order to generate benchmarks for you, versus live benchmarks generated by, you know, whoever is coming to arena.ai. That's of course, these are of course two different populations and so they're going to be measuring slightly different behaviors.

What I'm going to talk to you guys about, in addition to just describing the generic leaderboard methodology, is how to approach from a methodological perspective, and maybe my particular methodological perspective. There's many people who have done research in this area about how to deal with, for example, targeting, how to take the organic prompt distribution and then get better targeted evaluations, the sort of natural usage data, how do we get more granular information about why people vote the way they do and how they vote. And then how to do more personalization around, you know, people's individual preferences so that we understand who they are as opposed to them just being, you know, an IID user, which is never the case.

So I'm first going to address this problem about extracting analytics from usage data. So let's just try to get one level deeper in sort of the academic topic to try to understand what's hard about this problem. What's tough is that the goal is very lofty. It's to try to basically achieve maximum granularity about users' preferences and about the utility of these models to real people. But what we have is this natural stream of conversations with one bit of information. So we have this high-dimensional covariate which is the sort of conversation that's happened and we only have one bit of information which is model A, model B, at least in the case of voting feedback. And so how are we supposed to use that super coarse information where we have one bit of information per conversation to answer questions like which models are best for me, or which models are safe, or for what queries is my model strong or weak, or why do people vote for a model? Well, we really need new methodology. So it needs regular, solid, and interpretable statistics and machine learning in order to help us address these questions.

And so before we get into the sort of deeper, more granular personalized leaderboard questions, I'm first going to describe how we do it overall. So how do we calculate the leaderboards that you see if you go to arena.ai/leaderboard? And to answer this question, I'm going to go into a type of log-linear modeling often called the Bradley-Terry model or the Bradley-Terry-Luce model, which was introduced, you know, quite a long time ago, decades ago, in the seminal paper by Bradley and Terry. Here's the description of the mathematical setting. The data that we're going to get is battles i = 1 through n. So this notation means like the bracket n is going to be like the set 1 through n.

And then for each i I'm going to observe models a_i and b_i and I'm going to observe a feedback signal y_i which is zero if I vote for model A and one if I vote for model B. That means for every battle, think about it as an IID sequence of data points. My observation model is, you know, at index i = 1 I'm going to observe model A which is let's say ChatGPT and model B which is let's say Claude 4.6. And then if Claude wins I'm going to give a one and if ChatGPT wins I'm going to give a zero. That's the definition of these random variables.

So what Bradley and Terry found is that there's a simple way of modeling essentially the probability that a model wins in a battle through a linear model, a generalized linear model, and the modeling strategy is basically I'm going to take y_i and assume that it is drawn Bernoulli from a function, like from a mean that is sigmoid of theta_b minus theta_a. Now what is theta? I'm going to draw it out here. Theta is going to be a vector of length M. It's in R^M where M is the number of models. And each entry in the vector, so theta_1, theta_2, blah blah blah through theta_M, is going to be the strength, the strength of the jth model. So the jth entry corresponds to the jth model. So if I think about Claude 4.6, Claude Opus 4.6 as being the fourth model in the list, you know, this being, not the list, the list is not in any particular order a priori, but if you think about Opus 4.6 as being the fourth one in the list, theta_sub_4, this would be like strength of Opus 4.6. And the higher it is, the better, the more likely it is to win a battle.

Now in the statistical model basically theta is a construction that we've created to signify the strength. And the way that it works is if you assume that these strengths actually exist, then what we do is we model the win rate of model B versus model A as sigmoid of theta_B minus theta_A. So the difference in strengths corresponds to the likelihood that, you know, through the sigmoid transform, the probability that you see a one versus a zero. Okay, so that's the statistical model. That's the Bradley-Terry model.

And long story short, the Bradley-Terry model can be re-expressed as a generalized linear model which allows it to be efficiently estimated. In fact, it can be expressed as a logistic regression. And the way that this works is, and I'm going to point this out here, if you take sigma of theta_b minus theta_a, what you can do is you can rephrase that as sigma of x_i transpose theta for an x_i which is a particular structured covariate which is basically zeros everywhere except for it's plus one on the index of model B and otherwise that's, you know, in other words that's b_i, and minus one on model A on a_i, okay. So that's the covariate. The x transpose theta becomes, if you basically, if you take x transpose theta, you're taking the, you know, you can take the dot product, turn it into a sum, and you're plucking out the b_i component and the a_i component, subtracting them from each other. That's how this works because it's zero everywhere.

And because you've done that now you've said, oh, x transpose theta, that looks like a linear model, and indeed it's just a logistic regression. You can train it that way. So you can just run a logistic regression solver, you can calculate this leaderboard, and that's exactly what we do. So if you look at our arena-elo package on our GitHub, our team has open-sourced the methodology for the leaderboard and efficient calculations of this estimator, but at the end of the day it's just the empirical risk minimizer of this loss, that's all it is. And it's equivalent to a logistic. And because it has such a simple form, you can basically calculate confidence intervals however you want with any standard method. That means you can use a bootstrap. We were doing that for a while. Now we've transitioned to using a limit theorem because there's easily available CLTs for this and it works great.

So now let's go try to think about how we go beyond this marginal leaderboard because the downside of this estimator... So plus side, it's super efficient, it's easy to calculate confidence intervals, it's kind of clear what it's estimating. On the downside, because this argument happens over a sum that treats every data point as being weighted equally, what happens is that it's just calculating an average leaderboard over everybody. But what if I want to do something that's more granular? What if I want to say which leaderboard is best for coders or for mathematicians or for different languages and so on? Well, I can't use this expression for that because it's mixing over all those, you know, different subcategory leaderboards.

So how do we go deeper and deeper into that? Well, one idea, and this might be familiar to some of you guys, is the old Chatbot Arena leaderboard from back when we were in the Berkeley days. Even back then, we introduced categories. So I'm going to, you know, if you clicked at the time, and this is now on arena.ai such leaderboard in the category list, what you can see is overall, math, instruction following, multi-turn, coding, hard prompts, English, Chinese, French, German, Russian, Spanish, all these languages. And we have even more nowadays on arena.ai.

The way that you can calculate this is pretty simple. All you do is substratify. All you do is you take the sum from earlier and you say, "Okay, I'm only going to calculate the leaderboard over prompts that are in a certain category." And then it's just, you know, basically the same arguments hold in that subpopulation that it converges to the right leaderboard.

Now this is great when the categories are big. In other words, when there's a lot of data in each category, but there is an issue which is variance. If the categories are very small, let's say I'm looking for a very particular, a rare language in a very small population where we might only be getting, you know, hundreds of data points every month. Well, you're going to get high variance, which means that the leaderboard, you know, the ranking of the leaderboard is going to be kind of unstable and it won't converge for quite a while. And the real issue with this is that because of that estimation error induced by the, you know, fragmentation of the data, the smaller the category is, the harder it is to get a solid estimate. Okay, so it's a fundamental trade-off. The less data you have, the more variance that your estimator has. And so how are you supposed to solve this? You know, what if I wanted to answer questions basically that are much more granular than a category would allow? What if I wanted to do adaptive rankings, data-driven evaluations that tell me even on a personal basis which model is best for me, for my use case, for people like me, or even for my specific prompt?

It sounds kind of like an impossible problem because what happens is that I'm asking you to do something like take a prompt that you've never seen before and generate a leaderboard that's specific to that prompt. But it turns out that this is, you know, although not possible in general, we can find ways of using our massive dataset that we've collected in order to essentially train models that output these leaderboards. Okay. And this is going to be, I'm now talking about work that we did with a group of people at Arena and at Berkeley, including, you know, Evan, Connor, Joseph, Wayin, Yan and I, Tim, and many others. Okay. So this is called prompt-to-leaderboard.

The core idea of this methodology is that I'm going to parameterize the Bradley-Terry coefficients from earlier, the Bradley-Terry-Luce coefficients, as a language model. So I'm going to show you a way that I can train a language model specifically to output a leaderboard. Okay, I just mean that it takes the prompt as input and then produces the leaderboard as an output, such that when I get a new prompt, I can just pass it through and at inference time I can generate leaderboards on the fly.

And so again, this problem sounds like an impossible problem because how are you supposed to supervise this thing? You can't supervise it with leaderboards because I don't have ground truth leaderboards. I only have pairwise preference data. So how am I supposed to do this? Well, here's the core idea. What I'm going to do is I'm going to parameterize the language model by adding a layer, an M-dimensional layer at the end, theta_p. So this is going to be for every prompt p. So I take the prompt p, I pass it through the language model, and then I'm going to produce theta of p, and that theta of p is going to be in R^M. So it's going to have the same shape as a leaderboard even though it might be initialized to random noise at the beginning. I'm going to just, you know, pretend it's a leaderboard for now.

And then what I'm going to do is I'm going to plug it into the logistic regression loss from earlier. So I'm going to say this being the binary cross-entropy loss. I'm going to say I'm going to take the loss of sigma of theta transpose x of y where theta depends on p. And where theta depends on p is because I passed p through the network in order to get that. Okay, so in other words, I'm basically just going to pretend the leaderboard is actually correct even though I know it's not. It's sort of initialized randomly and it's the output of this neural net. And then I'm going to plug it into my loss function and measure how well it's doing.

Of course at the beginning that's going to do nothing. It's going to just give you a bad loss. But the insight here is that this entire loss function is differentiable with respect to the parameters of this network. And the fact that it's differentiable means that I can backprop through the loss. I can backprop through the sigmoid. I can backprop through theta, through x, through theta transpose x, all the way to the parameters of theta. Okay. So that's great. That means that all we have to do is just train this whole thing with backpropagation. I'm going to say I'm going to construct my empirical risk R-hat of phi. I'm going to consider theta of phi which is the leaderboard theta that's parameterized by parameters phi, and then I'm going to optimize this empirical risk by just like running gradient descent standard way. And turns out that this works and it's super good and theta of p then becomes a prompt-specific leaderboard and the vote prediction capability of this model improves with data.

Basically, there's a scaling law that comes with this that allows us to get more and more granular evaluations that are more and more personalized to the individual and to the prompt the more training data points that we've seen. So, this is a plot that we have in the paper. And what you'll see is that this line here is the constant. It's like the best possible constant predictor of a vote. And remember, it's like randomized and blinded which models are shown on the left and on the right. So, the best constant predictor is 0.5. It's like 50-50 whether you vote. And when you look at the validation loss of 0.5 on an independent validation set, that's like whatever, 1.38.

The green line is the marginal regression that you get when you just calculate the leaderboard overall in terms of loss. So you can see it doesn't really improve with data much because the coefficients converge super fast, at least at this scale of training data points from the hundreds of thousands to millions. And then this here is an increasing ladder of model size. So with a 135 million parameter model, 360 million parameter model, 0.5B, 1.5B, 3B, 7B, the bigger your model gets, the better the performance is getting. And the more data you get, the more each of these models benefits. So it's kind of incredible. If you look at the end performance, the best 7B model is giving you way more lift in terms of the benefit on the validation loss. Like the gap between the overall leaderboard and the prompt leaderboard method is like three or four times bigger than the gap between doing nothing and calculating a leaderboard. That's huge. I mean it was unexpected to me when we first wrote the paper. And you know, you look at plots of the validation loss of the mean squared error and they kind of follow the same trend.

Now the other thing that you can do when you have a system like prompt-to-leaderboard is that you can then, by virtue of the fact that it is a regression and you can interpret these things probabilistically as regression coefficients, you can start doing optimal, like optimization problems on top of it. So I can start solving problems like, you know, I'm going to construct a randomized policy that takes my prompt in and then it outputs a distribution over models that optimizes the following thing, solves the optimization problem, like the convex problem, which is maximize my routing policy in terms of its performance subject to constraints, for example constraints on cost. And it turns out that can be reformulated as linear programs and it's really easy to do even at runtime.

And when you do this, this is sort of a picture of the Pareto curve. What happens is that the prompt-to-leaderboard model at different cost levels just dominates any individual model that's used inside the router, which means that on a sort of a bang-for-your-buck basis, you're better off, you know, routing to different models depending on your prompt than using any individual model. Okay, so here's on the left, that's overall, and then on the right, that's when we subset the creative prompts. So you can see it still dominates potentially, you know, arguably even more so.

Now I'm going to shift gears here because I've talked a little bit about the methodology for getting more, you know, granular estimates of performance through adaptive leaderboards. Now I'm going to talk a little bit about basically causal inference. Why do people vote the way they do? And that's not to say that we have a complete answer to this, but I'm going to share a little bit about the way that we've approached the problem. And there's a lot more to do. So what we've done so far is we've said, hey, one common question about leaderboard methodology is, well, how do people factor in things like style, length, you know, the amount of markdown headers, how paragraphs are formatted, the sentiment or the sycophancy of the response, and can we come up with a leaderboard estimate that controls for these factors? And so what we've done is created a method that's called style control. It's pretty simple. We're just trying to address this sort of confounding problem that has to do with style and separate out style from substance.

And so what we've done is we have modeled y as being Bernoulli of sigmoid x transpose theta plus z transpose gamma where z is a vector of style features. So it would be something like, you know, the difference in length between response A and response B, the number of headers, the sort of sentiment as judged by an LLM, and so on. So you can collect all these style features and we have several. You can include them in your regression and what you'll get is that the effect of style will sort of be absorbed into these coefficients and that the theta coefficients will be representative more of the model strength.

Well, this isn't exactly true because it's only true if the model's well specified, and then if you want like more rigorous techniques under, you know, the model not being well specified, then you need to start doing interventional experiments and causal inference and so on. All things that we've been working on, but this method is sort of a first cut at it. And what you see is that there's actually a pretty significant effect on the leaderboard in particular. What happens is that, and this is an old leaderboard, so, you know, not reflective of today, but the models that tend to be sort of more concise and not as RLHF'd for human preference tend to go up, and, you know, the ones that are meant to be a bit chattier tend to go down. Not to say that there's a value judgment here as to which one's better. It's important, and we believe at Arena that it's important for models to be able to communicate well with humans and satisfy humans and so on. But this certainly does have an effect, and so it's interesting to see that come out.

In terms of sentiment, it's actually pretty funny to see how widespread the sentiment can be between models that leads to these changes in the leaderboard. So on the left-hand side I have a model with very positive sentiment in a response. So we've asked her the question, which actor had a Doberman Pinscher in it called Kirk? And the assistant says, a very specific and delightful question, and has all these emojis. The answer is James Dean. That's right. Blah blah blah. And on the right-hand side, it's a very negative sentiment. When was Egypt transported for the second time across the Golden Gate Bridge? And the assistant says, Egypt was never transported, not even once, and the Golden Gate Bridge is not wide enough to accommodate blah blah blah. This question is nonsense. I mean, you know, it's so mean that you just feel like not talking to it. You know, I get why people don't vote for this. So yeah, sentiment also has quite a big effect and we've done some studies of sentiment in style control.

Of course, I won't talk about this much, but the question that immediately arises is how about we do prospective studies that ask LLMs to intervene because we have control over the full platform. So we can potentially run interventional studies that allow us to assess the effects of style. And that's certainly, you know, one of the ideas that's on the horizon for us. And if you're interested in working on stuff like that, you know, obviously we're just trying to run here. But, you know, might be something you might reach out to us about and we're happy to work on it together.

So what I hope that you get from this is not only an understanding of the platform and the methodology that we use to construct the leaderboard, but also the fact that these sort of old-school and, you know, rigorous formal ideas from generalized linear models to causal inference to experimental design to learning-to-rank, as well as these contemporary techniques, deep learning, you know, reward modeling and so on, they really are of a lot of use at Arena. Benchmarking is one of these timeless topics that's drawing from basically the entire history of machine learning and statistics in order to do cutting-edge work. And it's actually quite exciting and vibrant as a research area where we are really proud to be doing some of the leading research in this area here at Arena.

We have a great team doing it. So, you know, I don't want to be a shill, but if you are interested in working on this stuff together, you should just reach out because there's room for you here. Thanks so much. Appreciate it. Nice to meet you.