LMArena CEO Anastasios Angelopoulos - Keynote at UC Berkeley AI Hackathon

Hello everyone. You can hear me? Yeah. Well, thanks so much to the organizers for inviting me. It's great to meet all of you. First thing I'd like to say is it smells amazing in here. It's just wonderful. You know, it's the combination of body odor and cologne that you only expect from a hackathon. So, wonderful to meet all of you and exciting, you know, to be speaking to this crowd.

Yeah. First of all, you know, my speech is like the least interesting thing that you guys are going to be seeing today. Congratulations to all of you on an incredible event. Um, and to the organizers for a wonderful hackathon. I know that this is a huge amount of work to put on. So, let's have a hand for them.

Yeah, really incredible work. Getting into it, let me tell you a little bit about me before we go into the content of the talk. So, uh, I'm a bear. Go Bears. Uh I was born in Berkeley then I moved to LA later grew up there and then came back. Um came back for my PhD in machine learning. Uh I fell in love in Berkeley. My uh future wife, my fiance sitting in the audience right here. Uh we met as part of the PhD program. Um and I started my company as a you know quintessential Berkeley company with a group of students and collaborators and professors here. Um so Berkeley has given me a lot. Uh it's an incredible place. I have a lot of love for it. Um and I'm excited to be here speaking.

So now let me tell you a little bit about LMArena, which is the company that I've been working on, you know, as a research project through Berkeley for the past couple of years, um, and now newly as a startup. So I'll show it to you via demo. Uh if you go to lmarena.ai, this is what you're going to see. It's just a chat interface. I'm sure you guys are familiar with these given it's the AI hackathon, right? And if you type something, you can say, 'Hey, what are some good ideas for an AI hackathon? Keep it concise and give me five. We'll see if any of them parallel things that you guys came up with.' Okay, so it says an accessibility tool,

you know, a medical symptom journal, a code review, and then I'll say which one would win the real Berkeley AI hackathon. Okay, we'll see what happens here. Just a quick it's thinking and it says okay the AI powered accessibility tool. So you'll see these are two anonymous LLMs sampled from a pool, assistant A and assistant B. And then for assistant B, which I'm going to choose because it's a little more concise and I like the mental health tool. That was Claude 3.7 at B, Claude 4.0. Now there's three valuable pieces of data there that we used to construct a leaderboard based on this. There's the prompt, there's the two responses, and then there's the actual human vote. And you'll see what we do is we use regressions like Bradley-Terry regressions and so on to turn this into a leaderboard. The number one model right now is Gemini 2.5 Pro. Then you get into o3, the last version of Pro. The number six model is Claude Opus at 1419. The score higher is better. It's basically a pair. It's like a reward model if you guys use those. Um, and then there's also different categories. There's creative writing, there's coding, and so on.

Um, and then we also run certain like controlled regressions on it. So if you want to understand, hey, which model's best for me and I don't care about style, you can remove style control. A style control is a filter that we put on to say, can we disentangle style and substance and only show the style? And you'll see it has a big effect on the leaderboard. Like when you take away the effect of style, Claude goes up quite a bit.

Now it's not just text that we do. It's also image generation, image editing, and other modalities. I'll do a fun one here. So, I'm going to go on the internet and find a picture of the Chancellor Lion. I'm going to put it in there and say, 'Put his tongue out.' And we'll see what happens. We'll see what it generates. It's thinking here. I've sped it up. Oh, nice. Okay. So, on the left, it's wonderful. It's beautiful. Scooby-Doo tongue. Okay. Or taco tongue. Taco tongue is also good. But I'm going to choose the one on the left.

Okay. And then the identities are again revealed. So you can see it's like a blind forced-choice testing platform for the purpose of evaluating AIs. Now another thing because you guys are hackers, um, you guys must be familiar with tools like Replit, things that are going to, you know, allow you to build an app in like, you know, however many seconds instead of coding it yourself. Now this is a Replit arena. We call it WebDev Arena that we built here and it was actually built by I think an alum of this program, Aryan Vichare. Um, they're saying yep, yep, and so it's like it'll pop up two and then you can choose between them. I said make a hackathon submission. This one's a little weird. It's not exactly a submission. It's more like a website. Okay. It's got the prizes. Okay. I can't find my talk in there, but that's all right.

Alex Johnson, CTO of DeepMind. I don't know who that is. Uh, and the buttons don't exactly work. They'll take you back to the top, but it's okay. At least it compiled. So, I'll click right is better. That's where we're at. So, if this tells you anything, your jobs at the moment are safe because neither of these websites or submissions is really good. So, can an AI win Berkeley Hackathon? Probably not.

Now the platform that I just showed you, you know, it's a bunch of AI evaluations across different modalities: text, image, image generation, web development. We also have red teaming. We have in-context coding like a Copilot Arena. And it's been run, it started as an open source project at Berkeley. Um, it started just as a bunch of students, um, working together and after two years it's grown into a pretty large platform. Um, it's got more than a million monthly users. Um, over 150 million conversations have happened on the platform. About one to two million more conversations every week happen and it's a trusted leaderboard for basically all major model providers. So we work with all major model providers to evaluate their models, sometimes pre-release, sometimes at the moment of release. That includes the OpenAIs and Googles and Anthropics, the Mistral, the Metas, the XAIs of the world. And you might have seen LMArena online. It's kind of a, you know, we're active on Twitter and a lot of people are, you know, generating conversation about it.

Um, now the question is how did we get here? You know, how did LMArena become what it is? Um, and how did it grow out of the open source project that it was, a bunch of PhD students working together, into the platform that it is today and, you know, where is it going? And the first-order answer to that question is it started in a hackathon. Okay, so what happened is that some students at Berkeley and at SkyLab were working on this open-source chatbot, Vicuna, um, that was one of the first open-source chat-tuned LLMs. And in the process of that they realized like, hey, how are we going to prove that this LLM is actually any good? How are we going to prove that it's better than, you know, other offerings that are out there or even non-chat-tuned things? And what they realized is that a lot of it is subjective. The LLM is interacting with the human and so you need human evaluations in order to tell what's happening. And so that's the birthplace of LMArena. Um, and you can see that's the first tweet about it on April 24th, 2023, says introducing Chatbot Arena, you know, we've released Vicuna and there's all these other LLMs, come try them in one interface. And, you know, I initially approached the project, you know, I was doing machine learning, I was doing like statistical theory and stuff like that, and so when I saw this project I was like, hey, what a cool opportunity to like develop methodology, new statistical methods in order to try and address the problem of evaluation in the world of LLMs. You know, it was exciting to me and also I liked the people and it was like, hey, I have a little bit of time on my hands. It was referred through my PhD advisor. Why not spend some time doing this like wacky side project? And so I first met with my co-founder Wei and Lianmin Zheng and we filled up three whiteboards filled with, you know, just math problems, statistics problems that, you know, we weren't necessarily developing new things, but how are we going to take all these votes and turn them into a regression? How are we going to construct a leaderboard? How are we going to form the confidence intervals? Um, and you know, two years since now I spend like all my time running this platform, um, along with a wonderful team.

Now a hackathon is not enough. Um, after the hackathon the project went ooh and you know there weren't as many votes. There was some initial interest and then look at the plot, it's like you start in May, you go to July and September and the curve is down, guys. That's not like, this is not the hockey stick that we were, you know, necessarily anticipating. But credit to Wei, you know, he had enormous consistency in making sure that this platform continued developing. At some point in the middle of the summer he said, hey, I want this to, you know, I want to work on this full-time. Um, and he started adding models, improving the platform, tweeting about it, increasing the distribution, you know, every day little improvements. And I, you know, I joined him around that time too. Um, and then, you know, that led to a little bit of growth, growth, growth, growth, growth. Then we were like, hey, let's take it to the next level. And we started living together. So, we moved in. This is Wei's birthday like last year. Um, and you know along the way it wasn't just us, it was a huge group of Berkeley PhD students and undergrads and master's students that were working on this platform as it continues to grow. And you know you can see the spikes there. These often have to do with competition. So we were at the right place, right time. Uh, as you know, maybe it was just ChatGPT at the beginning, but once Meta got in the game and Anthropic got in the game and so on and so forth, interest in comparisons and evaluations skyrocketed. So that's the way, you know, the curve grew and grew and grew and grew. And we also started pre-release testing models. Our community loves seeing new models before anyone else and so they come onto our platform with code names. I don't know if you remember like, you know, 'I'm a little GPT' and that whole debacle, but that happened on our platform and it was amazing. That's the other spike in traffic that you see there. And since last year we've, you know, continued to grow it and improve it.

Um, now another question that you might have is why did LMArena even resonate? You know, I showed you the polished version of the platform, but the early version of the platform that we built was a research prototype. It was a Gradio app and you know, credit to Gradio. Gradio, I'm sure a lot of you guys are way better at interface design than we are. Um, and the Gradio app that we had allowed us from a research perspective to take the platform from just an idea we had in our head directly into code and scale to like millions of users. And then only recently have we transitioned into like Next.js and all these frameworks that you guys are probably using for your hackathon. Um, but why did it resonate? Why did, you know, despite the fact that we're not exactly the world's best consumer product people, why did this turn into, you know, such a phenomenon? Um, and I think that for that you need to zoom out. We are in a new age of software. Software is changing. Unit tests are not enough for us to evaluate our software anymore because the way that our software works, you have humans inputting these super subjective prompts asking you to do things that maybe don't even make sense, they're not precisely defined, and so ultimately at the end of the day, because they're interacting with humans, you need humans at the other end evaluating them. The space of things that you're doing is too broad. It can't be just ImageNet-like classify or MMLU, I'm going to look at multiple-choice question answering. You need to have something that goes beyond and captures the diversity of uses, you know, that LLMs are being subjected to on a day-to-day basis.

So, this leads us into LMArena. You know, with static benchmarks, the way they work is you look at a vertical. I'm going to collect a data set, um, and then like I'm going to test the model on the test that I come up with. But the issue with there's just differences. Okay, these static benchmarks are useful for evaluating these narrow verticals. At the end of the day, live benchmarks may also be useful. And in fact, I would say that these are the benchmarks of the future. Not just ours, but many. Um, so what are the ways that they differ? Static benchmarks, for example, lend themselves to overfitting. If you create one test and you test the LLM over and over and over again on the same test, what ends up happening is it memorizes the answers and then the test doesn't become a good, you know, a good judge of whether or not the LLM is good at the underlying task distribution. Live benchmarks though, like LMArena, are different. We have millions of people coming to the platform giving fresh prompts all the time. And because the data is fresh, that guards against this overfitting because you're constantly getting new questions from the distribution of test questions that you're supposed to be good at.

Now, another issue with static benchmarks that we're starting to see is contamination risk. You know, as soon as you create a static benchmark, if you publish it on the internet, chances are that because the LLMs are trained on the whole internet, that somebody's seeing it and training on it. Um, but with live benchmarks that are created via, you know, people like us going to the website and asking whatever questions are on our mind, there's no contamination based on those, right? Because the questions are always new.

Now, one benefit of static benchmarks is that they're targeted. Whereas live benchmarks are organic, right? You're getting whatever people want to ask. It might be something that you're not even interested in, or it might be something that you didn't know that you were interested in, or that somebody else in the world is interested in and someone else is not. Um, static benchmarks also lend themselves well to rubrics, right? People don't come to LMArena to say, I want to, you know, give data. People are coming to LMArena because they love using the best AIs and comparing them and looking at the leaderboard and getting multiple opinions from all the best AIs in the world. Um, so natural user data often does not lend itself well towards asking people to do things they don't already automatically want to do. Um, and static benchmarks, you know, often they're constructed via paid annotators, whether that's a graduate student that's sitting for days and days labeling images of dogs and cats, or that's somebody that you actually hire to say, 'Hey, you know, you're a radiologist. Come in and label images.' Um, live benchmarks are different though. They're just like random people on the internet.

So, in other words, like this is where we are with the platform today. But there's a number of challenges that we need to resolve for the future. How are we going to take these features of live benchmarks, the fact that they're organic, the fact that they're determined by, you know, just random people on the internet doing whatever they want to do, and turn those into features? And I'm going to dig in a little bit on one of them, which is targeted evaluations. How can we take all this organic data that we have and turn it actually, you know, basically invert the problem of evaluation? Turn it into a problem of I'm going to go target a task and I'm going to collect, you know, spend a lot of money to collect a data set for it. Instead of doing that, maybe what I want to do is say, 'Hey, go use the LLMs however you want. I'm going to generate a bunch of data from this organic usage and I'm going to analyze it. I'm going to basically mine it for insights as to which LLMs are good at different parts of the space.' And then you start getting into the questions that are really interesting to people like us, developers and scientists, which is how do we create very personalized evaluations that are telling you which model is best for you? Which model is best for your use case? For people like me, how are we supposed to answer this question? And it's a tough question. You know, one way that you can ask the question would be to say, you give me a prompt. You give me, I want my hackathon project about, you know, a personal assistant that's going to help me schedule things in my calendar, and then I'm going to tell you which model is best exactly for that use case. But the problem is you might be the first person doing what you're doing. That might be the only time I've ever seen that prompt ever. And so the standard paradigm of regression won't work because regressions don't work on data points that I've never seen before, right? You need to like run empirical risk minimization, you need to calculate, you need to have your loss function, you need to minimize, and so on. That's not the way that this is going to work.

So, it seems like a hard problem, but it turns out that there's a way to solve it. And this is where it's useful to have, you know, Berkeley people on the team that are great scientists because what our team has done is we've developed a way of training a large language model to take prompts as input and output leaderboards even if those prompts have never been seen before. So the idea basically is parameterizing the Bradley-Terry coefficients, these regression coefficients, as a language model. Can we train language models to intrinsically output leaderboards by nature? And the way that it works is basically a creative way of supervising via your loss function. Okay, so the only thing we observe, remember, is pairwise preference data from Chatbot Arena left-right votes. Is model A better or is model B? And what we end up doing is we're saying, hey, let's call the prompt P. Let's put it through an LLM. You'll get theta of P. Theta representing the leaderboard. It's a vector of length m where m is the number of models in your leaderboard. And then what we're going to do is we're going to supervise it via binary cross-entropy in order to try to predict the votes that we see. So if theta P is good, this model, which is like sigma of theta P transpose x, that model is going to be good at predicting whether or not you voted left or right. And if the leaderboard is bad, it won't be good. And so the loss function is going to tell you that. Loss function is high when sigmoid of blah blah blah does not agree with y, right? That's how the loss function works. And it turns out that that loss function is differentiable, so you can apply the standard, you know, toolkit of deep learning in order to make sure that loss goes down and you're learning leaderboards that are good at predicting. Okay. So what we end up doing is we actually just train a language model end-to-end, supervise it in order to do this task, and you train it with backprop just as normal, and theta P then becomes a prompt-specific leaderboard.

Now what's interesting about this? Why did I just tell you this technical story? It's because as we're developing these AI apps, there's a lot of interesting and important research questions to do along the way that are really going to change the way that we build our software. So, one of the insights that we have while doing this is to say, hey, by doing this, by turning evaluation into a problem of learning instead of a problem of averaging as you would normally think about it, we benefit from scaling laws. We can have scaling laws for evaluations. And that's exactly what you see. If you train, you know, a half-billion parameter model to do this leaderboard prediction task, it'll do well. It'll do better than the standard regression, but not nearly as well as a seven-billion parameter model, and so on and so on. And the more data you get, the better it gets. So, we're benefiting from ever, ever, ever more granular evaluations that are better at telling you which model is best for you and your app via this like new scientific methodology that we developed open source at Berkeley.

So last thing I'll say, you know, takeaways from this is this was just a hackathon project at first and now it's become a real company, right? We have, you know, more than a dozen employees. Um, we've raised money. Uh, more money, more problems, of course, but things are off to the races. Now we have to demonstrate that this platform is useful. We have our million users, our community that we're digging into.

The first thing I would say is you have to stay consistent. If you looked at the plot I showed you earlier, the platform almost died and it would have died if not for, you know, Wei and others supporting it along the way. So if you don't, if you come out of this and you build your hackathon project and you say, hey, I'm not going to maintain this, it will die and then nothing will come of it. But if you do maintain it, who knows, it might go dip down and then come back up just like ours did. The second thing I'll say is just find the best team. You know, one of the things that we're really proud of is that our team is built with like pretty low-ego people that are looking for people better than them at every task. Wei and I know that we're not the best consumer product people. We know that we're not necessarily, um, the best at UI. We know that we're not the best at backend, but we need to find people that are better than us at this. Um, and in fact, you want to find other people that continue to create this low-ego environment where you guys are building a team serving a greater mission than yourself, right? So, go take your hackathon project, go find the best people in the room, work together on something that's going to become greater than the project that you built over the last couple days. And the last thing I'd say is keep adapting. The world's going to teach you how to change, right? You have to listen to the feedback the universe is giving you in order to continue improving the application that you guys are building. So, those are my three takeaways from my project. And, you know, your mileage may vary. It's not like we've had massive success yet, but, uh, we'll see where it goes. And looking forward to hearing from any of you on how your projects, you know, go to the moon.

[Applause]