The “Big Effing Race” Mentality Is Reshaping AI Research - Anastasios Angelopoulos (LMArena)

Oh, I came in today. I came in for this.

I'll be hanging out today and tomorrow.

Well, I mean the whole ecosystem, it's not just buzzing, it's flying, right? Like we're way past buzzing now. For me, why am I mostly here? It's to find great people to work on Arena with us. It's to try to recruit some top talent to join our, you know, high-performance team building the future of evaluations. Cool. That's my main objective.

LMArena is a gold standard benchmark for the real-world usage of humans with AI. We focus on measuring and advancing the frontier of AI for real-world usage. We run a consumer platform, LMArena, that's got millions and millions of users. They, you know, use our platform for their real daily tasks and we take the feedback that they provide, we turn it into leaderboards and analytics to set the standard of the industry.

You would know it well too. I mean, it's the SkyLab story or the Rise Lab story or the AMPLab story, what have you. But, you know, LMArena started two and a half plus years ago as a research project in, you know, Soda Hall at UC Berkeley, computer science building. And there, you know, there's a group of people in SkyLab even before I was, you know, part of the project that were working on how to build the next open-source language model, you know, an alternative to ChatGPT, inspired by some folks at Stanford actually. And they didn't know how to evaluate it. They were asking, you know, it's clear that A is better than B, but how can we tell? They do equally well on all of these benchmarks, but when you chat with them in a conversation, one is better than the other. And so what emerged from that is this blind testing platform that has turned into LMArena. It's a system where the user can come in, they can input a prompt, they get more than one response and they can vote for the one they like better. So this platform started from nothing and then, you know, Wei-Lin and Ion and Yang Song and all these other people that were working on the project, also Joey Gonzalez and Lisa Dunlap, they, you know, and myself, grew the project from a tiny, tiny thing that it was in Justin Soda to this massive platform with a million users even while we were still students. Wow. You know, many of the, for GPT-4o debuted on our platform before it was in ChatGPT even while we were at Berkeley.

Yeah. I mean, to their credit, I think Frontier Labs were interested in supporting us. They were interested in having a neutral academic system for telling which models are better than the others and believed in the methodology that we created. So credit goes really to them as well. But yes, it became a focal point and remains a focal point for the industry in terms of telling which AIs are best on economically valuable tasks.

To begin, I think it's worth saying I really love the AI space. I think that what's happening in AI is extraordinarily exciting and that this, I view it as the opportunity of a lifetime to be working here. And I came into it by complete accident. You know, it's not like I came in here wanting to do this. I started working in like medicine and then, yeah, and then I transitioned. I did most of my PhD in theory. I did like theoretical statistics, theoretical. Yeah. Undergrad. Yeah. I was doing like devices. I did my undergrad in DUB. I worked with like Gordon Wetzstein and some folks at USC, Mihaim, devices, medical devices and then stats. And then I decided I wanted to do something more fundamental and like really build my fundamental understanding of mathematics and statistics, the foundations of data, so that I could give myself a really rigorous foundation which I think from moving forward a long longevity in terms of my career. So I went and then did theoretical work in statistics, machine learning, conformal prediction, the theory basically of estimation. And that turned into building a suite of techniques along with my, you know, many wonderful collaborators about how do we properly use AI and synthetic data to reliably draw conclusions from synthetic data about the real world, you know, make predictions that can be used in high-risk downstream decisions safely in formal mathematical guarantees of validity. That was what I did most of my PhD on. And then I started getting into this like LMArena stuff sort of as a side project because I was interested in the methodological aspects of it. How do we run these regressions? How do we learn from preference data? That is a whole another thread that we might want to explore. So I feel very, very lucky to be here. I had a very nonlinear path to this point and I've been amazed by the quality and strength of all of the different people in industry and in academia working on AI. It's a privilege really to be here. I think one of the things that worries me to your point is that in the quest to, you know, develop artificial intelligence, we've been so successful as a community that it's attracted a huge amount of money, resources, fame, relevance, the whole world is talking about AI right now and I feel that that is starting to touch the academic world. In fact, it's very much infiltrated the academic world in a way that could pose a risk to this sort of endangered species of researcher, which is people that don't care about any of this. People that are like doing, they'll have 5,000 citations over a 40-year career, but they're doing something that's actually very deep, very impactful. They're not talking about it much. They're not, you know, they're not like me. They're not on a podcast, right? They're like not shipping their research either, which I believe in, right? I believe in the whole talk about, you talk a lot about this and I believe that you should ship your research. I'm an example of that. But it shouldn't take being someone like a you or a me or a Ion Stoica or a Fei or, you know, whatever in order to have a home in the academic world. We need space for pure people who are just thinking about how to push the frontier of human knowledge without any regard for, you know, fame and wealth. And I believe citations as a metric have really exacerbated this too.

It's not just about theory. I mean, I'm not talking about something that's just theory here. I'm talking about there's almost like a personality type. They want to make sort of 10-year career-level bets, 20, 30 years. Think about these people as like the sort of rare spider that's making this ornate web. You know, they're like creating this technology that may or may not have immediate economic value or be interesting to anybody other than them and five other people in their community over the next 5 years or 10 years. Those kinds of people, they're actually gems and we need them in our world.

Well, at least short-term impact, right? Yes. And I often think that academia traditionally has had impact through the procreation of the academic personality itself, of the researcher itself. So we have advisors and advisees and we have the job talk process and we have the whole institutional subsystem that lets us have researchers beget more researchers, like give us this millennia-old family tree. That's another form of impact that you tend to see at the end of a faculty member's career when they put their last slide. They're like, here's my accolades. Here's what I did. When you summarize my whole career in one slide, often it's the list of their PhD students and sometimes numbered like, and training PhD students is great, it's a great mass, but that's almost like the most indirect and potentially the most profound type of impact. And then the next one over you could say is this curiosity fundamental knowledge pursuit. You have things that are very abstract where you don't need them to justify that this certainly would be able to have impact. Yeah, exactly. Or near-term, let's say, could it ever turn into a product. You wouldn't want to require them to justify research.