The AI Leaderboard Every Top Lab Watches - Inside LMArena's Real-World AI Battleground

I never did any of this because I wanted to have commercial anything. We're at Berkeley, you know, we're eating pizza, free pizza, making no money, you know, but Sam Altman cares what we're doing. Our goal has always been to be stewards of the ecosystem, to have as much positive impact as we can. We're academics by nature and we found ourselves serendipitously at the center of it. Well, the vision has always been to measure and advance the frontier of AI for real world usage as opposed to some, you know, academic benchmark that somebody has, you know, collected perhaps painstakingly, but that doesn't necessarily reflect real users. Who cares how well it does on a math olympiad? I do mathematics in my work and even I don't care about it because it doesn't reflect the type of math that I do.

Thanks for having me, Boris, and excited to be here. Honored to be your guest on the podcast. First 10 has been a wonderful partner to us at LMArena too.

Great. Yeah.

Please.

I root for whatever school I'm at. So, and what ended up happening is that all the years that I was at Stanford, Stanford won. And then all the years that I was at Berkeley, Berkeley won.

Yeah. I don't know. I mean, yeah. You'll have to ask the players because I never saw any of the games.

Yeah.

I've been busy doing statistics instead of football. That's the problem.

I don't know how hidden, I don't hide much. I kind of wear my heart on my sleeve, but I guess outside of the work, you know, outside of professional work, the main thing that I do is music. So, I have a little music studio at home. I play guitar, piano, I sing, I play bass. I mean, bass and guitar are kind of the same thing anyway because it's electric bass. Yeah. And yeah, I just hang out, play music with my friends, produce. It was my first love. I thought I was going to go to college and major in music. I ended up majoring in like electrical engineering because of signal processing, which had this connection to music because you do a lot of signal processing when you produce songs. And I'd say that's maybe a little known talent to the rest of the world. Not clear that it's a talent really. I'm obviously not a professional, but I do love doing it.

That's funny.

Maybe Voodoo by D'Angelo.

Yeah. One of my favorite songs is Really Love by D'Angelo. So I'd say I don't remember if that's on Voodoo or Black Messiah. Maybe Black Messiah.

Yeah, he's awesome.

Yeah. Do it. He recently passed away, right?

Yeah, I think he died. He died this year. Yeah, he died a couple months ago. Super sad, at 50, 51.

Yeah, rest in peace, D'Angelo.

No, you know, I never did any of this because I wanted to have commercial anything. I had done my undergraduate in EE. I was working primarily on medicine, like medical devices, trying to help, you know, disabled people and people with diseases of the eye primarily, like, you know, people that are going blind, try to help them walk and, you know, use devices better. And after I finished my undergrad, I felt like I didn't have enough foundational, fundamental understanding to be able to engage with science for my whole life. And so to acquire some of that extra knowledge and expertise, I decided to do a PhD. It was really only about knowledge acquisition and, you know, to sort of train my thinking muscle.

But I decided to do a PhD in something fundamental, and also because I thought maybe I'll do a master's, but that seems expensive. So a PhD, I'll be able to maybe get it for free. And if I don't like it, I'll just drop out with the master's. So I applied to PhD programs. I had a few mentors at the time in Stanford like Joseph Kahn, Stephen Boyd, people like that, Gordon Wetzstein, who really, really encouraged me. And so then I ended up going and doing a PhD at Berkeley under Mike Jordan and Jen Malik, who were two huge luminaries in the AI field. It was before, again, any of this like big commercial stuff. I started my PhD in 2019, I think.

So it's before any of the ChatGPT generative AI boom. I guess at that time it was all about self-driving. But it was orders of magnitude smaller in terms of the compensation that you could receive with the skill set that you get with an AI PhD. And the work that I concentrated on was all about reliability. It was about developing the mathematical and statistical foundations of reliable AI. And I started by working on this field called conformal prediction. Conformal prediction is a methodology for placing uncertainty bounds on black-box algorithms. One of the most important applications being to AI, helping know when AI is wrong and use it therefore with confidence. And that's what I spent the majority of my PhD on, as well as synthetic data, trying to understand how to properly leverage synthetic data to draw valid scientific conclusions. And again, without even a thought of turning it into a company or anything.

While at Berkeley, yeah. Well, what happened was that I got introduced to Wei through my advisor Mike, and they were kind of working on this pairwise preference thing, and it was very, very early stages at the time, and needed to know how to do proper statistical inference on the data that they were gathering through this platform, how to construct the leaderboard, how to construct confidence intervals for the leaderboard, you know, what methodology to be using, and I helped shape that. It wasn't really anybody's main thing except for maybe Wei's at the time. And then we wrote a paper on it, and I think we kind of thought that would be that, like we would continue to run the platform, try to have an impact with it, but we never expected it to be a company. And then it just grew and grew and grew up until the point that even while we were students, GPT-4o, for example, released on our platform before it got into ChatGPT. The same thing happened with a bunch of the Gemini models, and, you know, at that time it was like an explosion of diversity in the AI ecosystem around models. So many new models, so much competition as to who was going to be the best. And we found ourselves serendipitously at the center of it because we had built this competitive platform. And then it became latched onto by media, by labs, by researchers as a battleground for these new and really exciting corporate competitive markets to duke it out. And we were like, 'Wow, we're at Berkeley, you know, we're eating pizza, free pizza, going to seminars, eating free pizza.' And making no money, you know, but like Sam Altman cares what we're doing.

Wow.

And not only did they come, but they stayed. Like the attention only grew and grew on the platform over time. And it started becoming this really central gold standard benchmark for measuring the real world utility of AI to real users.

Which is something that we really believed in. And so eventually Wei and I, we kind of got together and said we want to turn this into a company and we want to do it together.

There was no moment. It took a long time for us to think about what to do with the project because our goal was never really to make money. Our goal has always been to be stewards of the ecosystem, to have as much positive impact as we can. We're academics by nature. You know, we don't care about aggrandizing or enriching ourselves. And so we thought about, hey, what's the right way to do it? Should we do it as a nonprofit? Should we do it as a company? Should we just donate the whole thing to somebody? And what ended up happening is that it became clear that the only way to make it work was to have the scale and resources that a company could provide, and that there was a business model that would allow this to scale to become a really central piece of the ecosystem.