Anastasios Angelopoulos
0:00 ↗
Anthropic revenue has been booming, booming, booming on the API, but to some extent it's also easy come, easy go. If you are spending that much money that quickly, there's no chance that you're not able to switch vendors. And so if an open source model comes by and it is able to do 90% of the work at 10% of the cost, which is totally feasible that that might happen over the next couple years, I think that's a pretty structural problem. People are assuming a degree of lock-in to the APIs that I don't think is necessarily going to exist. We work with labs to help them improve their models. The product that we give to them is basically an analysis and insights product called an evaluation. In the process of developing models, labs will come to us and say, "Hey, we have like 50 checkpoints this week. How good or bad are these at different types of tasks? Is this good at front end? Is this good at full stack? Is it bad on tool calls?" We take those checkpoints, we evaluate them on our data, which represents real usage, and then we hand those results back to them as an insights dashboard. And so this is the product that scaled us to over 100 million in revenue. This has proven so useful to the labs. How can we take an Arena and put it within every business in America and every business globally?
Welcome to Grit. I'm Jubin, partner at Kleiner Perkins, a show where we go beyond the highlight reel and explore the personal and professional challenges of building history-making companies. Today on Grit, I'm talking with Anastasios Angelopoulos, co-founder and CEO of Arena, the platform the entire AI industry watches to answer one question: which model is actually best? Anastasios went from a Berkeley statistician to building Chatbot Arena as a research side project to running a company that hit $100 million in annualized revenue this June at a $1.7 billion valuation. My partner Mamoon Hamid joins me as guest host.
You, you were born here. Was born here.
Anastasios Angelopoulos
1:51 ↗
Yeah. So, LA kid.
LA kid. Is the best place to grow up, you know? It's like...
Anastasios Angelopoulos
1:56 ↗
It was awesome. A lot of art, a lot of music.
Anastasios Angelopoulos
1:59 ↗
I didn't think I was going to be in tech. Yeah. Then I went to Stanford and I was like, this is what...
Did you know he still has, he has a plaque at Stanford as a professor?
Anastasios Angelopoulos
2:10 ↗
Oh, did I tell you that I was going to be, before this, a professor at Stanford?
You got the offer to be a professor, right?
Anastasios Angelopoulos
2:16 ↗
Yeah.
Anastasios Angelopoulos
2:17 ↗
I got it and then I accepted it and then I deferred it.
Anastasios Angelopoulos
2:20 ↗
And my wife is there and I have an office next door with my name on it.
Anastasios Angelopoulos
2:24 ↗
And it's empty. Yeah.
Yeah. You know, I was talking to the dean about this particular problem.
Anastasios Angelopoulos
2:32 ↗
I don't know what I'm going to do with respect to them. I guess I'm just going to like let it go.
I asked, I asked like, "Hey." Yeah. Yeah. She was, remember this conversation about like, "Hey, are you losing a lot of students?" Like, no. Well, I'm losing some students, but worse than that, I'm losing faculty.
Anastasios Angelopoulos
2:49 ↗
Yeah, and I need them to be here. We're a small department. Especially CS, it's actually a small department.
And if like 10 of you decide you don't, you want to start a company, that's a problem for Stanford.
Anastasios Angelopoulos
3:02 ↗
Yeah, for sure.
Anastasios Angelopoulos
3:04 ↗
It's a huge problem.
But they also need to find a way to make it work because the highest value people are just going to leave.
Anastasios Angelopoulos
3:09 ↗
Can you still get away with like... I'm just not going to go back unless they find a way to make it work.
Anastasios Angelopoulos
3:15 ↗
Oh yeah.
Yeah. She's dealing with this right now for sure. Right. Um, and it has happened to be all in the same week where I'm just talking to her, talking to Dean Whidham about it. But anyway...
Anastasios Angelopoulos
3:26 ↗
Has anyone found solutions?
She's like, like, can you help? I don't know how to... I think the dean has to solve this. Actually has to come up with a plan.
Anastasios Angelopoulos
3:39 ↗
Makes sense.
Is it, is it worth having to come like helping? I think it's worth it. There's a lot of us during that situation and...
Anastasios Angelopoulos
3:47 ↗
Like how many is a lot of us?
Anastasios Angelopoulos
3:53 ↗
Um, there's... I always said, but I guess I probably said it wrong. How was it?
I don't know. I always say, what's the situation? What's the... what do you feel stuck in?
Anastasios Angelopoulos
4:03 ↗
Oh, there's like you have a job at Stanford as faculty.
You're like trying to take your research and turn it into real impact.
Anastasios Angelopoulos
4:10 ↗
Right.
Right, like you're trying to do what your job is as an academic, which is to develop a new technology or a new idea and then translate it into something that like shapes the world. Right? As part of that you build a company.
Anastasios Angelopoulos
4:24 ↗
You become very important to the company.
And then Stanford will not let you take an indefinite leave. They require you to come back after two years.
Anastasios Angelopoulos
4:35 ↗
And they just are completely inflexible with what an arrangement looks like. So they won't allow you to like retain your position at Stanford and, you know, teach one semester out of every two and, you know, so that you can run your company and still juggle these responsibilities, which, you know, would be something that I would be happy to do. I'm sure Fei would be happy to do, or like, you know, buy yourself out of teaching, or have some like... there's, when, when you're running like, let's say, a multi-billion dollar company, there has to be some way of like making an arrangement work so that the teaching can happen and the department can function and all that stuff can work, and also your faculty that are having the largest impact on the world through these companies can continue to do that and not be crippled.
But what's the counterargument from Stanford? Like, wouldn't it be in their best interest to have you go out, do what you're doing now, and then come back and spend some time on campus with students talking about the experience that you're having in the real world?
Anastasios Angelopoulos
5:33 ↗
Well, you know, the counterarguments...
Yeah. I mean, I think they, they don't have unlimited faculty positions, right? So they make a bet on Anastasios and others. Uh, and, and the bar is like yay high, and you know, there's not many people that are crossing the bar. Once you've decided this, this person's in, like, you know, you can't like make offers to, you know, 2x as many people who just have built flux into the system because, you know, a professorship costs money to the department and, uh, it's like, it's managing a business. It's like, you know, I'm allocating dollars, uh, to this much headcount and if you all of a sudden say I have to have two, two x the headcount to make my department work just because I've got a bunch of faculty members who are off on leave. Yeah. It, it really makes it a kind of a logistical nightmare. And then, and a lot of the junior faculty are the ones teaching, doing the teaching, right?
Anastasios Angelopoulos
6:25 ↗
And so...
Everybody needs to teach.
Anastasios Angelopoulos
6:28 ↗
Yeah.
Even the senior faculty need to teach.
Anastasios Angelopoulos
6:30 ↗
You mean they need to teach or they're...
They're required to teach.
Anastasios Angelopoulos
6:32 ↗
They're required to teach. Yeah.
Are you stuck in that right now?
Anastasios Angelopoulos
6:36 ↗
Well, I mean, I'm not stuck in anything. I have a lot of great opportunities. I think it's more like Stanford's stuck because they don't have a way to like flexibly retain some of their best. It's not, I mean, like, you know, I'm, I haven't even started yet. I'm not really a problem for them.
Anastasios Angelopoulos
6:53 ↗
But Fei leaving is a problem for Stanford. Andrew Ng already left. Daphne Koller already left.
And these people are like luminaries.
Anastasios Angelopoulos
7:01 ↗
Luminaries.
Anastasios Angelopoulos
7:03 ↗
Um...
Yeah. Like Chris is sort of...
Anastasios Angelopoulos
7:05 ↗
Chris Re has kind of his own arrangement. Percy. Yeah.
What's going to happen with Percy?
Anastasios Angelopoulos
7:10 ↗
And those, those are some of the best that Stanford has.
Anastasios Angelopoulos
7:13 ↗
Um, for me, I, you know, it's kind of like a no because I haven't even joined. And so maybe one day I could be important at Stanford, but it's more like lost opportunity. For some of these people, it's like they're losing their anchors and it's because they have great opportunities elsewhere and they have opportunities. It's not about necessarily making more money, but it's also about making more impact. And so the need to solve it...
Anastasios Angelopoulos
7:38 ↗
Yeah. Well, the first time we met was actually on the Berkeley campus.
Anastasios Angelopoulos
7:45 ↗
In a, I don't know if I should call it charming or depressing little rectangular box with...
Anastasios Angelopoulos
7:54 ↗
The, the like, uh, yeah. We were like in the basement of Soda Hall and yeah. That's what we like pitched you guys.
That's where you pitched us. It was awesome. Me, Ion, and then...
Anastasios Angelopoulos
8:04 ↗
Yeah. And a couple of us.
Anastasios Angelopoulos
8:06 ↗
And, and Ilya, I think, was it right?
Anastasios Angelopoulos
8:09 ↗
Bucky. Yeah, Buck.
Anastasios Angelopoulos
8:11 ↗
Why did you meet at Berkeley, not Stanford?
Well, he was finishing off his PhD at...
Anastasios Angelopoulos
8:17 ↗
This was never meant to be a company. We were like, we started this off as a student project like three years ago at Berkeley because at the time we didn't know how to evaluate. It was like kind of the, a side project of a side project. I meanwhile was doing my PhD in theoretical ML. I was like proving theorems in a basement at the time when really I was happy doing that and that's also how my, my wife, because we were both theorists and that's how we both also got, you know, our jobs at Stanford and that's...
Um, and then we, you know, in the meantime we started Arena and it just grew and grew and grew into this project and then I remember what happened was that at the beginning it was just OpenAI and then Google came in the mix, Anthropic came into the mix and it just exploded because people wanted to compare, people wanted to understand the differences and they were kind of surprising because you would test them on like MMLU or whatever multiple choice questions and it would be like the model that does well on the test is not the same model that I like to use and that's doing well in the real world. It's because they all train at the test. Um, and so we had this philosophy that it's really about reality. Reality is the only judge that you can actually trust. So we built Arena which is this platform where people go to use AI in the real world and by using it they're giving us the feedback that we need to make these leaderboards which you'll see online. Uh, and it just exploded from there. Um, but we met very early in the days back when we were still basement dwellers.
Yeah. And do you think, uh, it was at that time, you know, this is a year and a half ago or so now, you had OpenAI, Anthropic, Google and, um, DeepSeek was relatively new. Was there a moment around DeepSeek that made you think, hey, like I need to go start a company because there will be not just three models but there'll be, you know, not infinite but lots of models?
I wouldn't say it's DeepSeek specifically, but I, there, you know, throughout the history of Arena, there have been all these times when people come to me being like, do you think that they're, they're going to consolidate? Do you think that the models are going to consolidate? People still ask me that question like once a week I get, oh, but what if the models consolidate? And it's always the opposite.
People are just not understanding that AI, because it transforms every industry, is going to lead to like many flowers blooming.
Um, and so yeah, for sure that, that was a trend that we didn't fully appreciate at the time, but has proven to be really helpful for our business.
Yeah. Because at, at the time it was still the leaderboard for those, the obvious company.
Yeah. And we had like, you know, 20 models, you know, we started with like eight models.
Yeah. How many models do you have today?
And when you say have, can you explain what have means?
No, we just like rank them on the leaderboard.
We've started deprecating some of the older models.
Um, but you know what's happening now is that every week there will be like 10 releases.
It's insane. Today new releases from Qwen, Qwen Max and there's more coming from Qwen. The dragon has awoken.
Yeah. And you've become somewhat of an expert on, uh, some of these open source models.
For sure. And like I think you guys are sort of leading the charge on, you know, almost like not anointing, but like really understanding deeply like what's happening with Kimi and Qwen and others.
Where does that expertise within the company come from?
Well, it all comes from the users. The point is that, like, the point of Arena is that real users and the utility that these models provide to real users should be at the center of the AI conversation. There shouldn't be some random benchmark that's made by some dude that thinks they're smart, some researcher out there that's like, "Oh, how does AI do on this like weird prediction task?" And I think that like long-running finance workflows of XYZ type are the thing that determines AI progress. It's like, dude, get your head out of your ass.
You're the Consumer Reports for AI.
We're the Consumer Reports for AI. There you go. I'm serious though. People make all these up benchmarks.
Why are they making those benchmarks? Because benchmarks attract attention and, you know, and people don't know how AI models... People don't know how to rank them. People don't know what performs best for their task and so they're creating all of these like wacky ways of measuring them. We were all like for the last year on Twitter looking at pelicans riding bicycles because people were making benchmarks around SVGs. You know what I'm talking about, right? The pelican riding the bicycle.
Well, there was this common benchmark that was made, you know, there's a particular individual who likes benchmarking things and made a pelican riding a bicycle. And then everyone on Twitter is looking at this every time the new model comes out. Oh, where's the pelican? It's like, let's show you the pelican. Who the cares about the pelican, dog? I'm not kidding you. I mean, it sounds ridiculous because you guys haven't heard about it. But it's like...
No, you should. You should be... I'm so happy to hear that. You should...
It reminds me of like when you were in the graphics industry in, I don't know, circa 2000. Everyone had like the spinning, um, like video of Venus Demo. That was, that was a test.
So this is the pelican riding the bicycle. This is how far we've gotten is from mythology to...
Yeah. 100%. Now I think that benchmark has been saturated because it's made its way into the training data now. People know how to generate SVGs and pelicans riding bicycles. But people create all sorts of like random little tests and toy examples, but what it doesn't speak to is real user value. What happens when you go try to use the model for your actual workflow? You're trying to code with it. You're trying to do your finance workflows on it. Spreadsheets about which companies to fund and which companies not to fund, which you guys do every day. That kind of stuff. There should be benchmarks that measure it. And it can't come from just somebody writing down a set of questions. It needs to come from you. You need to be the user and you need to tell me, hey, this is good, this is bad. And it's not because you, uh, are trying to benchmark, but because you're in the natural flow of work and you're getting your job done. That's what Arena is all about. It's measuring utility by providing utility. The point is that the user incentive and the platform incentive are perfectly aligned because we're trying to measure utility and the only way to measure utility is to give utility to people. So it's like Yelp meets Reddit where you can do some hierarchy upvoting to make sure that the users are the ones that are actually deciding what's good or not.
Yeah. And I think, um, a different way of thinking about it would be like a, uh, model-agnostic ChatGPT, um, or a model-agnostic Claude coworker because, uh, the insight is that the user of this product should never feel like they have to give feedback. They shouldn't feel like they have to upvote, downvote, and a lot of times the feedback that they give is not of that nature. It would be implicit conversational feedback. You know, I like this. Can you keep going? Or, you know, what a terrible response. You know, undo it and give me something else. Anytime you're doing that, that's explicit natural language textual feedback that we can use for our rankings.
And then there's implicit natural language feedback. You ask, you ask a question, you try to get your response. It doesn't work. So, you try to ask it another way. Almost like in the days of search, you have query reformulation. Like, there's query reformulation also for coworker type products where you're just trying to get, like, how many times have you had the experience where it's like I'm just trying to do X with my AI and I keep prompting it to do X and it's doing it wrong or it's doing X plus Y and the Y I didn't ask for it. It's like I've been trying to like, you know, figure out where to put a couch in my living room and it's like I want the painting here and I want the couch there. Can you do that? And it's like sure and then it's like I'll do that and I'll also modify the window. What are you doing? And so I'll try it again. No, do it again but don't mod. And so that is a piece of feedback. Every interaction is a piece of feedback. Download button. That's a piece of feedback. And so if you take all this together, you can learn about which models provide utility to users. You know, which ones are they actually getting their job done where they're not having to reformulate their query 50 times? Which ones are they downloading the files and then actually using them? Which PRs are they merging into their codebase? All this stuff we measure and it becomes the highest quality signal of model performance on the market.
What kind of people use Arena on a regular basis?
So this is very interesting. Um, what would you guess? Actually, you don't know our business. What would you guess about, uh, Arena users?
Why do you guess engineer?
Like, uh, assuming maybe that if the breakaway application is coding and having spent a bunch of time with engineers over the last year, they seem to care absolutely more than anyone being on the frontier depending on what coding model they want to use.
Totally, and you're right. It tends to be more knowledge worker professionals that are using Arena. 28% of them are software engineers.
Mhm. Um, by the way, at the scale of tens of millions of users, this is a lot of freaking software engineers.
Yeah.
So, 28% software engineers, 17% scientists, 15% finance, 6% legal, 6% media.
The categories that are like you should expect as these new categories.
Like if Harvey and Sierra ever catch up to Cursor, you would expect those numbers to probably start going up in Arena.
No joke. So, it's not just a leaderboard.
It's not just a leaderboard. There's this massive, uh, community consumer following, uh, that powers all the, all the data and analytics that we show.
And I'm curious like, okay, let's just assume that you have more ground truth on model quality and progress than maybe anybody in the world. What has surprised you over the last year on how the leaderboards have changed?
You know, the, the easy answer would be that the pace of development is just extraordinary. I think it's faster than any of us could have predicted. But I think the number one thing that's starting to happen now is there's a phase shift between closed and open models. And I think that the implications of this are not well understood by our industry. Um, the fact that Kimi was number one on our web development arena was huge national, international news. I don't know if you saw it online. Um, but the fact that Kimi is number one on, on web development, it, it's the first time really that we're seeing a violation of the distillation narrative. The distillation narrative being that the only reason why China is keeping up is because they're distilling American intelligence. But how can that be true if they're number one and America's number two in a very important category by the way. Most software developers are doing web development.
So it's not a nothing category. It's where most of the economic value is coming from.
And so what that means is that yes, of course they may be distilling, but distillation is not the only part of the story. There's something else that they're doing actually to push the frontier beyond American labs. And so this is what we're getting into. You know, I don't think Americans are fully appreciating the intelligence and creativity that's coming from Chinese scientists, but that is being shown. It's being proven as we speak. And Qwen is coming out. You know, Qwen just came out with their Max model. It's a frontier model. Um, and there's going to be many more of these. And so, I think that we're, we are starting to get into the flipping, the flipping of the American and Chinese. And so what happens when this actually comes to fruition if we actually get a Chinese model that's as dominant as Claude has been over the past couple months in the sense that it's topping every leaderboard. You know, you should tell me what's happening to the capital markets. I think it's not going to be good for us. Um, but the other aspect of it is that there is a silver lining which is that I think that there is room for a trillion dollar American company based on open source.
I think everybody would probably agree with that.
I don't know. Would they?
Well, I mean, I think everyone here would like to agree with that.
Like, I'm curious if the leaderboard suggests that there's anything even close.
Even close? No. Thinking Machines the closest. If you, if you, uh, look at, uh, open source models, they would be number 10.
