All right, we're here with Anastasia from Arena. I actually don't actually know your last name. It's like very...
Anastasios Angelopoulos
0:17 ↗
Angelopoulos.
Yeah, there you go. Um, congrats on all the success. You got the arena handle.
Anastasios Angelopoulos
0:22 ↗
Yeah, we did. We got the arena handle. Thank you. Big branding moment.
Uh, I mean, I think X is like being more commercial. So obviously you bought it, but like at least you like have a place to go to where you can be like, 'Hey, like we really like this.' But I do think like dropping LMArena has changed the feel of it. I don't feel... I don't know.
Anastasios Angelopoulos
0:42 ↗
I mean, the reason we kept the LMArena at the beginning is because we were like, we started as LMSYS, right, out of the LMSYS sort of, you know, conglomerate at Berkeley. So we decided...
Anastasios Angelopoulos
0:56 ↗
Exactly. So, so we wanted to maybe broaden a little bit. Yeah. And we were the first arena, so we feel like let's kind of try to own that.
Yeah. Last time we had you guys on, you hadn't really spun out yet. And I did a call with Alessio and I was like, these guys are going to start a company. [laughter] And I didn't know. I think you actually were already started at the time.
Anastasios Angelopoulos
1:15 ↗
I don't remember. Uh, maybe.
Because I chatted with An and he said he was your founding CEO.
Anastasios Angelopoulos
1:23 ↗
He was indeed, uh, which like people don't know.
An is a very interesting character. We have a podcast scheduled with him. Yeah. Yeah. He does a lot more than normal VCs.
Anastasios Angelopoulos
1:33 ↗
He does. He's been incredible to us.
You want to shout out some stuff that he did?
Anastasios Angelopoulos
1:36 ↗
Yeah, absolutely. So, you know, the way the company started was as an incubation by An. So, what he did is he kind of like found us at Berkeley and picked us out of the basement and was like, 'Hey, these guys seem like they're on to something.' And started working with us really early. Gave us, you know, some grants. He was not, you know, A16Z was not the only one to do this. We also had a great grant from Sequoia, but An was in particular quite supportive of us and, you know, gave us some resources in order to continue building out Arena before we even were committed to starting a business. And in that capacity he sort of like, you know, formed an entity for us and this and that. And, you know, he was like, hey, you guys can walk away at any time if you guys don't want to start a business. I mean, it was really incredible. Very aggressive investment move by him, [laughter] right? Because of course, any money that he spent at the end of the day, like we could walk away and leave him with nothing.
But I think he knew wisely that the right thing for Arena was to start a company out of it. It was the only way that we could scale and that Wei-An and I would ultimately see that and be excited about doing it ourselves, which ended up being the case.
Was there a moment for you where you, I'm sure you were debating it yourself, you had other opportunities. What was the deciding factor for you?
Anastasios Angelopoulos
2:49 ↗
It became clear that the only way to scale what we were building was to build a company out of it. That the world really needed something like Arena. Arena being really a place to sort of measure, understand and advance the frontier AI capabilities on real world users, on real world usage based on organic feedback. And that in order to achieve the scale and, you know, distribution necessary and the quality, of course, of the platform necessary to do this effectively, we would need to start a company out of it. You know, we considered other options. Are we going to keep doing this as an academic project? Are we going to do it as a nonprofit? Blah, blah, blah. But ultimately under those constructs we didn't feel like we'd have the resources necessary to accomplish our mission.
So you raised a hundred million or 80... 100... $100 million. That's a lot of resources. It's great. Yeah. What's it for?
Anastasios Angelopoulos
3:44 ↗
Well, you know, obviously...
On behalf of like everyone who like... Yeah. Everybody, dude, it's an arena. How are you going to spend your money? [laughter] Yeah, it's great. Yeah, I think so.
Anastasios Angelopoulos
3:52 ↗
First of all, we don't necessarily need to spend all that money, right? The purpose of money at a company is to give you cards to flip. It's to say, hey, you have enough resources necessary so that if your first bet fails, you can make another bet and another bet. Of course. So that's not to say that we're going to spend all of it. Of course, you want to spend things responsibly. Having said that, the platform is actually quite expensive to run. We fund all of the inference on the platform. You know, the way that it works is the platform...
You pay market rates. They don't give you a discount.
Anastasios Angelopoulos
4:19 ↗
No, no, we get discounts, but they are standard enterprise discounts, the same that would be given to any other customer.
And have you disclosed any, I don't know, any numbers? I see numbers of votes, but like what's that in like monthly tokens or I don't know.
Anastasios Angelopoulos
4:36 ↗
I don't know about tokens, you know. I'd have to back that out. We have like, you know, let's say, again, this is off the cuff, but I can safely say we have more than 5 million now, 5, 6 million. We have probably 250 million conversations that happen over the course of the platform. We're on the order of, you know, mid tens of millions of conversations every month that are happening on the platform. Um, it's actually quite a large, you know, it's one of the largest consumer platforms for LLM. Of course, nothing is really comparing to ChatGPT, but you know...
But no, still, I mean, I think the largest scaled ones...
Anastasios Angelopoulos
5:07 ↗
The benefit of this is that like it's actually quite a diverse population. So 25% of the people on our platform, for example, do software for a living.
Still at this scale. How do you know?
Anastasios Angelopoulos
5:18 ↗
Because we like do all sorts of... we either survey them or we'll like analyze the prompt distribution that's coming into the platform. Happy also to share more. We've done something called Expert Arena, which is like trying to understand the distribution of experts that are coming to the platform. A lot of that can be like unauthenticated or whatever usage it is. But about half of our users now are logged in.
Anastasios Angelopoulos
5:37 ↗
And so we have some ability to understand them, and we also have surveys that we run on the platform that tell us a bit more about who are the actual users. Of course, there's always like response bias in surveys, so you have to take it with a grain of salt, but nonetheless...
And if you don't know Anastasia's background, like he's the guy to correct for response bias.
Anastasios Angelopoulos
5:56 ↗
Yeah. Okay. Yeah, there's a lot of guys like that and girls.
Guys and girls. Um, you're not the only player. Uh, there's Artificial Analysis, LMArena. Yep. AI, it's like some crypto people that started this. I don't know if you have a conversation with them on like, well, hey, this is our thing or like, let's work together on something. I don't know if you've... what the...
Anastasios Angelopoulos
6:17 ↗
No, you know, so I've talked to Artificial... I've actually talked to both groups. Both seem, you know...
Am I missing any major players? It's just those...
Anastasios Angelopoulos
6:24 ↗
No, I think those... I don't know. Yeah, I don't think so. Okay. I think those are like some of course large, depending on how you define the term. Artificial Analysis obviously has like huge market mind share around the analysis of different AI systems.
Yeah, they told me they were just like, we are going to be Gartner of AI.
Anastasios Angelopoulos
6:44 ↗
Yeah, that's kind of their goal, and I think they're going after that consulting market and so on. Artificial, from what I understand, is like a team of consultants that are doing this.
Anastasios Angelopoulos
6:51 ↗
Seem like really, really nice guys going after that particular market. It's different from our platform in the sense that their analysis is based on sort of like aggregating public benchmarks and turning those into analytics.
And independently rerunning.
Anastasios Angelopoulos
7:09 ↗
And independently rerunning, which... Yeah. Yeah. Yeah. Which matters. Yes. And using those in order to compile sort of reports and so on that educate the field on the performance of all these different models.
But they also have arenas.
Anastasios Angelopoulos
7:23 ↗
They have arenas, but the arenas are not based on organic usage. Like the thing that distinguishes our platform versus theirs is that the users are actually inputting their own use case. They're actually asking their own question. And that gives a level of realism that platform doesn't have. Of course, they specialize in a slightly different thing, but I see those platforms kind of diverging in that sense.
Yeah, and sometimes it's like it's the only way to do this. So like for example for AA, their video arena, it's pre-generated videos. You can't enter in your own video.
Anastasios Angelopoulos
7:55 ↗
That's correct. But we are doing it organically.
Anastasios Angelopoulos
7:58 ↗
And so as a voter it does help in terms of like I don't have to wait.
It does, but also why would you go? Do you actually care about like other people's videos to form your own intuition?
Anastasios Angelopoulos
8:10 ↗
Maybe. Yeah. Maybe you're like interested in comparing.
Like I'm a shitty prompter, right?
Anastasios Angelopoulos
8:14 ↗
Are you? I don't believe that.
I'm a terrible prompter. I learn by example.
Anastasios Angelopoulos
8:18 ↗
Don't denigrate yourself. Don't denigrate yourself.
Um, there are many prompts much better than me. Let's say that. Right. That's a fact.
Anastasios Angelopoulos
8:24 ↗
People have all sorts of cool ways of prompting. Yeah. It's educational to see.
The only way to learn is by like, well, look at other people's prompts and see the results and go like, 'Oh, I didn't know you could do that.' And that's...
Anastasios Angelopoulos
8:35 ↗
Totally. Yeah. Yeah. Yeah.
Okay. So, let's go back to Arena. Oh, one thing I do want to say is the number one use of funds is getting off Gradio.
Anastasios Angelopoulos
8:43 ↗
Oh, yeah. Yes. Well, listen, [laughter] Gradio, incredible platform. Gradio scaled us to a million MAU.
Yeah, that's incredible. And of course, you tell the Hugging Face fans that...
Anastasios Angelopoulos
8:52 ↗
Of course. Yeah, we're really, really grateful to Gradio for taking this so far. Eventually, you know, it became time for us to move off of that and go to React.
I'm sure Hugging Face would have loved you to stay on.
Anastasios Angelopoulos
9:04 ↗
They would have. I'm sure.
Um, was there a technical like reason you just couldn't get the performance?
Anastasios Angelopoulos
9:09 ↗
Yeah, it just became hard to develop and there were all these tools that we wanted in React and, you know, to do all the fancy things that you can do in React became kind of like...
One example, I don't know, what's a feature that you really wanted?
Anastasios Angelopoulos
9:21 ↗
Let's say we wanted to create like our own custom like loading icons for video with notifications.
Anastasios Angelopoulos
9:27 ↗
How are we going to do that in React? It's hard.
Uh, yeah, I mean, make a custom component. I'm sure the Hugging Face guys are going to come in and say like, hey, you can do that in Gradio, which maybe you can, but also fewer developers know how. Are we going to hire for that? We have to reskill them. People are less familiar with that stack, you know.
Anastasios Angelopoulos
9:45 ↗
So it's full React, all this. Yeah, yeah, yeah, all that.
Okay, cool. Other use of funds that might be interesting?
Anastasios Angelopoulos
9:51 ↗
Like, uh, you know, basically that's basically deployed resources. Okay, yeah, it's primarily inference that funds the free usage of the platform and then also, uh, hiring, of course, headcount. Yeah, we have an office. You know, it's in SF.
I'll tackle one of the major things this year, which I'm sure you're tired of thinking about, but for people who are not in the loop, this is going to be news to them. Uh, the leaderboard illusion, the whole thing with Cohere. Let's summarize what they said and then your response.
Mhm. So, Leaderboard Illusion is a paper that critiques LMArena and the main...
Well, you know, I would say unscientifically. [laughter] Um, and let's be clear, Cohere wasn't doing that well on...
Cohere was like 74. It's all good.
You know, it's actually not. It's a respectable place that they had on the leaderboard. I don't even think it was really Cohere people, like the Cohere model developers doing this. It was more their research side. But in any case, what does the Leaderboard Illusion say? It says that LMArena was... what their claim is that LMArena was doing this undisclosed, quote unquote, private testing on our platform, that model providers will send us pre-release models and we'll expose them and so on and so forth, and that this creates so-called inequities in the leaderboard, you know, due to that pre-release testing. For example, they cited that Meta at some point tested a amount of models with us. Of course, we can't disclose all of the details of how all that was done, but that is the main claim of the paper. Now, our response to that paper, it's online. You can find it on response to Leaderboard Illusion. And our response to that paper is essentially pointing out a series of factual mistakes in the paper that question the validity of the claims. So, you can go look at the first version of the paper on arXiv yourself and you'll see the claims. I think most scientists would view that as...
Oh, they've corrected it.
They've corrected, of course, because we... I mean, but they didn't correct everything. They just corrected some aspects that were just blatantly unscientific and false. But you know, there, for example, um, said that we only sampled like 9% open source models and like, you know, 60% like closed source models and this created a gap between open and closed source. But in reality, we're actually really supportive of open source models and it was more like 60/40. And so that was, you know, one of the examples of an error in the claims. Another example is that they were claiming there was some sort of bias introduced by this pre-release testing and that it was undisclosed. In reality, as you probably know, we've been doing this pre-release testing for a long time. Our community loves it. They love basically getting the secret code names.
