Anastasios Angelopoulos
0:00 ↗
There needs to be a neutral player that's in charge of incentivizing AI to benefit people. That is always going to be our mission.
Arena just crossed a hundred million in ARR in eight months, making it one of the fastest growing companies of all time. They raised a hundred fifty million at a one point seven billion dollar valuation to become the referee of the entire AI industry. In 2023, it started as a research project at UC Berkeley with the simple idea of measuring which AI models people actually prefer. You type a question, two anonymous models answer, and you vote on the better one. AI labs used to just tell you that their model was the best, but now Arena has become the neutral source of truth for which AI is actually best and has turned that into a billion dollar company. The person running it isn't your typical founder. Anastasios Angelopoulos didn't set out to be a CEO. He was actually a PhD student at Berkeley studying the math of uncertainty advised by Michael Jordan, one of the most cited AI researchers alive, and Jitendra Malik, a founding figure of modern computer vision. Anastasios' own work is focused on conformal prediction, which is essentially the science of how confident you should be in a prediction. And he wrote one of the most widely read primers in the field. So, he spent years on the math of measuring things rigorously and then built a company whose entire job is to be the best at measuring AI.
Maybe even just like zooming out a little bit more for people who are at a high level, understand AI, understand what evaluations are, but don't really understand the significance. Like, why are evaluations so important?
Anastasios Angelopoulos
1:31 ↗
Well, evaluations are very fundamental because what they do is they're telling you the definition of good.
Anastasios Angelopoulos
1:37 ↗
Like, an evaluation tells you, "Hey, if you're doing well on this metric, that means that your model's a good model." And so, in some sense, they're setting the North Star of the industry. By picking a good evaluation and climbing that hill, you better make sure you're climbing the right hill.
Anastasios Angelopoulos
1:51 ↗
And so, that's what Arena is all about, is making sure that we're climbing the right hills in AI, incentivizing model labs to benefit real humans. And our philosophy for doing that is to put AI in the hands of real people and then measure whether or not they're getting value out of it.
What were the flaws with existing evaluations at the time?
Anastasios Angelopoulos
2:09 ↗
Well, at the time, those evaluations have their place, but I would say that people were thinking about evaluation in a very classical way. They were thinking about it in the old school machine learning algorithms like classifiers and saying, "Hey, how well is this language model doing on multiple choice question answering?" Like if we have it sit down and take a test and fill in the bubbles, how good is it? But it's just like students in school. How they do on the test is not necessarily reflective of how they do in real life. And so what we found is that you put these models in front of people, and the model that did well on the test is not the model that does well in real life. Cuz a user goes and talks to it and it's like, "This model's crap."
Anastasios Angelopoulos
2:45 ↗
It does well on the test, but it's like impossible to interact with.
Anastasios Angelopoulos
2:49 ↗
And so that's what we noticed and that's why Arena sort of emerged as a platform in a different evaluation paradigm that's all about how do you measure real-world value when the model's in the hands of actual users.
Before something like an Arena existed, these labs were just publishing, self-publishing some of these benchmarks. You kind of had to trust them. And now with Arena, what does that ecosystem look like?
Anastasios Angelopoulos
3:11 ↗
Well, with Arena, what happens is that anytime a model is released, we'll immediately be evaluating it on Arena. And what you can do is you can go to the leaderboard and see how well it's doing on basically every subcategory of real-world task usage. So, math, coding, instruction following, multi-turn, all these different occupational categories like legal and medicine and software engineering. Agentic coding, we have agent mode that we've recently released. And agent mode is collecting multiple millions of traces, like agentic traces with like thousands of tool calls per session, where you have people using these agents on Arena and then we're measuring how good they are at, you know, all sorts of different things. Hallucination rates, objective task completion, basically one source of, you know, ground truth information about all of these different factors.
Started obviously as this research project. When did you know you wanted to start a business?
Anastasios Angelopoulos
4:05 ↗
Well, at some point we realized that we were really passionate about this mission. There needs to be a neutral party that's in charge of measuring the capabilities of the different AI systems on real users, communicating that to the rest of the world, acting as an interface and a source of information between the broader AI community, developers, businesses, and the AI labs. And that if we did this properly, we would fill a really important role in the ecosystem, which is incentivizing these providers to benefit humans. And once we realized that, then we thought, what's the right way to do it? Should we do it nonprofit? Should we do it as an open-source project, so on and so forth. But we realized the only way to really get the resources that we needed to do this effectively was to start a business.
With AgentMode, I want you to elaborate a little bit more on what you're, you mentioned a couple of things, but elaborate more on what you're measuring, and what do you feel like this unlocks that wasn't possible before?
Anastasios Angelopoulos
5:01 ↗
AgentMode is the premier evaluation for agents. It is essentially the only real-world evaluation that exists for general-purpose agents. And what makes it special is that we have this huge platform in arena.ai, which is actually one of the largest consumer AI apps in the world. And our users are on there using agents every day. I'm talking millions of traces a week. And it's growing like 10% week-over-week.
Anastasios Angelopoulos
5:31 ↗
And all of these different categories of usage, people are doing workflow automation, people are doing software engineering, people are doing document analysis, you know, research and lookup tasks. All these, you know, basically professional tasks that are relevant to not just developers, but also organizations and businesses, just the broader, you know, scope of agentic work. And by using agents in that context, we're able to measure how useful they are for all these different types of tasks. And what we do is we mine signals through this process of causal inference. We call it causal tracing. Because what happens is that we look at every trace, and then we take all these explicit and implicit sources of feedback throughout, and then we run causal inference pipelines on top of them to help understand what's the effect of different subcomponents of the agent. What's the effect of the harness? What's the effect of the orchestrator? What's the effect of different tools? Are they increasing or decreasing task completion rates, hallucination rates, steerability rates, so on and so forth. And we're able to see all of that in the data through pretty complicated LLM judge and, you know, conversion-based pipelines. And we're able to report that to the world.
I want to talk a little bit more about scale, and I'm going to pull up the numbers cuz it's pretty impressive. You all now have 10 million monthly users, 150 countries, 700 million conversations to date. Curious what that scale unlocks for you, and it sounds like, you know, a lot of times users don't even know that they're contributing to this AI research. Sometimes they're just chatting. It's a very like dynamic, seamless experience. Break down a little bit more how the platform actually works.
Anastasios Angelopoulos
7:02 ↗
We try to be very transparent with our users that when they're coming in the site, they are participating in this evaluation ecosystem. That's part of the value transfer. That by using the platform, they are contributing to these public benchmarks. And our users love us for that, actually. They love going to the leaderboard and looking at what's happening and, you know, being able to contribute to the future of AI. Linking together evaluation along with the real usage has really exploded tens of millions of monthly users.
Anastasios Angelopoulos
7:28 ↗
Hundreds of millions of conversations. In terms of votes in battle mode, tens of millions of votes. What that means is that the moment a model comes out, we're able to evaluate it very quickly. The speed of evaluation is probably the number one biggest effect of this scale. Because if you think about it, if our platform is 10 times smaller in scale, it means we're 10 times slower in being able to evaluate a model, which is why the sort of network effects accrue to our business. Because the bigger and bigger and bigger we get, and it's not just about scale, it's also about quality. Our users, 28% of them are in software engineering. Tens of percents of them are in areas like math or legal or medicine. Tons of people in business. And so we focus on those sort of like valuable consumer areas and growing those areas where we really want high-quality evaluations. We're also then able to take all of that data and train evaluation models on top of that. Models that help us tease apart finer and finer grain signals about what makes AI good or bad, what are its strengths and weaknesses.
Another part of that scale means that you all have gone from again that research project to now 100 million ARR in under 8 months, which is absolutely insane after launching Enterprise recently. What do you think, just as CEO of this company steering this, what do you think has contributed to that success and trajectory?
Anastasios Angelopoulos
8:47 ↗
I think the revenue milestone, of course, we're so proud of it. It's like unbelievable. I never thought that coming from just a student project at Berkeley that we would ever be here. But what I think it really speaks to is the focus that the AI community and labs have on real utility to people. Because that's what our platform can provide that other evaluations can only do indirectly. If you think about it, the only way to measure real value to people is to put AI or AI agents in front of them. And then measure whether or not they're actually getting the value that's purported. And that's exactly what our platform does. And because we're able to loop together the AI developers along with the people that are on our platform in order to measure these quantities, of course, we're able to provide quite a bit of value that we can also turn into revenue.
The space is evolving so quickly. You all have done a very successful job. We've seen this paradigm shift towards agents, launching agent mode. How do you keep up with all of that change and know where to build?
Anastasios Angelopoulos
9:49 ↗
Well, it's very difficult. The space is moving extremely quickly. The only way to keep up is to build an incredible team that's able to turn on a dime because things are unexpected. You know, who could have known that OpenClaw would have in January just like become the most viral open source project in history. Nobody could have known that. Who could have known when the capability jump would have happened that made coding agents useful? Of course, you can do some level of prediction.
But predictions are so often wrong that you just have to be ready to like be humble and look at what's happening and say, "Hey, that's where the world is going. What am I going to do to respond?" You go to the team, you say, "Hey, here's what's happening. How does this change our roadmap?" And then you just whoop, which is how agent mode happened.
New agents, by the way, were going to be a big deal. We just had no idea that it would explode so quickly. We thought we had a few months.
