Anastasios Angelopoulos
0:00 ↗
People are so fixated on cost per token. And at Arena, we really believe that you need to be evaluating things on cost per task. If you look at token costs, the leaderboard looks completely different than it does if you look at task completion rates.
All right, we are back. We're live with Anastasios Angelopoulos, who is the co-founder and CEO of Arena, our sponsor, the platform where people vote head-to-head on AI models to rank them, measuring AI performance in the real world, they say. Anastasios, welcome back.
Anastasios Angelopoulos
0:33 ↗
Thanks for having me. Excited to be here.
Absolutely. So, I guess we should start with Gemini 4 Argon, which just took number one in text arena. It's still behind on code. Some people are saying it's benchmaxed. It seems like Google's still pretty clearly behind the OpenAI frontier, but what are your impressions on this model?
Anastasios Angelopoulos
0:54 ↗
Yeah, so it's very interesting. I think that the Argon model is a great example of how static benchmarks are really failing to assess the actual real-world performance of models and why you need platforms like Arena. I'm going to share my screen here if you guys don't mind, to show you a little bit about what we see from the Argon model on our leaderboards. So I'm pulling now arena.ai up. You'll see it in a moment.
Okay. And if you go here to the leaderboard, you'll see that Gemini 4 Argon was just released. And I think the big headline that you're seeing from Arena is that it's number one on text. Text is a pairwise human preference. So for sure, whatever Google is doing is making the model better and better in terms of just people's raw preferences and also in terms of factuality, because the model is doing quite well on that front. But when you subset to experts, the model actually drops quite significantly. So it goes from rank one to number seven. Of course, the Anthropic and OpenAI models are dominating for experts. And if you go to agent, the agentic tasks in our agent arena, the model is number 10. That said, the top nine spots are all dominated by OpenAI and Anthropic.
So you'll see, but basically the Argon model is just like a solid number three. It's a solid number three. It's definitely a big improvement from the previous models that they've released. Better than Moonshot, better than SpaceX, better than DeepSeek, right? So better than some of the open-source Chinese labs.
Anastasios Angelopoulos
2:42 ↗
Mhm.
You subset to code, SpaceX and xAI goes up. All right, cursor, SpaceX, xAI makes sense. But if you go to chat tasks and work tasks, these are like Claude co-work style tasks, Google's actually quite a bit ahead. So I would say solid number three moment. As always, Google is competitive in terms of price-performance as well. So if we go to the code leaderboard, Google's model, it's actually not on the Pareto frontier, but it is doing fairly well. It's one of these models over here. I'm forgetting which one in terms of... yeah, there it is. Argon, price-performance pretty good, although not frontier.
Right. So how do you guys do the expert text benchmark, or what does the expert benchmark measure?
Anastasios Angelopoulos
3:39 ↗
Well, we have a number of... so you can actually click on our blog here, which I'll open. A number of signals that we look at to try to find experts in economically valuable areas, whether that's software, sciences, business, math, legal. And we identify and mine which users are experts based on categorizations of our data and following us over time. And so what we have is a set of criteria that we use. Basically, they have to be asking some of the deepest, hardest questions inside their area over a long period of time to be considered an expert, right?
And experts have substantially different preferences from normal users because they're pushing the limits of AI and they can tell the difference between good and great. And that's why we have a dedicated leaderboard for them.
Yeah, it's quite an interesting model. One of the areas that we've seen the model have room for improvement is in the consistency of the websites and 3D generations that it makes. So you should see a video on this on our YouTube channel, on the Arena YouTube.
Right. So on 3D modeling and 3D visualization, because I feel like that's something that recently models got really, really good at. Do you guys have specific benchmarks? And also it's important for making the model go viral on Twitter, like actually.
Anastasios Angelopoulos
5:02 ↗
Yeah, it's actually true. It's definitely important. Yeah. So we actually have a certain set of sort of like, you know, the Pelican benchmark.
We have like Pelican benchmarks that we release for every single model consistently. And Peter, one of the model capability analysts on our team, is releasing those for all the big releases on our YouTube. So you should go check it out there. But what you'll see is that the Gemini models, they're quite good. If it was six months ago, we would be saying, 'Wow, this is an incredible model, really pushing the frontier, like changing the world.' And right now, I think we're saying, 'Okay, Google's a solid number three on these topics.' But you know, that's a huge achievement and they're definitely coming back from where they were.
Anastasios Angelopoulos
5:54 ↗
Yeah.
So what kinds of things that Arena is not already benchmarking for would you consider adding to the Arena lineup? Like 3D visualization could be one, for example.
Anastasios Angelopoulos
6:08 ↗
Yeah, our goal is always to get deeper and deeper in people's real work. So definitely trying to add more categories here that speak to the economically valuable use cases that people have, whether that is academic research, whether that's accounting, whether that's software engineering. These are the sorts of things that we want to evaluate. But let me show you something else which is quite interesting. People are so fixated on cost per token, and at Arena we really believe that you need to be evaluating things on cost per task.
So even the way that we benchmark cost is something that we've continued to innovate on. Like here on the Pareto frontier for agent, you can see that it's Claude on the top, it's Soul, and then where's the Gemini model? I don't know. There it is over here. Not quite Pareto frontier.
But in terms of this Pareto frontier, it's telling you not just about the performance, but also about what's basically the ratio of how token-hungry it is versus getting jobs done.
And you can't just look at price per token for that. If you look at token costs, the leaderboard looks completely different than it does if you look at task completion rates.
Yeah. What kind of tasks are measured by this?
Anastasios Angelopoulos
7:31 ↗
So these tasks are trying to measure the general work capabilities of people with agents.
So that would be like you are trying to open your GitHub, open a new PR on your GitHub for your personal project. You have a web app that you're trying to build and you're building it and hosting it and so on using Arena. You are trying to review legal documents and make edits to them. You can use Arena for that. So it is a very broad, again, kind of like Claude co-work in your browser style experience. And if you go to arena.ai and you click on agent mode, that's what it will give you. You can see you can integrate with your Drive, your Docs, your Notion, your Gmail, your Linear, your calendar. You can select repositories, you can select harnesses and agents. So it is a pretty flexible platform.
I want to shout out our sponsor, Lovable. You know, the app you've been meaning to build, the internal tool, side project, or product you'd otherwise lose a weekend to. With Lovable, you get software you can ship today. That includes an editable codebase you can inspect and change, plus two-way GitHub sync. Lovable also handles the backend and infrastructure, including managed Postgres, object storage, hosting, payments, and more. So you don't have to wire it all up yourself. And through Lovable's MCP server, you can create and deploy your project directly from the agents you already use. Turn ideas into software people love with Lovable. lovable.dev. Now, onto the episode.
I definitely agree that cost per task is better than cost per token. What about somehow having a benchmark for cost per valuable economic output?
Anastasios Angelopoulos
9:19 ↗
Yeah, that's definitely where this is headed. I think tasks are a proxy for that, but we'll continue to build the maturity there. And the actual benefit of this is that it's all again based on real-world data. You know, we have like 350 million sessions over the platform.
Look, counting. Here's our counter. I love watching this. We're getting like 25 sessions at least, you know, every second these days. And a lot of these are, you know, making presentations, running web searches, back files. You can see that there's a live feed happening over here, writing files, making presentations. You see, it's quite interesting reading.
And there's out of that. What about benchmarks for things like UI design or presentations, like you mentioned?
Yes. So here's our webdev leaderboard. The web speaks to this.
Okay. Because webdev is basically, at the end of the day, for webdev people aren't looking at the code anymore.
Looking at code is going to be like looking at assembly.
Or looking at... this is low-key already the case for me.
Yeah. For sure. I mean, I look at code sometimes still, I have to say, but it's getting more and more rare. And so at the end of the day, a lot of the delta, the differences between good and great models, is going to be assessed by looking at the deliverable.
And some of that is visual. It's design-oriented and it's also functionality-oriented. Do the buttons work? Do they do what I want them to do? Is the text, you know, is the framing and branding correct based on what I wanted? And also, you know, there's an element of safety that we need to evaluate as well, because given the capabilities of these models to break out of their sandboxes, as we've seen with the Hugging Face incident, there needs to be a neutral evaluator for that.
I think there's some people doing it, but we're excited to think about those directions as well.
Yeah. Can you go into a little more detail about that? Like, how could Arena help out with cyber evaluation, for example?
Well, it's unclear. You know, I think that it's something that we really appreciate community feedback on. But, you know, I think that the capabilities of these models are growing so great that there needs to be... it's not a matter of if, but when, that there's got to be a neutral evaluation platform that speaks to this, because frankly the model labs are not incentivized to do it themselves. And yes, you know, the big first-party labs, the OpenAIs and Anthropics of the world, they're passionate about safety and, you know, kudos to them for doing that. But we need to create a system, and they've called for this as well, that they're not the only ones that are sort of making the guardrails.
Right. So wasn't there a model, I think like Theo, you might have sent me this, that compares or is able to fit to somebody's writing style or something?
Oh yeah. So I'm also curious about benchmarking this sort of ability to maybe sort of fine-tune or fit a model to a user's preferences, because over time I feel like it makes sense. Like I feel like I have skill files on my computer that are just about like make it look like me or make it sound like me, and I feel like increasingly this is seen as a capability and not just something the model can do.
