Ο Αναστάσιος Αγγελόπουλος, CEO της Arena, στο «ΠρΑΙμ»

You know, Greece for me, it represents an extraordinary cultural heritage that I'm really proud to be a part of. There's all these amazing Greek scientists. There's, you know, Aristotle, there's Archimedes, there's... you know, all of us as Greeks are very proud of this, I'm sure. You know, everybody watching is just an amazing... It's a blessing to be part of this culture. And Greek, you know, the legacy lives on. I think that Greece still represents the birthplace of democracy and the birthplace of modern science. And there are many Greek people today that I continue to admire. Of course, the Demis Hassabises of the world and Anastasis Germanidis and Yiannis Antoniou Glou from the current AI wave. These are great people that are changing the world, even today, Greeks globally. So, very proud to be a part of it. And for me, you know, my journey in science probably started with my father as a physicist who taught me the beauty of science and mathematics. He's an amazing scientist himself. And then I don't know exactly how I found myself here. I just, you know, I certainly didn't know that AI would explode in this way. And sort of by chance, I found myself in the center of this incredible competition just because I was trying to enter the world of science because I wanted to help people and build tools that would help humanity. And because you can build that, new ideas can really change the world in science. I was attracted to this. But I had no idea how far it would go.

Well, I have to tell you, it was a complete surprise to me. But when it happened, I just grabbed it and didn't let go. And that's how I ended up here. I was at the beginning of my PhD, I was basically a theorist. The product of my work was theorems, mathematical theorems that tell us about the foundations of reliability in AI, how to use AI safely and responsibly. And it's through that that I got into the research area of AI evaluation, which is all about how do we measure the real-world capabilities of AI so that we can understand how to safely use it and to use it in a way that benefits humans. And that's why we started this project that at the time was called Chatbot Arena. And it was really only supposed to be a paper. It was supposed to be an academic publication. But then what happened is that at the same time that we were working on this, it was right after ChatGPT, Google and OpenAI and Anthropic were all fighting in order to build the best AI model. And they were fighting on our platform. Basically, we were the neutral playing field where everybody was looking to say who's the winner, who's the loser, and how good are they, not just on answering multiple-choice questions, but on real people, basically the voice of real people. And we were passionate about this mission.

Well, as I was sort of alluding to this earlier, but the thing that distinguishes Arena from a standard benchmark is that Arena is based on the voice of real people. It is a post-deployment evaluation of AI. You put AI in the real world and then you measure how humans react and whether it's providing value to them in reality. It's basically the difference between taking the test in school and then you put someone in the real world and see how they perform. And Arena is the latter. And it's really, we're very unique. We're the only benchmark that operates in this way even to this day. And therefore we fill an important role in the world because it's not just incentivizing AI to take a test well, but rather by setting the North Star, we are incentivizing all of these companies in order to provide value to real people. And because reality is the only benchmark that you can't game. It's the only benchmark that you can't manipulate. Because reality is grounded in objective truth.

There's many factors that go into the score that you see when you go to arena.ai/leaderboard. One of them is the preference of humans. But there's also all of these different factors that we're building into the system. There's whether the model can be steered correctly, there's its ability to generate code, and we're building more and more of these signals that actually go beyond human preference these days. Some of these have been released already. We have a code arena where you can see not just if, you know, humans like the way a model talks to them, but whether the model's producing code artifacts and websites and full-stack artifacts that are doing a good job for people. But we are also releasing a lot of this stuff in the coming days and weeks.

It's very important that we keep the dynamics of the ecosystem in mind as the central arbiter. Basically, we must be a fair and balanced judge. That's the most important thing on our platform. And so one of the things that we've been doing is we have been looking at why do people vote the way they do? What causes people to prefer one model over the other? And sometimes it's style, and sometimes it's content. But we created the statistical methodology called style control. And what style control does is it looks at all the stylistic factors, the response length, it looks at the number of headers, it looks at the sentiment of the response, and it looks at whether the response is sycophantic. And it takes all those factors into account and regresses them away so that on the main leaderboard that you see on arena.ai, you'll see a leaderboard that is corrected for the stylistic factors that go into human preference. Even though I actually believe that these are important, you can see them on a separate tab.

Yes, it's a way of trying to be clearer about the fact that we expect the performance of models to be grounded in objective truth, not simply the convincingness of the model.

It's a great question. Obviously, as the judges and the auditors, we have a huge responsibility, and a lot of it is actually up to us to make sure that we build our system in a way that is fair and balanced. But also, there's a huge amount of scrutiny on us. It's just like a ref in a football game. People care what you say. And so, there's a lot of emotion that goes back and forth, and that keeps us accountable. Basically, we have a community, whether that's the model providers themselves or whether that's just random people on the internet that are always looking at our results, and for better or for worse, they're always telling us what's going well, what's going poorly, and how we can improve. And engaging with them is a huge part of maintaining trust in our platform.

Absolutely. We are working on that now, and in fact, if you go to arena.ai, the majority of our population on Arena has access to something that's called agent mode. And agent mode is something that goes beyond chatbots. It's evaluating end-to-end agents that can do economically useful work for you. And it's actually not pairwise preference anymore. It's a single-threaded chat where we analyze based on the properties of your conversation how useful the agent is for you. And so, what is an agent? An agent is an AI model plus tools. And those tools are things that allow it to, for example, have access to a full computer. And they have access to bash commands, and they have access to the ability to create slide decks and websites and CSVs and Excel files. And in so doing, they can actually produce work products for you. Let's say you want to do your financial modeling, or you want to make a presentation, and so on. The agent can do all of this for you, and we measure along the way whether it's helpful, whether you're frustrated with the responses, and whether it's, you know, difficult for you to get the artifact that you'd like. And these are really rich signals that we're excited to expose to the world because they speak to the real utility of AI to people.

Of course. Absolutely. Without an evaluation, you don't know where to go. The evaluation is setting the North Star of the industry because it is telling AI what to optimize for. It is the incentive system. It is basically the natural selection is the evaluation because it determines which AIs survive and which ones die. Therefore, in this climate, do best.

Well, it's certainly going to be a competitive space, but our goal at Arena is to provide a global evaluation platform that takes into account the opinions of everybody. We have a huge global population. We have many, many users in Greece. We have many users in the United States, you know, in India, in the continent of Africa, and all over. And for that reason, you know, the benefits of the type of evaluation that we have accrued with scale. By building a platform that can really be a big tent and include everybody's voices, we can act as one. And I think the way that I would see it if I were somebody in the, you know, global population that might not be a technical expert in AI, I would say it like this. We think a lot about AI, but what we should really be thinking about is people. And the reason we have to think about people is because ultimately, what's the point of AI? It's to help people. It's to give people value. It's to help them with their work. It's to make our society better, more efficient. And that's really what our platform is meant to measure because we put humans at the center. It's the global population that's using the platform. And it's their voices that shape the leaderboard. And in so doing, they're creating the structure and the infrastructure, as you pointed out, that's going to incentivize AI for human value and ground it in objective reality.

I don't worry about things like that. I only worry about the science. I worry about doing the best job evaluating AI and placing humans at the center. And I just want to fulfill the mission of our company and I have no concern about ourselves or unseating ourselves. And if somebody does a better job than us at our mission, I'm happy for them to take the lead.