Probably that would be a correction because humans are emotional and given the magnitude of the capital which was invested in AI. I think that any correction, even a small correction, will have a reasonably big impact. I am choosing all the top ones: Anthropic, Claude, I'm using OpenAI, Google Gemini, and for the most important tasks and comparing them, so you have to make a bet and hopefully if you are right, then you are going to have tremendous advantage because others didn't do it.
Hello guys. So we are now at the summit in Paris and here with you Anna Tutava, founder of Cryptomines, and our guest Ion Stoica, who is a serial entrepreneur and as well he is a professor and he is co-founder of Databricks, AnyScale, and he has now his new AI startup, Arena. So great to have you here.
Thanks for having me. So you've literally run all your unicorn companies. So I guess now your biggest focus is with the Arena.
Yeah, you can say that, you know, on the front barrels. Yes.
Yeah. So basically you allow testing of different AI models on your platform. So can you tell more about that? Yeah.
This is coming from a research project we had at Berkeley started in 2023, and when we were trying to fine-tune models on LMSys, and we wanted to evaluate how good are these models compared, and we actually released the models as chatbots. So that's the problem we try to solve with Arena. We started from that need about how do you test a chatbot against another chatbot, and the main idea we came up with is that you have people coming to the site, asking questions, and then we give them answers from two random anonymized language models, and then the users can pick which one is better.
So based on this information we compute a rating very similar to Elo rating like in chess or maybe ATP in tennis, and that became very popular. We had more and more models from big model providers like Google and xAI coming and testing their models before being released. And you have quite a lot of model selection. You have both American models like you mentioned, OpenAI, and you have Chinese models. Yeah, we are testing over French models. We are having Mistral and so forth. So we've seen hundreds of models.
Do you have any specific preference on your own which model is what?
I think it depends on what you want. Some models are better in coding, some models are better in creative writing, and so forth. Typically I use more than one and then I see the big differences. The top ones I'm trying: Anthropic, Claude, I'm using OpenAI, Google Gemini, and for the most important tasks I compare them using Arena, because for the same questions they are going to emphasize different things in the answers even if all answers are high quality, and it's good to contrast the answers to get an even better answer.
What's your business model with Arena?
The business model is quite simple. We provide free access to the most powerful models. So you can go there and you can use any of these models to see which models are better for your tasks. And then we get in exchange the data you share with us, the prompts or preferences like voting, and based on that we build data products, in particular evaluations for model providers, helping them not only evaluate but also improve their models.
And can you share the future plans for the development of Arena? What's next for you?
I think you can see from the announcements we had recently. It's a lot about agentic. We have a new evaluation for agentic workloads. We have a new offering enabling people to use the agentic mode on our site, similar to like OpenAI's o1 and so forth, which is very popular and going very fast. That's one angle. The other angle is to always refine our evaluations to be more precise and more accurate. That's a continuous effort. In addition to this evaluation driven by human feedback, we also build our own models to simulate the users so they can predict user preferences. So now we can use this kind of model for evaluations not only by humans, and also we are using this for other products like intelligent routing. So you get a prompt to pick the best model for your prompt, maybe under some cost constraints. Actually our users already have access to it, it's called Max Out.
And as well you yourself are a big supporter of open source models. So what challenges with them do you see?
I think obviously you want open source, people will not use an open source model that is not open source at least in most cases. The opportunities and challenges: they have to be very high quality. We do have many open source models which are very high quality, a lot of them coming from China today. But I think that we are going to see, at least with the current restrictions to some of the most powerful models in the US and the announcement from China about maybe having export control on their best models, I think you are going to see more and more organizations and enterprises working on open source models because they provide the most reliable access. They can run these models themselves.
And at the same time we see a lot of Chinese models are open source while American-based companies are closed source. Where do you see this rivalry between US and China and who leads the race?
