Anastasios Angelopoulos
0:00 ↗
In fact, there was one problem that my wife and I had been trying to solve for a long time. Like she actually started working on this problem like early in her PhD. Now, she's a professor at Stanford by the way. It turns out there was this like longstanding open conjecture and uh it was recently proven by um an OpenAI model one shot, which is... and my wife was trying to work on it earlier in her PhD. Then she and I together had been trying over the past year, you know, just on weekends to try to work on it together. But one of our colleagues actually solved it using Soul. Um it it was a pretty amazing moment and kind of a moment that was like hey you know even the best theorists in the world are being outclassed by these models.
With Anastasios Angelopoulos again on MTS. This time he's our sponsor. So congrats.
Anastasios Angelopoulos
0:47 ↗
Congrats. Well congrats to us [laughter]
Karina.
It is our honor.
Yeah. Thank you so much for coming into the studio in real life.
Anastasios Angelopoulos
0:54 ↗
Of course. Yeah. It's great to see you guys. I hope uh it's one of many
Anastasios Angelopoulos
0:58 ↗
certainly.
So yeah, there's so much to talk about when with uh state of models right now.
Uh I guess we could start with Qwen 3.8 Max which just came out over the last weekend. Um it looks like a very good model. Um what are your first impressions of
Anastasios Angelopoulos
1:14 ↗
Yeah, first impressions are great. So just to like zoom out a little bit, there's a few different models that are coming out. There's Qwen, there's DeepSeek Flash before
and then there's the new Flux 3 model.
Anastasios Angelopoulos
1:24 ↗
Okay. So, we'll definitely talk about all three. But on the note of Qwen, you know, I'm starting to call this the Chinese invasion because it is just like a slew of these Chinese models open source that are frontier level models. If you look across the Arena leaderboards, what you'll see is that the Qwen Max model, the one that was just released, is roughly between the Anthropic models and the OpenAI models in terms of performance. So it's like Anthropic models, a stack of them of the most recent ones, then Qwen, then it's like 5.6 Soul.
Um, and you know, it's kind of hard to distinguish these with these between these models at the very highest level. So we'll need to see the scores bake, but that is true definitely among experts, um, among among coders, uh, and to some extent also among chatters.
Anastasios Angelopoulos
2:16 ↗
So we can go to the leaderboards if you guys are interested. Here I can share my screen. Cool.
Yeah, I mean we can certainly do that. Uh it just looks like with a lot of these like open source releases, they like end up on the Pareto frontier increasingly. So
Anastasios Angelopoulos
2:29 ↗
So check this out.
They kind of have to, right? Otherwise, why release it?
Anastasios Angelopoulos
2:32 ↗
I mean, I guess that's true. Yeah.
Plenty of companies are releasing models that are not on the Pareto frontier. For example, Inkling is not on the Pareto.
Anastasios Angelopoulos
2:38 ↗
Yeah. Yeah. We had Ink in Inkling like Neotron,
Anastasios Angelopoulos
2:43 ↗
I think there's a lot of labs that are sort of like trying to catch up and there's this dynamic America versus China that the American open source ecosystem is so behind that being number 10, which is what Thinking Machines was, is actually an accomplishment.
Right, I know, yeah [laughter]
Anastasios Angelopoulos
2:56 ↗
people kind of seeing that as like a positive signal that American open source might not be dead yet
um and that it might be having a resurgence in fact, which I do believe the commercial incentives will lead to a resurgence in this. But take a look at this. I'm I'm screen sharing here the Text Arena chat leaderboard. It goes Fable, Fable 4, 6, 4, 7. Um, and then [laughter] I love that
it's not because we didn't even capture on the stream.
67
Anastasios Angelopoulos
3:22 ↗
Fable 4, 6, 4, 7, 6, 7 and uh then uh Qwen 3.8 Max right there. So it's about Yeah, that that model also um likely soon to be open sourced,
Anastasios Angelopoulos
3:35 ↗
Um so incredible to see it. Number five model on the Text Arena. And so this leaderboard, by the way, is um is it's based on chatting, but not just chatting in terms of style. It's also adjusting for the effect of things like emojis and length and so on.
Oh, sure.
So it's like the more substantive aspects of a chatbot,
other languages as well.
Anastasios Angelopoulos
3:54 ↗
Other languages. Yeah. So it's very much multilingual. Um English, Chinese, Russian, Spanish, German, and so on.
Uh and so and then you get a bunch more Claude models and then Muse Spark and then Gemini. And then eventually number 15 is GPT Soul. Surprising to me that the best OpenAI model is so low. Uh especially below models like Muse Spark. The consensus seems to be that it's mid. Same with Gemini 3 Pro,
Anastasios Angelopoulos
4:18 ↗
That's because this is for chat chatbot related tasks. And then if you go to experts, the story is a little bit different, right? Because I actually use 5.6 Soul. And this is one of the questions that we got asked actually in the in the X thread where we were asking for questions like people asked their questions. What's the most underrated model that I love to use but that isn't reflected as highly on the leaderboards? For me, it's actually Soul xHigh. And that's because I do a lot of like advanced math when I do my chatbot tasks because like, you know, I have a background in statistics. My wife is also a statistician. So, she and I like to prove theorems together. It's like our bed, you know, bedside conversations. It's awful. It's so terrible.
Um,
yeah, I'm sure everyone's like vomiting.
Anastasios Angelopoulos
4:59 ↗
But uh but Soul is like really good for this. Like all these like theorems that are coming out, they're all using OpenAI models, right? And that's for a reason.
Yeah. Yeah. They just published their uh like a paper like August 1st about there are 10 different mathematical like
Anastasios Angelopoulos
5:14 ↗
some of which are a little bit questionable but yes.
Yeah. Maybe. Wait, can you speak to that actually?
Anastasios Angelopoulos
5:18 ↗
Well, yeah. I mean, I don't know. People have been saying that some of the proofs are not not quite correct yet.
Anastasios Angelopoulos
5:23 ↗
Um but it's still like an extraordinary achievement. I mean, some of these problems I've actually heard of. In fact, there was one problem that my wife and I had been trying to solve for a long time. Like she actually started working on this problem like early in her PhD. Now she's a professor at Stanford by the way. So she's like first rate in what she does.
Um and and it's this problem about a false discovery rate control that's you know basically you have many hypotheses like which genes genes cause cancer. There might be like you know hundreds of thousands of different options for associations between different genes and different types of cancers. How do you discover the associations that are actually true in the context where there's many many possibilities there? And so if you if you test each at like a hypothesis level like of 0.05 then the chance that you're going to discover one that's false is going to be really really high. And so people have methods for controlling these false discovery rate control methods. So it turns out there was this like longstanding open conjecture about whether the dominant method for FDR control false discovery rate control actually works for Gaussian random variables. And uh it was recently proven by um an OpenAI model
Anastasios Angelopoulos
6:31 ↗
which is and my my wife was trying to work on it earlier in her PhD then she and I together had been trying over the past year you know just on weekends to try to work on it together and then one of our colleagues you know it's a small area this like theoretical statistics thing so we know everybody there but one of our colleagues actually solved it using Soul um it it was a pretty amazing moment and kind of a moment that was like hey you know even the best theorists in the world are being outclassed by these models and so um it makes sense to me that uh Soul is number 10 on expert and you know right behind the Anthropic models um it's amazing to see Qwen 3.8 Max there too. We have to try it on some of these more advanced mathematics tasks to see if it holds up there.
Um, but of course the Anthropic the Anthropic dominance has been holding. I think Anthropic probably has the longest lead of any model provider in the arena.
Wow. Well, how does it perform? How does uh Qwen perform on math?
Anastasios Angelopoulos
7:24 ↗
Yeah, let's look at math. But this is this won't be like the advanced math. Um, and also it looks like there's not enough data yet here. So 3.8 hasn't been ranked yet. But you know, we'll see. We'll see once that's a smaller category. So we'll see once the more votes come in. Um yeah, it's been amazing to see. Now another area that would be interesting for you guys is code. Um so here it's quite interesting. We have the the Anthropic models.
Uh Opus 5, Kimi K3, Opus 5 High and then 3.8 Max is number four. So it's just you know alternating Anthropic and open source.
Anastasios Angelopoulos
8:00 ↗
And then OpenAI is number number six here with Soul.
Yeah, these models all seem quite neck and neck um in the Vibe Wars as well. Like I've heard um potentially Fable is still better at like you know very hard very long horizon coding tasks like um the Epoch Mirror code benchmark showed that like Fable was much better than other models at doing like extremely long and complicated coding tasks. So maybe it still has like that
Anastasios Angelopoulos
8:30 ↗
big model smell to it that other Qwen does does like famously pride itself on long running tasks.
So
autonomous protein synthesis.
Anastasios Angelopoulos
8:38 ↗
Autonomous protein synthesis. Yeah.
Anastasios Angelopoulos
8:41 ↗
Yeah. Well, I think that the um the thing that's most interesting about this is the price performance trade-offs because the leaderboard shows you just like the raw score. And it is true that like Opus 5 Max, I believe it's even statistically significant the difference between that and Kimi K3. It's, by the way, uh, good that they released this soon after Kimi was released because if it was released like three weeks later, I think that would have been a pretty long lead for Kimi. Um, but this was just like one one week afterwards. So, it shows that Anthropic is definitely like uh still in the race here with Kimi to win it all.
Yep. Um, but then what you see when you look at the cost performance trade-offs is that, you know, for the most expensive tasks, it's still Anthropic, that's number one. Then you go down the list and it's Kimi, it's Qwen, it's GLM, it's DeepSeek. And look at that gap for DeepSeek. This is one of the largest bends in the Pareto Frontier that we've ever seen.
Yeah. And this speaks to I think Brent's question um that he was asking which is like hey like you could barely see the cost on the plot you know when you when you know you looked at Andrew's Andrew's post and it's deprecated so many models like this model used to be the Pareto Frontier DeepSeek V4 Flash High preview deprecated like not like not relevant anymore. Milo
V 2.5 not relevant anymore. High three not relevant anymore. 28 cents for a million output tokens is just incredible.
It's so insane. It's so insane.
Um, so Flash is, uh, you know, for like a pretty big band of tasks between basically $5 to 20 per million tokens.
Um, it is the uh best model in the world.
So tell us about uh IBM Granite 4.18b. [laughter] We were actually just playing, we had the Arena Pareto Frontier on stream like yesterday and we were like, "What is this model? I've never heard of it. Why would people use it?"
Uh, I don't know that anybody uses this model. I think this is a pretty old model. Um, and you know, it's so funny with Pareto Frontiers that it's like easy to be on the Pareto Frontier by like pricing yourself at zero.
