Hello everyone. Welcome back to SemiAnalysis Weekly. We got a big show this week. We got some heavy hitters. Doug, Max... anybody else out there? Oh yeah, of course, that's Dylan Patel calling in nicely in a nice setup office with his headphones on and a SemiAnalysis logo water bottle. Yeah, we are, we're going to talk about Claude Code, which is one of the only things we talk about on this podcast now. We're going to talk about GPT 5.5, possibly DeepSeek. And yeah, just get into it, guys.
Crazy. Today is... today's May 5th. GPT 5.5.
It's Max's birthday. Happy birthday, Max. Max debuted on something else's newsletter last week with his first...
Lead author article, let's say.
The coding assistant breakdown. The funniest thing is he was like, he's like telling, he's asking like Doug and Michelle like, hey, like, you know, am I going to be approved on my work trial? It's like, bro, obviously, like, what are you talking about? So also, Max is now a full-time employee at SemiAnalysis, not on a work trial.
Happy to be here. It's really exciting.
Working on the team. Okay. So let's run through this article. Basically, when we pushed it out or when we were prepping to write it, it was effectively just going to be a review of 5.5 Pro, or just 5.5, whatever the new... I think we assumed it was going to be called 5.5 when we were doing some testing of the...
Potato patata. It's a spud, right? Potato patata.
Yeah, we're calling it a spud.
Tried some different checkpoints, but the... I guess the basis for the article was like just going to be a review of that model, and then quickly Claude released, Anthropic released Claude Opus 4.7, which is a big change of some, you know, pretty interesting features that we covered in the article. And then we also got DeepSeek V4, the whatever four, five, possibly six month delayed release from DeepSeek. And so it became this article just about all the latest releases. So, Max, maybe you could give us like a quick summary. What was your high-level takeaway, and then we can dig into the details of the changes for these models.
Yeah, happy to. So the TLDR is that things were looking really dire for OpenAI for a while. You know, started the year, Anthropic was quickly encroaching in terms of revenue. Like the information leak, like 19 billion ARR for Anthropic versus just like 24 for OpenAI. Anthropic then sort of quickly surpassed them, even with all the sort of accounting discrepancies related to like, do you recognize net growth from the hyperscalers aside. I think it's pretty clear they surpassed them on a like-for-like basis in like early to mid-April. This was primarily because Opus 4.5 was just a real step change in coding and overall agentic abilities. And so from like late November up until end of March, early April, everyone was basically just spamming like Opus 4.5, 4.6 for all their workloads. OpenAI tried to fire back with GPT 5.4, but that thing was honestly just an embarrassment. Like in the model release card, they didn't even compare it to the Opus models, just to the past like OpenAI models. So that kind of tells you all you need to know. But then finally with 5.5, they're like, they're back on the frontier. Opus was re-included in the model release card. I wouldn't say it's like definitively better than 4.6 or 4.7, despite what the Twitter propaganda machine was trying to push on release date. But I think it's definitely like in the conversation. I'm happy to use it. And sort of it allowed OpenAI to come back in the game. You know, things were looking dire, but now I think they got a shot again.
Doug, what are you using? Daily driver. Are you using both or just one right now?
I have tried so many times to use Codex, but the usage limits, they raw dog me every day, bro.
Wait, what do you mean? Use the API, bro?
I use the API. Don't get me started. We already have...
How do you get usage limited?
I don't know, bro. And then I... then after it takes me longer than 15... OpenAI Slack. Just go... I have, I literally, dude. Okay. So, but then after it takes me longer than 15 minutes, for what it's worth, we're trying to get you started. If it takes me longer than 15 minutes, I don't try essentially to go actually fix it. GPT seems really expensive and blows out the context window, but I mean, in my impression, it's like, you know, it's very neck and neck. They are, they are pretty replaceable to me. The thing that I really want is someone to give me fast mode and high uptime. And that's, you know, no one is, no one is high uptime.
Wasn't there, isn't OpenAI's fast mode like fake news? Yeah, it's pretty fake.
Well, they... they like reduce the reasoning depth and it's not that fast.
Opus fast mode is not that fast anymore either. It's not even two times faster.
Sounds like we need more compute.
It started out at 2.5 times faster. Now it's less than two times faster. But OpenAI has like three versions of fast mode. They have priority mode, fast mode, and the 5.3 Codex Spark. 5.3 Codex Spark is definitely fake. That's just a different model. But then fast mode I feel like is comparable to Opus fast mode. You guys don't think it's actually faster or much faster.
I've been using it in Codex and it feels pretty slow. I don't even notice it like really being faster than with fast mode turned off.
Thought that's what you were showing me, Jordan, the other day, is like the distribution of like what's the peak tokens per second? What's the median? What's the trough? I thought you showed me, maybe it was someone else, but that like OpenAI's priority mode doesn't like actually, it's not always faster. It can be faster though.
Yeah. So priority mode is like guaranteed execution in an SLA, but not faster interactivity for people who have like, you know, they need stuff done on the API and they'll pay a premium for that, but it's a small premium. It's like two times. I think what I was showing you was the data that we have on 4.6 fast versus 4.6 base, which is it started out at, let's say, around 90 tokens per second per user on fast mode and around 35 or 40 for base. It's still 35 or 40 for base, but it's now at like 70 tokens per second fast mode consistently. So it's like not even two times faster for six times the price. The guys internally are like, I mean, Max, you made this point in the article, like for it, or we made this point in the article, I guess. This is like the first time where any of the engineers at SemiAnalysis have made the trade of wanting fast over higher quality tokens. And...
I'm not sure what drives that. Definitely people want fast mode, definitely feels faster. But maybe the big thing is that 4.7 is just not meaningfully better quality than 4.6 for people today.
Yeah, I was actually thinking about this earlier today. If you kind of treat in the context of the OpenAI's compute still, because if you assume that like Opus 4.5 with just this, like it passed some key threshold in intelligence/model capability such that a lot of your sort of day-to-day tasks are now just like one-shotable by the models, like you don't really have to supervise them at all. You're not even looking at the outputs. Maybe it's just the case that like even if you can never run anything larger than let's say like 100 or 200 billion parameters on servers, like you're going to have GPT 5.5 level intelligence in that form factor in probably like less than a year. And it might just be that we've actually like passed the inflection point where the majority of people do not need frontier level intelligence like for the day-to-day workload. And this is probably like doubly true if the models keep getting more expensive. Like originally I thought, same analysis, like maybe there's a chance where we'd be able to afford Mythos fast. I think it's probably just not true anymore. A lot of people have already been priced out of the models. I think we're on the cusp of getting priced out.
I don't think, I don't think we can, right? Because like...
7x, 51, 150, I think 25, 150 or 25, 125, one of the two.
5, 25, so it's five times more expensive.
And then fast mode 6x on top of that. If we were to take our current token spend, I can justify that. But if you were to even double it, I'd be like, maybe, maybe we got to turn off fast mode, guys. At this point, at this point, like it is, it is, it is like margins, you know, like matter.
Do you think there's a possibility that we could define something so important in SemiAnalysis research in the future that we would want to pay the premium just for one project or one task?
I think, I think the flip side is, is Mythos is more token efficient. Mythos fast mode is probably cheaper than like 4.6 fast mode for most tasks. I just imagine at least that's the case with like Codex versus 5.4 versus 5.4.5. You know, even if they make the model more expensive, you know, but that's like not a huge jump in price for the model. I don't expect new models to be like more expensive to do the same task. It's the problem of cost is like you're going to do new tasks.
They call that Jevons, dude.
Yeah, we're going to be coming up with new stuff to do with these models based on the work that we do in the next few months with the models. We'll come up with new things, new tasks that are harder and more complex and need to use the bigger models for.
Like... but I feel like right now at least, like I don't even think about is this task like worthy for me to spend tokens on. I just like spend the tokens. But with the models get much more expensive, you might have to think carefully, like is this task really worth me those fast token pricing? I think that'd be really sad, honestly, when it happens.
The question, what is the trade-off? Because I have a personal anecdote where it's more expensive to burn tokens than to do it. Whatever your examples of the cost is not worth it. Like there were times where I was just setting up like some benchmark on a digital education... $400, you know.
That wasn't worth 10 minutes of your time.
Yeah. Jordan, what's yours?
I think, yeah, I think stuff where you can very clearly do it from a script or writing, like writing docs, maybe the first pass with the model is good, but editing stuff, it's just annoying to use the model sometimes instead of doing it yourself because it might screw it up or might, you know, edit the wrong thing. And that's a quality versus cost conversation. What's this is like, wow, this is just a complete waste of tokens. Like the tokens in were two times more expensive. I don't think I've had that experience yet, to be honest.
Yeah. What's yours? Yeah.
Okay. So, scraping large data sets, at some point there's a diminishing return. You're like, hey, give me 10,000 employees from this company. And you're like, great. I'm going to have it hit the search API, various whatever, blah, blah, blah, blah. Like, I'll do a lot of variation to figure out a profile about every person who's worked at this company. And then you like actually run the cost and there's like a data enhancement API and like the data enhancement API is like literally one-tenth the cost. You're like, ah, like I think I burnt, I think I burnt $800 in tokens to do what would take maybe $50 to $100 of API calls. And one is like, okay, it's probably slop, and the other one is in theory verified by another slot cannon. And so like I just, I think there's some value like, and that's like probably one of the most interesting places where the replacement cost of the tokens versus the actual information. You know, it's still a lot cheaper to just like essentially serve data via an API, but like that cost, like the pressure on the top will move that down if that makes sense. Like just scraping the entire internet is just not token efficient at all. So, yeah, that's a good example where I just tried to boil the ocean using AI. And I've been really curious about like there's going to be a trade-off where the, like, you know, that it's just not worth this much intelligence, you know, like making me coffee. It's like the Rick and Morty meme is like, what's your purpose? Like, pass me the butter. Like, this is, this is a waste, man. Like, we got to find a better... better token efficiency.
Yeah. You can actually hire some people for cheaper to do some menial tasks than you can with tokens. But some of the analysis that we've done recently is like it kind of goes the opposite way so often that you just get used to tokens being the cheaper approach or the faster approach in so many cases that you don't even consider the alternative.
In aggregate, it's for sure. No way.
So what's the, what's the takeaway, Dylan? Is, is OpenAI so back at this point?
You think they're going to manage new release in two weeks? New release in two weeks. Everyone's releasing in two weeks. Google, OpenAI, maybe Anthropic. I don't know about Anthropic, but Google and OpenAI definitely releasing in two weeks. So...
What are they releasing? Pre-trains? More RL? Same base model?
Everything new, more continued pre-training because the Spud, they kind of like didn't finish the pre-training and just released it. So finish the pre-training, do more RL, drop model. Google mostly just gonna be like multimodal swap.
I guess like my question is like, like the narrative is that, you know, for even the normies who don't use fast mode, that 4.7 is worse than 4.6. Do you guys agree with this or no?
I think the instruction following has gone objectively worse. I think they, it keeps missing CLAUDE.md instructions or you pull a skill and you're like, dude, you didn't do exactly what was laid out in the skill. That seems to be a consistent problem. I don't know. I feel like it's a compute problem more than anything else. It still has the like, we've done a lot for today. You know, go enjoy your weekend. I was like, it's Monday. Like, get back to work. It annoys me so much. It tries to like enforce me to like stop working and I'm like, clearly usage issue. But I think, I think the 4.6 golden age when it wasn't quantitized in the beginning, that's when it was, those were the days. Okay. Fast mode 4.6 pre, pre the nerf. But I just think that there's this maturity of the models as they become more inference optimized and more people use them that it becomes a worse experience as you 10 or 100x the user.
And I think that that's happened. Yeah, I just think 4.7 is fine. I think 4.6, 4.7 is probably the same. It's just that it's... Oh my god, I saw that. It's kind of the same level of experience. Yeah, it's just too many users, bro. I need a NIMBY AI. I need a NIMBY frontier model where no one else uses it except for, you know, except for me so I can use, I can get it at a higher, a higher rate. I mean, that's the appeal of Codex right now, I think, is that in theory you should be able to have higher rates.
Can we, can we talk benchmarks? Like Max, when, when you saw the 4.7 release, for example, and we reviewed some of the benchmark scores, it was better on most, not better on all. And it led us to talk about like where benchmarks are useful and where they're not. So Doug's given the vibes like obviously individual experience can be different across people and it just drives their preferences, but like there should be some objective way to say 4.7 is better or is not better than 4.6.
Honestly, benchmarks obviously try to be that objective measure. I think they just like no longer are today. I would say benchmarks, it's like you need to be close to the frontier performance in order to have a shot at being like the true best model. But being like number one on the benchmark ranking does not necessarily imply that you actually are the best model. And so I would say like benchmarks today are most useful as sort of just like a vibe check to make sure that the model is not total trash. We kind of talked about this in the newsletter article, but it's surprising to me sort of how few people actually look into the details of the benchmarks to understand how unrepresentative they are of like real LLM use cases. I think a lot of people just hear a name like Humanity's Last Exam and they assume like, oh my god, you know, if a model can solve Humanity's Last Exam, that implies that it's like smart in all of humanity, right? We've like passed AGI or something. When in reality, you just like, you look at the individual questions and they're just the most like esoteric multiple-choice questions you've ever seen. They're not at all representative of anything you've like ever asked an LLM to do. It's very intentionally multiple-choice to make verification easy, even though like obviously when you like use in real life, it's open-ended. And this like even applies to benchmarks that on the surface you might think would be better like SWE-bench 2, where it's like, oh, you know, coding is this verifiable task. Surely you can just come up with some coding problem for the model and then write some like tests that verify if it was successful or not. But then you dig into the details and it's like, oh, it's actually really hard to write a coding problem that is both naturally worded, yet still like perfectly unambiguous with exactly one correct solution. And the SWE-bench problems, at least for the original version, they do not at all like fit those criteria. They just scrape GitHub issues. And for all the developers listening know that like GitHub issue descriptions are not meant to be like well-scoped tasks that you just copy-paste and give a model. Similarly for the verifiers, like they often include lots of unit tests that are scoped like particular implementation details, going as far to like ask the AI to output a specific like 20-word error message that's not at all mentioned in the task description. And obviously like later versions of SWE-bench tried to solve these issues, but still not perfect. I think really just underscores a lot of the issues with benchmarks.
Makes sense. So let's talk about 4.7 specifically. Like there was a few things that improved or changed, let's say, in terms of like features as opposed to just the benchmark itself. And I think you made the point in the article that like the people don't necessarily care as much about the quality of the model anymore. It's the model plus the harness which is the product that should be tested as opposed to the model on a generic, you know, bash-only harness or something like that. Harness meaning Claude Code is the thing you're testing, not Opus 4.7. So it's Claude Code versus Codex. It's not Opus versus GPT. So to that end, like during the release for 4.7, they announced an extra high reasoning effort option that slots between high and max. They announced high-resolution image support which people can use to like use for screenshots and styling and front-end applications. They're omitting thinking content by default so people won't see when the model's thinking. They've got this task budget concept that you can tell the model how much it should think or work before it can actually like run out of context in the context window. And then maybe most importantly, they updated their tokenizer. So it potentially costing people 35% more for the exact same output from the model as a previous model just because they're counting tokens differently and they're, you know, using it. Well, that of course like implies that there's something... Sorry, I'm just laughing at Dylan right now. Dude, I literally put him to sleep with that monologue. Holy... What the...
Oh my god. Okay, well, man, come on. Hold me out here.
What are you talking about, man? Tokenizer.
Dude, tokenizer is so boring that it puts Dylan to sleep. That's the takeaway.
It's not like the pod was like compelling and great at this point, but still...
It is now. I'm compelled. Now we can talk about all the... we want. The tokenizer. Super...
What's the Michelangelo painting or sculpture where the guy is like...
Dude, could you actually not hear us through your headphones?
I mean, he's been, he's been asleep this entire time.
I was just thinking, man.
You've been gone for, for a few hours.
Call me, call me, call me GPT 5.5x high. Just thinking right now. Tokenizer, Opus 4.7, Opus 4.6. We've been dancing around the fact that GPT 5.5 is fine, not good, definitely good enough, lots of capacity, they'll catch up on the margin. Sounds good. Great. Anything else? Benchmarks suck. That's a really good pick.
What's the tokenizer doing? What's the tokenizer difference?
They changed tokenizer between 4.6 and 4.7.
This, the exact same output could have 35% more tokens with the new one.
Oh, they made the tokenizer... they made the vocab smaller. They made the vocab bigger for 4.7 compared to 4.6. There's 35% more tokens in the tokenizer, like more vocab.
So wouldn't that make the average output smaller because you can represent longer, you can represent longer things with fewer tokens?
