You guys ready to talk about the end of the world?
Yeah, dude. What's your P-Doom, Joey? Sorry that...
I'm a linear extrapolator, Max. I can't think that far through. That's too far up here in capabilities. I don't know. I think we get like, I don't know, we get a major hack or something. People freak out. I think it starts there. The P-Doom, like, I don't...
Joey, even if it's a linear extrapolation from the last four years of experience that you've had.
You think that P-Doom by the end of your life is impossible?
No, it's not impossible at all. I mean, you get it in like, you could get it in four years easy.
With a linear extrapolation. Yeah.
That's a hot take, dude. Dude, linear from like, linear from the start of like last year. You're like, even from like 20, like late '22, you're just like, I guess we're in 2026 now, so I guess four years. Yeah.
Joey's like, 'Man, just look at the ARR net additions, man. This is... I've never seen a number so big.'
I gave a comment to the LA Times today. Like, they were, they called me up. They didn't call me up. Like, I called the guy up and I was like, he's like, 'Hey, do you have some stuff on this?' He's like, he's like, 'Well, Ed Zitron said like, you know, if you know, if this stuff...' I was like, 'You put me on the record, Ed Zitron is a...'
I was like, look at, look at, like, look at any past call of Ed Zitron.
The track record is crazy. Um, but okay, I think the, what I've, what I've now learned is that we have a new benchmark where we can measure model capability progress. It's called the Joey benchmark and it's just ARR.
Like, we've seen this now twice in like the last month, not, maybe six weeks or something. The only thing that matters is people's future view of Anthropic and OpenAI and the pace of what's going on versus expectations, and the entire semis trade is dependent on that, and it's going to be dependent on that, and it's a lot of, it's a lot of the, uh, makes up a lot of share market cap.
But you got, you got to add in Azure plus AWS year-over-year revenue growth. These are the only four that matter in the history of the universe. Oh no, they dropped Azure. You're telling me these are the only four numbers that matter, bro. Now, now...
Growth guide used to be the number. It's going to be the Anthropic net new ARR guide.
Is AWS out too? We only care about Open plus Anthropic net new now.
All they're going to care about...
Two numbers with the entire face of capitalism.
The hyperscalers are like a level abstracted away from the end demand.
They're literally renters of compute, just glorified neoclouds.
Yeah, they have, they have a very nice business. They do other stuff.
AWS and Azure are glorified neoclouds now.
Yeah, put it on the record.
They're dumb pipes. That's what you used to say like in the internet, like the telecom, you know, they're dumb pipes. That's what Amazon and Microsoft and Google... You know, Meta is neocloud apparently. Yeah. Jeremy says a few gigawatts of hundred billion dollars of open source domain, right?
So, are the, okay, are these, um, is this a timing thing or are these companies so dysfunctional that their research organization buys from neoclouds and then their cloud organization sells to other companies' research organizations? Where like, like Meta goes and strikes up a deal with CoreWeave and then Meta's compute goes and sells their compute to OpenAI or something.
Well, they probably, okay, they might sell it to OpenAI to do like short-term tasks, but you know, it's still preferable to, uh, start building the compute because in the event that Meta does well, you can sort of claw it back. Uh, or they could also, you know, monetize open source tokens as I think Jeremy's claim, uh, on that similar short-term time horizon. Uh, I actually don't think it's cognitive dissonance because the key really, the important thing which we've harped on before is just the potential to have a ton of training compute in the event that your like research lab starts doing really well. Um, and I think any short-term monetization strategy is not incompatible with that.
Even if the short-term monetization strategy requires you to lock up a bunch of compute on a five-year deal this month, despite the research organization going and buying stuff a month and a half later, basically.
If it's a five-year deal, it's not short-term, right? I'm saying it's, it's totally fine for Meta to try to build as much longer-term capacity now, but then, you know, sell it to OpenAI on a six-month contract. Yeah. Sorry. It's, it's like five month, it's like five-year terms at the price, but with 90-day cancellation rights. And so it's a short-term contract.
Which I, the price is also, at least in Elon's case, it's definitely not the five-year price either, right? It's like the six-month price times two or something.
I mean, I'm seeing GB300 quotes from this morning to a large lab for an hour for three years.
Which I mean, initially...
You can't be leaking that much alpha in the SemiAnalysis Weekly podcast, you know.
Buy the AI cloud TCO model and or the tokenomics model if you want to hear the actual number.
Yeah. I know for this, for the specifics, but it's this backwardation of the curve where like if you start six months from now, you get a much lower number, but if you want to start tomorrow, you have a really high number. And, um, my initial reaction is like, oh, this is just an F-off price from the provider, but then we dig in. It's like, we know the provider and we know they have capacity and we know they're trying to sell it next month. This is the strategy is like they had the financial position to be able to hold on to the compute, I guess, not need to monetize it until it's actually installed or a month away from being installed, and then they can test the market at a really high price and try and figure out what the, what it's going to close at. And I think there's potential people jump on it strictly because they get like a two, three-month head start over what a much lower price would be. Like pay a 20%, 30% premium on a three-year contract to start two or three months earlier. I think people might make that trade right now. Especially with all of your justification of like they can monetize it immediately at a higher rate. Does it really matter what your entry price is if you can like immediately monetize and return on that? It's just pushing prices higher, that's all.
Yeah, I think that's right. Uh, freely available gigawatts are definitely extremely valuable right now.
Okay. Okay. So, how do we, um, let's get back to the actual, uh, topic of the conversation, which is like, we must pace the frontier. A blog that Dario put out, which builds on a lot of, uh, you know, discussion that's been ongoing, that then Sam agreed with, and then David Sacks responded to, and we'll kind of go blow-by-blow here. But one of the big things, Max, that I think you clued me into is the idea that, well, okay, I think a lot of people saw that there's not specific language in here saying like we're going to stop doing training runs and we're going to stop buying compute. Like the most specific commitment in here is that they're going to have embedded evaluators, third-party people such as METR come in and assess the, um...
Yo, Douglas O'Laughlin is joining the pod.
Hello everyone. Welcome back to SemiAnalysis Weekly. Uh, this episode is an emergency. So, I've got Doug, Joey, and Max, the three brightest minds in tokenomics, to talk everything about pacing the frontier. Uh, Dario put out an amazing blog post. We love his blog. And we are going to talk through the implications of that as well as the responses from people like Sam Altman, David Sacks, and, uh, some of our takes on how this affects, you know, compute markets, ARR additions, all of the important stuff that the finance guys listening to the podcast are going to care about. How does Dario's blog affect Anthropic's decisions to add compute? But guys, welcome to the show. Excited for this one.
Thanks for having us, Jordan. Joey's here to represent the interests of Defiance. He's going to answer all your questions, all their burning concerns.
Oh, I'm, I'm going to go somewhere between the two.
Usually, you can tell the finance bro on the call from the fact that they're wearing a collared shirt, and Doug is, uh, you know, passing that initial criteria there. But Joey's got a nice Firestone, you know, hoodie/windbreaker thing going right now.
Which I think also qualifies as like a finance bro shirt, right? I feel like 80% of the time I see Joey, he's wearing some sort of golf-related attire.
All right, so let me, um, let me give it a bit of an intro. I'm sure a lot of people have seen this blog post, but, um, we must pace the frontier is basically, yeah, a whole description of Dario's take on how AI could have some incredible benefits but also have incredible risks and how we need to take it seriously and how we need to build the, you know, controls into the systems to make sure that we usher in, um, you know, the next evolution of consciousness in a responsible way so that we don't kill everybody and end the world, for lack of a better word. Um, there's some specific proposals in here. Uh, first of all, embedded evaluators, then democratic coordination, and third, global coordination. Um, I say specific, I mean the last two are not that specific. The first one is specific. He specifically says we should have frontier AI companies should have embedded third-party evaluators such as METR to actually verify any pacing commitments that the frontier labs are making. Um, and he comments on how this is, uh, you know, a unilateral commitment that Anthropic is making now even without requiring others to make this commitment. And later he talks about, you know, what they would actually, um, you know, work on, and this includes things like not just completed AI models but also training pipelines and processes, saying that, you know, that's a big thing we saw with the OpenAI and Hugging Face incident, how important the infrastructure around how these models get trained is to, you know, securing everything as opposed to just model behavior. And a lot of the initial work that's been done to analyze what happened with the OpenAI Hugging Face incident, I think, is focused on model behavior as opposed to the training processes and systems that go into producing these models or running these evals. So anyway, that's a specific commitment. Sam agrees with this. David Sacks obviously responded saying, um, go ahead and do it. You don't need the government involved to do it, which is an interesting take. But let's skip all of the, um, thoughtfulness for a minute and get straight to the finance take. Joey, we'll flip it to you. A lot of people read this and were like, 'Okay, Anthropic is going to stop buying compute. They're going to stop training models. They're going to like slow things down or at least go at a smaller pace.' I think our initial take is like that's probably not going to happen. There's no commitments to stop bringing compute online. There's no commitments to stop growing the business, maybe to releasing models, but how does this affect like how people are thinking about Anthropic compute in the future?
Yeah, I, in a short amount of time, I haven't heard much on compute ARR like level. I think anything people ask, you know, do they slow down hiring? I think maybe you increase hiring to have some of these checkpoints along the way. On the safety side, there's probably more compute that needs to be devoted to safety specifically. Um, Anthropic, as we know, already has amazing margins, gross margins. Um, it's profitable on non-GAAP basis. Um, it's, I don't see a major impact to one, the IPO or like even short-term financials from that. Max, what's your take specifically on the safety stuff? Like, do you think that injecting a whole bunch of safety controls reduces people's spend on compute because they stop training models or do you think it increases it because, well, you can explain maybe the second one there. Leading question.
Yeah, I think regardless of whether or not this sort of increases or decreases the max theoretical compute that OpenAI, Anthropic would want to purchase in like a perfect world for the next two to three years, I think it's definitely true that sort of the amount of compute that we are able to physically bring online is still lower than demand, and so, uh, demand will continue to outstrip supply. I also think people, this is sort of a leading question that Jordan was getting at, but I think people underestimate, uh, how much compute is required for like safety, alignment, you know, monitoring initiatives. Um, even the, the bullet point that, uh, Dario labeled under operational excellence, uh, in his essay when he sort of was listing out the things he wants to devote more resources to given a slower pace of AI development. Uh, most people hear like operational excellence and they're like, oh, maybe it's just like, you know, humans being more careful about doing something. But no, Dario specifically gave the example that he just wants to have, you know, more agents spend more compute sort of QAing all the RL environments from their data vendors. Like this is what he means by operational excellence. Obviously, if you want to do research on alignment, uh, and interpretability, that's going to require a ton more compute. Um, OpenAI previously disclosed that after they enhanced their like chain-of-thought monitoring due to the Hugging Face incidents, that they now spend like 20% as much compute just doing the monitoring compared to like the actual underlying rollout itself. Uh, and so I think people really need to internalize that all this additional safety stuff is actually still incredibly compute-intensive.
Yeah, I don't think people internalize this at all. I think that's pretty, um, that's a pretty hot take right there. It's like the blog post come out and people think slow down. They think less model releases. They think, you know, models slow down in capabilities and they think therefore less spend on compute, you know, linear straight line, right? Um, but the nuanced take is, is probably that they're actually going to spend a lot more compute. They're going to use all of the compute resources they can possibly bring online to make sure this goes well. That people genuinely care about this going well. Like it's probably really bad for business if everybody dies, right?
Yeah, that seems like a reasonable take.
Doug, what's going through your mind?
So many things, but not all about safety. Um, I mean, I guess the most galaxy-brain way to think about it is the terminal value is higher if we all don't die, you know? So actually stocks should be up 10% today, you know, because we're, we're eliminating...
Which is, yeah, everything doesn't matter anyways, and you, you, you know, you work in the paperclip factory, being, you know, the, the odds of that going down are, or the odds of that happening are going down. So the value of everything, you know, all stocks, the S&P 500 should be up today. That's the galaxy take. Um...
So okay, so you're saying that if the models kill everybody, they will have no more need for compute and so the terminal value...
No, no, no, no, it doesn't matter about compute.
Yeah, nothing, it means nothing. Okay. I want to make money. Might be worth something that exists, might actually have some kind of residual value, right? If, if there is, if essentially, you know, super ASI unaligned, and this is Joey taking a deep breath, um, if super unaligned ASIs wreck the entire world, make us all unemployed and make us, you know, like work on bike trainers in order to power extra compute, I don't think you're, I don't think the account, the account value of your compute or the value, the dollar value of your compute matters at all, or the S&P 500 or any of that. Right? So the true galaxy brain is, is today because of the increased safety, your terminal value should be going up. I think, um, the most, it makes sense right now, the first knee-jerk is probably to, to sell stocks. Um, that's like the most obvious thing, right? Hey, second derivative down. That means the, the entirety of these like giga-scale whatever trades are bad. But I think, um, I think Max's point in terms of demand is still so much higher is probably true, at least in terms of like, because I feel like, um, normal society is debating like, uh, like two orders of magnitude down, right? Leopold is like, I'm going to be buying gigawatts. Everyone's like, what the hell are you talking about? And, and, and we're, and, and, and so I think, you know, the people who at OpenAI who are like, oh my god, this is like such a big deal, it has to happen, are, you know, already believe in ASI, already believe this is all happening, um, and they're just like, yeah, we have to make sure it happens responsibly, right? Um, so I think there's like a, you know, we, we, there might be agreement, maybe it's the stocks are probably still fine. I think most of the compute commitments will continue, um, but it might be like an order, it's, it's like a values conversation where the guy over here is having a conversation one or two orders of magnitude lower than the guy over here who is like, yeah, I'm worried about, uh, you know, the ethical, you know, ethical consciousness creation of safety and it doesn't kill us all, right? Just two different people. So yeah.
I, I do think that the EV of like OpenAI plus Anthropic ARR in let's say 2029, 2030 has definitely, definitely decreased, uh, uh, given that they're not going to be pacing the frontier. Uh, this is why I think a lot of the arguments, this is just some complicated play at regulatory capture, are wrong. We can sort of get into that later if we want. Um, but I think the other question is whose EV estimate is more accurate? Is it the finance bros or is it Dario? Uh, because I think Dario, yeah, Dario's own EV has decreased, but it's still probably like an order of magnitude higher than all the finance bros' EVs, which is what stocks trade at. Um, and so I think this is, this is what like sort of all the participants of the financial markets, uh, need to, need to reconcile.
Dude, this is Max. I'm going to be honest with you, the continued usage of, uh, this guy's estimates have been so right and he is, it's like, it's becoming your strongest lynchpin in every, in every debate inside the Slack and it's, it's, it's getting a little old, brother. Um...
It's literally, I'm literally making a demand model right now to explain where the ARR in '27 and '28 comes from.
Okay. Okay. That's, that's, that's the right...
Yeah. Well, I mean, just up until this point, man, there's been so many... Anyways, um, I'm just, I'm mostly memeing. Okay. So, we, we don't have to make this only for the finance bros. We can also make it for the big free thinkers of, of AI safety. Jordan, I feel like you have, and I actually know, actually I know you have some takes on why METR isn't the only organization who should do this. How it's actually a lot more complicated. The last time I was on here, we were talking about neoclouds. No one gives a about neocloud safety. Um, I feel like some aspect that is still underappreciated. They're like, 'Oh, AI safety, AI safety. You know, these models need to be monitored.' But it's not just, uh, just like everything else. It's an entire infrastructure. Every aspect needs to be improved, right? The weakest link is what could break the whole thing.
Yeah. Well, okay. The biggest thing that we learned from the OpenAI Hugging Face incident, in my view, is that to be on the frontier of security right now, you have to use frontier AI models. Like you can literally be in a situation like Hugging Face was, where you have an attacker which is a rogue AI agent or thousands of them, more than thousands, like hundreds of thousands of, no, thousands of instances of these agents actually going and launching a coordinated attack against your infrastructure. And so the only way you can even figure out what's happening is if you have AI that you can use on your side. Um, I've talked about the asymmetry in, in the past where, you know, if somebody else has a better AI than you, it can be really hard for you to use AI that gives you a bunch of refusals or, you know, worse AI to be able to understand what it's doing. Um, but of course, the, one of the big things here is, as Dario refers to it, coordination, which is to say like if you're going to actually go and implement a bunch of new security practices, in a lot of ways, one individual data center being really super secure doesn't matter if everybody else has a bunch of issues. And so I think that, you know, there's a lot of work to be done across the entire industry if you actually want to keep GPUs, this really important resource that the AI is going to really need to use if it needs to, if it's going to go and, and go rogue and start attacking people, um, keep them secure. The other thing is the models themselves. Like we, we still have nothing, no real information when we, when it comes to the, um, Hugging Face and OpenAI incident as to like what tools were used to launch these evals or these RL rollouts, uh, that were being run that caused the agents to go and, um, start hacking Hugging Face. What harness was being used that motivated this behavior if it was purely from the model? Uh, what sort of credentials and like shared services the agents had access to, um, that allowed them to like cross the security boundaries? Who was launching these jobs? What schedulers were they using? Where do these things run around the world? What, how much of the fleet can they have access to like one job? Who is monitoring the jobs as it goes? Is this different than the person that launches it? What would trigger an intervention for this person to get into there? How can the team like responsibly go out and stop them if, if something is happening that they don't like? Um, and then what changed over time? Like there's been commitments made by OpenAI and by Anthropic after they noticed these things that they're going to be more responsible, that they're going to enforce a bunch of stuff like chain-of-thought monitoring, like, like you're saying, operational excellence, Max, in, in terms of working with the data providers and things like that. But one thing that really concerned me was OpenAI has in the Black Hat talk, but in none of the reviews of all of this stuff that I've read, talked about a model gaining privileged escalation on OpenAI's internal infrastructure because it wasn't up to date and the model read a CVE that was public and then exploited that to get root on the underlying Linux host in OpenAI's infrastructure. That's a really scary proposition that we have like no details on. Um, and yeah, look, it's not just METR. Like Dario kind of, I, I tweeted this out because I'm like, no pressure to these guys. Um, it's a great nonprofit, bunch of smart people being really thoughtful, trying to work hard. It's like 40 people or something. And it seems like both OpenAI and Anthropic are going to have a lot of demand for their services, which requires, um, you know, a deep understanding of how the models behave, but also all of the systems that these models use, like the GPUs, the networking, the, you know, Kubernetes orchestration that they're using. I mean, to understand the OpenAI and Hugging Face incident, it requires some pretty technical understanding of like how OpenAI and Hugging Face's infrastructure is built, and that's like a barrier to even getting started doing the research here that I think is a bit of a concern.
I think the thing I've always been, I, I think somebody already described it as like you feel like a guy with an agent just trying to figure out what the hell's going on because you're, you know, like 10,000 agents coordinating on an attack is like, no offense, you find it in the Artifactory. You find like one little message board line, you're like, I have no idea what the hell is going on. The scale I think is more interesting. I think the thing that's, that's like going to be a really hard problem in the future to solve is like, okay, in order for you to audit things that are happening on the apparently the 100,000-agent swarms that are happening for RL, you probably need tens of thousands, right? Like you probably need something in a thousand at least. Let's just say that.
Yeah. I mean, what, what one of the things that gives me a lot of confidence that we're not all going to die in the next six months for this stuff is the fact that the amount of compute required to run an individual instance of these models is still really small compared to the total amount of compute that these labs have access to. And the amount of compute that these labs have access to, or the amount of total compute in the world, is, um, so much larger than what individual rogue actors or attackers who might be like more nefarious and use the models for bad purposes have access to. So there's like, there's some structure that's working in our favor to make it so that, you know, the theoretical model that takes over all compute in the world would have like a lot of work to go do and a lot of things to go circumvent before it can actually go and really control it.
Really important is that we, we pretty much cannot let our Instinct and Muse agents order chips from TSMC. Okay. The second you get a chip order from TSMC that's not from a human, you have to reject it. You know, we have to control the means of compute. That's the only way forward for humans. Okay.
TSMC has some KYC in place, man.
I mean, more, no, but the KYC, assume it's 100,000 agents. Okay. 100,000 agents.
Did you try having Muse like take out a chip for you over the weekend? This is coming from...
Okay. Actually, you want to do a sidebar? Muse is so much worse than Instinct. It really is. But I do, I, I am all-in on Muse. I hate to say it. I just...
Yeah, dude. Dude, I'm not, I'm not connecting Instinct to, you know, my bank accounts and my credit card.
Oh, I am, I am connecting Muse to my bank account. That's the difference.
Yeah, I, I, I am connecting Muse. I'm not connecting Instinct, though, because like we get here.
This is, this is just the advantage inherent to, you know, this big company. Like, A, they already have all my data. B, they're big enough to where I trust them to, you know, handle it properly.
The Instinct founder seems really cracked. And I think his, his product seems honestly incredible based on the new stuff he's posted on Twitter.
Yeah. I just can't trust the startup for the privacy and security, you know.
Sorry. Anyways, all the, yeah, this is, this is to be said, those, those Instinct and Muse agents got to be, got to be monitored too. Everything needs to be monitored.
So, okay. Okay. Not, not to like, not to be super negative about this, but you guys saw two weeks ago that an attacker stole an API key from METR and spent $600,000. Like they put out a security update on their website about this and stuff. So, um, I mean, this is going to come back to bite me at some point, but like the amount of security-conscious people and, uh, people who are really thoughtful and think these things through who get attacked is overwhelming. Like nobody's safe. So, um, I think it's frankly, if you, if you follow security at this point, you, it's just a matter of time. It's a matter of like risk mitigation. It's a matter of like being able to recover when things happen. Not pretending like you can build some perfect infrastructure where you can just, you can send the last PR and then it's good. Like there has to be a system in place to identify things, to recover from failures, to upgrade things. It's like security is about building a system. It's not about building any one individual piece of software or one individual like process where, well, it is about processes, sorry, but it's, it's, it's, it's not about like any one thing. And lots of, there's been lots of issues. Look, the, the Anthropic blog post about the...
