The most boring interpretation of this is probably the correct one, which is that the world will probably only accept superintelligence at a pace that it can absorb. So the labs are slowing down because they have to, and not because they're plotting anything. And the result is probably intelligence that's, you know, decently fast, decently safe, decently commoditized, and it's spreading through best practices and even like distillation until the models are just basically a reasonable integration into the world.
Welcome to Prof G Markets. I'm Ed Elson. It is September 15th. Let's check in on yesterday's market vitals. The major indices declined with chip makers selling off on fears of an AI slowdown. More on that in a minute. Meanwhile, CrowdStrike rallied 14% as investors piled into cybersecurity stocks. Brent crude remained elevated at $105 per barrel. And finally, the yield on 10-year Treasuries topped 5% for the first time in three years. We will be discussing that news tomorrow.
Okay, what else is happening? The AI apocalypse debate intensified over the weekend, and both Anthropic and OpenAI have now officially weighed in. On Saturday, Anthropic CEO Dario Amodei published an essay titled, quote, "We must pace the frontier." He wrote that we must slow the pace at which we improve the capabilities of AI models. He also warned that within six to 12 months, a swarm of rogue AI agents could take over the entire internet with a persistent botnet. Sam Altman also posted, quote, "I agree with Dario that we need to pace the frontier." And Elon Musk concurred, posting, quote, "Dario is right." Altman also told Fortune that OpenAI will not go public in 2026, calling this, quote, "an ill-advised moment to go public." The White House, however, is not on board with slowing things down. President Trump, speaking to reporters in Ireland on Sunday, said, quote, "Whoever wins AI wins," and called the people raising these alarms, quote, "negative forces." Still, a slew of AI-adjacent companies sold off on Monday on concerns that a slowdown might affect AI spending. Nvidia closed down 3%, Oracle closed down 4%, CoreWeave down 7%, and SoftBank, which of course is a big investor in OpenAI, down 14%.
So, here to break down what all of this means, we are speaking with Charlie O'Neill, co-head of model training at Base 10. Charlie, thank you for joining us again on Prof G Markets. You work in AI, you are an AI developer, you work with these models, you've been in this game a long time. Suddenly, everyone is very upset about this. It's interesting how the debate has evolved. But we're now reaching a place where the leaders of these AI companies are saying, "We need to actually slow everything down," which I'm not sure many people would have predicted. And now it's the president, President Trump, saying, "No, that's the wrong approach. We need to speed things up." Where do you land on this? What is your perspective?
Yeah, I think it's really interesting to see the reaction to something that's kind of been, I guess, like linearly scaling for a long time in terms of the calls for pacing the frontier from people such as Dario and Sam. I think the best way to view what they're actually asking for is not necessarily we're going to put a halt to capability development, we're going to put all these stringent checks and like, you know, first-party kind of things that are slowing us down. The best way to view this is we are going to keep advancing the capabilities of the models. We are just going to allocate a little bit more compute to making sure those models are safe. And then it's very interesting to see all the sell-offs and so on today. OpenAI and Anthropic are planning on spending more compute probably than they were a few weeks ago. You know, there's rumors that OpenAI is going to allocate up to 20% of internal compute for monitoring and safety. So, you know, as they're doing these training runs, as they're deploying these models in the real world, things like the Hugging Face incident, they don't want to happen again. And so, if you allocate more compute to monitoring those models as they're doing their rollouts and as they're doing inference, then you're more likely to catch it. Anthropic is similar, and some people have suggested they're going to be a much higher than 20% figure. So when you consider that and the fact that Anthropic and OpenAI really don't want to move off their, you know, model roadmaps, they want to keep training bigger and bigger models, they want to keep scaling the reinforcement learning they're doing on top of these big pre-training bases, they just have to allocate more compute to, you know, monitoring and safety research, then I think we might even see the labs be even more aggressive with compute buildouts and securing compute. And it's certainly not going to be a, you know, a bearish sign for like the amount of compute the world is going to need over the next few years. So that's probably the best way to view it is like, you know, capabilities will keep progressing at roughly the same rate. It's just that we're going to allocate more compute to safety and monitoring.
If everything that they are doing is sort of within their own power, as you're kind of describing, why are they saying anything right now? What are they trying to get at? Because there are a lot of people, especially in government, David Sacks has said this, who's the former AI czar, and the president is saying this too. It sounds like what they're asking for is for the government to do something about it, for there to be more regulation that is imposed on themselves. So what do you think they are asking for exactly in this moment?
There have been some, you know, people like David Sacks and so on saying that these labs are using this as an opportunity for regulatory capture to crowd out open source, to shut down competitors which are behind. I don't really think that's the case, to be honest. I think like when you look at the main things that these labs are calling for, or at least instantiating on their own accord, it's things like third-party evaluators and this commitment of compute to monitoring and safety. You know, those third-party evaluators won't have necessarily any license or legal standpoint to, you know, shut down model development if they find things that they don't like. Like that's still going to be up to the internal labs themselves. I think like, to be honest, like the best read here isn't a conspiracy theory on either side. It's not the labs trying to crowd out open source. And you know, it's not some like, you know, kind of global cooperation between OpenAI and Anthropic to crowd everyone else out either. I think it's simply a case of, you know, these models are getting very, very good. The closed-source models got there first. You know, in the next three to nine months, open-source models are going to reach these capability points, and we're now at the point where that could have significant impacts on the world. Even if it's not malicious and, you know, rogue AI is going off and trying to like kill humans, there'll probably be significant annoyances. Like the most likely scenario here is that things like the Hugging Face incident happen and the internet is overrun by like swarms of AI agents that are trying to get some like arbitrary task done that are not trying to be malicious or evil. And so like I think the labs just recognize this. Some people have more extreme views, of course, but realistically capabilities are going to progress. Like the most boring interpretation of this is probably the correct one, which is that the world will probably only accept superintelligence at a pace that it can absorb. So the labs are slowing down because they have to, and not because they're plotting anything. And the result is probably intelligence that's, you know, decently fast, decently safe, decently commoditized, and it's spreading through best practices and even like distillation until the models are just basically a reasonable integration into the world.
It's interesting to hear you say it sounds like you're not worried about this much at all. And that is interesting because you work with open-source models. That is a lot of what you do. And of course, as you mentioned, that is what David Sacks has been accusing, and a lot of people have been accusing Anthropic and OpenAI of, that they are saying, oh, now we need regulation because that way it might create some level of regulatory capture which might crowd out the availability of open-source models and open-source model providers. But you don't believe that is an issue. I wonder, do you think that this whole thing is overblown? Do you think that the Jacob Steinhardt tweet that really started this debate, saying that AI could kill us all by the end of the decade, is your view that that isn't something to worry about much either?
Yeah. I mean, like I'm definitely not one of those extremists who think that, you know, the AI has a more than 10% chance of killing all of humanity. Yeah, I do believe that there are tail risks that like some people should seriously be considering, but I think that what we're seeing with the current LM paradigm is like, as I said before, a risk of swarms doing very, very annoying things to humanity and things that will be quite painful in the short term. However, I believe the benefits of AI outweigh the annoyances and even the pain that those like, you know, short-term things can cause, and that at some point the world is going to have to harden to these systems. And I think that the pace we're currently progressing at, and particularly if we do things like, you know, dedicating more compute to safety, is going to allow us to integrate AI into a world at a pace we can handle. A really good example of this is like, you know, the cybersecurity arguments that people have been making. You know, like we worry that when AI got to this point, or even the point it was at six months ago, the world will be overrun by cybersecurity attacks, and that just hasn't happened. And a large part of the reason is that, you know, the frontier labs, the closed-source labs, have kind of been the canary in the coal mine. And we've understood where the model's capabilities are going to be at six months for the open source, and we spent six months preparing. These have been slowly released, and you know, like Greg Brockman describes using Astra to like repeatedly harden, you know, all the vulnerabilities in OpenAI's codebases, and every single model that they release, they do this with. And I think the world will look the same, and cybersecurity is just one example, but it's also one example that's very important where we haven't seen this like massive pain play out, and we have definitely seen the benefits. So yes, I think there's risk to consider on any side of the spectrum, but you know, I sit firmly in the middle, and I believe that the pace we're currently progressing at is a healthy pace. And like I also have trust in like not only the closed lab leaders, but also the open-source lab leaders to make sure that pace continues at an appropriate pace.
One of the other conspiracy theories going around about why they are saying all of this, saying that we need to slow it down, we need to be regulated now. One of the theories is that maybe Anthropic and OpenAI are trying to sort of preempt a slowdown before they go public, either because they want to have a reason as to why maybe their growth started to slow down, or maybe because they want to get out of some of the larger spending contracts which have been quite onerous in terms of their income statements, certainly for OpenAI because they have to pay billions and billions of dollars for these data centers, for these GPUs, and to rent them. What do you make of that conspiracy theory that now they say, "Well, make the government make us stop spending, and then we'll be in a better financial position"?
Yeah, from the spending aspect, completely don't buy it at all. And I think there's a lot of evidence for that. From the optics aspect, I definitely buy it. I think like, you know, even Dario, who I believe, you know, fundamentally thinks that this is like a real risk, and like he's going to act in accordance with that and as rationally as he can to protect humanity from what he believes to be this real risk. I think both Sam and Dario benefit from the public optics of saying, okay, we're going to treat this technology carefully and not race to the end. And there's also a bit of game theory here, like one of them can't say it without the other saying it, because then again, you're the evil corporation in this duopoly which is currently running the frontier. From the spending perspective, I think that like, you know, it's very clear that like the value of a gigawatt or a megawatt even of compute is only going to get like more valuable. Like Anthropic and OpenAI are both squeezing increasing margins out of each megawatt of compute they buy. There's a reason that each megawatt is going, you know, from $10 million per megawatt to probably 15 at the moment to probably 20 to 25 next year and perhaps even higher. That's kind of a conservative estimate. Running a model on compute has never been more valuable. And I don't think there's any world in which any lab, let alone Anthropic and OpenAI, are going to want to stop spending on compute because they realize how compute the world is going to be over the next few years and how valuable this intelligence is. So optics, yes. Spending, no.
You mentioned that the boring explanation is probably the most true. But a lot of the conversation has been anything but boring. I mean, this debate is exploding everywhere, on podcasts, on tech podcasts, on cable news networks, and now it's literally occupying the mind of the president. What is it about this moment that is so triggering to everyone? And not just people who are afraid of AI, people who are excited about AI, people who are optimistic about AI. I mean, this has got everyone riled up in a way that I have rarely seen. What is it about this moment that explains that?
The reason it's so visceral is not because there has been a discontinuity in terms of capability advancement, in terms of what people have been saying about where these models would be at, in terms of the press around these models. I think that most people really fundamentally involved in the technical side have been able to draw the straight lines on the scaling graphs and say, okay, at this point in time we are going to hit here. I think it's just a confluence of a lot of things, and like to some people looks like a discontinuity because it's managed to permeate the public consciousness in a proper way for the first time. I think a big part of that is the Hugging Face incident and all the discourse that that generated, and then you know, you add things in like OpenAI's marketing around Astra being AGI. Like that's probably not going to help either, right? Like people have become familiar with this term now, and if you claim that your model's finally there, then they might see that as a phase transition or discontinuity in of itself. But you know, like this misalignment research has been very clear for a long time. Like go back to Anthropic's 2023 papers with much worse models, with much, much smaller amounts of RL, if any, and it's clear that like models would behave poorly in incidents like the Hugging Face swarm incident if you put them in weird situations. And you know, the RL environments that the labs have been buying at scale, some are good, but some are also very, very poor quality. Some are intentionally engineered to be like, you know, impossible to do, and this is leading the models to do really weird, misaligned things. I think that's pretty common sense and pretty clear. I also am not trying to downweight the risks that that comes with. Obviously, like the Hugging Face incident could have had real-world impact, but at the same time, like I don't think that there has been this discontinuity. So I'm glad that the world is recognizing it, but I do think that the reaction to it will calm down as people start to understand exactly how to interpret these things and exactly what it means.
