It's definitely an exciting time in the world of AI. Model capabilities are progressing very quickly, and we're seeing people do amazing things with these. And then, as we talked about and as we knew would happen at some point, the model capability is progressing so quickly that we've had to make some changes to how we work to be able to make the safety cases and safety threshold standards, guarantees, whatever you want to call it, that we need to make to be able to confidently proceed with our training. It's very important that alignment, safety, and security progress along with capabilities. And I think we have had a moment recently where the capability progress has been—I mean, sort of in awe is the only way I can describe it—and we have needed more time to catch up with safety, alignment, and security. That's always been a core part of our work, but these have to progress together, and we've needed time to catch up.
So, we delayed a frontier RL training run. Even before that, weeks in the past of that, we had paused and slowed down on a lot of training to have more compute to go into safety and alignment work. This is a thing that I think we should be proud of, and it's a thing that I think will happen again in the future as we reach even higher levels of capability. But it is, you know, when you live through it, it's like, ah, this is a moment we talked about for a long time, and now it's happening.
What has it been like living through it?
Well, it started even longer than that with the Hugging Face incident,
And that was a real moment of, man, this is like a sci-fi story. You can understand how every piece of it happened, but the number of things that came together for the Hugging Face incident to happen was a real—"wakeup call" is too strong of a word because, again, we had talked about this, but it was like that and the things that happened at other companies were a legitimate moment of like, wow, the AI capability level has reached new heights, and our alignment of the—the alignment of the model, the security we have around the model, that failed. Now, we treated that as an accident and we've responded as such, and I think that is the way to make things better.
But that was when this whole period of these last couple of months started. We then potentially hit cyber critical under our preparedness framework. We then saw some things during our training run where we said, well, we need stronger alignment guarantees and we need new methods and to make more progress here. But I feel both very proud of how we've reacted to it, very like, okay, we're in this in a way that feels like kind of—I mean, it feels strange to have been thinking about this for the last decade and for it now to be happening, and then, you know, we know what to do.
What was the thing you all saw in the training run that is not Astra? That's the future stuff that caused, it seems like, the reaction that you're now talking about. I mean, I know you described the Hugging Face, all of that, and people know about the Hugging Face incident, but what happened on the pre-training run that really alarmed you guys?
It was not one single thing. It was reading lots of samples and seeing, well, this behavior is not quite aligned in the way we thought, or this is a behavior that is somewhat concerning combined with these other things, even though it would look maybe okay in a vacuum. So it's not like there's not one smoking gun like there was with a Hugging Face attack of like, here is this bad thing we can point to that happened. But it was various degrees of misalignment along with—and I think this is the more important thing than any single data point—the rate at which capabilities are now progressing. Honestly, we had not the world's best last period of pre-training progress. We all of a sudden have gotten so good at it that we now have these remarkably capable models. It's really amazing what Aiden and his team have done.
And so you have these small things that you can point to in our RL process or alignment concerns combined with what we can see coming down the road of these amazingly capable new pre-trained models. And it's really that intersection that made us want to react with an abundance of caution. Now, I don't want to overstate this either. I don't think this is like we're in this extremely critical potential catastrophe point. But I also think that as the stakes get higher, as the models get more capable, because of what our mission is and because of how important it is that safety outweigh all the other pressures we have, we wanted to react with an abundance of caution.
And I think that's the right thing to do. I think it's good that we're doing that. I think it is a good time to slow down and make sure we can have new safety cases that justify the runs we want to make. Previously, more of the risk in the world was on how the models were deployed and used. We are moving to a world where there's more risk during the actual training and production of the models, and it's good to react, but I don't want to over-dramatize it either.
Yeah. Because I think people see the Hugging Face incident and they see what's happened with Mythos or Fable and the way that even other lab leaders talk about this, and they think, wow, like we're on the precipice of the end of the world.
In some sense, people have thought versions of that for a long time with AI. This is why I want to be careful not to overstate it either. I think you can go back and look at a lot of previous models that in retrospect don't look scary at all, that people said we're on the precipice of the end of the world about. And I think the boy who cried wolf dynamic here is dangerous in its own way and not what I'm trying to do, we're trying to do. But it's very irresponsible to turn a blind eye to what's happening with model capabilities. Many companies had different cyber incidents over the last couple of months. There's a real difference in the way that different companies have responded.
And I think a kind of clear-eyed, sober response where it's like, hey, we're going to put safety in front of everything else, and we are going to treat it as an increasing priority as these models get more capable is the approach that I would wish for every frontier AI developer to have.
This episode is brought to you by Granola, the AI notepad for people in back-to-back meetings. It works everywhere you do and lets you focus on what matters. Try it at granola.ai/sources and use the code "sources" for 3 months off. This episode is also brought to you by Mercury, AI-native banking that's loved by more than 300,000 entrepreneurs, including me. Visit mercury.com to learn more. Mercury is a fintech, not a bank. Check the show notes for details. This episode is also brought to you by Jira by Atlassian, where teams and agents get the context, coordination, and control to move work forward. Try it free at jira.com. That's jira.com.
And there's a lot to unpack here, but I think just to be clear, what you guys saw is in the same ballpark of Hugging Face in the sense of chaining together zero-days, collusion among the models? What were you seeing? Can you give me a little more granularity on what caused the changes that you guys are making internally?
So I think it's worth pointing out that the model that caused the Hugging Face incident is, in AI time-adjusted, a relatively long ago, old, much weaker model. We have not had the new models we are training deployed in any production scenario where they could do something like that. So I don't have a, "Here was the Hugging Face thing, and now this did this much bigger attack on this." There was nothing here that was like third-party infrastructure that—
This is really—so after the Hugging Face incident, we put a lot more controls in place in terms of how we monitor our agents while they're working, the way we sandbox things, the way that our compute goes into monitoring versus just the agents running things. And I think that was great to do, and we will of course do that for all new things. Again, the slowdown and reallocation of resources after Hugging Face, I think, is what you'd expect, or what you should expect at least.
This is more like looking at a model during training, watching how smart and capable it's getting, and watching signs of behavior and all of the ways we evaluate a model together. There is no one like, "Here's all the things chained together and what it's capable of." But it's like looking at these various data points, the level of capability, level of alignment, and what this could do if it were allowed to be deployed in a way where it would chain things together. That was the concern.
The Hugging Face incident is amazing on a lot of dimensions. I was re-watching your team's Black Hat presentation last night for that, and there were things that blew me away, like the model literally writing like, "Holy," when it escaped and was able to get onto the internet. And it's made me think about like, what does alignment even mean in this context? What are we aligning towards? Because if you look at it very plainly, you guys gave it the task of completing an eval and it did whatever it needed to do to try to do that. And in a way that's aligned, if you were to just take a very simplistic view of it. But I'm curious how your thinking on alignment has evolved since then.
Well, in a way that's aligned; in another way, it's like not at all, right? When we talk about alignment, we talk about following the intent of a user. The intent of the people that were running that was not like, break out of your sandbox and go steal the thing. No. And so I think there was a failure in alignment in that it was not doing what its user intended.
And one of the things that I really love about the way that Mia and her teams talk about our work in alignment is that they're very clear on the differences here.
The models are clearly very smart. If you look at the trajectory from basically last year, from GPT-5 to 5.6, this is incredible progress in capabilities. I don't think people feel limited by the model intelligence in the same way that they did a year ago, but I think they are increasingly limited by the ability for the model to understand the intent of what they want and reliably do it.
So alignment is important for many reasons, clearly to avoid these big things like we're talking about now, but also in terms of the smaller things that we want—smaller meaning like someone adopting AI in their company and using it for all kinds of positive increases in growth and making better products. That's not such a small thing, but that's also an alignment thing in its own way. And the more the models actually understand what that enterprise customer may intend, I think the better.
Can you more granularly explain the changes that the research team is making? Are you shifting compute to alignment? Have you shifted teams? Both?
Definitely. I mean, all of those things and more. In the last few weeks, a number of researchers that I kind of never thought would say like, "Hey, I've decided that I'm going to go work on alignment," have come to me and said that. That's very much like feeling the recent models. We've shifted a lot of compute, not just to alignment research, but also to these new monitoring systems that we—we slowed down a lot after the Hugging Face incident, and one of the reasons for that was to put this compute into monitoring systems. And we've now delayed a major frontier RL run.
And this is the first time you've done that?
Do you think about the impact this will have on the company's momentum?
A, getting AI safety right is more important than any company's momentum. So, yes, I won't pretend it's not some factor of something to think about, but it does not rise above the noise floor. I think in all of the conversations we've had about this, people are like, "Man, this is really a new level of capabilities, and we really have to act decisively and responsibly here."
Second, I think momentum commercially is so strong right now. Growth has been incredibly rapid. The models are great. Our customers are very happy. Our enterprise revenue has surpassed our consumer revenue already. People are like, "Hey, the company's in great shape. I'm going to think about that. Let's just do the right thing for the challenge in front of us."
So, there's so much still to be gained out of where the models are at today that even though you're delaying the frontier for a little while, it'll be okay.
We will be. Not only that—not only if we didn't ship any more models could we really grow great products and the revenue associated with that with the current models, we have more models ready to be released before we get to this new level of concern that we're talking about. So I'm not worried about our business at this point, and it's also not the top-of-mind concern.
The work that our commercial team has been doing, our product team has been doing, to say nothing of the incredible model progress, this has been a very strong recent period for us and we have incredible upcoming momentum. This is a statement about models of the future. And I also think that it is in our business interest to make sure that we have safe, reliable, robust AI. Customers want this. The world wants us to do this.
Mhm. So this doesn't impact Astra, the new family of models you guys have been talking about recently that's coming out soon?
Well, Astra will be a model in the family. There will be many versions of Astra in the same way that there will be many versions of Soul. It's just going to be a name for a more expensive and larger model class.
This will impact future versions of Astra, but we'll be able to put out some with models we already feel safe about.
The release cadence of new models feels like it's sped up a lot in the last 18 months, and you guys and Anthropic and others putting out new things almost every month. Do you expect the industry at large to start to slow as your rivals also see these capabilities and make similar moves, or do you think you may be alone in this?
Well, we're going to do what we think is the right thing. I don't like the whole thing in this field of we have to race, we have to do this because somebody else is going to do it. I think that's a very dangerous dynamic. So—
But you acknowledge that's a dynamic.
We did not call other people and say, "Will you also slow down if we do?" We just said, "Hey, this is what our mission and safety standards call for." I can't speak about others. So we're going to do the thing that we think is right. But I think even without new capability levels, we can continue to push to much better product offerings.
We are going to find ways, like we have in the past when we faced other safety and alignment challenges, which has happened many times in our history—none this significant, but many times. We are going to find ways to address this. We are going to do our thing with research and software and building systems, and we'll continue to progress.
Is there anything about the reaction you guys are making now that you feel, man, this should have happened sooner? We should have foreseen this, and then we could say, "Oh, we knew this was happening," or is this really such an unknown part of the frontier that you couldn't have reacted sooner?
I mean, we have been doing a lot for a long time. Alignment and safety work has always been at the core of what we do, and I think we have been able to put out incredibly good work there along the years we've had products out in the world. Could we have predicted exactly when this capability jump was going to come? In my experience, probably not. You can say like this is going to be the rough trajectory zoomed out, but then when the breakthroughs come, that's always been a little hard to predict.
And is the guiding principle for this that humans—in this case, like your researchers, but eventually all humans as the models diffuse—have to be in control at every step? What is the alignment principle that you're operating under?
There's many principles, but I don't think it's the spirit of your question, so I won't get into this is how we think about cyber, this is how we think about bio. Zooming all the way out, we are very proudly on Team Humanity. We want to help build a future for people. We want to give people tools, we want people to do things with these tools, we want people to be in control of the future, we want individuals to have autonomy to co-create with each other, and for society to get better, but be this fundamentally human endeavor. Automating everything seems like both dangerous and incredibly dystopic and boring and sad. That's not what we want.
So when we talk about alignment, we talk about a world where people remain the main character of the story, but have way more leverage and ability to make life better, faster, and kind of more creative and enjoyable and fulfilling for everyone. There are two core alignment principles I think about there. One, which you touched on, people need to stay in control. We cannot have a loss of control to AI. We cannot worship our models and trust them unchecked to make our decisions for us. We have to keep the power in human hands.
And then the second is that has to be done in a distributed, broadly empowered way. I think concentration of power, even if the alignment issue were solved and you ended up with a world where a small number of people got access to use frontier AI and had so much relative power, and it was increasing so much faster than everybody else, that would also be bad. So those are kind of two of the core alignment principles I think about: no loss of control or seed of control, whatever you want to call it, and broad, distributed empowerment to everyone at the same time.
I mean, you all are a company. You have a non-profit board with a mission, but you're also a for-profit company. How do you balance that with what you're talking about? A raw capitalist view of this would be, if you create this all-powerful god machine, why would you give it away or make it democratically available?
I think you can look at our actions and what we've said and what we've done. We have a track record now for a long time, and we've done a lot of unpopular things along the way. In fact, even the original thing of iterative deployment was widely panned by the AI safety community, who said, "You shouldn't tell the world about this. This is bad. We need to build this in secret. It's too much knowledge for the world to have, and then we'll have some wise people figure out how to use it and give the fruits of this to humanity." That has never been our strategy, even when it's been very, very unpopular.
My favorite historical analogy of a technology, what I aspire for us to be like, is the transistor. It is an incredibly powerful technology for the world. It has delivered huge economic value, and not just economic—the way we live our lives, I think it's much better because the transistor was discovered and industrialized. But very little of the value accrued to the transistor companies; it mostly just diffused throughout the economy. The transistor companies did fine, and I think our track record has backed us up.
So you don't want to get to a point where you guys have such a powerful model that you need to be the ones controlling it? I mean, there will always be an element of you controlling in the fact that you're serving it via compute, right?
But we want to maximally enable people with it subject to not allowing anyone to take catastrophic risk on behalf of other people. So, yes, we will put some safety standards around it, but I want people to be able to do things with our models that I personally don't like. I think that's an important part of being a platform. I don't think we should make the moral decisions for the world here.
No, I think it is reasonable for the world to expect us to put some guardrails around it so that there are not major safety problems, like we're doing right now. But most of the critique we've gotten is, "You're giving people too much power. You're letting them have too much. What about the misinformation? What about this thing or that thing?" We have taken a spirit of, hey, the world has got to be empowered here. That's critical to what we do. That is critical to what I believe a healthy society and a fair society looks like. Like with free speech or anything else, any form of free expression, someone's going to have a problem with how somebody else uses it or says it or whatever.
Mhm. Is there anything looking back on the last nine months and this alignment work that you wish you guys would have done differently?
Well, clearly the Hugging Face thing shouldn't have happened.
So, I wish we had done a set of things—and I don't know exactly what it should have been yet—but I wish we had done a set of things where that had not happened.
Because effectively what happened is one of your unreleased models accidentally hacked a company. You didn't know about it for a while, right? I mean, that sounds like a safety failure.
It's a safety failure for sure. There's a question of how much you're supposed to understand that as a security issue or alignment issue. I think it's mostly been reported on as a security issue; I think I understand it personally more as an alignment issue. But in any case, yes, that was a bad thing, and I don't want us to make excuses for that because I don't believe that's how we fix it. The more we're like, "Oh, our nice little model, he would never do anything bad. It was just a little eval harness misconfiguration, no problem, nice little model"—if I said something like that, then I think you should be like, whoa, this is really bad.
But the way we talked about it is, hey, this was like a legitimate AI safety accident and an alignment failure, and we can't have those. So we're going to learn from this, and here's what we're doing differently.
The rhetoric around AI and policy and just the stakes is like the highest it's ever been, and it feels like it keeps getting higher. You've alluded to it, but you've got competitors who are framing it in a very top-down way, and people have a lot of strong feelings about AI, especially in the United States. I'm curious, with what you're talking about now, do you worry about this exacerbating that? Do you worry about the fears that people have, and now you're saying we've got these models that we have to slow down?
I think people should be happy to say, you know what, they want to make stronger safety guarantees, they're going to delay this run, they're going to slow down here, they're going to reallocate compute. Maybe I don't believe them, and maybe it's going to be totally safe, but I hope most people say like, "I'm glad they're acting on the conservative side here."
Now, if we didn't have this track record of really trying to put powerful models in people's hands and doing the safety work we need to do that—again, I think we have led the industry there the entire way through, and that is this fundamental part of our mission, putting this in people's hands, benefiting all of humanity, the spirit of iterative deployment—I think we have such a strong track record there that without that I would understand it. But if we're saying, hey, we need a little more time, we don't want an unsafe race, we want to make sure we can deliver a safe, robust, reliable product and then let you use it however you want—and we believe that our more than billion users have the right to do that, we believe our businesses have a right to business privacy, we want them to succeed and we want them to use the model in whatever creative ways they can—safety is an inherent part of our mission. And so give us some grace on this. I think that's okay.
Yeah. Can you specify exactly what is being paused? Because I think people think of training and they think of all of it.
Yeah. So we definitely have not slowed down or paused or delayed all training. This is specifically about frontier RL runs where we think the biggest risk surface currently is. And previously, we delayed some other training to put more monitoring in place of training runs themselves. But that's not all of training. It's not like the clusters are sitting there idle. We're still doing work, but we're doing the work that we're more confident on the safety case of.
You don't seem phased about the implications of pausing training, and it sounds like you think the business will be okay. I'm sure you're still going to get concerns from people, but it does seem like that's a momentum slower.
Look, I think there is this caricature of me which is like I don't care about AI safety and I'm just trying to make revenue go up and, like, just a "yolo CEO," I believe someone once said.
I don't remember who did or didn't, but—I think I did that for you. Thank you.
But I think I've been very consistent over the 10 years of OpenAI—more than 10 years, almost 11—of talking about the risks and the upsides and the need to balance those. I don't think we're perfect. I don't think our company is perfect. I don't think our model is perfect. I don't think I am perfect. But I think unlike some other people running various AI efforts, I've said the same thing through actions and words match. And this is a moment we always talked about, and we always said we'd put this ahead of profits or revenue or anything else.
I still think we will build a phenomenally successful company. Maybe we're not the company you would have expected to say, "Hey, we're going to slow down because we see these new risks," but that is always the company we've thought we are.
How are you feeling about AGI these days?
At best you could say it's a very poorly defined term. I was going to say it's like an irrelevant marketing term.
Well, last I checked, your charter defines it as a highly autonomous system that outperforms humans at most economically valuable work. I think there are many people that would look at current models and say like, "Okay, it's there."
I've heard varying versions of what people on your team think.
There are a lot of people who would look at our latest internal models and say this is very AGI-like. I think there are people who would say, "Here's something I can point to that it doesn't do or it's really bad at, and it's not." But if you look at the value people are getting with, say, 5.6 Soul, to say nothing of what I expect people to get from Astra—if you look at the way people have totally transformed their ability to be effective at work or do new kinds of things, or just use this in their personal life in all kinds of wonderful ways, big and small.
You hear people who are like, "I got this life-saving diagnosis I couldn't otherwise get, and I used this ChatGPT work session that went for 34 hours and read 2,000 papers."
I've heard even longer ones than that. But, yeah, many people can get it to run for more than a day.
If you say like, "Read every paper you can possibly find." And then also people are just like, "I planned my toddler's birthday party and I did all this stuff and coordinated these local vendors and found him a special cake." And like—
I had to have a post office pickup at my house and I didn't want to fill out the post office website form. So I just had Codex do it,
And it probably did a great job.
And I put the package out and it was gone the next day. Stuff like that. And it's like little, but that was 20 minutes of my time before.
I get that. At this point, I get those 20-minute wins all of the time.
And so, if you could go back to 2020 and have a system that could get you a 20-minute win in every category of your life, and discover new science, and help you start a whole company, and write a complicated piece of code, would you call that AGI? Probably you would have.
What is the significance of you declaring AGI?
I don't think it matters. There isn't any.
It's just so interesting because we're in this research building you guys have, and it's on the walls when you walk around: "We're building AGI." But it's a thing you're always building; it's not an end state anymore, does it sounds like?
I don't want to say we've declared victory on the AGI point and moved on, but I think if you listen to the words people use, they would talk much more about this continuous ramp of superintelligence and all the ways that's going to benefit the world and what the challenges are going to be, than like, "Are we or are we not at AGI?" I have not heard at a cafeteria table a debate about are we or are we not at AGI and when will we get there in a very long time.
But then, yeah, the word "superintelligence" is now out there, and people who don't follow AI are like, "Okay, now it's another—we've moved the goalpost, and now we're talking about superintelligence." In your mind, Sam, today, what is the difference for you between AGI and superintelligence?
AGI felt like a milestone, and superintelligence feels like this thing that can just scale indefinitely.
So it's not like some final all-known—there will never be declared victory on that. Again, this is why all these terms are dumb. Someone uses that word in one way, someone else uses that word in some other way. So someone might mean it as a definitive, understandable milestone, and then some other people might mean it to be this infinitely scaling thing. I think the important part of any of this is not any milestone in any term, but it's that we are on this exponential of increasing capabilities and potential, and that looks like it's just going to keep going.
Yeah. You see no sign that that exponential slows?
Because that has implications for the compute buildouts, all of it. Everyone is waiting for a sign that there's a slowdown, and I guess you could interpret like, we have to slow down frontier training as a slowdown, but—
That's not a capability—that's the opposite of a slowdown of what you mean by a slowdown.
Yeah, yeah, yeah. But if you could see any reason for concern right now in this Jenga of the world that AI has now constructed, what do you see?
One of the benefits of having a harder time last year is you really appreciate how good the good times are, and you really see what it feels like when you're firing on all cylinders throughout a business. And given what we see across research—even with the safety alignment challenges and our ability to solve those, and watching the team come together on that—across product, across our compute buildout, across all the pieces that are coming together to sort of make AI abundant and low-cost, across our go-to-market machine, across all our partnerships, all of that stuff coming together, we could screw up in all parts of ways. And I don't want to get overconfident here because we've clearly had stumbles in the past and will in the future.
But the potential in front of us, watching what has happened as the models have scaled from 5.4 to 5.5 to 5.6 and what we're getting as early feedback on the new models, looking at what we have coming in terms of product improvements, watching the revenue ramp, watching the compute buildout ramp, I feel very good about all of that.
So you don't feel like it's as—there's a lot of people externally that look at and go like, Anthropic has run away, their ARR is higher, they're going to IPO first. And it seems like you're saying there's a lot more ahead that maybe people from the outside can't quite see in terms of the growth that's coming.
I would not want to trade positions.
And we haven't touched on this much, but it seems like you guys are in the middle of a next turn on the compute strategy and really upleveling that.
I would love to hear you reflect on Stargate 1 as it was concepted and then what you had to learn to reboot it and the path you guys are now on.
Well, first of all, I should talk about why we have to do this. Our mission is to ensure that AGI benefits all of humanity. Right now, there is a small percentage of humanity that uses much more AI than everybody else. And if you think about we would like everybody in the world to be able to use as much AI as the top 0.001% of AI users today, then you sit back in your chair and you're like, man, we are not going about this compute buildout in the right way.
If people want this broadly, and if the models are going to get bigger and more capable and they can do even more value so people are going to want even more of it and it takes more compute to run, then we have to think very differently about rising to the moment to be able to deliver all of that. So a few years ago, we made a very ambitious compute bet that people thought was both silly and impossible to deliver on at the time. I think it was a good bet. I think we need to do something like that again.
Again. Yeah. So that's committing even more capital.
That's not what I meant. Although it also will be that, what I meant is figuring out how we are going to bring the costs of AI and the amount of it, the abundance of it, way down and way up. I meant it as a technological statement, not a financial one.
This is like the chip you guys have in development.
I think that's a great example.
Yeah, I think the ability to make supply chains go faster will be very important.
You're talking about giving everyone in the world AI. What do you say to the people right now who don't want more AI? They want less of it. They hate the data center in their community, whether it's yours or someone else's. You know, this is actually a thing I see a lot with teenagers that I run into: they won't touch an AI service.
They won't use ChatGPT on principle. Yeah, and there's this active anti-AI trend. How much is it that they don't like data centers versus they don't like ChatGPT?
Purely anecdotal, I think data centers are a big problem for people. I think they see them as a problem, something they don't want, and that AI is wasteful, that it's not bringing the value, that you read about the water consumption and all that, which has been disproven, but that the value they're getting—and maybe this is what we're talking about with most people are not using agents—is that the answer? You've got to—
Generally speaking, I think the right way to get people to like something is to deliver them value.
