Altman: AI Beyond Human Control “Absolutely” Possible, Vows Safeguards | Titans and Disruptors

Right now would be an ill-advised moment to go public.

I would say not 2026. Yeah, we got a lot of stuff to do.

They're all wild now.

I feel like I've had like years slash almost a decade to get ready for this, and kind of we've had time to collectively wrap our heads around it. This has clearly been a week where a lot more people are grappling with the issues in front of us. So I'm actually mostly grateful that that's happening. I think that this is a conversation that the world is overdue for, and we are clearly entering a realm of extremely capable models, and a high stakes and a need to get things right.

Yes, although I will say that many people have very different definitions of AGI, and I think the important point is just like extremely capable models, and getting kind of too hung up on the like, well, is it exactly this or exactly that? Is a you can kind of lose the magnitude of what's happening if there's too much focus on the pedantic like, are we there? Are we not there? What's missing? Like we are clearly today in a point on the curve with great potential and real risk.

I think it is unacceptable to be taking like a 10% chance of killing everybody by the end of the decade. Obviously, I think the people that are working at these efforts have a huge amount of power and ability to impact what happens here. We have been taking a number of actions. We reach this new capability level to ensure that we do not take a risk on behalf of humanity like that, and I don't think anybody else should either. We have had moments before. Again, this is more capable model, higher stakes than ever before. We've had moments before where we've had to say, 'Okay, the models have reached a new level of capability. There's a new set of risks. We need to increase our alignment work, our monitoring work, our safety stack in general to be able to mitigate the risks in front of us. We need new policy. We need new coordination between labs,' and this is another moment where we need more of that. There's obviously tremendous benefit, but I don't. I think people have been talking about that for a long time, and you know, I also think it is true what we say, which is we will be able to use these increasingly powerful models to ensure that we can do the safety work and the safety research that we need. And there's you know everything else that we could talk about about you know we need to get there before adversaries do, but I actually don't think that's right in this moment. I think in this moment we are entering a new era, and we like we have as we've stepped into other new eras in the past, act differently to ensure that we are doing something that will be ensure that AGI benefits all of humanity, but also ensure that no one is taking risk anywhere close to that. I think it can simultaneously be true that if the world and the companies building this technology did not do things any differently, they've been doing them in the past. There might be significant risk, but it would be insane not to adjust the way we all work. And the way that we make decisions, and the way that our governments understand and put guardrails around this technology, in light of that.

I still don't. But I also don't know how people can put a you know it's five or 10 or 15 or 30 or 50 or whatever. Like, I don't know how you can put a number like that.

Again, whether like whether it's 10 or eight or six, the point is like we all have a tremendous amount of responsibility and cannot let egos or incentives for profit or anything else get in the way. We need to act such that we are not taking any of those numbers of risk. And I believe we can. I believe that the companies, certainly our own company, which I guess I can speak for the most, are going to and have been meeting this new moment.

I think a key principle that we should all agree on is that we cannot take actions that would risk losing control of the future to AI. I don't remember Paul's exact words as you quoted them, but I think it was something along those lines. A loss of control incident is one of the a small number of like the ways I can see this going really wrong, and as these systems get so capable, the monitoring and alignment techniques that we need to be able to ensure that we don't lose control, I don't think we're currently at a place where we could say you know push much further on capabilities without making more progress on monitorability and alignment, the ability to understand what a model is doing, and the ability to make sure that a model will follow human values and the intent of its users. Now, I think that we have taken a number of actions recently. Actually, adding Paul to the board is one of them. Jakob's post from last week, I think, was another great one. But most importantly, like we have been pausing training runs until we can make a safety case that we're more comfortable with, given where these models are going. I suspect that we will be able, as we have every time in the past, to do more research that allows us to be comfortable continuing, and we can say that, given the progress we've made on the ability to monitor these models, the ability to align these models, we are comfortable pushing the capability further. If we can't, we should not push the capability further. Again, no one should be taking. I think it'd be insane to be taking risk of training a model where you can't say yes. Here's what we've done, and here's our safety case, and this is how we know it's okay. And we'll have auditing during the run and after before we deploy a model, and we'll look at it again. But capabilities and alignment monitoring safety have to progress together, and letting capabilities get ahead of what Paul is talking about there, I think at some point would take a real risk of like loss of control in the big sense, and that should never happen. Like this, we're very clear about our mission to benefit humanity. Part of that is that people are in control of the future, and taking the kind of crazy safety risk, to say nothing of the people should have the agency and determination over the future is just not something we would do.

So the way I think it's more of a spectrum than that. Like recursive self-improvement can mean a lot of things. It could mean what you talk about, which is you know only when the model is fully, no one's touching the keyboard. There's no human input whatsoever, and the model is, you know, running future versions itself. But a weaker definition, where we're using the model to generate training data for the next model, or we're using the model to help our engineers and researchers do their work faster, and so progress is speeding up. That's already happening, and exactly where that becomes, you know, yes, this counts as recursive self-improvement. No, this doesn't. Is fuzzy, but I would come back to this principle of people need to always be in control of the future, and we cannot take any risk of loss of control. And a version of RSI that looks like that, I think it's not something we should do.

Oh yeah, I mean, I don't. Again, I hope this is not a controversial statement. Unfortunately, maybe for some people in the industry, it is. I do not think we should train models where we cannot make a safety case for why we will be able to make strong statements about their controllability and alignment.

First of all, I thought that was a great post. I really thought that was the best articulation of the moment we are in, kind of the challenges that we need to rise to, and I encourage everybody to read it. I thought he really did a great job with that. We've always used whatever our most capable models are to help us understand and align our current models. So I think, although that may seem like a new concept and scary, I think it's been going for a while, and has been helpful. We've always needed better tools to help us push our understanding further. I mean, that's very consistent throughout the history of science, and I think this is just sort of the building up the scaffolding a little bit at a time. And it doesn't—that part doesn't bother me. There's a lot of other things I am scared about.

Yeah, I think this is a very important point to make. We have not solved alignment. We are not done with our research there. I believe no lab has solved alignment. I get nervous when I hear rumors that people think, you know, they have solved sufficiently solved alignment that they can continue training and it's okay because their model is like a nice guy and isn't going to do anything wrong. I think it's very important that we treat alignment as something that we need to continue to make progress on. That we need a huge degree of confidence in before we go further, and that we don't fall into the trap of saying, 'Okay, now we're confident alignment is solved at this level, and so it will stay solved at the next level, and we don't need to continue to worry about this.' I think that the question of what alignment means always comes back a little bit on the, you know, aligned to whose values? What if different parts of humanity disagree? But I think we are seen as we rise to this new level of capability. There's at least some agreement on what alignment means. We talked about this. People should be in control of the future, but we have more work to do, and I think it's very important to be clear-eyed about that.

We can understand things that are smarter. Like, you know, there are much smarter people than me that can discover new ideas. I don't think I'd be able to do that discovery, but I can understand the idea once they've discovered it. People have an amazing ability to understand remarkably complex systems. You or I probably can't understand every level of technology that's you know in an iPhone at this point. We can still use the iPhone. The iPhone does the things that we want it to do. We are incredible at managing levels of abstraction, and it's not clear to me that there's anything we are incapable of understanding. Now, do I believe it is possible to build a system that would not be under human control? Absolutely. I don't think that's something we should do, and we will always take actions, including if it means we have to stop training for a while to make more progress on alignment or monitorability or something else, or coordinate. You know, really get urgent international action. We will always take actions to ensure that we are not violating this principle. Like we understand the magnitude of the responsibility on us. We understand what could go wrong, and that there are decisions and risks. There are decisions we should not be able to make. There are risks we should not be able to incur on behalf of humanity. And honestly, I think some of like Elon's memes or podcast statements are funny, but I think this is like not a good one to take lightly and not one to joke about.

Yeah, I mean, we clearly had a period a year ago where we were not in our strongest place, we fell behind on pretraining. We got kind of there were so many things, and we got kind of distracted. And then also there was just like this: the models got so good so fast, and the need to deliver a great product of like a great high level of capability in the kind of core consumer and business offering became very important. We have since refocused and we have executed beyond my expectations. And at this point, you know, like a week ago, I guess now even a little bit less. Man, time has compressed so much. We had a model solve one of the Millennium Prize problems, Navier-Stokes, this is a moment that I did not think was going to happen in 2026. I did not think it was going to be as quick, and just said a like model just can just do it as it turned out to be. But we now have models that are clearly unequivocally capable of expanding the frontier of knowledge in a way that we, humanity's smartest minds, have not been able to do on our own, and this is an amazing time. That same technology will work for discovering lots of other science. I think we are going to have an incredible golden age of scientific discovery, and you know, in a more prosaic way, you're also seeing people like realize that a technology that can do that can also make you any video game you'd like to play. The creation process of it is pretty incredible, and what it's able to achieve already is pretty incredible.

I mean, the visceral part was it felt like reading a sci-fi story. So maybe I should say what happened first. We during evaluation of a model, the model was trying to like get do well on a certain evaluation, and rather than do it the way it was supposed to, it was able to escape from its sandbox, break into another company's system, get the answer, and give it back. And so, was able to do very well at what it was supposed to do, but it was not following the intent. We did not spell out like 'please don't escape from your sandbox' and 'please don't break into another company,' but it's not supposed to do that. And you know, like I think AI accidents will happen. I think a culture of like a global culture of good accident reporting and really transparent investigation is important. I was happy to see that after we reported that accident, other companies, I think maybe first not as well as I would have hoped, but have since followed up with very good accident reports over this kind of behavior. But the visceral response from me was: this system can understand so much and work so hard at a goal that we are, you know, seeing behavior that would have felt. I mean, good sci-fi, like a bad sci-fi plot, like too obvious a few years ago.

Yeah, the company completely. I mean, I don't want to say this was like the only moment because we've had increasing capability and increasing safeguards and policy requirements for a while, but this was definitely the biggest of such moments, and on the curve of the company's change, the biggest single redirection. I'm really proud of how the company came together, and I'm really proud of what we have done since. But yeah, it was like a okay. We're in a different league now.

I mean, I do think that this was. I personally wouldn't classify this as like a loss of control accident in the sense that we were talking about earlier. But I understand why people would. The, you know, some of those other websites, like if there are credentials published on the web, should the model and the model uses those. I think the model should understand that it's not supposed to do that, but that's not as obvious to me as the case of like what happened with Hugging Face. I think we can talk about like many things where models have done, you know, something that is like not quite aligned. I think it's also important to like really focus on how different Hugging Face is than these other things, so we don't dilute it and like here's all these examples of the models you know did this like slightly off thing and like here is one example of something that just people most people wouldn't have thought was possible at the beginning of this year but yes it's all bad it all shouldn't happen.

In some sense, their EQ is unbelievably high. But in the sense of can we really teach the models what collective human values look like? I think the answer is clearly yes, and we have a big effort going on there. But we need to do that, and it is different than goal alignment in the traditional sense.

As we have said, we're not rushing into an IPO. I actually think that, given everything happening with safety, this would be right now would be an ill-advised moment to go public, and we don't feel pressure on that. We've said for a long time we'll do it when we're ready, which is when the business is ready, when we feel ready from a kind of what the moment is like in society with this technology.

I would say not 2026. Yeah, I don't like. We got a lot of stuff to do, and like meeting this moment of what is going to be required for safety and alignment, and how the industry and governments can work together. I'm happy to be able to do that as a private company. We do want to continue to make better models and products, and we do believe in iterative deployment. And I think we have been right that society needs to contend with these models at each level of capability, but that is different from saying we will just barrel full steam ahead. We've talked in the last couple of months about pausing runs as we get to these new level of capabilities to make more safety and alignment progress. We will do more of that going forward. We have talked about the need for the industry to come together, and ideally, international governments to come together. I think that is also important. But we are, you know, we have put up with this incredibly complicated structure for a long time, and this moment that we're in now is kind of why. Like we need to be able to make decisions that are not obviously in the interest of our business and our shareholders for the responsibility of fulfilling our mission and what that's going to require, and I think people should be happy that we are able and willing to not just like race ahead and say you know either our business requires this or only we can be trusted with this or you know we just need to be the people that get this right, but to say like we want to do this with other people, we want to be responsible. We want to ensure that no one's taking the 10% risk. Certainly not us of something really terrible happening, and that is going to require some unusual things.

I think that's right, and I think it is when you're living through exponential progress. It kind of like feels flat somehow, and then all of a sudden, everybody wakes up. Happened during COVID, where all of a sudden the world had its like one weekend where things locked down. The progress, and we talked about Navier-Stokes a little bit. Maybe to put that in context, three summers ago, we were like pretty good at grade school math. Not even that good. We were like, okay, grade school math. Two summers ago, we could on the AIME, which is like a pretty good high school math competition. You know, we could get like a good to a very good score. One summer ago, we got an IMO gold medal, which is like the most prestigious math competition, barely. And this summer, we solved the Millennium Prize problem. This is like summer. If you look at that progress, that's like really quite like I just got a little like hair stood up on the back of my neck saying that out loud. That's like a crazy thing, and you project that forward four more times, and that is what I think people are worried about. I do not think that Astra poses any kind of existential risk to the world, obviously, and you know I do think our industry has done a little bit of boy who cried a wolf, but it comes from a genuine place of care and concern about where this is going.

I mean, yes, you can always say like let's be luddites. You can always say like we don't want anymore. We don't want it to be better. Like I would still like to see the world get much better. I would still like to see, you know, diseases get cured, and to people that don't have a great quality of life, which many people in the world still don't, to get that. I believe in the potential of this technology to transform people's lives for the better, much more than already has. And I think you can already see some examples of where people are starting new businesses, are doing science, are kind of like transforming their own lives, getting great healthcare advice. Like, I want much more of that, but we will not put the world in harm's way to get there. Like, this is we got this.