We're now in the AGI era. Astra has really hit something that I'm like, 'Okay, I think this is pretty reasonable to call it AGI.' We've seen it run coherently for 24 hours to go accomplish tasks that I think are quite amazing.
The models will be plenty powerful, but it'll be hard to get to everybody given that we won't have enough compute to serve it all.
You don't win the Super Bowl by saying, 'I want to win the Super Bowl.' You win it by blocking, tackling. You really have to make sure that safety, security, alignment, those are all standards that you're constantly upleveling. I think that's going to be a huge challenge people are underestimating.
You've made two very big bets in your career: helping build Stripe early and helping, of course, co-found OpenAI.
I throughout OpenAI have always focused on whatever is the most important problem. For the past two years, it's been the data centers, the infrastructure, the machine learning.
Do you know what the next year we'll focus on, or...
I think this is going to become the most important conversation.
Greg, welcome to the A&Z podcast.
So, Greg, you've made two very big bets in your career, helping build Stripe early and helping, of course, co-found OpenAI. If we were talking 10 years ago and you were predicting what would the world look like in 2026 as it relates to AI, would you be able to predict that we would be making the breakthroughs that you've, you know, made today, or what would you tell them about what you would expect?
Well, so Ilya and I actually spent a lot of time trying to predict what it would look like, what the timelines would be. And I remember we did some math on compute in around 2016, 2017. And we kind of came to the conclusion that if you look at Moore's law progress, that kind of thing, 15 years felt like about the timeline to AGI. And if you really squinted at it, you're willing to scale up and build massive supercomputers, spend the hundreds of billions of dollars, that kind of thing, that maybe it'd be 10. And so I actually feel like in some ways, obviously what's happening, it's remarkable. It's this amazing sort of moment for everyone to be a part of and to be able to help shape collectively. But it also feels a little bit like maybe it's kind of the conclusion of like a lot of forces that are all coming together for this moment. If you step back and really take that sort of macro view, it kind of makes sense it's happening now.
And do you think we're currently, because you guys slightly underestimated the timeline, or I guess it was basically on point. Do you think we're still on the timeline given we're now starting to drive real shortages on the supply chain?
Well, look, I do think that we are in a world where it is hard for compute to keep up with the demand that we're already seeing in the market, right? And just in terms of how people are going to use this technology, benefit from it. I do think it's going to be very hard for us to scale the raw potential and capability of these models to everyone, and that's part of what we try to do. And so I think that the progress I do see, like we have line of sight to continue to make the models much more capable, safe and aligned, but also really distributing that power and the benefits and the empowerment to everyone. I think that's going to be a huge challenge people are underestimating.
Right. Ah, interesting. So we'll have the models will be plenty powerful, or they'll continue apace, but it'll be hard to get to everybody, certainly in an affordable way, given that we won't have enough compute to serve it all.
I think that's true, and I do think we're at a point now where we have to really start thinking about what we call pacing the frontier. And so thinking about as we move to more capable models, you really have to make sure that safety, security, alignment, those are all standards that you're constantly upleveling. And those actually become almost the bottleneck to progress, or sort of the part that you have to spend a lot of your effort to make sure you've gotten right. And so I think in my mind it's more those constraints and the compute. I think we can make it happen. And then on the flip side, I think, yeah, this bringing it to everyone, which is ultimately about our mission, right? It's empower everyone, ensure it benefits everyone. That's something that I think deserves a lot more airtime than it's gotten.
Yeah. Actually, let's get a little deeper on the safety thing because it's been very interesting to me in that it felt like in the beginning, safety was like, okay, let's make these things not say nasty stuff that people don't like. And so the approach that was taken was kind of a surface around the edges. Okay, we'll, you know, put some filters on this guy and we'll RLHF, you know, around the edges. But like if you get deep into the thing, you'll be able to get the bad words out. But if somebody wants to go through that to hear bad words themselves, who cares? But then, you know, when you get into, okay, now these things are really good at cyber hacking and other kinds of ideas. Now you need kind of a more architectural idea where the model itself knows not to reward hack in a way that's going to be dangerous and so forth. And how do you feel like, yes, we can make progress against that fast, or is that like a really hard different category of problem, or how are you thinking about that?
Well, I absolutely think we can and are making very rapid progress on this problem. I think that there's a lot of both great ideas and research that we've been investing in for many years. Actually, if you rewind to 2017, I think people underappreciate some of the most key results that came out of OpenAI in the field at the time. So, both kind of the first inklings of modern language models. You can find a paper from 2017 that kind of laid that out with LSTMs, and you know, it's like kind of this very baby result, but also reward learning, reinforcement learning from human preferences that was also created in 2017 to start thinking about how can you align a model to match what humans want, right, by providing feedback from people.
Yeah. Just for usability.
Exactly. In 2017, 2018, we had ideas for if you have something that's very smart and capable, how can you actually supervise what it's doing, how can you provide feedback and ensure that it's staying aligned with you. And we had ideas such as debate or iterative amplification. So these are ideas that were really sort of at this phase before these systems existed, and you can start to see the sort of trickle down of those ideas into modern systems and investment. And so it's in some ways that I think that there was this early phase when OpenAI started where we really were thinking about AGI safety, things like that, and that was very front and center in even the comms. And then I think that as things like ChatGPT took off, then people started to see, okay, well, we're not at this point yet, and so the is the AI politically neutral and questions like that start to become to the front and center.
And now that we're here, all these other ideas that we've been talking about for a long time, they're taking the main stage again, and I think we've been sort of thinking about this moment for a long time.
That's really good news. And when you think about, and we don't have like a really great community yet amongst the soda models, but it seems like those kinds of ideas, you and Google and Anthropic and SpaceX would want to share, and Meta now, as opposed to like, okay, this is a proprietary idea that's a way to keep these models safe since you're all on related architectures. Or how do you see that unfolding, or is everybody going to do it independently?
Well, I think there's nuance here, and I do think the coordination is going to be a very important theme, right? To really think about within the frontier labs and really just thinking broadly about what has to happen for humanity as a whole to sort of navigate this technology in the best way. I think that we're going to have to really think hard about those kinds of questions. And we published a lot of our thoughts. And again, some of this is about pacing the frontier. Some of this is about unilateral actions that we can take and how we think about how do you make safety cases for even training and developing and evaluating these kinds of models. All that's new. No one's ever really had to operationalize this before. And I think it is not at all unique to OpenAI. Like there's a whole world that is basically developing this technology. And I think one thing it's easy to miss is that what we're building is almost a sort of thing that falls out of compute progress. And in some ways, compute progress is something that falls out of technological progress. And so there's this massive wave that's been building for a very long time. And we're starting to see the leading edges of this technology. And companies like OpenAI can lead by a bit in order to kind of peer into this future and really understand what is possible. How do we shape this technology? But we can't do that alone. And I think that having coordination and especially the more that we can talk about safety techniques and share what we're seeing, alignment failures, those kinds of things, all of that is going to again take a very front seat for this next phase.
Right. Very interesting.
You've called the OpenAI Hugging Face recent incident a watershed moment and talked about how the defender window is now open. Can you explain that statement and the significance behind it?
So I think Hugging Face shows two things. One is call it a something for us in terms of how we monitor sandbox and control the models during evaluation, and that's something we've really risen to that occasion. Our team has totally changed so much of our internal standards and really implemented a lot of controls that I think are very important and very critical as we look to future more capable models. But there's a second thing that I think is also very valuable for the world that came out of this, which is an insight into what future capabilities will be like when they are broadly diffused and in the hands of threat actors. And that will happen, right? That there are so many people who are building these models, and again, there's something very important and good about the diffusion broadly of AI capabilities because there's a risk of concentration of power if one or a few entities, people, huge risk, right? It's something not to at all write off, but you also have to prepare for if everyone is empowered with tools that are cyber capable. And in the case of Hugging Face, you saw both an AI that was able to hack out of a secure environment and hack into a company's production environment. And I think that the takeaway...
Very cleverly, right? And it's like that the things that it found were quite sophisticated.
And this capability broadly diffused, I think, is something that will really empower threat actors in new ways. And I think that defenders need to use this time before that technology is broadly available to secure themselves. And the nice thing about it is it's a dual use, right? It's something where if you can find vulnerabilities, if you're an attacker, you can use it for no good. But if you're a defender, you can patch, right? If you're a defender, you control the battleground, right? You control the setup of your systems. And so our belief right now is that there's this window of you have frontier capabilities, you have the broadly diffused capabilities, and you as a defender by default, you know your security is probably pretty static, been static for the past five, ten years, that kind of thing. You need to move, use these frontier capabilities that you'll have differential access to, right, where we have trusted access programs, things like that, to bring these capabilities to defenders, and you can use that to move yourself up so that as the frontier capabilities get better, you get pulled along too, right?
Okay, so I've got a comment and a question on it. I would say there's a third thing that we learn, which is like these things have capabilities that I don't know that we all understood before, which on the good side, like, oh, I can deploy 10,000 agents and they can talk to each other and organize themselves and do stuff for me. Like that's pretty amazing. So that was on the good side. On the other side, so I agree that we've got a kind of defense window. However, we have like 50 years of code and architectural ideas and deployment ideas that weren't built for this world. And so yes, the AI can help us like, okay, find a bug, patch a bug, and so forth. But it seems like there's, you know, maybe a bigger issue, which is we have these huge, you know, massive honeypots of consumer data and all these things lying all over the internet. And, you know, from a consumer standpoint, it's like, okay, I can't protect my stuff. These all these companies have to get their act together, which seems a bit worrisome. And do you think kind of in the future we need a, do we need a decentralized consumer architecture? Like will this kind of current world that we live in with all these centralized data repositories be viable in a world of AI?
So several pieces to the answer, and first to your point on what you can get out of 10,000 agents, we actually used 10,000 agents to solve the Navier-Stokes problem.
Yeah, that was pretty awesome, by the way. Congratulations on that.
Thank you. Thank you. And it's both an important problem for what it is, has significant implications and applications to fluid dynamics, to how you think about ocean currents, all these things, but for what it represents, right, of new knowledge created by AI and it unlocking a whole wave of scientific discovery, medicines, all those things, they're on the table now. So I think there's something really amazing to think about what can happen through the power of AI that is able to really help solve problems. And in the case of cybersecurity, how I think about it, we at OpenAI took our models and applied them to finding vulnerabilities. We took 25% of our production engineers and said, 'Sorry, all your projects are on hold. You are now defending. You are now upleveling our security architecture. You're going to use the models to find all the holes.' And we found a number of serious issues and we fixed them. And I've talked to a number of CISOs over the past couple weeks and months, and there are many companies who are also telling me that, yeah, that they've applied these models, they found some very significant issues, but that they're able to fix them. And one positive sort of part of the story is that when we took Astra pointed at our systems, we found some new problems, but eventually it saturated. We basically found, to our knowledge, all of the P0s, all of the critical problems that Astra is smart enough to find.
And of course, there will be a new model. There will be a new round.
But I think that that's the world that we'll be in, is that you'll be in a world where you want to be in this tight loop of new cyber capability drops. You deploy it against your systems. You find the new holes, and ideally you've managed to automate this what we call defense factory. And that's what we're building internally, this end-to-end of both find vulnerability, triage it, remediate, deploy, validate, right? That end-to-end. And if you can do that at machine speed, I think the defenders will be advantaged in deeply significant ways. And there are ideas, for example, formally verifying all of software that are possible with AI.
Yeah, we never, that's always been a dream. We've had these formal languages and all these kinds of things, but they never kind of took off.
That's right. Because it's just intractable for people. It's just so hard. But we have these that are solving these crazy impossible math problems. And so one application of that, that proving power, right, actually one of the things to know about the Navier-Stokes problem is that we formalized it, right? We formalized it into Lean, and so...
So the AIs can write verifiable code.
Yeah. Very nice. Yeah, that's a great idea.
So I think there's real hope, but I think that our view is that the world needs to act with urgency because we're in a very dangerous window right now.
Closing the loop on this incident. Is there anything you felt that the narrative got wrong in an important way, or is there any preferred way of talking about what happened or this window that's important to get across when you think about the public narrative? Or is there anything just inside the company else that changed in terms of how you're approaching the set of issues?
Well, two things. I think that one big theme that people should take away from it is a question of access, right? That these capabilities exist right now in the world, but that they are in a small number of frontier companies, and the frontier companies have a trusted access program, which means that anyone who's not in the trusted access program is not really able to benefit from the fact that there's this differential. The people we're in, they get a benefit if they use it. And so I think that there's something we need to do as a field and as a society to really scale up the number of defenders that have access to these technologies because it's like every day matters, and to use those days you need access to these tools. And one thing that was actually interesting about the Hugging Face response was that they said that they used frontier models to try to look over the logs of what had happened because that's the only way to actually analyze an attack like this. And that they said the frontier models refused, but they didn't actually try our frontier models, and they actually believe that ours would have permitted it. And so there is something too about the default stance of providers. I think there's something here about using these capabilities for good with this urgency and this sort of real sense that it has to happen. I'll tell a quick story, by the way, which is unrelated but maybe also shows a little bit about how I think about this. I remember when we trained GPT-3. It was the beginning of December 2019. So everyone's about to head out on vacation. Y's like, 'Okay, we can train the model.' And I just remember feeling like this model is sitting on a shelf. No one is using it. It's this like amazing technology, new to the world, new to humanity. It's like every day that no one is exploring what it's capable of and trying to understand it, figure out what to do with it, that's a day that is lost to the world. And so I was just like, I canceled basically all my holiday plans. I spent the whole time just playing with the model, building interfaces around it, trying to see what it was capable of. I remember I was trying to teach it how to sort lists of numbers. It didn't work very well. But it was just like this really try to probe it and see what's possible. And I think that that spirit and ethos is something I think we should bring to what we're building today. It's obviously at much larger scale, much larger impact, but we as a world have the opportunity to understand this technology in this moment, which then helps us shape and steer where it will go next.
And very good point.
You said two things. What, do you have another one? One was access, or did you say both of them?
I think I said both of them. Yes. Yes.
Let's go back to Astra. It's incredible to see all the excitement on X, all sorts of cases. People are excited about computer use. You've said that in some ways it is bringing us closer to AGI. What do you talk about what you find most compelling in Astra, or what you think the breakthrough there in light of that statement, and where we have still left to go?
Actually, wait, sorry, let me actually revise my answer, and I can say, so both access, and let me also tell another story about how I have used the models personally. So after Hugging Face, I was thinking about how can I use these models in my personal life? What can I do to secure myself? And I have a website. It's a very simple website, gregbrockman.com. Not the most popular website.
Got a good blog post on it.
Exactly. You got some blog post. It's a static site. It's very simple. Like what kind of vulnerabilities could be there? So I took my Codex and asked it, go check out gregbrockman.com. Tell me if there's any vulnerabilities. So I did a pen test and it came back with 13 findings. And these findings were things like I had set my SPF record so that you would prevent people from spoofing emails, right? That there was some, it was serving over HTTP without forcing people to HTTPS, things like that. And individually, these things are maybe not the biggest deal, but if you think about with an AI that's able to chain together many small vulnerabilities into a big one, I'm like, do I really want a hole where someone can scoop emails for me? Probably not. So 15 minutes for it to find these 13 findings. But then I asked it, can you fix these? Right. Because fixing is so annoying.
Exactly. And so 45 minutes, it opened up my Cloudflare control panel. It clicked around, set all the headers. It migrated me to Cloudflare Pages. It set everything correctly. It started the DMARC process, which apparently you have to do like a 48-hour window of whatever. And that was 45 minutes of fixing. And I felt so protected. I felt like, wow.
Did you ask it to find the vulnerabilities?
There you go. No, I added... Yeah. So it actually does do that automatically. This was by six soul. It said, I just checked that this one's fixed, this one's fixed, this one's fixed, and in 48 hours I'm going to have to run and set up a little automation. So in 48 hours, it would check back in to complete the DMARC process. And I was like, all right, this is, we're in business now.
All right. All right. Craig Brockman.com.
There we go. You too can be protected.
Awesome.
Let's transition to Astra. It's incredible to see all the excitement online and the use cases. People really excited about computer use among other things. You've said it's sort of closer to the way along to AGI. I'm curious what you find most groundbreaking with it and where do you think we still left to go?
Well, I think that Astra is really a step function on so many axes, and in many ways it is the sum of a number of research bets that we've been making for years, and to see them come into one model at one time, it's been absolutely incredible. And so just one thing to know about how we do numbering is that we kind of have been wanting to have GPT-6 represent something that's worthy of it. And the problem we always have is that our models are kind of incrementally getting better. And so it just never feels like it's a right moment to go for a major version bump. You always have to be like, it's 5.6 now, should be 5.7. And this one just happened to be because all these things came together at once, the first time that we actually had this almost discontinuous step in a way that we could have predicted, but it just was like all these factors happened to line up at once. And so it was a real positive moment. And to me, the computer use is the headline thing that we've talked about. And part of the reason computer use is so significant is that for agentic use cases, it really comes down to tools. It's like, is the model smart enough to use the tools, and then does it have access to the context that it needs to through these tools? And so people have been building these MCP servers and these CLIs and just really sort of taking the world of software and making it accessible in this almost stilted way that is not really meant for humans, right? It's like we're kind of retooling the world, right? You made it. It's kind of like, oh, we'll build an API like it's a software, but like what if it's really more behaving like a human? Can it just use a computer? And the result of building that other layer, you have now another layer of security challenges, this, that, the other.
