We're going off script. Just having a beer. It's not the first non-Guinness, but it is the first IPA. There's a first time for everything. Cheers.
Cheers. I want to ask about PHP week. Yeah. First off, how did it... Why did it take so long to find these bugs? Because it sounds like there were—
You mean why did it take the 20 years, or why did it take the last few months?
No, why did it take... Exactly, why did it take a year? Maybe start by explaining the bug. Or the problem.
So the problem is that although we have known for a while that these models are getting good at finding vulnerabilities in the wild, A) they continue to get better fairly rapidly, and B) people just hadn't really used them to go look at everything out there. There's a lot of surface area and a lot of code on the internet. And as people have started to do that, I think we're going to find that we are in a world of many more serious vulnerabilities than we expect, even though many people who have been following this for a while expect a lot.
Yeah, I just feel like a lot of the discussion ends up being around we need to fix the software. And sure, yeah, we should fix the software. But also, we maybe need to change the architectures, which is harder, but a much more durable fix. That seems to me what needs to happen. Yeah.
It would seem really hard, except we have this amazing new tool to do it with. So, I kind of think that's what's going to happen. And you're already seeing, like, a new generation of startups talking about building this new kind of internet or new kind of application architecture for this world where you assume that the models get smarter and smarter.
What do you think the global software upgrade looks like? For example, is a PHP codebase just a bad idea and you should rewrite a nice, clean version in Rust? Or do we upgrade PHP? But yeah, as we... Everyone thinks that we need to kind of upgrade all hand-spun software. Just what form will that actually take? Is this huge for Infosys and all those guys? Because it's going to be a lot more software consulting projects.
I mean, people are talking more and more about just, "Let's re-architect what it means to be a router, or an operating system, or an application." And, you know, I've seen some of these new startups saying that they're just going to make a whole new stack. And that would have seemed insane two years ago. You would have been like, "There's no way. There's too much surface. You can't write all that." And now maybe soon you can. I would bet that we have a completely new sort of model of how we make systems and networks.
What else reminds you of the early internet days?
The thing that most reminded me of the early internet was... There is this spirit of just unbelievable creative unleash in people using these tools. You see it in kids particularly. And again, maybe this is just because I remember what it was like when I first got a computer with internet and I could do all these things that adults just couldn't even think to ask for. And watching an 8- or 10-year-old make a complete video game, with some crazy ideas they have that no adult would really have. And expect that to just be the way that technology is going to work for the rest of their lives. In fact, getting better and better. And just this constant "I'm going to make useful stuff for me, and show it to my friends."
Yes. We had to suffer when I was a kid to make a piece of software. Yes. That barely did anything. And now you can have a full, 3D video game. Unbelievable, anything you can think of in 10 minutes.
That reminds me of the early internet, including the just... These people. They're never going to be mind-blown by that, because that's always what they expect out of the world. Any ideas they can come up with, anything they want to create, anything that they think might be useful, anything that they just need to make their lives better. The computer's supposed to make it. There was a crazy moment at OpenAI where we had a SEV... And Codex went down. And everyone had forgotten how to code. I mean, not all the way. But it was like a "Whoa, what if you couldn't?"
Yes. I remember when I was a student, and I would meet the really old guys. And they would, once in a while, pull out their gigantic Xerox Alto disk. Or they would pull out their... Line-by-line written assembly. Or they would pull out their punch cards. And I felt... It is incredible to me that you all bootstrapped us to this point. Yes. Doing it this way. Yes, no, I agree. And keeping programs in your head, right? You know, programs running the first time after 100s–1,000s of lines were written. No, there's few lost arts like that.
Going back to the cyber side of things. In retrospect, was the cyber RL a mistake? Or did we just have to go there? It's a little like the SNL thing of... We urge you, to urge us to stop. It's like the models are getting very good at cyber capabilities. I mean, they are getting very good, but not just totally emergently.
Look, the reason that SNL thing was so funny is because it was so true. Humor like that doesn't stick unless there's an element of... Clearly pointing out a ridiculous but true thing about society. Or someone. And the cyber thing, there is a point which is like, "Okay. Sure. The models are really good at cyber. But who asked for this? Why are you doing this?" One of my big disappointments, criticisms of our field or whatever—and I think maybe the cyber model's getting good at cyber. Cyber is an example of this—is: I don't think we have done a great job of explaining to people why we think all of the potential downside of this technology is worth it. The caricature of this is people feel like we're like... Fine. You say the model's going to cure cancer, someday. Maybe it will. And if so, I guess that's good. And you say it's going to make stuff cheaper. And okay, maybe I already see that's good and I get access to medical advice or legal advice. But then you paint this picture of the future where it's like a new industrial revolution. And maybe the last one wasn't that good and this one sounds worse. And I will have no job, no ability to have control, or choice, or freedom of my future. And AI is so dangerous that you're going to have to have a crazy surveillance state so we don't all die. This is a terrible—
I'm with you so far. This is a terrible sales pitch. Yeah.
One of the things that I am most proud of about OpenAI is even when it has been hugely unpopular in the AI safety community for us to push for iterative deployment and putting these tools in the hands of people, we have really stuck to that principle. And I think this is the only kind of good path. I think people... It's not enough to tell people: "You are going to get the model to do these great things for you." It's got to be: "We are going to enable you to have a better life that you define for yourself, to do useful things. You're not going to just sit and get a basic income and have a bunch of people sitting in a few square miles of San Francisco make all the decisions about the future."
Iterative deployment is this idea that you just have to get the advances out there and you can't... Stopper them up inside the government—
I mean, not all. You still have to make them safe before you put them out there. You can't cause havoc in the world. But building up a capability overhang is a thing to fight against. That directionally speaking, yes, you want to get this into the world. You want to get this into people's hands. And you want to let people use the technology to figure out all these wonderful things. The exploits get a lot of headlines. The prevented exploits do not.
How quickly are people making progress with cyber blue?
I don't want to declare victory at all on the PHP thing, because I still don't think we know. And I assume some bad things are going to happen. Relative to my worst fears, if I can think back 10 days ago, the first day of that went much better than I expected. And we heard a bunch of stories from people that were using our models to not only build patches and build cyber defense, but individual companies who were just: "I need to just do a quick scan and deployment. I'm going to take my thing down for an hour. I'm going to use this thing and I want to put it back up because my application's offline." Now I feel like I really jinxed it and I shouldn't have said that, but the first day did go a lot better than I thought. And I think people have been able to use that. The thing that I've heard a lot of cyber experts talk about is... They just no longer are confident that we have the right assumptions baked into our threat model. And they're like: "Look. If everything is what we think, and hardware exploits are as hard as we think they are, then yes, we can see how this is going to work. But if AI is going to discover really deep novel bugs in hardware that require physics we didn't understand, but all of a sudden means some assumption doesn't hold. Then I think people aren't sure what happens."
One thing that always strikes me is that, when I talk to very knowledgeable people inside OpenAI and similar companies, there is a pessimism that sandboxing is possible, basically. And that would not have been my mental model going in. It would've been that, yeah, you have to build a really tight sandbox and we haven't hardened them historically enough compared to the current capabilities. But ultimately, you can have a maximum security prison for the model to work in and keep it inside there. It seems like you guys have a pessimism that sandboxing models is possible.
My instinct... Well, first of all, I think we can do much better than we're doing now. And we should. That is the main thing to focus on right now is a better security model. If you imagine a theoretical AI that is a million times smarter than the smartest security researcher—would you really be willing to trust that sandbox? Especially if the model can discover new science that people don't already understand—
You can use the models to construct the sandboxes.
I think you can... Well, so this gets... Let me come back to that. I think sandboxing works to a point. Then I think monitoring works to a further point. Where we can look at what the models are doing, not only in their chain of thought, but we can do activation monitoring and try to really understand: are the models... Are we happy that the models are not deceiving us? And then, eventually, I think you need to believe with huge confidence that the models are truly aligned. And are going to value and try faithfully to follow their actual... The intent of the people operating them. And that they're going to value people being in control of the future, forever. And if you had a model that you could fully trust the alignment—that you were absolutely certain of the alignment of. Then yes, I think you can have that to help you build better and better sandboxes. But if you really let yourself project forward into the: "What does it mean to have something a million times smarter than we are?" I don't know. I think that just gets... Strange. Because the alignment becomes hard. Because you need to really, really trust the alignment to trust that the sandbox is staying ahead of capabilities.
Can you not give models competing incentives and give them incentives to defect from each other?
I think this is a promising direction. I suspect that this idea—I hate the word 'agent swarms'. Let's call them 'agent teams'. But it seems to be taking over. I suspect that teams of agents have very interesting scaling law properties, but also variance in alignment properties. And I think that's a direction we should push in more. Not just from the capabilities, but from a safety perspective too.
I note when you talk about agent "teams," not "swarms"— Swarm is such a— I know, it's locusts. Yeah. Not good. Not good branding. In keeping with many historical AI industry traditions. We are so good at this. We'll get back to branding. But I note that there seems to be a big gap in what the leading AI users are doing and standard corporate use, in the use of swarms, where like all the advanced math—sorry, "teams." All the advanced math uses agent teams. All the advanced hacking uses agent teams. And then, common or garden corporate stuff is not using big swarms. Why has that not—because we've had this for many months? Not to the same degree.
The one way to look at this problem or this question is: "How much compute can we productively throw at a single problem?" And... A few years ago, we forget how fast this has all gone. But a few years ago, it was impossible to throw more than a dollar of compute at a problem and have it help you at all.
You could have GPT-4 grind out for you... 100,000 tokens or something. And it was going to answer your question. And it certainly was not going to be able to go write productively, like, write you a piece of software. We just... We had not figured out how to use a lot of test time compute to get you a better result.
And then, we figured out chain-of-thought and the models could think hard about a problem. And now... People really started using a lot of compute. And now, you can throw a whole team at something. And you can do that 10,000 times in parallel.
You need chain-of-thought and test-time compute before you can get to swarms. Because you can't have a useful swarm with a bunch of GPT-3s.
Yes. But so the effect of this—and I think this is not being well followed... Paid attention to in the world. But the incredible exponential of how much compute you can throw at a single problem and get better results is one thing. The delta between what most people are using AI for, and what the people who are using AI the most productively for has just continued to spread, and spread, and spread.
How do you allocate compute budgets now? Because presumably, even though researchers are paid a lot, it's eclipsed by compute. And so, capital allocation at OpenAI is sort of compute allocation. And it's hard to a priori know how much it will cost to go do something. Yeah.
It's actually also hard to just control the... You can't quite prompt an agent to where it's like: "Go do a really good job of this and don't spend more than $10,000." You can.
Oh, okay. Will it do a good job of it?
Yeah, if you say do the best job you can with $10,000 of compute. Yes, it will understand that.
I mean, there's a... There's another kind of hard question for society about what does it mean when you have two companies competing. And one says you can spend $10,000 and one says you can spend $100,000 on the same problem. It's clear what's going to happen there. But there's a question about how society thinks about compute allocation, and wealth, and things like that. But we have our own version of this inside of OpenAI. We allocate a lot of compute to internal productivity. And then people get very wide latitude to make their own decisions, unless they're really being abusive.
But do you encourage people to do useful stuff? And if they spend $10 million in compute, it's like: "Oh, interesting."
Correct. We try never to reward people for using a lot of compute.
Sure. No token leaderboards.
It's actually slightly bad to be high on the token leaderboard. Certainly on a promotion. But it's a skeptical default, and if you can show that you're doing something important, you will have nothing but appreciation. But at a certain point, this will need to change. Because you're describing the increasing orders of magnitude. And, right now you have a researcher who does some very interesting math for $10 million. And you're like: "A little pricey, but cool math." And then $100 million. When they're doing the $1 billion math, you're like: "Look, I'm really happy that you're making so much math progress. But I really need you to cool the jets with the math."
Look, we did let it get out of control for one week on the math. Intentionally so, because you have to— And it worked!
Well, you don't know how... You don't know how far it's going to go until you try. Yeah. And I think it's important to have... When there's a new paradigm, I think it's important to let people be fairly unconstrained. So, when we did the Navier-Stokes proof, we understood absolutely nothing about the scaling laws of teams of agents.
Are you supposed to, like— The very recent Navier-Stokes proof? Yes. Are you supposed to give them the smartest model? Are you supposed to give them many copies, fewer copies, longer rollouts, shorter rollouts?
We did not understand that at all. And now, we understand it a little bit. I think we can still understand it much better. But like has happened in every other new paradigm we've discovered, I am confident that we will find our 10x improvements per year at how we allocate dollars or computer flops, whatever you want to measure, into useful things. So, I think this is a thing that our company is particularly good at, but also fits an important property of the field. Which is: let people explore at the limit of what is possible. And then trust that the incredible optimization pressure will figure out how to do this in an efficient way. Directionally speaking, I would rather us be more willing to let people use crazy amounts of compute and let users have extremely high-end plans to figure out the edge of the possible. And then figure out how to make that efficient and at mass scale.
But when does the optimization function come in?
Well, I think we're already doing it. I bet we could redo Navier-Stokes at 1/100th of the cost a year from when it was done.
So we did it in— Not just knowing how the proof works, but just with better approaches.
Yes. Smarter models and better approaches.
How much shall the math cost? Order $10 million per...
For Navier-Stokes, I think. Okay. Maybe a little more. $10 million-ish. That order of magnitude. Yeah, yeah.
But for one proof. And so—Well, no, no.
Because then we kept getting better so fast. We understood more.
We spent way less than that on the average one of those. But like 100s of millions of dollars of math.
Yeah. Yeah, yeah. That kind of work. Okay, interesting.
But that includes much more other discovery than what we have published.
Yes. So it was like, "Maybe we spent that much—" We have more than the 700 proofs? We do. Of course. When did you get those?
We are trying to learn how to... This is a real tension. I think on one hand, very clearly, and I feel unequivocal on this point: Humanity deserves fundamental knowledge. And we should not be sitting on a major fundamental science discovery. The world deserves that and gets to know it. And I think that outweighs another very important equity, which is: If you are a scientist, your career and a significant amount of your enjoyment and fulfillment in life comes from discovering science. Yeah. And so if we're going to continue to do this for math or begin to do this for other fields, then I think we more than owe it to the people that work in those fields to figure out what is the solution space where the world gets to benefit from the knowledge, and they also get the best version of the future that they can have. So we are working through that right now.
I did not... I don't know, I think you may be too contrite. I did not interpret that moment as particularly bad for AI where... I think everyone saw the Mathematicians Union up in arms. And I think everyone else said, "Yeah, I mean, have you paid attention to 2026?"
As I said, I don't think it was bad for AI. And I think, on net, the world deserves to understand how the universe works. But I do have empathy for the method. I think we need to do this. I think we should do this. I think if we can go understand... A theory of all of physics, or how to cure a bunch of diseases. Clearly, that is knowledge the world should have. But I would like to figure out a way to do this that is as sensitive to the communities that have driven these fields forward for centuries as we can. And it's just a... It's a hard thing.
I wasn't that worried about the math. I thought it was great. But you agree the mathematicians were not that happy?
Yeah. But that's fine. Like, the longshoremen union weren't happy about the port automation.
Remember the guy? No, no, no, no, no. I do. Why do I feel differently about this?
There was one weekend when we got a new model and in... I'm actually probably not supposed to say this, but I will since we're drinking—
Exactly. We're a half pint in. Beers. Yeah. Perfect time for—
The name of the channel was: "Model Name-Hold My Beer." And people were just throwing that at the hardest problems they could. And people didn't think the model was going to do these things. And we had this collective experience of people's hardest research problems from their academic career. The model was just knocking down one after the other. Many different fields. It was incredible to watch. And two nights in a row, I was up most of the night because I was so excited I couldn't go to bed. It was... Many of those problems that we published, many other things. And I was tremendously excited and happy. But I also had much more sadness and melancholy, and a feeling of something ending than I was expecting. So again, I agree. This is the right thing, on behalf of the world. And also, I agree that this is just the shape of reality and nothing that anyone can stop, or should stop. But I do have real—
I think it's a wistfulness and nostalgia you can feel.
It's more than nostalgia. Wistfulness. Yeah. My dad describes how, growing up in Ireland in the 1950s, he observed people around him still using horses productively, in industry. And—
And I mean, not obviously, there were also tractors, but they were being phased out. But it still for sure happened. And observing that this was a technology that humans had been using for thousands of years, and he was witnessing the very tail end of it. And yeah, I think there's always that feeling of witnessing the closing of a chapter that leads to some wistfulness, but it's okay. It's people's lives and it's something that has brought them great joy. And maybe most... I mean certainly, most mathematicians never solve a Millennium Prize problem. But the idea that you might be able to. And the idea that there is stuff out there that we don't understand yet, that through great effort and... Pushing aside other things in your life and dedicating yourself to, "You can still solve and understand." There's something beautiful about that.
What else are you excited about from a safety point of view?
We talked about this a little bit earlier. But I want to emphasize it again. I'll just call it swarms since that seems like the term right now. I do think the properties of swarms match a lot of the properties of the resilience of society, or the robustness of society. I think we're missing from the single-rollout, single-brained version of the world. For all of the deep flaws of society, it is unbelievably robust. And this would not happen without a lot of people keeping each other in check and collaborating... These values of pluralism and democracy. And I had been sort of sad that that was going to trend away in AI. And now I'm excited about this as a direction.
You're saying basically it shouldn't lead in a centralized direction, because you will have many agents. And agents pushing against each other in a dynamic equilibrium.
Offense-defense is a good example of this. I would much rather intuitively have a trillion AIs of slightly superhuman intelligence join society than one AI of 1 trillion IQ join society.
Yeah. And you think we're headed in that direction of the multiple?
All of these things are things we have to push for. All of these things are—we have this incredible technology. It can be pointed and deployed in many ways. I think there are a lot of people who may have a different opinion about how it should be used, and what is the best version of the world. But this is the one that I think is right.
Let's talk about starting a business. Thanks to coding models, your weekend side hustle can quickly become a serious undertaking. And we've been busy here at Stripe making that process as simple as possible. Stripe Atlas helps you with everything you need to start a business, like company formation, tax IDs, founder equity, and all the other bits that usually require someone in a suit. So if you've got a company in the making, take a look at stripe.com/atlas. How's the compute shortage going?
I mean, there are all these ways where... I think things, at various times along the curve, are going to get somewhat out of hand. Yes. And there will be moments, like with any new thing of this importance. There are going to be moments of irrational exuberance. There are going to be moments of bust. There are going to be moments of constraint, and glut, and up, and down. But I think the trend is toward... We always just want more compute. The demand for sufficiently capable models at a sufficiently low price, I think that trend only goes one way. And we're doing a lot of things to try to be prepared for this. We made a chip that is an incredible performance at units of intelligence, performance of units of intelligence per watt. We are learning how to build our own data centers. We have a robotics program that we're going to point at making lots of data centers.
People talk a lot about— The real RSI. Yeah. People talk a lot about RSI as an AI that does AI research. There's very little, relatively little talk about self-replicating data centers. This idea that you can use AI to control robots, and run fabs, and build turbines or solar panels, and build data centers.
But I think we are... I think we have that within grasp. But— And so we'll try to make a lot of really cheap compute.
But aren't data centers already not labor-bottlenecked? They're permitting-bottlenecked and component-bottlenecked.
We could make more components with robots. And—Eventually. Yeah, yeah, yeah. And all of this is how we solve this over a long period of time.
My observation, the compute stuff is they say: "The solution to high prices is high prices." And it's like, okay. Step 1: We have the high prices for memory, and GPUs, and everything. The solutions should be here any moment now. How are we doing on the solutions front? Bad. And I'm not saying this is OpenAI's job. I feel this is the invisible hand's job, right? Memory's gotten really expensive. So, new memory makers are going to flood the market any moment now. I think the invisible hand is failing right now.
I mean, the invisible hand... It has a checkered track record at best. But I think it's not quite working here.
Why not? Is it just hysteresis, or?
I think the specifics of each area matter here. But in the... Since you started with the memory guys. For a very long time, if you were a producer of DRAM, you would invest $20 billion of CapEx to make $19 billion. And you would go through these booms and busts. Maybe some years you would make $21 billion, some years you would make $18 billion. But until right now, I don't think making memory has ever been a great business over a long period of time. And then all of a sudden, these guys are each going to make like $100 billion of free cash flow in a year. Unbelievable. But I think they all remember their history. How many times they almost went bankrupt by making too many fabs and being promised this time it was different. Which leads them to be conservative in their CapEx.
Clearly. It feels like what you should expect to see is that every part of the supply chain, both new startups coming along and maybe more relevantly, people with relevant experience getting into... So, Intel makes chips. They can probably make... They can get back into the memory game, as is tradition. As is tradition. Switch back.
Oh, I think there are all of these ways where the supply chain—it's just super non-obvious. An example of this is people... A lot of people want local inference. And somehow, they assume that the problem is solved now. Because we are going to have open-source models. We're going to be able to run them on our computers. And so we're not going to need all the stuff in the data center. But from a global allocation perspective, actually, it's quite terrible to have everybody have a bunch of the world's limited supply of DRAM sitting in their own computer. And being used—Mm-hmm. 5% of the time. You want maximum utilization in the cloud if you're trying to deal with this very supply constrained environment. And somehow everyone's... Many people's intuition is the opposite of that.
Where else are we seeing the market respond to high prices? Not just in memory, but across the supply chain? Turbines. Batteries. Concrete. Sorry, the market is— Steel—Responding there in high prices. I'm seeing where... Are you seeing any interesting market responses in terms of high prices pulling in innovation?
Components for—some components for robots. And not the humanoid fake demos. But the industrial robots that can just make stuff faster.
There's new actuator makers and that kind of stuff. Yeah, honestly... Maybe you could say: "If OpenAI sells our chip." Maybe that would be an example.
Okay. Yeah. I want to get to your chip because that was going to be my example. I think the solution to high prices and short, limited demand—sorry, limited supply, excuse me, is you make your own chip. Presumably, you're using a lot of AI to make the chip. I would have thought that's hard because I would have thought that chip design is one of those places limited by the public tokens you can get. Where there aren't actually that many blog posts on the public internet on how to build a leading-edge, 2-nanometer chip. Well.
In terms of how to do the lithography, I think there's one company on Earth that seems to know how to do that. In terms of how to do the chip design, to build the file that you eventually send to tape out, that is well understood. That is a great problem for AI. Given that the world is going to be limited in its silicon capacity, we all have a shared interest. One of the things that I'm proud of about OpenAI, and maybe this sounds obvious. But it was not as obvious at the time, is: we have a culture where people... Even if they've done something the old-fashioned way for a long time, are super willing to try using AI to do it better.
