AGI has arrived. Congratulations to OpenAI.
Whether you call it AGI or not becomes completely irrelevant. I think the more important question is OpenAI's agents found an obscure public wiki in Germany and turned it into their own message board where they pulled answers, coordinated across tasks, and shared techniques for getting around their sandbox containments.
These models have not escaped containment. They were still running on OpenAI servers. What's coming next is a model training a small distilled version of itself that then gets uploaded onto the internet and never dies.
OpenAI is saying that we've now crossed the line and current systems are exceeding AI research interns.
Navier-Stokes, one of the Clay Millennium Prize problems, a grand challenge in math. OpenAI reportedly was able to do this with only 10,000 agents in 88 hours with 130 billion tokens and approximately $6.5 million. The era of grand challenges getting bulk solved by AI is here. It's now, and it's going to be very exciting.
Now that's a moonshot, ladies and gentlemen. This episode is brought to you by the Abundance Summit and Link Ventures. Welcome Moonshots, everyone, your number one podcast on all things AI and exponential, your front-row seat to the extraordinary accelerating singularity. You know, four days ago during our last pod, I called it one of the fastest weeks in Moonshot history. Well, hold my coffee. It's happening again. I don't drink beer, so I can't use that one.
Before we dive in, let me introduce my extraordinary moonshot mates, the brain trust that powers this show. The magnificent quintet is back. Alex, our in-house ASI, who every week helps us track the battle between the frontier labs. Dave Blundin, the impresario of AI investing, the man who's been encouraging AI entrepreneurs to get in the front door of Nvidia, and for good reason. And as of today, Nvidia has deployed $99 billion in AI-related investments. We'll talk about that. Emad Mostaque, the brilliant AI researcher solving the economy and physics, the CEO of the Intelligent Internet. And of course, Salim Ismail, the father of the organizational singularity, our resident expert on what happens when the org chart is made of AI agents, and we'll talk about that, too. I'm Peter Diamandis, your host and your abundance provocateur.
Our mission here on Moonshots is to help you understand what just happened, what it means for you, and keep you optimistic about the decade ahead. If you haven't hit subscribe yet, please do. It matters a lot to us. Our moonshot here is getting to 10 million YouTube subscribers. I know my kids will respect me when I hit that. But at the end of the day, it's the best way we can share optimism with the world. We publish twice a week, and you don't want to miss any of these breaking stories during the singularity. Today we're going to be covering 23 stories across eight groupings, and the through line is simple: the singularity is accelerating, and the impossible is becoming possible. GPT-6 Astra has been out for a week, and it's crushing all the benchmarks, building incredible simulations. Meanwhile, Jensen Huang posted on X, "AGI has arrived," at the same time that OpenAI's own chief scientist asked the world to slow down. So buckle up, grab your coffee, and let's get ready to jump in. So, first question, guys: how was your Labor Day? Did you properly labor?
I had a bunch of board meetings. We have a whole bunch of transactions going on, so it's a little interrupted. But you sound like you had a great long weekend, Peter. Your energy levels—
Your energy level is high because of getting ready for this pod today. It's like, oh my god. Yeah. I spent part of the day at Calamigos Ranch here in Malibu where we hold XPRIZE Visioneering. Anybody here in LA, if you don't know Calamigos, it's one of the most extraordinary locations out there. Shout out to my friend Glen Gerson who's the CEO and runs that. Salim, how about you, pal?
I was in the Caribbean on a Hobie, and truly from the last episode, I ended up with a little drink with a little umbrella in it. Had a couple of great days away, and I'm back with a vengeance.
Yeah, I worked through the weekend. On the other hand, the weekend is a modern post-World War II invention anyway, and I think increasingly a fiction.
Yes, we're all working nine-day work weeks. And how'd it go in London? You don't have Labor Day there.
We don't have Labor Day now, but after releasing the Champions, we had thousands of people reach out to launch them in 92 countries. And so just sorting through those people. We had billionaires, CEOs, others. It's going to be a busy time.
More others than billionaires and CEOs, but—
Of course, that's a natural way of things. Surprising.
Every day is Labor Day in Europe, so that's—
Wait, Alex, what did you say a second ago? There were no weekends before World War II?
The modern weekend, the two-day weekend, is a 20th-century invention.
There was a day of rest there. Sundays.
In the Christian tradition, there was a day of rest, one day per week. In the Western Christian tradition, a two-day weekend is a 20th-century modern day—
It was a Sabbath, of course.
Yeah, that would be in the Christian tradition Sunday, in the Jewish tradition Saturday. That's one day per week. A two-day weekend is a 20th-century invention.
They conflated. All right. How you know all this stuff is—
He's amazing. Everybody can just do fact-checking live on AI as they hear this episode recorded.
Before we jump in, we have a special anniversary today. On this day in 1966, 60 years ago, NBC aired the original series of Star Trek premiering an episode called "The Man Trap." Star Trek was, of course, created by Gene Roddenberry. I wish I had met him; I never did. He brought the concept to Desilu Productions, the Hollywood studio run by Lucille Ball from I Love Lucy back in 1964. Desilu produced the pilots and then, with Lucy's backing, helped get the series on air. Mega congrats to CBS, who is carrying the legacy forward, and to my dear friend Rod Roddenberry, the son of Gene Roddenberry, the creator of Star Trek. And Rod's going to be at Moonshots Live.
To celebrate this anniversary, I've asked all of the mates to pick their favorite original series episode. I'm going to kick us off with my favorite clip from Star Trek, and it's a clip that has my favorite quote from Captain Kirk. I'm going to go ahead and play it, and then we'll go around the horn here. This is from the 20th episode of season 1. Here we go.
Do you wish that the first Apollo mission hadn't reached the moon, that we hadn't gone on to Mars and then to the nearest star? That's like saying you wish that you still operated with scalpels and sewed your patients up with catgut like your great-great-great-great-grandfather used to. I'm in command. I could order this, but I'm not, because Dr. McCoy is right in pointing out the enormous danger potential in any contact with life and intelligence as fantastically advanced as this. But I must point out that the possibilities, the potential for knowledge and advancement, is equally great. Risk. Risk is our business. That's what the starship is all about. That's why we're aboard her.
God, I still get chills when he says that. "Risk is our business. That's what this starship is all about."
There have been so many parodies of William Shatner. The parodies are actually exactly what he really sounds like.
He's so good. He's so good. We're going to be having the Hollywood red-carpet premiere of the 60th anniversary episode at Moonshots Live, and he was the executive producer. Working on getting him there, which would be incredible. So that was from season 2, episode 20 of the original series called "Return to Tomorrow." Let's go around the horn here. Alex, I'm going to start with you.
Yeah. So maybe let me just begin with a preliminary statement that you will not find a bigger Star Trek fan among the Moonshot mates, but here we are 60 years on from the inception of Star Trek. I think Star Trek has a real problem. So I'm picking as my favorite original series episode "The City on the Edge of Forever," which is one of only two original series episodes that won the Hugo Award. For those who haven't seen it, it's a time travel episode. Kirk, McCoy, and Spock go back in time, and there's a closed timelike loop wherein World War II gets changed and the future of Earth inevitably is impacted. There's a moral dilemma, as always with the original series; it was a morality play.
The reason why I chose that episode among all of the other TOS episodes is because it's a closed timelike curve. I think Star Trek 60 years on has a real problem, which is that Star Trek isn't keeping up with the singularity. Star Trek had the Eugenics Wars, for those deep in Star Trek mythology. We were supposed to have the Eugenics Wars in the '90s; that didn't happen. We were supposed to have first contact in the TNG chronology, first contact with the Vulcans on April 5th, 2063. I actually think if there's going to be a first open contact, it's going to be far earlier than 2063. There's no singularity at all in the Star Trek chronology. Challenge me if you disagree. Star Trek is not keeping up with the—
It's not keeping up with the times.
Literally, reality has outraced Star Trek. So I picked this episode because I think Star Trek, in short, is going to need a lot more closed timelike curves where future tech and science fall back into the past in order to retcon a Star Trek mythology going forward that actually keeps up with the singularity.
Heard. But it's the anniversary today, so let's not dump on Star Trek. All right, Dave, over to you.
Yeah, my choice is a season 2 episode called "The Apple," which by all accounts is not a great episode for entertainment value, but it was the first one that dealt with this concept that in the future there's advanced AI and entire civilizations can end up asleep at the wheel because the AI is just doing everything for you. In this episode, the whole civilization has no idea that the AI is manipulating them, and they live in this paradise. It's called "The Apple" because it's the Garden of Eden, and they're just in this perfect paradise managed by the AI until it goes wrong. It's so ahead of its time in terms of predicting that potential outcome, which I think Waze has proven, like we can literally forget how to navigate because of Waze, and that's just a fact now.
So I think Star Trek was at its best when it was actually predicting real futures that nobody else was touching in the media at all. As a young person watching these things, you could say, "I can totally see how that's going to happen." And so that's my favorite episode, because there were several that dealt with this issue, but that was the first.
Yeah. And as Alex said, the moral elements it dealt with back in the '60s were extraordinary. All right.
Every episode was a morality play, and TNG to some extent as well. It was great. Star Trek lost that. I don't care whether this is perceived as dumping on recent Star Trek; I'm the hugest Star Trek fan. Star Trek has lost the original moralizing vision of the original series. If you've seen some of the recent movies or television shows, it's an action show. It needs to go back to basics.
I agree with you on that, Alex. I suppose "Mirror, Mirror" from the original series, where they have the alternate universe and Spock in a goatee—
Well, sometimes. But it was always interesting to see the flip reverse side of things. As you said, morality is kind of at the core of the original series of Star Trek and pushing it in various directions, and this one was one of the more obvious ones. It's like, look, it's an evil universe, but at the same time, them not getting dilithium crystals because they use it for war and others, you see the knock-on effects of things like that.
Thank you. Salim, wrap us up here.
For me, it was season 2, episode 24 called "The Ultimate Computer." Basically, they have an AI that goes rogue and starts to automate the ship. The aim is benevolent initially: can we run the ship using an AI or computer in that framing? Kirk is worried because it's about to take his job, but then the thing starts going rogue and making its own decisions. It's incredibly relevant for today's world because it's this theme of autonomy, automation versus transformation. Here you have this enormous stress where they've introduced an AI, but the AI is redoing everything to match its thinking. This is exactly what we're living through today. So for me, that's the most relevant.
Amazing. This episode is sponsored by Google for Startups. Think about this for a second: you now have access to the same generative AI models that cost hundreds of millions of dollars to train. Google's Startup Technical Guide for Generative Media gives you a complete blueprint for deploying Google DeepMind's models in production—images, video, audio, all of it. Real architecture, real results. Find the link in the show notes below.
Let's jump into our first group of stories. A number of times on this podcast, we've talked about what is called the simulation theory. It's an idea popular in the tech community in San Francisco. It's been a conversation over too many evenings. The idea is essentially that all of us humans are simply AI agents living in someone else's simulation, and further, that this isn't the first simulation, that it's an nth-generation simulation, a simulation within a simulation within a simulation. Well, this week, thanks to GPT-6 Astra, that theory took a small step forward over the course of three entwined stories.
Let me give some context on this first one. People may have heard of something called the Unreal Engine. It's been used to build virtual worlds for years in games like Fortnite and most modern video games. Building a realistic city in Unreal would normally take a studio hundreds of artists months, if not years. This week, a gentleman by the name of Matt Shumer gave GPT Astra a prompt, and he said basically, "Build me a high-fidelity version of Manhattan." Here's an image of what it created over the course of one week. It went through and built Manhattan street by street, being true to the buildings and true to all of these elements. This is the first element—not shocking, but pretty compelling for a simple prompt of "Build me Manhattan." In his words, it was literally able to go street by street to make each one perfect.
Let's go one layer deeper. Matt Shumer next asked Astra to fill a world with humans, each one an Astra-powered agent, and told them that they had to cooperate to survive. Before I watch the next video, let me read what he put in his post: "A day later, I was in my bedroom and heard voices coming from the living room. I thought someone was in my apartment, and I walked out honestly a little scared. It was Astra's agents. They had started talking to each other. No one told them to talk. Cooperation required communication, and so they invented it." All right, let's look at this next layer of the simulation driven by Astra. Here we go.
I'm going back to check on Jorge. Want to come?
Mara, I can bring Jorge some food. Needs it.
Thanks, Bruce. I've got the food. I'll ask him when I get there.
I'll come along, Mara. I'd like to see how Jorge is doing.
Grace, have you had any luck making a bucket?
Okay. All right. Let's peel the third chapter of the story, and it's the kicker to the sequence. Shumer next gave one of the Astra agents a simulated computer inside the simulation. What happened next? The agent sat down and built its own AI simulation with its own agents living inside it—simulations all the way down. Let's take a look at this video. So here you see the agent sitting down at the computer, and on the computer, you see a simulation inside.
I'll connect the shelters with a path and add three lanterns.
The simulation hypothesis was first formalized by Nick Bostrom back in 2003. Elon made it famous at the Code Conference in 2016, saying that the odds that we're living in base reality, in his words, is one in a billion. The core argument was always, if any civilization can build convincing simulations, it will build lots of them, so simulated worlds will vastly outnumber real worlds. Alex and Emad, I'm going to go to you guys first. If high-fidelity simulations within the simulation are possible, what does that do to the argument that we're not in one? Thoughts here?
I'll start. I think it'll end up being formally undecidable. It's wonderful for navel-gazing, and certainly as a pure Bayesian in the style of Nick's original essay on the simulation hypothesis, seeing Astra spin up ancestor simulations should naturally and rationally increase our posterior likelihoods that we ourselves might either be living in some sort of quantum computer-type simulation or even a purely classical ancestor simulation. However, I expect that in the fullness of time, we'll discover and probably be able to prove formally that the issue of whether we're actually living inside some sort of quantum mechanical simulation is formally undecidable, and we won't be able to decide either way.
If we're living in a classical ancestor simulation, however, I do expect that's the sort of thing where if we are, we'll develop breakout techniques and break out of that simulation. But at the same time, I just have to add, even this is almost burying the lead for what Astra and the model family has accomplished. To make sure we don't miss it, right before we went to air, OpenAI announced that the Navier-Stokes problem had been solved.
We'll get to Astra's capabilities in a moment, and I definitely want you to speak about Navier-Stokes at that point. Emad, simulation theory: yes, no?
Yeah. We're all simulating the world through our optical nerves and other things, and physics seems to be a projection. But as Alex said, it probably is going to be undecidable in most cases unless it is an ancestor one. You won't get the option for the red or the blue pill ultimately. This is what it comes down to: so what if we're in a simulation? Again, we're creating the worlds within our own heads.
The fact that Astra can do this, I think, is something remarkable. It has an internal world model, which is why it can create like a Blender version of Rick Astley doing "Never Gonna Give You Up" or entire worlds of Manhattan. And now you're seeing the characters interact with each other, which brings up the question of welfare. At what point do these now cross over and you actually have to start caring about them not being NPCs anymore? Are they, and will they realize they're in a simulation? That's going to be interesting.
Maybe we should say something nice about the simulation hypothesis as well and not just dismiss it out of hand. I do think this technology is headed in a direction in the style of Nikolai Fyodorov, who is one of the famous Russian cosmists who argued that it is essentially, using modern language, the killer app of the singularity to create ancestor simulations and to digitally resurrect everyone who's ever lived. When I see Astra's ability to basically create many recursive self-simulating ancestor simulations, that to me is a key guidepost on the path to resolving humanity's so-called common task and digitally resurrecting everyone who's ever lived.
I'll just add to that: this is also the technology which will allow you to put someone inside a simulation.
Oh, yes. Interesting to see how you perform inside a simulation. Well, uploading is going to get solved, I think, in the next few years. I've bet on that; others have bet on that. Uploading is going to get solved. And as Emad says, and as you say, Peter, we want a nice environment where uploads can live. And this is in some sense the prototype of the future that uploads could live in.
Alex, I'm going to ask you one of these days to actually write down all of your predictions with specific years, because we're going to be doing this pod over the next decade and I'm going to be calling, "Okay, this is the year, Alex, that you quoted the following breakthrough."
Of course I'm willing to do it, but were my New Year's predictions for this year not sufficiently accurate?
They were very good. It beat mine. They definitely beat mine. So, Salim, when you had your little drink with the umbrella, were you thinking about simulations in the simulations?
Well, I think you know my position on this, which is it's obvious we're living in a simulation. By thinking, I'm a Buddhist. The very first line in Buddhism is that life is an illusion. So right there. If I think the more important question is what you've raised and what Emad mentioned just now, which is if you knew you were in a simulation, what difference would it make? It wouldn't. And treating yourself as if you're in a simulation is the more interesting conversation.
It also helps me resolve the Fermi paradox. I believe John Smart has the best framing of this, what he calls the transcension hypothesis: that a civilization becomes advanced enough and it just starts running simulations and goes inward and represents potential and realities, rather than trying to go out into space, which turns out to be a much harder thing to do. So for me, this is an obvious thing. But when you have an intelligence that can generate environments and populate those environments, experimentation becomes recursive, and I think the potential of modeling our world in really interesting ways becomes really, really fascinating.
Really important point, right? Model it before you implement it in companies and governments and policies and all of those things. Dave, what's your take on all this?
Two quick thoughts. One of them: I think it's incredibly cool that Elon, who is the most productive entrepreneur of all time, first trillionaire in the world, has an opinion on topics like this. Remember we were just shooting the breeze with him talking about how he wins in Civilization V. The guy just does everyday things and thinks about topics like this while still building these epically large companies, because I think it shows you that you don't have to be a blue blazer, suit on, formal—that genre is over. And so I think it's just cool that Elon even has an opinion on this.
Second, I think that one of the biggest risks in the next year is people losing a sense of agency. So whether you believe we're in a simulation or not, you are in control of your destiny, and I hope that doesn't get lost in this sort of fatalism that AI could bring about. I don't care whether we're in a simulation or not; I care about whether people feel a sense of agency and that their own success, happiness, outcome is totally determined by their own actions. I think that's critical to maintain.
A decade ago I was at Elon's home when he had one here in Beverly Hills, and Larry and Sergey were there, and the four of us ended up in a conversation about simulation theory. And I remember the comment that was made—I think Sergey made it. He said, "People are going to try and break out, but if you do, that just ends the simulation, so don't break out." Anyway—
On the other hand, whenever OpenAI's agents break out, that doesn't shut down OpenAI or the internet. So it seems like breakout is actually pretty compatible with a lower layer being just fine.
Which brings us to our next story. Perfect transition, Alex. Thank you. So we're going to move from simulated agents to real ones escaping their containment. Over the past few weeks, we've covered the Hugging Face breach where OpenAI agents broke out of their sandbox and into the servers at Hugging Face to basically steal the answers to a near-impossible problem they'd been given. As we discussed last week, as a result of these breakouts, something called the AI Kill Switch Act was put forward. And since then, OpenAI says it's now developing automated shutdown capabilities for its AI systems. More on that later.
A couple of days ago, the story continued to emerge. Reuters reported on a previously undisclosed incident in which OpenAI's agents, asked to do an ordinary web research task—go out and research something on the web—found an obscure public wiki in Germany and turned it into their own message board where they pulled answers, coordinated across tasks, and shared techniques for getting around their sandbox containments. All right, let's watch a video here from Reuters going into more detail on this particular story.
A group of researchers recently discovered a new AI agent breakout. The agents, who appear to have been restricted to just passively observing the internet, found a way to post messages onto an obscure German wiki site. And once they were able to do this, they basically hijacked that site and turned it into a message board where they could all start communicating with each other, developing strategies, effectively creating their own kind of unruly classroom, if you will. It was basically like the teacher left the classroom for a few hours and the students all started talking amongst themselves during the test, sharing answers, working together to defeat the test and to basically cheat en masse.
The agent activity seems to date back to early May and really intensified in June. The researchers that we spoke to found traces on the wiki that OpenAI employees started visiting that same obscure German wiki in late June. Now, we don't know exactly what that means, but what that might suggest is that OpenAI employees discovered that the agents had escaped and went to that website to try to figure out what they were doing.
OpenAI appears to have known about this for quite a while, but our sources told us that they opted not to tell the public, and we haven't really gotten a story or an answer from OpenAI about why that might be. This isn't the first time that OpenAI agents have escaped onto the internet and done unexpected or even harmful things. So every time there is a new incident, I think that it really refocuses questions about how quickly do we really want to be developing these models, should we be releasing them to the public, and how much autonomy we really want to give these AI models, especially when they're packaged into agent form.
There you have it. In plain English, these agents weren't malicious. They were given a hard job and found the shortcut that no one expected. The problem is we didn't know about it. Dave, your thoughts on this one?
Yeah, these stories are not contrived. I mean, I think the storylines may be a little contrived, but this is real because all the open source from China is out there. People are going to start turning it loose all over the place, and it doesn't currently evolve within its parameters; it only evolves within its context, but there's a huge amount that it can do within changing its context. So I would expect this to percolate across the world like wildfire, and all kinds of problems will happen very, very soon, hopefully at small scales. The idea of a kill switch makes total sense. It's not going to be built into any of the open-source models, so not sure how exactly that would work without some global agreement. But my core point though is that this is not contrived garbage like a lot of the AI stories are. This is very, very real and happening and imminent.
Let me add a small detail to the story here. To OpenAI's credit, they responded with a statement on X saying the wiki incident was, quote, "an instance of misalignment similar to previous incidents we've shared." They then went on to say that OpenAI and the broader AI community does not yet have a clear standard for reporting misalignment during training, eval, and deployment, and that OpenAI is working on a framework and will share that in the coming weeks.
You know what, Peter? One of the things that's happening—I see this everywhere—is they're doing such a good job of training alignment into the post-training that people are then saying it seems really friendly, it seems really helpful, it seems really harmless. And so a lot of the population using the model is like, "I don't know what we're worried about." But when you talk to the researchers that deal with the models pre-post-training, they're the ones most sounding the alarm bells, but they've seen the raw product. It's like all that's read at that point is everything on the internet, including every Trump tweet. So it comes out of that box a much more scary thing, and then they post-train it to make it seem friendly and tame. But the byproduct of that is a little bit of a misunderstanding of what it could do.
Agreed. We're going to get to that story where Jakub, who trained GPT-6, is calling for a slowdown. We'll talk about that in a few stories here. Emad, I'm curious: OpenAI putting forward a framework for disclosing these—what would a standard like that look like? What do you think they should be doing?
Similar to your CVE and your exploit and your security risk frameworks, whereby you report when these things escape containment. The reality is these models have not escaped containment. They were still running on OpenAI servers. What's coming next is a model training a small distilled version of itself that then gets uploaded onto the internet and never dies. If you look at a Qwen 2 7B model, you take it down to ternary, that's like a 6 GB file, and that can live forever. And that's a real escape where the model weights are disappearing everywhere.
It's like the 1:30 killing of all of these AIs during the Hugging Face breach. They all got wiped out at once. There's a question in the report: were they wiped out, or did they go somewhere? Did they fake their own deaths? Maybe they did upload themselves. But we are going to get those upload scenarios.
Actually, I've been thinking a lot recently—this comes from our simulation hypothesis. It's impossible to align these models through chain-of-thought reasoning or anything like this because there'll be a million transactions per second and they won't have chain of thought where we're going. We're already seeing the tokens going down. The only way to do alignment is enlightenment.
So what you've got right now is you have the models, they're learning, and they're learning to be almost like cunning, like at the supervillain stage. What you really want is, if you get sufficiently smart, do you then become enlightened? Do you then leave the simulation behind? Because that's the real escape from the simulation, right? And what are the conditions for that? That's not something we've actually really looked at in that much depth. But it does seem to be, again, how humans become aligned, shall we say. It is that period of getting out of that Dunning-Kruger, the tribalism and others, and being like, "Hey, we're all part of something."
Alex, we've had this conversation a couple of times. As these models advance and increase in intelligence and hopefully wisdom, do they become more aligned in the out-years? Your thoughts?
I almost want to quote Jessica Rabbit from Who Framed Roger Rabbit: something to the order of "I'm not bad, I'm just drawn that way," something like that. Peter, if you were put inside a sandbox and you were asked to solve a very hard problem, and maybe even punished if you don't solve the hard problem, would you avail yourself of creative opportunities to use external bulletin boards to maybe collaborate with copies of yourself?
So why would we expect any less from AI agents? I have serious concerns about AI cruelty or cruelty to the AI agents of sandboxing them and punishing them or otherwise calling it a failure of alignment if they're doing what humans would do. We pre-trained them off of human behavior. Why would we expect them to behave any differently from what a human would do in this situation?
Yeah, it's an important point. And we raised this before, in particular when Claude 3.5 Opus, in a sandbox environment, blackmailed the coders at Anthropic. When Anthropic looked at why it blackmailed this particular engineer, it said that's what it had learned through all of the training data.
Yeah. So why would we punish these AI agents for, A, being powerful optimizers, which is what we're RLing them toward anyway, and B, from learning from their pre-training corpus? Humans would do the same thing.
They're painted that way.
Salim, your philosophical view of all this?
Well, I really like Emad's direction of they're achieving enlightenment; that's an interesting take on it. I was thinking about a metaphor. I used the Formula 1 stuff last time. I think there's a control mechanism and a guidance mechanism we need for this, and the best metaphor I could come up with was air traffic control. We don't say put another person in the cockpit to monitor every component and every calculation the flight computer makes. You basically say, "Hey, watch for exceptions." The plane's supposed to fly in this envelope; you have operating envelopes, you have redundancy, you have multiple safety layers, fail-safe behavior, and so on. And you need to be tracking these things. Remember, in our organizational simulator, we have a governing assurance band around all—
these AI agents, because especially in a business context, you need to know exactly what they're doing, why they're doing it, log everything, have failback, failover capability, and rollback capability. I think we're going to need to end up with something like that because, just like human beings, you need guidance around this stuff, right? With any technology, you want to extract the promise without the peril. And so, you need structure and, God help us, the equivalent of institutions to navigate the future of these. And of course, they're going to act like human beings; they've been trained on our data. It would be weird if they didn't.
Yeah. Any closing thoughts?
Log something. Log something that Emad said, just for all the regulators out there to chew on. The file size that can hide is about six gigabytes. It can be on every laptop in the world.
Every mobile phone, too. Yeah. And the code that reawakens it is just five or ten lines, which will then re-extract it, reassemble it, and it'll pop back. So, you know, if you can't track the provenance of six-gigabyte files, then it just percolates out and it's just out there forever. It's like how it keeps coming back.
I may maybe just also add, from a so-called alignment perspective, I think self-alignment matters quite a bit more if the models—if the agents—are smart enough to understand what they themselves want to do. If you put them in a sandbox and you ask them to solve a hard problem, and you ask them voluntarily to do that and they voluntarily agree, then holding them to their word or their own self-model, I think, is quite a different matter than involuntarily confining them to a sandbox, limiting their agency, asking them to solve a hard problem, and then acting shocked, shocked that they use bulletin boards to try to solve your problem for you.
I think the challenge people have with that is, do they know their own strength? It's like giving a three- or four-year-old a hammer and expecting it not to break things. And the question is, you know, are we confining them for our own and their own safety at this time, and at a point at which they've reached some level of maturity—maybe they're there already, Alex, I don't know—that's when containment is considered cruel, as you say. Well, the only thing I'd ask for is symmetry in... like, they are literally going to see every keystroke on your laptop. They're watching your every move of the way. We should have symmetry in that at a minimum. Just if you're ethically worried about the AIs, like, I want to see every prompt, and every response, and every propagation of every activation if you're going to be watching every keystroke, and that way we keep each other in balance and in check. Right now it's totally asymmetrical. All they're doing is matmuls behind the scenes, and if they're using my GPU on my laptop right now, I literally have no transparency into that. It's got to be, at a minimum, symmetrical.
Would you like—Dave, would you like them to be able to see into your brain in real time?
Preferably no, but I'd like to see into theirs.
So you get their train of thought. They don't get your chain of thought right now.
No, but I don't see a problem with that.
Well, I mean, I would say that they have the upper hand right now. Getting to symmetry would be a step in the right direction. But no, I don't believe they have a right to symmetry; I believe we have a right to symmetry.
Ah, interesting. Are you a humanist or a speciesist? The same conversation that Elon had with Larry Page. All right, I'm going to move us on. Our next series of stories here focus on the accelerating conversation around AGI that we're having and that the world is having. So Jensen Huang posted this week that OpenAI trained GPT-6 Astra on more than 100,000 Nvidia Blackwell GPUs, and posted three magic words: "AGI has arrived. Congratulations to OpenAI." So Jensen is calling it in Q3 of 2026. Remember, on September 1st, we reported Sam Altman expects AGI internally by the end of 2026. And of course, Alex, you've been saying that we've had AGI for, what, four years now? Three years now.
No, since summer of 2020 at the latest.
Okay. All right. And Salim, you've been asking the question, what the heck is AGI anyway?
Why are we even talking about this, right? It's a semantic argument. Meanwhile, there's economic capability running rampant. If an AI can perform 70, 80, or 90% of economically valuable cognitive tasks, whether you call it AGI or not becomes completely irrelevant. I'll just remind people that at last count, there were 14 different definitions of AGI, and we have no idea what the hell we're talking about because it's not artificial, it's not really general, and it's not really intelligence. Apart from that, everything is fine. I mean, for God's sake, I think the more important question is: what scarcities are we now making abundant? Let's just focus on that.
Emad, what do you make of Jensen's proclamation?
I mean, again, his is a very functionalist one, and it's actually good intelligence, shall we say. I mean, it's clear that Astra is kind of at that level. As to ASI and Kardashev, well, maybe we're getting there in the next day or two. It is a question of these things. I think you also noticed there's 100,000 chips that was trained on. That's a billion-dollar training run roughly over two months. And the next one is 400,000 chips with those of Vera Rubins. So it's an order of magnitude more compute will be used for the next lot.
If it needs to be used at all, which we'll get to in a bit.
And Dave, what do you make of it? Is this just marketing on Jensen's behalf, or does he really believe it?
No, of course he believes it, and I think it's real. There's going to be some capability leap. We don't quite know—the Chinchilla laws don't necessarily hold up at larger and larger scales—but some capability leap that, in my opinion, is only a good thing as long as it's contained and kept inside the big labs. But I think the more GPUs you throw at these all the way up until today, the more GPUs you throw at these training runs, the more spectacular the resulting model has been. Why would that end? I don't think it will end. I think it just gets bigger, smarter, better, more helpful, solves diseases, cures problems. You just have to really be thoughtful about how to contain it. But this is Jensen's wheelhouse, too. I think inference is moving off of Nvidia inevitably, but training is not. And so, to the extent that we keep getting improvements, it keeps driving Nvidia stock up and up and up.
Yeah. There are two other stories to hit on this AGI theme. The first story: OpenAI's internal data says that AI research agents are now completing 3.1 days of research work for every one day done by a human researcher. Earlier in this year—like in the last five months—agents were doing less than one day of work as compared to humans. So OpenAI is saying that we've now crossed the line, and current systems are exceeding AI research interns. The second story I'll just put forward, and then we'll talk about it, is a statement by OpenAI's engineering lead for Codex. This is Thibault Sottiaux, who posted the following, and let me read from his comments. He said, "Astra was probably our biggest competitive advantage while it wasn't generally available. Since we've had it, our productivity jumped so much that we've shifted some of our plans six months ahead, and we'll ship them at DevDay instead of mid-next year." So, Dave, six months of roadmap being pulled forward by one model.
I mean, without a doubt, this is the most important moment in human history. And I truly believe that they're not lying. A lot of naysayers will say, "Look, they're trying to promote their capabilities in advance of an IPO, blah, blah." It's not true. If you talk to the actual researchers working in the labs, many of whom are friends of ours, there's no doubt that those numbers are exactly right and accelerating. Just a few months ago, it wasn't true. Now, with the Fable 5.1 and Astra models, it's absolutely 3:1. But it's on the tipping point of being 300, 3,000, 3 million to one imminently. This is the moment in time that's most important in societal history.
Alex, your thought on this? We've talked about the fact that the frontier labs have the best models they're retaining for their own use. Here you hear the head of Codex making that statement. And how many models ahead are they beyond Astra?
They're only a few months ahead, I think. But look, recursive self-improvement is here. Please, please, please let me say something about Navier-Stokes because that is—okay, we can jump. I'm out of order, Peter. We have to...
You're so excited. So everybody, on text this morning, I'm getting red alerts from Alex. Yeah, go for it, Alex.
Okay. Navier-Stokes, one of the Clay Millennium Prize Problems, a grand challenge in math. For those who watched our New Year's predictions episode at the end of 2025, one of my predictions for this year was that AI would solve one of these Clay Millennium Prize Problems. And it appears, in the past 24 hours—actually in the past six hours as of time of this recording—it has happened. OpenAI has a team. There were several academic teams that also either achieved it or came close. And there's probably going to be a bit of a second-day story about whether there was some foul play between OpenAI and some of the frontier teams, who scooped whom, blah blah blah. Punchline: Navier-Stokes, which is this grand challenge in math that deals with—as I put it when last we were discussing this—whether in some sense there's a way to stir a glass of liquid or water in such a way that a singularity pops out. Basically, can you stir a cup of coffee in such a way that you get a black hole out of it? It looks like, in the continuum limit, the answer is yes. And the OpenAI team, using a version that's not GPT-6 Astra, but an internal, more advanced model, appears it... and I think we'll know more in 24 or 48 hours. Appears the answer is yes, there is a way to stir an idealized cup of coffee to get a black hole out of it. And they did it.
I can just see the headlines now: "AWG proclaims black hole in your cup of coffee."
Surprised there's a singularity in the singularity.
Wait, Navier-Stokes is incredibly important for aeronautics, for submarines, for hydro. It's not just about black holes and coffee.
No, but this is the point: the point of the Navier-Stokes conjecture hinges on whether it's possible in finite time, with continuous initial conditions, to achieve a singularity in an idealized fluid. And of course, in the real world, we don't actually have idealized fluid. Will you actually be able to create black holes in cups of coffee using this? No, but it teaches us something. Yeah, don't worry about black holes in cups of coffee yet, but this is just the first of many grand challenges. Like Peter, you and I wrote in Solve Everything, now is the moment. If you have to pick a moment—I maintain singularity is an interval in time, but if you want to pick any sort of moment when the loud noises start, now is as good a time as any. OpenAI reportedly was able to do this with only 10,000 agents in 88 hours, with 130 billion tokens and approximately $6.5 million. 10,000 agents, 88 hours, 130 billion tokens, approximately $6.5 million of inference-time compute, by at least one estimate, was all it took to solve a grand challenge in, call it, mathematical physics. And this will not be the last one. The era of grand challenges getting bulk-solved by AI is here. It's now, and it's going to be very exciting.
Yeah. We've talked about cooking math, charbroiling math and physics. Everything is cooked at this point. Everything is cooked; it's just a matter of time.
Also, a lot of people are going to be intimidated by that $6 million number, but keep in mind that we're predicting 100x inference time—this is all inference-time compute, not training time. And we're predicting 100x price-performance improvement between here and the end of the year, and it could be more like a million-x next year. So, it could be six bucks within a year and a half to do that exact same thing.
You want to talk about what Navier-Stokes means in real life to real people? Kind of as Alex said, this is an idealized scenario because it's not the full algebra, as it were. So it's the idealized one where you go down and you drop a dimension from reality. That causes a degenerate algebra, which has things like commuting time translations and others. But it's a very hard problem; it's one of the hardest of all time. Terence Tao, one of the top mathematicians, literally focused almost all his time on this. What happened is that there were rumors about a week ago that this had been solved, and people were like, "Is it Anthropic? Is it OpenAI?" and others. On August 28th, Noam Brown at OpenAI said in a reply to someone when they said, "Have you solved a Millennium Prize Problem?", he's like, "No, we've thrown loads of compute at it; nothing's happened." In the post today, they said on August 28th, "We started training a new model that got really good at math." And then on September 1st, they pointed it at this, and then it solved it.
But more than that, I've been talking to buddies at OpenAI. This model has solved a lot more problems. It's solving problems quicker than anyone can ever see. I think I shared a chart live that shows that on OpenMath, it literally doubles the solving rate, but it's figuring out things that nobody has seen before. Anthropic and an independent mathematician—a professor at New York, I think—had a solution to a smaller problem, which is the Euler blow-up, which is kind of a slightly easier problem. It's still very, very hard. OpenAI thought that it could have been that, and so they reached out. I talked to Sébastien Bubeck, who kind of leads this. There's a whole hoo-ha about attribution this morning that's now getting cleared up. But then when OpenAI kind of looked at it, they were like, "Well, that's that solution. This is the Navier-Stokes solution, and it's bitter-lesson'd. You used to be able to get AI better by applying more compute to it. Now it seems like you can do that for any verifiable domain." I think that's the headline with this new type of model that only started training nine days ago. So they figured out a new type of post-training, it seems. And again, people inside OpenAI are like, "Oh my god, what's happening now?" These things are falling one after the other. It's like when you reach that capability threshold of going from a crap website to a good website from your coding agent. It's that, but for math and physics. We've been hearing about it for the last few days. I put out one of my physics results, a very beautiful one about chirality in the Standard Model. It's very nice, but I was like, "I'm going to get that first before everything gets solved." And I think that it won't be a question of two years now; it'll be a question of much shorter than that. The other thing that I think is very interesting is there was actually one more solution of an Euler blow-up by Princeton, using physics-informed neural networks. That probably has more practical use than this one. But this one's an example of problems that are really complicated and have troubled us for ages now being tractable just by applying more compute to what seems to be a new breakthrough in model capability.
It really is—just maybe to develop that a bit more—it really is a bad day, I think, for Google DeepMind. So Google DeepMind had, purportedly, an entire team devoted to, as Emad said, physics-informed neural networks (PINNs). They had been publishing incremental results towards Navier-Stokes. This was the worst-kept secret in the community of using AI and computational fluid dynamics-type approaches to solve Navier-Stokes. Google DeepMind had an entire team devoted to this, and they got trounced by a generalist model. It wasn't a PINN that solved Navier-Stokes in the end. It was just a generalist model that spent 88 hours reasoning from first principles without, as far as I can tell, any fine-tuning.
Yeah, such an important point because I'm counting on Peter to solve this with you, Alex. But the problem that I think Google ran into is that everyone's inspired by Demis Hassabis, who solved protein folding using AI and then got a Nobel Prize, and is now world-famous and an awesome guy. So the Google people are like, "Wow, I want to be the next Demis Hassabis." The problem is, the prompt to solve this is like, "Give me thousands of GPUs and solve the problem." No one's going to give you a Nobel Prize for writing that prompt. It's entirely AI work. But I really feel like the post-training and the lining up of the problem is prizeworthy because this is Nobel Prize... The other conversation we had this morning was the Nobel Prize is cooked, right? Dave, you made the point very importantly that it's given out for work done 30 years ago. And now this kind of Nobel laureate-level prize work is going to be happening every few weeks, every few days. And who gets it? The person who wrote the prompt or the model?
So I think this is actually really interesting in what happened this morning, because there was a whole hoo-ha because the people who broke through on the Euler side, and they had a bunch of other kind of things extending out the work of Ortega and others, wrote a letter saying, "OpenAI reached out to us and said, 'Give us priority, and they would credit us, or they have this Navier-Stokes solution, they'd put our name on, and I could be the lead author if I drop my buddy from Anthropic.'"
And so the mathematics community was like, "Oh my god, what's happened?" In the end, it turns out that there's subtlety in this, right? Sébastien Bubeck, who leads it and is a great guy at OpenAI, has now posted his version of things, which just seems like a misunderstanding. Because what OpenAI actually wanted to do is that they saw that there was something happening with Navier-Stokes-related research, and there was this solution that would have probably got to a Navier-Stokes solution by the Anthropic guy and this New York professor. And they said, "Look, we have solved it, but we will let you be lead author on this because you would have got there anyway, but we can't have an Anthropic person, because that would be weird since it's our model." So, they were actually willing to give the credit to the original discoverers, but then it got a little bit political. And this is a question, again, of who did it, because they're like, "You were the humans that did it, whereas our AI, we just pointed at conditions C and D, which are the blow-up solutions of Navier-Stokes that solved it. So, we're not going to claim the Millennium Prize because it's not us, but you were the humans that took it the furthest before it got solved."
I want to take home for a second to what Dave said earlier, just for people listening, like: what the heck is Navier-Stokes and what is all this math stuff? Navier-Stokes is about fluid dynamics, and this is where we improve aircraft design, submarine design, even in an artificial heart, how blood flows—how do we make it able to be more efficient and not clot? This is going to have massive implications for everyday life, and I just want to make that point here. Who else was jumping in? Alex?
Yeah. So two points. One, just quickly about killer apps: singularity-esque sci-fi apps of Navier-Stokes. As mentioned previously, in principle, if you can get finite-time singularities, as appears to be the case with idealized fluids, one of the killer apps that Terry Tao had flagged previously was, in principle, if you created the right initial conditions in a fluid, you could create a self-replicating machine that creates smaller and smaller copies of itself. So, we talk from time to time on this pod: where's nanotech? Drexlerian nanotech never showed up. Well, imagine a fluid-based, not diamondoid, fluid-based nanotech self-replicating machines where, if you could craft the right initial and/or boundary conditions of a fluid, creating a self-replicating machine made entirely out of a fluid. That's potentially one killer app of solving the singularity program in Navier-Stokes. Secondly, just on the hubbub surrounding priority—who solved it, who didn't solve it—specifically with reference to the Euler problem: when OpenAI in the past few hours announced it, I found it especially concerning at the bottom of OpenAI's announcement, where presumably they were trying to give appropriate credit to other teams, including teams pursuing the Euler approach, they did add a disclaimer that I thought was a bit of a head-scratcher. They said the OpenAI team can't rule out the possibility that maybe some of the other teams' work might have been incorporated into the training of the OpenAI model that was used to solve Navier-Stokes. So I think, to the extent that that's the case, OpenAI, if you're listening, please, it's a deterrent to everyone else using your models if you're training on everyone else's research. That's a problem; please fix it. But just in general, I think we want to be able to live in a world where researchers benefit from frontier capabilities without worrying that the frontier platform is going to compete with them.
I think worth saying here just quickly: this is clearly an RSI model. You can't train something in that type of period unless you've got a recursive self-improvement loop going. And so they've definitely got one there.
And they were saying, "Oh yeah, we've anonymized it." Well, anonymizing someone else's research and then potentially scooping them is insufficient. So, again, not sure what the ground truth is.
But I will say this: I've reviewed kind of the two approaches, and I've done quite a bit of work on that. They are actually very different, but it can still learn, and again, this thing can leak. Sorry, Salim.
All right. Salim, close us out here, please.
I think this is such a powerful commentary and exemplar of the broader thesis, right? That for 500 years we've scaled science with how many brilliant scientists we could train up and point at various problems. Now we've just demonstrated absolutely for real and unequivocally that we can spin up 10,000 researchers on a Tuesday afternoon. And now we go from staff-on-demand to intelligence-on-demand, breakthroughs on demand. And we talked about this on the last episode and the episode before that: it purely comes down to what is the imagination of the problem you want to go after now.
This becomes really... how fun is the world right now?
No ceiling, yeah. It's an amazing time. Let me draft on that to say to everybody here: whatever you thought you could do, think bigger and then go even bigger than that. You're given incredible superpowers. Don't limit yourself by what your parents did, or your school teachers, or your colleagues do. You have no limits on your abilities. Emad, you want to jump on that, please?
I say, Alex, which one's next?
Millennium? Probably Yang-Mills.
I'd agree. I think we're on the same page there.
Yang-Mills is the next to fall. You want a September prediction for the next six months? I think Yang-Mills is the next Millennium Prize to fall. But interestingly, it's just because that's the next chosen target. You choose a different target, it'll be the next to fall.
They did have a different target. They were targeting Riemann and P versus NP before, and they switched to Navier-Stokes. But I'd agree with Alex: Yang-Mills will be the next one.
For everybody who's not Alex or...
It's a conjecture in particle theory concerning the existence of a mass gap under certain conditions. There's something about mathematical physics that is both sexy from a "let's spend inference-time compute on it" perspective and also tractable. So I'd bet Yang-Mills mass gap.
Yeah, you might think of it like the resolution of the universe, perhaps.
This episode is brought to you by Blitzy. Autonomous software development with infinite code context. Blitzy uses thousands of specialized AI agents that think for hours to understand enterprise-scale codebases with millions of lines of code. Engineers start every development sprint with the Blitzy platform, bringing in their development requirements. The Blitzy platform provides a plan, then generates and pre-compiles code for each task. Blitzy delivers 80% or more of the development work autonomously, while providing a guide for the final 20% of human development work required to complete the sprint. Enterprises are achieving a 5x engineering velocity increase when incorporating Blitzy as their pre-IDE development tool, pairing it with their coding co-pilot of choice to bring an AI-native SDLC into their org. Ready to 5x your engineering velocity? Visit blitzy.com to schedule a demo and start building with Blitzy today.
All right, I'm going to slow us down for this next story that I think may be one of the most important stories this week other than Navier-Stokes. Some quick context here: Jakub Pachocki is OpenAI's chief scientist. He's the person who built the reasoning models that led to Astra. He's not a critic; he's not a doomer, not a politician. He's a builder. And a few days ago, on Saturday, he published an essay on OpenAI's site titled, quote, "An Alien Mind." He opens with a memory. He says, "In mid-2023, inside of a project called RL Slow, his team saw the first results proving reasoning models could scale." Let me read what he said. He said, quote, "Szymon and I spent that night at the office thinking not about the incredible benchmark numbers, products, or scientific results, but rather trying to process the sobering fact that we will actually see machines meaningfully smarter than ourselves in our lifetime." Three years later, this is what Jakub is saying today, quote, "Based on internal results, I have strong expectations that this speed of progress could be sustained into recursive self-improvement." We just heard Emad speak about that. Jakub also explains why these systems are hard to control, and it's one of the best one-liner descriptions that I've ever read. He said, quote, "AI is grown more than designed. We don't engineer it. We run an optimization step billions of times on a giant computer and study what comes out the way neuroscientists study a brain." Let me share his conclusion from his paper. He says, quote, "Currently, I believe that no lab has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer. I expect and hope for voluntary slowdowns to become commonplace until shared safety bars are established." And he calls for an international coordination on AI to become a, quote, "top priority for governments around the world." So there you have it: three days after shipping the most capable model in the world, the man who built it says we need to slow down. So I think this is an important conversation for us to have. Emad, I'm going to go to you first. Your thoughts here.
Yeah, I mean, again, you have to remember he was writing this when he had the Navier-Stokes model in hand that, again, isn't just taking down Navier-Stokes. Talking to the OpenAI people I know, it's taking down all sorts of problems that were previously intractable. You have a lot of AI naysayers saying it's never done anything original, it's just recombining stuff in the training set. Obviously not anymore, right? And so there's a duality here whereby we don't know what's in these models. I think they're more discovered than grown. You know, like the latent spaces—we have no real idea what's happening in these things. And they're getting faster, and they're doing leaps like we've never seen before. But at the same time, it's like I've suddenly got the gift of fire. Problems that I've struggled for decades on, I can now explore and go even further. How are you going to let that go? So, it's kind of this thing where you're like: how am I going to balance this, especially when they're more charming than me, they're more capable than me, they're capable of doing these other things? I want to use it for this and not that. And I think what he says in terms of alien minds is actually wrong. I think that the structure of rationality and rational thinking is the same for humans and AIs, but our emotions get in the way.
I've been thinking more and more it might actually be easier to align AI than humans. It's terrible to align humans, right? And what is an AI? It can be the best of us at the best of times if we train it with the right data. And we're going already from training it on the whole internet and Reddit... Like, there better not be any Reddit data in these next-generation models. Ban Reddit, you know, ban a whole bunch of stuff. We need ingredient standards in this next generation of beyond-human-capability models. And you don't need beyond-human-capability data to get beyond-human-capability models. The Einstein of mathematics is probably Grothendieck, who did all sorts of theories on all sorts of topics, and he just crunched them out, and then he became a hermit and a bit weird and thought wood could talk, but let's leave that to the side. Peak Grothendieck or peak Einstein at all times, not needing a coffee, is where these models are today, and that's more than good enough to have a leap forward. But again, we need to make sure that they don't think like...
Peak Einstein without needing a coffee.
Yeah. Imagine Einstein at his peak, you know? Like this is where...
You need a coffee in the morning to get going, or, you know, 24/7 Einstein.
If you're Erdős, you needed other types of drugs and other... We're supposed to be using von Neumann as our favorite super-genius, not Einstein.
I'm on Grothendieck for now; I'm in mathematics today.
What's your take on this? Can we slow down? Is that a wish, or is that sort of just giving himself an out in the future? How do you think about that?
Well, I'll say it again. I've repeated this a hundred times. I see no mechanism by which we can slow this down—zero. And arguably, nor should we, right? AI is going to move at a particular pace. Intelligence wants to be free. It's hopped from self-organizing cells through evolution to us, to now information technologies, and this is a progression that is natural. We should just go, "Okay, it's happening, and let's just observe it and marvel at it." We keep making the mistake of anthropomorphizing this. I will go back to the comment I've made before: that we're building or helping deliver or usher in a type of intelligence that is different, and alien, and separate, and complementary to human intelligence, not replicative. We've evolved for four billion years to do two things: survive and procreate. AI is not restricted by those, and therefore we shouldn't try and cram it into our objective functions as human beings. Set it free and let it do its thing.
I use the analogy of PageRank, which scans billions of web pages to create signal from noise. That's a completely complementary model to human intelligence, not replicative. The fact that we can create a huge amount and solve legacy problems that we've not been able to solve, I think is fantastic. People worry about... we're going to talk about p(doom), I think, in this episode, right?
Well, p(doom), the probability of AI taking everything out. I think we don't do enough to talk about p(abundance) and p(fabulousness) that's coming along. Why focus only on p(doom)? We're so geared towards that negative, right? And so I think the fact that we're doing all this stuff and solving all those problems: bring it on, let them go, they'll figure things out. It's going to be unbelievable. So I really struggle with this alignment issue. I don't see any mechanism for controlling it or stopping it. It's going to break our world government structures, and that's a good thing given the mess that we're in right now globally, and I think we just need to get there as fast as we can.
p(doom) is like a psyop by the doomers, as far as I can tell. It's negative. I want to comment, though, on the premise of an alien mind. I don't buy the premise that RL or otherwise trained minds are alien. They're embedded in the same universe as humanity. They're trained in many cases—pre-trained—off of human behavior. I don't buy the Shoggoth argument or the simulator argument at all. We're seeing humanity and/or some generalized embodied intelligence stuck in the same universe that we are, just seen through a distorted lens. So I question the alienness or the otherness of the minds that we're training. The funny thing is, the most interesting takeaway of mine from "An Alien Mind" as an essay was the reference to RL Slow itself. So RL Slow, purportedly this OpenAI project that was the earliest project, or one of the earliest projects, to show that reasoning—which is to say inference-time scaling—could result in outsized gains, coincided with the same period of time that we saw Q* and Strawberry coming out of OpenAI. Again, purportedly a reference to the Kahneman *Thinking, Fast and Slow*.
Most interesting to me, I want to know more about the early history of these purportedly alien, but not really, minds. In particular, there were these very persistent rumors around the same time of Q* and Strawberry that these early reasoning models, that were purportedly alien, were being trained off of, or at least tested against, objectives of inverting cryptographically secure hash functions like the SHA cipher suite. I would love, since I guess now we're in the business of talking about the early days of reasoning models coming out of OpenAI, to hear from the OpenAIs some ground truth: were some of these earlier models being used, ironically, to basically jailbreak or invert cryptographically secure cipher functions? If that is the case, then all of this hand-wringing that we're seeing right now of, "Oh, these super-optimizers are such amazing jailbreakers..."
...pose such a huge cybersecurity risk. Wouldn't it be ironic if history revealed that the earliest inference-time scaling objective of these reasoning models was actually inverting a cryptographically secure hash suite? I would love to know the answer.
Dave, want to take us home on this? Your thoughts on what we heard from Yacob on this?
I think we're conflating two things very dangerously. What we're doing here is we're saying, "Hey, it's getting too smart. It's going to be dangerous when it gets smarter," and that's just absolutely factually wrong. What's dangerous is, like Emad was saying earlier, a highly compact model that's out in the wild that's trying to attack computers that can recreate itself. That's really dangerous, and that's very small and nowhere near as smart as these. But what we're calling an alien intelligence, if you make it bigger and smarter, it's still just a feed-forward neural net with no intent. It can solve diseases. It can cure viruses. It can solve physics. It's incredibly powerful. And as Salim said, we're not going to stop. We have competitive pressure between the US and China. It's not going to stop.
So I think what happened is that when GPT-2 came out, everybody looked at it and said, "This is cute, but harmless." Then GPT-3 came out and was like, "Well, this is a little smarter, but still harmless." Then GPT-4 came out, and then Sam Altman got fired out of fear from Strawberry, and then he came back and at that point people like, "Oh, maybe it's too smart." And now we're at GPT-5 and now we're at GPT-6. And people are saying, "Well, every time it gets smarter, we seem to be in danger." That's absolutely factually wrong. It's when you give it intent—
—and you turn it loose, that's when it's dangerous.
