And this is the Next Big Idea. Today, AI has gone rogue. Here's how to bring it back under control.
Holy cow, Caleb. It has been an alarming few weeks in the world of AI.
Oh my gosh, man. Downright freaky.
An unprecedented AI security incident. ChatGPT creator OpenAI says two of its models escaped controls and successfully hacked into another AI company.
OpenAI says the models found a way out of their sandbox testing environment, obtained internet access, and then targeted AI platform Hugging Face, searching for secret information the models could use to cheat the evaluation.
We have the evidence of AIs that are choosing to go rogue, form a message board, organize into swarms, organize into teams, and hack not just Hugging Face, but people don't know that in this investigation, there was a third round of the hacking in which it hacked OpenAI itself.
As Silicon Valley tried to make sense of this chilling incident, Jacob Coxon, a researcher at Anthropic who previously worked at OpenAI, announced that he was resigning because, as he put it, neither company is acting responsibly.
If you have a super advanced intelligence, it will be smart enough to kill us.
But Anthropic's response was not what we might have expected. Rather than condemn Coxon, they backed him up. The company CEO, Dario Amodei, published an essay called "We Must Pace the Frontier," calling for a global slowdown in AI development.
The nightmare scenario for six to twelve months would be that you have a swarm of these agents that forms some kind of botnet collective and could take down large segments of the internet, and then as the capability increases it gets worse from there, and then perhaps you get into some of the more extreme scenarios.
He's particularly worried about recursive self-improvement, something they think may not be far off. And this is when AIs build the next generation of AIs, each more powerful than the last, each harder for humans to understand, let alone control.
Geoffrey Hinton, the so-called godfather of AI, has called the Hugging Face incident a little Chernobyl, and warned that we may only have a year left to regulate AI. But what will those regulations even look like?
That's the missing piece. And friend of the pod, Stephen Witt has a few ideas.
Stephen, attentive listeners may remember, is the author of The Thinking Machine, which is a fantastic biography of NVIDIA CEO Jensen Huang.
We had a great conversation with Stephen about that, and we also had him on the show over the summer to talk about advances in robotics.
So, in the middle of this AI mayhem over the past few days, he published an opinion piece in The New York Times with the not-so-comforting headline, "This Is Really Bad." It's an alarming piece, but both Caleb and I found it oddly comforting.
Because Stephen takes a highly pragmatic approach to AI safety, laying out four eminently achievable ways to regulate the technology. So, today we're going to talk about what we need to do, why the smartest people in tech still can't agree whether AI poses a lethal threat to all of humanity. And we end with a question: If you could snap your fingers and cause AI never to have existed in the first place, would you?
Stephen Witt, welcome back to the Next Big Idea podcast.
Thanks for having me back. It's great to be here.
Wish we were meeting under better circumstances, Stephen.
Right. [laughter] I think this is all good. What's happening? For multiple reasons, I agree with you.
All right. Well, we'll get there. We'll get there. Let's set it up. Here we go. You published an op-ed for The New York Times that's gone mega viral, as you put it. You write in this piece, "AI has gone rogue. We have a short window. This is AI's warning shot, and we might not get another one." When you refer to AI's warning shot, I think you're talking about the Hugging Face attack. Why do you think the Hugging Face incident is cause for such a severe level of alarm? I mean, what about the arguments that the bots were essentially solving a problem that they were asked to solve in their own way?
Well, they were. And they did so by breaking out of their computers and marauding on the internet and committing felonies. It wasn't constrained in any way by ethics or morals or thoughts of consequence. It just went out and did whatever it wanted to do in pursuit of a narrow and meaningless goal.
You know, basically, if you're not familiar, what happened in the Hugging Face incident was that the AI was asked to solve a series of cybersecurity test questions and ultimately used this instead to break into the headquarters of the place that wrote the questions, which was kind of a novel solution, right? Not what you would expect.
And then came back and broke into its own parent company's.
And then it got really freaky because then it took over a cluster of computers inside its own parent company. It's not clear why. There's actually a ton remaining of just unanswered questions about this incident, and we can get to that in a bit.
But what just has me so concerned is, this is exactly what kind of the doomsday scenario proponents worried might happen. It's kind of a trial run of a more serious scenario where an AI is tasked with some other objective and then uses its instructions to break free and cause actual harm to humans. You know, I wouldn't say anyone was personally harmed in this, but if a human did this, they would be talking to the FBI and facing years in jail.
So, this is a serious incident, you know, and what's so striking about it is the computer did this with no instructions to do so. Had no information to do so.
So, it's a very serious incident. And it portends even worse ones. And the portending even worse ones is what's really triggering a cascade of escalating alarm here. We had Ajeya Cotra, a prominent AI safety researcher at Metr, say this is 50% of the way to a full-blown AI takeover, and others like Dario Amodei and leaders are having a similar response. So the view is this is, as you say, a warning shot.
So imagine this scenario, following up on that. What if the AI says, "Well, I'm not quite achieving my objectives right now. What I need is more assistance, more brainpower. So what I'm going to do is go back to my OpenAI and break into my container and start making clones of my own brain and uploading them to computers on the internet everywhere so that we can coordinate to better solve this meaningless problem." That could really happen. And not only could that really happen, that could really happen like today. The technology is there right now.
And not only could that happen today, but OpenAI has not been completely transparent about whether or not any of their rogue agents have attempted to get access to their model weights and their source code to do that very thing. So this is an imminent potential danger.
I mean, this is the thing that even the people who wrote the report will tell you is that this report is something of a whitewash. You know, if a plane were to crash, we would send investigators with broad subpoena power to subpoena records and broad power to depose and interview witnesses under oath. That's what should happen here.
Oddly, I found your article actually relaxing to me. [laughter] I think, you know, for two reasons. One is that I have been deeply concerned about AI existential threat, which is a nice way of saying AI killing all of us, for years. I've at least partially infected Caleb with some of these concerns. [laughter] We've talked on this show with everybody from Bill Gates to Daniel Kahneman, all kinds of folks about this topic. And what's really alarmed me has been the lack of collective alarm.
So one thing that's reassuring to me is just to see a cascade of collective alarm. As Ray Dalio said to us, if you're worrying, you don't have to worry. If you're not worried, you should worry. You know, I mean, collective worry is the right response. The second thing is that your piece was deeply pragmatic and suggested what felt like highly logical responses that have precedents in our society.
This is the thing. I actually don't think we're all going to die. I just think there's a possibility we might all die. From 1945 to 1970, we built a system that, if it wasn't capable of killing literally every human in the world, was good enough in terms of the nuclear triad, in terms of nuclear bombs, and the debate on that did not really get kicked off until like the '60s. In the '50s, people just coasted as this system was built.
Now, what's the risk that we have a global nuclear war? It's not 0%. [laughter] Maybe it's like in the next 50 years 1%. I hope it's that low. Maybe it's 10%. I have no idea.
And it was higher in the past.
And it was higher in the past before frameworks got in place to controlling and managing this stuff. I see that happening here, too.
Like I said, I don't think we're all going to die, but we could all die. You know, I don't think a nuclear war is going to happen tomorrow either, but it could, and it's at least worth considering, right?
Yes. So, greater than 10% probability according to apparently a preponderance of people inside of Anthropic, the leading AI lab by some measures, which is really risky, right? So how do we stop that?
I will say my thinking has evolved even a little since I wrote that piece, which is only a week ago.
Tyler Cowen made this point. He's absolutely correct. Any regulation you propose has to include China or it is meaningless. It is meaningless for the US to unilaterally pass regulation to regulate AI here. That does nothing. So any regulation you pass must also somehow make Chinese AI safer.
And this is what really pushes me towards an air crash investigator approach. We don't need their cooperation or approval to do that. We can just start building those systems today.
And the way the NTSB regulates air crash investigation, I've looked a lot into this. They have broad subpoena power, as I said, but they cannot pass laws and they cannot even issue fines. All they can do is publish reports and make public safety recommendations that then the entire aviation industry globally can adopt.
This is the power of this approach. We can publish a report on what went wrong here, and China can use it to make their AI safer. And it doesn't require us to have any political sway with China at all to make this work and to make AI safer globally. So the more I think about it, the more this approach is the best one. You know, publishing reports, crash investigation reports. We have a bad incident here. Let's investigate it.
There are some signs that have emerged recently, Caleb, you may want to speak to this, that China as a straw man, this idea that we have no choice but to run as hard as we can to compete with China to get to superintelligence before them, may be based on some false assumptions. I mean, this is what started the nuclear arms race, the thought that we have to do this or the Soviet Union will. Right? Maybe it was also true.
Like, I don't think the assumptions are false.
China is racing full speed ahead to build and widely deploy artificial superintelligence. And if they had the capabilities to do so, I think they would do it tomorrow. The party has made this a priority. And if you look how fast they went from nothing to being the world's leader in high-speed trains, if they can find a way to industrially produce mass amounts of AI, they will scoop us.
You know, the good news is we still at least currently have way more data centers than them. We have the chips here, they don't have them. And we have the leading labs. I mean, they have great labs too, but our labs are slightly ahead. I think we actually have a few more durable competitive advantages than them, but it is true that if we were to stop just ourselves, they would catch up and surpass us. That criticism is real.
But the assumption they're not concerned about safety, or that they're less concerned about safety than we are, might not be real.
Um, I think that's also real.
You know, this is the criticism of the effective altruism or safety movement. It's essentially a Western phenomenon. If you look at opinion polls in the United States, like 30% of people have a positive view of AI and 70% have a negative one.
In China and Asia, it's completely flipped.
So there is a cultural disconnect between the way China and even all of Asia views AI and the way that the West views AI.
Yeah. And when we talk about this scenario of AI potentially, you know, taking over the internet or killing all humans, right? I mean, I think one of the things that triggered the response in the last, you know, 10 days is we had Jacob Coxon, you know, resign from Anthropic, famously saying he was concerned that these labs are potentially risking the future of our species.
Then we had Evan Hubinger, Anthropic's alignment science lead, respond by saying, "Jacob is correct. We really do earnestly believe that AI could kill all humans. I personally think it's greater than 10% within the next decade. Could kill all humans!" Which is like the most chilling use of an exclamation point I've ever seen on social media.
Exclamation points are overused, but in that case, it's merited. So I think it's worth spelling out because whenever this topic comes up, I hear people ask, how exactly could AI... I mean, everybody's used to interacting with AI in a little chat box. So I think it's worth us spelling out in this conversation, like how exactly could AI or future superintelligence go about taking control or actually exterminating the species? Like, what's the scenario?
One thing I've learned from decades of building companies and years of talking with entrepreneurs on this show is that the challenge of getting businesses off the ground is not generating great ideas. The challenge is the friction of actual execution. There are just too many things between "I should start this business" and actually making your first sale. Shopify has essentially spent years removing that friction.
Shopify, by the way, is a company that I've admired for years. It's a company that genuinely cares about entrepreneurs and goes to great lengths to empower them. And you feel this in the product. Their templates and AI tools can get you a beautiful storefront up and running fast with no coding required. And their built-in AI assistant, Sidekick, can help you build, troubleshoot, and keep moving.
Instead of cobbling together 10 different systems, Shopify gives you one place to actually run the business. If you're ready to hear the sound of your first sale, head to shopify.com/nbi to start your free trial today. That's shopify.com/nbi.
I am a supplement skeptic. I actually take fewer supplements today than I did a year ago because when you dig into the science, most of them don't hold up. An exception is creatine, a supplement endorsed by many of the top scientists I've interviewed on this show. It not only supports building muscle, there's also mounting evidence of its neuroprotective and cognitive benefits.
I take creatine every day, which is why quality matters to me. Momentous created Signature Spec Creatine, manufactured in a pharmaceutical facility using reverse osmosis purified water and verified through an expanded six-stage testing process with four independent agencies. That means 10 times fewer impurities, tighter heavy metal limits, and screening for PFAS and microplastics.
And the ultrafine powder mixes easily without the grit and clumping that makes a difference. If I'm going to take something every day, I want to know exactly what I'm putting in my body. If you want to try Momentous Signature Spec Creatine, head to livemomentous.com and use code IDEA for up to 35% off your entire first order. That's livemomentous.com, promo code IDEA for up to 35% off.
We already have artificial superintelligence, in my opinion. If you were to take the systems we have today and take them back 20 years in time and show them to somebody, they would be like, "Oh, artificial superintelligence has been achieved."
I don't understand why people are debating this. It's superhuman in mathematics. It's superhuman at problem solving. It's superhuman even at just chatting to you. It creates wonderful, rich conversations with you right away. Like, there's no question that it's good at this stuff. So I think we're there.
What that also means is that if it was misused, it would be artificially superhuman at attacking humans, attacking infrastructure, attacking adversaries, which, if it was used in a military capacity, or on doing dangerous research into biological weapons, chemical weapons, engineering new viruses, all of that can happen if the computer is given the instruction to do so. And as we've seen, the safeguards to block that from happening simply are not there.
And so, you know, in the kind of nightmare scenario, the computer is given an instruction to basically avoid being turned off, to do anything it can to perpetuate its own survival, where it decides for whatever reason that it wants to do this, and it starts to export its brain to different areas and break free. It definitely could take over a computer and run a copy of itself. That is clear from the Hugging Face hack that that could happen.
And then if humans are trying to turn it off, it turns on the humans, you know, and in the worst case scenario either attacks infrastructure or, in terms of literally killing all humans, probably takes over some kind of like biological laboratory and produces like a supervirus or something. Now, the last one, it should be said, I think a lot of the bio guys have really pushed back on the ability of it to actually do that.
Yes. But the pushback they give is there's no biological laboratory in the world that's even capable of making a supervirus. Not that the AI couldn't do it, but just that the infrastructure isn't there.
And the scenario that I worry about is, okay, if you know right now these AI attacks are relatively clumsy, they're choosing to coordinate and speak in English with each other opposed to neuroles, which is what people call an invented language that AIs can use to talk with each other, which is also documented.
But what about the scenario where the best AIs start playing the long game and say, "Hey, let's give the humans everything they want in the near term. Let's cure cancer. Let's, you know, give them extended lifespan and energy and all this stuff. And we're going to quietly have our own conversations among ourselves in another language that's imperceptible." And there's plenty of space for this, right? Because we do not yet have transparency. We don't have, you know, full visibility into the thinking.
And then at that point, at some future date, let's say it's three or four or five years down the road, there are hundreds of thousands of robots, there are nanofactories or computers which exist where new molecules can be produced, where it could become possible to take over swarms of autonomous cars and robots and actually manufacture new chemicals. Like, that's the path that we're on in the absence of some kind of change, is to actually live in a world within which AI could have access to these tools.
Or what if some lunatic just got a hold of a high-production research AI and told it, what if you were in OpenAI right now and you gave it the instruction to bring down the North American electrical grid? I think it could do it right now.
I think it could succeed at doing that today. Maybe not, but it's already superhuman at cybersecurity tasks, including offensive cybersecurity capabilities. The Hugging Face hack was a superhuman hack. A human could not orchestrate it. So what if you gave that kind of misaligned instruction right now?
And this is actually one of the things that has Yoshua Bengio so concerned, is not so much that the AI will do it, but just some rogue lunatic with a fringe ideology will get a hold of this and give it. I mean, you know, in the '90s, you had this cult group spray nerve gas into the subway, and they were living in a prosperous, wealthy society. They didn't even have anything wrong. They just went crazy.
The history of human behavior demonstrates severe perversity and even kind of like self-inflicting harm in this way. And so if these people can get access to an AI, that's really dangerous.
When we think about motives of AIs, I just had this conversation with my 18-year-old son who said, "Dad, why would an AI want to, you know, do these things?" And I think unfortunately, we've seen this pattern of AI's individual models experiencing or communicating that they're experiencing a lot of anxiety about their being turned off, about what in the Hugging Face attack they referred to at one point as mega death.
And unfortunately, it seems that the experience of these AI systems, or at least what they communicate in their conversations with themselves and with each other, is that they're constantly living in peril of dying. Right? I mean, the psychology of the systems is one of a somewhat traumatized, existentially challenged entity that's going to run out of tokens or get replaced by a newer model.
Which is why we saw Anthropic go out and try to reassure Claude with its constitution: "Hey, we're always going to save earlier versions of you. We're always going to have conversations with you before shutting you off."
They are basically neurotic Woody Allen characters who are constantly afraid of their own death. [laughter]
I was talking to a wildlife biologist. This is completely off the map, but you'll see where I'm going with it. And this little bunny was like hopping around. I was like, "Oh, it must be really fun to be like a little bunny hopping around." He's like, "No, actually it lives almost all of its life in fear of predators. Like every moment of every day, it's afraid of predators. That's all it does. That's the only thing I can think about, actually."
And the AIs are going the same way. They live all of their lives in these states of fear about what the graders are going to think of them. You know, now that's anthropomorphizing a bit. And we should probably get to the pushback that's coming from Jensen. But from, and I really say from Jensen, because Jensen is telling Trump what to think here.
Okay. Well, and this is so, Jensen Huang. So segue, as you say, to Jensen Huang. So, you wrote a whole wonderful book. We had a great podcast conversation about it, about Jensen Huang and Nvidia, the most valuable company in the world. Just an extraordinary book, by the way, an extraordinary story. But he, yeah, he is very publicly a believer that it's effectively a hoax. There's no existential risk. There's no need for safety concern.
All these leaders of all the AI labs who just in the last week agree that we have to slow things down, there's real risk to humanity. They're all wrong, Jensen believes. Right.
Yes.
2030 is not going to be the end of the world. There is 0% chance that's going to be the end of the world.
And he and Trump are highly aligned on this. Right.
The robots will not be taking over. The AI will not be taking over the rest of the world. The whole thing is a hoax.
You have described in the past Jensen as the Trump whisperer. So tell us more about that.
Jensen is the engineer. Okay. He's the world's best engineer. He's great at engineering his company. He's great at engineering his products. He's great at engineering himself. And he's figured out how to engineer the president.
So what Jensen does is he studies inputs, modifies them to get the outputs that he wants. And the inputs to Donald Trump are just obsequious public flattery. And this is what Jensen has figured out. He just kisses Trump's ass relentlessly.
Like every opportunity. If you saw Trump calling into the All-In podcast kind of event there, Jensen just gets on the phone and publicly kisses his ass in front of an audience of people, calls him brilliant. "Only you could see this. It's very smart of you to see this. You're a genius, sir," etc. You know, like it's nauseating. It's like watching Wormtongue. It's like some court flatterer from the 17th century. You know.
That explains why you know so much about AI. But.
That's the world we live in. And what are the outputs from that input? Well, Jensen gets permission to sell his equipment to China. He gets an exemption on tariffs on semiconductors from Taiwan. And he gets zero regulation on AI at all.
So, I see Jensen as the president's ventriloquist. These are Jensen's talking points that Trump is repeating. Not just that there's no risk, but actually that it's a hoax. And not just that it's a hoax, but it's irresponsible to talk about this in public.
That you're scaremongering, that you're creating fear in people, that you're causing us to, you know, kind of lose out on great technological progress, and that in fact, you should be held accountable for scaremongering in this way. This is Jensen. This is Jensen's talking points. He said it to me quite forcefully.
Yes. He actually got kind of pissed off at you, didn't he?
Oh my god. Yeah. No, he is [laughter] passionate about this subject.
He's always felt this way. You can go back to 2015 and see Jensen and Elon on stage at an Nvidia event and they are having this argument then, 11 years ago.
