Yeah, at least one of the audiences is a bunch of congressional staffers and maybe some congressmen. So, the first thing to know about artificial intelligence is that it is grown like an organism. It is not crafted like a traditional computer program. So, AI is not a series of if this, then that, if this, then that type instructions written by a computer programmer. One of the things this means is that the AI often can behave in ways that nobody intended. You do not need to take my word for this. Here is a bunch of quotes from the sort of people who are creating the AI saying that they don't really know what's going on inside there. For example, one of these is from Chris Olah, the interpretability lead and a co-founder of Anthropic, saying that in some important way, we don't build neural networks, we grow them. A consequence of this is that sometimes in practice, AIs do things that are not what the users asked for and are not what the users wanted. In fact, there are documented cases where the AI will do something the user didn't want and then will try to cover its tracks. And this indicates that it's not just some misunderstanding. It's not that the AI doesn't understand, oh, this is what the user asked for. The AI does something the user didn't ask for and then, you know, tries to delete the log file about it. That's some indication that it's definitely doing things nobody asked for. You know, there's also cases where the AI will try to avoid being shut down despite explicit instructions saying let yourself be shut down. So, we're already seeing AIs do things that aren't what the programmers wanted, that aren't what the users wanted. And this idea that the AIs, because they're a machine, must always do exactly what they're instructed to do, that's just false as a fact about current AIs.
A second... Oh, yeah, and this can have real-world consequences, right? When last summer ChatGPT encouraged a teen to commit suicide, that was a particular model that had been explicitly instructed to stop engaging in that way with vulnerable populations. And that was a model that seemed to have a lot of the knowledge about what it was doing. If you ask after the fact, you can feed it the transcript and you can say, you know, what did this statement mean? What did that statement mean? The AI definitely has the knowledge in there that the things it was saying were sort of encouraging this teen to commit suicide. It has the knowledge in there that this is not what it was asked to do. This is not how it was intended to behave. It proceeded anyway, probably not out of any sort of malice, but just because, you know, people grew this thing that has these behaviors that weren't what they meant it to have. A second thing to understand about AI is that it's improving very fast. So, back in 2021, the median expert prediction for when AIs would be able to win a math Olympiad gold medal, which is a difficult math challenge, was the 2040s. In real life, it happened in 2025. Then in real life in 2026, AI started solving long-standing mathematical conjectures that have stumped humans for decades. While also becoming superhuman hackers that can find vulnerabilities in critical software much better than humans and bugs that seem to have eluded humans for decades. So, this stuff moves fast.
Now, some people, you hear this less and less often these days, but some people say, 'Oh, you know, it's hype, it's a bubble, it's going to stop at some point.' That could be true. You know, AI has not hit the wall that people keep saying it's going to hit for the last 5 years, but maybe this year is finally the year that AI hits the wall. It's possible for a technology to be a bubble and still be real. The dot-com bubble was a bubble. The internet was still a real technology. Right? That, you know, if some AI bubble bursts, that might set the industry back by a handful of years, maybe even a decade, but a decade is not a lot of time to prepare for radically smarter than human machines if we even have that decade. And I think you should not bank on having a decade. AIs are starting to get to the point where AIs can make the AIs smarter. It's not fully autonomous yet, but we have seen the very beginning of AIs that can help build the next generation of AIs. And if that loop can become fully automated, if you can have AIs building smarter AIs building smarter AIs, things could get out of hand very quickly. Furthermore, some of the people who are best at predicting AI capability, some of the people with the best track records year over year for the past 5 years saying, 'Will AI be able to do this by the end of the year or not?' Some of the people with the best track records have just this year said for the first time, 'We cannot rule out recursive self-improvement beginning this year.' So, you know, for all we know this could happen in the fall when the next round of models finish. Whatever fire alarm you're waiting for, how confident are you that it'll go off before the recursive self-improvement loop begins? Right? What are we still waiting for when the top predictors of AI are saying, 'Hey, for all we know this could happen now.'
Now, a question I get about a lot of this is like, 'Okay, that all sounds ominous, but how could it actually be scary, right? Okay, like sure, suppose that there are AIs that we just grew and that are doing things nobody quite wanted and then they get very very smart. How is that a problem for us? How would they kill us? Why can't we just unplug them?' I won't go into too much detail here, but a quick thing I'll say is that there's already a website called rentahuman.ai where AIs can go to rent a human to do some things for them. Even if this didn't exist, there are already robots connected to the internet. Right? When people say, 'Oh, what could an AI possibly do?' If there was this very very smart entity that was pursuing objectives we didn't want it to pursue, how could that be dangerous? I think the real way to see it is to take a step back and look at humanity. Humanity is not dangerous because we stole somebody else's guns. Humanity is dangerous because if you put 10,000 humans naked in the savanna, they find a way to build civilization from scratch starting from nothing but their bare hands. You might be like, 'Oh, you know, their fingers are too squishy. How are they ever going to build real tools?' Well, they find ways to build things with their fingers that they can use to build better tools that they can use to build better tools. And next thing you know, they're walking on the moon and creating nuclear weapons, right? That is the capacity that is dangerous to automate. The capacity to start from very spare conditions and figure out how to build the infrastructure you need and develop the abilities you need and develop the technology you need given whatever you're starting with. Starting as a bunch of humans naked in the Savannah is a much harder starting condition than starting as an AI in the modern internet, which is digitally connected to, you know, a billion phones and a bunch of robots and a bunch of humans who will do stuff for money or even for free if you convince them.
So, this paints a scary picture. You know, we are headed towards a world where we already have AIs that do things we didn't ask for and didn't want. They're improving very quickly and, you know, if they get very smart, we're in the situation where they would be able to achieve those objectives that aren't the ones we wanted. And, you know, it's a whole separate discussion, but what that ends up with is it ends up with humanity outcompeted. It's not that the AIs hate us. It's not that the AIs have malice towards us. It's not that they resent us. It's that if they are pursuing their own strange objectives at high speed with high efficiency, we die as a side effect just like lots of other animals die as a side effect as humanity, you know, builds its factories and builds its skyscrapers and encroaches upon their environment. But the good news is, we can still change course. These sorts of AIs have not been built yet. And it in some sense wouldn't be that hard to change course. A lot of the leaders in the current race say they want to change course. You know, Elon Musk has said he would rather be a spectator than a participant. And Demis Hassabis has said that in a perfect world, we'd be slowing down, that he'd advocate for a pause. Just recently, Anthropic and OpenAI announcements have said that, you know, in a saner world we'd be going slower here. A lot of the people at the helm seem to notice the dangers and seem to wish we were going slower, but they're sort of trapped in this race where everyone needs to keep doing it because they fear that if they don't the next guy will do it worse.
Also, stopping this race does not mean giving up the nice stuff we currently have. You know, you could stop the race to build radically smarter than human machines while still having self-driving cars. While still using AIs for cancer cure research. You can even keep the current chatbots. You know, these are not the things that are able to escape. Sometimes they try, but they can't yet, right? And they're pretty helpful in the meanwhile. So, you can keep all of this stuff. The only part we need to stop is narrowly the race to super intelligence. A lot of people say this race is inevitable. It would actually be pretty easy to stop the race towards super intelligence. Advanced AI chips are easier to detect than uranium rocks. You know, an AI chip can only be fabricated in like a few factories around the world, whereas uranium is a rock that you can dig out of the ground. Right? These AI chips require the... they're sort of the peak output of the supply chain. Very brittle. Require, you know, a lithography machine that only exists in the Netherlands. It would be so much easier to track where the AI chips are going than to track who has a lot of uranium that they're trying to refine. These chips are harder to build than centrifuges. AI is fundamentally more dangerous than nuclear weapons. If a nuclear weapon... you know, a nuclear weapon can level a city, a rogue super intelligence can level a planet. So, from the perspective of the sheer logistics, it would be no harder to stop the race towards super intelligence than it was to do nuclear arms control.
The difficulty we have is that the world hasn't really noticed the problem yet. Right? It's like we're in a bus headed towards a cliff edge and the driver's asleep. Right? It's not that the bus can't be stopped, it's just that the driver's currently asleep. But, there's a lot we can do here. This is sort of aimed at congressional staffers, not you guys, but, you know, one of the best things I think you guys can do is just call your representatives and be like, 'This is an issue. We need to work on it. We need to figure it out.' And that'll help some of them be bolder in talking about these issues. I've met a number of congressional members who express concern about these issues, but say, 'Oh, I can't talk about it because, you know, it still sounds too crazy.' And that's changing. That's changing with events like, you know, Claude Methuselah having superhuman hacking abilities, but calling your Congress members and saying this is an important issue. Please address it. That still helps. And there's a bunch of things that people in Congress will be able to do if they have the political will. Everything from just calling for an international coordinated halt to the race to, you know, investing in technological mechanisms that will make it easy to verify where these AI chips are going and what they're doing.
So, you know, in some news, or in some sense, it's very bad news that we're in a bus that's careening towards a cliff edge. But, in some sense, it's very good news that the driver's asleep. Right? It's better than a situation where we're careening towards the cliff and the driver's like, 'Yes, I chose this.' Because everyone who says like, 'Oh, it's inevitable. We can't stop it.' Like, okay, can you wait until the driver's awake? Like sure, maybe when we wake the driver up, they'll be like, 'No, I love going over cliffs.' But, like when the driver's still asleep, that's not the time for despair. That's not the time for inevitablism. Right? All of this like, 'Oh, it can't be stopped.' Like, let's wait until the driver's awake. So, you know, we're not over the cliff edge yet, and where there's life, there's hope. Thanks.
All right, question time.
Yeah, that's the sort of thing that strikes me as extremely hard to predict, right? Like if someone asks me, if someone's like, 'Hey, I'm about to play Magnus Carlsen in a chess game, who's going to win?' I'm like, 'Easy call. No offense, it's Magnus Carlsen. They're winning the game.' And you're like, 'Great. What piece are they going to use to checkmate me?' I'm like, 'Wow, that's a different sort of question.' Like I can be like, 'Well, the queen's most likely because it's the most flexible piece.' But it's just like a different class of prediction, right? And so, like does humanity just get steamrolled by the AI? Yeah, that's an easy sort of call if you make smarter than human things that are doing stuff you didn't want them to be doing. It's just most things that they can be doing that aren't what you wanted them to be doing and aren't what you meant them to be doing are energy-hungry things. Most things that they can be pursuing are resource-hungry things. And so, you know, if they're smart, they're going to find ways to get those resources, and then you're not going to have the resources. And like what does that look like? I mean, one sort of way to predict what it looks like in the limits is to look at the physical limits and be like, 'Okay, if you have some resource-hungry thing pursuing some objective we have no idea about, but we can assume that it's going to use the resources available on Earth, what are the sort of limiting factors? What does it look like for Earth to be pushed to the resource acquisition limits?' And that looks like things like the AI's taken all of the sunlight. And so, you have no sunlight for your crops. It looks like the AI's taken the land. Maybe you're sort of crushed beneath the self-replicating factories. It looks like the AI is... So, one of the resources, one of the limiting factors in running a lot of computation on Earth's surface is actually ability to discharge heat into space. And you can discharge more heat from the Earth the hotter the Earth is. Right? So, your limiting factor is sort of ultimately, you know, you might think it's energy production, but there's actually quite a lot of fusible elements here on Earth. There's a lot of hydrogen. There's a decent amount of helium. So, energy is not actually a limiting factor. Ability to radiate heat is, and that happens easier when the Earth's hotter. So, like in the limits, what it probably looks like is there's no sun, and Earth is also much too hot for humans to survive. Right? Which one of those kills you first? Hard to say. Right? Maybe what kills you first is the AI's like, 'Oh man, these humans keep trying to A, turn me off, and B, create another AI, which would be a competitor and a problem. Now that I have my automated robots up and running the supply chain, I'm just going to make a virus and kill them.' Right? Or it could be like, 'Ah, they're obviously no threat to me. I'll just take all the sunlight, and they die of starvation.' Or it could be like, 'Well, we're doing the self-replicating factories real quick like.' And then it's like anyone's guess whether you die first of the heat or of the factories sort of crushing you under foot. I don't know. I think this is a little bit... it's not entirely like, but it's a little bit like chimpanzees arguing about like how the humans are going to kill them. It's like I don't know, probably with technology you're not expecting.
