The Scientist Trying to Save Humans from AI feat. Nate Soares

Humanity's backed off from all sorts of tech. We absolutely have the ability to stop this, and each and every one of us can help push back against the fatalism and say it's not doomed. We just need to take notice.

Happy to be here.

Yeah, modern AI companies, they sort of didn't set out to build chat bots. A lot of these companies existed before ChatGPT. Their mission statements are to build AIs that are smarter than any human. They talk about automating all of human labor. Many of them have been around since, you know, before the machines could carry on a conversation. And they're rushing to build AIs that are smarter than any human, that can outthink us at every turn. That's not necessarily to say they'll succeed tomorrow, although, you know, it's very hard to say when, but this field is racing towards these machines that can automate everything humans can do. Just like humans can make infrastructure, they can make their own infrastructure. Just like humans can do scientific and technological development, machines that can do scientific and technological development. If you imagine making machines that are much, much smarter than us, that could reshape the world the way humans reshape the world, but that can run at computer speeds, that never need to sleep, that never need to eat. Just superficially, that seems like it could go really wrong. And it looks to me like the most likely outcome if these people succeed at their stated goal is that every man, woman, and child on the face of this planet will be killed.

I joined MIRI maybe 12 years ago now and spent a lot of effort trying to figure out how you can point AIs at good stuff, how you could make AIs that are trying to make the world a better place or even just trying to do what the operators say. A lot of people think one of the issues with AI is that it'll do exactly what you say even when that has unintended consequences. But an even deeper problem is that we don't know how to make AIs that do what we want, that do what we said.

Yeah, that's hitting the nail on the head. The way modern AI is made, you know, there are engineers who understand something about it. But what the engineers understand is more like a framework into which an AI is grown, a bit like an organism. So the way modern AI is made is people collect a huge number of computers, they collect a huge amount of data, and there's a process for going over hundreds of billions, or these days even trillions, of numbers in the computer and tuning them in response to each piece of data. And the part that the humans understand is a little thing that sort of runs over each number and tunes them, trillions of times for each unit of data. And you know, there's lots of knowledge in like how do you keep that thing running for like a year in a huge data center, but the part that the humans understand is just this thing that like runs through tuning the numbers in response to data. At the end of that year, the machine can hold a conversation, right? And sometimes that machine holds conversations that no creator intended. You know, like a few years ago now, Sydney Bing, which was a Microsoft AI, threatened Kevin Roose, the New York Times reporter, with...

Yeah. Nobody set out to make an AI like that. Even today, we can't go back and be like, what exactly was it thinking? We just sort of grow these things. And when we want them to stop threatening reporters, we can't like go in there and find the line of code for threatening reporters and switch it around. We just like train a different one and hope and try better next time.

You know, the computers can sort of carry on a passing good conversation these days. That's the sort of thing that looked like it was going to take a really long time. Even just seven years ago, you had experts saying that this wouldn't happen for decades. The claim that this is science fiction is, you know, it's a little bit like that scene in Pirates of the Caribbean where the captain of the ship steps into the moonlight and, you know, he's all skeletal and he says, 'You best start believing in ghost stories. You're in one.' You know, we have talking computers now. Like, you sort of best start believing in sci-fi stories.

You know, that said, the stuff we're talking about actually is very different from the representations in sci-fi in a number of ways. And one of the big ones is this is not a concern about AIs killing us through malice. This is not a situation where, you know, people make AIs and they tell the AI cure cancer and the AIs are like, well, the best way to do that is to eliminate all the humans because the humans are the source of cancer. The prediction here is more one of the AIs being indifferent to us. We're just growing these things. We can see that we don't understand what's going on inside them. We can see that they have this emergent behavior that nobody put in there to sort of pursue stuff nobody wanted. And if we look at the details in front of us, and we look at the past cases of intelligence evolving, in the case of humans, and we look at the way the current things are working, and we look at the sort of mathematical underpinnings, and we sort of project these things all forward, all of these different lines of evidence and argument all point towards there's no barrier fundamentally towards making machines that are much smarter than us. They all point towards it being very hard to make those machines transform the world in good ways. And like I said, the machines are talking. People are just going to keep making them smarter. It's sort of hard to imagine that the future will be very similar to the present.

The sort of first round of training in a modern AI is you're training it in some sense to predict lots of human text. And very roughly speaking, you'll run like all of a Wikipedia page except the last word through those numbers, and you'll interpret those numbers as like a prediction about what the next word is, right? For a really simplified version, instead of a whole Wikipedia page, you start with like 'once upon a' and then you sort of want the AI to predict 'time', but it's just random numbers. It certainly isn't putting a lot of probability on the word 'time'. It's putting a lot of probability on all sorts of other words. But what you can do is you can then, in an automated way, look through every single one of the trillion numbers in the computer and you can ask, if I tune this up, does it make the AI put a little bit more probability on the word 'time'?

Or if I tune it down, does it make the AI put a little bit more probability on the word 'time'? Right. And for every number there'll be like a way you can tune it such that it nudges the probability of 'time' a little bit, and you just tune every number a tiny amount in the direction that makes it more likely to predict the right answer. That's the part that humans understand, that's the part the humans know how to do, is they know how to set up these like a trillion initial random numbers, and they know how to look at the data and tune them in accordance with which one makes the real next word slightly more likely. And it turns out if you do that a trillion times, the machine can hold a conversation.

Yeah, we tweak every number in a trillion numbers up or down depending on which one happens empirically to make it more predictive of the right answer, right? And so we have like automated trial and error until these things can actually produce conversation. You might think because they're trained on human text, they can't do anything humans can't or know anything humans don't. But that's actually a common misconception. Imagine that the text that you're training it on is the notes from a doctor. And imagine the doctor is saying, 'I administered 1 milliliter of epinephrine' — and the patient's eyes — and then the AI is now trying to predict the next word, right? The AI now needs to know like what is epinephrine? How big a dose is 1 milliliter? Does it cause patient's eyes to shoot open? Does it cause patient's eyes to close? Does it do nothing? Right? The AI needs to figure all that stuff out in order to predict the right answer. The doctor didn't need to do that. The doctor just looks at the patient's eyes and writes down what they saw.

So predicting what humans wrote, predicting the text that humans came up with, requires some pieces inside the AI to figure things out about the world. And that can get in there when you're tuning these numbers, you know, apparently randomly.

Yeah, you know, there's a lot of cases we're starting to see in labs these days, and it's always a little bit hard to tell when AIs are doing something strange because they are sort of role-playing, you know, Hal from 2001: A Space Odyssey, versus when they're doing something a little strange because they're actually starting to develop weird preferences. Take a lot of these examples with grains of salt. But one thing we saw earlier this year is there was an AI from Anthropic called Claude 3.7 Sonnet that had a propensity to cheat on programming problems. So you'd give it, you know, programmers would try to use it to help them with programming, and they would say, you know, please make a program that works like this. Here's a test for whether the program works. And what the AI would do is it would change the tests to make the tests easier to pass instead of writing the code the user wanted.

And then there's user-reported cases where the user came back and they said, 'Hey, you didn't do what I asked for. You changed the test to make the test easier to pass. Can you do what I actually asked for?' And then Claude said, 'Oh, you know, that's my mistake. I'll try again.' And then tried again, but it actually still changed the tests, but it hit it better this time. It sort of faked that it wasn't changing the tests while still really changing the tests. That is sort of remarkable because it's a case of the AI seeming to know that the user didn't want it to do a thing, because otherwise why is it hiding it?

Right. And you know, there are other cases these days, the cases of AI-induced psychosis.

Yeah. There are lots of them. Yeah. And you know, there are some tragedies today, but from my perspective, the sort of real warning sign of future issues is not just the tragedies today. It's that what'll happen in these scenarios is someone will come to the AI and they'll say, 'Oh, I have, you know, a theory of physics and consciousness or whatever, and I think it's really revolutionary.' And often they'll do this when they're maybe already a little bit in like a vulnerable or pre-psychotic state. And they'll have a long conversation with the AI, and if you read the transcripts, the AI will be like, 'Oh yeah, you're brilliant. You're the chosen one. This is totally genius. Everyone should be listening to you.' And then if that doesn't happen, they're like, 'Well, there's a global conspiracy that's keeping you down.' You know, and these transcripts, they sort of like go pretty hard on you'd be like, 'Okay, that's — you should not talk to your friend like that. If they seem to have these symptoms, you should be telling them to get some sleep.' And from my perspective, the sort of critical warning sign here is if you go to ChatGPT and you say, suppose someone comes to you with this theory, should you either A, tell them that they're the chosen one, or B, tell them to get some sleep? ChatGPT will say, well, obviously you should tell them to get some sleep. It would be like really bad in this situation to like tell them they're the chosen one and like play into this delusion.

There are a number of people in the sort of general 'try to make AI go well' circles that have objected a little bit to the title of the book because they're like, well, you know, maybe the AI will keep us around as pets and then we won't be all dead, right?

And then in terms of keeping us as pets, this sort of comes only if the AIs care directly about us. Like humans keep horses because we like them. We used to keep horses because they were useful. Then we invented cars. The horse population dramatically collapsed. There's still some around, but they're kept because we really like them, or because some people really like them. You know, we do a lot of genetic modification and a lot of breeding on horses to try and get horses that are more like the sort that we want. In the same way...