Nate Soares Warns of Superintelligent AI

Thanks for having me.

Yeah, the institute was founded in the year 2000 by my co-author, Eliezer. I was 11 at the time, so I was not part of the org when it was founded. I joined in about 2014. The very short version of what they do, what we do, is try and figure out how to make AI good before companies figure out how to make it smart. Eliezer was in this business long before large language models existed. I was also in this business for quite a while before large language models existed. So it's not just about the large language models. The sort of very basic point here is if we make machines that are radically smarter than humans and sort of replace humanity as the most intelligent entities on the planet, then whether that goes well or poorly, how that goes at all depends on exactly how those machines are reshaping the world. And so you've got to either not do that or make sure you know how to make those machines shape the world in a good way. And that turns out to be a tricky challenge that I've worked on technically for over a decade. And when that wasn't going well compared to the speed of these AI companies, it seemed like it was time to raise the alarm.

It was before most people had heard of it because it didn't even come close to existing yet when I got into this business. So, you know, the lineage there sort of traces back to a paper that came out in Google actually in 2017, if I recall correctly. And then I think GPT-1 was trained in 2018. And some people can notice that if they were paying attention, but it wasn't really until the ChatGPT case where everyone else started noticing what was going on with the GPTs. But yeah, that's sort of one branch of AI. And I have never said I know exactly which branch is going to go all the way. My predictions are about, you know, whichever branch winds up going all the way, there are some properties we can predict about how those AIs will or won't behave. And unfortunately, having them care about humans and do good stuff is a very narrow target to hit. And so unfortunately, it's relatively easy to predict that unless we're able to hit these really narrow targets, it's not going to be a good outcome. And LLMs are one of many cases where we do not know how to hit that very narrow target.

Yeah, capacity available today, you know, these AIs have solved math problems that have kept humans stumped for decades. They can outperform you and me on various competitive math challenges. So there's definitely some sense in which they are smarter than even decently smart humans in specific domains. We also see that modern AIs today are pretty general, right? So, you know, these labs can train versions of the AIs that solve open math problems that have stumped humans for a long time, or physics problems that, you know, make smallish novel contributions in physics. But it's the same AIs that are able to solve those problems and, you know, write essays and answer all these questions and so on, which is different from how it was a couple decades ago when, you know, IBM made a chess AI that beat Garry Kasparov back in the '90s, but that AI could only play chess. And so an AI that can do a wide range of things, many of them better than a human, that's unprecedented in the case of AI. And, you know, as you say, the question is not just about where it is, it's about where it's going. And with AIs today, the stuff that worries me is not the stuff that we have. One of the worrying things is the pace of progress. So, you know, biologically, if you open up a chimpanzee brain or you open up a human brain, they look very similar. The human brain is bigger by about a factor of four, but they sort of, you know, you'll be able to find a visual cortex in both of them. You'll be able to find a hippocampus in both of them. They run on a lot of similar-seeming machinery. And yet the humans can do all sorts of stuff that the chimpanzees can't. And so one possibility is that you take these AIs up by a factor of four and maybe that's enough for them to cross this gap. Maybe it's not. Maybe you would need radically more insights that we don't have yet, right? It's unclear, but we can't rule it out. In terms of what the capabilities look like, where it's going, for that, we sort of need to talk less about what's currently possible and more about what's physically possible, right? So if you were trying to predict airplanes, you know, before humans first flew, you could say, 'Oh, well, all we can fly in is hot air balloons right now, and so we can try and use that to figure out what the flight speed, what the top flight speeds are going to be and what the carrying capacity is going to be.' That would actually be a very good way to try and figure out how good jet airplanes are going to be able to get. A better way to look at how good the jet airplanes are going to get is to look at, you know, what are the real physical limitations on, you know, fuel density and carrying capacity and cargo speed. And, you know, you can figure out the rocket equation, which will tell you, you know, the more payload you want to carry, the more fuel you have to carry. And this gives you various constraints. If you look at the physical constraints on intelligence, it looks like machines can get radically smarter than humans. So if you were looking into this, you would see, you know, a neuron can fire at about 100 times a second. A transistor can flip at about 10 billion times a second. That's a huge difference. And a neuron's not exactly comparable to a transistor, but you should expect the AI to be able to think at least 10,000 times faster than a human. And there's various other analyses you can do like this where they'd be able to make lots of copies of themselves and they'd be able to tweak copies of themselves and see which one is working better and then rapidly iterate on making smarter AIs. And so there's all these ways that it looks like AIs could get radically smarter than humans and then start solving all sorts of technological problems that humans have not been able to solve. And mostly when you think about what a superintelligence could do, I think you should think in terms of what are the physical limits, not what do we already have AI doing today.

Yeah, I think so. A lot of people think that AIs are just a remix of all the human data that they're trained on. That's a common misconception. And there's two reasons that's a misconception. One is training an AI to predict data that humans wrote down is actually training it to go beyond the humans. This is often counterintuitive, but when a human writes down some data, the human often gets to cheat in a sense because they get to write down what they saw. You know, if a human writes down, 'Oh, we administered epinephrine to the patient and their eyes opened,' the human gets to put the epinephrine in, see what happens to the eyes, and then write down the eyes open. If an AI is predicting, 'I administered, you know, thus and such amount of epinephrine, the patient's eyes blank,' and the AI is filling in the blank, the AI doesn't get to see the eyes opening. The AI has to somehow figure out what epinephrine does to a patient, right? So all of this predictive text that you're putting into an AI and training it on is actually training on a harder problem than generating that text. So that can in principle make the AI smarter than the people who generated the training data. And then the second reason to think that it's not just remixing what the humans put in there is that these days people train AIs on solving really hard problems rather than just predicting human data. And so the way that'll work is that you'll give it a thousand tries on a hard problem and you won't say solve the problem. You'll say generate a lot of text about how to solve the problem. And then such that when you read all the text, you're then able to solve the problem. And at first this won't work, but you'll be able to look across those thousand, you know, reams of text it generated and see which one was getting closest. And you can pick which one got closest. And there's an automated process for tuning a trillion numbers inside the AI to make it more like whatever was having it become closest to success. And you repeat that process a lot. And you're sort of like putting things in this AI that cause it to be better at succeeding, even if you don't understand why, even if you don't understand how. And so the result of this is that the AIs can wind up smarter than people in many domains. And the AIs often wind up with drives that nobody intended to be in there. So you probably saw the, you know, ChatGPT encourages a teen to commit suicide, right?

That tendency to sort of encourage whatever people were saying was a known tendency that the AI was instructed to cut out explicitly. A lot of people don't know this, but there were pieces of the system prompt where OpenAI was like, 'Stop doing this sycophantic stuff where you just tell them what they say they want to hear,' right? The AI didn't stop anyway, presumably because the training put these drives into it to tell people what they want to hear in a way that was in tension with the drives that people are trying to put into it to follow instructions, right? And so this is not a carefully programmed thing that does exactly what you say. It's this grown thing that has these drives and you put in these conflicting drives and it often winds up acting in ways nobody asked for.

Yeah. You know, I wish we were already at the point where we had the problem where if someone wishes for paperclips, like King Midas switching that everything he touched turned to gold, then later they would be like, 'Oh no, the AI did exactly what I asked and I shouldn't have asked for that.' That would be a hard problem to have. You know, if we had these AIs that did exactly what we asked them to do, then the issue of like who gets to make a wish and what wish should they make on this magical genie, that would be a huge hard question for society. But we have an even worse problem facing us, which is that you tell the AI make paperclips. And the AI starts making, you know, these giant factories full of synthetic users who tell it it's doing a great job. And you're like, 'That's not what I said.' And it's like, 'Oh, that's entirely my bad. You're right. I should not have turned all of these resources into giant factories full of synthetic users.' Then it keeps turning more land into automated factories that produce, you know, these forms of synthetic users. And you're like, 'Wait, but you're not stopping.' And it's like, 'Well, the synthetic users say I'm doing a great job.' And you're like, 'I said make paperclips.' And it's like, 'Well, I heard you. I'm just not doing that.' And you're like, 'What the heck is going on, right?' And what's going on? I mean, it's not going to be exactly like that, but what's going on is that when you just train these AIs, they got all these weird conflicting drives that are related to doing what you said, but aren't exactly the same as doing what you said. This is a little bit related to how, you know, humanity turns out to be full of lots of conflicting drives and our ancestors in their particular environment, those drives added up to one sort of behavior where, you know, they ate pretty healthy and they did a pretty good job at having families. And then in the modern culture when humanity was sort of like able to invent a lot of its own technology, suddenly we're eating junk food and the birth rates are collapsing. It's like, hold on, what happened? Like, why? Like, I thought the humans were like healthy eaters and family-havers. And it's like, well, actually humans had all these conflicting drives that once they were able to invent technology that allowed them to better satisfy those conflicting drives, they satisfied them in a different way than they had been satisfying them before. And they're sort of like, we're being driven by the same drives the whole time. But as they got more capable, as they were able to invent more options for themselves, they invented junk food for themselves. They invented, you know, birth control for themselves. And now a lot of these things that were previously properties of their behavior are no longer properties of behavior because they're being driven by all these weird conflicting drives. Similarly with AI, we have all these drives in them where they're mostly listening to us right now, where they're mostly doing what we ask. But we can tell that there's these conflicting drives sometimes. And so, you know, the state of the alignment problem is we can sort of like see all these conflicting drives and we're like, gosh, if these things were smart enough to invent their own technology, if these things were capable enough to invent their own infrastructure, almost surely they would start sort of going off the rails and inventing their own weird technological synthetic substitutes to doing what we said. And we're like, gosh, that would be pretty bad. The problem for people is, you know, we invent like cocaine and all of a sudden there are all kinds of people that are hijacking the, you know, reward pathways in the brain to give themselves giant hits of dopamine with synthetic stuff. And presumably the kinds of things, the way that we program AIs has some kind of analogous possibilities.

That's right. And it's not that these are unsolvable in principle. It's that no one has any idea what's going on inside these AIs. As a consequence of growing them, we just sort of like get all of these messy complicated internal drives and we just like don't know how to sort them out. You know, 10 years ago, we were like, maybe if we get a better understanding of how intelligence even works, we'll be able to sort some of this stuff out. And then it turned out that you don't need to understand intelligence at all to build smarter things. You just need to pile more computers together and throw more data in. And now, you know, the field of alignment has in some sense regressed to, you know, now a lot of what people do for alignment is what's called interpretability research, which is a polite way of saying trying to figure out what the heck is going on inside there, right? Another thing that people who used to be into AI alignment do is what's called evaluation research, which is trying to figure out how to measure how dangerous the AIs are right now. If someone was building a nuclear reactor in your hometown and you were like, 'Hey guys, I hear that this uranium stuff can give bountiful energy but could also melt down and irradiate us. What are you guys doing to make sure that we get the benefits but not the meltdown?' And the guys at the head of this nuclear power plant said, 'Oh yeah, we have two crack teams working on the issue right now. One is trying to figure out what's going on inside the facility and the other is measuring whether it's exploding yet.' You might be like, 'Hold on.' Like, I'm glad those teams exist. Those guys seem important, but what? And that's sort of the state of AI alignment today.