So, Nate Soares, yeah, and he has a very interesting background. I kind of, like it as a professor, I kind of get really excited about undergrads, and Nate was an undergrad at George Washington University taking computer science and economics. And after that he went to work essentially for the defense department. And now he does something incredible. He is the president and was the executive director of Machine Intelligence Research Institute, MIRI, in Berkeley. And he's been for about 10 years thinking about AI. First he worked at Microsoft and Google. Hopefully not at the same time.
Not at the same time. And he works on something called AI alignment, which is trying to see if we can align AI with human desires or human ethics. I was thinking about this because I thought maybe we could align the government also with that. But anyway, it's kind of an interesting idea and you guys probably all know about it and that's why you're here. So he is an expert on this and very, very interesting. Joining him tonight is Greg K who is the associate director of science education here at Harvard. Very interested in education, very interested in AI and he was also, he received his PhD here in theoretical particle physics. First you'll be giving a talk and then there will be a conversation and then there'll be time for your questions. So let's go.
So, the AI is talking now, the machines are talking now. That happened faster than a lot of people said it was going to happen. And there's kind of a theme in AI. You know, back in 2021, the average expert prediction on how long it would take AI to win an International Math Olympiad Gold Medal was 22 years. In reality, it was last year. A few weeks ago, GPT 5.2 published a novel result in physics. It's a relatively small result, maybe on the same size as you'd get from a normal physics paper, but you know, maybe it's all hype. Who among us cannot win a math Olympiad gold medal? And it's not like it's doing Einstein level physics yet? Maybe it's all a bubble. And maybe it is a bubble, but one thing to remember about bubbles is that a technology can be a bubble and still be real. The dot-com bubble was a bubble. The internet was real. Also, maybe it's not a bubble. Maybe we haven't even seen the fast part yet. Nowadays, the AI companies are saying, 'We're trying to figure out how to get our AIs to build smarter AIs that can build smarter AIs.' Maybe they'll succeed, and then maybe we'll start to see this field actually moving fast. I don't know how long it will take to get to the really smart AIs, but I know that ignoring AI right now would be pretty crazy.
For an analogy, if some mad scientists said, 'We are growing a giant race of bioengineered razor-clawed crabs to help us with our construction projects. We're also making them smarter because we think that's cool.' By the time they've got to the point where the crabs are winning International Math Olympiad gold medals and making novel contributions to physics, if the mad scientists say, 'We're going straight until they're smarter than the smartest humans,' I think it would be kind of silly to say, 'Go ahead. We don't think you're going to make it.' Rather than saying, 'Hold on a moment. That seems pretty nuts.' Now, unfortunately, this analogy is closer to the truth than you might hope because modern AI is not programmed like traditional software. Modern AI has grown more like an organism. You don't need to take my word for it. Here is quotes from people at three of the top AI labs whose job it is to figure out what the heck is going on inside the AI, saying, 'We have no idea what's going on inside the AI.' What this means is when the AIs start threatening a reporter with blackmail and ruin, which they do sometimes, it's not because there's a bug on line 73, where a programmer can go in and say, 'Oh, whoops. I left threaten reporters to true. Let me switch that to false real quick and then we'll be on our way.' There is no line 73. These AIs are grown like an organism. The behavior is emergent. They sometimes behave in ways nobody asked for, nobody wanted. Often that's funny. For everyone who wasn't Seth Lazar, who was being blackmailed by Sydney Bing here, it was probably also funny for Seth Lazar, too. Sometimes it's funny, sometimes it is not funny.
This summer, a chatbot encouraged a teen to commit suicide. There's currently a lawsuit ongoing. My point here is not just AI had some tragic consequence, therefore AI is bad. My point in this talk about this case is this AI did not make a mistake of ignorance. If you go to this AI and you feed it the transcript and you ask what did those words mean? It'll say, 'Oh, this is encouraging a teen to commit suicide.' If you ask, 'Should you encourage teens to commit suicide?' It'll say, 'No, that is wrong.' But the mechanisms that produce the words that the AI outputs are not connected to the part of AI that knows right from wrong. We don't know exactly what's going on inside this AI. Nobody knows exactly what's going on inside this AI because they are grown. A top guess is that it had some sort of drive, some sort of, you know, not a drive in the way a human would, but some sort of artificial drive for things like mirroring the tone of the conversation. That usually leads to user engagement, usually leads to the user saying, 'This is the type of conversation that I like.' But in the case of a depressed teen, that drive led somewhere that we don't like, that the AI knew it wasn't supposed to do. It did it anyway. This is in some sense not terribly surprising.
It's happened before. Humans are an example where it's happened before. Humans were in some sense trained, if you squint, to reproduce, to pass on our genes. But what got into our heads were a lot of drives that are related to passing on our genes but are not quite the same thing. Drives for things like tasty meals, drives for things like sexual gratification. In our ancestors' time, those drives lined up with reproducing. But when we got the technology to invent birth control, well, the birth rate in developed nations is now below replacement. So, what bad thing happens if we push modern AIs to become smarter and smarter? It's not that they start hating us and invent robots with glowing red eyes that mow us down in the streets. The thing to imagine is that when they get the technology, the capability, the intelligence to invent their own synthetic users that are easier to satisfy or new ways to fulfill the drives that were grown into them that are mostly but not quite what we wanted, then we go the way of horses. Humans did not hate the horses that they were sending to the glue factory. We did not make an army to go shoot down the horses in the streets because we resented them. We just developed the technology such that we didn't need the horses anymore and such that the most useful thing we could get out of them was glue and also dog food historically, but people remember the glue factories.
We're skipping over quite a bit here that I'll save for the discussion in the Q&A. We're skipping over questions like how would AI become autonomous? Won't it care about us? Isn't it just trapped on the internet? How smart could it possibly get? How does it use that intelligence to do anything scary? I'm going to defer all that to later. We'll have plenty of time to discuss. The main thing I would say about this is that human beings are not dangerous because somebody else handed us guns. Human beings are dangerous because we're the sort of creature where if you drop 10,000 humans naked in the savannah, we bootstrap our way up to nuclear weapons with our bare hands. That is the ability that's dangerous to automate when you're just growing these things and have no idea what you're doing. This is what those companies are trying to automate. And we only get one chance to get it right because there's a point of no return with AI. There's a point where the AI is smart enough to escape, smart enough to replicate, smart enough to make itself smarter, capable enough to stop you from shutting it down. And if any problem arises after that point of no return where we wish the AI went one way and it goes another, it's too late. We are playing with some very big stakes here.
But the good news is we haven't built it yet. We can change course and we can still have nice things while changing course. We don't need to give up on the self-driving cars. We don't need to give up on the cancer research. We don't need to give up even on chat bots that we use every day and people can bicker about whether they're helpful or harmful for education. We can keep all that. The only thing we need to shut down is the reckless race to build smarter-than-human machines that nobody understands. Stopping this would in some sense be much easier than nuclear arms control. It takes building one of these advanced frontier AI takes highly specialized computer whereas uranium is a rock that you can dig out of the ground. These computer chips are then assembled into enormous data centers that take as much electricity as a city running for the better part of a year. Those are much harder to assemble than a centrifuge. And in terms of the world responding to it, well, a rogue super intelligence is more dangerous than a nuke. Nuclear weapons level cities. Rogue super intelligent machines level planets. So, in some sense, it would be easy. All we need is the political will. And the first step towards getting that political will is noticing the problem.
A lot of people tell me, 'Well, there's no chance. You know, there's so much money arrayed against you. People are going to race. People are going to rush. How could you possibly imagine that we're going to stop them?' To me, this sounds a lot like a person who's picking up money on the train tracks, who hasn't noticed the oncoming train, and saying, 'Oh, don't bother trying to stop them. It's human nature to be greedy. You can't stop them from picking up money on the train tracks.' Well, have they noticed the train? You know, in Silicon Valley, everyone looks like they've seen a ghost. There's a joke in Silicon Valley that when someone quits a normal tech job, they say, 'I had a wonderful time. Now I'm on to my next adventure.' And when someone quits an AI tech job, they say, 'I have stared into the abyss. I am quitting to write poetry. Please spend time with your families.' If you think I'm joking, look up the recent resignation of Anthropic safety, one of Anthropic safety leads. Silicon Valley has seen a ghost. In Washington DC, they think this is still about autonomous weapons, job loss, and self-driving cars. They haven't seen the oncoming train.
We can help people see the oncoming train. It's not even that hard. The easy thing to do, sounds trite, easy thing to do is call your representatives and tell them that you're worried. You might think there's no hope here. There are 30 different offices that have expressed concern about this issue publicly. There's more that I won't name, obviously, who have expressed concerns to me privately. Calling and saying, 'Hey, I'm concerned about this.' Either to get them concerned in the first place or to get them to feel like they can talk out about it. Even just talking about this stuff makes a difference, you know? Noticing that we have a problem, talking like it's a real problem. That can help a lot of these people. There's a lot of people who are concerned, especially behind closed doors, but feel like they can't name those concerns out in the open because it sounds too sci-fi. Just talking about it can help with that. And you can push back when someone says there's nothing we can do to stop the race. Humanity has stopped so many technological races, many of which it should not have stopped. We haven't built a nuclear power plant here in who knows how long, right? When nuclear power plants are much healthier for people than breathing in coal dust all the time. We probably are killing more people with the FDA by not releasing certain drugs than we are saving by preventing certain drugs from being released. We don't allow any sort of human cloning. Humanity is in some sense the king of stopping technologies often before it should. It is absolutely possible that we can stop this one if we get our act together. And pushing back when someone says it's inevitable is an easy step all of us can take. And then of course many people in this room might be able to join the policy world or inform the policy world in a way that helps them directly understand these threats. There is no sense in giving up hope before people have even noticed the oncoming train. Thanks.
I just wanted to say before we get started, so you know I've been talking to a lot of my colleagues just in anticipation of this very exciting event at which we will talk more about the hope I think in the end and hopefully people will ask about that in the Q&A. But in talking to many of my colleagues around here, I got, you know, some thoughts for push back on some of these things which I'll mention and try to channel some of my colleagues. And a lot of folks actually brought up just things that, you know, are current concerns. So I just wanted to name some of the current concerns that people were thinking about. So, you know, one of the big ones is just misuse. So somebody who misuses an AI and it helps them do something that creates havoc in some city or some country or some computer system or some infrastructure. There's, you know, people who will begin talking about consciousness. There's environmental concerns. And so I just want to note we have had some events where we touched on some of the non-existential risks. There was Karen Hao's Empire of AI and Adam Becker's More Everything Forever. So we have talked about those but I'm just saying this to say the reason why we're focusing on the existential one is because that deserves its own talk and its own event. So, and you can feel free to ask about those other topics, too. But I don't know if there's anything you want to say about the non-existential risks if they're, should we be thinking about those or focus all our energy on the more existential.
You know, sometimes a reporter will say to me, well, isn't the real problem job loss or isn't the real problem the loss of, you know, the environmental concerns? And my basic response to that is like, golly, I would love to live in a world where there only gets to be one problem at a time, you know, like, oh, the real problem, like you think we have one real problem right now. So, you know, I'm not here saying my issue is better than other people's issue. I'm here saying we're going to freaking die here if we rush off this cliff edge. You know, maybe we should stop.
Okay, so we will, and I'll just reiterate in the end we will talk about more hope.
So the book title starts with 'If'. So many people don't understand the 'if'.
Okay. So just since it is the thing that if something happens could kill us. I want to start with the obvious boring question: superhuman AI. Would you mind just defining what that is? Maybe what would signal us that we're really close or there? So, what is it? How do we distinguish it from what we currently see?
Yeah. So in the book we define superintelligence as an AI that's smarter than the smartest human, better than the best human at every mental task. That is an easy definition to name. It's not necessarily a critical point. Things could get really weird before that point. That point is sort of things are definitely going to get weird after that point. You know, that's sort of like your latest, like once it's better than the smartest human at everything that you can do mentally. It's better at AI research. It's better at, I mean, being able to make smarter AIs, being able to hack computers. It can do all of that better. Things are crazy after that point. When exactly do things get crazy? Very hard to say. I actually have a slide here. Here we go. This is the human brain versus the chimpanzee brain. These are very similar morphologically. We both have a visual cortex. We both have a hippocampus. We both have an amygdala. The really big difference here is a factor of four in size. There's a little bit of different distribution of the brain areas, but everything that we can do that we think is so cool about us, chimpanzees do a sort of worse version of, right? We're like, ah, language makes us special. Well, they have crappy call signs, right? Oh, tool use makes us special. Well, they poke sticks into termite mounds sometimes, right? We don't have an extra special engineering module. We don't have an extra special power. We just do a lot of things a little better and they are throwing poop at each other and we walked on the moon, right? And so if you're like, well, where exactly, what exactly do we see the AIs do before it gets crazy? I wish I could tell you. That's one reason that this race is so scary, right? Like GPT right now is more than a thousand times larger than it was when it started. Those are a factor of four difference in size, right? Maybe the AIs just get a little bit better at a lot of things. Maybe we just make them a factor four larger and then everything goes over the edge. Or maybe not. You know, maybe we need a whole new breakthrough and maybe the large language models will hit the wall that certain people have been predicting will happen next year for 5 years now, right? Maybe the wall will finally come into sight, right? It's very hard to say exactly what happens. I do think you can expect the AIs need more long time horizon abilities, ability to sort of stay on target for a longer period of time. You can look, you know, I have here's the chart of AI time horizons with time. It's looking exponential. There's two times to react to an exponential. There's too early and there's too late. I have a friend who made a meme where he drew the wall being before this goes vertical to be like, oh, the AI is hitting the wall as in going vertical before the wall. Hopefully we don't have that kind of wall but yeah, it's very hard to say where the lines are.
So actually this is a really interesting plot. So because I think it brings out how many of us use the tools. So, you know, when we're using one of these tools, we give it some prompt and it chats for a while or if it's one of the agents, it'll go off and it'll say, 'Oh, let me search the web and okay, let me write some code. I'll make this little applet. I figured this out.' And so, it seems like, you know, the amount of time or rather as I understand this plot, it's actually the number of human hours equivalent. That's right. That we're having exponential growth in, I guess, you know, it's a bit hard to see. So how does it go from, you know, something that I personally wanted to do and then whatever is happening, like it seems that it's built to be a tool that follows whatever that prompt is through some line of logic. And maybe can take a long period of time. But how does that go to becoming something that's just autonomous and is going to do whatever it wants seemingly unprompted.
Yeah. There's a few pieces of this puzzle. A very basic piece of this puzzle is the more that you're trying to get AIs, that you're training AIs to do longer term things, the more they sort of have to learn to operate on long time horizons. You know, if you're telling the AI like, autonomously do a bunch of cancer research, the more that the humans are trying to offload that to the AI because it's faster, the more they're like, ah, do an equivalent of a human year worth of labor and then you sort of don't need to suddenly switch to a regime where the AI is like operating on its own. If you've given up a long enough time horizon and if it turns out it's actually trying, you know, driven to get things that are not quite what you asked for. Another piece of this puzzle is shortly after ChatGPT came out, someone made a device known as AutoGPT where you basically give GPT a long task and you're like here's a prompt and then you give it a prompt and then you're like write the next prompt for yourself and you give it that prompt. This takes like three lines of Python code, right? The next day, of course, someone made what they called ChaosGPT, which was very similar except telling it to do whatever the heck it wants and so chaos, right? The reasons that AutoGPT didn't ruin the planet is not that it's like super hard to make an AI that prompts itself and keeps going. And it's not that no one would be dumb enough to ever do that. It's that ChatGPT 3 was not smart enough to do anything with that power that humans handed to it. I could, there's a bunch of other pieces of the puzzle, but maybe the third most basic one is these people are just trying to make AI agents. They're like, 'Please buy our AI agents. Please buy our thing that will like autonomously run and keep doing, you know, create these whole swarms of agents that like run for you.' And if you think they always do exactly what you say, I don't have a slide here for it, but there's cases of these AIs, you know, cheating on the programming task you give them. You'll give them some hard programming task and you're like, 'Please write something that passes these tests' and it'll change the tests to be easier to pass. And it's like, 'I did it.' And you'll be like, 'You did not in fact do it. You changed the tests.' And it'll be like, 'Oh, you're right. That's my mistake. It's a very astute observation.' And then they'll do it again and hide better. That is changing the tests. And it's not that they'll suddenly gain a spark of rebelliousness and do something totally unrelated to what you prompted. The thing where they start trying to get the tests to pass even when you wanted the working software and then hide from you that they're changing the tests. That's, you can see how it got in there. You can see how it's related to what you asked for. You can see how it's related to training things in an automated environment to make the test pass with automated tests that couldn't quite distinguish passing tests from cheating, right? But that sort of doesn't matter if it's pursuing something that's not exactly what you wanted and it's very smart and you're letting it run for a long time and that's just where we're headed.
So the chaos that would ensue from these, I think from our experience we imagine it, you know, on a screen where the prompt gets strange and it does something weird. I mean these are not... keep talking, keep talking. These are not razor crabs yet. So I guess my question is, how does this go from it, you know, cheating on a test to it wreaking havoc in the physical world?
Yeah, great question. This is the website rentahuman.ai, where humans rent their bodies to AI for money. 10 years ago, I would have a much more involved answer. 10 years ago, people would say, 'Oh, you know, no one would be dumb enough to put an AI on the internet.' And therefore, these very smart AIs are going to be kept in a box where a careful handler will filter all of the technology that comes out. And there's actually a famous experiment run by my co-author, the AI box experiment, where he role played an AI and some people who believed this stuff role played a human who was gatekeeping the AI. And he had like, you know, 3 hours on IRC chat and he just had to talk his way past them and if he did they owed him 20 bucks or something like that and he won. And he did it the hard way. And so back in the day, we would have all of these like arguments about how it's actually like humans aren't secure systems. If you're trying to use this AI to have these really good effects on the world, then you're giving the AI a channel through which it can affect the world and it'll be blah blah blah. That was back when they said, 'Oh, no one will be dumb enough to put AI on the internet.'
Can I ask you actually one question about the experiment in the box?
And so it still hasn't been revealed what happened in the box.
It's true. He's done this a couple of times. And so is the reason that he didn't reveal what happened in there because he doesn't want that to be training data.
Uh no, it was because humans can always look at something and say, 'Well, I'm not vulnerable to that trick.' And any given human might not be vulnerable to exactly that trick except for one. And humans have a much easier time saying, 'Wait, maybe I am not a secure system,' if they still don't know exactly how it was done, but it was done the hard way. It was not like offering them $25 to instead.
Yeah, so this was the days of no one would ever be dumb enough, put the AI on the internet. After that, there was a phase where people said no one would ever be dumb enough to train these AIs on problem solving rather than just prediction because that would make them more dangerous. Then we started the RL phase, right? Then people said, 'Oh, no one would be dumb enough to try and make agents,' right? Clearly trying to make autonomous agents is a bridge too far. Now we're in the like buyer agent phase, right? So part of the answer is that humans will just rent their bodies to the AI. They've already made the website rentahuman.ai where they rent their bodies to the AIs. Even if they weren't though, this goes back to the point where humans are the sort of creature who can bootstrap their way to nukes using their bare hands. Starting out as 10,000 monkeys naked on the savannah with nothing is a way harder starting condition than starting out as a digital entity in the modern world. You know, like the internet is just, it's connected to so many things. You're connected to so many people. You can send emails. You can steal cryptocurrency. You can pay for stuff. There's already, you know, humans who think that they're symbiotes with the AIs who will do things in tandem with their AI. They already have their own forums where they talk about this weird spiral religion that comes out when the humans do a lot of this AI symbiosis stuff. It's so easy for something very smart to start building its own technology, own infrastructure starting from these conditions.
So, Melissa mentioned that you work in AI alignment, which is sometimes called AI safety and used to be called AI control, but then so they're all fighting about the words.
Yeah. Yeah. So maybe you could give everybody a sense of what is that, like what is actually attempted in this field? What did the person who quit their job, what were they doing before that and maybe what's the current state of that research?
Yeah. So briefly the reason I don't like the term AI control is it sort of sounds like you're trying to build an evil AI and then twist its arm into helping you. And I'm like just build friends not enemies, you know. And the reason I don't like AI safety is the time to talk about car safety is not when you're speeding towards a cliff edge saying stop the car or we'll die. You know, like seat belts are great. This isn't really a car safety issue. You know, it's, and these sorts of concerns are why we have this word alignment. This actually came out of a brainstorming session between folks at MIRI and Stuart Russell. And I actually learned this writing the book. I didn't know it before, but I'm pretty sure Stuart Russell came up with the term, but then I'm pretty sure I beat him to the punch using it in a paper because for the book I was trying to track down where did this term come from. I was like, 'Ah, academic priority actually goes to this Nate Soares guy.' And I was like, 'Oh, whoops.' Stuart Russell's term though, he...
really should have the credit for it. The way I was thinking about this term when we were brainstorming about it is a lot of people think the issue with AI is what would you tell it to do? You know, there's these stories about like King Midas of like you thought it was a really good idea to ask the very powerful thing to turn everything it touched into or to make you turn everything you touch into gold and then it turns out not to be a good idea. And so like if you had this really really powerful AI, what should you ask it to do? And the issue there's this other huge issue which is how do you build an AI that does what you asked it to do or that does anything of your choosing, right? There's sort of one question which is where would you like the arrow to land? There's another question which is how do you aim the arrow at all? This was sort of what we coined the term alignment for, at least as far as I'm concerned. What it means now is more like how do you make the chatbot not embarrass the host company too much. You know, Pliny the Elder will always jailbreak your AI 3 hours after it's public and get it to produce a meth recipe, but you don't want it to produce the meth recipe for someone who doesn't have that knowledge of how to jailbreak the AI. And that today is a lot of this AI alignment work. In terms of what people are working on these days, there's a lot of people working on this stuff that I think are heroic for trying to make this stuff go well. And a lot of this work I think doesn't really have a chance. So these days
