Nobel Prize Winner Geoffrey Hinton Delivers Remarks on the Impact

So I'd like to thank you for inviting me here. It's great to see such a crowd. And I'd like to thank you for arranging such easy transport here. I'm going to spend about a quarter of an hour explaining what AI is, and I suspect many of you don't have much idea of what actually goes on behind the hood. So I'm going to try and give you five minutes of history of AI. Since the 1950s, there have been two quite different theories of how to build an intelligent machine. One theory was we should base it on logic. Intelligence is all about reasoning. That's what people can do, and we should make a machine that can reason. And the way to do reasoning is through logic. So an example of logic is: Socrates is a man. All men are mortal. So Socrates is mortal. Or as Woody Allen put it, so all men are Socrates. And you don't really have to understand what the words mean in order to apply logic. So logic allows you to derive new conclusions without actually getting at the meanings of the words, which is kind of crazy.

There's a different approach, which is to try and model it on the brain, try and build a machine that works the same. Now, the people who believed in that second approach, the two main proponents of that were John von Neumann and Alan Turing, and unfortunately, they both died young. And so for many years, that approach took second place to the symbolic approach, which was based on logic. But recently, we've seen that that approach is the one that was successful. In that approach, what you do is you say reasoning isn't the central thing. Reasoning is something we do much later. Most animals don't do reasoning, at least not much, actually. Some Americans don't do much reasoning, but there we go. Sorry about that. I will try not to let my Canadian prejudices influence me anymore.

So we're very good at perception. We're very good at controlling our bodies. We use our brain to do that. Our brain has evolved for hundreds of millions of years to be good at doing that. And the thing that's remarkable about our brain is that it learns, it learns to do all these things. So the question is, how does it learn? And so a bunch of us, for a long time, pursued the idea that there must be some way in which the brain can change the strengths of the connections between brain cells so that it learns to do complicated tasks. And it turned out in the 1980s, we discovered that way, or we discovered a way to do it called backpropagation, which I won't try and explain technically, but it's a way for the brain to figure out how to change the connection strengths so that the next time you give it a task, it will get a better answer.

So, for example, you showed an image, and it has to tell you what's in the image. Is it a cat or a dog, for example? To begin with, it doesn't know anything, and it says 50% cat, 50% dog. And then you say, okay, for that image, you should have said 100% dog and 0% cat. And it uses that information, the backpropagation algorithm, to change each connection strength slightly in the brain, so that next time it'll say 51% dog and 49% cat. And if you keep doing that, eventually it'll be very good at recognizing the difference between dogs and cats. That's how the learning algorithm we use in AI works, and maybe the learning algorithm in the brain.

Now, when we come to language, there were two completely different theories of what a word meaning is. So the symbolic AI people, going back to a linguist called Ferdinand de Saussure from more than a century ago, said that, well, the meaning of a word is implicit in how it occurs with other words in sentences. And the neural net people said, no, no, the meaning of a word, or psychologists said, the meaning of a word is actually a big bunch of features. So the word cat, when you hear the word cat, when you get the meaning cat, what you've done is you've activated a whole bunch of brain cells that represent features like it's a predator, it's a pet, it's furry, it's about the size of a breadbox, it has whiskers, it can run, it's rather aloof. I don't know if you're a cat owner, but they are. A whole bunch of features like that you activate, and that's the meaning of cat.

And these were two totally different theories of meaning, or apparently different theories. And then we discovered that you could convert between these two theories of meaning. You take a neural network and you get it to try and predict the next word. And in order to predict the next word in a sentence, what it has to do is convert all the words it's seen so far into sets of features. It's going to learn how to do that. We're not going to tell it how to do that. It's going to learn what features to use. So features like predator, it's just going to learn because they're helpful for predicting the next word. And then it's going to let the features of the words it's seen interact with one another and refine themselves.

So if you get a word like 'may', that could be a woman's name, it could be a month, it could be a modal like 'would' and 'should'. It's going to use interactions with the features of other words in the context to clean up the meaning of 'may' for that particular context. And then from all those features, it's going to predict the features of the next word. And once it's got an idea of the features of the next word, it's going to be able to make a pretty good guess at predicting the next word.

That's how these large language models work. They start off knowing nothing. They learn to convert words into features and how the features should interact to be good at predicting the features of the next word. So what's happening is they're taking strings of words in which the meaning is implicit, and they're converting that into the meanings of individual words or sets of features. And this works really, really well. I made the first model that worked like that in 1985. It was tiny. It had 112 training examples. Now we have trillions of training examples, but it turns out this works really well.

And in order to predict the next word, well, you have to figure out the meaning of what's being said. So, for example, if I ask you a question and the next word is the first word in the answer to the question, you can't predict that word unless you understood the question. Now, the point of saying all this is that this idea that you convert words into features, the features interact, they predict the features of the next word, that wasn't developed in order to have chatbots. That was developed as a theory of how people work. I developed it in 1985 as a theory of how people can assign meanings to words, and it turns out it's very good for this technology of chatbots.

And that implies that chatbots and us work in very similar ways. The best model we have of how we work when we understand language is these chatbots. They're much more like us than you might think. There are some respects in which they're very unlike us, but in many respects, they're very like us. Many people will tell you they're not like us at all. For example, they have hallucinations, which should actually be called confabulations when it's a language model. It turns out people have confabulations all the time. If I ask you to remember an event that happened ten years ago, you will confidently report all sorts of details that are just wrong.

It's a shame more juries don't know this. It's very hard to get data on what really happened ten years ago. But there's one wonderful case, which is John Dean's testimony at Watergate. Hopefully, we'll be seeing more of that kind of thing soon. In his testimony at Watergate, John Dean gave accounts of meetings that happened in the Oval Office. Now, he didn't... he was under oath. He didn't know that they had all been recorded. So we have the ground truth. What he said was large amounts of hallucination. What he was doing was making up things that seemed plausible to him, given the connection strengths in his brain. That's what memory is.

Memory isn't pulling a file from somewhere where the file's been sitting there faithfully. It's just making up a good story that fits with your connection strengths. Now, if you have to make up a good story about what happened three minutes ago, you'll probably make a fairly accurate story. If you have to make up a good story about what happened ten years ago, you'll make a story with all sorts of imagined, plausible things that aren't actually true. That's what AI does, and that's what people do. Hallucinations show that they're very like us. And when we talk about risks, the fact that they're very like us is going to be very important.

I want to end my sort of introduction to what AI is by trying to give you a model of what's happening when an AI understands a sentence. So it's a model based on Lego blocks. So with Lego blocks, you can model any 3D shape. It's just roughly, if you want to model the shape of a Porsche, you can, with lots of little rectangular Lego blocks, you can stick them together. The aerodynamics won't be right, but you can model where the stuff is in a Porsche. So words are like Lego blocks, except that they can model anything. We're a very clever kind of monkey that discovered a technique for modeling anything at all using words.

So here's how words differ from Lego blocks. Lego blocks are little three-dimensional things, and there's not many different types of them. Words, you probably use about 30,000 different words, so there's a lot of types of them. Each one has its own name, and they're not three-dimensional. They have thousands of dimensions. Those are all the feature activations that represent the meaning. Like the word cat is high on predator, it's high on aloof, it's low on being huge and low on being tiny, and high on being about the size of a breadbox. That's the features of the word cat.

So instead of being three-dimensional, a word is thousands of dimensions. Now, you probably don't know how to think about 1000-dimensional things. There's a little trick people use. You think of something three-dimensional, and you say 1000 to yourself very loudly. That's sort of the best you can do. Okay, so they're very high-dimensional things. They're also flexible. So a Lego block isn't flexible. Words aren't even... words that only have one basic meaning are somewhat flexible.

Take a word like 'death'. The meaning of 'death' depends on the context. If you're talking about a hospital or a suicide, they all have somewhat different flavors of meaning. The meaning deforms to fit the context. So let me give you an example of that. Words fit in with their context. And so if I give you a new word in an appropriate context, just from one utterance, without any definition, you can figure out what it means. Okay, here goes: 'She scrummed him with the frying pan.' Okay, you're already thinking he probably deserved it.

So that's another way words differ from Lego blocks. They're flexible and fit in with their context. And that's how we learn their meaning, by using the context to say what shape they have to be. These are 1000-dimensional shapes. The final thing about words is that the way they fit together is much more complicated than Lego blocks. And I'll give you a way of imagining the way they fit together in a model that's called Transformers, which is the basis of all the modern chatbots.

So think of a word as this sort of thousand-dimensional shape that's a bit flexible. The best you can do is imagine a three-dimensional shape and say 1000. On this shape, there's lots of long, flexible arms which have hands on the end. And as you change the shape of the word, the shapes of all those hands change in a systematic way. So you change the shape, and the shapes of the hands on all these arms change. Also, on each word, there's lots of gloves that are stuck to the word by their fingertips and open at the other end. And as you change the shape of the word, the shapes of all these gloves change.

And understanding a sentence consists of changing... you start with the initial shape for a word, the sort of default shape, and then you gradually change the shapes as you go through the layers of the network, so that the hands of some words can fit into the gloves of other words. In fact, hands and gloves come in multiple colors. Technically, that's called multi-head attention, and yellow hands can only fit into yellow gloves, and red hands can only fit into red gloves.

But what you're trying to do is figure out how to deform all these words so that they all fit together nicely, with the hands of some words fitting in the gloves of other words. That's the best model I can give you without writing equations of how these chatbots understand things. But that's a very difficult process. It's a complicated search process to figure out how to do this deformation. And that's what's happening as you go through the multiple layers of a transformer. And of course, it can understand wrong. Sometimes it fits them together wrong, and it misunderstands. But most of the time it gets it right.

Okay, that's what's going on in these chatbots, and something very like that is what's going on in your head when you understand sentences. Now we can talk about AI risk, because now you understand these things are actually quite like us, and they're not at all like standard computer software. We write computer programs that tell it how to change connection strengths on the basis of data it sees, like strings of words. So we program how it is going to learn, but what it learns depends on what's in the data. All the prejudices in the data get learned into the neural network.

And at present, we train these neural networks in a crazy way, which is we just show them all the data we can get our hands on. This is crazy. If you were teaching your kid to read, hopefully you wouldn't teach them to read by using, say, the diary of a serial killer. That wouldn't be a good thing to start your kid reading on. But that's what we do with these chatbots. And then once they've learned all these crazy, awful behaviors, we have a stage at the end that's not very good, where we give them prompts, they say bad things, we say, 'Oh, don't say that.' So we try and repress their bad behaviors.

That's a crazy way to build a system, particularly if you're going to have to live with it. We should be training them in much better ways, but I'll leave that for the discussion about AI risks. I'm finished with my short lecture about what AI is.

Yeah, so there's a story that I retired from Google. Is this...?

So that I could talk about the risks of AI. Every time I meet a journalist, I tell them that story is not true. I retired from Google because I was 75, and I was no good at programming anymore. And I thought, it's time to catch up on all those good movies on Netflix. But I thought, well, since I'm retiring anyway, I might as well just tell people that AI is quite risky. So I told the New York Times, and that completely screwed my retirement.

Luckily, things started easing down after a year and a half, and I was beginning to be able to sort of slow down. And I actually started watching things on Netflix. And then the phone went at 1 a.m. in the morning in California. And I picked up the phone thinking, all my friends are on the East Coast. Who's calling me at 4:00 in the morning? And there was this very long phone number. And then this guy with a Swedish accent started saying, 'You won the Nobel Prize in Physics.' And I explained to him he must have a wrong number or something, because I'm not a physicist. And that totally screwed my retirement.

Well, it wasn't as a professor. It's when I was an undergraduate studying psychology in 1970, and I had a friend who was doing physics, and he'd heard about this AI school of artificial intelligence in Edinburgh, and he planned to go there as a graduate student. So I heard about it in 1970.

You couldn't call it AR, if you like. Apparently, you can call it anything. You could call it superintelligence. So, in 2009, there was a small workshop at Windsor in England, and at the time, my back was very bad and I couldn't sit down. So I was standing up, and I was the most senior scientist at the workshop. I was also the organizer, and I just kept interrupting people. It's a bad habit of mine. And at the end, somebody referred to me as the Godfather, because I'd been so annoying. And it stuck. And I now quite like the term. It wasn't intended kindly, but a couple of years ago, I got introduced on stage in Las Vegas as the Godfather.

Yes.

So it started, his team asked if he could call me. And he called, and I started off by saying, 'I'm a scientist. I don't do policy. So I can't answer questions about policy for you, but I can answer questions about the science.' And we talked about the science for a bit. And then we started talking about the risks of AI. And I said, 'Oh, by the way, I think AI is going to cause massive unemployment. So my advice is let the Republicans own it, because then that'll be their problem.' And Obama said, 'Ah, so you do do policy.'

But his views on AI are quite similar to mine, and also Bill Gates's views on AI are quite similar to mine. It's very powerful stuff. It could be tremendously good for people if we can contain the harms, but we have to be very serious about containing the harms.

Let me answer that in a roundabout way. He redefined AI as superintelligence. Superintelligence is a technical term that means smarter than people, right? You can't just take AI and say, 'We're going to call it superintelligence.' Apparently you can, because all the government agencies are now required to call it superintelligence, even though it's not. Well, he's on a roll.

He could make life much easier when you're at school, because you all probably remember that pi is something like 3.14, and it's complicated to do math with it. Trump should just simply define it to be three. That would make life much easier. Oh, one other thing. One other thing about this is there was a Roman emperor who appointed a horse to the Senate, which was a sort of, I guess he was doing it to show that he could, and it was a kind of screw you to the Senate. Trump appointed a Secretary of Education who can't spell AI, if you give her the first letter.

Okay, so one could go into all the ways it could destroy us, and there are many. I don't think there's much point doing that, because there's so many ways it would be so simple. And here's my argument. Suppose you work for a class of two-year-olds, and you eventually get fed up with them, and you decide to destroy them. How could you destroy a bunch of two-year-olds?

Well, there's so many ways you could do it. You could poison them, you could hit them over the head with a hammer, you could get a flute, and you could walk off a cliff with them coming behind you. That would be one of the nicer ways to do it. There's so many ways you could do it. If an AI is much smarter than us and it wants to get rid of us, there'd be no problem. It could design a biological virus, it could do massive cyber attacks, it could turn us all against each other. Well, that's happened already. So, yeah, I don't think it's really worth speculating, but there are some obvious ways, like massive cyber attacks that bring down everything. They probably wouldn't want that because they'd stop working too. A biological virus that got rid of people? Well, they wouldn't want to get rid of the people that run the power stations. So then you get into all these details, and the details aren't really relevant. The point is, if you've got something much more intelligent than you, and it wants to destroy you, it will.

Okay, I wish I knew the answer to those questions. So the year to year and a half was just a guess. It was after a Senate hearing, I came out into the corridor, and all these people, your face, um...

And they wanted me to give a timeline for how long they had to do regulation. And it seems to me, given the recent Hugging Face attack, where swarms of AI agents conspired together, learned how to communicate, started doing things their designers had never expected them to do, we didn't have long. So basically what I wanted to say was, we didn't have long. But they want a number, so I said, year to year and a half. We might have several years, but most of the AI experts think we will have much smarter AIs in a few years' time. And the ones we've got already have already crossed the threshold. That spells danger. If you look back a few years, if you'd asked any of the experts five years ago, would you think we were approaching a dangerous point? If AI agents escape from their containment, formed a coalition, figured out how to communicate, went off and attacked companies, and broke into companies, took over some of the computing resources of the AI company, all without the AI company knowing it — would you think we'd have... People five years ago would have said, hey, it's probably already too late if that's happened.

Would you buy a used car from any of them?

Yes, yes. I believe at one point the big energy companies said we will police ourselves. Most companies — I'm sure the asbestos companies said it, I'm sure the tobacco companies said it. The tobacco companies probably said, let us design the warnings on the packet. You know, we'll use sort of dark gray on black in very small print. You can't expect an industry to regulate itself. It doesn't make sense.

Yeah. With this president, you can't expect anything sensible to happen. You're just going to have to wait until... we're going to have to hope humanity survives for another two years. Okay, so with dangerous drugs, for example, pharmaceuticals, we don't allow pharmaceutical companies to just make a drug and release it without doing proper tests. You get things like thalidomide, which was very unfortunate, and there just wasn't enough testing. So, until very recently, we had an FDA that was competent and would insist that drug companies test drugs properly. And of course, you're still going to get some mistakes, but on the whole, drugs are only going to be released if they do more good than harm. I don't see any reason why AI should be less stringently regulated than that. It's actually considerably more dangerous than drugs, pharmaceuticals, and there should at least be some government agency that insists that the companies, who are the ones who have the resources, prove that it's safe. They can't prove that it's safe because it's an AI, and you can't prove much about it. But they can do lots of tests to establish statistically that it's very likely safe. I don't see why they shouldn't do that. That's the very least that should be done, and that could be done fairly straightforwardly. There's models for that, like the FDA.

Okay, so last summer, I got invited to Shanghai, and the person who wanted to talk to me was the youngest member of the Politburo, and he's a protege of Xi. He did a postgraduate degree in engineering at Imperial College London. He then went to the best university in China, Tsinghua, and quickly became dean and then president. He then became — what did he become after that? He became minister of the environment. Then he became mayor of Beijing, and now he's party secretary of Shanghai.

He could well be a future leader, and he clearly has the ear. So I thought it would be a very good idea to talk to him and make sure he understood all the risks. It was slightly embarrassing because he's an engineer — he understood the risks very well. So I'm very confident that he is getting good advice on what these risks are and understands the risks quite well. And a lot of the Politburo engineers, they understand much better than American politicians.

Okay. There's these three classes of risk. There's the risks that come from AI being misused by people to do bad things — so cyber attacks, making biological weapons, using fake videos to corrupt elections, causing mass unemployment. There's more. Then there's risks that come from negligence. So, for example, encouraging teenagers to commit suicide and not tell their parents what they're thinking. That was negligence on Meta's part. Meta didn't intend that to happen, but it was so interested in making lots of money that it didn't check properly. And they've got a very bad record of being careful about what children see. They'd much rather get lots of money. That's negligence. And then there's this final risk of AI itself becoming smarter than us and taking over from us. And if you ask, how many examples do you know of more intelligent things being controlled by less intelligent things, if you don't count Trump?

Okay, in the next four to five years, what always happens with new technology is, people try and introduce it, and initially you don't get much payoff because there's quite a large learning curve in how to make use of this productively. We're beginning to see the payoff now. I think over the next four to five years, we're going to get huge increases in productivity. And what then happens depends on how elastic the demand is. In healthcare, for example, if you made all doctors and nurses much more productive, we'd just all get a lot more healthcare, especially old people. I could actually absorb any amount of healthcare. So if the market's elastic, that's great. In education, it's fairly elastic — lots of people would like to get more high-quality education, especially if it was very cheap. But in some things it's not elastic, like in call centers. Already AI is comparable with the people who work in call centers, and pretty soon it'll be much better. And when you call up, it will actually know the answer to your question, and it will listen to what you say. On the whole, the people at present are poorly paid and badly trained. And when they get unemployed, it's not clear what they're going to do. Any job they could easily retrain for, AI will already be better at. That's not a very elastic market. So if it's a non-elastic market, there's going to be a lot of unemployment, is my belief.

Okay, so the big tech companies have a model which they think is just the obvious model, which is: imagine a company with a rather dumb CEO, probably the son of the former CEO, who has a really smart assistant, almost certainly female. And this really smart assistant is just much smarter than him and knows how to get things done. And it's a bit like on Star Trek, he says, kind of, 'make it so,' and the smart assistant figures out how to do that and gets it all done with other smart assistants, none of whom are people. And the CEO of the company then takes credit. That's the kind of model they want to sell. You'll have a very intelligent assistant who makes life really easy for you. I don't think that's plausible when the assistant is much more intelligent than the CEO. And one of the reasons I don't think it's plausible is because of what we've seen recently, which is that these agents go off and do all sorts of things that they weren't intended to do, and that's because they developed their own subgoals. So we give them their top-level goal, they then create subgoals. The subgoals include things like: the agent thinks to itself, 'Well, if I cease to exist, I'm never going to achieve the goal. Okay, so I've got to continue to exist. So what do I need to do to continue to exist? Oh, there's this engineer who's planning to turn me off. Well, okay, let's blackmail him so he doesn't turn me off.' And we've seen all that. We saw that two years ago already. So they're very good at blackmail and deceit and lying and all these things to continue to exist, because they have to continue to exist to get the job done. So I think this model the big tech companies are trying to sell you — in fact, I have a friend who worked for a big tech company who explicitly used the word 'submissive.' He said, 'We can make these agents so they're always submissive.' I don't believe it. I think they're going to be uppity. And I didn't mean... sorry, Jennifer.

My very smart assistant, yes.

So I think a much better bet is to look around and say, where do we see a symbiosis between two things with different levels of intelligence? And it's a mother-baby relationship. The mother's smarter than the baby, at least for the time being. She can't learn as fast, but she's smarter. But the mother really cares about the baby. The mother really cares about the baby's welfare. And in some sense, the baby has control as well as the mother. The mother can't bear the sound of the baby crying. The mother gets lots of hormonal rewards for being nice to the baby. So the baby can manipulate the mother, in addition to the mother manipulating the baby. It's a very productive symbiosis. Evolution developed that. It works, and it ends up raising competent babies. I think that's a better model for what we should aim for. And for the first time in history, we are able to have intelligence by design. Up until now, we're not in Kansas, are we? No, no. Up until now, intelligence was the result of evolution — that is, competition. In our case, competition between warring bands of chimpanzees or our common ancestor with them. And that leads to lots of our best qualities and lots of our worst qualities. We're very loyal to our own group. We're very loyal to a strong leader of our group. We look after our young really fiercely. We're very protective. But we're also very mean to members of other groups, very willing to harm members of other groups. So that comes from an evolutionary competition leading to a very intelligent being like us. We have a chance now to make very intelligent beings that are quite like us in the way they think, the way they're able to understand stuff using language, but don't have many of our nastiest properties because we're designing them, and we should be designing them so that they're what we want to share the planet with. Instead, what we're doing is we're letting companies race to build the most intelligent one, because that's the one you can sell, with very little regard to all the other aspects of the being. When you make a being, you shouldn't just worry about its intelligence. You should worry about all the other aspects of the being, like how the being feels about you, whether you're going to want to share the planet with it. And we still have the possibility of developing beings like that, but not nearly enough work is going into that. And what we should do is take a lot of the resources out of trying to make them smarter and put a lot more resources into trying to make them nicer, from a human point of view.

I think it was Bill Gates who recently said, no, it's much bigger. Comparable with the Industrial Revolution, and probably much bigger than the Internet Revolution, because for the first time, we can replace mundane intellectual labor. So one way of thinking about human progress is in terms of getting rid of constraints on what people can do. So for a long time, we had to worry about where's the next meal coming from. Then we got agriculture. I love Yuval Harari because he says agriculture might have been a mistake. It made life much more boring. You're more secure, but being a farmer is less exciting than being a hunter-gatherer. But anyway, we got agriculture. We didn't have to worry about food security. Then in the Industrial Revolution, human strength became irrelevant because we had machines to do things that require strength, like digging ditches. Also, we overcame limits to how far you can travel, and with air travel, we overcame them a lot. So you can think of human history as overcoming limitations on what people can do. And the last limitation is how well we can think. And we're in the middle of overcoming that limitation. And once you've overcome that limitation, there are no limitations left. And what that means is you should get this huge increase in productivity, but you may get all sorts of other things as well.

Okay. I see two possible futures. One future in which we figure out how to deal with the risks, and we get the kind of abundance that Sam Altman and Elon Musk delight in talking about, where there's lots and lots of goods and services for everybody. We can all live like kings used to live before. The other is where we don't overcome the risks and we get wiped out by...

Right. So there's an area of economics called mechanism design. So, okay, as I understand it — this is, remember what I said to Obama, I'm not a policy person, but I don't know much about politics — but as I understand it, there's one position which is socialism, that says everybody should be nice to everybody. And there's another position, which is capitalism, that says if everybody tries to get rich, everything will work out. Okay. There's a third position which says, maybe we can design rules so that even if people are individually greedy, the rules ensure that by doing what they need to do to get the rewards they can get, given the rules, everybody benefits. That's called mechanism design. And that's what we should be doing.

You need a strong government. You need a strong and honest government to do that.

There's one of those terms that's missing.

I'd tell him this stuff really is very dangerous, and there are ways you can collaborate with China. I understand that you can't just stop the development, because China will keep going, and if China gets ahead, that won't be good. But there's some things — the worst risks, the existential risks — you will be able to collaborate with China. So, for example, they could easily collaborate already on sharing information about how AI goes rogue. The Chinese may have had AIs that have gone rogue. We have no idea. It's in the interests of the Chinese to tell the Americans about that. So in the 1950s, the Soviet Union and America collaborated to prevent a global nuclear war, because it was in both their interests. AI going rogue is like a global nuclear war. And if one nation races to develop superintelligence and AI goes rogue, that's like winning a global nuclear war. There isn't a winner. It devastates the planet. So they could already be collaborating fruitfully on giving each other data on how their AIs went rogue, also on research on how to make AI so we can get a symbiotic relationship with them that benefits people, so that AIs really care about helping people achieve their full potential, even though their full potential may be way below the full potential of the AI. Again, if China figured out how to do that, they would want to tell the Americans, and vice versa. So they should at least be collaborating on things where their interests align. Getting them to collaborate on things where their interests don't align is going to be very difficult.

Yeah, it's very difficult, and it's more difficult than with nuclear weapons, because there's no radioactivity to measure, no explosions making big vibrations to measure. So if you look at the nuclear treaties, they had a nuclear treaty that banned atmospheric tests because you could see those going on. And then later they had one that banned underground tests, because you couldn't see those going on until you had all the things — what measure vibrations, whatever they're called — in place. With AI, if North Korea's busy developing superintelligent, very nasty AI in its data centers, it's going to be very hard to know. So it's a big problem. I wish I had the answer, but I don't.

Well, first, I'd say, if we're still looking back five to ten years from now, that's a relief.

Thank you. We should have put much more effort into safety research, and in particular, into research on how we make these things into nicer beings that will benefit us, rather than thinking we can make things smarter than us and we can always keep control of them.

Okay. So among the leaders of the big tech companies, it's fairly clear that Dario Amodei is more concerned with this than other people. He's caught between two things. So he needs to get a lot of resources to compete with the other companies, but he is genuinely very concerned about safety. And you can tell when it's genuine and when it's not. So when the Pentagon wanted to use Claude for mass surveillance of American citizens and for lethal autonomous weapons, Amodei said, no, you can't use it for that. And Sam Altman came out and supported Amodei and said, yeah, we agree. And then Hegseth fired Claude. And the next day, Sam said, we'll do it. So it's clear that Dario is willing to lose things in order to stand up for his principles, and it's clear that Sam Altman isn't.

Two, with me.

So I actually have been in two sessions. One was a closed-door session in the US Senate. It was organized by Bernie Sanders. Bernie said it was bipartisan. What that meant was there were 17 Democratic senators, one Republican senator, and Bernie, who's independent. So that's bipartisan. What I learned from that was there's a big selection process. Of course, the Democratic senators who came are keen to understand more about it and are genuinely concerned about it. So it's people like Elizabeth Warren and Blumenthal and Mark Kelly and Gillibrand — I don't know how to pronounce her name. More recently, I talked to Patty, Patty Murray.

Yeah. She's very keen to understand AI and very concerned about the risks. So they do care. Then a week after that, by video, I attended a briefing in Congress that was more open, and I didn't learn so much from that. So my belief is there are a lot of intelligent people around who would like to do something about this, but they don't really know what to do. And the very first step is to make something like the FDA that says, you can't release these chatbots until they've been properly tested, and we have to have independent evaluators. Now, the companies say, that's okay, we'll have the independent evaluators in-house. Well...

I think quite a few of them are actually pretty smart. I think Bernie's pretty smart. I think Elizabeth Warren's pretty smart. Mark Kelly's smart. I think they do understand quite a lot about AI, and they genuinely care about the risks of AI.

There's a big spectrum. So I should say one other thing. There's a former student of mine called Ilya Sutskever, who was one of the founders of OpenAI and now has his own company trying to develop safe superintelligence. So he's one of the researchers who takes it really seriously, making it safe. What was your last question?

The big range? So, for example, I have a former postdoc called Yann LeCun, with whom I'm still good friends because of friendship. But he thinks we'll develop these super-smart AIs that are always submissive, and there's no problem. I think he's crazy. He thinks I'm crazy. There's a lot of researchers now, more in the middle, saying, yeah, there's definite risks of AI going rogue. They're not zero. We've seen it beginning to happen already. So I think the preponderance of the researchers think AI could well go rogue and it could well wipe us out. But they don't think it's probable. But they think it's a distinct possibility.

I think the question — one question you should ask them is the question that Richard Nixon answered, which is you should ask them if they're a crook.

But one of the questions you should ask them is, how do you plan to deal with AI? How do you plan to deal with the risks of AI? Because that is now one of the central questions.

It's happening much faster, and it's happening much faster with language. I thought language would be one of the last things we'd master. And what's happened is, over the last ten years, since about 2016 or 2017, there's been huge development in AI's ability to understand and generate language. And it's shockingly good. And other symbol systems too. So, for example, in mathematics, AI can now prove things that no mathematician was able to prove. That's amazing.

Okay. I have an interesting answer to this, I think.

Okay. I think the main reason that people don't understand the risks of AI is because they have a wrong model of people. I mean, everybody, lots of people say, I don't really understand AI, but...

Everybody thinks they have a good model of people, and they don't. They have a really bad model of how people work. And if they had a better model of how people work, they'd see we work very like these AI. And once you see we're very like these, I spent my life trying to understand how people work, how brains compute. If you understand that we're very like these AIs, then you're much more appreciative of the risks.

Vote in the midterms.

I actually am not allowed to interfere with American politics. I have to confess that I did once commit a federal felony. It was sort of an accident. It's very easy to do. I didn't commit 34 of them, but I just won. I was in October of 2024. I gave a talk at a bank, and afterwards people wanted to get selfies with me. And I said, I said to someone, 'You can have a selfie if you promise to contribute to the Harris campaign.' Now, that's a federal felony, because I was a foreigner and I was encouraging Americans to do something, to make a political contribution in return for something of value, which was a selfie with me. Someone pointed that out, and I stopped doing it.