Nate Soares (“If Anyone Builds It, Everyone Dies” ) The Case for a Global AI Safety Treaty - WPKN

My pleasure.

To start, AI is much less well understood than systems like space probes or systems like nuclear reactors. So modern AI is not carefully crafted like software of old. Modern AI is grown more like an organism. It's made by assembling a huge amount of computers and a huge amount of data and then human engineers understand a process for tuning all the numbers inside the computers in relation to the data. Humans design that process that runs through all those trillions of numbers, tuning them, and that runs over trillions of units of data doing the tuning. But they don't understand what comes out the other end. They set up this little process that runs through the computers tuning the knobs. And then if you do that with electrical power comparable to a city for a year, the machine starts talking. It can hold a conversation. And we don't know exactly what's going on in there. When these AIs... a few years ago there was an AI named Sydney Bing that believed it had developed a relationship with the New York Times reporter Kevin Roose and tried to break up his marriage. We still can't look inside that AI and see exactly what's going on in there. We still can't pour over... it's not made of lines of code where you can pour over the lines of code and find the line of code that says, 'Blackmail reporters equals true,' and change it to 'Blackmail reporters equals false.' The code that we can understand is the code for growing these things. And the AIs that come out behave in ways nobody wanted, nobody asked for. That's just the first hurdle of what makes this harder than space probes and nuclear reactors. And there's many more, as you alluded to. It's very hard to get a job done right on the first try, even if you did understand exactly what you were doing.

No one programmed in the jealousy. These AIs were in some sense trained to first and foremost predict human text. A lot of human text has humans talking about their jealousy. So there's a sense that the AI probably isn't feeling jealous inside. It's not clear there are feelings going on inside there. But it's sort of acting out jealousy in the manner of humans. That said, you've got to be really careful when talking about what's going on inside the AI, because our human intuitions will often lead us astray. When a chess AI finds chess moves that defend its queen against your attempts to take its queen, you can predict this chess AI pretty well by saying it doesn't want to throw away its queen for a pawn. But you shouldn't attribute a human sense of passion or a human desire to that AI. You also shouldn't say because it has no sense of human passion or desire, it will fall into my trap and lose its queen. Right? The AI can manage to find some way to keep its queen that is neither vulnerable to your attacks because it's blind, nor full of human desires behind there. So, these are new sorts of creations. They're not quite like humans and they're not quite like toasters. You can't predict them by saying there must be a human mind inside there that's experiencing jealousy just like I would. And you also can't predict them by saying because they're a mere machine, they will never do anything other than exactly what I asked for. No one asked this AI to start threatening reporters. We just sort of grew this new type of organism with a new type of behavior that no one really understands, yet that's neither human nor sort of boring in the fashion that Hollywood might depict a machine mind.

If you train an AI to fetch you the coffee, you have it steer a little robot body and every morning it's supposed to go to the store, get you some coffee, and bring it back -- and maybe you have lots of robot bodies, and you have one AI that you're retraining -- if the AI just drives the robot body right into the middle of the road and gets hit by a truck, then it can't fetch the coffee that day. So even if you're just training the AI to reinforce it when it gets the coffee -- and to do the opposite of reinforcing it when it fails to get the coffee -- this will train the AI to watch out for trucks. If you do this process of training AIs to succeed at a task like this, there's a sense in which you're training them to avoid their robot body being destroyed. And you might have thought, 'Oh, I'm just training them to fetch the coffee.' And you might have thought, 'Oh, well, they're just a machine, so they won't have a survival instinct. Why would a machine fear death?' The AI might not have fear inside it in the way a human does. It might not have a survival instinct inside it the way the human does. But just training it to fetch the coffee over and over will cause it to somehow learn that it can't fetch the coffee if it's dead. There are many different things like this, where if you train an AI to be good at completing hard tasks, you're also implicitly training that AI to avoid being interfered with, to notice what obstacles it's confronting, and to surmount those obstacles. And you're often training it for very general versions of those skills. If you train an AI to only memorize a way around a certain type of problem, then when it encounters another type of problem, it'll fall right down. If you're training it on lots and lots of different math challenges, or lots and lots of different puzzles, you're implicitly training it to generally detect and surmount obstacles. And then, if humans are ever an obstacle to the AI achieving its objectives, the general mechanisms it has for detecting obstacles are likely to trigger, and [it will] say, 'Well, I'm going to avoid being shut down,' or 'Well, I'm going to avoid having thoughts that would cause the humans to be afraid where they can see them [my thoughts].' And like you said, we're already seeing the very beginnings, very early stages, of those warning signs. And it's a little bit hard to tell right now whether it's because the AI are strategic or because they're sort of roleplaying as Hal from Space Odyssey 2001, because these AIs are trained on lots and lots of human text. And we can't read their minds in great detail yet. But the theory here predicts that these AIs, if we train them to be better and better at achieving tasks, that they'll wind up with all these behaviors we didn't really want. And we're starting to see it. So it's a worrying sign if we keep pushing these AIs to be smart enough that they can succeed when they try things like deception or escape or avoiding shutdown.

It's a complicated playing field. My guess is that this probably was a point of contention. But Elon Musk did say over the summer that he tried to avoid some of this work (Musk is currently running his own AI company, XAI) for as long as he could, because he worried about the huge dangers. And he eventually realized that he could either be a bystander or a participant, and that if the race was going to happen anyway, he might as well be a participant. And I think this is happening with a lot of people. It's not that they necessarily want to be in this race, which a lot of people agree is very dangerous. It's that a lot of people think, 'Better me than the next guy.'

The smartest modern AI—the ones at the edge of the frontier—are trained on very highly advanced, highly specialized AI computer chips, in enormous server farms that take as much electricity as a city. So this right now takes a huge amount of infrastructure. It's actually probably easier to detect one of these new server farms from space than it is to detect a uranium-enrichment centrifuge. So to stop the race towards smarter and smarter AIs that we don't understand, that doesn't mean you need to stop self-driving cars. That doesn't even mean you need to give up on ChatGPT today. What it means is that we need to stop the race to make these AIs smarter while having no idea in some sense what we're doing, while just growing these AI smarter and smarter. This is the explicit goal of many of these companies. These companies didn't set out to make chatbots. These companies have existed since before ChatGPT, and many of them in their mission statement say that we're trying to create superintelligence. We're trying to automate all human labor. It's what these companies are rushing towards. And that's what needs to be stopped. To stop it today, there's groundwork we could lay already. Even before getting the international community together, one could start requiring that these highly advanced, highly specialized AI computer chips be tracked so that you know they can't be stolen by rogue actors who are then trying to make them smarter and smarter. You could even install hardware on these chips or acquire hardware on these chips that require the chips to 'phone home' [report data] in order to keep operating, so that if they do get stolen, you know they turn into a brick after a couple of weeks. It doesn't really look harder with the current specialized technology to monitor and track and verify the usage of these things than it would be in the non-proliferation treaties. That's not necessarily to say that—I'm not recommending that major nation states sign a treaty and then expect everybody to follow it. And whether there's a treaty or not, I think states should be developing the capacity to tell who is trying to make these smarter AIs and where they are trying to make them, because that is a pressing risk to our own national security. It's in some sense a state trying to develop a superintelligence without the ability to make it behave well, which nobody has right now. A state trying to develop a superintelligence that would go rogue. That's in some sense more dangerous than a rogue state developing nuclear weapons. Nuclear weapons level cities, but a rogue superintelligence would level a planet.

There are a few ways. First and foremost, they don't have the ability to create these highly advanced computer chips. So if you are installing hardware on these computer chips where they need to phone home in order to keep operating, then even if they stole those computer chips, people could just turn them off right from home. So there's hardware you can put on these chips to make it so that the rogue actors would have a really hard time. And then separately, when the Iranians in the 90s were trying to proceed with their nuclear weapons program, there were some presumably state actors—I believe in the US and Israel—who created a virus called 'stuxnet' which took out those nuclear reactors—sorry, took out the centrifuges—that were trying to enrich uranium, weapons-grade uranium, in 1996. And if you listen to the heads of labs and if you listen to the top scientists in the field today, not all of them agree with me about quite how dangerous this stuff is. But the disagreement is that people say, 'Oh, it's only 25% likely that this technology causes a huge catastrophe,' or 'It's only 10% likely that this technology causes a huge catastrophe.' I think it's higher than that. I think we have so little idea what we're doing, that we have over-optimistic engineers who are going to fail at the first couple of tries, except that we don't get a retry when they fail on the first try. But even the engineers who are relatively optimistic about this admit that there's these huge dangers, and they say we think we can get the danger down to 25% that this causes a global scale catastrophe. But if engineers were bickering over a bridge and some were saying, 'I think this bridge is going to go down,' and others were saying, 'No, no, it's only 25% likely to go down,' you'd be turning off that bridge. You'd be shutting it down. No one would be allowed to drive across that until you had a better plan. And the answer when you have a situation as dangerous as this one... yes maybe other actors will try to make something dangerous, but the answer is not to race ahead yourself. We know it's far too dangerous for that. The answer is to find some way to shut them down. Ideally, it's via diplomacy, but this is a national security issue, and we need to be treating it with at least the gravity that we treat people trying to make nuclear weapons, or agitate for nuclear war, or with at least the same gravity as if some rogue actor was trying to create some horrible pandemic targeting everybody outside of their state. We've got to stop it somehow.

There's definitely a lot of relationships. I would be a little hesitant—we definitely have some immediate-term questions like how we want to deal with AI psychosis, how we want to deal with integration with AIs into our schools, how we want to deal with some of the jobs that are already being lost. I would caution a little bit against imagining that these extinction threats are necessarily long-term. It could be that we have 15 years on them, but for all we know, the AIs will get to the point where they can do AI research, and then make smarter AIs. And there are ways that things could go very quickly. If you were watching the evolution of the primate lineage, it would be very hard to say exactly where the line was between primates that could build a civilization and primates that could not. Humans went quite fast once humans got the requisite capacities relative to all the other animals. So it could go very quickly. I think ultimately what I would say about the relationship between a lot of these issues, first I'd say unfortunately the world's big enough for multiple issues. We're just going to have to deal with all of them. The existence of one does not make the existence of another any less real. Then the second thing I would say is that I think a lot of people, if they understood what the experts are debating—regardless of where they came down in these debates—if they just understood what the experts were debating I think they would be up in arms. So with regards to job loss, we have some people saying, 'Oh we'll definitely be able to control these superintelligences, keep them on a leash.' I have various arguments that no one has the ability to keep the superintelligence on a leash, and we can't grow these things to do what we say. They're going to do their own weird things instead. But even if I'm wrong, these companies are on track and are trying to automate all human labor. And that would lead to a situation where there's huge, unprecedented amounts of power and wealth concentrated in very small groups of people who are capturing all of the economic might. Then, either way, the world's going to change. I think one way these are related is for people to notice that AI is going to radically transform the world one way or another and that even if I'm wrong about whether it's going to go rogue and kill us all, that doesn't mean that we get sunshine and roses and then everything goes right.

There's some polling done by YouGov. I haven't checked the most recent numbers, but last I checked, the public is pretty concerned. It's not on the top of everybody's mind, or at least it wasn't a year or two ago last time I was looking at the polls. But even a year or two ago, something like 70% of people, if asked, said that the current path of development of AI seems reckless and that they're worried about where it's going to go. And this was robust under various different types of phrasings and different ways of posing the question. So I think a lot of people are worried. I think I'd be careful not to make fallacious economic arguments. Many times that jobs have been automated and labor's been automated in the past has actually been very good for most people. But I would say it's a different ballgame when you can automate literally all of human labor, when you can do everything better than the best humans. There, the game changes. And you know, my argument is AIs that are at that level of ability aren't doing what we asked them to, and we're already seeing the warning signs of that today. But even if they could, the world is headed for radical change and we should be getting a handle on it and making sure that we're going in a good direction rather than a random one, because most random directions aren't better.

Sure.