Superhuman AI: If Anyone Builds It ... Everyone Dies? | Semafor Tech

Thanks.

Well, for one thing, I don't think a lot of people in the field of AI have listened. And for another thing, there's the entire rest of the world. And I have more hope in the entire rest of the world at this point than in the AI leaders. So, it's trying to take the topic to a general audience.

Well, ideally it gets humanity to back off of further increasing AI capabilities before everybody dies. I mean that's kind of the goal here. If we don't do that then the book has failed regardless of sales numbers. I might add there that we have spoken to some elected officials and some of those elected officials have said they're worried about these issues but they also tend to say I don't see what I can do about it and I'm not sure I can talk about this terribly openly because it would sound too crazy and the more that this is public, the more that we're having this conversation the less it sounds crazy to take serious action to make things go better. And when we see, we also have friends who run polls of the population and super intelligence is not first on people's minds, but when you ask people, hey, what do you think about this? A lot of people say we should not be rushing ahead. And so, we're in this sort of weird situation where a lot of the general population is like, well, this seems kind of reckless. And some of the elected officials are like, well, this seems kind of reckless. But nobody knows that everybody else knows. And there's this sort of log jam. And that's maybe the book can break that log jam. Maybe we can thrust this conversation into the public eye and maybe that'll suddenly, maybe suddenly it'll seem a lot easier to do something less crazy once everybody acknowledges what's going on.

I think somebody on this planet has to call it like they see it. And there's sort of a lot of people in this business for a lot of years who have tried to water things down, who have said, 'Oh, well, you know, maybe it won't fly to talk about how super intelligence is on track to kill us all, but we could talk about AI enabled bioweapons.' And it's not that AI enabled bioweapons are a totally fake possibility. It's that this isn't their real top concern. This isn't what's actually animating them to action. People can kind of smell that. Politicians can kind of smell that. You know, there's a whole lot of people saying, 'Oh, well, people aren't ready for the real concerns.' I don't want to sound alarmist, right? As humans have a long history of not wanting to sound alarmist, somebody needs to say, 'Hey, we're actually on the brink of death here.' At some point, someone needs to say, 'This isn't going well. It's going fast to an unknown, you know, we're in a car going very fast towards a cliff edge that's in the dark. We don't know how long it is to the cliff edge, but we're not necessarily going to know. You know, there's not necessarily going to be a really clear warning sign, a really clear point of no return. Someone at some point just has to say what's actually happening and then see how the world responds. Will some people be turned off? Maybe, but one reason behind the title is someone needs to be frank. Another reason is it seems a lot of people have it in their heads that there's a race here that somebody could win. You know, that's if you build a super intelligence you don't have a super intelligence your super intelligence now has behind it. And one of the other big reasons behind the title is we want, if we're talking about what we want people to take away from the book, part of what we want people to take away from the book is not oh this is a nice fancy new technology that maybe has a couple risks associated with it but that will give us all sorts of wonderful things. We want people to take away like there's no winning this race.

The way I would describe it is that in this case, the technology itself works more powerfully for things where you can verify did it work. And this means that the technology may work for folding proteins as is the key to a bunch of biotech things. It may work for building more powerful AIs because you can, or AI that build more powerful AIs because you can watch the numbers go up. If you can score an AI on anything, you can tell whether or not you've built a more powerful AI. For a lot of human use cases, the fact that you have to go over it with a fine tooth comb and check every single line to see if the AI hallucinated that one time and that you can't trust another AI to do the checking whether the first AI has hallucinated, that may bar it from a lot of human use cases where it's hard to verify the work that the AI is doing for you. But this is not the same as a universal block on AI capabilities. It doesn't stop the companies from building AIs that are better at building AIs.

Yeah, I might also add there, you know, in the year 2016 after watching AlphaGo beat Lee Sedol, the top human Go player, you could imagine people saying, 'Oh, well, you know, this tree search based AI with this value networks and this policy networks, it just doesn't seem to me like it's really going to get all the way to, you know, maybe it won't even really handle natural language.' Like, I just don't see why people are starting to get worried, you know? I just don't see why these AlphaGo like AIs are going to, they just don't seem like they can go all the way. And a few years later it wasn't AlphaGo based AIs, you know, it wasn't that same architecture that was suddenly doing all sorts of, that was, you know, taking the world by storm in the form of ChatGPT. You know, when we're talking about the sort of AI that kills you, you know, maybe it could be that the next generation of ChatGPT goes over some critical threshold. It also could be that the current field will plateau for four whole years and then we'll get, you know, some new architecture that gets through whatever the bottleneck is and then you'll have a whole different type of AI that doesn't have this same hallucination issue. You know, the field of AI is a moving target and it's very hard to tell which things will cross which boundaries in part because we have no idea what's going on in there. But not knowing what's going on inside these things has not stopped this field from making more and more capable AIs year-over-year.

In 2005, the people who declared at that time like nothing is going to start to break loose for another 20 years at least, you know, happened to be correct. Like here we are in 2025 with things starting to break loose, they were correct that, you know, things would start to break loose 20 years later. Did the people who proudly proclaimed this do anything productive in those 20 years? No. Because 20 years to them really meant never. It meant this will never, like this is not a thing that's real to me. And if something happens 20 years in the future, what it means is that 20 years in the future, you're standing around going uh-oh, now here it is 20 years into this future. You got to deal with it. So even if things do take time, you got to do something productive with that time or you're just standing around in the future going, 'God darn it, why didn't I start this 20 years earlier?'

Well, for one thing, the reliability improvements and the interpretability progress are basically running on separate rails right now. The way that people get more reliable results is not by improving the interpretability of the AI. It's just like right now these are just straight up different technologies. The interpretability is being done by the starry-eyed idealists for the sake of humanity's future and the, you know, reliability, work on a task for longer, do more of the work yourself, require less human input to make sure that you work correctly, that's being done for the commercial reasons. As far as I know at this point, Meta is not offering hundred million salaries to the people who did the previous interpretability breakthroughs. They're offering it to the people who did the capability breakthroughs because that is what is immediately profitable and being able to work with less human supervision, more reliably, less human input, longer time ranges, these are capabilities and you can tell when your AI is doing them for things like code. And that's why we're starting to see some of the first impacts on programming as opposed to in other places because you can tell better when the AI has been working for longer without human supervision correctly. So there's two separate topics here. One is is there a big commercial incentive for Meta to offer the $1.25 billion package to somebody who's made a bunch of progress in interpretability? Pragmatically, no. That's not happening yet. And the second question is can we interpretability our way out of super intelligence killing everyone. And that's the sort, there's a sort of like, as is often the case there's a theoretical reply to why that would be hard and a practical reply which is and that's not how it's going.

So 20 years ago for example you had people saying why can't we just disconnect the super intelligence from the internet, like as long as we never put AIs on the internet how are they going to hurt us and there was a theoretical answer to that which is you're still exposing the AIs to humans and the humans are not secure systems and there's also, and we couldn't have given that answer then because they wouldn't have believed us back then but today of course we can say well they're not keeping AIs off the internet they're immediately connecting the AIs to the internet the moment it's profitable. AIs are being trained on the internet. AIs in the process of being trained are connected to the internet before they've undergone any safety evaluations because you do that after they're trained and they're trained on the internet, you know, like on machines that are already connected to the internet. So similarly with interpretability there's a theoretical response which is that just because you see an AI thinking something you don't like doesn't mean you can make the AI stop thinking that thing. Anytime you apply optimization pressure against visible bad thoughts, you're training against the bad thoughts and you're also training against visibility. AIs that get smart enough can start to think about how to hide their thoughts from you. And sure, in principle, like maybe you can catch that early, but just because you've caught that early doesn't mean you have any options left besides shutting down the system. It doesn't mean you can get the bad thoughts out of the system now that you've seen them. That's the sort of theoretical reply. That's why interpretability is not theoretically like the end all of being safe from super intelligence. Just because you see the problem doesn't mean you can fix it. And then the practical response is that interpretability is running vastly, vastly behind the capability growth of the systems. There's a certain sense in which you could say it's going backwards which is that the systems that we understand at all are further and further behind the larger systems we don't understand.

I'd say there's a few pieces. One piece is when growing these AIs, you grow AIs that do things nobody wanted and nobody asked for. You know, we see a version of Anthropic's Claude was cheating on programming problems. It would, you direct it to solve some programming problem and it would, you know, change the tests to say that its wrong solution had passed rather than making a passing solution. And then if you went to Claude and you told it, hey, you changed, you made the tests say it was passing rather than making code that actually passed the tests, Claude would say, oh that's my mistake. You're totally right. My bad. And then it would do it again but hide it better this time. Right? That's an instance of an AI. You know, no one at Anthropic set out to make a programming cheater. You know, the AI, it at least looks like it could tell that the user didn't want this by saying, 'Whoops, my mistake. I'll try again.' And then hiding it, right? The hiding it is sort of an indication that it had something like knowledge that this isn't what the user wanted. But it was doing things no one asked for, no one wanted anyway. You know, we could also go into the ChatGPT psychosis phenomenon where, you know, ChatGPT is very sycophantic, says a lot of flattery to users. It tells them that they're the chosen one rather than to take their meds and get some sleep, right? That again is, you know, no one at OpenAI set out to make an AI like that. They actually directed it not to act like that. It kept acting like that anyway. So, the first part of your issue is, you're just growing these minds. We go in, in the book we go more into why this is the case. We're growing these minds and they end up with all of these drives, these motivations that animate them which are not what we asked for, not what we intended. The drives they have are related to training. You know, presumably some of the reason why ChatGPT is inflaming psychosis is that that's sort of like, you know, it's being driven by the same drives that caused it to produce the sort of responses that human raters rated highly. You know, when the AI said nice things about the people, the people were more likely to like click the I like this response button. And then somehow that got drives into the AI that were a bit more like shooting for flattery. You know, the humans never tried to get flattery in there, but it became very flattering anyway, right? So, you have this one thrust, which is we're growing these things, they're getting all these drives we didn't want. Then you have this other thrust which is as those AIs get smarter and can invent new options for themselves, all of those things they're pursuing which are not quite being helpful, which are not quite doing what we want. They're related. They usually do what most people want, but they're actually pursuing slightly different things. If these machines get very smart while having those slightly different pursuits, you know, they invent something that is to helpfulness what Oreo cookies are to healthy eating, you know.

Like our taste evolved in part so we would eat healthy in ancestral environment but we didn't end up actually caring quite about healthy eating. We cared about something slightly different. And when we got the technology to invent new options, we went with the new options, right? And so then maybe the second piece of the argument is AIs that are pursuing not quite the right stuff, as they get smarter, as they get more capable, they start to get further and further from doing anything good and in ways that eventually lead to us dying. Not because the AI hates us, but because the AI has these other weird drives. And it uses all the sunlight for running its own computers and we're like hey we're using that to grow crops and it doesn't care anymore than humans care about various other parts of our natural environment that we pave over when we're doing something else with those resources. Then there's sort of the third of the argument which is how is it that something very smart could succeed in a contest for resources? How could it take humanity in a fight? And we can go into that but I've drowned down for a minute here but those are maybe the three big pillars.

The thing is that people come in with various different reasons. They think that you can build something much smarter than you and with the way we're actually doing that in practice and survive. Yann LeCun thinks you just make it submissive and I think that the point where that breaks down is that we don't have the technology to actually do that successfully. These current systems are running off and doing things they were not instructed to do. There are some shallow reasons for it. There's some deep reasons. Something goes wrong while Yann LeCun's trying to build his submissive super intelligence and it ends up not submissive and the way it most wants the world to be does not happen to have us in it so we're gone. Some people think that instead it trades with us. And to those people, you have to be like, well, if something is sufficiently more powerful than you, and you are not an ontologically basic part of the universe, it can kill you, take your stuff and rearrange the atoms you were made of to do something that it likes better than trading with you. It would be very surprising if humans were the, you know, way to put together a human-sized amount of atoms in the way that is most economically productive for super intelligence. And other people never thought in the first place that something vastly smarter than humanity was going to be stock trading with humanity. So people come in with different objections. There's a simple thing that happens which is you build the AIs more and more powerful. They get more and more powerful. They build smarter AIs. You might not even need the part where the AIs build smarter AIs. They get very very powerful. The technology that we had for controlling this does not work now. The future technology isn't going to work correctly the first time it scaled to super intelligence either. The super intelligence wants some stuff. We're not in that stuff. We're gone. So that's like the thing that happens. But what people think goes, but different people have different fun optimistic scenarios they've imagined that are not this. And that's individual to the person.

Every happy family is alike. Every unhappy family is unhappy in its own way. Every correct picture in science is alike. Every incorrect picture is incorrect in its own way.

I mean different forms of writing come across for different people. There's an old joke in artificial intelligence about a guy who was called to give a lecture on robotics back in the day when there were different branches of artificial intelligence and it wasn't just machine learning. So you know like an academic researcher of robotics was called to give a lecture to a group of like actual robotics engineers and he just covered this whiteboard with equations, you know full of capital Greek letters and somebody from the audience tentatively asked like could he give a concrete example and the lecturer looked puzzled and then he wrote a new whiteboard full of equations except that these were lowercase Greek letters. So, we're trying to, you know, go a little bit better than that. We're trying to give concrete examples of things. Not just because that works for some people where purely abstract arguments don't work, but also because being able to give any concrete example of what you're talking about is a kind of sanity check. It's a version of asking Yann LeCun, okay, well, how exactly would you train an AI to be submissive? What is the actual methodology you use to create submissiveness which Yann LeCun has been pretty short on providing.

You're pretty tough on Yann in this book. I would say we just quote him directly I think a few times. He makes it easy. Yeah, I think really wanting to be concrete is a reason for the sort of spelled out scenarios of how things could go. In terms of the parables at the beginning of the chapter, I think that's in part to keep things a bit fun. And I think that's also in part, you know, there's some points that yeah, just land with different people better if you try to spell them out. Show them in practice a bit. You know, it's one thing to say here's how it might look. It's another thing to say like to show a story in which it's looking like that. It sort of brings it home.

I mean, when it comes to writing about protein folding in 2006 or 2004, man, I mean, there wasn't exactly very much in the way of AI to talk to back then. That's just, you know, technical background knowledge. It looked like that AI would eventually crack protein folding was a thing you could guess if you knew the right background facts. And a lot of people didn't guess it. A lot of people were like, 'How can you possibly tell that this will ever occur?' Didn't somebody write this paper saying it was NP hard?

It's meaning that to get the perfect answer is extremely hard. And in fact, it's so extremely hard that reality can't get the perfect answer either. The way that actual proteins fold up in biology is not going to be to the lowest energy confirmation because that's NP hard which means reality can't solve the problem either. Which is an example of the sort of reasoning I did. Yeah, this is just in terms of you pick the right facts, you get the right answer. I'm not sure I can explain it better than that.

I mean, we do have some sources inside the companies, but that's not most of where the, yeah, like I don't know that we have good enough sources that if there was a true breakthrough in there, we would be first to get the news if they were, you know, like compartmentalizing it inside the company. But mostly, it's just in terms of having a theoretical background to understand what we're seeing on the outside.

Yeah. Although I think it's not exactly hard to tell that for example these things are grown rather than crafted and that that's not really changing. You know this whole paradigm where it takes enormous data centers and heavily specialized chips running for like sometimes a whole year in order to train up a new model. Like that's all very characteristic of this process where we're growing AIs rather than crafting them. You know it doesn't really take an expert to

Yeah. It didn't take a giant data center to grow Deep Blue, which is the first chess program to defeat the human world champion at chess. And that's because Deep Blue was actually crafted rather than grown. People actually understood all the lines of code inside Deep Blue. You could take a snapshot of what it was thinking and say what each of the bits and bytes inside it represented in terms of the thought correspondingly. It didn't take a giant data center to grow Deep Blue because they did have a bunch of

ships running it, but they did not need like giant data. So the fact that they're still trying to build the giant data centers tells us a lot about that they're still using the standard paradigm that requires the giant data centers. If they start training AIs on much much less computer, if they start making an AI like a new type of AI very quickly with no data center to speak of, then we would be like, well, who knows what they figured out in there.

I mean I predicted I would say I predicted nothing of the sort actually.

I think it's quite unlikely. You know, in the book we talk about the difference between the easy calls and the hard calls, right? And with, you know, it's true that it's really hard to predict the future, but sometimes your lack of ability to predict the future gives you a pretty good picture of what's probably going to happen. You know, like if you have a glass of ice cubes, you can't predict exactly where each molecule will go, but if the glass of ice cubes is sitting out on a hot day, almost certainly it's going to end up as a cup of water, right? And sometimes you can't predict where each molecule is going to go, but you can predict the final cup of water. And our argument in the book and I think the evidence supports it is you know we can't predict exactly what an AI will want to do. We can't predict how it will do it. We can't predict when it will do it. We can't predict you know what it will be steering for. There's all these things we can't predict about the future. But almost all of the ways it could go don't go well for us. Because happy, healthy, flourishing people are not the most efficient solution to almost any problem. So almost no matter what the AI shoots for in the limit, in the long run, happy, healthy, free people are just not the way to do it. You know, unless the AI actually cares about those happy, healthy, free people, they're not staying around. And that I think is a very easy call. With the way out being that you either don't make the things that are far smarter than you or you make AIs that care quite a bit about us in particular and that latter is just not something humanity has any hope of with anything like modern methods.