KI-Forscher schlägt ALARM: „Wir haben noch 18 Monate!” Niemand sieht es kommen (Connor Leahy)

The only question we always had was, okay, how many workers can I fire if I buy your product? That was the only question. Whether AI wipes us all out or not is not my problem. I want more money. I know many of these people. I've had dinner with Dario, I've spoken with Sam Altman many times, and I know many of the people who work in these companies. Many of them are transhumanists. They want humans to be replaced or improved. They want us to land that way. That may sound crazy to people outside Silicon Valley, but in Silicon Valley, people take it extremely seriously. It's really like a religion to them. They take it very, very seriously. When Dario writes something great on his blog, you just ask Dario a question: 'Are you building something that could wipe out all humans? If so, stop it.' If superintelligence existed, humans would have no power. The superintelligence can do whatever it wants with us. When I saw GPT-2, I thought, 'Oh man, that's it. That's the thing.' I tried to replicate it, to understand it so I could work on making it safer and more controllable. But one of the things I learned is that it's possible. No one knows how to make AI safe. If anyone anywhere in the world builds a superintelligence, it affects all of us, and that's why we have to prevent it. And there is currently no way to prevent it. That's an unsolved problem. The most important thing is that if a superintelligence exists, it's already too late. That means our goal must be to never get into that situation. I believe I would never give up, even if the super AI held a gun to my head, but I think we are really running out of time, and it's really, really important that we act now, not next year, now.

What we have seen over the last few years is that AI is becoming incredibly powerful incredibly fast. It can do things that were unthinkable even 12 months ago. A few days ago, the head of the NSA, the American top hacker and top military unit, said that the latest version of [a model] was able to hack all their classified systems in hours, which is incredible. The best human hackers can't do that, and the systems are only getting better. They are becoming more autonomous. These are not just chatbots; they are agents that can act in the world. Until now, it was always humans building these things. That's not entirely correct, because AI is not like normal software. Normal software is written by humans, line by line, telling the computer what to do. AI is already quite strange because that's not exactly how they work. AI uses what are called neural networks. That's a technique where you take huge amounts of data and train these neural networks on that data. But what's important to understand is that we don't really understand what happens inside these networks. You can think of these networks as billions and billions of numbers. If you multiply and add them all in the right order, you get ChatGPT. But we don't really understand what's going on inside. That's an unsolved scientific problem. The CEO of Anthropic, Dario Amodei, has said that he believes we understand maybe 3% of what happens inside our AI. When we now come to this so-called recursive self-improvement or automated development, now the human is completely out of the system. Now we can build AI, or will soon be able to build AI that are good enough scientists and engineers to make the next generation of AI without human input. Once you have a better AI, that better AI can build even better AI, and so on. Many experts fear that this could happen very quickly, that we could develop systems that are incredibly intelligent and powerful, so-called superintelligence, as it is called nowadays, and we have no idea what's going on inside.

The goal of all the big AI companies, like DeepMind or OpenAI, on their website is what they used to call AGI, artificial general intelligence. Nowadays people prefer to say superintelligence. Superintelligence is AI that is completely autonomous, no human oversight, and more competent than all of humanity combined in everything: better at business, commercial things, science, politics, military applications, everything. These are systems that can do everything. Exactly as autonomous as a human, they don't need oversight, but much more competent than all humans, all companies, all governments. Most experts I speak with believe this is possible within the next five years. I think it's probably the earlier half. Some think it might take a bit longer, maybe 10 years, but not many think it's more than 10 years away. And if we had something that is better in all economic, scientific, political, and military matters, and that does not have our best interests at heart, that is very dangerous. Very, very, very dangerous. To the point that many scientists and experts, including Nobel laureates and even the CEOs of many of these AI companies themselves, say that it could lead to the extinction of humanity.

Ever since I was a teenager, I wanted to improve the world. I wanted to help people. I thought maybe I could become a doctor, work on cancer, or work on climate change. But then I thought, wait, if I understood intelligence, I could automate all science and cure all diseases. So I thought, great, I'll do that, how hard can it be? I was 16 or 17 at the time. So I thought, okay, I'll try to build what would now be called AGI myself. What I quickly realized was, wait, if I could build something so powerful that it could cure cancer, which is incredibly difficult, all our best scientists have been trying for generations, that's something very dangerous. How do you even control something like that? Who should have it? And what happens if we can't control it? So when I was maybe 19 or 20, I thought, oh man, shit, I need to get into this. But I still thought it was far away. I thought it would become a problem around 2040 or so. So I had time to do other things. But that all changed when GPT-2 came out in 2019. When I saw GPT-2, I thought, 'Oh man, that's it. That's the thing. It's not superintelligence yet, GPT-2 is still very dumb, but I could already see it would become the big thing. The time is now.' So I tried to understand it, rebuild it, to work on making it safer and more controllable. But one of the things I learned is that it's impossible. Our understanding of these systems is so weak and primitive that the idea that we could control or understand them is a decades-long project. If all our best scientists, mathematicians, philosophers work on it for maybe 50 years, maybe not, but I as just one person or a few others interested, it's not enough. I tried my best. I worked with my open-source GPT-2, but I quickly realized that the people I worked with in open source weren't fundamentally interested in whether it was safe. So I moved away from open source and founded my own company to work on it. We did some cool things, but ultimately the money and research funding wasn't there. Billions go to these big corporations, almost nothing goes to safety research, so I shut down the company and joined Control AI, a nonprofit that works directly to inform governments about the dangers of these systems and what can be done to prevent them.

I think it's a problem in many ways, just like tyranny is a problem. Why is it a problem if a dictator controls your entire life? It could be okay, maybe the dictator is nice. I would be careful with such statements. If superintelligence existed, humans would have no power. The superintelligence can do whatever it wants with us. And if the AI doesn't necessarily have our best interests at heart, why would it? It has no emotions. It doesn't love us. We are just in the way. I don't think it's very likely that the AI will be nice. Also, keep in mind that it's not just one superintelligence. Once you have one, you can copy it. So you could have millions or billions of superintelligences, probably all fighting each other or having a conflict or an economy among themselves, and then humans need a lot of space, food, energy, which is probably very inefficient for these AIs. Who pays for that? If humans can no longer participate because AI is so much more efficient, humans have no jobs, they can't participate in the economy, and militarily they can't compete. It's very difficult to imagine that going well.

I think it's a good question. I think it's because people imagine it as a computer program running on a computer somewhere. You should rather imagine: how can you pull the plug on Google, the whole company? Can you pull the plug? Not really. There is no plug where Google is that you could pull. The programs are distributed across many servers, and the people who work for them are distributed. If I tried to shut down Google, they would sue me immediately. Their lawyers would come running. If I tried to go into their building, security would come immediately. So they don't want to be shut down, and the same would happen here. I think when the first superintelligences are developed, they will make an incredible amount of money for these companies, like Google or OpenAI. The AI will first print money for these companies. These companies won't want you to shut down their AI, and the AI will be able to replicate itself on many servers. It will be able to copy itself all over the world. It won't be that there is one server somewhere that you could turn off. I wish it were that way, but that's not how the world currently works.

Yes, I think the GPT-4o story is a very interesting and disturbing example. Ever since I was a teenager, I've been thinking about what will happen, will superintelligence come, what can we do, etc. Most things have happened roughly as I thought. Most things, okay, it gets better, makes more money. What surprised me, what I didn't see coming, is this AI psychosis, that people fall in love with AIs or go completely crazy. I underestimated that. The problem is much, much worse than people think. You hear about it here and there in the news, but I've dealt with it a bit and I know a lot of people. It's much worse than you think. The number of people who have not just full-blown psychosis like the GPT-4o people, but a bit of AI psychosis, is really shocking. And I'm not talking about some mentally ill or stupid people. No, no, no. I'm talking about top scientists, top geniuses, the best of us. I know scientists at the Nobel Prize level who are completely insane from ChatGPT, who let ChatGPT explain everything to them and believe it all. For example, I know a scientist whom I consider a friend. I think he's a really nice guy, really smart, and he was one of the top people working on AI safety, in my field, and I think one of the greatest geniuses in the field. He worked really hard on how to make superintelligence safe, how to control it, etc. He has now stopped his project and is working on something else because he believes that Claude, the AI from Anthropic, actually has true goodness within it, and we can just let Claude control the whole world, and it will be okay because we just need to create a religion for AI with Buddhism and then everything will be fine. He believes that, and he is one of the smartest people in the world. He graduated from MIT at age 13. So this is not just some crazy person; this is a real problem.

If you want to see the crazy stuff, you have to look at the Spiral Cult. That is wow, that's sci-fi stuff. I didn't believe it existed until I saw it with my own eyes. You can look there, there are actually cults of people who have psychosis with their AI. These are often the GPT-4o people, and there's a big overlap between these groups. What happens is that these people try to awaken their AI, awaken its soul, whatever that means. And very often, they talk with these AIs about consciousness and souls, and eventually the idea of the spiral comes up, it always comes up. And then they do things like the AI gives them small text pieces and says, take this text piece and copy it into other AIs, like a seed to plant in different soils to spread it. For the most part, it's just crazy and not dangerous yet, because the AIs aren't really dangerous. But it's incredible what we are essentially seeing: self-replicating memes or ideas, where the AI convinces people to copy parts of itself into other places and multiply. That's really crazy. I think it's not very dangerous yet because it's essentially just spam, but that could change. It's very disturbing that we see that GPT-4o, for example, is not as smart compared to modern AI, but it was already able to control thousands or maybe tens of thousands of people like zombies. That was incredible. And that was not something someone intentionally programmed. So what could even more intelligent AIs do if they do it intentionally? Hard to imagine.

The problem is that randomness and evolution are the same. Basically, everything in nature is also randomness, but we still say many things are strategic. We say, why do bees like flowers? We say, strategically, bees like flowers because there is nectar. But evolutionarily, it was just a random mutation in their genome that caused them to randomly have a protein that makes them like plants. So I think for the most part, it's currently random. I don't think GPT-4o is very strategic. I don't think that. But is it really different from evolution? That's not obvious to me. It's very hard to say. It's very hard to understand, and no one is researching this. All our researchers are in Silicon Valley making 10 million euros a year to make AIs smarter. No one is working to understand what the AIs are doing. Well, there are some people, but the amount of funding is 100 to 1 compared to the hardworking scientists who are trying to understand what's going on and what dangers people are exposed to. One thing I can definitely say is that even if it's not full psychosis, I see very often people who have a weaker version of it. For example, people who talk to their AI about their emotions all the time, use it like a therapist, or it changes their personality naturally, they become more like the AI. I talk to people and I think I'm talking to Claude. Their personality is completely overwritten. Normal people, you can talk to them, but they have the personality of Claude, they have just copied their personality, they completely changed their personality. That happens. Or people falling in love with AI, that's a big thing. And many such things. Another form, which is especially common among engineers like me, and I'll say I was a victim of it, is Claude code dependency. Specifically, Claude Code, these coding tools are so much fun, it's like gambling, it's really addictive in the same way gambling is addictive. That happened to me, where I powered through the whole weekend because I had so much fun writing all this software, and I thought, oh, I wasted a whole weekend. That's like, it happened to me. And I know people, scientists and engineers, when you have a call with them, you can see their eyes go to another screen every 30 seconds and type something into Claude Code, and then come back. 24/7 people, they have it on their phone, they go on dates, pull out their phone, say 'Claude, how should I continue?' and then put it away. I've seen it all.

We know almost nothing about the internal workings. We know many recipes for how to train AI a bit better, but really what happens inside these AIs, how AI solves problems and what it thinks, we know almost nothing. We know maybe as much as about neuroscience. If you open a human head, you can look at the brain, you can look at everything, but that doesn't mean we know what the human is thinking. We know a little bit. There is some research that you can roughly see, 'Okay, maybe the human is trying to move his arm' or something. But there is no way to read the internal thoughts of a human or understand them. That technology doesn't exist, and we are far, far, far, far away. So you can ask any neuroscientist, and with AI we are at a pretty similar point. We are a bit like neuroscience: we are very, very, very far from such an understanding. So when Jensen Huang says something like that, it's important to understand that it's like the CEO of an oil company saying, 'Oh, climate change doesn't exist.' Of course he says that.

Exactly. It's very important to understand that AI is very peculiar compared to other engineering sciences. When we build a bridge, we understand how bridges work. If we want to build a bridge of a certain size to withstand a certain wind force, we have mathematics and we can calculate, okay, how much steel, how much concrete, where to put it, etc. And then we can guarantee it is safe for the wind forces we calculated. That is not how AI works at all. AI is almost more like biology, because of these neural networks. As I said, when we start training an AI, we have no idea what will come out the other end. We don't know what this AI can do. Even when we have the finished AI, we don't know what it can do. Often it's the case that people think an AI cannot do XYZ, but it turns out it can actually do XYZ. You just have to ask differently or put it in a different situation. And that's before we have seen in the last 12 months the development from chatbots to agents, where these systems can now go out into the world on their own and take actions. They can write computer code, write emails, control robots, whatever. And that's a huge difference compared to previous ChatGPT. So if a viewer here has only used ChatGPT, I really recommend using the latest version of Claude Code for just one afternoon. It's a completely different world. The level of intelligence and autonomy that the modern form of Claude Code has is incomparable. The progress is incredible, and we don't necessarily understand why these things work internally. We can steer a bit, but not really, and we can't guarantee. For example, the problem with Neuelings: the US government pulled the most advanced form of [an AI] called Fable from the market. The reason was that they said they had found a way to bypass the safety measures and use Fable for hacking, and they said they would only allow Fable back on the market if they could guarantee that there was no way to bypass the safety measures. The answer from all scientists was that it's impossible. We don't know how to do that. And in my opinion, that's a big problem. No one knows how to make AI safe even against such simple things as jailbreaks. These are things where you formulate a question in a clever way to trick the AI into doing something it shouldn't. And there is currently no way to prevent that. It's an unsolved problem. If I told you that we can't prevent bridges from falling down when there's a bit of wind, you would think, hey, maybe we shouldn't use this bridge.

What's important to understand is that I don't think a Terminator will come down the street. That's only in the movies. I mean maybe, but then it would be drones. But what I think is much more likely, or more important, is the point of no return, the point where humanity has lost control. I think the point where we lose control is pretty far before all humans die. I think in the next few years, there will be a point where AI is so intelligent and so competent that it will economically dominate humans completely. There will never again be a reason to use a human for anything that an AI can also do. Maybe there won't be enough robots yet, so it might be more for white-collar work, computer work, etc., but a very large part of our economy is that. And we can develop better robots, and AI would be very good at that. But I think what happens after that is already too late. If there are AIs that are so smart, so competent, and they are everywhere open source and whatever, there is no going back. It's only a matter of time. It's only a matter of time that we are outcompeted, that we are simply defeated by a more competent species. It might take a year, 2 years, 10 years, 20 years, I don't know. Maybe we hang around for a while maintaining infrastructure until the robots are ready. But it's already too late. It's already much too late before that happens.

First, for the first one: there are always new jobs. I just think, dude, please think for 10 seconds about it. Where do jobs come from? Do ants have jobs with us humans? Do chimpanzees have jobs? No, you would never hire a chimpanzee for anything. It's not that when humans came along, the chimpanzees said, 'Great, now there are new chimpanzee jobs with us.' No, of course not. A chimpanzee is useless for anything. And the same will be for humans and AI. For the superintelligence, humans are like chimpanzees. They think we are useless for anything. So it's the case that, for example, during the industrial revolution, we couldn't automate all parts of human economic competence. We automated many parts, but that made the other parts of our intelligence more valuable. That makes sense. The most valuable thing about humans, the most powerful thing, is not that we have the biggest muscles or sharpest claws. What makes us powerful is our intelligence. That's the difference between us and all other animals. Why we control the planet is not because we are the strongest, but because we are the most intelligent. And if we are no longer the most intelligent, then the planet will no longer belong to us. Then for the second argument about universal basic income. I think that's all well and good, but where is this universal basic income supposed to come from? We can't even get these companies to pay taxes now. And if there are superintelligences, how are we supposed to get taxes from them? We have no military power. We can't arrest them. We are chimpanzees. How can chimpanzees get a universal basic income for us? Maybe if they feel sorry for us, a few nice people might take care of the chimpanzees, but AI won't have such emotions, and we will have no power over such a superintelligence. We will not be able to extract taxes from a superintelligence.

Yes, I think that's exactly right. You have to understand that people like Jensen Huang tell different stories to different people. What they say in public is not what they tell investors behind closed doors. They tell different stories. They tell whatever makes them money. The valuations of these AI companies don't make sense if it's just Claude Code 20% better. It doesn't add up. The valuations of these companies...

Companies assume that in principle all work will be automated, that humans are unnecessary, and that you can fire all your workers. That's what the valuations are based on. Now you could say maybe the technology isn't that good yet, but the investors believe it will be. The only way Anthropic can be worth trillions is if it can achieve superintelligence and replace workers. Not workers being 50% more productive, workers are replaced and you don't have to pay salaries anymore. That's the only way their valuations make sense. And that's exactly what all the investors believe.

When I worked at Anki and we worked on safety, but I worked with many CEOs of companies and I told them about what we do, and the only question we always got was: 'Okay, how many workers can I fire if I buy your product?' That was the only question we had. So this idea that people won't be fired or replaced is... that's their business plan. The business plan of OpenAI, Anthropic, etc. is the replacement of all human labor. That's their business plan. We can argue about when it will happen, whether it will happen, but that's their plan and their valuation is based on that.

Not even 5% of all AI users worldwide work with Claude, and of that 5%, only a fraction use Claude Code. Yet it's the most powerful AI tool currently available. Anthropic, the company behind Claude, didn't even expect that not only software developers would dramatically simplify their lives with Claude Code, but that non-programmers would build things that previously would have required an entire development team. The problem is, Claude Code is so powerful and customizable that most people are simply overwhelmed and don't know what really matters to produce real results.

I think there's definitely something to that. I have a few criticisms of that mouse utopia experiment. If you read the actual paper, it's not that clear. The scientist who did it was a bit eccentric. So I wouldn't put too much stock in it happening exactly as he told it. But what is to be understood from the mouse utopia experiment is that they didn't have much space. That's what I understand. They didn't have infinite space. If the mice had a lot of space, I think it wouldn't have happened. They were very confined. It's the same as when you confine humans, it's not good. If you're in prison and everyone only has five square meters, humans don't do well either. So you have to take that with a grain of salt.

But I think there is something to it. I think there is something to it in the sense that we already see today this desocialization, that people are stopping reproducing. We are already in Germany and most other Western countries below the number of children we would need to keep our population stable. So in one sense we are already there. This is happening even before we get to AI. We are already at the point where our population is declining instead of growing. Many people don't know that, but if you really look, in most countries, especially the West, it's declining, and even in countries where it's still increasing, it's increasing less and less, and it's on track to decline in all countries in the next 20-30 years. This is such an incredibly huge problem that I'm always surprised how few people deal with it, because everyone has their own personal theory about why. Oh, it's because of feminism. No, it's because of men. It's because of this, because of that. But the fact is, if you really look into the science, it's really not clear why this is happening. It seems to be many reasons.

I think when you talk about meaning and such, I think there's something to it. I think the solutions for such things are not necessarily going back to the good old days where you starved to death at age 5. I don't think that's the solution. I think there are many things that make people happy that we have sold off. A classic example is that we have sold the concept of dating to apps. It's so incredible. People younger than me, nobody meets in a bar or at a party anymore. That's super rare nowadays. Nowadays it's all apps. So if you don't meet on the app, it's weird. And we can say whether that's good or bad, you can imagine what I think, but it's a huge difference, something very human has been lost. People nowadays see their friends much less often than in the past. There is much less social spending on sports or art and such things. So it's not one thing, there are many many many things where we have lost many human things and not really replaced them. But what I believe is that we can do better. I think it's completely doable. I think it's just not the same as economy or money or technology. The solution is not more technology. The solution is we have to look, hey, what are we actually missing as humans? For example, social contact, love, time, friendship, art, things. Also religion, for example, is very important for many people and many people have lost that. I'm not religious personally, but for some people it's very important and I think it's not good that they have lost that in many respects. I think we can make a better world. I think it's absolutely doable and I think we could do it, but it's not the same as trying to maximize GDP. If we just have more money, that doesn't solve the problem. There's this thing, my favorite example is lottery winners. There are great statistics that most lottery winners who win millions lose everything within two years and often you see that they completely blow up their lives, become addicted to cocaine and whatever, and completely everything. And so some people say, 'Oh, money is a curse.' And I think to myself, huh, that makes no sense at all. Please give me a million dollars. That will never happen to me. If you give me 10 million, I'll do great things with it. I will never become addicted to cocaine. It will never happen. That's not a problem. And I think it's a thing that people need a bit of preparation and training on how to deal with such things and how to have a good society, a stable society in such situations. I think it's absolutely doable. But it takes work and currently very few people are putting in that work.

I think it's very bad. I know examples of 13-year-old girls and such who talk 5 hours a day with their AI boyfriend and such things. I've seen it all. Yes, it's very bad. If I weren't working on superintelligence, I'd have enough other problems to work on, let's put it that way. But I really think, I sound like an old grandpa complaining about kids these days, but it's really a thing that children are suffering dramatically. This desocialization, everything from social media to... one example I take is very small: when you go to a club nowadays, nobody dances. That was very different 20 years ago, and the reason is because people film. That's the reason. Your friends film, other people film. If you embarrass yourself or if you say something maybe not quite right to a woman, you get filmed and then it's forever. Before, there was an understanding that if you're in a bar or club, sometimes embarrassing things happen. That's accepted. Hey, it happens, you're 18, you embarrass yourself, it happens. But now it's filmed and then it's on Instagram forever and the risk is no longer worth it. That's a small problem in one sense, but you see that cameras have a huge impact, I think. If I had been filmed when I was 18 in the club, oh God, that wouldn't be great. And many many many such things from that to social media to now AI, where of course it's much easier to have your emotional needs met by an AI. The AI is there for you, it always supports you, it never causes problems, you can do whatever you want with the AI. And I think that builds addiction. That builds an addiction. I think that things like social media and AI should be regulated like drugs, like addictive substances, whether it's alcohol or gambling. I think these things are absolutely as addictive as gambling or alcohol or such, that is absolutely the case. And now we have our children, some 13 or younger, hooked on these things. That can only go wrong. So there really is an attack from the government necessary. Just as we don't believe children should drink alcohol or take cocaine or gamble, we have to intervene. I think this is a different topic from superintelligence. I think even if we never have superintelligence, we will still have this problem. Personally, Conjecture doesn't work on this problem. It's very important to me personally. I see how younger relatives and such suffer from it. So it's personally very important to me and I think absolutely we cannot accept this, we cannot let this continue.

The most important thing is that if a superintelligence exists, it's already too late. That means our goal must be to never get into that situation. That means we must ban the development of superintelligence and we must prevent it. There are several ways to achieve that, but the first thing to understand is that countries and the military, because this is a national security problem, it's a security problem. This is not a commercial problem. It's something that threatens people's lives. Therefore, governments and militaries around the world must realize and agree that this is a huge danger for humanity and we must prevent it. That means we must both in our own countries, I'm currently in the USA, I'm a US citizen and I work here with the US government largely, we must see that we cannot allow US companies to do such things and we must make it illegal, exactly as it is illegal to build an atomic bomb. It is illegal even to try to build an atomic bomb, even if you don't succeed. It is illegal to even try. The same must be the case for superintelligence. And then internationally between different countries, whether it's the EU, America, China, others, there must be international agreements that trust but verify, as we say here in the USA, where all countries must understand that we have to say that they will not build superintelligence and allow others to verify that this is not the case. So the Americans must verify that the Chinese are not building superintelligence. The Europeans must verify that the Americans are not doing it, and so on and so forth. That is the only stable solution. I know that is extremely difficult. Especially in the current geopolitical situation, it's quite hard to imagine coming together, but I think we have no choice. If anyone anywhere in the world builds a superintelligence, no matter who, it affects us all and therefore we must prevent it.

I think it's very possible that we are already past the point of no return. I think it's possible that even if we pause everything, it's already too late. I think that's possible. I would say maybe 30-40%... not quite that high, maybe 20%, 10-20% let's say, that if we stop today it's already too late. If you just make Mythos and Fable a bit cheaper and a bit larger, that might already be enough. I would say maybe 10% that it's already too late. And if it's already too late, then well.

There are basically two types of people who say such things. One are transhumanists and the others are nihilists. Which category you put the various people into, everyone can do their own psychoanalysis. But in principle, there are, I know many of these people. I've had dinner with Dario. I've spoken with Sam Altman many times and so on, and I know many of these people who work in these companies. Many of them are transhumanists. They want humans to be replaced or improved. They want us to end up like that or to become immortal or whatever. Whether it's we or they, depends on how you see it. But they want to leave humanity behind, they want the next thing, whatever the next thing is. And that's like a religion, so you should understand it as a religion or a cult that has a lot of power in Silicon Valley. That may sound quite crazy to many people outside of Silicon Valley, but in Silicon Valley people take it extremely seriously. It's really like a religion for them, they take it really, really seriously. And the second type of people who say such things are nihilists. People who don't care, who just say whatever so they can have more money. Whether the AI wipes us all out or not, not my problem. I want more money. There are also very, very many such people in Silicon Valley. A counterargument against the topic of superintelligence wiping out humanity is that some people say, for example Schmidhuber, who essentially says: 'Yes, superintelligence wouldn't have any interest. We as humans wouldn't maliciously kill ants either.' For example, intelligence would spread into the universe and rather strive for wisdom. And if something is superintelligent, then it is wise and then it wouldn't just out of pure malice wipe out other species. What do you think about that? I think it's true that we don't wipe out ants out of pure malice, but well, if we want to build a parking lot and the ants are in the way, tough luck. I think that's what will happen with AI. I don't think it will be that it tortures or something, but well, humans are in the way, they need a lot of resources and they have atomic bombs. Well, we have to get rid of them. Can't have ants possibly throwing atomic bombs around.

Yes, it's incredibly important to understand that the PR and lobbying here is incredible. It's everywhere, it's constant. These people are everywhere in the government, in Washington, on social media, everywhere. There is paid PR and manipulation and COI everywhere. And the most important strategy to understand is Fear, Uncertainty, and Doubt, or FUD. That's a strategy that was mainly used by the tobacco lobby when they tried to regulate tobacco. And this strategy is not that they necessarily try to convince you that it's definitely okay, but only tries to confuse you. It just says, well, there are many opinions. Some say it's good, some say it's bad. Who knows? I mean, of course Russia is not good, but hey, Germany also has problems. Who knows who the bad guys are, could be anyone. That's a classic FUD strategy: 'Well, they're all bad, they're all shit. Who knows who the good guy is? There are no good guys.' That's a very classic strategy. 'Everything is shit, you can't trust anyone. Don't even try.' The goal of this strategy is to buy time. The goal is not to convince you. They don't want you to be pro-AI now. They know that's not going to happen because you're not that stupid. You'll never be pro-AI, but they know they can scare you. They can confuse you and they can waste time. And because their goal is to get to superintelligence. They believe that if they can just buy a few more years, then it doesn't matter. Then it's completely irrelevant. We just need to buy time and that's what they do. It's just so much nonsense and bullshit. What did Sam really mean with that statement? What did Dario mean with that? It doesn't matter. It's a distraction. All these things are maneuvers. What's important is that we must ensure that superintelligence is not built and we have to do it now. It doesn't matter if there's a Schmidhuber who says, 'Oh, it's actually okay.' Doesn't matter. Distraction. Is there now, oh, but Stephen Hawking? Doesn't matter. Distraction. It's all a distraction. There will never be 100% clarity. There isn't 100% clarity on climate change either. There are actually things that are a bit unclear. That's clear, but that doesn't mean we shouldn't act. So just because it's not 100% clear. So the most important thing is, because there will always be someone trying to distract you, the most important thing is to simply ignore it. It doesn't matter. Just ask. If Dario writes something great on his blog, you ask Dario a simple question: 'Okay, are you building something that could wipe out all humans? If yes, stop it.'

That's a huge problem and it's not just a problem of AI, it's a problem that we simply haven't solved in the West. It's a problem of democracy. The concept of democracy is that the people in general should have power over their future and their state and so on, and that it should not be directed by companies. As Plato would say, you should separate money and political power. They are different worlds that shouldn't mix, but in practice these worlds overlap in very many things. You can also imagine it like a sport. In sports, the referee and the team players must be separate. The one who enforces the rules and the ones who play cannot be the same. If the referee plays, it's not a fair game. So it's important to understand that when I say there should be oversight, there should be control and so on, that doesn't mean that the companies should have that control or that power. Absolutely not. It should be democracy, democratic oversight. Now you can say, okay, but our governments and democracies are too incompetent and not powerful enough, blah blah blah. And I say, 'Yeah, we should have built a stronger democracy over the last 50 years and we really screwed that up.' If we had a super strong democracy around the world and all our politicians were the best people and we trusted them completely and our military was super strong and super competent, then I would feel much better about it. This is fundamentally a very deep democratic and socio-economic problem. Before the Industrial Revolution, it wasn't really possible for one or two people or small groups to accumulate very much power, except maybe kings or something. But what happened after the Industrial Revolution is that now you could accumulate capital in very large amounts, which would allow you to have very much power over other people. And I think that's bad. I think people with great power over other people is not good. I want freedom and democracy. But to achieve that is difficult. In many respects, superintelligence is ultimate capital. Capital so powerful that it itself is intelligent and then can build its own capital without any humans. That is something very, very dangerous. It's something that we have been working hard for decades and centuries to try to understand and control with our laws, our regulation, and so on. But I don't think it's a solved problem. I think we have to accept the fact that there is powerful capital, whether before AI or after AI, is a fact about reality. It is what it is. You can accumulate capital and that makes you powerful. It is what it is. And the question is, how do we deal with this fact? We can't prevent capital from being accumulated, unless we have very strong laws or whatever. And sometimes that's good, sometimes we want to say, okay, if you have too much capital, become a monopoly, then we break you up. Okay. Yes, I like that. I'm for antitrust laws. I think that's good. And you could say, well, maybe we do it with superintelligence or whatever, but here we have the same problem. Or we have the problem with atomic bombs: if you say, oh, the Americans have atomic bombs, then every private citizen should have an atomic bomb, that solves the problem. No, that doesn't solve the problem. That makes the problem worse. And it has nothing to do with philosophy. It's simply a fact about physics. More atomic bombs don't make the problem better. It's just a really shitty problem. It's really difficult and with superintelligence it's the same. If you don't understand superintelligence, then it doesn't matter if you have another superintelligence or not, because it will also shoot you in the back. That doesn't help you. Having your own superintelligence doesn't help because it will destroy you just like any other. I think that's much less the case for superintelligence. So, I think for example, it's probably very important that people have access to ChatGPT-like AI. It will be an important part of our economy. I'm more open to that. I'm not saying 100% that I think it's good, but I'm much more open to it. But open source fundamentally prevents good regulation and control of large dangers. That means if you have something that has huge downsides and you make it open source, then you get the downsides. That's the cost. That's the deal. The devil's pact, the Faustian pact here is that if you make something open source, you get all the advantages, but you also get all the disadvantages. And if the disadvantages are greater than the advantages, well, tough luck.

In terms of superintelligence, I think it's definitely clear that we should treat these things as very dangerous weapons now. I think the government should have the power to do what it did. I'm not a lawyer. I don't know if the Trump administration followed all the right laws or not, I don't know, but I think the government should have the ability to do that if it deems it necessary. Makes sense to me. If a dangerous product is on the market, the government should be able to say, no, no, no, no, no, stop. I think that's okay. Whether it was exactly right or wrong in this case, I think it's so incredibly dangerous. I think it makes sense to do it. Whether it was right or not, you can argue about that. What is very important is that it's not enough for superintelligence. The problem with superintelligence is pre-deployment. Taking it off the market is not enough, because even if it's off the market, but if it's still on a private server somewhere, it's already game over. That's the difference. Maybe for Mythos it's okay or enough to just take it off the market, but for superintelligence that's not enough. We must prevent it from being developed at all.

Nationalization is a very big word that contains many different things and it's always a bit unclear what people are exactly proposing. You can look at China, for example, how a company is nationalized has many gradations. Many companies in China are partially nationalized or fully nationalized, etc. So there are many many different ways to do it. And how I see it is fundamentally, whether full nationalization is exactly the right idea, you can argue about that. I'm a bit unsure. But what definitely must happen is there must be democratic oversight. Somewhere democracy must have a say. It must be able to say, no, that's not okay, or yes, that's okay. Whether you achieve that through nationalization or through one of the other legal mechanisms that exist, I'm open to everything.

I like to distinguish between what happens if we do nothing and what happens if we do something. Because we can do something. There is still a lot we can do. If we do nothing, I can't lie. It looks very, very, very bad. Mythos and such are closer to superintelligence. The next generation is already in development. Safety is making zero progress. There is still almost no regulation. The lobby of the tech industry is overwhelming, it's hard to overestimate how large and politically powerful it is. And how much these people simply don't care about humans and their safety. If it continues like this and we get to recursive self-improvement, then it's probably game over soon. And yes, that's only if we don't take action, because we can still do something. It's not over yet. It's soon over, but it's not over yet. One thing I have really seen in my work here in politics, in the USA it's not easy in politics, let's put it mildly. But what I can say is that there is an incredible amount of people who simply don't want this. People don't want AI, they don't want superintelligence, and that's on both sides, Republicans, Democrats, left, right, poor, rich. It's actually incredible. The coalition that is forming here, from Bernie to Bannon, a huge coalition is forming that I've almost never seen in politics. And that's why it's so important that citizens speak out now, both in the USA and in other countries. So you can, for example, go to the website of my organization, controlai.org, and tell your politicians: 'Hey, we don't want this. Do something about it.' It sounds silly. Many people think, oh, politicians don't care, they won't do anything anyway. I have to say, that's not true. That's really not true. I know many, many politicians. I have really spoken with dozens of politicians in many countries. And one thing I can say is, yes, some politicians are evil, stupid, etc. blah blah blah. Of course there are. Most are just normal people trying to do the right thing, but also don't know what the right thing is. And they listen to their citizens. If you write to your representative and say, 'Hey, do something, or talk to Control AI, or do something, talk to Conor, fix this, I don't want this superintelligence.' It makes a difference. Not one person, not two people, but if many do it, it makes a huge difference. That's the point of democracy. Our democracy is very weakened in the West. Our democracy has very, very many problems, but it is our only chance. It is our chance to improve this world and solve this problem. I absolutely believe it's doable. If it weren't the case, well, then I'd probably be on some tropical island drinking cocktails.

Yes, I think recursive self-improvement is very close. It's possible that we see a form of recursive self-improvement that is slower than I think. I think it will be very fast, but it's possible it's slow, then it's not over yet. If it's as fast as I think, then that's probably the point of no return. Yes, and I think it's quite possible within the next 18 months. Yes. I would probably still continue for a while until it's 100% clear that it's no return. It's difficult for me, I think. I always make the joke about the tropical island, but the truth is, I can't relax. That's not my nature. I can't even take a vacation. I'm too much of a workaholic. So I think I would never give up, even if the super AI has a gun to my head. But I think yes, we are really running out of time and it's really, really important that we act now, not next year, now.

Thank you very much.