Banning ASI, Germany's Role, and the $5B Question | Connor Leahy Q&A

This is a great question. Luckily there is a post that you can read which I'm sure Benjamin could potentially link in the chat, which you have recently released, about what we would do with 50 million US dollars, which we think would give us a 10% chance of banning ASI globally, and we think we have a 40% chance if we could get 500 million. As an example for how large these sums are, a large political campaign in the US, let's say for example the marijuana legalization campaign, was about $650 million, and a single presidential campaign is usually around 1.1 or 1.2 billion. So it seems pretty obvious to me that given the scope of ASI and extinction risk, it should be on the order of a large political campaign. So $5 billion is actually more than most of the large political campaigns in the world, which is great. That means we can do more things.

One of the things that's really great about money is you can exchange it for goods and services. I know it's pretty crazy, but it's great. So things I would do obviously is you would hire all the best people in the world. You poach all the best people in the world. Be like, 'Okay, we'll double your salary. No problem. We need you to work on this.' You get all the best lawyers, all the best campaign people, etc. We'd run the most epic advertising campaign you've ever seen in your life. We would be on everything. Everyone would hear us 10 times in six months. Everyone in the world would see our message 10 times in six months. We would get to all the different channels, all the different demographics. We would fund citizen groups in every state and all G20 countries at the minimum, maybe even more than that. So we would have grassroots organizations there that we would fund. We would set up a large-scale policy research think tank. We'd hire all the best legal scholars in the world and say, 'You figure the hell out how we ban ASI forever.'

We would of course obviously have massive doors open just because we have a lot of money. Even without spending it, just because you have a lot of money means that if I call someone, they will pick up. They're like, 'Oh, it's the guy with $5 billion. You should take this call.' This is really, really powerful. This kind of soft power. I would get a meeting with every senator in the US. I would get a meeting with almost every head of every agency. I would definitely get a meeting with Trump. I would probably get many meetings with Trump. He likes hanging around people with lots of money. So, I think it would be very easy for me to meet him potentially many, many times. I would be able to meet all the heads or at least top-ranking politicians of all G7 or G20 countries. We would hire lobbyists in all G20 countries to get all these issues out. We would produce world-class media. We'd do some funny stunts, probably hire Keanu Reeves or Arnold Schwarzenegger to do an ad for us. There are many, many things you can do at the order of five billion dollars.

And it also allows you to experiment. For example, let me give you a very concrete example. The thing we run into right now, there are many PR firms, many of them. And almost all of them are scams. Almost all of them will quote you $100,000 and they'll give you a bunch of PowerPoint presentations and nothing happens. And it's really hard to tell that nothing is happening until it's already too late. If you're restricted on budget, you don't want to do this. But if you have billions of dollars, just hire all the PR firms and fire all the ones that are bad after 6 to 12 months and then you have the good ones. You can do the same with employees. You don't know which of these employees is good. Hire all of them and fire all the ones that are bad after a year. No problem. And now you have the best employees in the world. There are many, many things you can do when you have money at this scale. You can do the kind of stuff that maybe Elon Musk does. It's really hard to underestimate how much you can do with $5 billion. It would change the trajectory of the world is how I would say it.

So, if anybody has an idea on how we can get these five billions, make sure to write an email, Benjamin or I will then forward this.

Great question. Unfortunately, my answer is going to be very frustrating, which is that if you want my true technical opinion, it's too long to be contained in this narrow segment. I have very strong technical opinions about what these things mean and why I think these terms are not good. The short version is I think AI is actually a bad word because it mixes several things. Usually when people nowadays talk about AI, what they're talking about specifically is neural networks, which is a very specific technology. Say 40 or 50 years ago, databases were considered AI because back then it wasn't clear how you would store facts and retrieve them from a computer. So people working on databases would call themselves AI researchers. Nowadays we would never think that, but this is basically a cultural thing. Nowadays really AI means software we don't understand is how I would describe it. I think the core problem of AI is that we don't understand it. It is that we're building software that is more powerful than what we can understand. And this is not the case for other areas. If someone wants to build a new airplane and it's twice as good as my airplane, I expect this means they know more about airplanes than I do. But this is not the case for AI. You can make an AI twice as good and not have learned anything about how AI looks internally. And this is in fact the default.

So there's the philosophical technical aspect which I think goes beyond the scope of this, but then I think more interesting is the legal and practical policy definitions here. The way I see it is that the intermediate solution or the short-term solution is we define ASI as systems that are powerful enough to disempower the government or significantly threaten global or national security stability. This is an important definition but it's a useful definition. This is very useful because national security agencies can think about these issues in very deep terms and what would be a threat, what level of threat do we as the military or as the national defense system want to accept from such systems. So this is for example a useful thing. Another thing we often think about is regulating precursors. Instead of saying exactly where the line is, we say what are precursors we're confident will be before ASI. We monitor them very heavily and we are very conservative about making progress on them. We don't, for example, self-replication, self-improvement, cyber capabilities, autonomy, and stuff like this. And we make very, very strong oversight and strong bounds on anything that is on the frontier of any of these capabilities. So if your system is like AlphaFold, it has no capability to self-improve, it doesn't fall under this, and it's fine.

The true long-term solution is what is called prospective security cases. A prospective security case is how we regulate nuclear reactors. Nuclear reactors, you're not allowed to just build a nuclear reactor and then see if it melts down. That's illegal. Instead, what you do is you have to write an extremely detailed security case, massive documents which describe in extreme detail proving that your reactor is definitely safe to build. You then submit this to the nuclear regulator which checks it very, very carefully by all their experts. This design is safe and only then do they issue a license. This is what we should do for highly advanced autonomous AI systems. And the good thing about prospective security rather than the regime we currently have is it puts the burden of proof on the developer. By default you can't build anything unless you can prove to us that it's not superintelligence beyond a shadow of a doubt. So for AlphaFold this would be really, really easy, writing this report would be super easy. For AlphaFold, writing it for GPT-5, actually very hard, maybe impossible. And my answer to that is that's the point. If you can't prove that your AI does not have these safety properties or these risk profiles, then you don't get to build it. Tough luck.

One of the benefits of this is that it would also align the incentives of the market to actually develop these techniques. Currently we don't have the techniques that we would need, so for example, to prove that GPT-5 is safe. We just haven't developed them and no one is trying to, or the few that are are vastly underfunded. But if we said you can't build GPT-6 until you can do this, this would immediately align the incentives of the market to actually develop these kinds of methods and these kinds of security cases. This is ambitious. It's not something that we can do immediately. There's a lot of details here that have not been figured out. If I had $5 billion, I'd be hiring a lot of very smart people to figure out the exact details. But this is kind of where I think it's going. And to say one more thing on that, I think another thing we should do is criminalize the intent. I think this is a thing you can do in law. For example, it is illegal to attempt to build a bomb or to attempt to murder someone even if you fail. You could do the same thing with ASI. It is illegal to attempt to build ASI even if you fail. I think this is another thing we should do.

Yep. So the true answer is that almost always when people bring up things like this they're not actually bringing up evidence. What they are doing is that they are bringing up emotions. So almost always, not always. There's two large cases. One thing is people who are just genuinely underinformed. They have never used ChatGPT. They have never seen Claude code. They have no idea. Often like older people and in my experience usually you just show them, here's a bunch of scientists with Nobel prizes who sign that this is a huge risk, or here's the NSA saying Mythos is a massive risk. You just show them stuff like this and in my experience most people are just like, oh yeah, that seems really bad. This is usually not that hard. What is harder is, and I was going to predict this, is that your friends are in tech. People are the hardest possible people to convince because their entire life depends on tech being a good thing. Their identity, their ego, their salary. All of this depends on tech being a good thing for the world. So they are the most resistant people in the entire world against realistically seeing the risks of the technology they're working on. It's the same thing as if you told me, well, the CEO of this oil company really doesn't see climate change as a big problem. My answer is, yeah, duh, obviously. Otherwise, he wouldn't be the CEO of an oil company. I'm like, yeah, duh, obviously. So, I usually just don't even bother. Like, if I see someone's like a tech guy and he's like, 'Well, no, tech's good, actually.' I'm just like, 'Yeah, okay, sure, whatever.' And I just move on. It's usually not worth your time.

If you want to convince people like this, the true thing is you need to do psychotherapy with them. Basically, you need to dig into why are they saying this. You need to figure out what are they scared of. What is the emotion? And then you have to hold their hand. You have to hold their emotions usually. And because they usually have the evidence or you can give them the evidence, but if you've given them the evidence and they are still fighting, kicking, screaming, it's probably an emotional issue. And I think it's rude to psychoanalyze or psychotherapize strangers. If it's your friends or family maybe. But then you probably have to talk to them about their feelings. You probably have to talk about why they feel this way, how they feel, and so on. And this is just messy.

I think 99.9% is general purpose. I basically think it's totally fine to work on other AI technologies. If you're working on AlphaFold or whatever, it's just for me, it's just a different thing. It's not a problem.

It's a great question. So, for people inclined to read papers, Control AI has a paper on this, middle power coalitions. There is a tempting thing to think that small countries don't have any power. I think this is wrong. I think they have a lot of soft power, especially if they form coalitions. So I think the main thing you should do, other than getting in contact with Control AI and seeing if we can help you, is contact your lawmakers, push them on these same kinds of issues as any other country, and urge them towards championing and or joining coalitions with other countries to call for global moratoria on ASI. The way I see things going is ultimately we need to get to a world where there are multilateral agreements to not build ASI that are instantiated through a trust but verify regime. And of course not just the US and China, they're the most important ones obviously, but they're not the only ones important. The support and the action of other countries to facilitate such deals, to push for such things, is very important. Small countries still have diplomats. They have power. They have connections. They have a lot of the ability to say things that potentially other countries can't say. Pushing the Overton window, just saying this should happen. If the head of any country, like the Czech Republic, if the president or prime minister, whoever the head of the Czech Republic, said it's a top priority of us that we must stop superintelligence, this would be super useful for me. I would go immediately to anyone and say, well, here's a respected politician who really takes this seriously. And then it will make it much easier for the next head of state to say the next thing. It'll make it easier for the next one. And so coalitions can be formed and it will be very helpful if there were large coalitions of countries already pushing for, already doing the work of figuring out how can we do this, how can we work with our American colleagues or our Chinese colleagues for how to make this happen. How can our diplomats talk with their diplomats and start floating these ideas and pushing these ideas. This is how diplomacy works. It requires lots of action from lots and lots of people. And it sounds trite but it is true. Every contribution like this does matter. If the Czech Republic tomorrow said it's our number one national priority to make sure this happens, yeah, I think it matters. I really think it does. I think it will make a difference. Will it solve the whole problem all at once? No. But I think it will really matter.

Sorry, maybe I misspoke. What I'm not saying is that evil isn't present. What I'm saying is it's not the problem. What I mean by this is that evil is a real thing in the world. No question about it. I have encountered evil that is like true evil. There's no other way to put it. What I'm saying is that it's not the dominant force. The dominant force is chaos. The main reason evil exists is because we don't have enough law. We don't have enough order. It's not that we're living in an extremely ordered reality that happens to be evil. It's not like we're living in a dictatorship where we have a planned economy and the planned economy is run by extremely evil, corrupt people who are coherently optimizing towards ASI. Thus, such worlds exist and they're very bad, but it's not the world we're living in right now. Currently, all these evil people are also at the whims of chaos. They're also acting in a chaotic environment. They're empowered by chaos.

If we have laws and the way to address, let's say we have an evil person running a company, and there are many such people, and they're running an oil company or Google or whatever. Let's take Google. And let's say the CEO is a super evil super villain. Now let's say we get him to step down or we get him to change his heart, he has a change of heart, you know, he's the Grinch, heart grows three sizes and he totally changes his mind. Now what? Well, let's say he's completely convinced, oh no, Google is a huge threat to humanity and we have to shut it down. So he runs into the office and says, shut it down, delete all the AI, destroy all the servers. What happens? He stops being CEO of Google and someone just as evil gets put in. The problem isn't there's an individual evil guy. This is a huge distraction. Even if Sam Altman or whatever just went to retire on an island, it doesn't matter. Someone doing the same thing will be put in the same position. It's the system. It's not any individual person being evil. There are evil people but they're being empowered by a system that is lacking in law, that is lacking in order. There are other industries for example where we do have regulation. Like I think the CEOs of nuclear companies are probably no less evil than any other, but they don't build dangerous nuclear reactors because we have the law to prevent them from doing that. We have the order, we have the systems, we have the institutions. The goal, the way to make nuclear safe is not put a really nice guy in charge of the nuclear program. The way you do it is through institutions and law. That's the solution to the problem. So maybe a better way to think about it is not that chaos is the enemy, more law is the solution.

I'm not thinking in terms of what's realistic or what's not because funding is very heavy-tailed. As someone who's done startups before, I had effectively $20 to my name for most of my adult life. And then very suddenly I had 30 million basically overnight for no reason, just because I did some stupid thing with my friends online. A lot of when it comes to fundraising and big projects like this, political projects or startup projects but especially political projects, is that at least I have not yet figured out how to make it predictable. A lot of my work is taking something that is unpredictable and making it predictable. Taking something that's chaos and turning it into order so that it can be reasoned about. A lot of what we do with Control AI is taking something chaotic like what do people think about AI and what is politics and trying to ground it into something that is reasonable, like okay how many people have signed this, how many people have said this publicly, what's the next step, etc. I have not yet figured out how to do this for fundraising. I'm not saying it's impossible, but I have not done it. I do not have a formula or a system that I have figured out of how to predictably predict in any bounded sense what funding I get.

It is completely in model for me that I run out of money in one or two years and that's it. What is also in model is that this time next year I have a billion dollars. Both of these are in model. I do think for example the 50 million are extremely doable. I think that is like, I have basically myself raised that much before on other projects that were much less good. So I think it's totally possible but it is chaotic especially because the nonprofit space is not an efficient market. If in the commercial space, if you have a company with really good metrics, you're making a lot of money, you're growing fast, you have a very solid business model, and you need more investment, you will find it. Like if you have a great company, you will find investors. This is not a problem. This is not the case for political or open projects. This is not at all the case. You can have the best policy in the world and the best plan and everything's perfect. All you need is money and you can get nothing. Or you could have the shittiest project in the world, but then you meet one eccentric billionaire who thinks you're really fun and he gives you $100 million. I dislike this. This is I think one of the problems I currently have with fundraising is that it is chaos. I'm pretty good at handling chaos. So, like if there is someone in the world that could do this, I'm like pretty confident, I'm not the worst person in the world to do it. Yeah, I think 50 million is totally doable. I think 500 million is also in model. If I had like five or 10 years, oh, of course, I'm getting 500 million, but on the shorter timelines, very chaotic.

Two important questions. So, first one is hardware versus software. The way I think about this is it's kind of the same thing. We don't know how much hardware is needed for an ASI. It's probably much less than is currently on the grid. I expect you can build it if you have the right algorithms, you can make an ASI with 1% of the GPUs on the grid or something like that. Hardware controls have many good reasons to have hardware controls that have nothing to do with ASI. For example, if you think ASI isn't real, but you're worried about Chinese war fighters using advanced targeting models, that's a good reason for chip exports, for example. Not my main concern, but it is a very good and valid concern. I don't think we should be selling weapons-grade hardware to our adversaries. Controversial opinion. But we have to be realistic here is that we don't know what the minimum amount of hardware is to get to ASI. Anyone who claimed otherwise is lying. I think hardware is a bit of a drunk in the street light effect. Like there's a thing where the drunk looks for his keys and the policeman sees him looking under a street light and he goes to the next street light. The cop asks him, 'What are you doing?' And he says, 'Well, I'm looking for my keys.' 'Why are you looking under the street light? It could be somewhere in between.' 'Well, that's where the light is.'

So, there's a similar thing here. It's easier to measure hardware. Therefore, it's very tempting to focus on it, but we have to be realistic. It's not enough. It doesn't work. It just doesn't. We have to do something we haven't done before, which is fundamentally we have to regulate and we have to work on pre-development software. This is a quite novel form of regulation. It's not entirely novel but it's quite novel. Pre-deployment is not good enough. If you have an ASI in a lab it's already too late. Hardware is not enough because if you have a good enough algorithm the hardware already in the grid is good enough. You need pre-development safety and regulation on software. The more you understand software, so to answer your second question, the more we can predict what software is safe or not safe, the basically the more breathing room we have on how strict we have to be on this legislation. Like if we know very clearly that ASI cannot happen under this level, then we can loosen legislation. And if we don't have that then we have to have very tight legislation. We have to be very careful. We have to ban things that are three or four or 10 steps away from ASI, not one step. If we're only one step away we're way too close. I think most technical AI safety work does not make sense in the current world. I think it is the kind of thing that makes sense after a ban exists. Similar to how a lot of the technologies to enforce the nuclear test ban treaty were developed after the treaty was signed. First the treaty produces the will that we can have DARPA spend $10 billion on it or something. These are this class level of problems. These should be problems that we were having. DARPA spends $10 billion. MIT and Area 51 should be working on this. It shouldn't be, no disrespect to hackers around the world. I was one myself, but it's nice, but they will not solve this tier of problem. And so, we should get people the funding and the support to be able to do this kind of work. And this is just currently lacking. And the bottleneck again is political will.

Don't have a specific plan specifically about the midterms.

I think it is definitely useful. I think its effectiveness depends on follow-up. I think one-off statements are good but not great. I think a sustained campaign where you for example reactivate these people regularly again to make sure they see a message seven to 10 times over six months is extremely valuable and extremely effective. I think this kind of thing, like if you get 10 million people to join your newsletter and to sign your statement or whatever, I think this is extremely effective. This is extremely good. Control AI does do stuff in this direction. We do have a large newsletter over 100K subscribers and so on. Getting to a million people is possible, but it's very expensive. Like it's just you need massive marketing efforts, you need massive advertising, you need massive engagement. It's definitely doable. It's just extremely expensive. Not extremely expensive. It's actually very cheap for what it is, but you should expect, if you want a million people to sign a statement, you should expect to spend close to a million dollars on that, maybe more, maybe less.

I mean, very similar question, very similar answer to other middle powers. Obviously Germany is barely a middle power. It's like the third largest economy in the world. So Germany is the largest non-superpower country that is very close to it. Germany obviously has an extremely leading role in the EU which is to many degrees a superpower. So I would say my advice to Germany is the same to the Czech Republic or other countries just skewed slightly towards leading the EU and leading with their extremely strong economy to help create such global machines.

I don't think right now, but it's the kind of problem that $2 billion of DARPA money can solve.

My understanding is that truly the frontier is at the big public labs and they're all extremely leaky. You can go to the parties in San Francisco and people will kind of just tell you everything. It's just super, super leaky. Intelligence agencies have fully penetrated all of these companies entirely. There's no ITAR restrictions and so on. These companies are completely hacked. They're completely full of holes. They leak all the time, etc. There is no credible reason to believe that there are government black projects that are competitive in any meaningful sense whatsoever. I talked to an expert on government secret projects and he had a fun way of saying it. His phrasing was there's only one thing that all secret projects have in common. They suck. Like once you do something that's secret and it's out in a desert somewhere, there's so much friction to doing anything that you just slow everything down by a factor of 100. So there's no credible reasons to believe that there currently is anything beyond the state-of-the-art there.

I mean, my true opinion is that I think we probably crossed the line somewhere by GPT-3 or 4. So I unfortunately do think that, yep, you're right. We don't know where the correct line is, like where we should stop. Mythos might have crossed it. And this is really bad. This is something that me and other people have been warning for many years is that the closer you get to the point of no return, the more expensive it is to stop. There's this idea that like, oh, we race as hard as possible and then we stop. But this is insane. As you get closer to the finish line, it becomes exponentially more expensive to stop because now there's much more diffusion and also the technology is like, it's much easier to go from Mythos to an ASI than it is from GPT-4 to an ASI. So even if we stop but Mythos is still out there, people will hack it, people will tweak it, people, even if there's no large project, some people will try and we will have to regulate them much stronger. If there was no GPT-4, then it's okay to own 100 GPUs probably, you can't make GPT-5 with 100 GPUs so it's fine, who cares. But if there is a system that's 99% ASI, then maybe 100 GPUs might be enough to get you the last 1% and then we have to regulate 100 GPUs. So the level of restriction we have to put in increases dramatically the closer we get to the line and we might have already crossed it. Very importantly I don't think we will know when we've crossed the line. It might already be too late. We don't know and we probably won't know. So therefore the policy remains the same. We have to act as soon as possible because it only will get worse. It will not get better.

Yeah, I think this is the big one. Most people just have not heard the issues or have not had it explained to them even once. I don't know any normal person who's had it explained to them seven times who does not care. And a lot of it is that you do need to give people things to do. You need to give them things that are reasonable for them to do given their constraints. Telling them to themselves figure out how to lobby Congress or something is just not sensible. So, this is why we have stuff like saymicrocommit.io, which is a platform that you can use where every week we'll send you things you can do in under five minutes that will help about these issues. And also why I for example run a community called Torchbearer, torchbearer.com community, where anyone who wants to dedicate two hours or more per week to doing something to make the world a better place, you can sign up there. It's a nice community. I think we're like 35 people now trying to do things like this. So these would be for example things that you can do that anyone can do. But I think most part activation is just very much bottlenecked by actually getting the information to people and having them see messages seven to 10 times and that just takes time and money.

I can only speak in terms of the US. Our endorsed policy and my recommended policy for the United States is not to pause the development of ASI and not to stop the development of data centers until we have multilateral disarmament. I do think there is a version of this which is that we continue the build of data centers, military AI systems, etc. while restricting development specifically of superintelligence. I think there is a version of this which is very good such as for example criminalizing intent. So for example you will continue to build military AI systems at the frontier so that we can compete with our geopolitical adversaries while also criminalizing the intent to build superintelligence that threatens human lives including American lives. This is a good compromised position in the sense that the creation of a rogue superintelligence is in fact a system that undermines national security. So kind of by definition this is a system that our national security system should oppose being built. I do think for example slowing down data centers is not productive fundamentally because it just leads to power leaking into other jurisdictions. Say the US has a moratorium on data center construction. This just means more data centers get built in China or UAE or whatever. I think for non-US countries it can make much more sense. Many of these countries do not have a meaningful frontier superintelligence effort and there the signaling value of criminalizing this kind of thing I think is actually very valuable. So there's quite a different answer between whether your country already has an ongoing superintelligence effort and what level of investment you want to say.

Generally, my top recommendation is criminalize the construction of uncontrollable superintelligence, including within your borders, but do not stop the buildout of data centers until a global multilateral agreement has been reached. And then you can consider whether that would make sense or not.