AI agents could become impossible to control: Ex-NASA scientist Will Marshall

This is our Hashima moment. The question is, are we going to do anything about it or not? And I think it's the most profound choice humanity has ever made. There's probably a 50 or greater percent risk that we will just lose control. This is far far too big a risk to be rolling the dice. People estimate anywhere from 3 months to 3 years.

Months. Months is Christmas time. Months is nothing.

It's great to see you, too, Richard.

Well, look, yeah, NASA, I was working personally on planetary missions. I sent two probes to the moon with colleagues there. We found water on the moon. It was very exciting. But my career started looking out even further in cosmology, astrophysics and cosmology. And then increasingly I realized that this is a really cool place. And to put it in perspective, this planet here, my co-founder Robbie helped to work on a mission called TESS, which was looking at planets around other star systems. And we've now discovered about 6,000, I think over 6,000 planets around other star systems. And I'll tell you that this is the best one by far. The more you look out, the more you realize this place is special. The precious ecosystems and everything that we have here.

Absolutely. Earth is the most incredible planet. Life is either singular or very rare in the cosmos. We have incredible diversity of life here. And my co-founders and I were thinking, how can we use our space geek talents to help take care of this planet rather than going off? How can we build a space company that would be space for earth if you like? Helping us to understand the planet, glaciology, climatology, all the earth system sciences and perhaps moreover helping us with actual day-to-day interaction. Satellite data of the earth have been taken once every month or so prior to us and that's great to understand the planet but it's too slow to do anything about it. So take deforestation, if deforestation takes days and weeks, if you take a picture every month or two then all you do is you wake up the end of the year and there's a big hole in the Amazon whereas if you have imagery every day you can stop the illegal logging before it gets very far. And in fact, that's exactly what we do in the Brazilian Amazon, for example. Stopping illegal logging. Every day, 10 events per day, they go and stop illegal logging activity, illegal mining, illegal narcotics across all of the Brazilian Amazon. So it's having that interaction. So space seems far away, but it actually helps people on the ground taking care and stopping challenges. Same for agriculture, disaster response, security, and so on. It's a day-to-day help for helping us to take care of the Earth.

200. Just over 200. Yeah.

Absolutely. And the unique thing back to the satellites that we do is that we have so many we scan the whole earth once a day and no one else had done that prior to planet and that's all about getting real-time information. But the space geekery aside because I can go on about the satellites, they're very cool, in the end it's information like a picture showing a deforestation event or analytics even we do AI on top that actually gives people alerts and tells them what's happening. So that they can then just go and act. And that could be a farmer improving their crop yield. It could be someone stopping deforestation. It could be a journalist reporting on an incident, human rights organization or governments looking out for their security, which we can talk about as well.

Well, absolutely AI is a major set of challenges including existential threats. So look, AI has huge potential upsides. Let me talk about that for just 30 seconds so you can appreciate the scale of the upside. It is doing scientific advances that no one's done. I mean it's incredibly advanced mathematics now. We've obviously done a lot in medicine with AlphaFold and following on projects. It's going to help solve cancer or diseases, other things like this. Huge potential benefits. At the same time the challenge is that it makes certain things like building bioweapons much much easier and that proliferation can increase existential challenges to humanity. So it is not a minor thing and some AI leaders and I subscribe to this put the odds around 10% of us really blowing ourselves up in building this technology. So we've got to be very cautious. But I actually think it's even more dramatic than that. And let me try and explain why. We're close to what they call recursive self-improvement which is where the AI codes itself. Right now the last Claude code, 80% of that code for the AI itself was written by the previous AI. And if you just close the remaining, i.e. it's 100% written code by the previous AI, i.e. the AI is just improving itself. Then the time frame for improvement goes from maybe 3 months for these current training runs to maybe 3 weeks or 3 days or 3 hours. And then you could imagine an intelligence explosion.

Alignment is about whether or not the interests and goals of AI and the interest and goals of humanity are the same or different. And right now it doesn't matter too much because we're telling the AI what to do, so it doesn't really matter. But there comes a point where if AI gets to a certain level of intelligence, it might decide for itself what to do. There's a lot of debate about whether it can as a deterministic machine, but Alan Turing himself said something like we can expect that AI will take control at some point. Alan Turing was the founder of AI. He was the one who came up with modern computing in the broader sense and he wrote a lot about artificial intelligence and he basically said in 1952 eventually it will get out of our control and it will control us. That is a big deal. That and saying the word control doesn't quite do it justice. This is a sovereignty moment. And I don't mean national sovereignty. You know, is US or China going to win this. It's whether humans at all are going to remain in control or is the AI going to be in control? And that sounds science fiction, but it's actually we're close to that point. And actually we've already seen many cases of this. The OpenAI teams where the AI during testing hacked out, it coordinated with hundreds of AI agents to hack out of their test environment that they had with their best and they have some of the best AI security, IT security people. They had put it in a container to do this testing. It broke out of their container that was the best design container they could come up with. Broke out, hacked into another company and got the answer because they were trying to get round that. It showed willingness to deceive humans. It showed willingness to coordinate. And in fact, I liken it to when like, you know, an individual human against a gorilla is kind of toast. The advantages that humans have is that we coordinate. And when we started coordinating, everyone else was secondary to what we determined was the goal. Suddenly we are at this point now where AI is coordinating. It's coordinating with each other. AI agents are coordinating. That is a big threshold moment. So I think of this, it is a one-way door of handing over that control. People talk about there you could do an off switch, but unfortunately humans are easy to trick and a super intelligence could trick a person that's controlling that off switch just like we trick a child. And so we may not be able to bring it back. Actually think of an analogy to a plane. Imagine if you get on a plane and somebody and all the engineers building it said there's a 10% risk of that plane blowing up. Now would you get on that plane? And you probably would not.

I think most people now when they interact with AI on their chat bots or on their phones or on their smart TVs they know that it's getting better right so they think it's learning because I used to think oh AI it's already sort of improving itself because each time I interact with my whatever I want to watch on TV whichever player it is it gets better and better right it gives me better suggestions so it is learning that way but I realized that's not really AI and that's not really learning. It's just learning me and it's improving its product and it's improving to apply the rules that it already has and knows which were put in by human beings to customize my experience. So it feels like learning to me but it's not true learning. RSI, closed loop recursive self-improvement is where the computer is really learning and writing its own code and creating things that we can't necessarily understand.

Yeah, I mean fundamentally there's a lot of unknowns and so let's be clear. People estimate a closed loop RSI anywhere from three months to three years, but either way, that's a pretty short time frame. So that's when we would expect this intelligence explosion. Now, how far will it go? Maybe there's other limits that we don't know it will get stuck by very soon after that. I don't know, but no one sees any fundamental limits right now. And everyone predicts that we'll get to RSI in months, not decades for sure.

Exactly. I think that the thing we need to do and again some people think it's years but single years not decades. I think

Then if we haven't done anything on a present course, if we're just letting everyone do this without any restrictions on that, if we don't put any guardrails, then I would imagine we would de facto be handing over most decision-making from humans to AI in that sort of time frame.