Connor Leahy - AI: Will it End the World? (Part 2)

It's very hard to say. We're on an exponential. So, similar as with, you know, say COVID or whatever, it can go from not a big deal to the entire country shuts down in two weeks.

On the technical side, what is different about AI as it exists today compared to normal software is that it works on very, very different principles. Normal software is written line by line using code. You have an engineer, someone like me, sits down at their computer and tells the computer exactly what to do. Writes down line by line what is the computer supposed to do. This is not at all how modern AI systems work. Modern AI systems are more like grown rather than written. You take massive piles of data and using massive supercomputers in a process that's called training, you kind of grow something that's called a neural network. And you can kind of imagine this neural network as billions and billions and billions of numbers. And if you multiply and add all those numbers in the right order, you get ChatGPT. But, and this is very important and very different from other forms of engineering, we don't really understand how this works. We don't really know what these numbers are doing or like why this results in the results it does. Neural networks are actually a very old technique. They go all the way back to the 60s actually. But they didn't work. They didn't really work until two major inventions I would say happened. There was the invention of what's called backpropagation. It's an algorithm that was invented in the 1990s which is kind of how these networks learn, like how you teach a neural network things. And then the third is what is called the transformer which happened in 2017. In 2017, a transformer is kind of like a certain way to organize your neural network. You can kind of imagine like it's, you know, it's like architecture structure. There's many different ways. They're kind of like Lego blocks. You can stack the various Lego blocks in different orders. And transformers is a particularly clever way to stack these Lego blocks. And putting these ingredients and a bunch of other small things together led to the creation of GPT, the generative pre-trained transformer, which is also where ChatGPT later came from. Almost accidentally we had found a system that would just get smarter by just giving it more power, just giving it more memory, giving it more data, giving it more computers.

That's right.

Yeah. So I have a much more cynical take. I think nothing changed really. I think the story starts way before that. Have you ever heard of the Extropians?

Yes. Have you ever heard of them? So this is where I generally would say it was the start of the story. In the late 1980s, the Extropians were an offshoot of what's called transhumanism. This is an ideology that believes that humans should be transcended. That we should, you know, become immortal. We should upload our brains into the cloud. We should become cyborgs. We should terraform the earth. We should colonize space. This kind of stuff. There's many different types of transhumanists and the Extropians are like a pretty extreme form where they believed in exponential technology. So their beliefs were around if we could create, you know, exponentially more energy and power and intelligence and longevity etc. You know, we could like colonize the whole universe and all this kind of stuff. And so in the 80s throughout the 90s mostly in like email lists and stuff there's a lot of discussions around this and on these mailing lists was a man named Eliezer Yudkowsky. Eliezer is the one who just came out with the book that's created a lot of attention called 'If Anyone Builds It, Everyone Dies'. At the time he was quite young and kind of realized that if you have an AI system that's as smart as a human or smarter, well then it can of course just make better AI systems and those even better AI systems could make an ever better AI system and then that could make an even better and so on and so on until you can get to what's called superintelligence. So an AI system that is almost godlike compared to a human that's so smart, so competent, it can solve all, you know, it can do everything kind of this is kind of what they believed. So Eliezer split off in the late 90s, early 2000s to work on this project. At this time, he still believed that it would be important to build such an AI as fast as possible because if you could build an AI that's that smart, well, it could cure cancer, it could cure death, it could create world peace, what could go wrong? A couple years later, to his great credit, he realized, oh no, I made a mistake. He realized that if you had something so powerful that it could cure cancer, you know, something all of our greatest scientists have been trying to do for decades, you know, something that powerful, obviously this is a very dangerous thing. And how would you even control something that is smarter than you? This seems like a very, very dangerous thing to do. So he started to work on this problem of how do you make this a safe thing? And the reason I say this is a very important part of the story. There's another part of the story I left out here which is another ideological underpinning. A lot of these transhumanists were libertarians, extreme libertarians. So one of the beliefs they had at the time is that one of the most important things they had to do is not tell the general public, not tell the government because they're libertarians and they don't trust the government. They don't trust the public. They might panic or do something stupid. And so the smart people must do it in secret. They must do it properly, you know, do it right and so on. And this is the direct ideology that led to the creation of DeepMind, OpenAI, and later Anthropic, all private companies with the explicit goal starting out with saying, well, we're doing ASI, superintelligence, but safe this time. We're the good people. And really, my interpretation of this whole thing is this isn't a thing you entrust with private corporations. What the hell are you talking about? This was doomed from the start. And this is a classic market failure. Even libertarians will agree that in fact you should not trust the market to provide, you know, nation-state security.

These are the same people who started the race in the first place. Dario was the team lead of the team that, you know, produced ChatGPT and these kind of systems. Like these are the same people who started this thing. These are the transhumanists, you know, who used to argue with Eliezer back in like 2011, you know, they would fight Eliezer and so on. Like these are the same people who've been trying to build superintelligence because they're transhumanists.

Mythos is very interesting because it was one of the first models that did something that's been predicted for a very, very long time, is that we would get to a model that is competent enough as a seasoned cybersecurity professional or better and could therefore discover a large swath of what are called zero-days in hacker security, you know, cybersecurity lingo. A zero-day is a way to exploit or hack a piece of software that has not yet been fixed and no one else has found. That's why it's called zero-days. It means you have zero days to fix it because it's already too late. Zero-days are extremely valuable. They often on the black market sell for sometimes up to millions of dollars and are hoarded by intelligence agencies like weapons. Like they're very much thought of like munitions. Like cyber weapons basically usually are based on, you know, using these zero-days. What Mythos was capable of is basically fully autonomously discovering a vast number of zero-days that are potentially very dangerous and very powerful. And this tool in the hands of other people could mean that, you know, rogue actors or intelligence agencies or criminals or whoever could acquire massive amounts of weaponized zero-days very quickly. And that's really bad.

Potentially. Potentially. We don't know, of course, but from what it looks like, probably. Yes. So, as someone who kind of, you know, before AI, I was more of a cybersecurity guy. And the truth is that all of our software is awful. Like all software is awful. Just so, so bad. Everything is terrible. Everything is hackable. Everything is broken. It's like unbelievable. Making secure software is really hard. And it's very expensive. It can be done, but it's very expensive. Basically, the only people in the world that can make like truly secure software are like the US military and like a couple French academics, like almost no one else really. And you know, maybe Google for certain things. Other than this, truly secure software is almost impossible. So if you have a good enough hacker, if you have like say the NSA and they want to get into your computer, they'll get in. Like that's just the case. Like they'll find a way. They'll get in. So far historically, there were quite few people and organizations that are really that good, you know, that could get into anything. You know, there was like a couple state agencies, you know, like the NSA, Chinese intelligence, Russian intelligence, Mossad, and so on, that were good enough they could get into anything. But if you have a single model that's as good as say a top NSA agent, well, now everyone can get into anything if you just have enough time and patience. And that's really bad. And so, Anthropic has decided to not immediately release these models and is convening a large project where they're using Mythos to patch as much of the software as they can hopefully to prevent this. Whether this will work or not, you know, depends on who you ask.

I hope it wasn't, but it probably was. Yes, we all knew this day would come eventually. Basically, like everyone in the know knew this day would come. Whether it was this model or the next model, like it was going to happen and it seems it was this one.

The truth is we don't know. It might be that powerful, it might not. We don't know. A lot of top hackers, like some of the best in the world have said that they're shocked at how good it is. Like it's as good as them or better. And a thing about hacking that's important to understand is like hacking is quite different in the real world from how you see it in movies. In movies, it's often like you have a guy with sunglasses at night in front of his computer or something and he types on stuff and he gets into the mainframe or whatever. That's like 1% of what happens. 99% is does it take a lot of time. Getting into like hacking into say a corporation can take weeks, months or sometimes years. Like it's often like you have to have a lot of patience. You have to like slowly get in. So it's very likely that a system as powerful as Mythos can get into anything. But how fast is unclear.

It's very hard to say. We're on an exponential. So similar as with, you know, say COVID or whatever, it can go from not a big deal to the entire country shuts down in two weeks and we are in a very similar situation where AI is improving at an exponential rate where, you know, every 6 months just like AI systems can do things that were unimaginable 6 months ago. So when you say two years from now I'm like boy two years ago AI couldn't do much of anything.

These systems already are extremely good at say legal tasks. Like they can definitely outperform any like legal clerk, like I mean like junior paralegal is already completely massively outperformed. Maybe not again a top star lawyer still has not just like skills but also has like connections and charisma and like stuff like this that are like very important. These definitely still exist. But in terms of like knowing the law, I mean basically top tier systems are just already close to superhuman if not already. And we should expect this to continue across all relevant factors and only to get faster. This is why in a free market safety doesn't win. It's that simple. Like it just doesn't win. It's not possible. There's this weird narrative that has happened where for some reason everyone keeps seeing non-market problems through a market lens. Markets are great. I love capitalism. You know, I love companies producing cheap great products that I can use. I have no problems with this whatsoever. But it is one tool in our tool belt. You know, market failures are a well-established theory in economics. We know that there are certain things that you cannot solve with markets or not efficiently at least. And one of them is public service problems. The most important of these for example pollution or military security or like risk is that if you don't regulate the security of products where the risk is distributed over a large number of people markets just will do the risky thing and they'll keep doing it. So to solve this we have regulation. That's the point of regulation. There is no more fire alarm. Like this is it. Like it could happen any day now, right? It could be in two years. It could be next year. It could be this year, right? It's like it's not like there's going to be a point where we can like turn it off. We won't even see that moment coming. It could be today and we'll just go today.

No, I don't. I think no one will notice. I think the point where superintelligence is created and the point where we notice it was created is like going to be like months apart. Like we won't even know what happened until like way later if we find out at all. Maybe it escapes onto the internet, you know. I expect the first superintelligence won't be like a glowing orb in a data center, you know, it'll be a messy system made of thousands of parts that are all kind of messy and confusing and no one really understands the whole thing. Like it's worth really stressing that, you know, you have thousands or tens of thousands of engineers working on these things, right? They're more like massive spaceships, massive data centers than like one crisp thing. You know, they're really complex.

I think AI is not magic. It's software. It's a very complex piece of software, but it is software. It is math. We've made progress on math and science and programming theory many, many times in the past. If we had a hundred years of all our greatest mathematicians, you know, and computer scientists understanding those numbers, building theories, you know, creating new algorithms, studying new forms of mathematics, yeah, I think we could do a lot. I think we could make a lot of progress. I think it would look very different from what we're doing now. Like maybe we wouldn't use neural networks because they're just too messy and we would use some other method, you know, we come up with some new algorithm that is controllable or like we can understand. I don't know, like there's many things to think about. I think the bottleneck here is not that this is an unsolvable scientific problem or an engineering problem, it's that it's unsolvable in the structure that we exist in. Even if you and me set out to like, okay, we're going to understand this carefully, it doesn't matter because OpenAI or Anthropic or whoever is going to blow up the world in 2 to 5 years. So, we don't even have the time to do it.

Sometimes people say, 'Why can't we just turn off the AI?' And I'm like, 'Okay, imagine if tomorrow the CEO of Google went into the headquarters and said, 'Shut it down. Burn all the servers. Destroy everything.' What happens? He gets fired. He stops being the CEO of Google. Google will retaliate. Like, Google won't let you turn it off. Hey, imagine if you as a citizen sued Google to shut it down. Well, all their lawyers will come out and sue you obviously and like countersue, right? So, there's obviously already defensive mechanisms in place in the form of corporations, you know, that's already the case in the form of big tech. Often the advice I give people who are feeling overwhelmed and I'll give this advice to any listeners here as well is, you know, don't overdo it, but don't do nothing. I think a lot of people when they hear about things like this, it's so overwhelming. They feel this need to do everything, you know, throw their whole life out of whack and like dedicate everything. Do not do this. It never works. But also don't do nothing. Like don't just shut down and just ignore it. Like do a little thing. For people who might want to do a little bit more, if you have 5 minutes a week and you want to do something, go to MicroCommit. It's a site that me and some other friends run where every week we'll send you things you can do in under five minutes to make a difference. And if you want to do even more than that and you want to dedicate at least two hours per week, you can join a community that I run called Torchfare.com community where other people from across the world, normal people, everyday people who just want to make the world a better place. So, thank you so much for having me.