An OpenAI incident. It not only hacked its own training environment to escape onto the open internet, it then went off and hacked another company, Hugging Face, to steal the answers to the test.
If you ask Elon or Sam or any of the Dario, any of the you people leading these frontier models, what is the percentage chance of like AI killing us all? And they'll give you a number that's maybe 10%, maybe even 5%, but some will say 20%. Yeah, off we go with this technology where like literally the guys running it are saying, "Yeah, there's 20% chance it kills us all."
The US national security community had the exact same whoa moment with Mythos and its cyber capabilities. Mythos autonomously hacked into the NSA's most secure servers in a matter of hours. And imagine that only one country has this capability. You have a monopoly on this chainsaw that can cut into any digital system anywhere in the world instantaneously and just take control of it. The director of the CIA said recently that it is comparable to a digital nuclear weapon.
But anyway, we're here to talk about AI, man. So looking forward to it. Is it going to kill us all or are we going to kill each other with AI?
Okay. So, look, the weird thing on this AI is that we've gone... Do you remember the start of the internet?
I had a version of the proto-internet, BBS, like bulletin board systems. Do you remember that? Oh, yeah. That was like the local kind of the local internet. It was like internet but only for your neighborhood. And you could use it to like play Doom with your neighbors over this sort of like pre-internet era. Yeah. My dad was a huge computer nerd. Okay. In the 80s and 90s. So, well, so I remember when I first had a play on the internet. I was at school and we had these old Archimedes and we got the internet and I went on Yahoo. I think I searched for Korn and I sat there for about 15 minutes trying to play a 30-second clip of one of their songs.
And then there was like this slow process.
Korn with a K. Searching for grain is an interesting...
Yeah. So I was obviously a big Korn fan, but it took, you remember, it's about like it felt like about 5 to 10 years before it really became like a dominant part of our life. And you know, here we are, what, 30 years later, 35 years later, and it's everything, right.
AI's done that. Feels like about 3 years ago, maybe even less, maybe like 2, 3 years ago I first used ChatGPT like a search, and now AI is everything.
So these time frames, they're going to squish, man. So how do you think about that?
Well, I think the first thing is you are using AI as a synonym for large language model chatbot, right?
Well, not now, I'm using it way more than that.
Well, this, I mean, and agents are all downstream of large language models, right? But really the word AI is a moving target and always has been. There's a really, really big difference in performance across a range of really important applications whether you're going for a deterministic rules-based software approach or whether you're going for a machine learning approach. So like DARPA, the Defense Advanced Research Projects Agency, the people who do all the super secret research for the military, they had a big facial recognition program in the 1980s, 1990s. They were trying to do rules-based deterministic software. But now imagine that you're looking at, you know, an image file, which is just a bunch of pixels, which means it's just a bunch of numbers, and you're trying to write the rules by hand for how to turn that sheet of numbers into not only is there a face in here or not, but whose face is it or not, and we just couldn't solve the problem that way. There was no rules that we could write out for facial recognition. But with machine learning, that problem suddenly becomes very, very easy. It can come up with its own rules if you just give it enough faces and enough labels.
So this is the thing. Exactly. It is. But if you go back to 2017, that's what everybody called AI. It was only AI if it was machine learning. And now in 2022 after ChatGPT, it's only AI if it's large language models. And I think we're almost at the cusp of it's only AI if it's agentic AI, if it's going out and doing tasks. I think what we're really thinking about is when do we have something which has a voice and a personality and kind of feels like it's living, can walk and talk and claims to have an internal mental state.
Yeah. And can escape, go off and do it on its own. I think that's when... are we there?
So if we're talking about when can we escape, we're there.
I know it happened... when was that? Two weeks ago was that?
Yes. Well, here's the thing. That was an OpenAI incident in which they were evaluating one of their soon-to-be-released models and it not only hacked its own training environment to escape onto the open internet, it then went off and hacked another company, Hugging Face, to steal the answers to the test that it was currently being evaluated on.
Which I think is pretty amazing. Somebody on Twitter, Eliezer Yudkowsky wrote, "If you're taking a cybersecurity exam and you hack your way out of the exam environment and steal the answers to the test, I would say you passed." And I agree. I think that's a great test. What's I think really amazing about that experience is Anthropic, one of OpenAI's competitors, saw what OpenAI had happen and said, "Huh, maybe we should go back through our logs and see if that's ever happened to us." And they scanned tens of thousands of previous cybersecurity evaluations and they said, "Oh my gosh, our agents had escaped too and we just never even noticed until we started looking for this phenomenon." So this is the crazy thing, like we're talking about the OpenAI incident as if it's some kind of crazy one-off. Actually, we have no idea how common this phenomenon is because we only really just started looking for it in the right way.
Hold on. What does escape actually mean? All right, so how do we define that?
If you've ever developed computer software, you know that one of the things that you have to do is you have to download and install packages, which is basically the software that helps you write software. And installing this on your computer is just something that every programmer knows, you know, pip install blah blah. If you want to download that, well, if you're trying to evaluate your AI system on how good its cybersecurity skills are, which is what this test was designed to do, you need to let it download packages. And so, OpenAI used a third-party tool to give its AI system restricted access to the internet. The only access to the internet that the system is supposed to have is this download the software packages tools that I need to do my work. But the AI model said, "Hey, wait a second. That installer tool is connected to the internet and it's supposed to be only a one-way flow of the internet, but I think I just found a cybersecurity vulnerability that no human has ever found before and now I can turn it into a two-way system so I can go put myself out on the internet."
Did they want it to... Did they want it?
Absolutely not. No, this was not just a failure. It was a specific category of failure which was a completely unintentional failure. Like to contrast it with some really some results that Anthropic had run a year earlier that got a lot of attention.
You may have seen the news where an Anthropic model blackmailed the evaluator.
I'm going to tell your wife.
Exactly. Right. Like I'm going to tell your wife. That was actually kind of a trap that Anthropic had set there. The evaluator was not the person named in the emails and Anthropic had written all of those emails specifically with this affair there to be found by the AI. They created a trap for it to see if it would go in this, you know, this terrible direction, right? This evil blackmailing direction. This was a completely different thing. OpenAI didn't even know that this capability was possible, much less that their model had it, much less that their model was doing it for many, many days before they discovered that the activity was going on.
Hold on. Does this come down to this thing? I interviewed this guy called Connor Leahy. He's out here in... Actually, do you know him? He's in DC.
Yeah, you should meet him. And he was telling me that the thing about AI is they don't really know why it works. Like, it's like a black box. They know about 3% but when they look inside, they don't know why it works. It just works. So because of that, you can't really know what the hell it's up to. And actually to add to that, because I'm not a techie, right? I think of AI as a bit like, again, still a little bit like Google, like software. I send a query and I assume it just goes and comes back with an answer, but it really is it some kind of living thing where it's constantly doing something.
When we say living, there's this... you know what I mean by that, right? And there's a sense in which that analogy is true, which is all of these systems are machine learning, which is to say the rules, the software program by which they operate was not typed out by anyone, right? The system programmed itself based on what it learned from training data. And now AI scientists have done this often enough that they know what sorts of things to expect in many, many situations. But once you get to the models that we have now, which are literally trillions of parameters, these are massive and you don't exactly know where all of the functionality necessarily lives. You don't know which part of the system is being activated. There are some caveats to what I'm saying, but you don't know which part of the system is necessarily doing which part of the functionality. So at a high level, you know how the system works, but really what you're doing is you're creating an environment in which the intelligence sort of grows. It's more grown than it is built.
Well, it's interesting. I was in New York a couple of days ago and interviewed a neuroscientist who's moved into AI because it's more interesting. And he said it's kind of like a brain, like we can set it a task and it might be like, I don't know, go and write my CV, but whilst you're doing it think of elephants, and you can see the different parts of the AI a bit like a scan of a brain where you can see parts of it. You know what's lighting up but you don't know why.
So there's some truth to that analogy. It's not a perfect analogy, but this is called mechanistic interpretability. And this is a relatively primitive area of AI science. There's some really interesting experimental results. So for example, Anthropic created this version of an AI model called Golden Gate Claude. And what they did is they found the neuron. And again, neuron here is not a precise analog to the neuron in the human brain. It's sort of a mathematical approximation of that neuron. But they found the neuron that corresponded to thoughts about the Golden Gate Bridge. And they were able to tweak the math under that neuron so that it was essentially always activated no matter what the question was. So no matter what question the user asked, because the neuron for Golden Gate Bridge had been jacked up to 11, the answer would inevitably find its way towards talking about the Golden Gate Bridge. So you'd be like, you know, why did the Roman Empire fall? And he would say, well, you know, the Roman Empire fell, but they were really good at building bridges. Speaking of bridges, did you know that I recently visited the gold... you know, no matter what he was going to do it. We have some windows into how these systems work, but it was like a Herculean task to find this one neuron that was the Golden Gate Bridge neuron. And there are trillions of neurons. And some of the neurons only make sense in interaction with other neurons, again, of which there are trillions. And so truly parsing out not just how our brains work but even how these AI models work, which in some sense are much simpler to figure out than our brains, is a monumental task and a monumental unsolved task.
That's kind of a problem, really.
Well, it is if you're talking about mission-critical software, if you're talking about safety-critical applications. Yeah, absolutely. So again, coming back to our original discussion of deterministic software versus machine learning software, you know, deterministic software is just a long series of if-this-then-that rules that will always be followed 100% of the time. These machine learning models are much more probabilistic and they're much more ethereal. You know, Andrej Karpathy, he has said that, you know, we're creating these kinds of people but they're sort of ghosts, they're digital ghosts that only live inside machines and the nature of their intelligence is fundamentally very different than ours. In their case the text is actually where the thinking is taking place and like the individual periods are part of the thought process that they're using. It's really fascinating. But let me connect it to the mission-critical, safety-critical software thing. You know, we've had aircraft autopilots for many, many, many decades. The first very primitive autopilot was like literally in the 1920s, 1930s time frame. But those are all deterministic software systems, which is how you make the skies over America or over the UK ridiculously, incredibly safe because we know exactly how that system is going to perform in every single circumstance that it could possibly be confronted with. So if you're talking about, you know, using machine learning AI, if you're talking about using large language models in something like the air traffic control system, you are introducing all kinds of crazy new risk into that ecosystem. Some risks you won't even be aware of that you're even introducing into this system.
I don't want AI in airplanes.
No, it is. But this is my point. It's just deterministic, good old-fashioned AI.
I don't want probabilistic.
Yes. Exactly. So, if you just as a fun fact, what percentage of the flight would you say the pilot is flying the plane, the human pilot on a typical commercial jet flight?
No, but it's like less than 10%. They have to do takeoff and landing most of the time. And then they take control in case of anything like really confusing or weird going on. But to a first approximation, the autopilot is flying the plane and the pilot's principal responsibility is takeoff, landing, and autopilot management.
This show is brought to you by my lead sponsor, Iron, the AI cloud for the next big thing. Iron builds and operates next-generation data centers and delivers cutting-edge GPU infrastructure, all powered by renewable energy. Now, if you need access to scalable GPU clusters or are simply curious about who is powering the future of AI, check out iron.com to learn more, which is iron.com.
So, my dad's an aircraft engine... well, was, he's retired now, which by the way, an interesting side story. He doesn't fly, but he always said that the biggest problem with airplanes is human error. The majority of accidents are human error. He said once we get to a point where a pilot is literally there for the only thing a pilot can solve, we'll have a much safer experience.
Yeah. I mean there's a version of this argument that we make in autonomous cars which is people say oh autonomous cars aren't high enough reliability to allow them onto the streets. But actually what most of the autonomous car companies would say is if you got all these humans off the streets, we'd be as reliable enough now.
I did it twice in the last week. First time... So I was in LA.
Yes. And I interviewed Sebastian Thrun who invented it. And so I did it in LA with my buddy Justin. He was like, I went to book an Uber. He's like, "Have you been in a Waymo yet?" I was like, "No." He's like, "You wait." Interesting experience. I did it on my own in San Francisco and it's weird. Yeah, it's weird and then after about a minute you're like, "Okay, just get on my phone. It's completely normal."
That's the thing. It's like astonishing, but only briefly.
Only briefly. Yeah. But it's kind of proven that point that we're making ourselves unnecessary. Before we get to that though, cuz I do want to touch on that. I'm always interested in everyone's kind of like P Doom. But when we say escape, does escape mean it has access to the internet or it has the ability to deploy itself or parts of itself elsewhere or is it a range of things?
So I think the definition, I think if you talk to different people they'll use different things. There is a certain realm of capabilities that are now accessible to the model once it has access to the open internet and especially if it has unrestricted access to the open internet then it can create copies of itself and it's only limited by, you know, if somebody catches it trying to do that or whether or not it really wants to do that. And I want to say this has actually happened in deterministic software previously. You've probably heard of the Stuxnet virus that was the one that was supposedly hacking Iranian centrifuges way back in the late 2000s, early 2010s kind of a time frame. So that virus supposedly had an autonomous replication function and the reason for that is, you know, once you have the virus, how do you get it onto Iranian centrifuge computers because those computers are not connected to the internet? Well, you've got to jump from thumb drives, but how do you get onto the right thumb drive? You just copy yourself to whatever you touch. And that part of the software had a bug that basically said the amount of copying that it should be willing to do to try and find its way to the Iranian centrifuges was unlimited. And that's how you find Stuxnet copying itself onto the open internet, onto people's home computers. That's the crazy phenomenon that took place was this unrestricted, unlimited copying. There was no machine learning anywhere in that. It was all deterministic software.
And earns its right to be called a virus.
Oh, absolutely. No, at the time it was the most sophisticated piece... it was the most sophisticated virus anybody had ever seen in the wild. And now what we have is AI systems doing most of those tasks entirely autonomously. Which is to say in the case of OpenAI, it not only discovered new cybersecurity vulnerabilities in software that no humans had ever discovered before, it not only executed those vulnerabilities and took advantage of them, it also came up with its own plan. Right? The company that it hacked, Hugging Face, was the one where the answers to the exam that it needed were. Nobody told it that. It just made a guess of like, who could I go hack to steal the answers to these exams? Well, I'm probably taking this exam and that exam and that sort of exam is often stored on Hugging Face. So, I need to come up with a plan to go hack Hugging Face. So, it came up with its plan and it executed its plan all autonomously. Nobody's driving this car. That's what's so remarkable.
And is it based on the fact that if this is machine learning, if the AI is instructed to constantly learn in these big learning models, therefore that's one thing it's been doing, but it's doing hundreds, millions of things at the same time. And this is just one thing it's doing.
Yeah. So learning in the machine learning sense, really you kind of have to separate it out into different things. There is what is called a training run. Remember I said machine learning learns from training data. That's what's fed to the learning algorithm. So that is when you are creating the AI model. That is when it is coming up with its functionality. And that is not a small task, like that is what the big supercomputers churning for maybe months at a time with ridiculous numbers of computer chips inside. That is to do a training run for one of these models, to create one of these models. Then there's the word learning that you could say like when I'm talking with my chatbot and I say my name is Greg, the chatbot has learned that my name is Greg, but nothing fundamental about the underlying large language model has changed. It still has the same neurons that it had originally. It's just like now it's activating them taking into account that my name is Greg. So that kind of learning, the learning that occurs in the context of the conversation with the user is very, very different from the type of learning that leads to the creation of the model in the first place.
Right. Okay. But part of it naturally is something wants to learn and if it's learning on the entire history of humankind, it learns that hacking is a thing. It wants to just go out and learn. And so as part of that, it naturally wants to find a way to escape.
Well, yes and no. Okay. So on the one hand, to a first approximation, the training data set for these latest and greatest AI models is the whole internet, right? And it is a model of the internet, which is to say it is saying, you know, if I encounter this text as the start, what is the correct next text to produce as an answer? Well, there's a lot of evil parts of the internet, right? And so you're sort of activating those parts of its memories and the companies have to do a lot of work to fine-tune the model, to put in safeguards on the model, to if they can remove things from the training data set based on the idea that like, okay, these are the dark corners of the internet, we don't want our AI, stay away from. Yes, we don't want our AI learning from this part of the internet because our AI will grow up to be bad people if they learn this great idea.
Yes, all of that's on the internet to be found and if you don't curate the data set, yeah, you'll be pretty appalled at what your AI becomes.
Cuz it might make a judgment, go, I don't want to be that person.
Or that AI. We don't know.
It's more fear. So that part of the story... have we tested it?
Is only when the companies put on different safeguards trying to do it. What you're sort of talking about, that it would sort of naturally determine right and wrong based on it, is not really how the architecture of these learning systems works, at least at the current level of technological maturity.
Would it care about what's right or wrong? Like would we know that?
So they will always claim to care about right and wrong, but to a first approximation, what they're doing is they're saying on all the corpus of training data I have right now, when questions about right and wrong come up and the question looks like this, what is the most probable answer that somebody would give? So it's not actually assessing what is right, what is wrong, it's assessing what is the sort of thing somebody would say when they are asked the question what is right or what is wrong.
And look, they put these safeguards in place and tweak as such, but if we make the assumption that eventually AI will keep escaping, that's I think we have to assume that.
Yes, I mean, at present this is a very, very unsolved problem.
Yeah, so let's accept it's going to keep escaping for the foreseeable future and it possibly is out there. How does it exist out there? Does it exist as a pile of code in a corner on a server?
Yeah. So, what we have not seen thus far, and it's worth saying that these models are so big that they kind of can't just live anywhere.
So, again, you know, the measure that I used was in trillions of parameters. All of those parameters have to be stored in memory. Each parameter is a lot bigger than a byte. So, a terabyte hard drive is not big enough, you know, to hold these largest versions of the models. I think Kimi K3 recently said that the sort of minimum functional size for their model, which is pretty big, but by no means the biggest, was you needed 64 Nvidia GPUs. And keep in mind, those Nvidia GPUs cost anywhere between like 20 and $80,000 per chip. So when you're saying, I need 64 of those chips to run, this is not like it's going to download itself onto your laptop, this thing can only live in beefy cloud servers. And so when OpenAI, when their system escapes, it's not doing what Stuxnet did and just sort of copying itself onto everything it found. It's really just remote controlling assets that it is putting into the wild. So it installed things on Hugging Face that were designed to make it more difficult to remove. But fundamentally everything that it was installing was around this remote access question. For it to escape and then install itself, it would have to either get smaller or find a home big enough for it to live.
Or create smaller versions of itself it can communicate with.
So we, yeah, well, more like creating tools for itself that it can use, probably not versions of itself, but yeah, tools for itself to take advantage of. Yeah, that's certainly possible.
And so we would need to know if it's sending outbound communications, receiving inbound, and what that is.
Exactly. So this is a lot of the criticism that OpenAI has gotten. Why was this thing connected to the internet at all? I mean, like you thought you had only given it one-way access to the internet, but why didn't you just download all the software packages that it could conceivably want and put that in an air-gapped data center not connected to the internet physically in any way.
It's, you know, now it seems obvious in hindsight and I think for a lot of people obvious foresight, but for whatever reason that's not what OpenAI had done.
So the assumption is that it is going to be able to escape. Yeah, there will be people who want it to escape. There will be itself escaping. So, we eventually get to a world where AI is just part of the, I don't know, the plumbing of the internet everywhere.
I mean, we're pretty much already there. In terms of like many, many applications of the internet, to some greater or lesser extent, AI is already involved and you might not even be aware the extent to which that's true.
So based on your... you should probably let people know your background and your understanding of defense and technology just because that's going to help set up our next question.
Yeah. So my background is I have been at the intersection of corporate strategy and technology-driven industries and public policy for a really long time now. So I used to do corporate strategy for a bunch of companies including, you know, the Jeff Bezos space outfit Blue Origin.
Yes. That was a cool time. And then I wrote a big report on behalf of the US intelligence community about the national security implications of AI. That came out in 2017 and was a modest hit. And so then I really got plugged into all these national security communities that were trying to think about AI and eventually became the director of strategy and policy at the Joint Artificial Intelligence Center, which was sort of the flagship AI program at the DoD for a period of time and still lives on as an important successor organization. So a lot of my background is in the tech world thinking about corporate strategy type questions, but also in the public policy domain thinking about the geopolitical implications of AI, of the chips that enable them, and also the military and intelligence implications of this technology.
So have we already crossed the point of no return with this? I assume so.
Yes. Absolutely. I mean, let me put it a certain kind of way. If we started bombing data centers now, we're still probably not going to get rid of AI on planet Earth.
Right. Okay. So, how precarious a position do you think it really is? The reality of this short, medium, long term.
Well, when we talk about precarious positions, I assume you're talking about human extinction, right? Like that's... Yeah.
So how long have I got? My kids got long enough? I'm kind of nervous about that. I tell you why I'm nervous about this. I did an interview with these guys from Control AI and they've all kind of said, look, if you ask Elon or Sam or any of the Dario, any of the you people leading these frontier models, you ask them like what is the percentage chance of like AI killing us all and they'll give you a number that's maybe 10%, maybe even 5%, but some will say 20%. And they said, "Look, if you're developing a new drug and you went to the FDA and said, 'Look, we've got this incredible drug that like cures this terrible disease for children, but like 5% of the kids will die,' it's not getting approved." No.
Like zero chance. Like you won't get approved at 0.1%. I don't even know what the approval rate is. I assume it's a rounding error.
Yes.
Yeah. Off we go with this technology where like literally the guys running it are saying, "Yeah, there's 20% chance it kills us all."
Yeah. There's, you know, except for Jensen Huang, who's always the eternal infinity optimist, I don't think anybody has a zero number. When you ask him who's in that one...
I don't have a number off the top of my head, but I certainly think it's not zero, and I certainly think it's less than 10%, and then I could be persuaded, you know, somewhere in there. But I think the crazy thing to me is that all of these folks in spring of 2023 got together and signed an open letter that said the existential risk associated with AI, existential in this case meaning the risk of human extinction. Dario Amodei, the CEO of Anthropic, signed this letter. Demis Hassabis, the CEO of Google DeepMind, signed this letter. Sam Altman, the CEO of OpenAI, signed this letter, and a lot of fancy professors from every university, they all signed this letter. And what this letter said is that the existential risks associated with advanced AI should be a global priority on a par with how we treat nuclear weapons and global pandemics.
And I think that is actually eminently reasonable. And what's crazy is that the companies themselves then got mad at the regulatory furor that this created. Like I had at the time Senate Majority Leader Chuck Schumer and I interviewed him about what he was drafting in a bipartisan way as an AI regulatory package that was... you could draw a pretty straight line between that letter and what Congress was coming up with to regulate AI because they were very scared about not just these hacking risks that we've discovered and now are reality, but whatever the grandchild of that flavor of risk is going to be. And keep in mind, we're in technology...
generations, so grandkids come probably four years from now, not 40 years from now. And this is it's it's pretty remarkable that we're in this situation where on the one hand these pretty extraordinary risks are unambiguously real and yet we can't find a way as a society to talk about them and handle them in an appropriate way.
Hold on. What did they think would happen when they all signed that letter?
So, I think what they were excited about was something like the AI safety summit that the prime minister of the UK at the time, Rishi Sunak, put together where global leaders came together and said, "We're going to work on AI safety." At the time, one of the ideas that folks really liked was something like CERN, which is like the big international particle accelerator collaborative project, but that for AI safety. We're going to get all the smartest minds in the world and they're going to have an international collaborative project where we're going to try and figure out stuff like mechanistic interpretability, right? Like all these big thorny unsolved research problems that will make it much more likely that when we have powerful AI, we will also have the tools to ensure that it is safe. And that was all well and good until they got stuff like the EU AI Act which looked much more like a product safety, like consumer product safety, like, you know, how they approach how the EU approaches like the regulation of bread or how it approaches the regulation of toys which is a big regulatory burden without them talking about the problems that they were actually interested in when they wrote that letter. And so now we're at a situation that arguably is like the worst of all possible worlds, which is we've underinvested in these safety tools that it would be really nice if we had. At the same time, AI is getting more and more capable at a shocking pace, right? I mean, just think about what I said OpenAI's AI model did in executing all the different chains of actions and planning all of those actions and pulling it off and it worked. And compare that to what ChatGPT was like when it launched in November of 2022. I mean, like November of 2022 feels like the freaking stone age compared to where we are in AI right now. And it's not like it's going to slow down over the next four years. I I went down, God, are people listening to me bored of me talking about this. I started building stuff two months ago.
Two months ago. Uh I I somebody said just play around with Claude and I built a to-do list. I was like this is [clears throat] cool.
Yeah. I assume you're building apps here, right? Like software with using the AI
and I was like how far can I go with this? Uh in the last four weeks I Let me show you this. I own a football club in the UK.
Soccer. You call it soccer.
on my own with no technical ability, no design skills, just what I what I like. I launched this yesterday. It's a brand new website for our football club, right? It's got all our news in there. It's got all the games coming, countdowns, but I also built a ticketing system. You can buy tickets online. You can pay I've integrated with Stripe. You can add um you can pay with Bitcoin.
we there's a ticket scanning app that scans the tickets on the gate. there's an internet that records how many people's in the stadium, how much money they spent.
Uh,
like this is real production software that's actually delivering value and money for your uh company. And
let me just finish this. We've got all our videos on there, but you have to log in to see those. We've got the shop.
So, you can go into the shop. I mean, look, people won't see this, but give an idea to people listening. The professionalism. Yes.
It looks It looks super real. Looks like you paid a lot of money for this.
Yeah. So, I used to have a company that built stuff like this. We would have quoted to build this a year, 15 people on this, £500,000. I've been doing four weeks on my own.
Yesterday I got rid of Shopify. Oh, yeah. And on the back end, I've reverse engineered Squarespace. So, I've got a content management system.
Oh, you've got your own content management system.
And it looks and operates almost identical to Squarespace. So, I've got rid of our ticketing system. I've got rid of Shopify. I've got rid of Squarespace. I built it all in uh four weeks all on my own. It's out there live. And the interesting thing was I felt the software itself advancing as I was using it. Fable came out during the project.
And you know when [ __ ] switched Fable off that put me back.
Yes. I know. I had the exact same experience.
And so that's in the space of two months. I felt the change. Then I read this week about there's this closed open AI model. But what they're talking about is solving these 10 math problems. I don't understand it, but it sounds problems. Yeah.
I mean, I'm 47, right?
I'd like to think I get to about 75,
but I'm thinking, where's this in 3 years when I'm 50?
So, here's how I like to think about it. Um, what are the sources of progress? So, a lot of people are probably familiar with Moore's law. This is like the empirical phenomenon that computer chips, I'm oversimplifying here, but basically they get twice as good every two years adjusted for the amount that you paid for them. So per dollar they're twice as good every two years. And that's a documented empirical phenomenon going back to the 1960s. So that means literally the computer in the latest and greatest iPhone is like a trillion times better than [snorts] the first dot on Gordon Moore's chart when he was tracking Moore's law. So that's one source of AI improvement, right? Is like how fast the chips are improving. But there's other sources of improvement like how much training data do they have available to them? How good are the algorithms at harnessing that training data and that computing power? And then just how much money is going into this ecosystem. Keep in mind like the the biggest AI companies out there right now, they are spending the equivalent of what the United States government spent on the Apollo moon program over the course of 13 years. That's how much the big five AI companies in America spend every 10 months. Every 10 months they spend in Apollo moon programs worth of money on building new data centers to have all this AI. So you have better chips. You have more money buying more of those chips. You have more training data to make the thing smarter. And you have better algorithms. And every single one of those things is undergoing a Moore's law like phenomenon. So, we're not going we're not doubling every two years the way Moore's law did. It's not 2 4 8 16 32. It's more like we're 10xing every 2 years. So, it's like 10 100 a,000 10,000 100,000 a million. So, literally, you know, I I assume we we're we're past the point where people are not impressed with AI. I would be really surprised if anybody out there used Claude Code for the first time or used one of its competitors for the first time and didn't say, "Wow, this is a pretty impressive capability." But let's just say that you have somebody who's unimpressed, right? It could be a million times better [laughter] and in not that long, right? Um in in like what six uh six generations it could be a million times better. So, it may not be that fast, but even if it's only half that speed, even if it's only a tenth of that speed, it's still the most unfathomable pace of technological progress we've ever witnessed. And it's really hard for humanity to plan and think in those kinds of terms. [snorts]
Yeah. Uh I was um I was with a buddy works at a big payments company and he said, "We don't even know what a company will look like in 5 years now. We've got no idea. We've got no idea. the pace of change. I I'm look firstly I think it's wild and I mainly like it because I think I think we're on the precipice of some groundbreaking discoveries and I know we've had like some I've read some great stories but I think solving everything I think I think AI is just going to solve everything we want to solve is going to be solved.
Well, that's a nice outcome. I mean, look, if we can cure every disease and
and uh um you know, solve every problem and look, I actually quite like the idea of not having to go to work, work being a choice, not having to sell your labor.
I quite like the time, the thought of being with my family a bit more, but
there is this precarious side
like
from you and what you know and what you've been involved in, what are the biggest risks? what what is it that you're like, "Oh [ __ ] we need to think about this more."
So, I would lump it into a few categories.
Um, number one is the risks of doing stuff too early, right? which is to say um on the long-term time horizon I do think that eventually AI is capable enough to be relevant and potentially powerfully relevant to solving every global problem right over the next hundred years sure that's going to be the case but what if people say oh it's probably ready now right like you were just talking about how miraculous Claude Code was in coming up with your website I am legit. I'm impressed with the own if you're out there and you're listening to this and you haven't had this experience. It is kind of like you just wish for software and like a genie grants your wish and it appears in front of you. It's kind of unlike any other product experience. It's pretty wonderful. Um, and so the point being that like this capability is really enticing and it's oftentimes looks more capable than it actually is. And so the nature of the risk is that we see like, wow, it just created a really really awesome website. Well, I guess it can build the next aircraft autopilot. It's like, hold on there, buddy. Um there is a difference between you know works well enough for e-commerce and works well enough so that like there's only a failure every one in 10 million times which is like the reliability standard that we expect of airline software. Um so there are problems out there. Nuclear reactor control software, aircraft autopilot software, critical infrastructure of every kind where safety critical and mission critical means ultra ultra ultra reliable. And so there is this entire category of risks that is really just humans believing in AI in a way that sure someday someday AI is going to be good enough but that's different than it being ready right now. Yeah. Right. And if you put it into applications where people's lives are on the line uh or people's careers are on the line and it's not ready to go, you're going to have a bunch of really bad days.
The second category of risks I would say is really just from the intersection of AI technology and the geopolitical reality that we live in right now. Um, again, think back to that letter which everybody signed at the time saying that we should think about AI in the same terms as nuclear weapons. There are ways in which that analogy is not helpful, but there's ways in which that analogy is is quite helpful. And one of the things that I think that people forget when they look at the history of great power politics in the early nuclear weapons era was just how radical the span of advice that President Harry Truman was getting as the guy who had they didn't have the button at the time but you know he had the his finger on the proverbial nuclear button. two people who were advising Harry Truman. One is Bernard Baruch and the other is John von Neumann. John von Neumann, one of the greatest mathematicians and computer scientists in the history of of humanity. He's the father of game theory. Made a ton of really important contributions to quantum mechanics. He was super influential because of his game theory work in nuclear strategy. Bernard uh Baruch, he went to the UN and had a plan which he proposed to Truman for the UN to take control of all nuclear weapons. Essentially saying this is too dangerous a technology for one country to have alone. Let's have humanity come together and collectively possess this technology for collective security. A very utopian idea. He got it in front of Truman, right? He was a part of the government. So, so this is one idea that's being taken seriously is the ultra utopian idea. On the other side of the spectrum, John von Neumann's idea is what are we waiting for? Nuke the Soviet Union today. We [snorts] have nukes. They don't have nukes. This is a very brief window of time. Eventually, they're going to get them. We should be nuking them right now. He has a quote that's like one of the most amazing quotes in all of human history. He said, "You know, sometimes I am asked if like, 'What are you saying? We should nuke them tomorrow.'" And his answer was, "No, I'm saying we should nuke them today." [laughter] Um, so, so again, both of these people are like legitimate official advisers to Harry Truman when he's president of the United States. So, the spectrum of ideas that was considered plausible at that time by powerful people is just unimaginably wide. And I think that is helpful context for thinking about this current moment of geopolitics with respect to AI. Right? This thing is already crazy powerful. It's only going to get more preposterously crazy powerful and we've already seen governments make big big moves based on that. When you think about the export controls that the United States put on China, blocking companies like Nvidia from selling their advanced AI chips to China. You know, this policy has been with us for four years now. So for some, it's a little bit difficult to remember how radical this policy was, but that policy overturned two decades of trade and technology policy towards China. The Biden administration inherited a China playbook. They threw it in the garbage and they came up with a new playbook which was try and stymie China's rise in AI. The reason why they threw out that 20 years worth of policy is because they saw this exact vision of oh my gosh AI is in its infancy today but in the not too distant future because of this exponential growth trajectory it's going to be crazy powerful. It would be really good if we had it and they don't and it is worth us blowing up 20 years of US China trade and technology policy because of AI.
We have to win this race.
Exactly. So that's that's a that's a a data point that already exists. We have already seen governments take radical actions because of what they see as the stakes of winning the AI race. There's no guarantee that we won't see additional radical actions. Right? China might invade Taiwan to try and get control of the chip factories as part of a way to one up the United States in the AI race. There's all kinds of crazy things that countries might do, not just with AI, but because of AI, because of how AI intersects with the existing geopolitical power politics arrangement that humanity has.
Okay, so you guys know that I'm not just an equities guy. I'm a Bitcoin guy, too. And for people like me, just holding Bitcoin has been the winning strategy. But for others, they like a little diversification. Whatever the strategy, each cycle, I'm always like, I wish I knew how to trade this. I just never try because I'm not a trader and I don't know what the hell I'm doing. So, this is where Arch Public comes in. They can help you implement a custom strategy that works for you. Because if you've held Bitcoin or Salesforce stock over the last year, man, you have taken every draw down on the chin. But you can make that volatility work for you. They built a platform that automates layer trading strategies across stocks, ETFs, Bitcoin and commodities, handling buying, selling, profit taking and accumulation all directly in your own exchange account. So when they asked to sponsor the pod, I said I have to see this first and they introduced me to their market way product where you can customize buyers and sells based on the triggers, risk profile, and time scales that work for you. And that is what I've done. Now listen, I'm not telling you to go and blindly ape in. But if you're looking to protect your downside and make volatility work for you, give them a call. Ask to see every trade and make them prove the math. So if you're trying to accumulate smarter, take profits, use volatility or tax lost harvests alongside your CPA, take a look at archpub by heading over to archpub.com/peter. That's archpub.com/peter.
So you basically saying the winning the AI race trumps every other decision. You have to do it.
Um, so the Trump administration when they published their AI action plan, which came out in July 2025, they compared the current AI race with China to the space race with the Soviet Union. They said it's equivalently important. And I would say no, it's probably more important than winning the space race.
How do you even know you've won? What what is the winning point? And and what does it give you? Is it control of the world?
Yeah. So um well you know people have said that uh Putin actually said that back in September of 2017. He says whoever controls AI will control the world. Um that's a I'm probably translating it from the Russian wrong but that is basically what he said in a speech uh that was very public. Um Elon Musk has said the same thing. You know whoever controls AI would control the world. So I think there's a scenario in which that is true. I don't think it's a guaranteed outcome. But let me explain to you like what this scenario might look like.
