Is It Already Too Late To Control AI? | Jack Clark, Co-Founder of Anthropic

The world we're in is one where AI companies are building the equivalent of nuclear power plants. And we just had our first case where when we upgrade the plant, a nuclear bomb also falls out of it. You say, 'Uh-oh, this has some implications.'

We're entering an era where you actually need to do serious coordination, including between governments, to keep having all gas, no brakes. It's not sustainable to keep this going. We have to figure out that coordination both within the industry and maybe more importantly and more challengingly with China. If everyone gets access to it, you're now rolling like a whole bunch of loaded dice with the lives of people around the planet. We have to somehow force a larger conversation on what do we want to do with this technology and how do we want it to be developed.

Thanks for having me.

Absolutely. It's great to be on a British podcast.

Gosh, I didn't think it was you that would be doing the purity test for me. I think my credentials are pretty good here. I mean, I grew up in Brighton. My mother was a nurse. My dad was a grumpy guy from South Shields who had become a copywriter and, you know, went to state school, primary, secondary, college. And then I went to the University of East Anglia. So, I've been English and being rained on for my formative years.

I studied literature and creative writing, which was a bit of a curveball because in school I did very well on the sciences and I got D-grades in English. So obviously I was like, you know what, this is telling me I'm going to make my fortune with creative writing and studying literature. But I'd always been a huge reader and I was very interested in going and getting an education which was actually very much oriented around self-study and reading and grappling with ideas through the form of fiction, which is my main hobby and always has been.

This would be 2006 or so.

So, I'd been reading science fiction from when I was about the age of 11 or so. And I'd always been very interested in studying different parts of science from, you know, how ant colonies worked to I briefly wanted to be a town planner. So, reading about cities and trying to model out cities with very basic computer programs to fractals and everything else. And I have this memory of when I was about 14 writing a story in which ants simulated in a supercomputer eventually tried to bootstrap their way out of the simulation by building like an AI to break them out. And then I read a story by the writer Greg Egan called Crystal Knights.

Which is about crabs in a computer building technology to break them out. And I was like, damn it, he's done it too. But it was because I was just already at a young age very obsessed with technology and what it meant. What I spent my time reading and writing about at university was what kind of stories can be told grappling with the implications of technology and how do you make it interesting because technology breaks a lot of narrative in a bunch of ways.

I had no illusions about my chance of making it as a fiction writer of short stories in the mid-2000s. I wanted to be a journalist because I was obsessed with technology and studying it and writing about it. I felt like the most important story in the world was what was happening even back then with things like data centers and the buildout of them. I became a technology journalist in 2009. And I remember, you know, I was reading papers about databases that were being used by Google to train machine learning models. I remember talking to the people that made that software that you click on that shows you're not a bot on Google where it says click all the bicycles. I interviewed them in 2009 and I said this is very interesting. When are you getting bought by Google? And they were bought by Google 3 months later. So I felt like I was on the money.

Yeah. So, I moved to Silicon Valley in 2013 to write about artificial intelligence. I'd become obsessed with it. It was clear to me that this was where the research was happening. In Silicon Valley, I started attending conferences, reading about neural networks, reading about deep learning. And pretty soon after that, you run into someone like Demis Hassabis, who at that time had a small startup called DeepMind that no one had heard of. But shortly after DeepMind was bought by Google in I suppose 2014 or so, I had gone to a conference in New York where Demis was and I introduced myself by explaining to Demis I'd read a lot of his research papers and sort of quoting them to him which I think led to him tolerating my interest as a journalist. And I pitched him and I did a lengthy interview a few months later and I asked him what he thought should ultimately be done with AI technology if we succeeded in building the sorts of powerful things which were then being imagined. And he said he thought it should be sort of managed by or supervised via the United Nations. That's on record in a Bloomberg Businessweek story from around that time.

Just to find this would be what my colleagues in Silicon Valley call cope. But no, I like to think that having a journalist mindset means that you ask a bunch of unusual questions. You're trained to ask questions. You're trained to be skeptical. I've always found that a lot of the work that I've done which involves supervising technical teams to ask like off-the-wall questions has been heavily informed by my background as a journalist where you're just trying to figure out what are the interesting questions to be asked. You know, in the early days of Anthropic I asked could AI manufacture bioweapons and we set up a team to do that and now I'm running a team of economists asking well what impact will AI or potentially very powerful AI systems have on the economy. I think just by asking those questions you might get some good answers.

Oh, I felt completely bizarre. You joined OpenAI and you remember talking to Greg and Ilya on the first day and we were figuring out what the whole job was. And soon after joining OpenAI I said to Greg Brockman, I think I'll just start going to DC and doing policy. He was like very supportive of that. And so I got on a plane to DC not knowing how to do policy but figuring out that it was going to be very important.

I was among the first, you know, 25 or 30 employees. Like Dario had joined a month or two before me if I recall correctly. He was on my interview loop when I interviewed there.

I'd been reading about all of these AI research papers and when OpenAI was announced, it's hard for me to describe just how unbelievably stacked that team was and how notorious they were. Not notorious to anyone who hadn't been following it. I was saying to my colleagues at Bloomberg, I was like, 'Good lord, they got Ilya Sutskever.' Everyone staring at me. I was like, 'Oh, and Dario Amodei.' And no one, Greg Brockman, no one knows who these people are. All they knew was like Sam Altman who was somewhat famous at the time.

Yes. You know, Ilya Sutskever had been on the team up in Toronto with Geoff Hinton, another famous Brit who had cracked the ImageNet challenge, an image recognition challenge in 2012 by training a system using neural networks on graphical processing units made by Nvidia which subsequently became quite important. This is something Matt told us about in the where we've been doing this AI series together where he explained this lovely moment where suddenly this is incredible improvement in performance. I at the time had been making charts as a journalist plotting performance on machine learning benchmarks. And I remember putting the chart in and being like oh my goodness it's happening. I have to move to Silicon Valley right now. So Ilya, notorious for that, had written many foundational papers at Google already. Dario had studied some of the early scaling work on things like speech recognition initially at BBN and then he'd worked at Google and written a paper called Concrete Problems in AI Safety grappling with future safety issues of powerful AI systems in 2015. And it's hard for me to express how like completely berserk it seems to be writing papers about the safety issues of powerful AI systems when all you have are computer vision systems that just barely work some of the time. And then there were, you know, Greg Brockman who was known as just notoriously one of the best, most talented engineers in the world. He'd helped found Stripe which is an amazingly like fast-growing big startup now and there were many, many other names like that. So from my perspective I was like this is the team, it has a mandate to try and make powerful AI systems for the benefit of humanity. If there was ever a moment to stack all my chips on AI it was then. So, I resigned from Bloomberg and I got many emails that day and comments from people I'd looked up to saying I was making the worst mistake of my life.

I think even at the time it was not that well-known. There were a bunch of adorable misfits and weirdos who believed in what was going to happen with AI but it was not even that high status or well-known in Silicon Valley initially and actually you were competing against DeepMind which was already very established. So there was question...

Yeah. How are you going to possibly, you know, build another lab when Google has DeepMind?

Exactly. And you know, OpenAI was announced with a $1 billion funding amount from commitments from Elon Musk and others. And so it was serious about the capital requirements. Now then I get there and you realize there isn't a plan. You're like, what's the plan to build artificial general intelligence and no one knows because no one has an idea of what to do at the time. You don't have language models. You were running an amazing series of experiments at the company which were basically all oriented around ambition. Can we train an AI system that could beat humans at a complex video game? Can we train an AI system to operate a robot hand? Can we train an AI system to end up being able to generate text? Now, that last one turned out to be very important. But all of the other projects were important for having an organization that was continually doing unbelievably ambitious things that seemed like outrageously bold goals. And through trying to work on those goals, you built bigger systems that had ever been built to work the problem. You developed infrastructure that hadn't existed before. You encountered bugs at the frontier of scale which no one had encountered. And you also started to build up this information of, huh, every time we dump more compute into these systems on these outrageous challenges, they get better. So you started to develop this important intuition that actually we could scale the performance of AI systems not just through having clever ideas as human researchers but allocating more compute to the training of them which turned out to be one of the most important insights for what subsequently has happened.

Taking safety of AI systems seriously requires you to hold in your head how powerful they might become in the future. And I think that this notion of the technology not being as it is today but moving very quickly to become a lot more advanced has been both important for motivating the research agendas because you build a different research agenda if you believe something is about to happen in the future, some change up in scale, but also safety has ended up being extremely coupled to our ability to deploy this technology. You know, as the technology gets more powerful, deployment into society is gated by, oh well, if it's good at hacking, how do you make sure that you can deploy the good parts of coding but not hacking? If it's good at biology research, how do you stop the proliferation of bioweapon capabilities, but how do you allow useful biology to happen? These are very subtle questions that drive very, very complicated research agendas. And so I think thinking about safety allows you to think about what I think of as the most like ambitious technological optimist version of the tech which is where you've been able to deploy it very, very broadly.

I would think of it more that the great thing about the world is you get to run a range of different experiments. I view all of these as organizational experiments. There's an organization that was already within one of the large tech companies trying to build AI. There was an organization that started as a nonprofit and then converted itself into sort of a for-profit for capital purposes which is OpenAI that is trying to build it. And then after a few years of sitting around, you know, working together, I think we realized that we had our own vision for how to build a safety-focused organization. And we viewed it very much as the time to build like run another experiment in this domain is now because back in 2020 you can get a sense of how expensive the whole project of building AI is about to be. OpenAI started in 2016 with a billion dollars. You're sitting there saying uh oh it's 4 years later we're going to need a ton of money to do this. So you had to do it then. But I wouldn't claim that it's because you're claiming on our end oh this approach is definitely wrong. It's more we have an approach that we think is going to be subtly different and we want to like run the experiment in the most pure way possible. Just form a company.

I mean, I'd say there was an era where you wanted to run a range of experiments on different organizational designs, different research agendas. That era has happened. I think we're entering an era where you actually need to do serious coordination including between governments. It's great that China and US actually recently discussed AI at the high level summit but also you're going to have to bring companies and governments and other parts of society together to talk about this because the era to keep having all gas no brakes is probably entering the rearview mirror, right? It's not sustainable to keep this going and we have to somehow force a larger conversation on what do we want to do with this technology and how do we want it to be developed.

Well, now we have the UK AISI evaluating both Mythos and GPT 5.5 from two separate companies on cyber security challenges which the UK AISI has built in partnership with the UK intelligence community.

An AISI is the AI Security Institute, formerly the AI Safety Institute, but they changed the name.

Yeah. And the AISI has built tests, cyber security tests which neither of our companies has seen. We, our technical staff, believe in the legitimacy of the tests AISI has built because we trust the talent there, which is extraordinary. And the AISI has built a test that governments around the world can trust for cyber risks because it wasn't built by a company. It is them impartially testing our systems. If you'd said to me in 2020, we're going to build an entirely new function within the UK government that does frontier testing of AI systems, the most powerful AI systems that will ever be built, and they will invent their own tests that won't come from the companies. And these tests will be better than the ones the companies build. I would have said that's impossible. There is no way the government can build that function. And yet it has and that function is being replicated in countries around the world. It's being replicated in the US. They could do with more money in the US and we'll get them back.

Well, I mean, two things I'd say. One, it's extraordinary how quickly the amounts of money have changed. You know, I remember securing the first 100 million for AISI or what became AISI in 2023 and it seemed like a colossal amount of money at the time and obviously now it's not. But I do think and this is slightly off topic, but I think it's worth saying. I do think that to me one of the lessons is that when governments actually have a real clear mission with a degree of urgency and top-down support from the prime minister, both Rishi Sunak and followed up by Keir Starmer, actually there are a lot of people even people whose opportunity cost is earning huge amounts of money in big tech firms that are willing to take up the challenge. So I'm not saying hiring at AISI has always been easy, obviously don't work there anymore, but I definitely think the caliber of talent that you've been able to get through the mission has been very impressive.

It's because you have common standards on things like aerospace safety and testing authorities which exist in each of these different countries or sometimes at a regional level and planes are able to take off in countries including countries which are at war with one another sometimes and land in each other's countries because you have reciprocal technocratic standards organizations that are actually facilitating some shared notion of safety. Now the things that like the UK AISI which have been replicated in the US, China has its own efforts here, other countries have their own is exactly the beginnings of what you need for some of this notion of what standards look like. Now is that going to cover everything? No. For some of the larger risks, you're going to need bilateral or multilateral agreements between countries and high level diplomatic discussions.

I would say the question of what you do about proliferation of national security capabilities that touch on cyber or bio or like nuclear is exactly the kind of thing that traditionally has been the domain of nations trying to talk with one another about non-proliferation regimes, safety regimes, testing regimes. That is clearly an area where we're going to need to have some agreements with people eventually.

The best way to think of it is that biology is inherently dual use, right? It's the science of what our bodies are made of. Now, the experts that can build things like vaccine candidates also have the same expertise needed to build things that could cause terrible havoc in the realm of viruses. Why don't they? Well, the world doesn't incentivize for this. The world doesn't want this to happen. And also, there are things like biological weapons treaties. This has been a topic of discussion among governments for a long time. And also the basic goodness of people. There aren't that many people that want to just visit harm on others. There are some, but not many.

Well, the number of people, this gets to the point, right? The number of people that want to visit harm on others in the world is relatively small for this kind of horrendous act. And they typically aren't trained biologists. And if they are, carrying out acts of terrorism usually doesn't require you to be a lone wolf. It requires some amount of coordination. Now the risks of AI systems are AI systems are universal educators and if you take either an individual or small set of people that want to commit some act of bioterrorism and they have the ability to access a universal educator which is versed in every aspect of biology then suddenly those people have been accelerated and they've been accelerated without paying the coordination costs or conspiracy costs that typically allow us to find groups of maniacs in the world. Additionally, at the state level, states are controlled in this domain by many agreements between states to restrict this area because there is no interest in things that have the potential for vast collateral uncontrollable damage. States prefer military capabilities that can be a lot more targeted. A bioweapon is almost the definition of a thing which is very hard to target and has huge spillover effects. So for bio, you need to basically solve the two challenges. One, how do you make it hard for individual actors to access knowledge like street-level AI that allows them access to knowledge that would allow them to cause harm. And I think that problem has been worked on by industry and government for many years now so far with some effectiveness. The harder challenge in our future though is what happens when we have capabilities that are the same as or better than the best single group of biologists in the world which is what we have right now with systems capable of cyber offense. And then the question becomes okay how do you think about access at the state level or the corporate level for these capabilities which are now transformative at the high end as well.

Anthropic trained a model recently called Mythos. It is a standard AI model that is, you know, uses standard techniques, same as the ones that the other frontier labs do. And it's very, very good at a range of skills, including aspects of cyber offense and cyber defense, hacking.

It also would be quite good at writing Shakespeare. It's great at creative writing. It's great at coding. It's great at biology. It's similar to a few months ago where AI systems got good enough at coding that suddenly loads of programmers started using them. It went through some hard to predict point of being sufficiently good at cyber that it gets interesting from the point of experts.