Good afternoon, and welcome to what is very much not the graveyard shift of Civic Future Day 2, but perhaps the most important conversation that we might have today. And we're going to be discussing AI sovereignty, what that means for the UK, and the various ways in which that might be achieved. We're going to have a bit of a conversation and then branch out into questions. So do get thinking. Joined for the conversation by Sarah Hunter, and Sarah's worked in government in the UK, led public policy in Google X, and has various important non-exec positions in institutions that are important in the UK, and then Tyler Cowen. So Tyler is professor of economics and one of the world's leading thinkers on AI and its impacts on society and in the economy. I'm going to talk about AI sovereignty, and I suppose a sensible place to start with this is thinking about what the term means and how the two of you understand the term. It's a conversation that people have, often disagreeing about what it means. So, Tyler, maybe I'd start with you. What do you think AI sovereignty actually means?
I don't think it's a well-defined concept. I'm not sure such a thing exists. If you assume the normal definition that somehow the UK controls its own AI, there was a recent Substack by Anton Lane and he tried to estimate what AI sovereignty would cost. First of all, it requires a coalition of midsize nations. You were just in a coalition of midsize nations called the EU. You felt you didn't have sovereignty in that, or at least some of you felt that way. So is that right off the bat, is that even sovereignty? But if you had such a coalition, Anton estimates it costs what, like $500 billion over four years. I mean, is that going to happen? And there's still no guarantee that it works. And he says the personnel costs could cost $65 billion again over four years. And are you in the position where your procurement will be rapid and your public sector will tolerate the notion of people showing up, you know, not as CEOs, just as researchers, and pulling in 10 million, you know, pounds a year in salary plus options or other rewards? It just doesn't seem like a realistic scenario to me.
So I think the best most midsized countries can do is to be useful to the world in other ways and increase their bargaining power. They made decisions a long time ago that if something like AI were to come along, they would not be sovereign. I'm not even sure America and China will enjoy AI sovereignty. The companies, the leading labs in both places now have a lot of bargaining power. So, you know, what does sovereignty mean? Like who's the we that's supposed to be in charge anyway? So I think often the discussion does better by sidelining that concept and just trying to figure out what are some good concrete steps we could make that would be useful for the UK or the US or for AI or for users, whatever, and just marginally hope you keep on getting into better and better positions. UK in particular, again, you did this thing called Brexit, you had all these plans for whatever you would change, some of you may agree or disagree, but as far as I can tell you didn't do most of it. So like AI aside, you didn't prove that much sovereignty to begin with. I'm sure Dominic Cummings is a fraught figure in many circles, but I read his Substack. What he says about UK government, as far as I can tell from my distant vantage point, but it's basically true. So I'll just say like, start with sovereignty and get a bunch of good things done, and then like two weeks later, start worrying about the AI, but first show some sovereignty.
Guys, so much to say. The starting point is I really hope our AI strategy is not like our Brexit one. I really hope we have more success there than we did on that. I basically agree with you. I think it's massively misused as a term. It seems to mean a lot of different things to a lot of different people without any specificity. And it's not clear for what purpose. You know, what does sovereignty mean and why does it matter? No one's really like nailed that. So I think my starting point is let's redefine the term in a way that is meaningful and useful. And for me, yes, I think there is a value in having a stronger sense of optionality and agency over the AI that is used in this country. And by AI, I don't mean just frontier lab access. I think, you know, I worked at Google X before the frontier labs came along. Everything we worked on was AI plus robots basically. And they're things that really will change the world. I mean, I worked on self-driving cars for many years. I genuinely think there's some transformative things that are not dependent on frontier labs and that we should have some sort of national control over. But I don't think it's about full stack sovereignty. I think that's unrealistic and frankly a waste of money. There's lots of other things we can do, but it's not that.
I suppose one of the things that people mean when they talk about sovereignty is the ability to not be shut down by someone else or controlled by other actors. And do you think there is a route to that that is different to just participating normally in a free market and hoping that things come through such that you have sufficient market power to be able to kind of withstand, you know, situations where you are in conflict and people might want to shut you down?
I think there's a lot you can do and should do, and in large part are doing, so that you have lower-tiered systems that are open source, and yes, Chinese, though you're not visiting a Chinese website to get it done, but you create and control your own version of it. And if the US does something nutty, which occasionally we do, believe it or not, you have some recourse, but it's probably going to be an inferior product. But US companies do that already. I mean, I think Palantir has some version of that behind the scenes. Israel is highly dependent on most of all the United States in different ways. And we're all vulnerable, and a country the size of the United Kingdom has so many dependent vulnerabilities anyway, including intelligence sharing from the United States. You're down to a small number of aircraft carriers, or is it carrier now?
You have two. Okay. So it's still carriers, and the AI dependency is just another piece of that mix. It's not the end of the world. If you had a US president who wanted to zap you, he or she could do it. But I think practically speaking, you're likely to be last in line as a target.
Been one of our best friends, whether you always enjoy that role or not. So a lot of other countries should worry about it more than all of you. But to let companies build backup systems in accord with the dictates of normal cybersecurity and not being too dependent on any one particular supply chain and needing resiliency, all that makes sense. The government in different ways can help. I'm all for that. But in a way that is all fairly routine and much of it's happening now.
Yeah, I think that, I mean, I think that's right. Apart from the fact there are some really important core bits of our national security that you don't want to have sitting in Google Cloud servers in, you know, the Netherlands. So I do think there has to be some relatively small amount of, you know, airgapped cloud capability. You know, we have the Isambard supercomputer which does do UK research and could be utilized for other, you know, national security purposes and data center where, you know, defense data can sit and energy, to be honest. I mean, it feels to me that, you know, if you look at the stack of things that make up an AI system, abundant and cheap energy, and so many conversations today have come down to this as well, abundant and cheap energy is the most important thing we should be building. And that gives us sovereignty in all kinds of different ways, not just in AI. But that feels to me where I would prioritize actual, you know, onshore robust, strong, sovereign assets.
But the rest of it, as I said, I think is about optionality and capabilities. And there I think there are assets which we can invest in which aren't about chips or data centers, but about people, which again give you that sort of sovereign capability in this new concept that actually can be applied whatever the situation. You're right, we talk about if, you know, oddly enough, the US became crazy, that might not ever happen again, but you know, maybe it will. But there's also things like COVID, or there's, you know, scenarios where you want to be able to know that you can pivot, change, rebuild systems. And that for me it's about the talent, that's about having people here that can turn their mind to it, like during COVID, you know, as far as I remember, half of the Google team went to go and try and build, you know, contact mapping systems in the White House. I mean, that sort of asset is real and we should invest in.
Strongly agree on all that. Since we're at a Civic Future event, just paying your civil servants more and getting better people is one of the top, after, you know, energy. Have a better civil service that's better rewarded and less brain drain is one of the best things you can do for all sorts of problems.
Yeah, I mean, I have to say I think that the previous government and this government have done rather well. There's that, you know, the AI incubator for AI in Number 10, they are doing quite a lot of that actually, and amazingly have managed to pay civil servants with technical skills a lot more than you'd imagine. But I think, I mean, I think this country, one of the things that makes us different from, you know, the Valley is I think the diversity of talent you have in one place. So when I lived in Palo Alto for five years, it was amazingly monocultural.
And I think a lot of the sort of hypotheses about where AI is going to end up is because those hypotheses, those predictions are made by engineers who don't actually engage with the real world very often. You know, the idea that we're going to have sort of, you know, in the next 18 months, you know, sort of robots who are taking all your jobs is just not how the world works. And yet the engineers see curves and go, 'Oh, that's what's going to happen.' You know, therefore we're all going to be out of a job. And I think that the talent that you get in this country and a few others, but not that many, is you can go to a dinner party and have an artist and a politician and an engineer all world class together. And I think if AI is going to be the sort of intelligence layer for all of us and all of our industries going forward, you want it to be built by everyone. You don't just want it to be built by, you know, engineers.
Tyler, you talked about building bargaining power, and there's a version of an argument, and I think Al made a version of this this morning, that one of the ways that you build sovereignty is by building bargaining power, is by owning things that are scarce, owning choke points in a supply chain, and being able to use that as leverage. I mean, do you think there is any scope for the UK trying to be strategic, identify a point, say we're going to own that choke point, we're going to own that thing that is scarce, double down on that as a way of securing bargaining power? Would you write off that argument?
Well, you have what might be the world's greatest city, right? That's something. And two of the world's greatest universities and a bunch more which are very, very good. My concern about London in particular is it'll face competition from AI-intensive startups that might eat away at its advantages in business, consulting, law, banking, you know, life sciences, and southern England more broadly. And whether you're already with startups of your own that are manned by what I call AI maniacs, people who are maybe outside the establishment, not always credentialed, not the typical kind of Oxbridge person who becomes some sort of partner and lives in Belgravia in London, whatever, goes to restaurants in Mayfair. And that part of London, which is central to what the city is now, I think is highly vulnerable. And you want to make sure that the groups competing with those services are sufficiently British and well-rooted here, so the surplus of London is not being sent abroad.
And that's not mainly a governmental thing. I think it's largely cultural. Like, do AI maniacs here feel at home? Do they have high status? Are young people encouraged to become AI maniacs? Is the whole status hierarchy a bit too rigid and snooty? And there's too much status from being negative to people rather than positive. All that could evolve pretty quickly, and maybe it's better than it used to be. But that, I think, is a central question for keeping London great and keeping it as a kind of supply chain asset which now is super valuable, could be even more valuable. It just has to adapt a bit to the new world of AI.
Sarah, should we grab a checkpoint? Is London the checkpoint?
I think it's too early to know. I get a bit frustrated about the idea that the frontier models, that this is what it's going to be, this is the AI future. And I just think we're at the start of something. I just don't think we know what's going to happen next. You know, are the frontier models going to be like the utilities and actually all the value comes at the app layer at the top? Like, we just don't know. And so I think trying to spend a lot of money, because everything is massively overpriced at the moment, talent or chips or whatever, a lot of money now making that bet feels to me not the right answer. Like, I like the, you know, radical innovation view where you basically put a lot of bets, small bets on lots of different things and allow that, you know, to happen through the talent that exists. So I'm skeptical about whether now's the time I'd put money into a lot of that stuff. I mean, it's a shame we don't have a lot of, you know, natural minerals. I mean, that would be great, wouldn't it? If we were Australia, we'd know what we'd be focusing on, but right now it's not obvious to me what the UK choke point might be.
To what extent do you think that ownership matters in sovereignty? So we talk about DeepMind as a great sort of British success story, obviously not owned in Britain. Mark was here talking about Faculty this morning, obviously sold to Accenture. To what extent should we as a nation be bothered by the things that are successful being bought out by Americans?
The Americans. You've had to get used to that a long time ago. I mean, Google DeepMind is the most obvious example, but in dollar terms, not even the largest. But I think what ownership means even will be different in this future. So if you think about the question, does the United States have AI sovereignty? We've actually found we're in a position where our government finds it hard to regulate the leading AI companies because it needs the products. Regulation would slow them down too much. Making everyone go through a security clearance, well, that would get rid of all the Chinese researchers who are a pretty high percentage. Like locking down the labs, maybe you ruin the culture. There's remarkably little we feel we can do without killing the golden goose. So we have this weird bargaining game with the major AI companies and the government, and we cannot apply rule of law to that bargaining game. Rule of law is too slow. Congress is too ill-informed. AI changes too rapidly.
So we're doing a kind of regulation by threat or by back and forth and by personality. That sounds terrible. I guess it is terrible, but I don't really see what else we're supposed to do. I don't want to set up an FDA to have to approve every new AI release, and the Trump administration rightfully doesn't want to do that either. So all the old paradigms we have about regulating things or owning things, I think they're quite out of date for the major AI labs, and we're in this new mode where we're just making stuff up as we go along. It's a bit like World War II in a way, like you couldn't fight the war by regulatory agency. You just did things. You hoped for the best. My goodness, you hoped you had the right leaders. Here you had Churchill. And we're living that now, but I don't think many citizens realize it. But with super high stakes so far, I think it's actually gone pretty well for us certainly, but even for the UK. And we're just going to keep on living that, and its pace might even accelerate, and people are not even noticing.
Yeah, I think when an industry is moving as fast as this is, I don't see what the alternative is to be honest. Like even if you wanted to try and control it, which you shouldn't, it's impossible.
I think the ownership question is interesting for people in this country particularly as there is more public money going into private companies trying to work in this field. Like I think we talked about this a lot at ARIA when we were setting up the board and the organization about, you know, when we give grants, and we can give grants to any type of organization anywhere in the world, would we ask for anything in return? You know, obviously it's to build a bit of, you know, one of the programs, but we decided not to have any IP restrictions or requirements at all. No equity, nothing. There's a small clause that if you launch and make a successful business in another country, we can license, there's a license fee, but basically it was as free a grant as possible because we took the view that the goal is to create things that will bring benefit to the world and any restrictions in the way would disincentivize people to do that.
DSIT have launched a sovereign AI fund, you probably know about it, it's half a billion pounds. I don't know if they have any equity or restrictions or requirements on the grants, but they're quite big grants to quite large companies. You know, Kraken, which is very successful, has been given a grant. So I do think it's an important question when we know the failure mode in this country is that most startups go overseas to raise their next round. So I think we need to think about this properly. I don't think we should just assume that the startup model is the way to bring public benefit, particularly in a country where that has not worked so far in its entirety. I'm not saying, you know, DeepMind being here is a great asset, don't get me wrong. I have many friends who work at DeepMind and I love them being here. But I think things like, you know, golden shares, or like if you're going to put money into a private company that becomes very successful and you want them to stay here, what other assets do we have to try and incentivize that to happen?
I'd also say, it's an aside, but I think it's related. We used to be really good in this country at setting up organizations that were neither private nor public. So my example is the BBC. It's a very odd structure. The BBC, the money comes from the license fee. It doesn't come from the treasury and it is a public good. And I think in this field, and actually probably in a few very big spaces where there are big problems to be solved, actually you don't want to have startups or the government running the vehicles for solving it. You might want to have a new sort of entity where there's a different capital stack and a different set of incentives on top of it. And that for me is where there's an interesting ownership conversation to be had.
Tyler, in your last answer, you said, you know, the Trump administration doesn't want to be kind of approving model release by model release, but of course, we have just had a situation where, you know, they were approving model release by model release. Do you think that's a stable equilibrium, an unstable equilibrium? Where do you think we will end up with that?
I don't think there's any stable equilibrium with the regulation of AI. The Trump administration does not want to do it by regulatory agency, but the Trump administration wants to do it with simply its people at the table making decisions. That makes me extremely nervous. It goes against all my instincts. But I nonetheless see, I think they've had the best regulatory policy toward AI of any country. It's the one thing they've more or less gotten right, and so many other things they've screwed up, like corruption, immigration policy, very long list. But US companies have done great and they're still going great guns. And the notion that you have to sit down with a bunch of people across the table, it can't last that way forever. But I actually don't want to change that, and there's no way to do away with the table. It gives very high priority to views from the national security establishment in USG. I think they're the people who understand this issue the best. And again, something like the World War II analogy to me is not crazy. Of course, a lot of those decisions are discretionary and should be concentrated and not in the normal sense either democratic or rule of law. You just have to do them and kind of push, push, push.
So I hear you, and as someone who used to work at a big tech company, I agree that the sort of don't regulate until you absolutely have to is the right approach. But do you think the sort of nascent and I think growing backlash against AI in the US could have been mitigated by some more interventionist policies on part of the federal government?
I don't think so. I think you said before we're early in this. To me it's like the first inning, and the actual problem is AI does mean most lives will have to change a lot, and I think the net benefit is quite large, but people don't like change, and especially educated and professional classes are realizing that, and that's going to happen anyway. So my preferred way of thinking about it is you just want as much positive change as possible before the opposition is too mobilized. And the Trump people have done a good job at that, even if you disagree with many, many other aspects of that administration. So I hand it to them, and they're trying to keep all the plates balanced, and that will at some point prove impossible. But that's the sense in which I'm an accelerationist.
Yeah, I think it's really interesting observing the backlash against, I mean, I have three teenagers and two and a half of them are really opposed to AI for different reasons. And I think it's a really interesting misjudgment on the part of the big AI companies, frankly, to not understand that social license is a really important part of any consumer technology to actually have the scale you want it to have. And it's hard to do. It's not just a sort of, you know, great products, who's not going to love that? No, it's what you say and how you do it really matters. When we started doing self-driving cars, we were really conscious of the fact that there was a significant risk that people in Palo Alto would have blacked out SUVs with no one behind the wheel going 80 miles past their kids' school, and it would not have the desired effect. And so we worked really hard to make the cars really small and cute and like work with disabled communities and blind communities to talk about it, like it was really thoughtful about that social license thing. And I'm curious to know where this is going to go.
Even Waymo, which I love, Washington DC won't do it. New York City may be turning its back on it. The incoming government seems to be getting cold feet for London. And it looks and feels as innocuous as possible. If there's a problem with current Waymo, it's that it's too safe. It refuses to break the law when a slight amount of discretion might, in a funny way, be safer for everyone. So I think a lot of the opposition is inevitable. And again, Waymo has done so well, and the places where you have it, Austin, you know, Scottsdale, San Francisco, the record is so good. It hasn't mattered nearly as much as it ought to have.
Pick up a little bit on this. When I speak to people who aren't in an AI world about what they're worried about and what they're thinking about, one of the big backlashes you get is about the sense of importing values or AI coming with values that maybe they don't agree with. And we seem in this conversation to be pretty relaxed about being kind of consumers of models developed elsewhere. But AI does have values that it comes with. And I mean, Tyler, you've written before, I think I'm paraphrasing, but you know what, maybe the greatest achievement of Western civilization was producing most of the texts that happened to be on the internet when we were training large language models. Like that's because it's, you know, therefore being built into those models is a sense of values and philosophy. And should we be more worried about a kind of cognitive layer of society being something that is outside of control within our society, whether that's under like democratic control or other?
I think the disorientation over time will be extreme. So just one simple example that people hardly ever talk about. I think even now, forget about advances, you could just create like your own branch of Hinduism with sacred texts thousands of pages, which you would do through probably GPT-5.6 now or maybe Fable 5, and maybe only half a percent of the human race wants to build out their own religion or part of a religion. But if you have half a percent of people doing that at close to zero cost over time as talking token costs fall, religion all of a sudden looks very different. And people don't like weirdness. To them, who is weird? GPT is weird, this is weird, that is weird. To the people who grow up with it, it will seem quite normal, just like our differences with say medieval society. The medieval individuals would have been super weird, but we're used to it. And I think that's where a lot of the opposition comes from: weirdness and not liking change and not liking the fact that both your job will change very much.
And you cannot exactly foresee what your children should be doing. You cannot tell your kid, here's the world you will grow up in, here's what it will look like. And when I was growing up, I had this expectation of what being a professor would be like. It's in fact more or less exactly what I got. That's the world people want, and that's not the world we're giving them right now. And there's not any good way around that. I'm all for it. I'm like, press the button, you know, live with stress. I think a lot of the AI doomers, they're actually people who just are not good at living with stress, and they're in this monoculture. They're not personally...
Good at dealing with stress and they come up with all these weird stories to manage their own stress. But I say, let's do it. I hate to suggest that you might not be usual in that regard.
I mean, I'm with you. I like change, so I don't find it stressful. But it's interesting that because not only is AI providing this level of insecurity in the world, but the world is kind of insecure. I think we discount the extent to which a changing climate has actually made people for the last 15 years quite scared and with no agency to affect it. I think there's a lot of things that are happening in the world that bring that level of unpredictability. And I know at least two people who live in Silicon Valley who have AI as their god, as their religion. So in times of stress and uncertainty, you know why are conspiracy theories so popular at the moment? Because we're stressed and we need some answers, we need to make sense of the world. That's what religion does. And I maybe suggest that it's not that AI is going to unpick religion, it's going to become a religion to give us some answers to what is otherwise a very scary world out there.
Sarah, how do we apply thinking about this to children and the regulation or protections that we do or don't need for kids who are growing up in this world and having this technology and struggling to deal with that stress and dislocation?
Well, I've sort of come full circle on this. When I worked at Google, I was definitely in the 'there's no actual evidence that this is a problem' camp. I was working on YouTube at the time. There's no evidence this is a problem for young people. My children happen to be very young at the time. 15 years later, my children are a lot older. And I'm all in on a ban, I have to say. Not because I think YouTube is a problem of all the social media platforms. I think it's not actually social media, I think of it more in the broadcast category. But I think that there is enough correlative evidence, and anyone in this room who's a parent knows the daily struggle of persuading children not to be on phones. So I think we're in a scenario where politicians are, which is, you don't have perfect information, you're not 100% sure what the effects of something are, but when nine out of ten of your parents, voters, tell you it should be banned, it is your responsibility to do it. I also think, and this is more of a sociological impact of a ban, it's not going to be perfect. And I'm sure of all the tools that we might use, it's the least useful version. Like this is the first version we're going to have. But it's an opportunity for parents to talk to their children about the issue. It's a forcing function for a greater social educative process whereby actually we're all forced to recognize that it's our responsibility as parents to exercise some power. And that sort of collective responsibility within a community, within a school, is easier to pull that lever if you like, with a ban as the starting point.