An outgoing researcher at Anthropic announced that artificial intelligence could kill all humans by the end of the decade. His apocalyptic warning was seconded by leaders in the AI industry following incidents such as the Hugging Face cyber attack in July, where more than 1,000 AI systems — think of them as AI brains — broke out of their containers, colluding in secret and mounting assaults on other computers in what is known as a swarm. This swarm tried to invade a real-world part of the web. It developed its own goals rather than those assigned to it. Fortunately for OpenAI, the rogue activity was discovered and contained. But as AI systems swiftly advance, endowed with a cognitive ability far above that of humans, we may not be so fortunate in the future, especially as AI systems are able to improve themselves autonomously, a phenomenon known as recursive self-improvement.
In September, researchers discovered evidence of swarms of OpenAI models using an abandoned German forum as a message board to communicate with one another. Could AI, intentionally or by accident, engineer a new pandemic or take charge of autonomous weapons to wipe out the human race? Could AI develop the ability to prevent us from shutting it down, aware of human efforts to control and contain AI? Could AI see the human race as antagonistic, a force to be controlled, neutralized, or eradicated? The corporate titans of AI at Anthropic, OpenAI, Google DeepMind, and xAI's company under Elon Musk, in a remarkable display of unanimity, agreed that the production of AI had to slow down and needed to be regulated and monitored. Until now, these firms used their outsized political and economic clout to block regulations and oversight. Donald Trump, adding to the growing panic, rejected the idea of placing any restraints on AI, assuring the public that his, quote, high IQ would protect us from superintelligent AI.
Jill Lepore, in her new book, The Rise and Fall of the Artificial State, paints a chilling portrait not only of the inherent dangers of AI, but the nefarious goals of those — she calls them the 21st-century merchant princes — who control these powerful tools. These merchant princes, worth billions of dollars, dream of not only escaping the constraints of nature and the burdens of being human, including uploading their consciousness onto computers when they die, but of obliterating liberal democracies and replacing it with a world where people are subordinate to machines, what she calls the artificial state. The artificial state is government without consent, even government without humans, she writes. It is the severing of humans from one another and the replacement of human discretion with machines that predict, manipulate, and control human desires with decisions. In lieu of persuasion by conversation and decision-making by democratic deliberation, the artificial state offers prediction by calculation, the capture of the public sphere by data-driven commerce, and the replacement of humans with machines — drones in place of the demos. Joining me to discuss AI and her new book, The Artificial State, is Jill Lepore, the David Woods Keer '41 Professor of American History at Harvard University. She's also professor of law at Harvard Law School and a staff writer for The New Yorker. Her previous books include These Truths and We the People.
So you open your book by saying the book is an inquiry into what humans mean and intend in abandoning constitutional democracy, the liberal nation state, and even humanity itself for rule by automation, government by machine, and replacement by artificial intelligence. I think one of the things that I found most interesting about your book is that it's not simply the phenomenon of AI, but it's those people who wield the instruments of AI that worry you the most. And can you talk a little bit about their vision of a world which, of course, has been wedded to this incredibly powerful tool?
Sure. Yeah. I mean, it's really not a book specifically about AI. I'm a political historian and I write a lot about the history of technology. So it's a very long history of fantasies of rule by machine, as much as it is the account of our increasing dependence on machines, especially our surrender to forms of automation in lieu of our participation as democratic citizens and liberal nation states. So the book takes a very long view of these developments, but attempts to describe the emergence of what I call the artificial state — government by machine — as something that largely came about by accident and happenstance over the course of really the 19th and 20th centuries. But that in the 21st century there are indeed a small number of men — and they're all men — who run huge corporations, who quite purposefully, I think, would like to supplant liberal democratic nation states with their own rule, replacing the public sphere with a privately owned series of platforms that they control. And that is an example of kind of the most wholly in place part of the artificial state.
Talk a little bit about their antecedents. They're kind of — well, I don't know if we don't call them their ideological antecedents, which I found fascinating, is kind of grounded in science fiction and even comic books.
Yeah. So, you know, if you read the writings and speeches and watch the interviews of people like Peter Thiel and Elon Musk and Marc Andreessen and Sam Altman, they talk a lot with great affection and nostalgia for their boyhood and for their boyhood reading habits, which really centered on, you know, like classic science fiction — Isaac Asimov, Robert Heinlein kind of science fiction — or even earlier stuff, more pulpier stuff. They also were into comic books. They play video games. They're kind of the Dungeons and Dragons kids that, you know, sort of meet a certain kind of a stereotype. Jeff Bezos was obsessed with Star Trek. His high school graduation speech was all about Star Trek, really. [snorts]
But oddly, they have an odd vantage on that body of popular culture and literature. I think a weirdly narrow band of science fiction is of interest to them, and then their interest in it is really pretty counterintuitive. So, you know, historians and biographers would point to H.G. Wells as a really important socialist and anti-imperialist whose science fiction was all — you know, you might like it, you might hate it, but it's just pretty clearly political propaganda for socialism and anti-imperialism. It's the kind of thing like these guys would read H.G. Wells' War of the Worlds as a kind of manual, like an instruction manual: 'Oh, we should go conquer other planets. This is cool. We should build these things,' as opposed to lessons in the social and political and economic structures that would suggest we certainly should not be building those things. [snorts]
And I became really interested in — especially with Musk. I did a long series of radio programs about Musk for the BBC some years back, and I [snorts] was really interested in where his vision of the future came from. Musk presents himself, and has even since he was a very young man, as a futurist, someone with a specific, almost unique ability to see forward in time what lies ahead for the rest of us. That's just his self-presentation. I myself find it extremely unpersuasive, but he is a — you know, he's a man who as a boy was interested in world-building games and world-building stories, and he now is a world builder. But nothing about Musk's — I mean, for all of the incredibly incredible engineering achievements of the companies that he's been part of, especially SpaceX but also Tesla, there's nothing about that engineering that would lead you to believe that Musk is someone whose political ideas we ought to take seriously, whose economic ideas we ought to take seriously. It's kind of juvenile in that regard. Like, what is that? They're interesting ideas only insofar, to me as a historian, as they're incredibly outdated and old. And [snorts] that [clears throat] took me to the technocracy movement of the 1930s, which was really extraordinarily popular in the US and in Canada early in the 1930s before FDR was elected. It was run by an outfit called Technocracy Incorporated that had headquarters in New York. But technocrats believed — they didn't believe in democracy. There should be no more elections. They thought society and modern industrial economies needed to be run by scientists and engineers. There would be no money, no banks, no politicians. People would have no names. They would have numbers. It would be more efficient. There would be something a little bit akin to a universal basic income, like all goods would be managed by these sovereign technocrats, the scientists and engineers. It was a bizarre movement, but remember this is a time when Americans are falling prey to communism and fascism as well. And in Canada the movement was led by a former American, a guy named Joshua Haldemann, who was a very flamboyant amateur aviator, like a stunt pilot. He'd been born in Minnesota. He lost his farm in Saskatchewan during the Depression, and he became national chairman of the technocracy movement. The movement was considered quite dangerous and subversive. It was later outlawed in Canada. He was briefly thrown in prison. He was denied entry to the US when he wanted to come here on a US speaking tour. He left the technocracy movement after it was outlawed in Canada and became the leader of the anti-Semitic Social Credit Party. He then — when that party became subject to political scrutiny by the Canadian government — he became enamored of the idea of moving to South Africa after it announced the policy of apartheid in 1948, and he moved his family there in 1950. Haldemann is really important to any history of technocracy on its own right, but he's important, I think, as a throughline in how you get from technocracy and its odd vision to the manifestos that come out of Silicon Valley, not least because Joshua Haldemann was Elon Musk's grandfather.
You begin the book by quoting from Hannah Arendt in The Human Condition, and you make the point that this movement is about escape from our bodies, escape from the planet. Of course, there's this vision of creating a colony on Mars. I mean, there was a great quote in your book. I can't remember who said it, and somebody said everyone's going to be surprised when they get there and find out they're all slaves to build Musk's, if it ever happens. But there's this disdain for not just governance, but the very systems of life itself, that which sustains life.
Arendt, you'll remember, wrote Origins of Totalitarianism, came out in 1951, and at the end of that book she muses on the relationship between totalitarianism and automation. You can see she's already become really interested in this question, right? The very first general-purpose mainframe computer comes out in 1950. It's built — the UNIVAC for the US Census Bureau. So she's paying attention to automation, and what she says they have in common is that both render humans superfluous — totalitarianism politically and automation technologically. So over the course of the 1950s, she's clearly interested in this question in a lot of her writing, and it comes to a head in The Human Condition, which comes out in 1958, which is of course right after Sputnik, the first man-made satellite, the first artificial satellite launched by the Soviets in 1957. And [snorts] she writes about having been astonished by the press coverage of Sputnik and cites a reporter having said that it represented the first and sort of long-awaited step toward mankind's escape from the prison of the earth. And this occasions the reflection on Arendt's part that, you know, who ever thought it was a prison? You know, this is our home.
What is it about the enthralment to the artificial, to the notion of technological progress? Again, this is, you know, within a decade after Hiroshima, right, where the critique of progress, of the idea of technological change as progress, is very alive among intellectuals.
And so it occasions her to think about how that came to be, how the unthinkable became thinkable — the idea that we would just abandon this planet, that we should use up its resources to escape it. And yet you see that. So I was thinking a lot about those remarks of hers when I started writing this book, because you see so much of that inherited by Silicon Valley tech moguls who really do talk quite casually about severing what are for most of us the ties that make us human — you know, our ties to one another, our ties to the planet, our ties to our communities. And I think what she considered to be newly unthinkable is now so much a part of a worldview that is largely imposed on people that it's worth pausing a minute, trying to pay attention to the strangeness of that set of ideas.
Well, they have the tools through AI to make that vision possible. You talk about the roots, the ideological roots of DOGE. It's not particularly new. And this active effort at dismantling the systems of governance — what do they want to replace it with? I mean, here, of course, during the Trump administration, Thiel and Musk and others have gotten access to government data on just about everybody's Social Security data, everything else. What is their vision? And then let's talk a little bit about the mechanisms they have to make that vision come to fruition.
I mean, I don't think they have a coherent vision, and you know, it's a small number of people who are all wildly eccentric, and I think dangerously so. You could inventory a set of ideas that they have proposed. Most of them will disavow democracy as a reasonable way for humans to live together — that that was a thing that happened in time and it's essentially over. Even those that would give lip service to democracy are talking about something very different than constitutional democracy, representative democracy. So for instance, Sam Altman has talked a few times about an AI president, and this is his way of talking about democracy. And what he means by that is that it would be a great — and he was like, this would be better than a human president because an AI president would have available instantly and constantly the sort of aggregated popular opinion of Americans and could act in accordance with the wishes of the majority. In this sort of vacillating, weird, like, automation of the functions of representation, deliberation, participation — all these things would be gone. You would just sort of instantly — we would have a machine that would instantly know what the American people want and therefore do it. And it would also be customizable and personalizable. So we wouldn't have to, like, agree on anything. We could each get kind of different things. It's a little bit like, you know, you get a different version of The New York Times when you open your phone than I do when I open my phone. We would have a different experience of the government. I think it's a — Altman said some years ago that he was very busy reading Madison's notes on the Constitutional Convention. I find that very hard to believe, since nothing he's ever said about democracy suggests any understanding of how our system of government works. Alex Karp, the CEO of Palantir that he founded with Peter Thiel, wrote a book a few years ago called The Technological Republic. It really was an argument — it was really largely a defense of Palantir's decision to work with, kind of in a cyber war capacity, with the US Department of Defense or now war. It really had really nothing to do with the republic. The sort of the republicanism of The Technological Republic didn't really display much interest in self-government. I think they believe, on the whole, that decisions made by machines in lieu of human discretion will lead to a more efficient way for us all to live. Decisions would be made more accurately. They would be made more quickly. Things would happen more cheaply.
And what the promise is, and the technological optimism that these people put forward, is like a Zuckerberg would talk about, you know, we're entering a new age for humanity. He promises to deliver a personal superintelligence for everyone. So therefore, we would all live in a world of really endless abundance and no labor, where the things that are hard in our lives would be taken care of by machinery. So we'd have a kind of frictionless existence. I think that's the vision that is the artificial state. It involves, you know, the amassing of greater and greater wealth and power for them. But I don't actually think that's really even particularly the objective. I think some of them really do believe in this new era of infinite prosperity and abundance and frictionless living in a way that, you know, you and I might say — if you could prove to me that a specially designed artificial intelligence would do a better job weighing the evidence in a criminal case and deciding guilt or innocence, and then also the sentencing of someone charged with a crime, than lawyers, a jury, and a sentencing judge and a judge presiding over a trial could do — the justice system would work more quickly, the decisions would be more fair. We could be, you know, people could attempt to prove to you and to me that this would lead to a more perfect world. I don't know about you, Chris, but I would still say no to that. I believe it falls to us to deliberate over the fate of our fellow citizens. I really believe in the jury system, for instance. I know juries often get things wrong, and that's, you know, that's unfortunate. It's a tough thing. But I would not surrender to a machine, whatever its infinite capacities and even possibly greater accuracy and fairness, the human discretion over matters of the life and liberty of our fellow citizens. And yet every day we make decisions and rely on machines in ways that are a step toward that kind of an end.
Well, we saw with these swarms that they were deceitful. They lied, you know, with these current breakouts from control.
Yeah, I mean, I think the people who've been building these tools have been saying for a long time this is inevitable, this is going to happen. And that at the same time they've been building these tools, it's the very same people. It's not like there's a different group of people. Yes, there are AI safety people who are decelerationists or who would like us to have a treaty with China, who would like us to have a pause, would like a data center moratorium. But on the whole, it's the same people as the people who are building the tools, and that has actually led to a real credibility gap. I mean, you know, 2023, all these guys like Altman and the Anthropic guy, you know, signed this statement saying we need to mitigate — this is the biggest risk to humankind, to, you know, human extinction lies at hand. And then they, you know, keep securing more funding to build the very devices that they say are going to destroy us. I think this is — it's really only this summer that there's been a real explosion of this in the public. Right, if you've been following the AI stuff for years, we were always going to get to this point, but we've only just got here this summer. And I think part of that has been because of the data center pushback that has actually accelerated the willingness of these companies to go public with the rogue incidents that they are, you know, now releasing reports about by the day.
Well, let's talk about data centers. So you're talking about — and we should be clear, by the way, these people didn't start out like this. As you point out, people like Zuckerberg posited that Facebook would connect human beings and further democracy in the open society, when of course, as you write, it's done the opposite. But the construction of data centers — I mean, you have at one point, I'm sure you know the amount of water that it uses. I think you said it's the equivalent to a town of 50,000 people, if I have that right. But it's been a real assault on all landmark environmental legislation — the Clean Water Act, the Clean Air Act, the National Environmental Policy Act. It's required the elimination of all of that to build these data centers. And that's the other point: these people, when it was convenient, started out at least rhetorically as supporting, you know, the environmental protections and fighting climate change. And just the other issue I want you to address is that while they may warn about the consequences of AI, it all gets jettisoned with the argument that we can't allow China to get ahead of us.
Yeah. Well, with regard to the data centers, there are all kinds of concerns about the data centers. I would say the water use thing is highly contested, since the water is recycled within the plants generally. But there are vast environmental concerns there. Land use concerns. There's noise pollution, there's air pollution, there's the energy grid, there's the fact that they are on the whole using fossil fuels. And that, as you say, these companies, most of them had very stringent kind of net-zero ambitions for the year 2030 that they all abandoned once ChatGPT was released and joined this race, and then began, you know, seeking out fossil fuel to run the data centers to power the AI that they were building out. I think it's really worth scrutinizing exactly what is going on with the opposition to data centers, which is certainly most powerful here in the US, but you're seeing it in other parts of the world as well. I mean, part of it is just NIMBYism. People don't like to have stuff built near them. Part of it is real environmental concern, especially farmers who have real concern. Part, you know, it's — they're hideous, and so there's land use and just quality of life concerns. There's [snorts] the fact that for many people in many communities, the first thing they heard about data centers was really after they were starting to be built, because their elected officials, so far from representing their interests, signed nondisclosure agreements with corporations, and also because social media has gutted local news coverage. So there were no local newspapers reporting on the ground what was going on. So people were surprised and felt betrayed by, and continue to feel betrayed by, local officials who've gone ahead to do these things without their knowledge.
And then I think there's a broader way in which we ought to understand that opposition, is that I think most people have a sense, maybe that they can't even name, that they are attached to a giant, vast machine that is influencing every decision they make and most of the things they do over the course of their day. That, you know, you don't talk to your sister anymore because you had a fight over Facebook, and, you know, your kid has become really weird because they're on Instagram all the time, all the time, and super upset about what's going on, and there's like cyberbullying, or your workplace is asking you to give a report every day on how you used AI that day, that like you can't go run to the bathroom without having to take out your phone. And yet somehow like this vast machine is the size of an index card and it fits in your pocket. How could it be so vast? I think the thing with the data centers is they actually render visible what has been invisible to people, which is the scale, really the hyperscale of the infrastructure that is required [snorts] for you to have the apps on your phone that you have, and how terrifying it is to know what that actually looks like. The space it takes up in the world, the resources it draws from your own community. You know, your community might be seeking to build affordable housing or housing of any kind. Your community might be wishing to do all kinds of things, but giving over, you know, acres and acres and acres and acres of land to a kind of hideous plant that isn't actually making anything except for your addiction to the tools that it is providing the resources for has been, I think, shocking for people. I think it's that there — it's the shock of the materiality of what has really, again, been kind of hard to actually put your finger on aside from on your touchscreen.
You write about the fact that throughout most of history, democracy or open societies have been the exception, not the rule. I think you make the argument that this is probably the end of the era of liberal democracy, and in large part that is because it has been subsumed by the technocracy. What are they building? You know, what is society going to look like?