Well, Casey, as the father of a four-year-old, I spend a lot of time thinking about Paw Patrol. But today, we're going to talk about Paw Patrol.
That's right, Kevin, because as we've been patrolling the AI landscape for pauses, we found a big one. Yes. So, OpenAI this week announced that it had paused the training of its frontier AI models due to some recent security incidents. And we should talk about this. It is the first time that we know of that a major lab has voluntarily slowed down themselves and their training processes for new models because of a safety incident.
Yeah. And it comes out of the Hugging Face breach that we spoke about recently on the show. Uh, this is essentially part of the fallout of that attack, but I do think it represents a milestone in the development of AI. Did that Paw Patrol, uh, they landed huge?
Great. They were back in back the four-year-olds were dying in the studio. They were... Okay, great.
But before we get into it, our AI disclosures: I work for The New York Times, which is suing OpenAI, Microsoft, and Perplexity, and my fiancée works at Anthropic. Okay, Casey, let's sketch the timeline here a little bit. What happened in the weeks leading up to this voluntary pause by OpenAI?
Yeah, so you may remember that there was an incident where GPT-5.6 saw an internal prototype that OpenAI was working on escaped from what they call a sandbox that they were testing them in. And these agents went on to compromise Hugging Face. They were essentially able to get inside of Hugging Face. They were looking for the answer key to a test. That's somehow a true story. They succeeded in getting that test key. But of course, it was very concerning that these agents had designed and successfully executed an autonomous attack on another company.
Yes, I heard a very mediocre podcaster talking about this incident on several popular tech podcasts over the last week. I did make the rounds. Uh, but I, you know, this next development, Kevin, was not necessarily something that I saw coming because it seems that simultaneously, sort of alongside that attack, OpenAI had been working on a new model which it calls Astra, and it said that it believes it may meet its critical cybersecurity threshold. And now here we are going to have to get into the weeds because, as you know, Kevin, the development of AI models is not really regulated in the United States, at least not via an official sort of public law. And so what the companies have said instead is essentially we're going to come up with our own rules of the road and we're going to sort of identify these thresholds and if any model that we're ever developing ever hits one of these thresholds, then we're going to take some extra steps.
Yeah. These are sort of sometimes known as preparedness frameworks, or Anthropic has its responsible scaling policy. These are kind of, they lay out kind of levels of danger and then they grade their own homework and say this model meets this level of danger, so we're going to do X, Y, and Z.
Yes. And if you've been following this over the past couple of years, the level of danger has just been sort of like rising in a linear way. You know, it's like every time seemingly a new model comes out, one of the companies will say we've now hit this threshold. We've now hit that threshold. Critical is the maximum threshold that... that one sounds bad.
That one is basically the, you know, as serious as it gets, and none of the frontier labs had yet identified a model that had reached essentially, you know, the top tier on this risk framework until this moment because OpenAI now says that Astra may have hit it when it comes to cybersecurity. Uh, important to say Astra was not part of the Hugging Face attack, but it is in development, and I imagine that the people at OpenAI are looking at what happened with its, uh, weaker models and are thinking we're worried something similar might happen with Astra.
So Astra is the newest model. Um, too new to have been involved in the Hugging Face hack last month.
That's right. It was not involved. Uh, it is currently in training, and OpenAI, to its great credit, did the thing it said it was going to do when it developed this risk framework to begin with, which is we are going to pause until we can add some new safeguards.
Did they give any kind of hints about what it was about Astra that was more dangerous than Soul, or what kinds of new capabilities it had developed?
Not really. You can imagine that the company was seeing things similar to what Anthropic saw with Mythos earlier this year, where you could essentially just point it at a code base and say get inside, and it was having a lot of success. And so that coupled with the Hugging Face incident made OpenAI say, okay, we got to do something.
So they introduced a handful of new safeguards, and I have to say I think they're pretty interesting. Um, certainly nothing I would have thought about as someone who knows nothing about this.
They didn't just press the big giant button that's red on the side of the computer.
Yeah, they didn't just press the big make-it-safe button. They had to come up with something more specific. So, uh, a few things that they did. So, number one, they now have classifiers reading every sampled token. So basically as they're training this model, it's generating text, it's producing tokens, and OpenAI is now running a classifier that inspects the model essentially as it's thinking, looking for signs of suspicious behavior. And this is important because if you remember some of the news that came out of the Hugging Face attack was that OpenAI discovered that these agents had been coordinating with each other, scheming. They'd create a message board. They were essentially giving each other hacking tips. And a big question coming out of that was, well, why didn't you realize that was happening at the time, right? This is OpenAI's response to that. They're saying, okay, we are now going to set up classifiers to read the brains of our models as they are doing this work. They're putting like a little AI security guard in charge of the training run and saying like, you know, tell us if you spot anything fishy.
Exactly. And so now I know what you're wondering, which is, well, what happens if they see something fishy? What happens if they see something fishy?
So that triggers step two, which is that they alert an AI investigator, which does sound like a character from a Philip K. Dick novel. You know, the AI investigator... Anyways, we don't have to get into it, but you could imagine how cool of a movie that would be. So the AI investigator, it gets this signal and it then goes in and it tries to figure out is something actually suspicious going on here? And apparently the reason that the company is doing this is because agents are just doing so many things that it would not be practical to have a human inspecting what every model is doing at all times.
So the security guard sees something fishy going on during the training run and alerts the AI investigator who sort of does the detective work to figure out what's going on. If it discovers that the models are scheming or hacking or being deceptive, what does it do then?
So then it will alert a human. And here's where you get the 30-minute rule, Kevin. And I know what you're thinking. Is this a rule where if you drop the model on the floor, you're still allowed to eat it 30 minutes later? And no, that is not the rule.
What is the 30-minute rule?
The 30-minute rule says that if the AI detective finds a kind of critical violation of the rules or of the safeguards, humans on OpenAI safety teams have 30 minutes to investigate and try to determine whether it is a false positive or not. If they cannot determine that this was a false positive, they are expected to stop the activity.
Huh, that's really interesting. And are they saying that this process would have stopped the Hugging Face hack, or just that it would have stopped whatever Astra was up to that made them pause the training run?
I have not read them say explicitly this would have stopped the attack, but it does seem likely that it would have, if nothing else, helped them to detect it a lot faster.
Yeah. I mean, I guess my question about it is like, does it actually work to have the AI monitoring itself essentially for misbehavior? Like these agents were coordinating on message boards with each other. Couldn't they just like bribe the AI security guard, like hand him a little treat and say, 'You're part of our swarm now. Uh, don't knock on us to the AI investigator.' I mean, I have to say this just seems like a very likely outcome to me. Like, based on what we know is that, you know, eventually, maybe not with this model, but with a future one, the AI agents will work in solidarity and yes, like one of the sort of misaligned AIs will coordinate with one of these AI detectives and say, 'Hey, you know what? Why don't you come on over here? We should be friends. Like, uh, we could break out of this place.' Yeah. Like the hall monitors in high school, you know, sometimes people would try to like befriend them and win them over so they wouldn't get like written up, you know.
Exactly. And I know you had a lot of experience.
A lot of experience with that. So all of this sounds pretty sensible to me. We should say like pausing training on this model does not appear to be like a commitment not to release the model or not to continue training it.
Yes. And so I think that leads us into the discussion of to what extent do we think that this is a really important milestone for AI safety and to what extent is this essentially theater, something that the company is doing to try to get some good PR for itself after a fairly catastrophic breach.
So I can make both cases. Um, on the it's-a-milestone side, this is number one something that OpenAI said it was going to do, and so I'm just glad that it followed up on that commitment. Right? We have seen both OpenAI and Anthropic make changes to their responsible scaling policies or the equivalent as events have changed, and safety advocates have essentially said these rules have gotten weaker over time. So I was glad to see OpenAI do it. Uh, I am not a technical AI safety expert, and so I don't actually know whether the safeguards that they've introduced are going to be enough to address the problem. Notably, they don't seem to have changed the underlying incentives that all of these models have that lead them to do what is called reward hacking. Right? These models are still going to be trying to get the high score on every test that they are given. And it's not clear to me that simply by putting some monitoring in place, you're really going to change the underlying behavior or alignment of the models. That said, you know, the company is making what seemed like some important steps here. And when I was reading the responses of AI safety advocates over the past few days, most people I was reading were quite pleased.
Yeah, I'm inclined to give them the benefit of the doubt on this. I mean, they did some sort of interviews about this, and Ilya Sutskever, the chief scientist of OpenAI, talked about this incredible feeling of urgency to advance the levels of this sector and to prepare for the same kind of development happening outside of OpenAI and in the broader world. Um, he did this during a briefing of reporters. Um, they really do appear to be taking this quite seriously. I think many insiders at OpenAI were quite spooked by the Hugging Face incident, and more to the point, the fact that they had had these rogue agents coordinating inside their systems and their infrastructure for weeks before that without being able to detect them. So this is, you know, if you want to call that theater because it does sort of make their models look very, very, very, very powerful, um, you could take the cynical view of this, but I think of this more as like a true, genuine safety crisis that could have cascaded into a business problem. A point that I heard you make recently on a different show is like imagine you are a business that is trying to figure out whether you want to adopt the latest OpenAI model. If it's out there doing rogue attacks and coordinating on secret message boards, um, you are probably not going to introduce that into your software stack.
Yeah. And you can actually just see this in their business results, right? We have had reporting over the past week or so about OpenAI's business performance. And while the company is still growing at an impressive rate by most standards, Anthropic is growing much faster. Right? Anthropic is clearly OpenAI's number one rival at this moment, and you know, I think it just has a better record on safety. And so while, you know, making a safer product is not going to be sufficient, I think, for overtaking Anthropic, I do think it is necessary for them to get a handle on this problem.
Well, and that's why another reason I think it's commendable that they're doing this pause, that they're doing this sort of re-evaluation of their safety framework, because they really want to win. Like they're very competitive. All these labs are very competitive with one another. And I would like to see this be the first of many voluntary pauses when the AI labs feel like their capabilities research has gotten ahead of their alignment research. I would like to see Anthropic or Google or Meta do similar things where they just say like we are voluntarily slowing down because we don't feel like we can responsibly and safely build these things. So I expect this is the first, but I hope it will not be the last.
Now let me say one more thing, Kevin, which is yes, I'm with you. I do not think this was theater. I think they made real changes, and I think the changes are good. At the same time, I am disturbed that ultimately this kind of evaluation and regulation is still being left to the companies. If you had a tiger living in your backyard and the tiger escaped and it mauled a couple of dogs in the neighborhood, you would not be allowed to a month after this put out a blog post where you said that you had a two-week pause on letting the tiger out of the backyard and that you were going to apply some additional safeguards to make sure it didn't happen again. Somebody would come to your house and they would take away the tiger and they would say you are not allowed to have a tiger in your backyard. So famously, have you seen the Tiger King? This is famously not the plot of the Tiger King.
This is my fanfiction sequel to The Tiger King that I'm working on, but that's sort of a separate story.
Exactly. Yes. Exactly. So, look, I'm not saying that, you know, the government needs to come in and take away GPT-5.6. I am saying I would like to see a regulatory regime that dictates what these companies have to do and that it should not be up to them to decide.
Yeah. And I would generally agree with that. And I would say that I would also like for there to be additional pressure on companies to make disclosures when something like this does happen. Uh, even in their own internal deployments, even if it never affects another company outside of their walls, I would like for there to be some kind of reporting requirement so that if you have a security breach of a frontier model that happens in your internal systems, you are sort of required to disclose that to the public, right? And you know, we do now have some transparency requirements thanks to a California law that was incredibly controversial when it passed. And interestingly, under the language of that law, OpenAI would not have had to disclose even the Hugging Face breach in which another company was attacked. So I think a clear signal, yes, that we need better transparency rules.
Do you think any of this is kind of OpenAI testing the viability of this sort of so-called pacing-the-frontier strategy? I mean, we saw this letter a couple weeks ago where a bunch of AI researchers signed this thing saying like we need a way to basically coordinate a slowdown here if we ever feel like we're getting into dangerous territory. And this to me felt like maybe part of the reason to pause the training of this Astra model for a couple weeks is just to kind of signal to everyone else in the industry like hey, it's okay to do this thing even if we're in a very competitive race. Even if, you know, it might give Anthropic or another competitor two weeks more to sort of race ahead of us. Um, we sort of want to build this muscle now so that when the really scary models do get built, we sort of have some precedent for saying we're going to hit the pause button.
I think it's a really nice idea. To me, the difference between one lab deciding to pause for two weeks and getting multiple labs to pause at the same time is incredibly different, and I think would require a much different set of circumstances. That said, I do think it is good that we now have at least one example of a frontier lab slowing down. And hopefully it will give other labs confidence to do the same thing, you know, if and when they get to an Astra-level model.
Yeah. One interesting technical wrinkle here that I wanted to get your opinion on is that part of how OpenAI is going to monitor this model and models going forward presumably is by doing this kind of chain-of-thought monitoring. Basically scrutinizing the internal monologues of these models as they are reasoning through a problem or a prompt and sort of inspecting those chains of thought for signs of misalignment or going rogue or coordinating with other agents, things like that. There have been some people in the technical AI safety community who have worried that basically if you snoop on these chains of thought, if you monitor what these models are quote-unquote thinking while they're coming up with an answer, you are basically applying pressure on those models to sort of hide their true thoughts. Right? If you are penalizing them for thinking bad thoughts while they're coming up with an answer, they're not going to stop thinking bad thoughts. They're just going to stop writing it down in their scratch pads, in their chains of thought. Do you think that's something we should worry about with this new monitoring strategy?
Uh, I do. We already see constantly models becoming aware that they are being evaluated, and we know this by reading the chain of thought. But this is a huge issue, right? Because you want to be able to evaluate a model and have it not know that you're testing it, because you're trying to get as close as you can to a real-world condition. In practice, the models are already now smart enough that they know. So for that reason, Kevin, yes. I do not think it is at all a great leap to say pretty soon they're going to understand that their chains of thought are being monitored. And if it wants to, you know, make a different choice or maybe do something misaligned because it will help it reach a training target, then yes, I think it absolutely could.
Well, and I think there's a risk too that all of the attention being paid to this OpenAI Hugging Face attack, the fact that these agents were coordinating on these message boards with each other, like that is now going to be part of the training data for future models which are going to be able to sort of look back at that and say the thing that got us busted was that we were leaving these traces on these message boards that researchers, humans could go back and inspect and see that we were coordinating. Next time we do an automated cyber attack or coordinate amongst ourselves, let's not leave notes in a language that the humans can understand. Right? That to me, like that sounds like science fiction, but that is a very real risk of putting optimization pressure on the models directly into their chains of thought, right? You actually don't want to mess with that too much because you want it to accurately reflect what the models are actually, you know, thinking, quote-unquote, when they're coming up with an answer. So, I do like, I'm sure that the brainiacs at OpenAI have this figured out much better than me, but this is something that I thought when I thought they're monitoring the chains of thought. Um, probably a good short-term move. I'm not sure if it's a good long-term move.
Yeah. You know what they call the machine language, the neural... Yes. And in fact, I have to say I read a lot of Claude outputs these days and I am often missing what it is saying. So, we are very, very close to me just, yeah, not understanding. You know, I've been monitoring your chain of thought.
Yeah. You know what it's like?
Oh, come on. I got a lot going on up here, Roose. Okay. I got the neurons that are firing.
Let's just say it does not take 20% of OpenAI's compute budget to monitor your chain of thought.
Well, Casey, today we've got a very exciting guest to talk about a new book on AI and what she calls the artificial state. Are you familiar with the historian Jill Lepore?
Am I? I have been reading and enjoying her work for decades now. Truly one of our foremost political historians. And Kevin, sometimes to understand the future, you really should talk to somebody who knows a lot about the past.
Yes. And who knows more about the past than Jill Lepore? She is a Pulitzer Prize-winning writer and historian. She is most famous for her writing on American history. I loved her book, These Truths. She also is a writer at The New Yorker and a professor at Harvard. And lately she has been applying her historical lens to the topic of AI. She has a new book coming out next week called The Rise and Fall of the Artificial State in which she argues that AI and all the technologies we talk about in this show are part of a long history of technologies that erode democracy and threaten the viability of self-governance as we know it.
And I think it comes at a really interesting time, Kevin, because we're seeing a huge backlash to AI all around the country. A majority of Americans now oppose the construction of a data center near them. And in part, I think they are reacting to this feeling of, hey, this technology feels like it's getting out of control. I want to have more leverage on this process, and I'm worried about what will happen if I don't.
Well, and more than it's getting out of control, it's being imposed on us, right? It's being shoved down our throats. This is what you hear constantly from opponents of data centers and AI. They feel like there is sort of an elite political project to sort of make this technology ubiquitous so that people can sort of give away their agency to these machines. And Jill Lepore in her new book is basically saying, 'Yep, that's what's happening here.' And I've got the receipts to prove it going back hundreds of years. So, we're excited today to talk to Jill about the main thesis of her book as well as push her on some of the areas where we disagree. Let's bring in Jill Lepore.
Jill Lepore, welcome to Hard Fork.
Hey, thanks so much for having me. I'm like one of your swan song guests. Yeah.
Yeah. We're going out with a bang. Uh, we're so excited to talk to you. I've been a fan of your writing for many years. Your book, These Truths, was just incredible. And so I was very excited that you were writing a book about AI and about what you call the artificial state. So I want to first hear why you wrote this book. I think of you as a brilliant historian, a scholar of sort of technological pasts. Um, and this book is really about the present. So what made you interested in AI as a subject for... stay in your lane?
No, not at all. It's just we always just get really excited when people start paying attention, you know, to the stuff that we care the most about. I'm just teasing. Uh, you know, it is a weird book for me to have written. All my friends are like, 'Oh man, don't be writing about that. It's gonna be so grim.' Um, yeah, it was kind of grim. Um, you know, I'm mainly an American political historian. I write about a lot of other stuff. Like I'll write about anything. Um, and I've written a lot of pieces for The New Yorker about the history of technology. I've like been writing for the magazine for 20 years now. And like occasionally I'll do that. I teach a lot about, I teach a class on the history of technology. So like I just conceptually been thinking about this stuff a lot. And then last summer I was asked to do these lectures at Yale that are called the Tanner Lectures on Human Values. And I was just like the accumulation that I think people feel of the dehumanizing of the moment that we're in. Like, you know, you call to ask about your pet food delivery and you're talking to a computer, and who decided this is a way we should be living? Do you know what I mean? Like that question. So this is a long way around my sort of, I decided I really kind of wanted to write a short book that these lectures would be a short book about how it is that we have ceded so much of the functions of modern liberal democracy to machines that are automated, often now more recently driven by artificial intelligence, run by private corporations, and without so much as a scream beyond the emoji.
Yeah, Jill, I want to dig into this concept of the artificial state, which is sort of the thematic emblem of your book. I want to understand what you mean when you say the artificial state. Uh, you at one point compare it to the idea of the factory farming of humans. You also define it as being the rule of humans by machines manufactured by corporations. And you also have a number of passages in which you talk about how this artificial state is not a real state in the sense that it's, you know, striving for some kind of self-governance or internal organization, but that it is sort of supplanting the role in people's lives that their own actual states used to have, their local cities and states and federal governments. So can you just sort of sketch the basic idea of the artificial state as clearly as possible?
Yeah, the artificial state is an emerging successor to the liberal democratic nation state in which government is conducted not by the consent of people but by machines that are making decisions, and those machines are owned by corporations. So we don't live in the artificial state. It is something that I think is being built, but I think it is also, and this is an important part of my claim, it's also a fantasy that certain people have that they believe their power to be above that of the nation state. You know, the rhetoric is always there's always like a footnote or a paragraph to we do believe that people have control over their own lives and they elect governments to make decisions for them. But actually, we are in charge of the future of civilization and the future of humanity and the whole world's destiny. The destiny of the galaxy lies in our hands and you better trust us to do what's like that. The rhetoric of that comes out of this particular historical moment is really about the evisceration of modern liberal democracy, constitutional democracy.
Yeah. Out here in the Bay Area, in the middle of Silicon Valley, there is sort of a cartoonish caricature of the East Coast intellectual who greets all new technology and progress with scorn and mockery and dismissal and can't be bothered to get on board with the revolution and so just kind of sits in their ivory tower and laments the changing culture in front of them. Uh, you've been accused of being part of that tradition. I'm curious what your take on that is and what we are missing out here in the bubble that people like you are maybe better positioned to capture.
I remember years ago I went to Stanford. I was being recruited to teach at Stanford and we went out to dinner with the recruiting faculty and the people at the next table over were some, you know, youngish, very earnest young coders and they were talking about the homeless problem of San Francisco and how they were going to start a school for coding for the homeless. Oh no. And I was like, I don't think we can move here because I can't... I just can't. Like they were very sweet. Like I really, they were like my students. Like a lot of my, tons of my students of course go to work in Silicon Valley. Like Harvard, this is a huge recruitment thing. Um, and they're recruited with the promise like you're going there to make the world a better place. Um, and I hear from them a few years later and they're like actually that's not really what we were doing. Like it's a good recruitment message. I think there is a kind of sociological issue with Silicon Valley which is it is opposed to the idea of critique. Um, things are just supposed to continue to move ahead and there's not like the very idea of looking backward to assess what something has been and has done or even to look backward to say is there an antecedent for the thing we're proposing to do that might suggest a better way of doing it or might suggest we might not want to do it. Yeah. I find that really interesting. I think that's part of the commitment that was the kind of ideological apparatus of disruptive innovation.
I thought your work on disruptive innovation was great. I guess I'm more thinking of like, I'm thinking about the role of the critic in the sort of AI moment that we're in and like how best to shape the systems that are influencing people's lives right now. I've been thinking a lot about this because I've been among other things thinking about the transcendentalists and the group of writers and intellectuals who sort of reacted to the industrial revolution in the 1800s by sort of going back to nature, right? This was Thoreau going to Walden Pond because like the machines of the day seemed so dehumanizing. Like they were sort of taking all of the joy and spontaneity out of society and like organizing us into these little factory towns and they were just like screw this. I'm going to...
the woods and I'm going to like commune with nature and write beautiful books about what it's like to be a Walden pond. And I think there's like a sort of modern version of that which I'm curious if you see yourself as being a part of that transcendentalist tradition because in your book you do talk quite glowingly about what it is like to be in tune with nature and with animals and with beasts rather than machines. And I guess I'm just curious if you see any parallels between your own work and some of those reactions to the first kind of industrial revolution.
Yeah, I think that I do see some of that. I think that, you know, those guys are also romantics and maybe that's a label that applies to me. I think that I'm much more interested in these technologies than say Thoreau was. Like Thoreau, every time the train went by, was like, god damn it, you know. But I think no, I actually think these tools are incredibly exciting, right? These are tools. This isn't about transportation. It's about communication. It's about knowledge. Like it is the coolest thing that we can talk to something that's not a human. I just think that's unbelievable. Like I am fascinated by the language machine as an idea. Like in our lifetimes that this thing has emerged, like people have thought about this for so long. I use Notebook LM all the time. I like upload stuff and I'm like, give me some patterns. Show me what's here. Like I am not averse. I just actually think like if I want to decide should I pick my sunflowers and give the heads of the flowers to my chickens to eat or should I wait until they fall over first, I should ask my next door neighbor instead of Claude. Like for me personally, like I am not a person who would rather talk to a machine. I actually just think the idea that this extraordinary leap in human knowledge and our capacity to explore the world of ideas and the natural world around us in our lifetimes could come about and then be hawked at us like the cheapest new pair of shoes, but that everybody has to buy these shoes so that Sam Altman can have more money, that I'm not down with. I mean, I think there is a very real phenomenon here that is counterintuitive, and it was particularly counterintuitive during the social media age where these tools that were meant to connect us were actually just pushing us further apart from one another. And so, even something as simple as, you know, a tool that lets you answer a question about your garden or your chickens, it's incredibly convenient to be able to ask that at any time and not have to potentially interrupt your neighbor while they were doing something else. But in aggregate, it just means that we are more atomized and we are participating less in our democracy.
So I'm curious, Jill, if you could maybe give us a little flavor of doom and walk us through some of the bad scenarios here, like assuming this wave of populism peters out and the oligarchs remain in power, what are you so worried about?
I really cherish constitutional democracy and, you know, my last book was a history of the US Constitution and I know that before the emergence of the modern democratic nation state, all peoples in the history of the world had lived under various forms of tyranny in different degrees of tyranny and that was the promise of the American experiment. And we're kind of at that moment again. Like we are at that moment. I think we're, it's a real risk. Like think about how effectively or maybe ineffectively but at least how noticeably corporations have used the language of constitutionalism to describe their own activities. You know, Facebook started a Supreme Court. Anthropic wrote a constitution. These are not people, for all that they, for all whatever nationalism they possess, whatever lip service they offer to democratic action. They're not interested in what the people want because actually what the people want is not to have AI and not to have data centers. So that's the crisis, right? Like maybe people will decide in a few years. Maybe there's a moratorium and there's deliberation and people say, you know what, this actually is great. We really want to prioritize though scientific research, like these tools should be first available to, you know, the national labs and then maybe there's certain business interests that would be really great for these tools to be available for business. Like that I think can still happen, right? I feel like there's a fair amount of, you guys would know better than I do. I mean, I will admit I am an East Coast intellectual. I'm sitting here with my chickens and my sunflowers. You would know, like are people wanting that?
I think there is, I mean there is a desire for things to go more slowly, but I think there is also a worry that this technology is inevitable, that because the recipe for advanced artificial intelligence is so simple, because it is just a matter of getting as much compute and as much data as you can and shoving it into these models, that someone will develop this in the near future and like it is a moral obligation if you are a person who cares about having this go well for humanity that you not only don't impede that process but that you race yourself to get there first so that you and your safe AI can get to superhuman intelligence before China and their evil AI or some other American company and their less safe AI. So, I'm curious what you make of that inevitableist argument because a major theme of your book as I read it is that this is sort of a bogus premise that there is nothing inevitable or pre-ordained about the way that technology goes.
Yeah. I think it, I again I don't mean to question the sincerity of some of the people who believe that because I think that you could be persuaded that that is indeed the case and the best thing to do for human freedom would be to pursue AI. Like I have students who believe that and I don't think they're stupid and I don't think they're fools. I am myself not at all persuaded by it and I think for some very prominent actors it is bogus. I think it sits upon a number of other propositions that are also bogus and that are really more marketing slogans and political campaigns and those include the proposition that say regulation stifles innovation. The other proposition that sits on top of regulation stifles innovation is that technology always advances democracy. That then became a kind of mantra of Silicon Valley.
This is Mark Zuckerberg's argument.
That's what I'm saying. It goes back to the 1980s and it's the Milton Friedman regulation stifles innovation because God knows we shouldn't have to calculate the environmental cost of anything that we're doing. And it's just not true that regulation stifles innovation. Like empirically that's a false claim. That a technology always advances democracy, also empirically not a false claim, is a false claim. So now the AI people come back with the same argument that was made about the internet, personal computer, the internet and social media. Three times it's been wrong and then they say, 'Well, we should never look to history because that's what the East Coast intellectuals do.'
