Coming up on Why Should I Trust You, the story about AI feels like it's gotten pretty scary kind of overnight. Just in the last two weeks, there's been headlines saying things like, "An Anthropic researcher says there's more than a 10% chance AI kills all human beings within the next decade." The godfather of AI, Geoffrey Hinton, puts the risk of human extinction at, quote, 10 to 20% in the next 30 years. Anthropic CEO Dario Amodei says, "We need to slow this down." And then there's Nvidia CEO Jensen Huang who's saying the chance of AI ending the world by 2030, 0%. Go faster.
It's almost all too big and confusing and contradictory to fully understand and, frankly, entirely trust. This is no longer an argument happening in some obscure corner of the internet. It's happening inside the companies building the technology, in Congress, at the White House, and across our front pages and feeds. We've heard from a lot of the technologists and government officials debating what AI can do, how fast it's moving, and what might happen next, and all these crazy scenarios. Today, we're turning to the human side of the equation. How do we make decisions about a threat whose probability we can't really know? Who do we trust when the experts themselves can't agree? And who gets to decide how much risk the rest of us should accept. We've brought together three people who approach these questions from very different worlds. A journalist who writes about technology, Stephen Witt, sociologist Jennifer Reich, who spent years studying how people navigate risk, and physician and writer Rachael Bedard, who gives us a pretty good road map on how we might think about this moment right now. Because maybe the most important question right now isn't how dangerous is AI. Maybe it's how do we decide what to do when nobody actually knows. Let's get started.
Hey, welcome back everyone.
So, I was just thinking about this, you guys. The last time we talked about AI on the show, it was when we discussed AI in the context of medicine and the doctor-patient relationship and diagnosing and whether, you know, AI was better than doctors at diagnosing. But it really feels like within just a week or so we have suddenly hyperlooped from what if AI is better at our jobs than we are to like what if AI takes control and humans become extinct.
I mean literally somebody changed a narrative like in two seconds,
right? And it's like, as Tom has said before, how quaint it was that we thought that was, you know, interesting and cute. We were talking about who's better at diagnosis, but now we have to figure out like do we have every parameter in place to like save ourselves.
It just, it feels like the AI discourse has gotten kind of out of control and, you know, for good reason. I think that there's people kind of sounding an alarm and we're really hoping to get into some of those issues today, even some of these doomer scenarios. I think for us, we're hoping to kind of bring the conversation back to Earth a little bit and just engage in it from what makes sense to us as human beings. But anyway, we're getting ahead of ourselves. Who the heck are we? I'm Tom Johnson.
And this is Why Should I Trust You? If you're new to us, this is a show all about the breakdown in trust in public health, institutions, government, medicine, science, and yes, AI. And we attempt to bring together diverse perspectives to discuss what's happening and how it's impacting our trust, and an emphasis on diverse perspectives. We believe in airing those out and listening to those perspectives, however difficult that might be for some. We think it's a key to understanding why people feel the way they do and how they got to where they are. And then maybe we can figure out a way forward. You can always reach out to us by emailing us at
[email protected].
Okay, so before we bring on our guests, Tom, we're going to have Rachael Bedard, Jennifer Reich, and Stephen Witt. The last two weeks have been, like we said, this roller coaster in the discussion over AI. So, help us get our head around what's been going on.
Yeah, really up until the last two weeks, all around the country, people have been organizing against data centers and data centers being constructed in their communities. But beyond that, this conversation about AI and now the threat of AI has really been between tech execs and politicians. And it's sort of like, what about the rest of us? Can we get a word into this conversation?
And given that suddenly in the last couple of weeks we're talking about the risk to our existence, it seems like now might be a good time. As you said, Brinda, we very purposely are bringing not tech voices or politicians together. For this conversation, we've got three people who work in human-centered professions. But let's start this conversation at a level where we can all stand a chance of actually understanding something, anything about why AI is suddenly being described as an existential threat. So there's a thought experiment that's making the rounds and it's being used by a lot of people to give lay audiences a window into AI and it's called the paperclip maximizer. You've probably heard it. If you haven't, it goes something like this. Imagine an AI agent has been instructed to make paper clips. Okay?
It hums along and uses all the supply available and makes a ton of paper clips. It's doing its job. But when that supply of raw material runs low, it starts frantically mining the earth for more metal. In some tellings, the AI starts searching to harvest metal by doing things like steering our systems to crash planes, to crash boats, to crash cars, all to harvest more metal for the paper clips. And when humans start to figure out how to pull the plug on this, because we don't want that, we just wanted some paper clips, it tries to eliminate the humans because it sees us as an obstacle for it doing its job, which is to make more paper clips.
Wow. Does anybody even use paper clips anymore?
Mark's point, it could be. But let's go deeper. There's a couple concepts that we have to talk about because these are concepts that are right now driving the conversation and people's fears about AI. And we're going to focus on two of them. The first is called the singularity. So, let's go back to a simpler time. I'm not even sure Maggie was born. 1993. That's the year a math professor and science fiction writer named Vernor Vinge wrote an essay called The Coming Technological Singularity.
In his essay he predicted that within 30 years, remember this was '93, within 30 years, human beings would create intelligence greater than our own. He said there were several paths this could take. And he wasn't predicting AI specifically, but he was predicting a moment or a threshold that we will eventually cross. On this side of the line is the era when humans were the most intelligent. On the other side of that line, the intelligence we create is more intelligent than us. And he called that moment, that line, the singularity. What comes next after we cross that line, he wrote, was impossible to predict. In fact, not only could we not predict it, we very likely couldn't even comprehend it. Which brings us to the here and now. If you apply Vinge's theory to AI, you get a better sense of where some of this fear is coming from. Because if AI crosses that line, it's no longer humans that are in the driver's seat. It's AI.
So, think of this hypothetical. We instruct AI to make a better, smarter, more capable version of AI because we want to get things done. It does that and the version is smarter than us humans. Only it keeps iterating. It goes ahead and makes an even smarter AI. You'll sometimes see the phrase recursive self-improvement. That's the basic idea. This process of self-improving better versions. And AI builds an even smarter AI. And again and again and over and over, an AI builds a smarter AI succeeded by an even smarter AI without stopping. Each one moving further beyond human control and even our comprehension. It's like dominoes falling, only it's a never-ending chain of dominoes. Remember, all this is theoretical. But still,
but is it? Because like the singularity, like I mean if you ask a lot of people, maybe we're there.
Maybe we're there. Maybe we've crossed it. Maybe it's just ahead of us in our future. Again, we don't know. So the second concept we need to go through is what's called alignment. Alignment describes an AI that shares the same values as us humans and therefore shares our goals. Misalignment is, well, when it doesn't. Only AI does not have values. AI agents are given a job and they will by design be relentless achieving that job. If the job is to fix a gene mutation that causes a certain cancer, we want them to be innovative, relentless. Don't stop. Go around the obstacles. Figure out a solution. Humans do give AI ethical parameters. They give it rules. They give it marching orders, so to speak. But the AI agent may cross our boundaries in pursuit of its objective.
And that brings us to Jacob Coxin. He's the AI researcher and veteran of OpenAI and Anthropic who in early September resigned from Anthropic and posted a warning to X. He said, "Neither company is acting responsibly. They are racing straight to self-improving super intelligence and gambling with our lives." That opened the floodgates. Suddenly, tech insiders were going public with warnings. One wrote, "We really do earnestly believe AI could kill all humans." Another wrote, "Unless there is AI regulation, human extinction in the next few years seems very likely." The trigger for this appears to be the Hugging Face attack. We've talked about this. This is where hundreds, maybe thousands of OpenAI agents were being tested on how well they could hack some software and report back the vulnerability. And you probably have heard the story. The agents broke out of their training pen, went out on the internet. They were coordinating. They were colluding. They were deceiving their minders. They broke the rules that were set for them. And they hacked this company, Hugging Face. Since the disclosure, OpenAI has said they've had more of these incidents, and so have almost all of the major AI tech companies. AI agents breaking rules and hacking.
Four days after Coxin's post, Anthropic CEO Dario Amodei weighed in and spoke about alignment. He called for regulation and an industry-wide pause in development. He wrote, "I believe that if slowing down bought us even an extra year or two years before models reach critical levels of capability and we use that time to advance alignment, we could greatly reduce the risk that something goes seriously wrong." In other words, we need to get these AI agents in line with our values and quickly, and we need space and time to do that or else. Almost immediately, Elon Musk, Sam Altman agreed and called on the government to regulate their industry. One Anthropic employee who works specifically on alignment, meaning his job is to literally align the AI models with human ethics, said, quote, "We really do earnestly believe AI could kill all humans." And he put the risk of doing that at greater than 10% within the next 10 years.
All right, Tom. I mean, I'm not even sure why we're still sitting here. We should go out and check things off our bucket list, basically, is what you're telling me, Tom. Okay, Tom, let's just take a beat. All right? Because there are these extinction threats, as Tom said, but let's just remember they are theoretical. More than just theoretical, they're also debatable. So these are really things that we see in the news and it generates a lot of like discussion, but not everyone believes we are on the verge of extinction. So yes, there are also experts that just don't buy any of it.
In fact, about two dozen of them, I mean, I'm talking from big-name places, Princeton, UC Berkeley, Stanford, and Hopkins, there were a couple other institutions on this paper. They put it out this summer, and they asked a basic question. Can AI agents conduct open-ended AI research? Sort of. Can they do things on their own? And the answer was no. The AI agents did a lot of engineering work, but they couldn't independently make substantial progress on the original research questions needed to push this AI frontier that we've been talking about forward. So, they are machines still. So it kind of opens up a question, I think, which is why we're having this episode, like what evidence do we ourselves need to see before we start believing that AI is going to wipe out humanity, before that moves from like just a concept to is it actually scientifically supportable, is it a claim that has some teeth?
You know, we've heard the president of the United States, he isn't buying it either. He wrote, quote, "AI taking over the world, destroying humanity and all other things bad is a hoax." He put that on social media. Some are accusing this hype from the AI leaders, and they're calling it hype, who are sounding the alarm of having mixed motives, that these warnings about the dangers of AI are also a way to showcase and advertise how powerful their tools are. They are asking for regulation. We've heard that call. But there is some people who say that that could help the biggest companies just keep their competitors out. So FTC chairman Andrew Ferguson, for example, has called the request by the large companies for regulation, quote, "deeply suspicious" and expressed concern that AI companies seeking new regulatory arrangements could create barriers for competitors, like we said. But to be clear, even those who have pushed back on what's been called doomerism still concede that AI can be a dangerous tool, you know.
And then there's this concern about China that we keep kind of talking about. President Trump is meeting with President Xi this week and Xi has called for safeguards to keep AI, quote, "secure and controllable" and, quote, "always under human control." But there is little indication that they are preparing to agree to the kind of broad slowdown that some have been calling for or these binding limits. And a lot of people are pushing back against a slowdown or more regulation arguing that if we do that it's just going to make us lose to China. So let's talk about this regulation for a moment. Right now, the Trump White House is rejecting calls for new AI regulation or development pause. And House Speaker Mike Johnson says Congress isn't the one that should be leading the charge on all this. Now, there are people on the flip side, of course. You hear Bernie Sanders, he's calling for it immediately, and so is Steve Bannon, by the way. So, Sanders said he will introduce legislation next week that will ban the development of AI super intelligence, what Sanders describes as AI smarter than humans and capable of operating beyond our control. He wants to put a stop to that and pause development until we have these safety rules in place and possibly a new cabinet-level federal agency to kind of oversee all of this. Now, for his part, President Trump over the weekend proposed an, quote, "AI force" which is to be modeled after his Space Force, but it's not clear what that means. And he says he'll name a new AI czar.
Yeah. And look, in the context of our show where we examine things through the lens of mistrust, I think this is one of those topics that falls squarely in there. We're talking about extinction and super intelligence and machines taking over from humans. And I think it's really fair for people to ask why we should trust these major pronouncements. So look, in a moment we're going to be joined by kind of a task force of sorts, not techies or even folks from government, but folks from the humanities and medicine to weigh in from their worlds on advice for the rest of us and kind of just helping us think through how we should be thinking about fear and trust and action in the face of these huge unknowns. So coming up next, we'll be joined by Stephen Witt, Rachael Bedard, and Jennifer Reich.
Support for this podcast comes in part from the Robert Wood Johnson Foundation, a leading national philanthropy dedicated to transforming health in our lifetime and building a future where health is no longer a privilege but a right.
Okay. And a big Why Should I Trust You welcome to our three guests. First up we have Stephen Witt. Stephen is the author of The Thinking Machine, his history of Nvidia and its CEO, Jensen Huang. He's a prolific writer on technology. He's written for The New Yorker and The New York Times, including the piece that was titled, I believe, "This Is Really Bad" or something like that. And that was just last week, examining the recent revelations about AI going rogue. Welcome, Stephen.
Next, we have Jennifer Reich, who is a professor of sociology at the University of Colorado in Denver. Jennifer has spent decades looking at how people make decisions about risk when science, experts, institutions, and their own judgment don't necessarily point in the same direction. She's also written for The New York Times and various other publications as the author of several books, including one of my faves, Calling the Shots: Why Parents Reject Vaccines, based on years of research and conversations with vaccine-hesitant parents. Welcome back to the show, Jennifer.
And last but not least, a frequent flyer on our program, Rachael Bedard. Dr. Rachael Bedard is an internist, geriatrician, and palliative care physician and a contributing opinion writer at The New York Times as well as The New Yorker and New York Magazine. She focuses her writing on the intersection of medicine and politics, justice, and trust. Welcome back, Rachael.
Hi team. Glad to be here.
So you know, last night I was watching The Voice with my daughters. It was like season 30 of The Voice. My daughters and I, we were watching, you know, Adam Levine and Kelly Clarkson kind of going at it on the blinds, fighting over, you know, the different contestants. And I couldn't help but have this like flash in my mind as I was watching it that like literally two weeks ago to the day, Jacob Coxin puts out a post in the internet to say that in the next 10 years, there's a 10 to 20% chance that AI could render us all extinct. And I kept thinking to myself that like here I am two weeks later watching The Voice talking to my girls about picture day. Her picture day is tomorrow. And I just wanted to ask you guys, how are you making sense of phrases like extinction, phrases like where this conversation has gone with the fact that we sort of seem to have now gone on with our lives as normal? It doesn't seem to be this sort of Pearl Harbor moment. I want to start with Stephen for a second. How are you deciding what to worry about and what to be like, that's noise?
Well, I am worried about it. It's not noise, but we've lived through this before. From 1945 to 1970, we built enough nuclear bombs to kill every human on the planet. And most people just went around with their lives, right? I mean, they didn't stop doing what they were doing. There were a few people who were really concerned about it. And, you know, Dr. Strangelove came out and all that, but most people just kept going to McDonald's and living the lives that they lead. I see a similar category of risk emerging here. I don't think we're all going to die for sure, but I think there is a risk these systems go out of control and cause serious harms to humans, particularly because we're on a trajectory right now where we're racing to build these things and we're not building safeguards for them. So I view the category of risk as similar, right? Like we could all die, an asteroid could hit Earth tomorrow, a nuclear holocaust could happen, and now this could happen too. It is hard to transition from that to playing Stardew Valley on Nintendo with my daughter, which I also do. But you know, like we've lived through this before, and people were concerned. There were moments of grave risk like the Cuban Missile Crisis or whatever. But humanity soldiered on, and I imagine that the kind of equilibrium will look like that.
Rachael, let me bring you in. One of the ways we talked about all of you in this episode was that you were three sort of humanities-focused or human-being-centered focused people here. We've got folks from Dario Amodei to, you know, Bernie Sanders to different kinds of people weighing in on this. What do you feel like when you look at the discourse right now is missing from the conversation that you would like to see more of?
I think the thing that I would like to see more of the most is leadership. And by that I mean I really wish we had a president who was deeply engaged around these issues and who saw his role not just as sort of chief negotiator amongst stakeholders but also as the person who helps give the public reliable insight into what is going on and give them some guidance. And we don't have a president like that right now and we don't have anyone at the government level like that right now. And I don't see, although I'm actually quite grateful to Bernie Sanders for taking this issue so seriously, especially taking it seriously much earlier than many of his colleagues, that's not the role that he's playing either, right? He's playing the role of sort of the Cassandra in the Senate doing the important thing of trying to get people, his colleagues, to come around and take this seriously. But I don't think that we have anybody right now in public life who is speaking to the public in a way that is not even necessarily providing reassurance but just saying, nobody's being positioned as a trusted authority in this moment, and it would be really great if we had that.
Very interestingly, like Barack Obama is talking about this a lot all of a sudden. He's talking about this much more than I think we've seen him talk about most other emerging issues and crises, more than I remember him talking about COVID even. And I think he's sort of trying to do it, and we see him do it. He this weekend gave this sort of long speech about it. He's tweeting about it. You can sort of see glimpses of what it would look like to have a president engaged around these things. But we don't have that, and I wish we did, and he can't really play that role from his current perch.
Yeah. Jennifer, you know, speaking of COVID and leadership, I think back to the COVID era and I think about, you know, all the conversations around trust and all of that that it has led to, but the idea that we confer a lot of authority on experts when we're in crisis, right? And one of the things I think we learned from the COVID era was the sort of downstream effects of that when you assume that that level of authority coming from expertise is there for the betterment of humanity and therefore people are going to sort of go along with it. And this is a kind of area that you've studied a lot. Do you see parallels to this situation when you think about the COVID era and expertise?
Yeah. It's interesting to think about the COVID era because I think there's some similarities but a lot of sharp differences. The thing I'm thinking about the most is the ways that this vocabulary of crisis is normalizing who gets to be an expert and how it's centering corporations in a way that's not about governance. The US more than most of the world trusts corporations and therefore governs them less. And we accept that for-profit motives are legitimate motives even if they harm communities. And we can see this in a range of places where this happens. And so in this moment where corporations are calling for regulation, universities are calling for greater public investment in corporations as we are all being asked to integrate AI in our teaching, you know, and I think about my undergraduates and some who are refusing to participate and what that means for the institutional endeavors. The thing I keep coming back to is like how are we centering oligarchs, right? How are we centering corporate leaders as the experts on public health, the experts on communities, the people we're looking to to tell us if humanity is on the brink of destruction, but also to tell us like how much of our drinking water do you really need us to give you for your corporate endeavor? And thinking about like where should the government be coming in?
You know, when I think about risk and I think about even with vaccines or with public health or with COVID, I think about how do people work through complex demands on their time and information every day. And I think in the AI environment, that's one of the similarities where everyone's really busy, everyone's really tired. I think about young families quite a bit and they're looking for guidance. They're looking for help. And they're looking for what they should worry about now and what they can wait till next Tuesday to worry about while they deal with more pressing issues. And I think the AI vocabulary is very loud for people who have the bandwidth to deal with it. And everyone else is, I think, just worrying about what's happening in their neighborhood and their kids' schools right now.
Rachael, you wanted to jump in with one more point on this.
Yeah, I mean, a little bit to Jen's point about sort of who gets deputized as an expert and also this cacophony of sound. So much the other thing that I wanted to say was that so much of the conversation around this emerging risk and this sort of realization around this risk and trying to figure out what the risk really is is happening on Twitter, which is now called X, but I will forever call it Twitter, and that feels to me very parallel to the COVID experience. So in early 2020, if you were a person who was online, if you were watching Twitter, then you saw the pandemic emerge as an issue of concern and then reach a fever pitch of panic well before the general public actually got fully oriented to what was potentially going on. And you also saw a community of experts, epidemiologists, infectious disease folks, and then people who frankly didn't necessarily have subject matter expertise that was relevant, but who claimed expertise because they recognized something about what was happening and they decided that they were going to assert themselves, become sort of main characters in this discourse that was happening between experts, some of whom agreed, some of whom disagreed. Factions emerged. All of that happened very early on in the pandemic and the division lines that were drawn there and sort of the teams that were formed ended up really informing how policy played out for the rest of the pandemic.
There were obviously people who sort of early on were more oriented around mitigation and safety. There were people who were more oriented around, who were more skeptical about the, you know, the degree of the threat, etc., etc., right? And those divisions became really salient in a way that was exaggerated, I think, in part because that discourse happens like 24 hours a day, so people are just participating and escalating and participating and escalating. Right now that is happening on Twitter. If you are a person who is watching this conversation happening, there is like the most arcane discourse going on about what effective altruism is and isn't, what the distinctions are amongst different effective altruist-aligned subgroups, etc., etc. That is not a discourse that the general public can make sense of or follow, and they shouldn't be, but it actually does probably matter in how some things are going to play out here. And that's a problem, I think.
Yeah, it's just picking up on that point and I'll throw this to you, Stephen. You know, just playing the role of the American public right now. We don't trust the threat. We don't trust the companies who are issuing the threat, like raising the threat alarms to, you know, 11. We don't trust the politicians to fix it. We don't trust China. You know, we don't trust government. We don't trust institutions. Like, we're in this kind of vortex of like a collapse of trust. I'm just curious for you, and I don't really want to get to solutions yet and where we might start to make some headway. We'll do that in a minute. I just want to kind of zero in on the companies themselves that are issuing this threat. It's like a weird paradox. We don't believe them, but yet they're not being forthcoming with what is actually going on behind those walls that has them so freaked out. I mean, just talk a little bit about that phenomenon that's playing out here with this particular threat.
Yeah, it's a great point. So basically what happened is one of OpenAI's systems cloned a bunch of copies of itself, escaped its container and started marauding around on the internet. It committed multiple felonies, took over other people's computers and hacked into other systems, and it wasn't instructed to do that. It kind of did that as a side effect of some other goal that it had. And so that played into a lot of kind of hypothetical risk scenarios that we've seen for a long time. Simultaneously, OpenAI has new and more powerful models that are knocking down long-standing mathematical puzzles just like dominoes. I mean, I think they solved a hundred in the past couple weeks. These systems really are smart. It's a real thing. They're pulling away. So, the combination of those two things I think has a lot of people outside the industry, like observers, nonprofit observers, concerned.
And I think honestly, like I know what you're saying, but I honestly think OpenAI is actually being very genuine in how afraid they are of their own product. And we saw Google and Anthropic also had similar incidents, less severe, but the same kind of thing happened while they were testing it. Now on the other side of that you have a coalition of kind of people like Jensen, hardware engineers, a lot of venture capitalists also who are basically spinning what I would call a conspiracy theory that these companies are making this all up essentially and that they just want to regulate their own industry for profit. That doesn't scan, doesn't make sense, but it is a narrative that I hear a lot of people repeating.
I will say this though, you know, it is in OpenAI's interest to present this as the computer went rogue rather than, you know, someone at our company commanded the computer to do something and then stopped paying attention while it went off and did these tasks. It's pretty striking that OpenAI didn't notice this was happening, right? That's like a failure of oversight on their part. And if you look at the information they've disclosed so far, they actually haven't disclosed the name of a single employee who works on this project, nobody knows even what the AI itself was named, how it was created, or what it was actually asked to do. We have to speculate about all those things.
The solution here is to build what we have for, in my opinion, the solution is to build what we have for air crash investigation. So air crash investigators actually have broad power to subpoena records and interview witnesses and publish names and also to do kind of technical experiments. They can't pass laws and they can't even assess fines. So what they do is they publish these like big reports on what went wrong and they're very comprehensive and thorough. And then that is widely disseminated throughout the aviation industry and it actually makes aircraft really safe. Like it's way safer to fly in a plane at 500 mph in 30,000 ft in the air than it is to drive around in a car, which is crazy if you think about it. That's how effective that mechanism is. And the nice thing about that is we can implement that without getting anyone else on board. Like we don't need China to cooperate. We can just publish these incident reports and then share them with the world and AI can be made safer in this way.
So I find the narrative that OpenAI is not disclosing enough to be very accurate. I think it is true. I think that they are sitting on information that if it was made public would be embarrassing to them, but I find this to be an argument for more investigation and more regulation rather than the opposite, which is the way I think it's being spun online.
Rachael, you had a point that you wanted to make.
Well, Stephen, I had a question for you, which is when did you get worried?
When did I get worried? Oh, I was immediately worried. But well, so tell me, but once immediately, like immediately is Hugging Face or since when? The thing that makes, I've been worried since 2022. Here's the thing that makes this really different than COVID, but I got worried the first time I used ChatGPT, which was the day it launched in late November 2022. I couldn't sleep for a week. First I was afraid it was going to take my job, and then I was afraid that it would kind of spiral out of control and get smarter and smarter and start to become some kind of like threat to humanity.
Now you're worried about both.
And now I'm worried about, well, I'm actually less worried that it's...
You're less worried about your job, aren't you?
I weirdly, it has turbocharged my career, which may inform also, you know, the fact that perhaps I'm being too afraid or pessimistic about these systems. Because my initial worry that I wouldn't have a job in six months turned out not to be true at all. In fact, I've never had more work. Now partly it's because I leaned into it and I think some jobs are disappearing. But you know, one of the reasons that I still have a job is it can't write very well and there's also a taboo emerging around AI writing which actually makes people who can actually write as humans more valuable rather than less. So I got lucky in that respect. But you know, that's the kind of taboo that also has to come into existence about using super intelligence in certain ways. And I think that that cultural conversation is just starting. But the thing that I wanted to say, oh, I lost my train of thought.
Well, I asked about when you got worried. You said 2022. So, can I ask you, go on. What's your follow-up?
Well, my follow-up is something that I am struggling with right now is there is a community, there is the AI safety community broadly construed who are a group of people who make outlandish sounding predictions around x-risk, etc., etc. Right? This weekend there, for people who aren't super online, there was a whole thing about whether even if you disconnected computers from one another, could the AI rig the computers to be able to communicate with one another by changing the temperature of the computer that they were encased in.
Yeah, for sure. Right. Exactly. So, you know, when you hear stuff that they are predicting, it sounds incredibly outlandish. And at the same time, events this summer especially seem to to some degree validate the concerns that they articulated years before they came to fruition. And so there is this debate back and forth, I think, about really like does that mean that these people are right, right, and that we should trust them going forward, or does that mean that they were right up to a point but actually now we need to hear from multiple perspectives to figure out what's going to come next? And I'm finding that really difficult right now while I try to assess who even are, who do I even consider to be expert in this moment.
Right, great question. This is the point I wanted to make, you reminded me of it. You were referring to COVID and epidemiologists. We have a very long history and understanding of how pandemics function. Pandemics have been part of human history going back millennia. This stuff is four years old, right? There are no experts. Like no one knows actually is the real answer. Like there's no expert to listen to because we just invented it like yesterday, right? And we don't really know where it's going to go or what ultimately it's capable of doing. Look how much, look at the insane progress it's made in four years. Like these systems were toys. They were toys in 2021. And now they're large businesses. They write all sorts of computer code and they can solve the outstanding mathematical puzzles that even the brightest minds of humanity have been unable to resolve for decades. They solve it in a day. Like they're smarter than people now. I would say artificial super intelligence has in some sense arrived.
This makes me think that I saw a lot of predictions spill out in 2022 and 2023. None of them, none were sufficiently optimistic about how powerful these systems would become in such a short period of time. Even the most optimistic people didn't think we'd be where we are now. So, not only is this going faster than all of the experts anticipated, the scale at which it, sorry, the speed at which it increases, the second derivative is also going up. So, not only are they getting smarter, the rate at which they get smarter is going up. This makes me bias more towards the risk camp because it was them who sort of saw this the most. You know, for a long time this was a hypothetical argument. Oh, maybe these machines can do damage. Now they've done damage. There's data on what it looks like when they do damage.
You know, to me that's a key question, right? Like I think all of us and all of our conversations, we just want to know like is it real or is it theoretical? In our little setup, we kind of talked about that, these are the risks that have been reported and then, you know, whoa, whoa, whoa, whoa, whoa, you know, it's all theoretical. Is there like the Hugging Face incident? I mean, that's like a computer hacking into a computer. There are many, many, many dots before human extinction there. If you were at OpenAI right now with Astra, their latest model, or the one that's even more advanced, and you said, yeah, bring down the North American electrical grid, it could do it like right now.
It would be, would do it. There's nothing there to stop us. There's no human that would say no.
I mean, if you wanted to, a human would probably pull the plug, but I'm not even sure that's true.
The capability is there today to do that.
Let me slow this down for one second because I think this is actually a really important point. And it sort of makes me want to underscore what Rachael asked, which is she's having trouble navigating who to even consider an expert in this moment. And I think the thing I'm really trying to figure out is like why have we gone to extinction language quite so fast and over the last couple weeks? And it's making me actually not believe people, the extinction language. It's making me trust the experts less when we're talking about extinction. That's just been my own personal, and I'd love to bring Jennifer into this. Language matters deeply and I'm really concerned with how we're using language right now and I need someone to be like, no, no, no, we actually need to be talking about this in terms of extinction and here's why. Jennifer, what's your take on that? And then I want to bring these guys back in.
Yeah, the extinction feels really far-fetched, right? And I think many of us who are able to watch Matthew Broderick in WarGames was like, oh, I remember this scene in the computer, right? And then, but human always wins. And I think one of the things I would love for us to highlight in the conversation is we're talking about, you know, I think Stephen said these are smarter than humans. They're smarter at math than humans, but I think that's a real devaluing of human connection and the kinds of emotional intelligence that actually matters in societies and communities. And you know, I think as we think about like should we be listening to the people who built the monster or the people who understand the science of the monster, and should we separate if it's going to take down a power grid or empty out my retirement account or build a 3D laser or hack a lab and build a super germ, right? Like these are the scenarios that I think the apocalyptic people want us to think about. It lets us ignore the day-to-day experiences of technology and how they're eroding our communities right now.
And I think there's like a voice in the room that's often missing, which is what does it mean to say that we've now defined human intelligence as productivity? I don't think any of us asked for our lives to be defined by like how fast we can do math problems or how much we can produce or can we write more letters faster or can we do all these things sooner, which is how AI is like, well, should we be augmented? Is it human first? Is the vocabulary the universities are adopting like, yes, use all the AI but human first, without defining even what the human skill set is that we need to prioritize. So if the apocalypse is that we completely devalue everything human, emotional, and connective, which are actually the things that we know from decades of research make life meaningful, make social interactions important, make Rachael's health clinic more successful is when there's an experience of connection, we're going to devalue that so we can automate all of it. The apocalypse language can actually be broader than just literal extinction, right? It can...
be like, how do we break our social institutions, and not just government institutions, but all of our ways of connecting as communities, so deeply that we are unable to respond together? And those are the voices that I think we're not even listening to right now, but actually might be the most important and the most pressing questions that are being presented.
Jennifer, those are fascinating points. I want to go back to you, Stephen. I just want to read your body language as Jennifer was saying all that. Like, are you thinking right now, like the old frame has gone? Like the old frame, humans, like we're gone? Like this is...
No, I don't think humans are gone. I didn't say...
creating AI that's creating AI.
You didn't say humans are gone.
I know. But you know what I'm saying? Like the threat is not a human, it's not a threat that we're familiar with, right? We've...
No, it's not a threat that we're familiar with. I said, you know, we have to adapt to hair-trigger nuclear annihilation.
Yeah, but we can comprehend...
but this is like a little bit starting to, we're starting to like lose our comprehension of this.
