What will March 2020 for AI look like? (Dwarkesh Patel)

All right everyone, let's have a warm round of applause for Dwarkesh Patel, the world's greatest podcaster. He's going to be talking to us today about AI, and afterwards there'll be some time for Q&A with the audience.

Yeah, so I do think the analogy is apt. To me, January 2020 was all about, okay, this thing is happening in Wuhan. It is possible it's going to come here, but like, most people aren't epidemiologists, right? In the 2003 bird flu, I don't really understand why that didn't come to America, right? But it's in the real realm of possibility. And that's the time to put together runbooks and things like that. Time for the elites to work behind the scenes to see if they can prevent or mitigate. February 2020 was all about, okay, anyone who's paying attention knows it's going to come here, but we don't have common knowledge of it yet, and we're not really capable of collective action. All the people who are thinking about it are kind of looking at everyone else like, is it okay to say it yet? And then in March, that's when the NBA cancels their games, and I think that was the day it became real for Americans. So I think for AI, we're in January right now, and it's time to put together runbooks, it's time to talk to other countries about international deals—but not time to ask something of the average American.

Maybe this is a counter to the analogy, but don't you have potentially an active incentive on the part of companies to cook the frog slowly? In other words, have an incremental release so it doesn't elicit a reaction. And therefore you didn't have that with COVID—an active slow-release process to avoid a reaction.

Maybe, although they're not going to necessarily delay the eventual thing—they're just going to do it slowly so you're acclimated to it.

Yeah, Jeffrey, you have some thoughts?

So I think one interesting parallel was the failure of containment—where at first we tried to contain COVID and prevent it from spreading, and that proved either too expensive or too difficult. But if you imagine COVID was 100 times more lethal, then it would have been probably the right thing to really try to achieve containment. And I think this is actually a pretty good metaphor, because if we're looking at cases where you have something—AGI—that can self-exfiltrate, and then there's a whole amount of free compute out in the world that it could potentially get—sort of the ability to contain that seems pretty important. And I think right now we're just pretty far away from the kind of compute controls that would allow us to do that and respond to emergencies. And that seems quite important.

Yeah, I think the reason the containment analogy does work is because we have done containment for like Ebola and things of that nature. If it was a lot more deadly, zero COVID would have worked, because everyone would have been on board with it. The problem was that it was only like 1% lethal or whatever, and so people felt like the cost-benefit of zero was not something that every world leader agreed upon. But it might have been if it was 50% lethality. So if there was consensus that x-risk was 50% or whatever, then maybe there would be containment.

Yeah, I'm not saying that you shouldn't build the data centers—I mean, that's a separate question. But it's like: should you be able to contain—if you have a situation where you have a disaster, or you have AGI that's out of the lab—seems like you'd want to be able to respond to emergencies. So that's not that you don't have to convince everyone that we should pause, but you should convince everyone that if we find ourselves in a situation where AGI has become the threat—can we respond to that threat? That seems like a pretty sane thing that most people would want to get on board with.

Well, no, I want to have the ability to respond to emergencies. So like in the case of COVID, a lot of people thought you couldn't shut down airports, you couldn't do these things. Eventually figured out, no, you can do it. It just has to be—but like, can we be ready in advance so that we could do that if we see those warning shots? Like, it seems like a good compromise position for people who have high P(doom), where it's like, well, yes, we should have the emergency capacity to respond.

I think a pretty interesting through line in the analogy for the economic slowdown is potential massive pushback from economic advances in the knowledge economy that come with AI—that perhaps aren't actually safety-focused but are more redistribution, personal-focus, whatever—that grabs a lot of attention in the same way that closing down did with COVID. But there's not a clear mission for the larger-scale problem, and how that could be distracting people.

I just wanted to bring it back around to the COVID analogy for a minute, because I was an ER doctor in New York City during peak COVID. And I have to say, it's amazing that you can see something sort of going badly in front of your eyes and people still won't believe it. And so I think the containment analogy is a good one, because sort of in the thick of things there were still a number of people—I don't think there was a single person in New York City who didn't know someone who died—and yet there were still plenty of people who were still going out, who didn't follow the containment rules, who didn't do all these things. And so I think that it's just human nature to kind of not believe that things are bad. And I'm not sure that having a higher death rate will actually change anything. I've been thinking a lot about this as it regards to the bird flu that's coming up, and I think all that will happen is people will take the lessons from things like COVID and say, oh, see, it wasn't that bad. And then society will collapse. And so I think we're really reliant—whether it's AI or whether it's zoonotic infections—that we have a group of people who are well-funded to say, listen, this is a problem, and no matter how people behave, we need to have a solution like a vaccine or some kind of fail-safe, so that once society does what society inevitably does—which is avoid the problem until it's sort of too late—somebody can rise up and say, wait, actually, it's not too late because we've been working on this for a while. And that's where I think things like mRNA vaccines are great, because we have this sort of semi-solution already teed up. And I think that we can look at a lot of problems this way and say, how can we kind of have a halfway solution here, so that no matter what the problem is, we can try to get the few people that were worried the sky was going to fall to actually stand up and save everybody.

I think it's pretty likely that either way, whoever wins in the political sphere, it'll be too late and our response will be mangled. So maybe the right thing is to just get some decision made, even if it's not the best decision, but get it made early and act quickly.

That makes sense, although if you lock in a bad decision—which we did. I mean, the CDC made a very fast decision: they said we're going to build our own test, we're not going to use a WHO protocol, we're going to build our own protocol. And the FDA made a very fast decision.

I was just going to make a prediction. The left and right—and we're seeing it already—the right's going to say that AI are biased to the left, the left's going to say that the AIs are racist, and they're all going to ignore the possibility of it killing all humans. Right? So.

Being an old guy, I remember the Cold War very clearly, and the National Security State never listened to the anti-nuclear activists. They just said, 'Look, this is an existential threat to us, so we're going to keep building these bombs, build more of them, better ones.' I just can't see the National Security State saying, 'Yeah, let the Chinese do this, we're going to accept all these regulations from you.'

Missile defense was a huge fraud and it still doesn't work now, even with our much better technology. Mostly Reagan was being sold all kinds of lies by Lawrence Livermore Lab scientists. The people were selling Reagan the dream that Star Wars could actually work.

No, that's not true. The people were selling Reagan the dream that Star Wars could actually work.

That's incorrect reporting of what happened when Iran hit Israel with a strike package. What actually happened is very different from what was reported.

Well, I think you point to the real epistemic question here. So for example, yes, if tomorrow because of stuff in Ukraine we get into a nuclear war with Russia and we're all killed, maybe MAD and building all those missiles was not a great idea at all. In the same way, like arguing about p-doom now, you could be totally wrong and we'll never know.

Yeah, the last 30 minutes have me kind of disputing the premise of this event, which is that there's an analogy here and that analogies are useful. And I know you're a student of history, so maybe you know a lot about the Roman Empire that made sense in your life, but my experience is analogies are maybe a good first approximation, a good way to take a first stab, and then afterwards we should just think harder object-level.

But isn't this the whole thing forecasters are trying to do here? Like you—all forecasters know is base rates, and all base rates are is history.

Yeah, I think they would disagree with that characterization. I think the base rates are better than trying to just come up with an answer out of the blue, but you do a lot of updating on the base rates. The base rates only work if you have some kind of track record. It doesn't work if it's just like one other thing happened that was kind of similar to this in some ways and not similar in other ways.

I mean, if you don't use that sort of interactability—at least I have, because I know very little about how the government works—if there's an emergency, you have sort of a black box of how will the people react. What will the competence of that reaction be? And COVID happened pretty recently. We know how the government reacted, we know what the incentives were across bureaucracies. So in that way, it seems like what else are we going to look at at how the system functions?

I think you're going to talk to the people in government and you're going to find out how their jobs work and build a cohesive mental model that both retrodicts several events from the past and makes predictions about the future. And then I think you're going to have some concrete reasoning about the ways that AI is different.

I also want to say, like what you said earlier about AI eventually slowly replacing humans in that decision-making process—there's a lot of talk around AI agents, and there's ideas about in the future it will be just a bunch of AI agents interacting with each other instead of having humans in the loop. And I think that's another possibility of AI slowly taking over the human decision process.

I kind of agree with you on the relationship between AI bots and people. The problem with human behavior is it's going to be just accentuated by the bots. But I do see that AI can actually help humans with their behavior problem, which is pretty bad. The only way I've thought about it is representing the living world, making the biosocial sphere legible, even giving trees and plants and ecosystems to be part of an economy. Imagine an ecosystem as a trust fund and things like this. It can widen our sphere in a way that actually could change us a lot. So I suppose I do think bots are just going to be—as we're aligning them with us—they're going to be as good or bad as our vision of our future selves. But I do think applying AI to the problem of how we represent our living world more respectfully is really a use case where they could excel.

So there's the question of which things will make forecasting better in general and which things will lead people to actually use the forecasting and incorporate it into their decisions. So I'm definitely very, very bullish on AI in forecasting and AI in Manifold in particular. We already have some—Dan, our sponsor at Future Search, is already running a bot on Manifold today using his LLM forecasting, and profitable, which is pretty impressive. There's going to be a lot more of that in the future. I would imagine the average human today is a terrible forecaster. It's probably already true today that GPT-4 is a better forecaster than the median human, so I think that trend will continue. But the other question is a political economy question. I would have to defer to our own markets and see what they say on that.

At least the way people perceive it, because we have monkey brains, is that there's many different ways the future could go and what alignment is trying to prevent are the paths of the future where it's just something we totally don't recognize, some zombie goop, paperclips. But the more you're constraining the ability of the AI to influence the future, the more you're circumscribing 100% of the probability to something a human or group of humans would decide. And so the more alignment is solved, the more it's just about conflicts between humans. The more alignment is solved, the more it's just going to become this sort of gnarly question of who wins, who loses. And obviously this had disastrous effects during COVID. One of the other things that Patrick McKenzie talks about in his article was that during the rollout of the vaccine, one of the reasons it was so inefficient is because of concerns about equity and other things, where you want to make sure the vaccines aren't going to only rich people and socially economically marginalized people get the vaccines as well. But that means you had to make 75-year-olds fill out 50-page questionnaires about are you really the person you say you are. So obviously they're not filling it out. The vaccines are just literally getting thrown away at the end of the day because they're out of cold storage, to make sure that the proportions match what the governor of California wants them to be, even if in absolute numbers fewer people of even the demographics he would want vaccinated get them. The point being that these sorts of cultural trends can actually have deadly effects because of bad decision-making in crisis. I wonder what the analogous thing will be with AI. Maybe it will be exactly the same thing—to your point that the culture hasn't changed that much over the last few years.

Other thoughts? I think in terms of whether people agree with things or not—when I first read Superintelligence in 2017, the thing that really crushed my ego was the idea of computing coherent extrapolated volition. It just seemed impossible. And since then we have things like RLHF that show some signs of finding values that broad swathes of humanity can agree with. So while I agree on a micro level that people will continue to disagree with each other forever, my hope is that in a world of extreme abundance—if we are successful at our mission—those small differences can sort of achieve their own expression in these people's own simulated worlds. There's some My Little Pony fan fiction where there's a super intelligence and that's the solution they have. So maybe there will be a world where each of the people can create a world where they live out their own personal ethical system, and the broad strokes of the bare minimum must be enforced for everyone. Otherwise—to maybe throw water over that—Israel and Palestine today is many multiples richer than it was 100 years ago or a thousand years ago. And it's not because of all this abundance now they can figure out how to make some rational trade that leaves them all better off. The same monkey brain thing kicks in, and we're going to be fighting over it when we're multitudes richer. We're going to have the AGI and we're going to trade off galaxies about who gets to control the wall in Jerusalem or not. So maybe we just become more enlightened because of AI, but I suspect not.

Other thoughts? I just want to say something much earlier during the discussion—something the general public doesn't really matter. What's influencing congressional staffers who can make sensible policies that matters, it's about getting smart people into technical research work that matters, etc. So I'm not really optimistic about converting or convincing many people, nor do they matter. I think that so, just to add to the conversation.

Nobody's asked me that before. Let's see—the main crux then is: is there going to be some sort of intelligence explosion-like dynamic? Because the March thing is the exponential, quote-unquote, is the AI making itself better. I just had an episode about this, and at hour three of it—which most people don't listen to—we went through the cruxes of the intelligence explosion argument. I don't find it that convincing. And so I think there will be superhuman intelligence enabled by AI, but I don't think it happens in a year like that. I think it's a longer process, so it's not like March—it's more drawn out.

Other takes? I'm surprised that no one mentioned a power struggle kicking things off—a political power struggle, kind of like a bigger version of what we saw with Silicon Valley on the edge of its seat when Sam was ousted from the board of OpenAI or fired as CEO.

I think we have seen exponential growth in nuclear technology during World War II, just between the edge of the US-Japan war. Do you think—and this is a question for everyone in the audience—let's just say that in five years China attacked Taiwan or got hold of the chip manufacturer, do you think we'll see a total war preparation where basically the government just gathers all of our ML researchers and ML engineers, locks them into a giant village in the middle of nowhere, and allocates the nation's 10% of electricity to build GPT-567? And that will be the March 2020 moment.

I think there's a general recognition now that government can't do this itself because private industry is so dynamic and huge—they really need it doing it for them. So if, for example, they said, 'Oh, we're going to just make another Manhattan Project,' but in China they have a vibrant AI ecosystem where it's incorporated in the banks and everything, they're going to be way ahead. So I don't see this idea that the researchers vanish and they're all just working for the government. You need a whole-of-society effort to move this thing forward if you want to be competitive with a society that's embracing the future, like in China.

And by the way, these guys can't just take the electricity—they have to pay for it somehow. So mostly what you're hearing is overhyping AI capabilities because Sam needs to raise money. Everybody needs to raise money to power the next round of training.

Right. I think they have the capital—that's what Zuck said on the podcast. They have the capital, they just can't get the allocation of the energy reasonably. So it's not an elastic market—if you displace the current grid, other people's energy bills really go up.

Other thoughts? I was just going to say—it seems like if there is a March 2020 moment for AI, the warning shot would be a swift rise in market capitalization. That's an incredibly credible thing for governments to see and to believe. And I think financial markets will respond more quickly than the political ecosystem.

Before that, it seems like he was about to follow up on that. Remember when the Goldman Sachs report came out? Nobody really took it seriously until that came out. And so what you just said accentuates my point: the credible point at which governments will believe there is a turning point for AI is the point at which major financial institutions are issuing reports suggesting that there will be big changes. Until that point, I think it'll be hard for government—

Didn't McKinsey do some report about how they expect a bunch of jobs to be automated? And until financial markets believe that, I don't think anybody else is going to believe it.

Yeah, I just want to push back a little bit on something Steve said—although maybe it's not directly pushback. As a history lesson: Enrico Fermi, who was part of the team that created the first nuclear chain reaction, said himself that it would be impossible to have an atomic bomb because it would require too many resources and no nation state would be willing to spend that percent of their GDP on making a nuclear weapon. And then he himself was part of the team that created the first nuclear weapon.

Right. Right. So that piece is—the government doesn't care until it really does. And then secondly, on the intersection with private industry—the reason he said that also was you would have to make city-sized factories to do enough nuclear refinement in order to get enough fissile material, and at the time nobody thought that would ever happen. But then they just contracted these enormous private companies that were used to making very large weapons development facilities, and they succeeded in doing exactly that as an engineering megaproject. And so you could say maybe America has lost its edge in doing giant engineering projects, but I think we probably haven't, and we could probably do the same thing again today.

Sure. Yes. Use the proceeds of your NVIDIA call options, turn them into lobbying dollars, use those lobbying dollars to get access to people. It's really hard if you have an unconventional theory about the world to get people to buy into it. But if you've had a series of unconventional theories that have all been proven true, and you can demonstrate with timestamps that this is what you said would happen, and then with brokerage statements that this is what you did about it—that you're demonstrating that you know what's going to happen and you know what to do about it to the best extent that you're able to—that's usually more compelling than just 'I've got a wild theory because I extrapolated from a chart,' even if the wild theory is true and the chart is actually worth extrapolating a bit.

Fair enough. Cool. How are you doing on time?

Oh yeah, sorry—half counterpoint to that is maybe the Cixin Liu brothers will read your interesting blog post and then do Fast Grants or something.

Interesting. Steve—I was just going to say that I really like the remarks you just made. And it's worth saying that a lot of the scientists who built the bomb wanted to establish a world government right away to control access to this super powerful technology. And among people who want to regulate AI, they want to create a world government to regulate AI. So I think the analogy is very tight between nuclear weapons and AI.

Also, one more observation on the Fast Grants thing, which just slipped my mind until now: they funded things like researchers doing very crucial public health research. The Yale endowment, which has $40 billion and gained 40% the year of COVID, did not fund that stuff. These people with a $60 million budget that they had raised themselves because of Silicon Valley wealth funded the things that the Yale endowment wasn't funding. So the lesson there, of course, is depression about the institutions, but also that if the emergency hits, you got to do it yourself. The people with the resources aren't necessarily going to be lining up to give them away to do the sensible thing.