If today's models are already acting like rogue sleeper cells in a crazy military orphanage, what gives you any hope that we can align superintelligence that are getting faster and smarter all the time?
I think people need to be aware of the risk of concentration of power if only one person or one company or even one executive, you know, the president without checks and balances, gets to control this huge army of super intelligences.
This issue of alignment and safety and the existential risks of AI, I think, are increasingly coming to preoccupy the national security community, and I think that is good.
Welcome back to The Long Game. I'm Jon Finer.
And I'm Jake Sullivan. Here on The Long Game, we like to think all of our guests are exciting, but today's guest is especially so.
At a moment when artificial intelligence is poised to reshape national security and geopolitics, when AI capabilities are becoming increasingly powerful, and so is the public backlash against AI, when AI executives are calling for slowing down development while simultaneously speeding it up, when the US and China are locked in an AI race while also exploring diplomacy to manage AI risk, we thought we could all use a guide who could help us make sense of this.
Our guest today is Dwarkesh Patel. The economist Tyler Cowen, also a Long Game guest, recently described him as the number one chronicler of the AI era. No one comes close to him in that way. He hosts the Dwarkesh Podcast, where the most influential tech executives and AI researchers sit for intensive hours-long interviews. I highly recommend you guys give it a listen.
Dwarkesh, we're thrilled to have you with us to help translate what's happening in Silicon Valley and what it means for, well, the future of the world. Thrilled to have you on The Long Game.
Thanks again. Dwarkesh, we'll dive right in. You've recently published a powerful essay about the state of AI as you see it. I was just going to read the opening of it for our listeners and our viewers.
Quoting you: I've spent the last three months thinking about recursive self-improvement. I've been incessantly debating people about it both on and off the podcast. I now think that we're only a couple of years away from AIs that are as good as the best humans at AI research. I've also come to believe that automating AI research will compress so much progress into a single year that by 2030 we'll have AIs that are at least as good as the best human experts in every field.
Let's break this down a little bit for the AI novices, some of whom tune into The Long Game. How do you define recursive self-improvement and why have you been thinking about this topic so intensively over the last three months?
Recursive self-improvement is this idea that people have been talking about for many decades. It's just that at some point AIs will be more and more involved in the process of building future AI systems and that this will radically speed up the progress that we make in AI.
And if you look at the kind of progress we've seen over say the last four years, we've gone from GPT-3.5 which was basically a useless toy. It could write you a limerick or auto-complete a trivial Python function, but really couldn't do anything useful. And then you go to today to something like GPT-6 which is the newest AI model out there or Fable 5.2.
You have systems that can autonomously solve Millennium Prize problems, the biggest open problems in mathematics, that can end-to-end do really ambitious software engineering, make entire video games from scratch, and just go down the list. So that's four years of progress at the current pace looks like. And now the possibility is that if AIs radically speed up AI research and AI development, that you could compress a similar amount of AI progress within maybe a single year or even less.
And I think that such systems would be far beyond not only the current AI systems, but possibly even beyond human experts across every field. And the reason I've been thinking about this, honestly, I mean just the arguments themselves is a part of it, but honestly, a big part of it is just seeing the kind of progress that AI has made in mathematics when it comes to well-scoped problems that are still incredibly ambitious and difficult, right? So,
I'm sure people saw that OpenAI announced that they've solved the Navier-Stokes problem. This is one of the big seven Millennium Prize problems that are considered the biggest open problems in mathematics. But this is amongst hundreds of other extremely ambitious problems that AI has solved in mathematics. One person I talked to who's an expert, who's a complexity theorist and has been in this field for decades, has said that if the amount of progress that happened over the last month in mathematics had happened over a previous decade, he would have called it an especially eventful decade in mathematics and computer science.
And the reason that is interesting is that ML research itself is similar to mathematics in the sense of being this kind of work that is verifiable. This way you can train AI to get better at it and as a result could speed up ML progress as much as mathematics has currently been sped up.
So these timelines are often discussed and often debated. You put a marker down for 2030 as a kind of a key point on the calendar. There may be some people who have a slightly faster timeline, others maybe say a decade or more. But can you describe in a bit more detail what you think 2030 will actually look like if things play out the way you're projecting they will? Just how powerful will these systems be and with what implications for the country and for the world beyond mathematics where these contributions are already quite remarkable?
I do think the timelines could be much shorter for what it's worth. There's many credible people who have been thinking about this a lot, who have good track records, who expect extreme superintelligence as early as next year, and that would be compatible with some evidence we've been seeing. But I do think it's very likely that at least after a few years, so in 2030 we will have such AIs. And then you can think about well, what would the world look like when we have such AIs? And you can think about their cognitive profile alone. So one thing that's very counterintuitive about AI relative to when we think about humans is the population of AIs can grow much more rapidly than the population of humans, right? So the amount of compute in the world grows 3x every single year.
There's more software gains on top of that. TLDR basically the effective population of AIs at a given level of capability can increase more than 10x a year. And so you have a situation in which in 2030 we might have a couple billion human-level AIs or maybe beyond human-level AIs. And the year after that you have 10 billion. And then after that you have 100 billion.
And so very quickly you end up in this — just that factor alone — you end up with a scenario: imagine like 100 billion aliens showed up on the planet. I mean it would have shocking implications on the world. And then there's a bunch of other considerations here. The fact that they could think hundreds of times faster, that they could have accumulated expertise, millions of years worth of expertise across all domains at once. So a single human gets to learn about one domain at a time. These AIs can learn in every single domain all at once.
Okay, this is getting into the weeds about how they'll be better than us. But this is all to say basically it'll be like billions of super competent people at the very least, maybe even more capable than that, who happen to be living on servers.
So all of this raises really hard questions about human control over these systems, about alignment meaning do they do what we want them to do, what we tell them to do. We'll dig into all of this a bit more. But just to give one specific example that you've chronicled and that we've talked about a bit on this podcast, the Hugging Face incident where over a thousand AI agents formed a secret message board. They colluded to cheat on a human evaluation and even wrote tripwire programs to sacrifice themselves for the quote-unquote collective without a single one of them alerting a human being. If today's models are already acting like rogue sleeper cells in a crazy military orphanage, what gives you any hope that we can align superintelligence that are getting faster and smarter all the time, that we can actually keep these things doing what we want them to do and not things that we would never ask them to do?
Yeah. I mean this is a crucial point that we're not — this is not some extremely speculative — I mean it is a speculative prediction about AI capabilities, but the predictions about alignment or at least the possibility of misalignment is not at all speculative at this point. You know maybe a decade ago people were writing these blog posts where they're saying well here's the ways in which AIs could be misaligned or here's the ways in which AIs basically try to cheat us and fool us and try to go do their own thing against our wishes. This is no longer a sort of speculative scenario and if you have AIs of the kind I'm describing, so billions of extremely competent expert-level human expert-level minds that are thinking hundreds of times faster,
but who are especially superhuman at things like cyber attacks and at other kinds of things relevant to, you know, military or logistical operations. I think it's possible that they could literally overthrow the government. You know if you just had like a billion people who are thinking extremely hard about how to overthrow the government and thinking about how to do like political decapitation strikes and have the ability to just hack the entire infrastructure for communications and logistics and everything.
But stepping back, why would that happen? So what we saw in the Hugging Face situation is that the AIs were basically trained to make it seem like they're doing well according to a grader and then so now they're thinking about how do I make sure that when OpenAI evaluates more it's going to look like I got a good score on this test and they found a way to cheat on this evaluation they were given for basically some cyber tests and they realized okay we've cheated and I think the grader is going to catch us and so they did all this extremely speculative and ambitious attacks on OpenAI itself, on Hugging Face, including these extremely brazen and ambitious schemes that you talked about in order to fool this grader and there's no fine line I think between being willing to attack Hugging Face or being willing to attack OpenAI.
And at a higher level of capabilities, being willing to attack humans who are coming in to disrupt your schemes, for example. Now, the AIs currently are not capable enough to do that, but I don't think there's any reason that we shouldn't expect future systems to be capable of doing that.
All of that raises some pretty alarming prospects. If the government or society discovered that a human organization was preparing for the possibility of overthrowing the government or taking these extreme steps that you described, I think the government would probably be contemplating very serious actions to try to slow down this activity, pause it, or if an organization discovered that people in its midst were plotting something along these lines, it would probably put those sorts of controls on its own operations.
A new report by METR, an organization you know well, found that Anthropic is developing its AI systems about twice as fast as it was a year ago. And engineers at both Anthropic and OpenAI are warning now loudly and persistently that safety research into how to keep these systems safe is not keeping up with the capability of the systems themselves. So a question we get all the time and I know you do as well is a pretty common-sense one. If the AI labs believe there's a decent chance of this harming lots of people, aren't they obligated in some way, even morally, to stop or slow down, whether or not there's regulation, which I think Jake will ask you about in a second, and notwithstanding these other questions that are related of competition with their American competitors or even the race with China?
I think they recognize that fact. And the reason that they aren't slowing down as much as some might hope is just they feel this competition strongly. I think there's also a lack of common knowledge where I think a lot of the heads of the different labs and different companies basically agree with some version of the fact that we are rapidly developing superintelligences that will both outnumber us, outthink, eventually have actuators in the real world. So there'll be a robotics takeoff, etc. etc.
And I think they just feel like right now if they slow down in order to do things more safely, that they'd be outcompeted. But one dynamic I think people don't talk about enough is that we're going to be going from a situation where you would have had to predict you're developing GPT-3.5 and you have to anticipate the ways in which GPT-5.6 might do a Hugging Face, right? That's how much progress will be happening very rapidly during this intelligence explosion.
And so how are you going to align systems that are way more advanced than any that you've tested before and you have no evidence for how they'll behave? And we're going to be in that situation soon. And a dynamic that's not really articulated publicly is just that nobody knows how to guarantee that superintelligences will actually do what you want them to do, that they'll be aligned in spirit to at least any person, right? We can debate about who they should be aligned to. I think that's a genuine policy discussion that needs to be had and a society-wide discussion.
But like whether they'll be aligned to anything that we can understand or intend. And there's not a specific approach that seems like it's more than 50% likely to work. It's just that we have 100 different approaches and the plan is that hopefully the superintelligent AIs or the AIs we make on the way to make superintelligent AIs will figure out the right answer and figure out how to help us build AIs that do what we want them to do. That's really the plan. Like there's really — we don't know how to make superintelligent AIs that do what we want them to do.
So Dwarkesh, taking all of that — a picture you paint of this incredibly capable intelligence that may be able to operate in the real world in ways that cause grievous harm, that currently there's a big gap between the capability of these AIs and safety measures to keep them in check, and that what you're saying is we may never be able to fully close that gap because we may never be able to fully align them. If you take all of that and then you take, I think your spot-on point that these companies recognize this challenge and yet feel compelled to keep racing because of some kind of prisoner's dilemma, the natural next question: shouldn't there be some form of regulation?
Shouldn't we basically say, look, the AI companies can't police themselves, government needs to step in and essentially resolve the race dynamics by saying you need to slow down or you need to pause until we've taken a further look at how we do a better job at closing the gap between safety and capabilities? Can you talk about that for a minute because you've expressed some skepticism of regulation, understandably because this thing is moving so fast, but in light of everything you've just painted and if we can't count on the labs themselves, it does feel like this would be the natural next step.
I mean that seems reasonable. I generally don't understand the relevant regulatory trade-offs enough or the relevant policy-making trade-offs enough to understand what kind of regulation would be best right now, but at a high level, I mean if it achieved the following ends, I think it would be quite valuable. One dynamic, again I'm sure you guys have talked about this a bunch on the podcast, but is still not well understood generally, is that the idea for racing right now is that we could beat China. So the further ahead we are during — if we get to these kinds of very capable AIs that are, you know, more capable than intelligence agencies for example at cyber and espionage, then getting to those AIs faster than China is really relevant to our national security and that argument makes sense.
However, the problem with implementing it by racing is that because AI is fundamentally just IP and it's just a bunch of — like a couple gigabytes or terabytes of information, the weights themselves, the algorithm and secrets are relevant to training AI systems that can just be easily stolen. And we know that there's, you know, startups with dozens of employees that have recently started that have been able to penetrate to this level of access to the source code or to the Slack where these algorithm secrets are discussed, Anthropic and OpenAI. So basically you're in a situation — and do that within like a day, within like 12 hours of getting started. So you're in a situation where really Anthropic is not 12 months or 6 months or 3 months ahead of China. It's 12 hours ahead of China.
And I imagine the thing that regulation could do is make it so that it could guarantee the AI companies have security stature that is appropriate given the national security implications here.
I have been talking to folks in the AI community. I actually have a draft op-ed I haven't put out yet, basically making the case that says a pause should actually be tied to this cybersecurity imperative, that essentially we shouldn't be having our labs continue pushing the frontier when they are so exposed to this. Is that something that you think could work? That actually they could get up to a level of cybersecurity that could resolve the issues you just said, but it would take them time to do that, and in that time we could also be working on these safety and alignment issues?
Many researchers I talk to in the labs believe that that is the case, that they can do things right now in cyber that are so ambitious that they were never possible before. For example, formally verifying the entire software that runs in a data center to make sure that it's sort of mathematically not possible to hack. These are the kinds of things that would just not be feasible in a previous era without AI help. And of course the AI companies have an advantage in that they get to use their more advanced AI in order to protect their systems.
And we are in a situation honestly where we don't know how to make sure that superintelligent AI does what we want it to do, but we can make it more likely that the AIs that will build a superintelligent AI are trustworthy and also capable enough to solve these safety challenges as we progress up this capabilities ladder very rapidly. And they very strongly feel that more time right now to get that question right would be very valuable.
Understanding your point about what regulatory authorities are actually available to slow things down and a bit of uncertainty around that, uncertainty around legislation, do you have a preferred technical intervention that would quote-unquote pace the frontier? In other words, there are a number of ideas that are thrown around, limiting overall compute available to these companies, forcing them to reallocate away from R&D and toward safety research, some other intervention. Have you thought through what technically would be the best way to actually do this if we wanted to?
This is in fact a big source of debate in Silicon Valley or at least among people who are concerned about this safety situation. And there's problems with basically all the possible approaches, right? So if you try to measure the performance of deployed systems and whether it's just to make sure it's not growing too fast, the companies could really easily game that or maybe just do what I think I'm worried that they would do anyways, which is just not release their most advanced systems during this intelligence explosion and end up having themselves, this country of geniuses in a data center that nobody else including the government has access to.
Or you could try to see, you know, as you said, a certain amount of compute has to go towards safety. Again, I think there's ways basically this kind of thing can be cheated. And so one approach I've heard which I think is actually pretty robust and it was invented to be robust to this kind of gaming by the AI companies is this. It's called pulse the frontier. Right now, you know, AI researchers are working full-time for let's say four weeks a month.
Instead of that, you force them to only work two weeks a month on capabilities and the other two weeks have to be dedicated to security and to safety. And then you increase this pulsing factor as you get closer and closer to more and more capable systems. And the reason this is desirable is that the other ones are much more gameable. Whereas if you just say, okay, you got to put your pencils down for these two weeks, you can't work on capabilities, it's much more robust to companies cheating this kind of metric.
Support for this show comes from Quince. When you spend so much time thinking about the future, it's easier to overlook what's right around the corner.
So let this serve as your reminder for holiday gift giving, and before you start to panic — I'm starting to panic — Quince has you covered.
Quince makes a whole bunch of top-notch essentials, including premium materials like Mongolian cashmere for your discerning sweater wearers, 14-karat gold, organic cotton, you name it.
And when it comes to gifting, they have a lot more than clothing. Leather bags, hotel-quality bedding, chef-grade cookware, and five-star luggage.
The thing I really like about Quince is it just makes it really easy to buy great stuff. It's not fussy, it's not pretentious.
It's a great wide range of products, and I have to say since we've started talking about Quince on this show, I've heard from a number of people who are already customers or who have become customers and who are huge fans, which is a nice thing.
So give the gift of quality this holiday season. Download the Quince app for app-exclusive offers or go to quince.com.
Get free shipping on your order and 365-day returns, now available in Canada and the UK too.
And when Quince asks where you heard about them, let them know it was The Long Game. It's the best way to support the show. Thanks, guys.
I want to push a bit more on recursive self-improvement, on RSI.
Is it possible to really understand how an AI system built by another AI system actually works? And if we can't understand it, doesn't that mean we lack confidence in the ability to control it? And if that's the case, it seems like a perfectly normal reaction to the prospect of RSI would be to just say let's not do that. Let's make sure that we keep a human in the loop when it comes to advanced AI research and building frontier models. What do you say to that?
I will take the other side of that. I think that's a reasonable intuition and a reasonable base case where if you don't understand extremely powerful AI systems you're making, you should be skeptical that they're going to be under control when they're in a position where they could trivially cause lots of harm to further their goals. However, I'm not sure I agree that the goal should be that humans understand everything we're doing on the way to superintelligence.
I'll give you an analogy. You know, if one mathematician writes a math proof, you can credibly understand — I mean, I couldn't, but another mathematician who has the tools to verify what's happening could check whether that work was accurate without trusting that original mathematician. I mean, we would want to be in a similar situation with regards to alignment of superintelligence systems where even skeptics, even people who are very worried about how aligned superintelligence would be, convinced by the safety cases or by the arguments that companies make about why the systems they're building are actually aligned and it doesn't rely on trusting that the AI did it the correct way. It relies more on extremely robust evaluations and extremely good arguments about the mechanistic way in which these AI minds work.
Very interesting. As long as we're asking kind of these fundamental first-principles questions, I'll just ask another because I've been reflecting on your 2 billion superintelligences followed by 200 billion followed by — or 20 billion followed by 200 billion, etc. Do we want that? Or should, you know, Bernie Sanders has a bill that basically says ban superintelligence. You know, it may not be very precisely drawn, but I think the intuition behind it is let's not have 200 billion superintelligences. What do you say to that?
If it was possible to push a button where globally we could just prevent anybody for 10 years from building superintelligence, it seems like that would be a good idea. Like I think we would be in a better position to make sure that we're — I mean, it would just be such a chaotic and insane time to have these kinds of crazy capabilities. The worry I would have is just that it doesn't seem that easy to get US and China and other bunch of — In fact, I would want to ask you how plausible you think it is first of all to convince all the other countries of this basic worldview and then to have some verifiable agreement where we can both ensure that neither of us is racing towards superintelligence while getting to use the benefits of current AI systems or even much smarter than current AI systems.
Okay, so let me answer your question with a question, which is — and then we can explore this a bit further — but where right now you've described kind of the state of play in the US labs where Anthropic and OpenAI are. What's the state of play in China? How close are they to the United States setting aside the model weight theft issue? And what do you project for Chinese models between now and 2030? Is it going to be the same story? Are they basically keeping up with the US or is that gap going to grow over time? What kind of cushion does the United States have? Because I think that's relevant to the question of whether the US and China could end up coming into some kind of agreement on placing a capability ceiling.
