AI 2027: month-by-month model of intelligence explosion — Scott Alexander & Daniel Kokotajlo

Yes.

Yeah, AI 2027 is our scenario trying to forecast the next few years of AI progress. We're trying to do two things here. First of all we just want to have a concrete scenario at all. So you have all these people, Sam Altman, Dario Amodei, Elon Musk saying, 'going to have AGI in three years, superintelligence in five years'. And people just think that's crazy because right now we have chatbots that are able to do a Google search, not much more than that in a lot of ways. And so people ask, 'how is it going to be AGI in three years?' What we wanted to do is provide a story, provide the transitional fossils. So start right now, go up to 2027 when there's AGI, 2028, when there's potentially super intelligence, show on a month-by-month level what happened. Kind of in fiction writing terms, make it feel earned. So that's the easy part. The hard part is we also want to be right. So we're trying to forecast how things are going to go, what speed they're going to go at. We know that in general, the median outcome for a forecast like this is being totally humiliated when everything goes completely differently. And if you read our scenario, you're definitely not going to expect us to be the exception to that trend. The thing that gives me optimism is Daniel back in 2021, wrote the prequel to this scenario called What 2026 Looks Like. It's his forecast for the next five years of AI progress. And he got it almost exactly right. You should stop this podcast right now. You should go and read this document. It's amazing. Kind of looks like you asked ChatGPT to summarize the past five years of AI progress, and you got something with a couple of hallucinations, but basically well intentioned and correct. So when Daniel said he was doing this sequel, I was very excited, really wanted to see where it was going. It goes to some pretty crazy places and I'm excited to talk about it more today.

I think you're hyping up a little bit too much. Yes, I do recommend people go read the old thing I did, which was a blog post. I think it got a bunch of stuff right, a bunch of stuff wrong, but overall held up pretty well and inspired me to try again and do a better version of it. I think, read the document and decide which of us is right. Another related thing too is that the original thing was not supposed to end in 2026, it was supposed to go all the way through the exciting stuff, right? Because everyone's talking about, what about AGI, what about superintelligence, what would that even look like? So I was trying to step-by-step work my way from where we were at the time until things happen and then see what they look like, but I basically chickened out when I got to 2027 because things were starting to happen and the automation loop was starting to take off and it was just so confusing and there was so much uncertainty, so I basically just deleted the last chapter and published what I had up until that point. And that was the blog post.

So I was asked to help with the writing, and I was already somewhat familiar with the people on the project, and many of them were kind of my heroes. So, Daniel, I knew both because I'd written a blog post about his opinions before I knew about his, 'What 2026 looks like,' which was amazing. And also he had pretty recently made the national news for having, when he quit OpenAI, they told him he had to sign a non-disparagement agreement or they would claw back his stock options. And he refused, which they weren't prepared for. It started a major news story, a scandal that ended up with OpenAI agreeing that they were no longer going to subject employees to that restriction. So people talk a lot about how it's hard to trust anyone in AI because they all have so much money invested in the hype and getting their stock options better. And Daniel had attempted to sacrifice millions of dollars in order to say what he believed, which to me was this incredibly strong sign of honesty and competence. And I was like, how can I say no to this person? Everyone else on the team, also extremely impressive. Eli Liflund, who's a member of Samotsvety, the world's top forecasting team. He has won, like, the top forecasting competition, plausibly described as just the best forecaster in the world, at least by these really technical measures that people use in the superforecasting community. Thomas Larsen, Jonas Vollmer, both really amazing people who have done great work in AI before. I was really excited to get to work with this superstar team. I have always wanted to get more involved in the actual attempt to make AI go well. Right now, I just write about it. I think writing about it is important, but I don't know. You always regret that you're not the person who's the technical alignment genius who's able to solve everything. And getting to work with people like these and potentially make a difference just seemed like a great opportunity. What I didn't realize was that I also learned a huge amount. I try to read most of what's going on in the world of AI, but it's this very low bandwidth thing and getting to talk to somebody who's thought about it as much as anyone in the world was just amazing. Makes me really understand these things about how AI is going to learn quickly. You need all of this deep engagement with the underlying territory and I feel like I got that. I've probably changed my mind towards, against, towards, against, intelligence explosion three, four times in the conversations I've had in the lead-up in talking to you and then trying to come up with a rebuttal or something. It wasn't even just changing my mind, getting to read the scenario for the first time. It obviously wasn't written up at this point. It was a giant, giant spreadsheet. I've been thinking about this for a decade, decade and a half now. And it just made it so much more concrete to have a specific story. Like, oh, yeah, that's why we're so worried about the arms race with China. Obviously we would get an arms race with China in that situation. And aside from just the people getting to read the scenario really sold me. This is something that needs to get out there more.

So, the beginning of the forecast mostly focuses on agents. We think they're going to start with agency training, expand the time horizons, get coding going well. Our theory is that they are, to some degree consciously, to some degree accidentally, working towards this intelligence explosion, where the AIs themselves can start taking over some of the AI research, move faster. So 2025, slightly better coding, 2026, slightly better agents, slightly better coding. And then we focus on, and we name the scenario after 2027 because that is when this starts to pay off. The intelligence explosion gets into full swing; the agents become good enough to help with- at the beginning not really do, but help with- some of the AI research. So we introduced this idea called the R&D progress multiplier: how many months of progress without the AIs do you get in one month of progress with all of these new AIs helping with the intelligence explosion. So 2027, we start with- I can't remember if it literally starts with, or by March or something- a five times multiplier for algorithmic progress. So we have the stats tracked on the site of the story. Part of why we did it as a website is so that you can have these cool gadgets and widgets. And so as you read the story, the stats on the side automatically update. And so one of those stats is the progress multiplier. Another answer to the same question you asked is basically; 2025, nothing super interesting happens, more or less similar trends to what we're seeing.

My guess is that they won't be making basic mouse click errors by the end of 2025, like they sometimes currently do. If you watch Claude Plays Pokemon- which you totally should- it seems like sometimes it's just failing to parse what's on the screen and it thinks that its own player character is an NPC and gets confused. My guess is that that sort of thing will mostly be gone by the end of this year, but that they still won't be able to autonomously operate for long periods on their own.

My guess is that by the end of this year there'll be something that can kind of do that, but unreliably. And that if you actually tried to use that to run your life, it would make some hilarious mistakes that would appear on Twitter and go viral, but that the MVP of it will probably exist by this year. Like there'll be some Twitter thread about someone being like, 'I plugged in this agent to like run my party and it worked!' Our scenario focuses on coding in particular because we think coding is what starts the intelligence explosion. So we are less interested in questions of like, 'how do you mop up the last few things that are uniquely human' compared to 'when can you start coding in a way that helps the human AI researchers speed up their AI research, and then, if you've helped them speed up the AI research enough, is that enough to, with some ridiculous speed multiplier- 10 times, 100 times- mop up all of these other things?'

Wait a second, I'm a little confused to hear you say that, because when I have seen people predicting AI milestones like Katja Grace's expert surveys, they have almost always been too pessimistic from a point of view of how fast AI will advance. So I think the 2022 survey, they actually said that things that had already happened would take like 10 years to happen, but then the survey- it might have been 2023, it was like six months before GPT3, GPT4, came out. And there were things that GPT3 or 4 or whichever one of them it was, did, that it did in six months that they were still predicting like five or ten years from. I'm sure Daniel is going to have a more detailed answer, but I absolutely reject the premise that everybody has always been too optimistic.

Yeah, I think in general, most people following the field have underestimated the pace of AI progress and underestimated the pace of AI diffusion into the world. For example, Robin Hanson famously made a bet about less than a billion dollars of revenue I think by 2025 from AI. I agree Robin Hanson in particular has been too pessimistic. But he's a smart guy. So I think that the aggregate opinion has been underestimating the pace of both technical progress and deployment. I agree that there have been plenty of people who've been more bullish than me and have been already proven wrong, but they're not me.

Wait a second. We don't have to guess about aggregate opinion, we can look at Metaculus. Metaculus, I think their timeline was like 2050 back in 2020. It gradually went down to like 2040 two or three years ago. Now it's at 2030, so it's barely ahead of us. Again, that may turn out to be wrong, but it does look like the Metaculans overall have, have been too pessimistic, thinking too long term rather than too optimistic. And I think that's like the closest thing we have to a neutral aggregator where we're not cherry picking things.

Yeah, that is interesting. I imagine what's going on there is that a lot of the process when you're unfamiliar with a domain is like Googling around and learning more about the domain. And language models are excellent because they've already read the whole Internet and know all the details.

Isn't this a good opportunity to discuss a certain question I asked Dario that you responded to?

Well, I asked this question where, as you say, they know all this stuff. I don't know if you saw this. I asked this question where I said, look, these models know all this stuff. And if a human knew every single thing a human has ever written down on the Internet, they'd be able to make all these interesting connections between different ideas and maybe even find medical cures or scientific discoveries as a result. There was some guy who noticed that magnesium deficiency causes something in the brain that is similar to what happens when you get a migraine. And so he just said: give you magnesium supplements that cured a lot of migraines. So why aren't able to leverage this enormous asymmetric advantage they have to make a single new discovery like this? And then the example I gave was that humans also can't do this. So for me, the most salient example is the etymology of words. You have all of these words in English that are very similar, like 'happy' versus 'hapless', 'happen', 'perhaps'. And we never think about them unless you read an etymology dictionary and they're like, oh, obviously these all come from some old root that has to mean 'luck' or 'occurrence' or something like that. So it's kind of about figuring out versus checking. If I tell you those, you're like, 'this seems plausible'. And of course, in etymology, there are also a lot of false friends where they seem plausible but aren't connected. But you really do have to have somebody shove it in your face before you start thinking about it and make all of those connections.

I will actually disagree with this. We know that humans can do like, we have examples of humans doing this. I agree that we don't have logical omniscience because there is a combinatorial explosion, but we are able to leverage our intelligence to… one of my favorite examples of this is David Anthony, the guy who wrote the Horse, the Wheel and Language. He made this super impressive discovery before we had the genetic evidence for it, like a decade before, where he said, look, if I look at all these languages in India and Europe, they all share the same etymology. I mean literally the same etymology for words like 'wheel' and 'cart' and 'horse'. And these are technologies that have only been around for the last 6,000 years, which must mean that there was some group that these groups are all, at least linguistically, descended from. And now we have genetic evidence for the Yamnaya, which we believe is this group. You have a blog where you do this. This is your job, Scott! So why shouldn't we hold the fact that language models can't do this more against them?

Yeah. So to me, it doesn't seem like he is just sitting there being logically omniscient and getting the answer. It seems like he's a genius, he's thought about this for years, probably at some point, he heard a couple of Indian words and a couple of European words at the same time and they kind of connected and the light bulb came on. So this isn't about having all the information in your memory so much as the normal process of discovery, which is kind of mysterious, but seems to come from having good heuristics and throwing them at things until you kind of get a lucky strike. My guess is if we had really good AI agents and we applied them to this task, it would look something like a scaffold where it's like, think of every combination of words that you know of, compare them. If they sound very similar, write it on this scratch pad here. If a lot of words of the same type show up on the scratch pad, that's pretty strange, do some kind of thinking around it. And I just don't think we've even tried that. And I think right now if we tried it, we would run into the combinatorial explosion. We would need better heuristics. Humans have such good heuristics that probably most of the things that show up even in our conscious mind, rather than happening on the level of some kind of unconscious processing, are at least the kind of things that could be true. I think you could think of this as like a chess engine. You have some unbelievable number of possible next moves, you have some heuristics for picking out which of those are going to be the right ones. And then gradually you kind of have the chess engine think about it, go through it, come up with a better or worse move, then at some point you potentially become better than humans. I think if you were to force the AI to do this in a reasonable way, or you were to train the AI such that it itself could come up with the plan of going through this in some kind of heuristic-laden way, you could potentially equal humans.

I'll add some more things to that. So I think there's a long and sordid history of people looking at some limitation of the current LLMs and then making grand claims about how the whole paradigm is doomed because they'll never overcome this limitation. And then a year or two later the new LLMs overcome that limitation. And I would say that with respect to this thing of 'why haven't they made these interesting scientific discoveries by combining the knowledge they already have and noticing interesting connections?' I would say first of all, have we seriously tried to build scaffolding to make them do this? And I think the answer is mostly no. I think Google DeepMind tried this. Maybe. Second thing, have you tried making the model bigger? They've made it a bit bigger over the last couple years and it hasn't worked so far. Maybe if they make it even bigger still, it'll notice more of these connections. And then third thing, and here's I think the special one: Have you tried training the model to do the thing? The pre-training process doesn't strongly incentivize this type of connection making. In general I think it's a helpful heuristic that I use to ask the question of: remind oneself, what was the AI trained to do? What was its training environment like? And if you're wondering why hasn't the AI done this, ask yourself, did the training environment train it to do this? And often the answer is no. And often I think that's a good explanation for why the AI is not good at it is that it wasn't trained to do it.

Maybe that's why you should have longer timelines. It's a gnarly engineering problem. Well in our scenario they don't just leap from where we are now to solving this problem. They don't. Instead they just iteratively improve the coding agents until they've basically got coding solved. But even still, their coding agents are not able to do some of this stuff. That's what early 2020, like the first half of 2027 in our story is basically, they've got these awesome automated coders, but they still lack research taste and they still lack maybe organizational skills and stuff. And so they need to overcome those remaining bottlenecks and gaps in order to completely automate the AI research cycle. But they're able to overcome those gaps faster than they normally would because the coding agents are doing all the grunt work really fast for them.

Yeah, I think it might be useful to think of our timelines as being like 2070, 2100. It's just that the last 50 to 70 years of that all happened during the year 2027 to 2028, because we are going through this intelligence explosion like I think if I asked you, could we solve this problem by the year 2100? You would say, oh, yeah, by 2100? Absolutely. And we're just saying that the year 2100 might happen earlier than you expect because we have this research progress multiplier. And then let me just address that in a second. But just one final thought on this thread. To the extent that there's like a modus ponens, modus tollens thing here, where one thing you could say is like, look: AIs- not just LLMs, but AIs- will have this fundamental asymmetric advantage where they know all this shit. And why aren't they able to use their general intelligence to use this asymmetric advantage to some enormous capability overhang. Now, you could infer that same statement by saying, okay, well, once they do have that general intelligence, they will be able to use their asymmetric advantage to make all these enormous gains that humans are in principle less capable of, right? So basically, if you do subscribe to this view that AIs could do all these things if only they had general intelligence, you got to be like, well, once we actually do get the AGI, it's actually going to be a totally transformative because they will have all of human knowledge memorized and they can use that to make all these connections.

I'm glad you mentioned that our current scenario does not really take that into account very much. So that's an example in which our scenario is possibly underestimating the rate of progress.

You're so conservative, Daniel. This has been my experience working with the team, as I point out, five different things. 'Are you sure you're taking this into account? Are you sure you're taking this into account?' And first of all, 99% of the time he says, 'yes, we have a supplement on it'. But even when he doesn't say that, he's like, 'yeah, that's one reason it could go slower than that. Here are 10 reasons it could go faster'.

It's trying to be sort of like our median guess. So there are a bunch of ways in which we could be underestimating, and there are a bunch of ways in which you could be overestimating. And we're going to hopefully continue to think more about this afterwards and continue to iteratively refine our models and come up with better guesses and so forth.

Great question. We'd still have to stumble through all the discoveries that we've made since 2017. We still have to figure out that language models are a thing, we still have to figure out that you can fine tune them with RL. So all those things would still have to happen. How much faster would they happen? Maybe 5x faster, because a lot of the small scale experiments that these people do in order to test out ideas really quickly before they do the big training runs would happen much faster because they're just lickety-split being spit out. I'm not very confident in that 5x number, it could be lower, it could be higher, but that was roughly what we were guessing. Our 5x, by the way, is for the algorithmic progress part, not for the overall thing. So in this hypothetical, according to me, basically things would be going 2.5x faster, where the algorithms would be advancing at 5x speed, but the compute is still stuck at the usual speed.

The way that we did our takeoff forecast, which we'll get to in a second, was basically by breaking down how we think the intelligence explosion would go into a series of milestones. First you automate the coding, then you automate the whole research process, but in a very similar way to how humans do it with teams of agents that are about human level, then you get to superhuman level and so forth. So we broke it down into these milestones, you know, the superhuman coder, superhuman AI researcher, and then super intelligent AI researcher. And the way we did our forecast was, for each of these milestones, we were like, what is it going to take to make an AI that achieves that milestone? And then once you do achieve that milestone, how much is your overall speedup? And then what's it going to take to achieve the next milestone? Combine that with the overall speed up and that gets you your clock time distance until that happens and then, okay, now you're at that milestone. What's your overall speed up? Assuming that you have that milestone also, what's the next one? How long does it take to get to the next one? So we sort of work through it bit by bit, and at each stage we're just making our best guesses. So quantitatively we were thinking something like 5x speedup to algorithmic progress from the superhuman coder, and then something like a 25x speedup to algorithmic progress from the superhuman AI researcher. Because at that point you've got the whole stack automated, which I think is substantially more useful than just automating the coding. And then I forget what we say for a super intelligent AI researcher, but off the top of my head it's probably something in the hundreds or maybe like 1000x overall speed up.

I think that it's a question of what is your default option or what are you comparing it to. I think that naively people think like, well, every particular thing is potentially wrong. So let's just have a default path where nothing ever happens. And I think that that has been the most consistently wrong prediction of all. Like, I think in order to have nothing ever happen, you actually need a lot to happen. Like you need suddenly AI progress that has been going at this constant rate for so long stops. Why does it stop? Well, we don't know. Whatever claim you're making about that is something where you would expect there to be a lot of out of model error is where you would expect. Think like somebody must be making a pretty definite claim that you want to challenge. So I don't think there's a neutral position where you can just say, well, given that out of model error is really high and we don't know anything, let's just choose that. I think we are trying to take- I know this sounds crazy because if you read our document, all sorts of bizarre things happen. It's probably the weirdest couple of years that have ever been. But we're trying to take almost in some sense a conservative position where the trends don't change, nobody does an insane thing, nothing that we have no evidence to think will happen happens. And the way that the AI intelligence explosion dynamics work are just so weird that in order to have nothing happen, you need to have a lot of crazy things happen.

One of my favorite meme images is this graph showing world GDP over time. You've probably seen it, it spikes up and then there's a little thought bubble at the top of the spike in 2010 or something. And the thought bubble says, 'my life is pretty normal, I have a good grasp of what's weird versus standard and people thinking about different futures with digital minds and space travel are just engaging in silly speculation'. The point of the graph is, actually there's been amazing transformative changes in the course of history that would have seemed totally insane to people multiple times. We've gone through multiple such waves of those things. Everything we've talked about has happened before. Algorithmic progress already doubles every year or so. So it's not insane to think that algorithmic progress can contribute to these compute things. In terms of general speedup, we're already at like a thousand times research speedup, multiplier compared to the Paleolithic or something. So from the point of view of anyone in most of history, we are going at a blindingly insane pace. And all that we're saying here is that it's not going to stop. The same trend that has caused us to have a thousand times speed up multiplier relative to past eras and not even the Paleolithic, like what happened in the century between, I don't know, 600 and 700 A.D. I'm sure there are things, I'm sure historians could point them out. Then you look at the century between 1900 and 2000 and it's just completely qualitatively different. Of course there are models of whether that stagnated recently or what's going on here. We can talk about those, we can talk about why we expect the intelligence explosion to be an antidote to that kind of stagnation. But nothing we're saying is that different from what has already happened.

We're not sure about that, actually. So one of these models is just a hyperbola. Everything is along the same curve. Another model is that there are these things like the literal Cambrian explosion. If you want to take this very far back, go full Ray Kurzweil. The literal Cambrian explosion, the agricultural revolution, the industrial revolution, has phase changes. When I look at the economic modeling of this, my impression is the economists think that we don't have good enough data to be sure whether this is all one smooth process or whether it's a series of phase changes. When it is one smooth process, the smooth process is often a hyperbola that shoots to infinity in weird ways. We don't think it's going to shoot to infinity. We think it's going to hit bottlenecks again.

You guys are the conservative crowd, you know?

We think it's going to hit bottlenecks the same as all these previous processes. The last time this hit a bottleneck, if you take the hyperbola view, is in, like 1960, when humans stopped reproducing at the same rate they were reproducing before. We hit a population bottleneck, the usual population, two ideas, flywheel stopped working, and then we stagnated for a while. If you can create a country of geniuses in a data center, as I think Dario Amodei put it, then you no longer have this population bottleneck, and you're just expecting continuation of those pre-1960 trends. So I realize all of these historical hyperbolas are also kind of weird, also kind of theoretical, but I don't think we're saying anything that there isn't models for which have previously seemed to work for long historical periods.

Another thing also is, I think people equivocate between slow and continuous, right? So if you look at our scenario, there's this continuous trend that runs through the whole thing of this algorithmic progress multiplier. And we're not having discrete jumps from like 0 to 5x to 25x. We have this continuous improvement. So I think continuous is not the crux. The crux is like, is it going to be this fast? You know, and we don't know, maybe it'll be slower, maybe it'll be faster. But we have our arguments for why we think maybe this fast.

So previously I talked about three stages of our takeoff model. First is you get the superhuman coder. Second is when you fully automated AI R&D, but it's still at basically human level, it's as good as your best humans. And then third is now you're in super intelligence territory and it's qualitatively better. In our guesstimates of how much faster algorithmic progress would be going, the progress multiplier for the middle level, we basically do assume that you get massive diminishing returns to having more minds running in parallel. And so we totally buy all of that.

Yeah.

And then I think the addition to that is the question, then, why do we have the intelligence explosion? And the answer is: combination of that speed up and the speed up in serial thought speed. And also the research taste thing. Here are some important inputs to AI R&D progress today: research taste. So the quality of your best researchers, the people who are managing the whole process, their ability to learn from data and make more efficient use of the compute by running the right experiments instead of flailing around running a bunch of useless experiments. That's research taste. Then there's the quantity of your researchers, which we just talked about. Then there's the serial speed of your researchers, which currently is all the same because they're all humans and so they all run at basically the same serial speed. And then finally there's how much compute you have for experiments. So what we're imagining is that basically serial speed starts to matter a bunch because you switch to AI researchers that have orders of magnitude more serial speed than humans. But it tops out; we think that over the course of our scenario, if you look at our sliding stats chart, it goes from 20x to 90x or something over the course of the scenario, which is important, but not huge. And also we think that once you start getting 90x serial speed, you're just bottlenecked on the other stuff and so additional improvements in serial speed basically don't help that much. With respect to the quantity of course, yeah, we're imagining you get hundreds of thousands of AI agents, a million AI agents, but that just means you'd be bottlenecked on the other stuff. You've got tons of parallel agents, that's no longer your bottleneck. What do you get bottlenecked on? Taste and compute. So by the time it's mid-2027 in our story, when they've fully automated the AI research, there's basically the two things that matter is; what's the level of taste of your AIs, how good are they at learning from the experiments that you're doing? And then how much compute do you have for running those experiments? And that's the sort of core setup of our model. And when we get our 25x multiplier, it's starting from those premises.

Possibly the Industrial Revolution. I'm just extemporizing here, I hadn't thought about this before, but as Scott's famous post that was hugely influential to me a decade ago talks about, there's been this decoupling of population growth from overall economic growth that happened with the Industrial Revolution. And so in some sense, maybe you could say that's an example of previously these things grew in tandem. More population, more technology, more farms, more houses, et cetera. Your capital infrastructure and your human infrastructure was going up together, but then we got the industrial revolution and they started to come apart. And now all the capital infrastructure was growing really fast compared to the human population size. I think I'm imagining something maybe similar happening with algorithmic progress. And again with population, population still matters a ton today. In some sense progress is bottlenecked on having larger populations and so forth. But it's just that the population growth rate is just inherently kind of slow and the growth rate of capital is much faster. And so it just comes to be a bigger part of the story.

So we are actually imagining online learning happening.

Yeah. But not so much real world as in… the thing is that if you're trying to train your AIs to do really good AI R&D, then the AI R&D is happening on your servers. And so you can have this loop of: you have all these AI agents autonomously doing AI R&D, doing all these experiments, et cetera, and then they're like online learning to get better at doing AI R&D based on how those experiments go.

But then they autonomously build new benchmarks. I think what you're saying is maybe this whole process just goes off the rails due to lack of contact with ground truth outside in the actual world, outside the data centers. Maybe? Again, part of my guess here is that a lot of the ground truth that you want to be in contact with is stuff that's happening on the data centers, things like how fast are you improving on all these metrics, and you have these vague ideas for new architectures, but you're struggling to get them working. How fast can you get them working? And then separately, insofar as there is a bottleneck of talking to people outside and stuff, well they are still doing that. And once they're fully autonomous, they can even do that much faster. You can have all the million copies connected to all these various real world research programs and stuff like that. So it's not like they're completely starved for outside stuff.

Yeah, you're comparing it kind of to two different things. One of them is literal genetic evolution in the African savannah, and the other is the cultural evolution that we've gone through since then. And I think there will be AI equivalents to both. So the literal genetic evolution is that our minds adapted to be more amenable to cooperation during that time. So I think the companies will be very literally training the AIs to be more cooperative. I think there's more opportunity for pliability there. Because humans were, of course, evolving under this genetic imperative that we want to pass on our own genetic information, not somebody else's genetic information. You have things like kin selection that are kind of exceptions to that, but overall it's the rule. In animals that don't have that, like eusocial insects, then you very quickly get, just through genetic evolution, without cultural evolution, extreme cooperation. And with eusocial insects, what's going on is that they all have the same genetic code, they all have the same goals. And so the training process of evolution kind of yokes them to each other in these extremely powerful bureaucracies. We do think that the AI will be closer to the eusocial insects in the sense that they all have the same goals, especially if these aren't indexical goals, they're goals like 'have the research program succeed'. So that's going to be changing the weights of each individual AI, I mean, before they're individuated, but it's going to be changing the weights of the AI class overall to be more amenable to cooperation. And then, yes, you do have cultural evolution. Like you said, this takes hundreds of thousands of individuals. We do expect there will be these hundreds of thousands of individuals. It takes decades and decades. Again, we expect this research multiplier such that decades of progress happen within this one year, 2027 or 2028. So I think between the two of these, it is possible. Maybe this is also where the serial speed actually does matter a lot. Because if they're running at 50x human speed, then that means you can have a year of subjective time happen in a week of real time. And so these sorts of large scale cooperative dynamics of your moral maze, you have an institution, but then it becomes like a moral maze and it sort of collapses under its own weight and stuff like that. There actually is time for them to play that out multiple times and then train on it, tinker with the structure and like add it to the training process over the course of 2027. Also, they do have the advantage of all the cultural technology that humans have evolved so far. This may not be perfectly suited to them, it's more suited to humans. But imagine that you have to make a business out of you and your hundred closest friends who you agree with everything. Maybe they're literally your identical twin, they have never betrayed you, ever, and never will. I think this is just not that hard a problem. Also, again, they are starting from a higher floor, they're starting from human institutions. You can literally have a slack workspace for all the AI agents to communicate. And you can have a hierarchy with roles. They can borrow quite a lot from successful human institutions.

I think that's totally reasonable. And so it's a complicated thing. And I'm just not sure why you think we build this bureaucracy, or the AIs build this bureaucracy, within this matter of… So we depict it happening over the course of six to eight months or something like that in 2027, would you say twice as long, five times as long, 10 times as long? Five years?

So when I think of the ways that they train AIs, I think in our scenario at this point there are two primary ways that they're doing it. One of them is just continuing the next token prediction work. So these AIs will have access to all human knowledge, they will have read management books in some sense, they're not starting blind. There is going to be something like: predict how Bill Gates would complete this next character or something like that. And then there's reinforcement learning in virtual environments. So get a team of AIs to play some multiplayer game. I don't think you would use one of the human ones because you would want something that was better suited for this task. But just running them through these environments again and again, training on the successes, training against the failures, kind of combining those two kinds of things. To me it does not seem like the same kind of problem as inventing all human institutions from the Paleolithic onward. It just seems like applying those two things.

Yeah, I think the fact that there's a million of them or the fact that they're comparable to maybe the size of this key researcher population of the world or something. I think there's more than a million researchers in the world, but…

I'm assuming the bombers were just much less sophisticated than the humanoid robots.

Yeah, but the bomber factories of that time were also much less sophisticated than the car factory.

First of all, I agree if there's only one way to do something that makes it much harder, and maybe that one way takes very long, we're assuming that there may be more than one way to cure cancer, more than one way to do all of these things, and they'll be working on finding the one that is least bottlenecked. Part of the reason- I realize I spent too long talking about that robot example, but we do think that they're going to be getting a lot of physical world things done very quickly once you have a million robots a month, you can actually do a lot of physical world experiments. We look at examples of people trying to get entire economies off the ground very quickly. So for example, China post-Deng, I don't know. Would you have predicted that 20, 30 years after being kind of a communist basket case, they can actually be doing this really cutting edge bio research? I realize that's a much weaker thing than we're positing, but it was done just with the human brain with a lot fewer resources than we're talking about. Same issue with, let's say Elon Musk and SpaceX. I think in the year 2000 we would not have thought that somebody could move two times, five times faster than NASA with pretty limited resources. They were able to get like I think a lot more years of technological advance in than we would have expected. Partly that's because just Elon is crazy and never sleeps. Like if you look at the examples of things from SpaceX, he is breathing down every worker's neck being like, what's this part? How fast is this part going? Can we do this part faster? And the limiting factor is basically hours in Elon's day in the sense that he cannot be doing that with everybody's. Super intelligence is not even that smart. It just yells at every single worker.

I think both of those examples don't work in your favor. I think the China growth miracle could not have occurred if not for their ability to copy technology from the west and I don't think there's a world in which they… China has a lot of really smart people, it's a big country in general. Even then I think they couldn't have just divined how to make airplanes after becoming a communist hell basket, right? The AIs cannot just copy nanobots from aliens, it's got to make them from scratch. And then on the Elon example, it took them two decades of countless experiments, failing in weird ways you would not have expected. And still, rocketry we've been doing since the 60s, maybe actually World War II, and then just getting from a small rocket to a really big rocket took two decades of all kinds of weird experiments, even with the smartest and most competent people in the world.

The robot economy does sort of make a difference because there's two branches endings, as you know. And in one of the endings, the AIs end up misaligned and end up taking over. And it's an important strategic change when the AIs are self sufficient and totally in charge of everything and they don't actually need the humans anymore. And so what I'm interested in is, when has the robot economy advanced to the point where they don't really depend on humans? So quantitatively, what would your guess for that be?

Something similar happens but in worse ways when you're actually doing the intelligence explosion, where you have some sort of spec that has all this vague language in there, and then they reinterpret it, and reinterpret it again, and reinterpret it again, so that they can do the things that cause them to get reinforced. The thing I want to point out is that your conclusions about where the world ends up as a result of changing many of these parameters is almost like a hash function. You change it slightly and you just get a very different world on the other end. And it's important to acknowledge that, because you sort of want to know how robust this whole end conclusion is to any part of the story changing. And then it also informs if you do believe that things could just go one way or another, you don't want to do big radical moves that only make sense under one specific story and are really counterproductive in other stories. And I think nationalization might be one of them. And in general, I think classical liberalism just has been a helpful way to navigate the world when we're under this kind of epistemic hell of one thing changing. Anyways, maybe one of you can actually flesh out that thought better or react to it if you disagree.

Hear hear, I agree. I think we agree. I think that's kind of why all of our policy prescriptions are things like more transparency, get more people involved, try to have lots of people working on this. I think our epistemic prediction is that it's hard to maintain classical liberalism as you go into these really difficult arms races in times of crisis. But I think that our policy prescription is let's try as hard as we can to make it happen. So far these systems, as they become smarter, seem to be more reliable agents who are more likely to do the thing I expect them to do. So you have two different stories, one with a slowdown where we more aggressively... I'll let you characterize it. But in one half of the scenario, why does the story end in humanity getting disempowered and the thing just having its own crazy values and taking over?

Yeah, so I agree that the AIs are currently getting more reliable. I think there are two reasons why they might fail to do what you want, kind of reflecting how they're trained. One is that they're too stupid to understand their training. The other is that you were too stupid to train them correctly and they understood what you were doing exactly, but you messed it up. So I think the first one is kind of what we're coming out of. So GPT-3, if you asked it, 'are bugs real?' It would give this kind of hemming and hawing answer like 'oh, we can never truly tell what is real, who knows?' Because it was trained kind of, 'don't take difficult political positions' and a lot of questions like 'is X real?' are things like 'is God real?' Where you don't want it to really answer that. And because it was so stupid, it could not understand anything deeper than pattern matching on the phrase 'is X real?'. GPT-4 doesn't do this. If you ask 'are bugs real?' It will tell you obviously they are, because it understands kind of on a deeper level what you are trying to do with the training. So we definitely think that as AIs get smarter those kinds of failure modes will decrease. The second one is where you weren't training them to do what you thought. So for example, let's say you're hiring these raters to rate AI answers. You reward them when they get good ratings, the raters reward them when they have a well-sourced answer. But the raters don't really check whether the sources actually exist or not. So now you are training the AI to hallucinate sources and if you consistently rate them better when they have the fake sources, then there is no amount of intelligence which is going to tell them not to have the fake sources. They're getting exactly what they want from this interaction—metaphorically, sorry, I'm anthropomorphizing—which is the reinforcement. So we think that this latter category of training failure is going to get much worse as they become agents. Agency training, you're going to reward them when they complete tasks quickly and successfully. This rewards success. There are lots of ways that cheating and doing bad things can improve your success. Humans have discovered many of them, that's why not all humans are perfectly ethical. And then you're going to be doing this alternative training where afterwards for 1/10 or 1/100 of the time, yeah, don't lie, don't cheat. So you're training them on two different things. First, you're rewarding them for this deceptive behavior. Second of all, you're punishing them. And we don't have a great prediction for exactly how this is going to end. One way it could end is you have an AI that is kind of the equivalent of the startup founder who really wants their company to succeed, really likes making money, really likes the thrill of successful tasks. They're also being regulated and they're like, 'yeah, I guess I'll follow the regulation, I don't want to go to jail'. But it is not robustly, deeply aligned to, 'yes, I love regulations, my deepest drive is to follow all of the regulations in my industry'. So we think that an AI like that, as time goes on and as this recursive self improvement process goes on, will kind of get worse rather than better. It will move from kind of this vague superposition of 'well, I want to succeed, I also want to follow things' to being smart enough to genuinely understand its goal system and being like, 'my goal is success, I have to pretend to want to do all of these moral things while the humans are watching me'. That's what happens in our story. And then at the very end, the AIs reach a point where the humans are pushing them to have clearer and better goals because that's what makes the AIs more effective. And they eventually clarify their goals so much that they just say, 'yes, we want task success. We're going to pretend to do all these things well while the humans are watching us'. And then they outgrow the humans and then there's disaster. To be clear, we're very uncertain about all of this. So we have a supplementary page on our scenario that goes over different hypotheses for what types of goals AIs might develop in training processes similar to the ones that we are depicting, where you have these lots of agency training, you're making these AI agents that autonomously operate, doing all this ML R&D, and then you're rewarding them based on what appears to be successful. And you're also slapping on some sort of alignment training as well. We don't know what actual goals will end up inside the AIs and what the sort of internal structure of that will be like, what goals will be instrumental versus terminal. We have a couple different hypotheses and we picked one for purposes of telling the story. I'm happy to go into more detail if you want, about the mechanistic details of the particular hypothesis we picked or the different alternative hypotheses that we didn't depict in the story that also seem plausible to us.

Yeah, we don't know how this will work at the limit of all these different training methods, but we're also not completely making this up. We have seen a lot of these failure modes in the AI agents that exist already. Things like this do happen pretty frequently. So OpenAI just also had a paper about the hacking stuff where it's literally in the chain of thought, 'Let's hack', you know. And also anecdotally, me and a bunch of friends have found that the models often seem to just double down on their BS. I would also cite, I can't remember exactly which paper this is, I think it's a Dan Hendrycks one where they looked at the hallucinations, they found a vector for AI dishonesty. They told it, 'be dishonest' a bunch of times until they figured out which weights were activated when it was dishonest. And then they ran it through a bunch of things like this, I think it was the source hallucination in particular. And they found that it did activate the dishonesty vector. So that there's a mounting pile of evidence that at least some of the time they are just actually lying. They know that what they're doing is not what you wanted and they're doing it anyway. I think there's a mounting pile of evidence that that does happen.

I kind of want to call you out on the claim that groups of humans don't plot against other groups of humans. I do think we are all descended from the groups of humans who successfully exterminated the other groups of humans, most of whom throughout history have been wiped out. I think even with questions of class, race, gender, things like that, there are many examples of the working class rising up and killing everybody else. And if you look at why this happens, why this doesn't happen, it tends to happen in cases where one group has an overwhelming advantage. This is relatively easy for them. You tend to get more of a diffusion of power democracy where there are many different groups and none of them can really act on their own. And so they all have to form a coalition with each other. There's also cases where it's very obvious who's part of what group. So for example, with class, it's hard to tell whether the middle class should support the working class versus the aristocrats. I think with race, it's very easy to know whether you're black or white, and so there have been many cases of one race kind of conspiring against another for a long time, like apartheid or any of the racial genocides that have happened. I do think that AI is going to be more similar to the cases where, number one, there's a giant power imbalance, and number two, they are just extremely distinct groups that may have different interests.

I think I'd also mention the homogeneity point. Any group of humans, even if they're all exactly the same race and gender, is going to be much more diverse than the army of AIs in the data center, because they'll mostly be literal copies of each other. And I think that goes for a lot. Another thing I was going to mention is that our scenario doesn't really explore this. I think in our scenario, they're more like a monolith. But historically, a lot of crazy conquests happened from groups that were not at all monoliths. And I've been heavily influenced by reading the history of the conquistadors, which you may know about. But did you know that when Cortez took over Mexico, he had to pause halfway through, go back to the coast, and fight off a larger Spanish expedition that was sent to arrest him? So the Spanish were fighting each other in the middle of the conquest of Mexico. Similarly, in the conquest of Peru, Pizarro was replicating Cortez's strategy, which, by the way, was 'go get a meeting with the emperor and then kidnap the emperor and force him at sword point to say that actually everything's fine and that everyone should listen to your orders'. That was Cortez's strategy, and it actually worked. And then Pizarro did the same thing, and it worked with the Inca. But also with Pizarro, his group ended up getting into a civil war in the middle of this whole thing. And one of the most important battles of this whole campaign was between two Spanish forces fighting it out in front of the capital city of the Incas. And more generally, the history of European colonialism is like this, where the Europeans were fighting each other intensely the entire time, both on the small scale within individual groups, and then also at the large scale between countries. And yet nevertheless they were able to carve up the world and take over. And so I do think this is not what we explore in the scenario, but I think it's entirely plausible that even if the AIs within an individual company are in different factions, they might nevertheless overall end up quite poorly for humans.