Okay, take one. All right, here we are, Saving Humanity, take one. Having, say, for instance, the United States disrupt China's ability to develop a super intelligence. The main way in which they would develop it is if they get the ability to automate AI research and development fully and take the human out of the loop. Then you go from human speed to machine speed. We've had people such as Dario, for instance, the CEO of Anthropic, talk about how such a recursive process like that could lead to an intelligence explosion, and that would lead to a durable edge where nobody will be able to catch up. And last week, Sam Altman discussed how this process could telescope a decade's worth of AI development in a year or potentially a month. There's been this very seductive argument that has appealed to all of these people, which is basically, well, it's probably going to happen anyway. If we don't do it, someone else will. Basically, all of these people sort of trust themselves more than they trust everyone else and have therefore convinced themselves that even though these risks are real, the best way to deal with them is for them to go as fast as possible and win the race. Why did they make OpenAI? Well, they were worried that they didn't trust Demis to handle all that power responsibly when he was in charge of the AI project and all the fate of the world rested in his hands. So they wanted to create OpenAI to be this countervailing force that could do it right and distribute it to everybody and not concentrate power so much in Demis' hands. And in fact, the emails that came up in the lawsuit, they were talking about how they were worried that Demis would become dictator using AI. On the whole, I think that they follow the instructions fairly reasonably. There are some other parts of it. Yeah, that scares the shit out of me, right? Fairly reasonably, you know, when these things are still in our hands, kind of, that's okay. But if you have them guiding weapons or something like that, there are many circumstances where fairly reasonably is not good enough.
This episode of MLST is sponsored by Tufa AI Labs. It is a research lab which is headquartered in Zurich. They're moving to San Francisco as well. These guys are number one on the ARC-AGI v2 leaderboard. They're genuinely fascinated in building the next generation of technology, the next innovation which will take large language models to the next stage. If that sounds like you, please get in touch with Benjamin Kruier. Go to tufaabs.ai.
This podcast is supported by Google. Hey everyone, David here, one of the product leads for Google Gemini. If you dream it and describe it, Veo 3 and Gemini can help you bring it to life as a video, now with incredible sound effects, background noise, and even dialogue. Try it with a Google AI Pro plan or get the highest access with the Ultra plan. Sign up at gemini.google to get started and show us what you create.
All right, so I'm Gary Marcus. I'm a cognitive scientist, an entrepreneur. I've written six books, most recently Taming Silicon Valley. I think that, and here we are in the heart of Silicon Valley as we record this in San Francisco. And I think we all want what is best for humanity and hope that AI can be positive for humanity and not negative. I think the three of us are maybe not known for agreeing on everything, but we actually have a lot of shared values around that, and I'm looking forward to the conversation.
Thanks, Gary. My name is Daniel Kokotajlo. I'm the executive director of the AI Futures Project. We are the people who made AI 2027, which is a comprehensive, detailed scenario forecast of the future of AI.
I'm Dan. I'm the director of the Center for AI Safety. I also advise Scale AI and xAI. I also have done research in machine learning, like I coined the GELU and the SiLU, a common activation function. More recently, I've done evaluations to evaluate capabilities such as MMLU and the math benchmark and Humanity's Last Exam. And I've been focusing on trying to measure aspects of intelligence and also AI systems' safety properties for mostly the course of my whole career.
The common things that we care about are what is going to be the outcome when AGI comes? How soon is it going to come? How do we forecast these things methodologically? One person wrote in a question that maybe we could actually start, which is like, what is the positive view? I take it that all three of us in the room are united in thinking we think AGI could be a good thing. None of us is sort of like standing, you know, down like people would in front of a construction machine saying, stop it all, period. I think we all think there's at least a possibility of a positive outcome, but you can stop me if I'm wrong. And that we're hoping to steer towards that positive outcome. So our first question is, what could be the upside here? And why aren't you just saying, let's just forget about AI altogether? I can start with that. Yeah. So first of all, I actually think it's quite reasonable to basically stand in front of the bulldozer and say stop. I think that eventually we want to build AI because it is possible to get it right and it is possible to make it in such a way that's, you know, beneficial to everyone and massively beneficial to everyone, in fact. But if we are currently not on a track to getting it right and we're currently on a track to make it in a way that's going to be horrible, then it makes sense to just sort of stop until we figure out a better way. But yeah, as for what the benefits could look like, well, AI 2027 has the slowdown ending in which things go really well for almost everybody. And since not everybody read, I think more people probably read the darker scenario than the positive scenario. Lay out for people who might not have read the positive scenario a little bit about what the spirit of that is.
Sure. So in the slowdown ending of AI 2027, they manage to devote just barely enough effort and time and resources towards the technical alignment problem that they solve it just barely in time so that they can still beat China as they have wanted to do. By they, I mean, you know, the leader of the leading AI company and the president and maybe some other companies, right? That group of people. So, you know, the slowdown ending is not our recommendation for what we should actually aim for. We think it's an... to aim for this sort of scenario would be to expose the world to incredible amounts of risk of various types. So we don't think it's our recommendation, but nevertheless, it's a coherent, plausible scenario for how things could sort of muddle through and end up pretty well for everybody.
And pretty well is, I mean, like it's the Peter Diamandis abundance scenario, for example. Is that what you have in mind?
I haven't actually read that, but I would say when you get the AIs that are super intelligent, so what I mean by that is better than the best humans at everything and also faster and cheaper, you can just completely transform the economy. You know, super intelligent designed robot factories constructed in record speed producing all these amazing robots and all these amazing new industrial equipment, which then are used to construct new types of factories and new types of laboratories to do all sorts of experiments to build all sorts of new technologies and blah, blah, blah, blah, blah, blah. Eventually you get, and by eventually I mean in only a few years, you get to a completely automated, wonderful economy of all sorts of new technologies that have been iterated on and designed by super intelligences. Material needs are basically just met for everybody. You know, there's just an incredible abundance of wealth to distribute and, you know, things like curing all sorts of diseases, putting up new settlements on Mars and so forth, all that stuff becomes possible. So that's the sort of like potential upside. And then of course, there's the question of can we actually achieve that, right? And if we do have the technology to achieve that, who's in control of the technology? And do they actually use their power over the army of super intelligences to make that sort of broadly distributed good future for everybody, or do they do something that's more dystopian, you know?
So I think we, well, first of all, we'll come back to the bulldozers and whether you think we're actually at that point or not. But I think we can agree probably, and we'll let you chime in too, that there is a technical alignment problem and we should talk about that and a political, if it's not alignment problem, maybe that's too close pairing of words, but a political issue about if this technology exists, who controls it, that's deeply important. Yeah. And so we should talk about both sides of that sort of political control and technical alignment. Let me get your answer though on that first question. You know, do you think we need to stand in front of a bulldozer now? And if you don't, you know, what do you think is the positive? Yeah. So I think there are two concepts. There's AGI and there's ASI, the artificial general intelligence and artificial super intelligence. I think AGI doesn't have a clear definition. So it's difficult for me to say for all definitions of that that would be worth standing in front of. Some people would say we have AGI already. Let's just start with what we have now. Yeah. You know, there could be an argument that right now we should just stop the train. Sorry, I've moved from bulldozers to trains. You know, there'd be an argument. I'm not making this argument, but some people have made it. The argument would be we're already so far along this path and if we don't resist it massively right now, it's going to be bad and there isn't a good outcome here, a good enough outcome to offset what seems inevitable. I'm not making that argument, but I've heard people make it. And so partly, you know, grounding exercise, where are we on that? So, you know, one argument would be we've already screwed up. We need to have every person on the planet get in front of this train or bulldozer and say, just stop it because there isn't a positive enough outcome to warrant it. Another position would be there's actually a really positive outcome here and if we can guide it in the right way, we can get to that positive outcome and here's why I think it's positive. That's the kind of set of issues I wanted you to speak to first.
So I primarily think about stopping super intelligence. The main reason for that is because that is much more implementable, geopolitically compatible with existing incentives and so on. So it's something that I can actually foresee. I think things stopping tomorrow is not something I can as easily foresee. So I don't really think about that.
But is that just a fatalism then? I mean, like if you're worried about super intelligence and you might think if we move even towards AGI, which presumably is closer in time, that that's a bad idea because we won't be able to stop the super intelligence, like maybe we don't want to risk going further. I mean, I couldn't conceive of how we would coordinate, you know, now to do that.
I think that the asks are instead having, say, for instance, the United States disrupt China's ability to develop a super intelligence. The main way in which they would develop it is if they get the ability to automate AI research and development fully and take the human out of the loop. Then you go from human speed to machine speed. And we've had people such as Dario, for instance, the CEO, talk about how that will give the US a... or such a recursive process like that could lead to an intelligence explosion and that would lead to a durable edge where nobody will be able to catch up. And last week, Sam Altman discussed how this process could telescope a decade's worth of AI development in a year or potentially a month. And I think that's extraordinarily destabilizing for two reasons. One, if they control it, then or if a state controls it, such as, say, China, then all the other countries are at substantial risk because that super intelligence could be weaponized and used to crush other countries. And if they don't control it, which I think would be fairly likely by a nearly unsupervised, extremely fast-moving process, then everybody's survival is also threatened. So either way, this very fast automated AI R&D loop is quite destabilizing whether a state controls it or not. And so I think it makes sense, not just because, you know, AI is scary in some vague sense, but I think that there are very strong geopolitical incentives for state self-preservation.
I got that. I want to come back to the geopolitical aspect of it. And I read, I skimmed the paper that you just did with Alexander and Eric. But I don't think I got an answer for the first part, which is the upside. I don't think I need you to lay out the upside.
Yeah, yeah, yeah. So for upside, I think people generally think that if we have AI, it will necessarily hollow out all other values or that we'll have an extreme pressure toward one of them. That was we'll be zoo animals or we'll be all pleasured out and just, you know, in our VR experiences constantly and have superficial experiences. I think you could set a society up so that people have the autonomy to choose between different ways that they would want to live their life given the resources that AI increasing GDP could provide. So I think that there's a way you could have a list of objective goods being met in society. So I don't think there's no equilibria where things can work out. I think there's a way of achieving a variety of goods and still preserving human autonomy and all that, like sort of what we have now. Imagine we had things now, but then we have the autonomy to choose how we want to live our lives now. And there'll be more resources, some of us in the West have.
Yes, that's right. That's right. That's right.
And there's other sorts of longer-term questions about how you might do that, like how are you going to distribute power? Maybe that would be with compute slices that people would rent out, for instance, where they would have the unique cryptographic key to activating that compute slice so that you're not just distributing wealth, but you're also distributing ways of generating that wealth. You know, there's a lot in this deep future that one could pull out. But I think there are some paths where things work out well.
I like your term equilibria. I think we probably all agree that there's multiple equilibria here and that we're all trying to steer towards positive equilibria. And I think we would all also agree that there's lots of different ways to get to both the positive and the negative. We might disagree about some of the paths, some of the likelihoods and so forth. But I think we're all three operating under that general assumption that there's multiple ways this could go and some of them are good and some of them are bad. I'm a little darker than I used to be even a couple years ago because of the economic and equality sorts of issues. So a couple years ago when I trusted Sam a lot more than I trust him now, I heard him talking about universal basic income and, you know, I've always thought that that was an important part of the equation. And it seems to me that the more that data has driven things, the more inquisitive the companies, and it's not just OpenAI, have been, the less realistic it has seemed to me that we're going to have anything like a universal basic income. I think there is some scenario under which there's so much wealth that everybody's subsistence gets met. I don't expect that the wealthy people are going to give up the beachfront property under any circumstance or their power. And I think the dynamics that we've seen in Washington lately have made me, though I know our politics may not be the same, have made me less optimistic about any kind of relatively equal distribution or even just not a relatively extreme distribution. I think that the positive outcomes do depend on getting some answers there to what would be the incentives, the mechanisms, the dynamics such that things are well distributed. And I guess I'm sort of making this up as I go, but I think there's an argument here for stopping the train if we can't see any solution to the political thing. Most of my own personal concerns are really about technical alignment, which we'll talk about soon enough. But if we don't have an even envisionable political solution, that does make me nervous. And I could see an argument for let's stop the train now because we just don't see how to do it. And going back to something you said at the very beginning, there's an interesting argument about delaying the train versus stopping the train. And I think all of us signed the pause letter. Maybe you didn't sign the pause letter.
Okay, only I signed the pause letter.
That's interesting. I think that that didn't call that. That's fine whether or not the part of the pause letter that made me sign it that I liked the best was it said, let's pause this particular thing that we know is problematic in certain ways. I mean, it was really about delaying the development of GPT-5, which ironically still doesn't exist two and a half years after some of us signed the pause letter. But the notion was we would pause the development of GPT-5 because we knew that GPT-4 had certain kinds of problems around alignment and that we would spend that time instead working on safety. So it was explicitly constructed as a delaying tactic. It wasn't saying never build AI. It wasn't saying don't do any more AI research. It was in fact saying do more AI safety research and wait. And that does still seem like maybe not a politically viable thing in this moment, but at least a strategy one might consider is, you know, maybe we should be constantly updating our estimates on, you know, how likely are you to get the positive outcomes versus the negative outcomes and how much would that change as a function, for example, of putting more resources into safety research as opposed to capabilities research, etc. And what's your take on what I just said?
Totally agree. Like I think, yeah, like I like the Pause AI framing more than the Stop AI framing for the reasons you just mentioned. And then there's a lot more to say about more nuanced proposals for what is to be done. I do think we should maybe save a lot of this for the later part of our discussion after we've gotten through the technical stuff like timelines, alignment, etc. But you're the moderator, so I'm going to use moderator's prerogative for now, but you can push me a little bit later if we don't get there.
I'm trying to find some sort of common ground. I think we have lots of common ground. And yeah, and then we'll get to the differences. Do you buy that notion that also this should be on the table of some kind of clause or no?
I think making time for technical research, I'm not expecting much of a return on investment from that. I don't think most success strategies in the foreseeable future particularly go through AI being fully controllable. So I do technical research.
I'll give you a comeback on that. Like an extreme version of the argument I just placed comes from the cognitive psychologist and evolutionary psychologist Geoffrey Miller, who had a tweet that I really liked, which was, if it, you know, we should wait until we can build this stuff safely, even if that takes, I forget what he said, I'll say 250 years. And it was an interesting framing because like everybody's thinking like, what should we do next week or should we sign this pause letter? And it was kind of deliberately extreme. Like I think it was maybe even 500, like if it takes us 500 years, we should just wait. And, you know, I could see an argument for that. And in fact, I wonder what the counterarguments are.
So I think that the process that I described earlier, that sort of recursive loop, I guess you could call it an intelligence recursion, which if it goes fast enough is an intelligence explosion. That is not something you can research your way out of. You can't just, you know, write an eight-page paper and then we've solved it, you know, in the news tomorrow. It's we figured out how to fully derisk some extremely fast-moving process that we've never done before. And all of its unknown unknowns have been anticipated.
Well, I mean, we could politically, I mean, it would take a lot of willpower and it's probably not likely, but we could, you know, try to have a global treaty. Don't go there. Don't work on these kinds of things. Let's report it if you do, right? Right. I mean, you could at least imagine that kind of scenario.
I don't see any technical way of coping necessarily with that set of problems right now.
But if we were, I mean, not just the three of us in the room, but if we as a society were convinced that, let's say, that was a red line, you suggested one red line, there are others. So we could decide as a society there are certain red lines. Maybe recursive self-improvement might be a reasonable one to consider, that we can say we won't cross any of the red lines. We will make treaties around it and we just shouldn't do it until we have answers to them.
So I think it would be worth states clarifying that we don't want anybody doing that type of a fully automated intelligence recursion because it'll be destabilizing. And now that doesn't necessarily immediately take the form of a treaty. So initially you have them exchanging words about that and articulating their preferences, potentially explicitly or potentially in internal policy at CIA and other things like that. And the other parties come to learn it through leaks or directly. And you eventually want to gain more confidence that nobody's trying to trigger something like this. And this process is multiple stages. It may involve things even like it requires a conversation. And it may even require a skirmish for people to think, okay, we need to do a verification now. And then maybe you get some type of treaty, but you can still have various forms of coordination through deterrence without anything like a treaty, just as we've had strategic stability on multiple issues without treaties necessarily. So that's something that is potentially a later stage thing, but you have to have the conversation advance far further.
Daniel's giving me a look on the...
No, I'm just looking back and forth.
You're just looking back and forth. Why am I in this? You have to like rub your neck because I'm right in the middle. Yeah, in the middle. Okay. So let's separate for a moment the deterrent stuff, although I know you're keen to talk about and know more about it than I do. And let's try to get to it. You could sort of separate what are the dynamics for which we would form treaties. Maybe we need some disturbance and skirmishes as you just talked about, but from the question of like, would the rational thing for civilization to do right now be in fact to sign these treaties because we're basically pretty close? I maybe take a longer timeline than you, but we're reasonably close to recursive self-improvements of at least some sort. And so like maybe we don't want to go there. Maybe that's what the rational thing to do be right now to make those treaties. Or even if like we thought that was 25 years away because we know treaties take 8, 10 years, like maybe we should be putting all our, you know, intellectual capital or political capital or whatever into doing that right now. What do you think?
Yes. So I think three red lines would be no recursion where that's a fully automated, not just some AI-assisted one, but one that has that explosive potential. No AI agents with expert-level virology skills or cyber offensive skills made accessible without some safeguards. And model weights need to have, model weights past some capability level need to have some good information security for containing them and making sure that they're not exfiltrated or stolen by rogue actors.
In a minute, I'm going to ask you how close we are on these various things. But what's your take on what he just said? Is that those same with the first?
I don't know about the second two, the other two, but definitely the first one. I think that if somehow we could coordinate on that, that'd be great. I'm not sure if that's the best red line to coordinate on, but it's an excellent place to start at least.
Do you want to throw in any others now, or you can later in the conversation?
I think that the type of thing that I'm probably going to end up advocating for is going to be more of a, rather than like, here's a line that we're all not going to cross, something more like we are going to gradually develop AIs with these capabilities, but we're going to do it in a way that's like mutually transparent to each other and that proceeds sort of slowly and cautiously where we all like debate whether it's safe to go to the next level, and then after we get there, we study it for a little bit and then debate whether it's safe to go to the next level and so forth. So the sort of thing that we're probably going to end up advocating for is going to look something more like that rather than... But yeah, in terms of like the thing that you really need to like stop from happening in the short term, that sort of recursive self-improvement thing is, I would say, the number one thing.
Right. So this conversation is a little bit depressing in the sense that many of the things that we seem to be worried about actually seem fairly close. Maybe the person in the room who's most pessimistic, if that's the word, let's not, the most extended in time on this is me. But all of us, I would say, think that... Well, no, let me rephrase this question. It's a dark set of answers relative to the reality right now. I think in the following sense, even if you think it's going to take a while to get to AGI or ASI or something like that, and we'll talk about that in a little bit, the things that are red lines, and I like your red lines, people are already pushing against them. They may not be breaking through them. Like depending on your definition of recursive self-improvement, I might give you different estimates. Mine might be a little longer than yours, but people are already trying to do that, right? And transparency is like 1990s talk. Like it's like it's in the rear... I'm exaggerating a little bit, but it's in the rearview mirror. I mean, like OpenAI was open originally. It is not open anymore. So, you know, there are elements still pushing for transparency, but there are certainly elements pushing against.
So I'm somewhat more optimistic about increased transparency and more situational awareness from governments. A reason for that is I think it's incentive compatible for the US to be more transparent about what's going on at the frontier because China already knows. Meanwhile, for PLA or People's Liberation Army developments, there's somewhat less transparency there. So the people to gain more from information would probably be the rest of the world. So China has somewhat more of an advantage there. And this sort of levels things and that having high transparency into the frontier is very useful for making more credible deterrence. Because then, yeah...
Let me interrupt for a second about a time frame question. I think the notion of a durable advantage in LLMs, if that's what the technology is, is a myth. Like if we stay on LLMs, nothing's going to be durable. But I think you think about these things in a little bit different way than I do. I think with the current paradigm, I am expecting them to continue to leapfrog each other. And so roughly parity is what I'm seeing, right? So there could be a different technology, you know, my personal favorite would be neurosymbolic AI, that somebody gets a durable advantage of. Maybe, but, you know, there's going to be espionage. People are going to share ideas and whatever. But if the paradigm stays roughly like it is, I don't see anybody getting a durable advantage. So that's part one is, do you agree, disagree with that?
I think it depends on what you mean by paradigm. So I would say like LLMs are just a part of this or a subset of this overall like AI research. You know, they're not the only possible AI design and progress is going to continue being made. I mean, I would say like in some sense, arguably, we're already seeing a move away from LLMs with things like these so-called reasoning agents that have access to tools and can write code and so forth. So there's going to be this continuous shift towards, I would say, rather than like discrete paradigm shifts, it'll be more like a continuous paradigm shift. And the AIs of 2027 are just going to be like quite different from the AI of 2023, you know. But taken as a whole, the AIs, I do think that the United States could potentially end up having a durable advantage over China and one particular tech company within the United States could end up having a durable advantage.
I see this as possible. I see that as unlikely unless somebody approaches the problem in a pretty different way. Like I can imagine a neurosymbolic approach that would be just so different from what anybody else has that I could see at least for a while an advantage in the current way of people are doing things. I don't even know what that would look like.
Perhaps I should clarify what I mean by durable advantage. I don't mean they never figure out what you already figured out. I mean that by the time they catch up, you've already moved ahead. So that it's like, you know, you keep running as fast as they're running so that even though they're only six months behind, they can never quite catch up because by the time they do, you're six months ahead again.
And that was part of the other part of that question I wanted to get to, which is like, does six months matter? Is that enough to change the world or not?
Right now it doesn't seem huge, but if you are the first to trigger a recursion, yeah, that can matter a lot.
That could be the moment where it matters. And that goes back to the other question, which is on the recursion thing. The fact is people are trying it, as far as I understand it. Like it's the plan.
It is. It's the plan. Yeah, it's still highly AI-assisted and there's nobody could try and spin this up right now and have it be foreseeably going to lead to something substantial. But yeah, I mean, what I see right now is like you could do your hyperparameter search faster or something like that and you could call that recursion if you want, but that's not really what we're talking about here. We're talking about like a system finds a fundamentally new idea that year 24 would only come from humans.
And that leverage of not only coming up with the ideas, but testing them, validating them, working out the tweaks, implementing them, scaling, I think that's exactly what people like Sam and Dario are now hinting that they're going to be able to do soon. I'm not really buying those claims, but I think that's what they want to do now.
One hundred percent. Yep. Yeah.
And that is destabilizing and scary and dangerous.
Let's say potentially destabilizing. I mean, if it's just hype and they can't really do it, it's not destabilizing. But if one of them achieves it, if it were technically feasible, that would be destabilizing.
And I mean, I think we can agree it probably is technically achievable. It's just a question of whether you can do it with current technology or later, how far in the future. There's no, you know, surely an argument that this can't be done that you need.
Right. I mean, I think we all think that's going to happen. And so one advantage for transparency then is if there were very high visibility as to what's going on at the frontier, if they're basically triggering this sort of process and the public is kept reasonably informed as it's happening, I think the world would be very much freaking out. Yeah. So I think that possibly sunlight might be a way of providing substantial pressure to prevent that.
Freaking out might be too strong, but it's worth mentioning that like polls of the American people, not necessarily in Asia, but American people are pretty worried, maybe not at the top of their set of worries, which might be economic and so forth. But you know, you look at these polls and like seventy-five percent of the American public is already worried. And I expect that that worry will increase, not decrease. At least I don't see any active thing that is going to decrease it. And the thing that might decrease it is propaganda from the tech companies. I think the tech companies seem to like scaring people as far as I can tell.
Oh, I disagree. I think that, I mean, there's probably some of that there to some extent for sure, but at least my experience at OpenAI was that there was more pressure to like not talk about the risks in public and to like, you know, sort of downplay that sort of thing than there was no pressure to play it up, at least not as far as I could tell. And you know, if you look at the public messaging by Sam over the years, he certainly started to like talk about it less and less and less as well.
When he, I mean, partly because I pushed him, but when we talked at the Senate, he didn't answer the question, what were we most worried about? I mean, I made Blumenthal come back, but Blumenthal says it's jobs. And Sam gives his explanation why he's not that worried about jobs. And I say to Blumenthal, you should really ask him a question. And Sam there said, my biggest worry is that we do, I can't remember the exact words, but substantial harm to humanity. And when he was back at the Senate a few weeks ago, his biggest harm was like in so many words, like we don't make all the money and extract all the value that we could or something like that, right? So he, I mean, I think it's true that he used to talk more about this kind of risk stuff and talks less now. So I mean, that's an interesting data point is around the time that I did my Senate appearance, which was May 2023. So reconstructing timelines a little bit after that was a second letter related to the pause letter, which I did not sign. That was the one about being, and maybe you wrote, I don't know, the extinction letter.
You made that's what we signed. So I didn't sign that one, and we should soon get to extinction risk, but we'll come back to it in a minute. Sam did, I think, sign that, right? So at that time, that was maybe June 23 or July 23 or something like that. It was after the Senate appearance. I knew that. At that time, it was still popular among CEOs to express concern about this, and maybe it's true that they do a little bit less. Dario still alludes to this.
So a lot of people that I have spoken to think that it's a deliberate mechanism to try to hype the product, to talk about the risk, like, look how great our stuff is, it might kill us.
Yeah, so I have lots of scientists sign that. I didn't, but yeah, I understand.
Yeah, so first of all, yeah, loads of people have these views who aren't conflicted and trying to hype up the companies, you know.
Yeah, agreed, agreed about that too.
But then to your point about the hype, you know, my take on what's been happening is that DeepMind, OpenAI, and Anthropic have been full of people who were thinking about superintelligence from the very beginning at the highest levels of leadership, and therefore they have been considering at least somewhat both the loss of control risk and the concentration of power risk, you know, who gets the control of the AIs from the beginning. And this is documented in all sorts of ways. You can look at their old writings and so forth, some of the leaked emails. And then you might ask, well, why are they building it if they've been taking these, if they've been thinking about these risks?
Can they see loss of control?
Loss of control is what if we don't solve the alignment problem in time and the AI take over. And then concentration of power is if we do solve the alignment problem, like who controls the AIs, what goals do we put in them? You know, do we risk becoming a dictatorship or some sort of crazy oligarchy or whatever? So like you can find writings from people at these companies, both senior researchers and also the CEOs and so forth, going back decades talking about these things. And then you might ask, well, why are they doing this? Why are they racing to build these things if they were... I mean, Dario seems like the most extreme version of that question. Like he seems still very much publicly saying that there's very serious risk, and he seems very much publicly pushing the models forward. Like I understand, though I didn't see it myself, that at some point he was saying we won't build frontier models because of these risks, and now he's obviously building frontier models. And so the key here is, I guess I would say in a single word, it would be rationalization. So there's been this very seductive argument that has appealed to all of these people, which is basically, well, it's probably going to happen anyway. If we don't do it, someone else will, and it's better for us to do it first than for someone else to do it because we're the responsible good guys who will wisely solve all the safety issues and then also beneficently, you know, give UBI or whatever to make sure that everything works out well. So basically all of these people sort of trust themselves more than they trust everyone else and have therefore convinced themselves that even though these risks are real, the best way to deal with them is for them to go as fast as possible and win the race. And this is, you know, DeepMind's plan, so to speak. Demis Hassabis's plan was basically be there, get there first with this big corporation, Google. And then because you have such a lead over everybody else, you can sort of slow down and get all the safety stuff right and sort of make sure that everything goes well before everybody else catches up. That seems naive. That plan was torpedoed when, you know, Elon and Sam and Ilya made OpenAI. Why did they make OpenAI? Well, they were worried that they didn't trust Demis to handle all that power responsibly when he was in charge of the AI project. You know, when all the fate of the world rested in his hands. So they wanted to create OpenAI to be this countervailing force that could do it right and make it, you know, distributed to everybody and not concentrate power so much in Demis' hands. And in fact, in the leaked emails or the emails that came up in the lawsuit, they were talking about how they were worried that Demis would become dictator using AGI. And well, we can see how well that's worked out. You know, all the Anthropic people basically split off from OpenAI because they didn't think OpenAI was going to handle the safety stuff responsibly. So then they're claiming that like they have the technical talent and they'll be able to like sort out the alignment issues better than everyone else, you know. It's a mess.
You have anything to add?
I agree that it's a mess.
Not that it's a mess. And I mean, I guess I don't know if you may not want to answer this on camera, but do you feel confident that we should trust any of these particular...
We definitely should not. I think pushing for things like transparency, as well as the government having people whose job it is to be keeping track of these, coming up with contingency plans, interviewing these labs for their plans and coming up with internal assessments, their probability of success, all of that seems useful. And I think at least some of the players here would probably be willing to push for that type of stuff, and I think others wouldn't. So I think it's, are they willing to help solve these collective action problems or are they going to continue defecting? And it's been a lot of defection.
I realized I never actually answered why this ties into your question. So early on when these companies were fresh and young and idealistic, their founding mythology was basically, yes, the risks are real, and that's why you should come work at our company because we're the good guys. And so when that was still fresh and still like, you know, the main thing they were saying, they were talking about it a lot. But then now when their sort of like founding myth is sort of like kind of laughable and it's like very much not something they can say with a straight face, it's eroded some, not something they can really say with a straight face so much anymore. And also they're under lots of political pressure, you know, to get investors and to ward off regulation and stuff like that. So the rationalization wheels are continuing to turn and they're coming up with new narratives to justify what they're doing, you know.
And there's also a lot more players at the table, both within the US industry, which is most of what we were talking about, but also in China. Then we have the whole, I don't know if we have time to go into it, but maybe we will, maybe we won't, the mechanisms of open sourcing or at least open weight models, which means that, you know, essentially anybody can get in this game to some degree.
I disagree with that to some...
I mean, to some degree, sure, but...
No, walk me through the disagreement.
Well, like hypothetically suppose that we achieved AGI, someone achieved AGI and immediately opened the weights for everybody. It's not going to happen like that. But suppose it did happen like that. Then for a brief glorious moment, anybody with enough GPUs could be able to run their own AGI right at the frontier of capability. However, AGI isn't like, you know, there's going to be AGI plus and AGI++ and so forth. And whoever's going to get to AGI plus is going to be the one who had the most GPUs so they can run the AGIs to do the research fastest, you know. So even if you gave everybody exactly the same starting point at the same level of capability, the people who had more GPUs would pull ahead slowly but surely over everybody else, right? So I think that there's this unfortunate... around that, but go ahead. So there's this unfortunate sort of inherent, I don't know if I want to say winner-take-all effect, but there's like this return to scale sort of thing inherent in the dynamics of an intelligence explosion. And that's part of what makes this so scary. All the actors are incented to push it as fast as possible and to have the maximal resources in order to do so. And they're also incentivized not to open it up and, you know, to not be transparent about it and so forth, unfortunately.
Yeah, I think we have a little disagreement there about transparency and how much we might expect. I think you're a little bit more optimistic, and I'm, Daniel, being a little bit less optimistic about transparency. I'm especially less optimistic about transparency as we get closer to having actual AGI or let's say differentiated AI. So right now I would say that what's there is not very differentiated. Everybody's kind of using LLMs with reasoning models and so forth. There's some differentiation in how people do their RL and what their data sets are, but I think there's not so much value in transparency until somebody has something that really is unique that they think other people aren't going to just reconstruct very rapidly. And as one gets to that point of having some unique piece of intellectual property, which I think will happen, there's even more reason to be less transparent. So I think I'd be optimistic about being transparent about the numbers of the best models internally, not necessarily the methods that were used to create them or the weights for them, but at least the public knowing what's going on, what's the peak capabilities that the models are exhibiting, and then being aware of that.
Are you being optimistic that they'll do this by default or that this would be good?
I think it's good and that there's more tractability for this than a lot of other asks.
I agree. More tractability, like people might agree to do it.
Yeah, yeah, yeah, I agree with that. That's why I've been making these asks.
Yeah, but I'm just saying that they're not going to do it by default. Like if nobody asks them to be transparent about these things.
No, I agree. So I mean, we can agree that like voluntary self-regulation is probably not enough to get to that.
Yeah, yeah, yeah. I wouldn't bank on it. Yeah. All right, let's talk a little bit about time and forecasting and scenarios. We've talked a bunch about strategy, what we should do going forward, what policy should be. Some of that depends on timelines. So if we thought that we had a thousand years before AGI, then we might make different choices than if we thought we had six months. If we thought that LLMs were the answer to AGI, we might make one set of choices. If we thought they were definitely not the answer to AGI, we might make a different set of choices, maybe focusing more on research. So let's talk about our forecasts around that, and also maybe I'll throw in there our forecasts around when we might sort out the safety problems, the alignment problems. I will make the argument that we've made some progress towards AGI and very little towards alignment, and we can see if we agree on that. So I want to start by using a notion that many in the audience will know, but not all, which is a distribution of probability mass, which is to say that you could make a simple prediction. You could say, I think AGI will be here in 2027 or 2039 or whatever. But I think we all understand that that's the unsophisticated thing to do. You can certainly say your best guess for which particular year you think it might come. But as people who are either scientists or know something about science, we know there's what we call confidence interval around that. So, you know, it might be this plus or minus that. The most sophisticated thing to do is to actually draw out a curve and say, you know, I think some of the probability mass will come, you know, before 2027 and some of it will come before 2037 and some of it will come after. In your, I guess it's sort of an appendix to your AI 2027, you go through this in a fair amount of detail. I'm not sure how many people got to the appendix. I think it was maybe a little hidden to find it, but it was there and it did a good job of that. And it gave the forecast for four different people in a sort of qualitative way. I don't know if it showed the full curves, but it said like, this is the chance that they think it will come before 2027. And I think for three of the four forecasters or something, you remember better than I, you know, some of the probability mass was like after 2040 or something like that. So maybe I'll start with you because I think you have maybe put the most work into trying to get detailed probability mass distributions, and maybe you can talk about what your own are and different techniques people have used and where you are on that.
Thank you for that excellent launch to this. Feel free to stop me if I ramble too long because it's a huge topic, lots to talk about. Okay, so first the exciting part, the actual numbers. When we were writing AI 2027, we had our different medians or 50% marks for AGI, or I think we were, we divide up, we didn't use the word AGI, we used different milestones. So superhuman coder, full automation of AI research, superintelligence. But for those things, let's say for superintelligence, I was thinking 50% chance by the end of 2027. And that's why AI 2027 depicts it happening at the end of 2027 is because I was sort of illustrating my median projection. Now, the other people at AI Futures Project tended to be somewhat more optimistic than me. They tended to think it would take longer to get to superintelligence.
Let's clarify, optimistic is that flips depending on how you think.
Sure. So the other, they thought it would take longer, like 2029, 2031, something like that.
Let's say more conservative, a few more years.
Yeah, that's right. And, but, you know, I was the boss, so we went with my timelines. But happily, by the time we actually published it, I had lengthened my timeline somewhat. So these days, I would say 50% by end of 2028. But, yeah, doesn't give me that much comfort.
I mean, I think you're wrong, but if we have an extra 12 months, like, I'm not sure that's enough to handle all the... It's still quite scary to prepare.
But in terms of what the shape of our distributions looks like, they tend to have a sort of hump in the next five years and then like a long tail. And the reason for that is because, well, the pace of AI progress has been quite fast over the last 15 years. And we understand something about the reasons for why it's been so fast. And basically the reason is scale. So they've been scaling up compute, they've been scaling up data, they've been drawing in ever more researchers into the field, especially compute is probably the main, the most important input that they're scaling up. And that's sort of turbocharged progress, but they simply won't, the companies simply won't be able to scale things up at the same pace after a couple years.
Is that a function of power or is it a function like electrical power? What is the function? What's the rate limiting step by which you think that kind of scaling won't continue?
It won't be like a sharp cutoff, but it'll be a couple things. So partly it'll be, you know, power supplies. Partly it'll just be compute production. Like much of the world's, even after building new fabs and even after converting much of the world's chip production into AI chips, they won't, they'll have to like produce 10 times more fabs in order to scale up by 10 times, right? Whereas previously they could just take chips designed for gaming and you repurpose them for AI. So in a bunch of little ways that are going to add up, there's going to be all these frictions that's going to start to bite that will make it harder for them to continue the crazy exponential rate of scale.
I think we're already seeing that with data. I think I don't even know the actual numbers, but let's say that GPT-2 used maybe 10% of the internet or something like that, or 5% or something like that. Maybe you guys know the actual numbers. And GPT-3, you know, used a significantly larger fraction. GPT-4 used like most of the internet, including transcriptions of videos and stuff like that. And so you can't just keep 100x-ing that because there just isn't enough data. There's new data generated every day, you know, so you can always eke out a little more. And people are turning to augmented data as well. They should, but that's not a kind of universal solvent. And it works better for things like math where you can verify that the augmented data are better. And so I think we're already running against that kind of like bottleneck on one of the, let's say, raw resources that go in at least into the current approaches.
And another thing I would add is that you can also just think about money, which is, which you can use to buy many of these things. And you can say, well, they've been scaling up the amount of money that they're spending on AI research and on training runs in particular over the last decades, but it'll be hard for them to continue scaling at the same pace. You know, they're probably already doing something like billion-dollar training runs, but, you know, if it goes at the same pace... What was the biggest training run in 2020? It was like what, like $3 million, something like that, you know, $4, $5 million. So they've gone up by like two and a half orders of magnitude in five years. If it's another two and a half orders of magnitude, we're doing a $500 billion training run in 2030. Like there starts to be not enough.
Yeah, there's just not enough money in the world, you know, like the tech companies just won't be able to afford it. Even if they've grown bigger than they are today, the economy just won't be able to afford it.
And so that's why we predict that, like, if you don't get to some sort of radical transformation, if you don't get to some sort of crazy AI-powered automation of the economy by the end of this decade, then there's going to be a bit of an AI winter. There's going to be at the very least a sort of tapering off of the pace of progress. And then that sort of stretches out a lot of probability mass into the future because that's a very different world. It's like, you know, could take forever. Well, not forever, it could take quite a long time to get to AGI once you're in that regime.
I'm going to ask you one or two more questions. I'm going to insert mine and then I'm going to come to Dan.
Although if you want to, I mean, one note on data was, yeah, we sort of ran into that bottleneck, I think maybe two or so years ago. And the main things that have been continuing the pace would be this, you know, thinking mode type of stuff outside of the trends that were existing previously. So I think it's sort of picking up slack in some ways for the fact that most of the internet has been trained on.
The methodology by which you came up with these curves, can you just tell us a little bit about them?
