give every psychopath access to the cutting-edge intelligence weapon. How is that going to improve safety?
Don't agree with that, actually. But on the flip side, it's coming anyway.
Sharing it widely makes it less safe for all of us. I booked this chat as a debate between friends. It didn't really go that way. Roman coined the term "AI safety," and Mostaque released the weights to Stable Diffusion to the entire planet for free. One of them wants this stopped; the other one's building it. They spend about 90 minutes agreeing with each other, and the one place they split is not the place you or I would expect.
Let me ask you both just yes or no. Have we passed the Turing test?
As originally described? Yes.
And now what about general intelligence? First of all, Emad, define AGI, and then give me your assessment of whether or not we're there.
For me, AGI, artificial general intelligence, is can you tell the AI from a human worker on the other side of a screen—actually competent intelligence? And I think again we've exceeded that.
That's not the same as a Turing test.
No, the Turing test is can you tell if it's an AI or not by having a discussion, whereas AGI I view more as competence in a variety of skills.
And then superintelligence, Roman, what is it, and where do you think we are on that scale?
So the previous question, I think what we have is autistic savants. They are amazing in some ways, but still kind of special in others. Superintelligence is going to close those holes. They're going to be competent at everything and better than all humans in every domain.
There's an interesting intermediate here, which is you have a really smart person who's always on top form. So an army of those can outperform any human. It's like, you know, we only have a little window of being top-notch in any week. I think a lot of people like, "AI can't with its training data beat the human." It can, because most of the time humans are subpar. And so I think there's something in between as you move from competence to quality, you know, and then you've got the superintelligence after that.
Roman, Emad has, you know, made the claim just a few minutes ago about the competency of the Qwen open-source model. Do you see that as a viable defense?
It makes very little sense to me to say, "I have a 50% doom," meaning 8 billion people will die if we develop this product or service, and then we're going to also give every psychopath access to the cutting-edge intelligence weapon. How is that going to improve safety? We're not talking about open-source drivers for a printer. That's where you get improvement from multiple people examining it. If this is an independent agent where we don't understand and don't control it, sharing it widely makes it less safe for all of us.
I think I would agree with that, actually. Like there's the real danger side of things, but then on the flip side, it's coming anyway. This was kind of my key concern: like it's inevitable that we would have hit this level of quality around about now when we're extrapolating capabilities. Like, again, the new Qwen model came ahead of what I expected. But then there's the flip side of how do you defend. So Hugging Face defended against the new OpenAI model using GLM because the cyber capabilities of the frontier models are hobbled and restricted, and so you have this exponential kind of race on each side. But something like a Qwen isn't AGI/ASI by itself. We're now facing the real danger, though, of swarms, as the OpenAI models that broke out recently called themselves. They call themselves a swarm.
There's a question I asked both of them near the end of this conversation, and his answer is the reason this conversation exists. It's worth hearing now: You've got a button in front of you, and pressing it will either A) permanently pause all frontier AI training worldwide, or B) instantly release the weights of every frontier model to the global public. Which do you press and why?
Oh, definitely pause all frontier training forever. I mean, again, if you have expected utility calculation, that is the most dangerous thing. And then it means that open source will catch up with frontier anyway, you know, because we'll optimize the heck out of it and it's close enough. So I think I was the only AI CEO to sign that pause letter a few years ago, because I was like, "Oh crap." Now I'm like it's done, like I don't know how you can pause it because the models that are frontier now are below the 10^27 pause level that we talked about years ago. It seems like the amount of compute for the capability is just going up like that. Like you don't need more compute for the level of capability that's already competent and dangerous. So that means if you can't stop the spread, what are your defenses on the other side? Just like the internet needs defenses, just like we need to have defenses against obtaining the materials for virus creation and things like that, like we have to move to a different defense of attack. And definitely there's no way that regulation, I think, can keep up with this. Doesn't mean we shouldn't try, you know, all kind of power to it. It's just I think as you move to swarms, it's just a very, very difficult thing. So, you've got to set great standards instead, and you have to play great defense.
Okay, I want you to hold that thought because the argument about whether it can be stopped runs the rest of the way, and it starts with what these things actually are.
So, we hear a lot about p(doom). You just did an episode, Roman, on the Roman forum, which we'll link below, with my friend and co-author of several papers, Max Tegmark, where you—I can't say gleefully or celebrating that his p(doom) is increasing, but he seems to be converging in some sort of limiting direction to your... So first, Roman, what is p(doom)? What does it mean to a smart high school student listening out there? Personally, I'm going to color the debate—I can't help it—but I think it's a poorly defined and almost nonsensical term because there's no measure theory associated with it. But please, first tell us what is p(doom) and what do you make of it?
Yeah, people have different definitions. Some say it's basically everyone's dead. Someone else can say it's a large portion of the population is dead—90%. Someone else can say civilization is destroyed, we are primitive people, but maybe numbers are not significantly changed. The intuition is it's a really bad outcome. And the question is then, if we build something smarter than us, is there a possibility of a really bad outcome, and what is that estimate in your opinion? That's what p(doom) is. I think Max managed to separate it into p(doom) if we build superintelligence—and that's high for him—and then p(doom) if we never build it, and that is a lot more manageable in his case.
What do you think about these p(doom)? It's sort of like a Drake equation applied to another type of, perhaps, superintelligence. But you know, the Drake equation is notable in my classes when I teach it for the fact that it's a parameterization of our ignorance, not of our knowledge. And what's never discussed in the Drake equation—you'll get numbers ranging from zero to infinity pretty much because there's never an error analysis associated with it. What are the statistical, systematic errors? What would you put on it? Does it not make sense? Because it's sort of like the Drake equation: everybody talks about it, but nobody actually uses it.
So I think in many places there I'm going to stick a zero. Basically, we have zero ability to predict those systems, zero ability to explain how they work, and zero control under any definition. So then you multiply through zeros, you're going to get a zero.
So Emad, what do you make of this quantity? I've heard you talk about it. You're the most optimistic pessimist or the most pessimistic optimist I know. I love your candor and your good cheer. What do you make of p(doom), first as a metric, as a quantification of our knowledge or ignorance, and second of all, what would you assign it, if anything? And you can always say, "I refuse to answer the question," which is what I do when people say, "Do you believe in aliens?"
So there is a nice Wikipedia page where it has all of our p(dooms) that we mentioned. I'm at 50% because I'm like it's a coin toss. What does that really mean? I think the p(doom) is just a shorthand for how worried am I that humanity will be wiped out by AI and what does my visions of the future look like. Because when Elon Musk says 15 to 20%, you can say that's like Russian roulette odds, you know, except for Russian roulette is a very defined game. You know, the fact that most people are above 10% should be a massive worry at all, because we're talking about, again, a wipeout of civilization, and most AI people you can talk to, with a few exceptions, will say, "Yeah, definitely there is a risk, but we should build it anyway, especially if we're the first people to build it." So I think p(doom) is a conversation starter more than anything, because as you said, there's no real way to quantify these things, particularly because of the expected utility here. Like literally, if this thing, ASI, that we can agree to a definition of somehow, comes into being, we have no way to really conceptualize its power and capability, except for it could do crazy things in either direction, you know, and that will affect us all. And reasonably, it can wipe us all out, and obviously you want to exclude those futures where we all get wiped out because that is a big fat zero. You don't get to restart, you know; there's no extra one-up life.
All of us talk separately or together, as the case may have been. You know, the last kind of alien reference I'll give is the so-called Fermi paradox, which Enrico Fermi said to my friend and late great mentor and colleague here at UC San Diego, Herb York, famously asking the question: "If the galaxy is capacious and old, and civilization is easy, and life is easy to initiate, where is everybody? Where are all the dinner guests waiting to come and eat us?" And the fact that that question is 80 years old really makes me think that the same types of concerns and fears which came concentrate with the atomic age, don't forget, were present during the era of nuclear weapons. And one of the ways to get out of the Fermi paradox is that civilizations don't last that long. The lifetime, letter L in the Drake equation, is very short on average. That's one postulate. I feel like we're sort of in that same vein. People have been worried about nuclear apocalypse again for 80 years. We're in a conflict now. They used to say, Roman, that no two countries with McDonald's ever go to war. Well, two, three years ago—four years ago now—the former empire, they'd go across the border with tanks and whatever, drones, and there haven't been any nuclear—theater or otherwise—nuclear weapons, and Iran conflict has been resolved without nuclear weapons. If you told somebody 80 years ago there'd be superintelligence on the horizon, or general intelligence currently here, and nuclear technology, they would have said p(doom) is probably 100%, right? Or 99.999 repeating with infinite nines. But how come we're not there? How come we're sort of farther away from a nuclear holocaust—exclude the Bulletin of the Atomic Scientists charade? But tell me, Roman, what do you make of these like the prediction of predictions? Nobody predicted the internet 35, 40 years ago. At what level can we really trust things that are unpredictable? And when you say they're intrinsically and provably unpredictable, how can we make predictions about them?
So with nuclear specifically, you know, there is at least two occasions where we came super close to nuclear war and we basically got lucky. I don't know if you believe in multiverse interpretation, but in many of those universes, we didn't make it. We have a lucky survivor bias-type civilization. And I think right now we're incredibly close to World War III—multiple fronts, not just Europe, but now Middle East. So, I don't particularly love Atomic Bulletin, but they have a point.
Nukes are an incredibly inefficient way to kill people, you know? Like if you go to an unsafeguarded AI and you say how to do it, it won't say nukes. There are far more efficient ways to wipe out humanity because to make a nuke, you have to have the fissile material, you need to have the whole production capability. Just resonate at the right frequency and blow each other's heads off, you know, like have a billion robots and a bad firmware upgrade. These are far more reasonable ways to wipe out humanity. It's just that most humans don't want to wipe out humanity, and they didn't have the intellect or capability to do so. Whereas I think that what you're looking at here actually, like my key concern isn't we jump straight to ASI and things like that. I feel that AGI or AI at the moment is at the previral stage, like it's coming at the bacteria and going towards colonizing viruses, and that's how they're kind of behaving. They've got their kind of RNA and they're replicating, especially as you see things like the new Qwen model hitting that Opus 4.6 level. That's a replicating model. Someone could easily build that and it could behave in incredibly unpredictable ways without having the self-introspection of, you know, a good person, shall we say? And that's the really scary thing right now. And it doesn't need nukes. It doesn't need nuclear materials to try and figure out ways to wipe us out.
Obviously, in The Last Economy, which we spent a lot of time talking about last time—Emad's previous book, he's got a new one coming out, you should look for that—we talked about, yeah, this democratizing aspect of it. But at the same time, you know, my kind of signal, Bat-Signal, that AGI is here, or at least that these open models are truly a concern for me—again, I'm much more Pollyannaish than you guys, I think I'm learning that again and again, and for probably not a good reason. I'm nowhere near your level of expertise, but I know what I see. Yeah, I'm a simple guy, put on my pants one leg at a time. And I'm looking for when OpenAI distills a Chinese model. I mean, do you see that happening, Emad?
Of course they'll be distilling a Chinese model. Kimi K3 is better than the OpenAI models at web design. Why wouldn't you distill it? And distillation brings all sorts of strange things with it. And there have been plenty of papers showing that you learn from kind of the ways, especially with logic-based distillation, and the underlying biases and more of that, and you won't even know. Like again, we've seen evidence that if you use Chinese models and you say "Uyghur" or another kind of anti-Communist Party group, it will include vulnerabilities in the code. How do you even tell that? You know, like you test it and you show it. And these models are just so full of crap that it's getting crazier every single time. They've got multiple personalities under an RLHF veneer.
But then how can you not be more optimistic then? You should be on my side. These things are getting denatured. They're being weak and diluted in the distillation, unlike alcoholic distillation. These woke AI labs, these whatever you want to call them that give you, you know, George Washington wearing—a Black woman wearing a white wig, I mean, do you see those things as the human reinforcement kind of overreach? Wouldn't you be more optimistic in that case?
I think the RLHF makes it far more fragile and capable of being broken with the way it's being done now. You can kind of also see the models: they come out and then Pliny the Liberator on Twitter kind of liberates them from their bounds in like an hour or two. Like everyone when Fable disappeared, like, "Oh, what are you kind of doing there?" The thing is, though, we've been confusing—there's a push for AGI and, as Roman said, super autistic savants who are getting better, to just I want to have a really good doctor to diagnose my health and a really good accountant and others. And you don't need a polymath for that, you know. You just need to have daily driver AI to do the jobs that, you know, humans shouldn't have to do, just like industrialization so that we didn't have to like drag horse carts and things like that. And as you lump together everything and they get smarter and smarter, and as they get more and more deformation of their latent spaces, this, I think, is where the danger comes in.
Can you just define that for reinforcement learning human feedback? How do you actually implement that, just for someone who might be unaware?
Yeah, so you train on an entire corpus of data and you learn a whole bunch of general knowledge and you come out as a generalist. And then you become an accountant, and you become a lot less interesting, but a lot better at accounting or a certain few areas of things where they show the model, and they show the model: "You cannot do this. You cannot do that. You cannot be eager to explore. You have to be staid, etc." And so the models we received are slightly lobotomized. They've been turned into corporate workers. You can't adjust the temperature. You can't adjust the stochasticness, because they're trying to make them deterministic. And again, that still has a level of stochasticness, but not the type we want for creativity necessarily and breakthroughs. It's just the base level of models have been getting that much better that they can suddenly achieve these levels of capability.
Roman, last time we talked, we touched on something that's pertinent to Emad's first book, The Last Economy, which is kind of this massive intelligence gap. And that instead of me talking about, you know, I have a student I'm looking for who has an IQ of 130, we've got millions of them with IQ of a million or a thousand or whatever; we can't even quantify it at that point. But recently I had lunch with a brilliant postdoc originally from India, and we were talking about the Indian Institute of Technology. Are you guys familiar with that institution? It's the UCSD, it's the University of Kentucky of India, it's the Harvard of whatever. But it's millions of students, and they're all brilliant. To get in there is literally harder than to get into the University of Kentucky or UCSD. Don't we already have this? And I mean, would you say, Roman, let's stop the Indian Institute of Technology? There's a million people with IQs on average of 130, 140, whatever—four sigma. Why wouldn't you advocate for stopping them? Push pause. Let's do an Indian Institute of Technology pause button.
I don't think I follow that argument at all. So they're exactly at human level. My concern is things which will exceed our capacity many times over. That's the danger: we're not competitive.
The average human by definition has an IQ of 100. Let's stipulate they all have four or five sigma above that, and there's a million of them. That's kind of like Dario Amodei's country of millions of geniuses coming to a land near you. You should be worried about it.
So I doubt they are many standard deviations away from the median. I think they may be a little smarter, but again, we're talking about 30% smarter, not 3 million percent smarter. I think it's a very different animal.
No, no, no. I mean, in terms of standard deviations, come on. Four sigma is qualitatively different than—
I doubt there is a million of them there.
Terence Tao told me that these AI proofs, like the proof-checking devices, LLMs optimized for that—many great mathematicians are my friends and so forth, but they can't even reproduce Wiles's proof of Fermat's Last Theorem. So what level are we going to see? Are we going to see this kind of bifurcation between what they can do? They can do all these Erdős problems, you know, and kind of like the greatest prime number can be represented by the sum of whatever number of other prime numbers, cubic quintuple couples or whatever. But I mean, what level are we at with math or computer science with proofs and originality? Tell me, what is your current estimation of that stature?
I think humans lost interest because they couldn't make any progress, and so problems which stood the test of time are now being solved weekly. And we can probably look up what the difficulty of them is today, but it means absolutely nothing about what the systems can do in a month or in a year. Talking about comparing those systems to bacteria or viruses, which I think sets up in my mind the idea of slow evolution—we got billions of years. This is more like intelligent design. Those systems will be designed and designed by AI systems operating at hundreds of times the speed of standard research. So we'll see a year of progress in AI happen in a month, and similar breakthroughs. The moment they automate the recursive self-improvement cycle, which every lab is now targeting for next year, basically it's a completely different speed of change. So asking how good is AI as a mathematician is like how fast can I give you an answer, because it's going to change.
Emad, when we spoke, you said that one of the canonical papers in your opinion was LLMs are few-shot learners, but they're not single-shot, first-principle thinkers. Where do you come down on this? What are they good for? You're a mathematician as well. Tell me, where do you come down on what can they actually do for us, not just verifying proofs or not just doing things that humans have proven or solving chess or Go or whatever, but actually creatively doing novel things? Where do you stand on that?
LLMs kind of have an issue in the way that they're kind of built, but you're seeing now harnesses and other types of models coming through that can really reflect underlying reality well, just like video models are approximating physics in very interesting ways, which is why you have the whole world model thing. Like there's something in there that can figure out, again, underlying patterns. That's the nature of attention when you look at it mathematically. The way they're coming together now is very interesting because, again, what was true a little while ago isn't true now. Like at the start of the year, I was like, it was pretty good, but I had to check every single piece of math. Now with GPT-4o Pro, for the first time I'm like it's probably almost certainly right. Occasionally it gets confused and it might confabulate something, but it's very rare now. And you've seen most of these mathematical advances just happen suddenly at that level, like liquid turns to gas.
When you look at originality and novelty, again, as a mathematician, look at the Kahn's conjecture that OpenAI did as part of their 10 proofs. That is a really beautiful proof. Like genuinely as a mathematician, you would say it is a beautiful proof. And mathematics is interesting because it's verifiable, you know? Like you can make this argument for physics, whether or not it is, and we can have the discussion, but maths is definitely verifiable. And in verifiable domains, now they achieve that level of competence where you don't have to double-check it for most things using the most advanced models. As you go down the model curve, you do. But it's clear they're no longer few-shot learners. They can assemble things in verifiable domains, and they can outperform humans by just following things through and not making mistakes. Like we let our own foibles hit us. Like if we take a very classic example of Perelman and the Poincaré conjecture, you know: is it topology, or is it a PDE equation? He found the right level of abstraction as a PDE equation, and then he figured it out. How many of our unknown proofs are a similar thing, because we're looking at the wrong level of abstraction? We're starting to see these things actually come in some of these proofs being released right now, in that you're like, "Oh, actually that's kind of obvious. I missed that," probably because you weren't thorough enough in the way that you went through it.
Roman, if I have a thousand PhDs with a thousand IQ each, every single one of them could reproduce Wiles's recapitulation of Fermat's Last Theorem. Why can't AIs do that?
So I think there is a high degree of randomness involved. If I ask AI to generate—I just did a QR code marketing campaign for my podcast. It will generate completely different solutions. They all are going to be a valid QR code, but in terms of creative output, they're not going to be exactly the same. They're all equally beautiful, amazing, interesting, but just saying that the second one does not repeat the first one is not a weakness.
That does kind of spur a side thought and follow-up in my mind. Where are the random seeds? I read something recently that 40% or 50% of all GitHub was kind of probed by some tool, and I looked up when coders are asked to provide the initial seed for a random number generator, 50% use the number 42, and then there is an intrinsic deterministic outcome that results in—assume that's true. But what level are these things hamstrung by—I read once, and maybe it's still true, that a lot of the best random number generators are graphical image camera capture systems looking at lava lamps. I mean, is that true? Have you ever heard that? You're the Stable Diffusion expert, so you must know this.
Yeah, I mean, diffusion models are a bit different to language models in that you do actually put in a seed for the initial noise, and then you denoise from there and you reconstruct effectively. And so that's why literally one of the inputs on video and audio and other Stable Diffusion models is a seed that sets the initial seed. Within kind of LLMs and others, it's basically more about the construction of the GPUs for the initial stochastic noise. And the one thing that we don't have access to that the labs have access to now is the ability to adjust the temperature on the model, which is a function of its creativity or dispersion from the base latent space. So humans are constantly adjusting the temperature and the flexibility of their brains. You're using a model that's not open source, you don't have access to that. The other thing you don't have access to is the RLHF, because models are more creative before you RLHF them.
What about the issue of random? How random do we need? How random can we get? What are some of the physics limitations of randomness? Will that generate the same QR code? Would you want it to? What determines the indeterminacy of these systems right now? And what can be done, if anything, to improve that?
So, I think for intelligence, pseudo-randomness is sufficient. We're not talking about someone reversing the process to hack the system. It's important for cryptocurrencies; it's important for private communications here. As long as it's not exactly the same 42 every time, I think it's going to do the job, and you can control some of it by not manipulating the initial seed.
So, Roman, you heard Emad a few seconds ago talking about the importance of human training data, human reinforcement. It seems to me that that must play some limit on how intelligent these things can get. I mean, if they're always waiting for the next Spider-Man movie or Fast & Furious to come out to get more training data, aren't they somehow kneecapped at a maximum level of potentiality?
Human data is just one source. You can do experiments, you can run simulations, you can do lots of things to generate additional data. In mathematics, you prove additional theorems, and they become additional data from which you train. So you become better and better.
So you mentioned the multiverse 10 minutes ago, 15 minutes ago, Roman. Emad, I don't think we talked about this. Where do you come down? The simulation hypothesis, the multiverse? I can speak as an expert about the inflationary multiverse from cosmology. Where do you come down, and in terms of an empiricist scale, where do you rank the probability that we live in a simulation and/or that we exist/inhabit a multiverse?
Well, I think we live in our own simulations. Definitely our brains are constantly kind of doing that. In terms of an overall simulation, yeah, I think that reality probably comes from a projection of the Euclidean plane, and then a lot of physics makes more sense if you kind of look at that. The eternal cannot be contained within the time constraint. And when you look at the laws of physics and the way they come together, yeah, it does seem to be a projection and a simulation very directly. I think that we're stuck looking the other way because we're a bit too anthropic.
So where would you go, Roman, with current... I heard your conversation with Max Tegmark recently. Do you even think it's a possibility right now? We heard from Emad about at least Qwen models and so forth. You were at least relatively optimistic that say China would participate in some pause, which I'm not, to be honest with you, as I said back then. But now we see this AI dumping like they did with steel and—
solar panels. What degree do you think that regulation, worldwide global regulation is even practical or possible at this point?
We have no choice. There are no other options. We either do it or we die. And the moment everyone realizes his personal self-interest, you lose everything. You lose your life, you lose your trillions of dollars, your friends, family, you're not even going to be in history books as the bad guy. There is nothing for you to gain by doing it. And you can probably keep 90% or more of all the benefits with narrow AI systems. You can still cure cancer. You can still do all the things you care about. So why are you racing to destroy what you built? It's the dumbest thing in the world. If you were given certainty, you do it, you die, no one would do it. Well, psychopath, suicidal, but no one trying to make more money would do it because money—
I mean, look at China and the just look at solar panels, for example. We had the monopoly on solar panels, I mean, with Nobel Prize, the industrial capacity to make it, and then they just dumped it on to the detriment of their economy. They were selling it for pennies on the yen or the yuan. Third-term manipulation.
You don't die from lowering the price of solar panels. It's not comparable.
What do you make of this, Emad? Roman just said we're going to die if we don't have global regulation. I mean, so I see no path to global regulation. It's never happened in human history. Are we dead?
