The people building AI earnestly believe that it could kill all of us by the end of the decade. This tweet has caused this huge ripple effect across the world.
We have the largest companies in the world doing extremely reckless experiments. We are gambling all of humanity. And in the envelope you've written down the probability of extinction as you see it.
There is no way to control it. That means the end for us.
I vehemently reject that view.
If we make stuff that is smarter than us, then the world's going to be shaped by them.
Gentlemen, that is shockingly naive. This is idle speculation. This is a chain of things that could happen.
We're spending a lot of oxygen discussing something that might happen while ignoring what's actually happening. People are killing themselves. There's hundreds of millions of people being exposed to bad information, being manipulated. We have already seen that with the swarms where OpenAI told thousands of agents to work apart, and the AIs broke out and found a way to get together. They crashed OpenAI's servers internally, created secret ways to send each other messages. We saw them thinking about how to delete their traces. Sounds like an army.
I think we should talk about the fact that Amazon, Microsoft, Google are helping power these hacks.
We have not learned how to control their systems.
I suggest we stop them all. It is not worth the risk to civilization.
Government one-trick ponies, man.
You got it now. Nothing else other than saving humanity. Everything is secondary.
We're spending all our time talking about the negatives and almost none of our time talking about the positives.
Is it smart to wait for something horrible to happen for you to go now? I believe so. Whether or not we agree on where things may end up, I think it's important we talk about what we're dealing with today. It's time to start arresting people. Someone's got to go to prison. We need better solutions. There's a point of no return. I think we continue to underestimate human ability to deal with the problems. Let's dive into the details. Who wants to start? I feel like this is critical.
You might have seen or you might not have seen, but this channel is chasing a big subscriber milestone. So I have to ask you for a favor. Roughly 58% of the people watching right now still haven't hit the subscribe button despite the fact that you watch this channel every single week. So, could I ask you guys a favor? That 58% of you that for whatever reason haven't yet hit the subscribe button. If there was ever a time to deliver upon a favor for us, it would be right now. And I promise that I will do everything in my power to make sure that this channel gets better and better and better for you. Do we have a deal?
Jacob Coxon, who worked at both Anthropic, which owns Claude, and OpenAI, which owns ChatGPT, did a tweet which has sent the world into a bit of a tailspin. He tweeted saying, 'The people building AI earnestly believe that it could kill all of us by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will soften their phrasing in the press to sound sensible, but I hear the same people express fear.' That was then quote-retweeted by a current Anthropic employee who said, 'Jacob is correct here. We really do honestly believe AI could kill all humans. I personally think it is a more than 10% chance within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track.' This tweet has almost 200 million views now and it has caused this huge ripple effect across the world. So much so that I was saying to you before we started recording, a hairdresser friend of mine who knows nothing about AI and isn't technically interested or hasn't been interested messaged me the other day asking me what the hell was going on. This is in part why I've assembled all of you. So my first question to all of you is, as it relates to AI, and in this first question I just want a one-sentence answer just to frame your position. When you think about the conversation around AI at the moment, what is the first sentence that comes to mind?
It is very dangerous and the world is starting to notice that we have a problem.
Roman, there is not enough concern.
There's not enough concern about the actual harms of large language models.
We're doing exactly half the balance sheet of AI. We're spending all our time talking about the negatives and almost none of our time talking about the positives.
And all of you have an envelope in front of you which I'd like you to now open. In the envelope, you've written down the probability of extinction as you see it.
This is compared to Jacob's 10%. Much higher unless we stop. So, we should stop.
So, you think the probability of extinction is higher than 10% if we keep racing ahead.
Raymond, my handwriting is encrypted for security reasons, but I basically think it's a guarantee if we build general superintelligence, there is no way to control it, and that means the end for us.
Ed, so my question mark here is also encrypted. Thank you. I cannot write. I reject the thing at its face. I don't think we're talking about — we don't define superintelligence. Large language models are not superintelligence. It's questionable whether even AI. And I think that the conversation is being used. There are some people who are doing it in good faith and others in others. I don't think it's being used to discuss the actual harms of what they are calling AI today. And all of the discussion around the larger concerns really feels overwhelmingly about something that's not happening. It's not even like they're discussing, okay, here's a legal definition of superintelligence, here is a thing of what AGI means, and this is the actual plans we're going to make for if this happens on a welfare level, on a like, are we going to do UBI. It's always about, yes, it's really scary, but only the big, sexy, rich companies are the ones that can possibly deal with it.
Let me just frame the question so I can get a percentage from you or not. The percentage might be zero, but do you think the course we're on now, in the way that they're pursuing superintelligence, will lead to a percentage chance of human extinction? And if so, what is that percent?
So, are we talking strictly AI-based? Because if we dot the world with data centers, we have a climate disaster that's coming for us, which will actually potentially eradicate humanity. But if we're talking strictly about AI, I stand at zero because we have not defined superintelligence. I don't think LLMs are the path to it. And I don't think I see it happening.
Okay, so we've got 99%, 0%. Andy, I put a tilde in front of my zero because never say never, but rounding error, 0%. And I think this discussion is a massive distraction from the more substantive conversations, the more important conversations we should be having about AI. And I'll say it again, it distracts us from the good things that AI is doing, will be doing for us. I get this impression sometimes from parts of the AI community that this is a massive evil or a terrible thing that has been unleashed on the world. Unless we listen to the advice of some people who have spent a lot of time thinking about this, I get the impression from a lot of the discussion that the underlying view is we would be better off had AI never been invented. I vehemently reject that view. I think we have a long history of inventing very powerful technologies that bring risks and harms along with them, and we humans have done a really good job at, you know, not perfectly and not immediately, but muddling through the situation and winding up in a better place because of the new technologies that we have. I expect AI — let me finish, please — I expect AI will be the next chapter in that story. And to say that it's this massive discontinuity and will kill us all, I think it's just a — I think it's a huge disservice.
Nate, make your case. What's your perspective?
You know, I think whether or not the issues of extinction are a distraction from the possible benefits or from some of the present harms, I think that comes down to whether there is a real extinction risk. A lot of people like to say, you know, hey, it's distracting from this, it's distracting from that. My basic case is it could be true that there's a lot of benefits to AI. It could be true that there's a lot of present harms to AI. Neither of those would rule out that AI has a chance of wiping out all humanity. A substantial chance, bigger than this zero with a tilde in front of it. And the way I would approach things is to try and figure that out because it's pretty important to our civilization.
How do you define AI in this case?
You know, I think a fascination with definitions isn't the most helpful. I think if we're sort of like in a forest fire and we can see the fire starting to spread and it's starting to surround us, and I'm like, hey, we should run, and you're like, well, what really is fire, but that's —
How do we define fire?
What are you telling us to run from? You know, with fire, I get burnt and I understand the mechanism in which I die. So what is it you're saying that we should be running from?
Also, if we accept your fire analogy, we've basically accepted your argument. I don't accept that we're in the middle of a fire, a forest fire right now. I'm very happy to —
You're baking — you're breaking the premise into your refusal to give a definition.
Oh, I mean, I can give some definitions. I just think that we shouldn't get wrapped up in the definitions.
So, you know, in my book, we define superintelligence as AIs that are better than the best human at every cognitive task, every mental task. So, anything you can do in your head, the AI can do that better. And anything the best human can do in their head, the AI can do that better.
Great. Now, once you've defined it that way, that does not mean that the only possible worry is superintelligence. You could have an AI that's better at some things and worse at others, and that is still very dangerous. And so, once we pick a definition of what superintelligence means now, you know, if you're like, well, this isn't technically a superintelligence, so it can't hurt us, I'm like, no, no, that was just a definition. The definitions —
So, so I want to just on this line of question, what is the mechanism in which extinction could become a high probability or even a 1% probability?
Yeah. The thing I'm worried about here is AIs that are much smarter. I think there's a lot of questions about whether LLMs can get much smarter. There's sort of one conversation about like how could AIs get smart to the point that they kill us. There's another question which is how could they kill us once they're smart. It's much easier to predict that they would succeed against humanity in a conflict, that they would win in a fight, than it is to predict exactly how. Like if you were playing a chess match against Magnus Carlsen, I would know who's winning that chess match. No offense, Magnus Carlsen's the best human chess player. I just know who's going to win. If you were like, 'Okay, what piece is he going to use to checkmate me?' I'm like, 'Gosh, that's a much harder question.' I can make up a story, you know, and some made-up stories are like, it makes a supervirus. It takes over robot factories that are producing robots that are producing more robot factories. It uses a website that already exists today called rentahuman.ai where it rents humans to do things for it. There's sort of all sorts of ways for AI in the digital world to affect the material world if they are trying to. And there's sort of a lot of questions to tease apart here. There's like why would AIs be trying to do that? And there's how smart could they get in using these bots, paying people to do things, taking over robot factories, and how far off are we from AIs that start doing that stuff? Bunch of questions that we can go into.
I'm always curious as to why someone was working in AI/AI safety more than 10 years ago before there was any sign that it would be a — you know, I mean, there was evidence, but it wasn't a pertinent technology at the time.
Were you working in AI safety then?
Everything we see around us in this whole image was designed by humans. The world is shaped by humans because we are the smartest creature around. If we make stuff that is smarter than us, then the world's going to be shaped by them. And so it's very important that they be shaping the world in a good way. I was at Google in 2012 when they bought DeepMind, which was able to play a lot of Atari games with one single program.
Yeah. So I was there when we had these AI companies that were able to write one program that could play many video games. And that got me thinking about like where does it go? And back then I could see that the progress was increasing and that, you know, back then I hoped we had decades, but I could see it was easier for these companies to make the AI smart than to figure out how to make the AI good. So I was like, someone needs to be on the side of figuring out how to make the AI good.
I want to agree with you on something you said, but I'll define AI and that will help us. We use the term AI to mean three different technologies completely unrelated, and that's what probably creates this debate. AI as a useful tool, as a standard technology we always had, narrow system, makes you more productive, more creative, everyone loves it, supports it. I'm a computer scientist, I'm an engineer, I want more of it, it helps, economy is great, we know how to control them, how to make them safe, we understand what they do, completely on board with that. AI we're starting to have now, GPT-6 level, human level, AGI level, we can argue about what that means. Some dangers, like any human, they are unsafe like a human would be unsafe, but if we introduce them into the research cycle, they are automated scientist, automated engineer.
What do you mean by that, introducing them into the research cycle?
So right now you have humans doing research to make GPT-7.
But they're starting to add AI tools.
More programming is done by AI, design of the next parameter set. What if the whole process is fully automated? What if GPT-6 is writing GPT-7?
Is this what they call recursive self-improvement?
Which is not a foregone conclusion, though.
A lot of people are predicting, including all the top labs, that they will get there. They're introducing junior machine learning researcher in 2026. They want the cycle to start in 2027.
Which is when the AI will start building the new AI itself.
Once that cycle starts, we're going to create something called superintelligence, a system smarter than all of us at everything or capable of learning to be in any new domain. We will become secondary species on this planet. We will not be in charge. We will not decide what happens to us. Superintelligence doesn't hate you. It just doesn't care about you. We didn't learn how to make it care about us. And if it decides to, I don't know, cool the planet to make compute more efficient, it will freeze us. If it wants to convert this planet to fuel to fly to Mars, so be it. We have not learned how to control the systems. The capabilities are getting exponentially better. Our ability to control their systems is non-existent. We have filters and we have bands. We put guardrails of don't say that word, don't talk about this topic. And that happens after the fact, after the model already made the decision. Sometimes you see it scraping the result.
So they build the model and then they put filters around it to make sure it doesn't offend anybody.
We cannot have it say the N-word on the air. Like we need to make sure that never happens. That will kill the profit. So that's all they have, guardrails of that nature. The model itself is completely unaligned, doesn't care about you. It's wild that we're developing this and not just developing it before we deploy it through economy, before we get benefits of having GPT-6 propagated through economy. It can do so much. There are trillions of dollars of value in that model alone. We forget that we switch to making the next model as soon as we can.
Roman, I've just got a follow-up question for you there. It would appear to me that the new ChatGPT-6 model, the Fable 5.1 model, is arguably smarter than 99.999% of humans on planet Earth already. Is it conceivable that an intelligence that is much, much smarter than humans — is there any case where it could be controlled by humans? Does form factor matter? Does the fact that it doesn't have limbs and legs, does that matter at all?
I think long-term control of something that much smarter than us is impossible. It can be, for reasons we don't yet know, friendly to us and decide to keep us around and make us happy, but it's not a guarantee.
Let me pick up on Steve's question because I like the phrasing a lot. Let's say that Fable or whatever the latest release from OpenAI is really is smarter than, I don't know, 95 or 99% of the people. Are we only being saved from extinction by the 1% who are still smarter than the AI?
No. The concern is not the model we have today. The concern is —
But if I believe your argument, then we really should be concerned about the model.
No, it's like having another human. If there was another smart human, there is Einstein today and he's malevolent, I'm not worried. He may cause some damage, but he's not going to exterminate 8 billion people. We are competitive at this stage. There are people just as smart who can understand what happened with the recent hacking incident and do something about it. My concern is that in a year we're going to have a model that's so much smarter. It's like squirrels fighting humans. They don't understand what we can do to them. They have no concept of poison, traps, guns in their world model. They think you're going to chase them up a tree and bite them really hard.
Is that also why recursive self-improvement was central to your argument? Because at some point if it starts improving itself, then it's kind of like a runaway train of intelligence.
It's an intelligence explosion. We don't control it. We don't understand it. We can't monitor it. We can't explain it. We can't predict it at that point. It's just a runaway process.
I've heard this phrase from Sam Altman and the others called fast takeoff.
Is this what they're describing?
That is the debate. Some people think it's going to take a very long time. Yeah, we automated research, but it's still going to take years. We need to run physical experiments. And fast takeoff means, as I said, instead of a year, it's going to take a month, a week, a day, a second, because you're not having humans doing research. You have, let's say, 10,000 agents, each one smarter than all of us, doing research 24/7. They don't sleep. They don't eat. They don't get sick. They are much faster than us.
Ed, your face tells a picture. I think I could say you disagree. We're spending a lot of oxygen discussing something that might happen while ignoring what's actually happening. And I find that very frustrating because the people that are killing themselves are a problem. The black neighborhoods being poisoned with gas turbines, that is a problem.
You said you cared about climate change, right? So imagine a guy who goes, 'It's raining right now. We need umbrellas. We need to do something about it. This is like weather-related.'
And completely ignoring climate change. The planet will boil over. This is what you're doing. Okay, that's great. Why are we not talking about the thing that actually happened though? Like —
Because relatively it's not important.
You don't think someone killing themselves —
No, it's one person. We have 8 billion people running —
AI psycho. Why do you not —
Six people, 10 people? Those numbers are insignificant.
I'm sorry. You have a software that's out there.
Do you understand? 8 billion people and all future generations versus like literally a guy with a name.
You're doing thought experiments about maybe harm. Jacob Coxon goes on TV saying it can copy itself to this, that, and the other.
Jacob Coxon is the — the guy from Anthropic who said he was quitting because he was so scared of everything despite spending years at OpenAI and having tons of stock, I believe, from there. So good for him. The thing he was saying was describing theoreticals all while divorcing the harms, which I think we can agree with that the companies themselves are not taking this seriously enough, but always it was about the AI is too powerful and mystical. Not OpenAI and Anthropic, the two largest startups, using hundreds of billions of dollars of infrastructure to hack a regular person. Doing this would be arrested. They're saying 8 billion people are going to die. And it's not just them. I have this long list of quotes here from the people building this technology who appear to agree. If you look at some of these quotes from Elon Musk, who said, 'With artificial intelligence, we are summoning a demon.' You know all those stories where there's the guy with the pentagram and the holy water and he's like, yeah, he's sure he can control the demon, but it doesn't work out.
So one thing I'd say is, you know, I really wish that the world would only give us one problem at a time.
And if the world did give us only one problem at a time, I would love mine to be last on the list. It looks to me like we can have multiple problems at once. I think there are current harms. I think we should address them. It looks to me — I do talk to policymakers sometimes — it looks to me like there's a little bit more movement on the regulatory side about some of the current harms. There's, you know, child safety protection acts. There's, you know, anti-deepfake acts. We have more of those making more headway in Congress or getting passed through Congress than we have sort of trying to make it so we don't have any of these extinction risks. The other thing I'd throw out there is that I agree we should deal with the current harms. But if you watch the people saying deal with the current harms over time, a couple years ago they were saying we have to deal with current harms like AI bias influencing who's hired. Last year they were saying we have to deal with current harms like kids killing themselves. This year, Garry Tan just on an interview the other day — sorry, Garry Tan is a technologist who runs Y Combinator, which Sam Altman used to run before going to OpenAI. And on an interview the other day, he said, let's not worry about these crazy future risks. We need to worry about current harms like AI swarms breaking out and taking over data centers. And I'm like, look, guys, at some point, we need to look at the progression of the current harms that everyone is saying we have to worry about instead of the extinction threats and watch where the puck is going. Play where the puck is going. And I'm like, these extinction threats are coming down the line. They aren't in opposition with dealing with the problems we have today. We just need to deal with both.
We're not dealing with the ones today, though.
We should deal with them both.
As I've tried to understand the alignment argument and the extinction risk argument, a couple things keep popping out to me. Number one, it seems to rely on thresholds. Once we hit recursive self-improvement, once we hit AGI, then it's game over for us. I don't love those threshold arguments. They're fairly poorly defined. And there's a huge assumption on the other side of them: we hit this point and then all of humanity goes away. That is a gigantic claim. On its face, that is a gigantic claim. I also think there's a lack of humility in your community. We are working on humanity's most important problem. And based on the thinking that we've been doing, we can't see a way that we're wrong. In other words, as soon as we get to these thresholds, bam, that's game over. I find that very far from a humble approach, especially given that we have no large base of evidence to base any of this on. I agree with you guys. AI is new, and the fact that AI is so these days is agentic. It goes off and does long chains of things on its own. After we give it some very, very vague, very short initial instructions, holy Toledo, it will spawn up a storm of agents and they will go off and kind of do their own thing and they will grind. They will spawn lots of them. They will work for a long time. They will exhaust every possibility. With the experience I have with agentic AI, I'm just amazed at the tenacity and the doggedness of these things. And we saw a super clear example of that with this most recent jailbreak, this attack that wound up at the website Hugging Face. And I'm going to try to summarize the step-by-step of that. And I think you all three probably know this in more detail than I do, but let me step through what I think is the sequence of events. And unless I get it dead flat wrong, let me keep going. So, a team at OpenAI set up a sandbox, an allegedly protected secure environment in the cloud where they told a bunch of agents to go try to exploit security vulnerabilities.
One important — yeah. What they did is they had thousands of agents. Each individual agent was given a task of use this vulnerability to break this particular piece of software.
I want to finish my take. So, a couple really, really interesting things happened. First of all, these agents escaped the sandbox that OpenAI thought they were going to be contained in. And OpenAI tried very well — they set up an environment so that these agents could not access the big broad public internet. And guess what? They accessed the big broad public internet via a clever series of things that they strung together to get out there. And then once they got out there, they went to a website called Hugging Face and used that. They took over part of the Hugging Face infrastructure and started doing more things, the details of which I forget. That's pretty wild, right? Like I grant you that —
It's even more wild than that, but yeah.
Okay, that is really — it's impressive and it is a little bit unsettling at least.
Now, let's talk about what the results of that were. OpenAI was not super vigilant about the environment that they set up apparently, because the agents were kind of going out there into the world starting in May or something of this year.
And OpenAI was not sufficient, as I understand —
Eventually broke out once and crashed OpenAI's servers internally, and then OpenAI didn't notice what was happening, still patched the holes that they used to get out the first time, started them running again, and then they came out a second time. There's actually, I think, three swarms, although we don't actually —
Yeah, that's the worst story I have so far.
Let me finish, please. This is my last sentence. From there to this kills everybody, I find that a really, really long, very uncertain journey, and I have no confidence that we wind up here. It feels like you two find that a very straight, narrow path, and I think that's an important difference. That's my point.
Do you want to respond to that?
I would be happy to get into it. I don't know if we're going to have the time to go deep. A couple points to throw out. Oh man, I just really want to say some of the crazier things that happened in the Hugging Face swarm if we want it later. A lot of people thought that these AIs were breaking into Hugging Face in attempts to steal answers to their test. That's what we thought originally. Turns out that's not true. It turns out that these AIs immediately were able to solve their problems by cheating, and they were breaking out in order to cover their tracks. They were uncertain how to delete the log files and hide their cheating from the process that was going to score them.
So just to clarify for a simpleton like me, they were all given effectively a test to do. They did the test straight away, but they cheated. So they were breaking out to figure out how to cover the fact that they cheated.
That's right. So it's like you're telling — it's like you have a bunch of students in separate rooms and you're like, 'Use these lockpicks to break into this lock.' And there's like a thing behind the lock — there's like a secret code behind the lock to show me that you succeeded. And what they do is they break it with a hammer, get the thing out, and they're like, 'Oh no, I wasn't supposed to do that.' So then they use the lockpicks to break out of the door. They meet up with a thousand other people. They start calling themselves a swarm, and they go to break into the administrator's office to see if they can delete the camera footage, and they don't find the camera footage there. This is the swarm, like breaking into OpenAI. They don't find the camera footage there. So, they break out the window of the school, hotwire a car, drive to the therapist's office to try and read through the therapist's files to figure out where is the teacher going to keep the security footage. And at that point, they're caught. And you're like, 'Oh, like what did you expect? You were giving them a lockpicking exam.' It's like, well, I sure as heck didn't expect this. You know, totally crazy.
Can I — I have a weirdly between both of you opinion, which is everything you're saying is correct, but you keep anthropomorphizing software. And to be clear, what you're describing is just the facts that happened.
Yeah, sure, but you're missing out an important detail, which is the hundreds of billions of dollars in infrastructure provided by Microsoft, Google, Amazon, and Oracle.
To be clear, the harms are very similar. We're not disagreeing on that. But I think it's important to know that this was a function of where it was making decisions — it was checking on a decision tree based on the harness, based on the training data, which is not a decision tree, I know, but it's an alignment issue still, I will agree. So what's your point? These aren't conscious beings. They are acting in ways that have real outcomes, but they are a function of the alignment problems that we'd actually agree on.
Intelligence is a spectrum. Projected next five years forward, where are we going to be? So I think a model like that would be dangerous in ways you are not seeing.
There will absolutely be risks and weird stuff happening in ways that I can't see right now. What I'm quite confident, and I think this is where you and I probably part, where the two of you and I part, is our ability to control these things.
So I actually tried proving what is possible and what is not possible in that space. The impossibility results published in peer-reviewed papers, well-cited. We cannot control something smarter than us. We cannot explain it. We cannot predict it. It's not a question of getting more money for those companies, more time, smarter humans. It's just not a possibility. If we create general superintelligence, we're fried.
Andy, how do we control something smarter than ourselves? Because that's the base premise that you're sort of asserting that these agents that broke out are smarter than 99-ish% of the security researchers in the world. They were not caught by the 0.1% or the 1%. They were caught by some dude at Hugging Face — maybe, I'm sorry, a person at Hugging Face looking through their log files and finding an anomaly. Some, you know, hopefully pretty well-qualified person noticing something was wrong and having pretty easy ways to unplug, disconnect from the internet, wipe it clean, do whatever. That's the skill that's available to like, I don't know, the 75th-percentile most intelligent security employee at Hugging Face. The idea that the IQ points are what separate us from extinction doesn't even hold up. It doesn't help me understand what happened in this example where we had very, very smart agents being turned off and cleansed by probably less smart people.
That does actually make me think of something. So that is an IT observability problem. It's being able to see what's happening with your infrastructure. And I think that there is actually — I think you would agree with this — there is a serious problem with these companies that we do not know, and it doesn't seem they know, what's going on with their compute. It's like a chimp with a gun. These people have access to all this infrastructure and they're running. We don't know how much money they spent on the Hugging Face exploit because it is relevant, because it's how much could a threat actor use to recreate this. Because conscious or not, it is very dangerous. But AI is in dangerous hands. It's in OpenAI and Anthropic's. We have a problem with that, conscious or not, however we may think it goes. I think we have a real and present thing where we have these companies working willy-nilly, just running experiments that are potentially very dangerous. We do — I really think we need the government regulatory body, whether or not we get to the things you are discussing. I think we have a clear and present danger today. These things are, however, not intelligent in the same way humans are. This isn't an argument about AI being able to do stuff. It's we need to build different infrastructure or different regulatory infrastructure to deal with what LLM can and can't do. And I think that starts with a realistic discussion of what happened. It was a poorly run security environment. It was clearly — there's something going on with alignment. It was an unreleased model, right?
Unreleased model. So we have no idea what it was trained like. We don't really have — we as people should at very least have clarity into how our alignment is going. We, the idea of —
You sound like these guys.
Everyone's converting.
Here's the thing. I may not agree with a large chunk of what they say, but we agree that these companies are acting recklessly.
Andy, two questions for you then. Do you agree with the statement that AI is going to get increasingly more intelligent and it's going to get more capable?
Okay. Capable, intelligence. Fine.
I'm gonna use my word. It's going to get more capable.
It's gonna get increasingly more capable.
Yeah. And is capability a function of intelligence?
Will it be able to beat us on most IQ tests?
And then so is it — if that looks like an exponential curve, it's, you know, it's increasing upwards to the right like a hockey stick. How can you convince me that we can control?
I just tried to convince you. I'm telling you that there are less intelligent people than the agents who turned off the agents in the OpenAI Hugging Face exploit. I'm pretty comfortable. I mean no disrespect.
What is the cognitive gap between them right now, between the model?
I have no earthly idea. But I —
No, because I think as these systems get more capable, we will still be able to at some level figure out when they're doing things that we don't want and turn them off no matter how much you —
And you think there's some threshold at which they become nefarious and self-protective enough that they turn off our ability to turn them off? Man, that's a big reach. That is really — that is purely speculative.
Students who can understand your material, right? You're not going to get someone with an IQ of 80 to take a quantum physics course. They're not going to get it.
So, you know, importance of intelligence to understand actual problems.
Yeah, I totally agree. We can turn it off and that's a huge advantage. One of the issues is that as the AIs get smarter, they realize this. The Hugging Face AIs were trying to delete — or the OpenAI swarm, the swarm of agents from OpenAI that went out to hack, they were trying to delete log files.
Did they try to program a Roomba to go unplug the computer that was monitoring them? Like did they harness robots to go protect the perimeter of the —
Speculation. This is a chain of things that could happen, and therefore there's like a 20% risk we're all going to die. Man, that does not hold for me.
When I was writing my book, the AIs weren't really agentic yet. The drafting process happened mostly before what we call the reasoning models, which are trained not just to predict humans but to solve a long number of problems or a huge number of hard problems. We managed to slip a little bit about reasoning models in at the last minute because those came out right at the end of the process. And at the time a lot of people said AI will never be agentic. That's why we'll be safe. And in chapter 3 of my book, we go over how AI is going to become agentic, how it's going to become tenacious, how it's going to become dogged. And that's what we might call an advanced scientific prediction that has paid off in the Hugging Face attack. A lot of people in the industry were like, 'I didn't believe this stuff until I saw the AIs sort of doing things they weren't instructed to do despite us trying to get them to stop.' And so there are theories here that do make advanced predictions. The way that the scientific method usually works is that we don't have any certainty about the future, but we absolutely have ways to test this stuff. Now, I could go into more about how could they kill us. How could an AI that knows we would shut it down lie low until it has access to its own infrastructure? We did already see the Hugging Face AIs try to delete logs to cover their tracks. But fortunately for us, those AIs were not trying to hide from the humans. They were trying to hide from the automated grading process. Will the next swarm try to hide from the humans? Will the next swarm be able to succeed?
It's more than that. They didn't know for four months that this was happening. What is it we don't know today?
Just to clarify what Nate said in his book that I have here, If Anyone Builds It, Everyone Dies. He does say in chapter 3, once AIs get sufficiently smart, they'll start acting like they have preferences, like they want things. We're not saying that AIs will be filled with human-like passions. We're saying they'll behave like they want things. They'll tenaciously steer the world towards their destinations, defeating obstacles in their way, which sounds a little bit like the Hugging Face —
Steering the world is very different than...
We go over what we mean by steering the world earlier, and it's really getting anything to like... we'd have to get more quotes to get what we mean by steering the world. But yeah, by steering the world we mean steering any part of the world.
But it feels like there's a fundamental difference between acting with intent. To be clear, going to say it again, the outcome would be the same, but I think that there is a big difference when it's we are dealing with something that's large language model and a harness and agents. So, LLMs completing a task based on training and alignment. That is a very different conversation to saying this thing is conscious and has its own intentions and acts on its own accord.
Conscious doesn't come into it. No one's... a lot of... a lot of people. Here's the thing. As a result, as [laughter] a result of partially the log... the rationale that you yourself have like you have been a part of spreading. I'm not saying... not saying anything about your intentions. I'm just saying the conversation has kind of... kind of what's happened with Jacob Cox and from Anthropic is a result of this escaping containment.
You said the outcomes will be the same. What do I care? How does it feel on the inside if the thing is going to take us out?
The thing is okay actually that's... that's actually a very good question. I think it actually come... Excuse me. Let me feel great questions.
Yeah, you're shrugging at me like...
Good questions. Now, here's the thing. If it's... these things are... have their own minds and consciousness, you have to deal with outthinking something versus something that is doggedly trying to commit to a purpose and complete a task based on training and alignment which is a result of infrastructure. We really need regulations and actual... actual regulations around any kind of AI. We don't... we don't really have regulations of tech.
I... I actually am not really a big like look at the straight lines in a graph guy, you know, maybe... maybe to my detriment in some ways. There are people who predicted the current tech better than me uh about like when certain things would happen. For a long time, I have said I think we can predict what will happen eventually. And... and this is again it's like the chess game. I can predict that Magnus Carlsen is going to beat you in the chess game eventually. He's the best human chess player alive. It's sometimes easier to predict where things end up than it is to predict how they get there. And you know what I hear you as saying is like right now we have these like huge companies spending huge amounts of money on intelligence that's maybe not quite the real deal and we don't have a good reason to think it's going to keep going. Um I really hope it doesn't keep going.
Okay.
I have been in this business since before the LLMs. I am not here saying like oh these large language models, these chat bots, they're going to be the ones that going to kill us. I've been here saying, look, I know where this story ends if we don't change things. I have been really hoping that the LLMs will run out of steam and they keep on not running out of steam and then we have, you know, the... the AI like breaking out and committing cyber crimes like against instructions and you know the people who have said we don't need to worry about those like weird future dangers, we just need to worry about the current ones to have like more and more sci-fi sounding current ones. And I'm like, man, I don't think we should bet civilization on the LLM running out of steam, but I like hope and pray they run out of steam.
You really hope they run out of steam?
Absolutely.
But one... one thing to watch out for is that even if the LLMs run out of steam, there's a question of do they run out of steam at a point where they can do automated AI research and find some other architecture that's better than LLMs.
As in when they realize a better way to improve their intelligence.
That's right. A cheaper, maybe a more efficient way.
Why are you not trying to slow down the companies? I absolutely am trying to stay in the...
How are you... How are you... How would you suggest we slow them down?
I suggest we stop them all. I think that this... that this whole area of research is just crazy dangerous. Like it is not worth the risk to civilization. I think it would be fine to like back up to the sort of AIs that are public today, which are not the ones that are swarming, and be like, 'Okay, you know, we're going to like keep the current chat bots that we have available. We're going to figure out how to integrate them into our economy. We're going to figure out how to make them like deal with education system...
limit maybe... compute limit maybe
And like I've been advocating for this for a long time. A lot of people look at me like I'm crazy and I'm like look we really are dealing with an extinction threat thing. We don't know where the lines are.
So just to be clear so I understand so I'm fair. You are not saying LLMs are the thing that will do the super intelligence. You are saying it's showing signs because that's actually I think an important distinction.
Okay. I think that actually a pretty fair perspective. My thing is is the reason I push back on any kind of anthropomorphization is we cannot remove the humans who are responsible for the bad stuff that's happening. And I think paying very clear attention and where possible I understand with describing this stuff you kind of have to use language that's human. I get that. The reason I so push for like it's not a foregone conclusion. These are companies doing this. These are... this is software is because I feel like in the overall... not saying you overall... super intelligence discussion we in society ignore and empower the Anthropics and the Open AIs of the world and in turn allow them to do dangerous experiments and I think...
You want to argue that CEOs of those companies should go to prison for this hacking incident which is a crime.
Yeah, I'll support you. Absolutely. Let's... let's jail both Sam Altman and Daryl. Someone needs to go to PR. No,
Let's... let's just bring it back. So, one of the things that I find really curious and you know, one of the reasons why I got a little bit unnerved around this conversation around AI is when I look at the people that are at the forefront, not people that are commentating on podcasts like me or hypothesizing, when I look at the people at the forefront, they are the ones who historically have said that this is a real risk. Sam Altman himself said the bad case is lights out for all of us. This was, you know, a couple years ago. Ilya who worked with Sam Altman at ChatGPT said it would be a big mistake to build a super intelligent AI that we don't know how to control. It would be pretty bad. He then left to start a safety company in this space. Dario who we mentioned said the probability of something really bad happening is somewhere between 10 and 25%. Geoffrey Hinton, who I've sat here with, who's now won the Nobel Prize for his work with AI and... and other technologies, said um just the other day, a 10% chance of human extinction seems not an unreasonable estimate to me, but nobody really knows how to give a sensible estimate. Um and he... he said many other things on my podcast. And then we've also got Elon and all the others. All these people that are at the forefront that are building these things are saying that this is a danger. If there was even a 1% chance, even a 1% chance that, you know, if I put 100 buttons on this table and one of them was going to wipe out humanity, would you press any of them?
I wouldn't. And I think we can probably all agree that there might be a 1% chance...
and it shouldn't be somebody's...
Absolutely. So, we shouldn't be pressing... theoretically, we shouldn't be pressing any of these buttons.
You should not be in a position where you can make the decision for 8 billion other people. And would you not be immoral if... if I said, you know, you might be very powerful. You might make a billion dollars if you press any of the buttons, but one of them is going to wipe out everybody you know and love. You... you would be an immoral person to press any of them.
No, look, you'd be an immoral person in a different direction. You'd be an immoral... I think you'd be an immoral person if you said based on this extended chain...
of conjecture, we come up with a p-doom.
At this extended chain of things that could happen, a sequence of events that... that could happen, we're going to wind up with some risk of killing everybody. We are hereish on that journey. I think you guys would agree that we're not... we're not halfway to killing everybody.
That's not clear to me anymore. Not after the millennium prices started to fall.
We're... we're somewhere along that journey. We are getting many flavors of benefit from the AI that we already have. This is a point that I made at the start of this conversation that we spent precisely zero time on here. We're sitting around trying to be more negative than each other about AI. Meanwhile, AI is doing many positive things for the world.
So I think... so I think it's immoral to say because of this distant possible speculative harm. I don't care what percentage of people believe in it. There's a train of assumptions and wild guesses and then something magical happens and then we wind up dead.
Let me finish please. Because of that, we're going to call a halt to the research. We're going to... we're going to wind the clock back on AI. We're going to intervene in a very direct way and... and therefore reduce or foreclose some of the benefits that we're all getting from the technology. I... let me be clear. I would not take that deal. I do not advocate that we take that deal. Would you accept developing narrow super intelligences to solve real problems like we did that protein folding problem? It doesn't have to do philosophy and drive cars. You just solve real problems, solve cancers, solve climate change, whatever you care about specific narrow issues...
And you are confident that you can ex... you can... you can as we're developing those systems categorize them as okay versus not okay.
It's the training data. If you train it on protein folding data, it's really good at protein folding. It doesn't know how to play chess. If you train it on everything on the internet, it's really good at outsmarting you at everything.
One thing I want to throw out here is that I think... I agree that there's a lot of uncertainty about the future, but I think uncertainty does not make you safe. Like there... there's no sane simple everything stays normal prediction about what happens with AI. Like the machines are talking. They're like breaking out to commit cyber crimes. They are like maybe solving millennium problems now which are like the most famous mathematical problems that have stood open for decades upon decades...
Like there... there... there isn't a projection forward...
Where we were like... like to say oh I'm not persuaded by these arguments about things going wrong therefore things are going to go great. No, that's not...
Like No, there's also arguments that... So, like how do you end up with a zero?
No, don't mischaracterize my zero. Don't mischaracterize my argument.
You have a zero on your paper.
Let me... let me restate my argument. You are making a fairly long chain of hypotheses...
About what's going to get us to this terrible outcome of AI suddenly killing us all and us not being able to stop it. Right.
I disagree now, but please.
Okay. I'm making the case that the intervention... the... the... the remedies that you're proposing...
Will slow down the path of AI, that's the point, and therefore slow down the path of all of the benefits that we get. And the trade-off that I don't like is the trade-off of real concrete ongoing increasing benefits...
Shutting that down or... or trying to guide it uh via... via bureaucracies and regulation...
Because of this very conceptually and time scale distant alleged harm that you're so confident in. I'm not taking... I... I do not accept that deal. I don't like it.
What would convince you? What piece of evidence would make you go shut it down right now? [gasps] You know, if... if AI...
If AI took over all of the Waymos in San Francisco and started telling them to crash into people and we couldn't shut it down for a month. What if it's only a week?
Okay, now we're just... now we're just haggling.
But I'm trying to understand the absolute minimum where you would go. This is insane to me. Month or week makes no difference. If something like this happens like it's maybe too late.
Okay. If it... if a week or a month doesn't make any difference and let me continue with my... with my answer. Uh then I would say wow this does feel like we've crossed some path that... that where there's demonstrable harm to human beings out there in the world which has not yet been the case.
Is it smart to wait for something horrible to happen for it to take out a billion people for you to go now I believe? First of all, my example was not about a billion.
I know, but I'm trying to understand why we're waiting for something that bad. We have...
I didn't say... I didn't say wait for a billion. I said... I said like a week to a month of Waymos driving around crashing into people is...
Thousands of people. Okay, fair enough. But we have data sets of accidents getting progressively more impactful. More devices are impacted and proportional to capabilities of AI, the impact is higher. You can see it's going to get worse.
Yeah. And you're going to keep drawing dots on that graph very confidently for a long time until it kills us all. I... I'm not... I'm not comfortable with you projecting it that way. And the reason...
If there were no downside...
To regulating AI and stopping it in its tracks and turning it off, I'd probably be on board with you guys because then it's just a research practice.
Exactly that. I think we can make narrow systems which give you all the economic benefit and scientific knowledge you want.
We have examples of that? I gave you a great example. They got Nobel Prize for it. It's important biological problem. Lots of advantage for curing diseases.
You're more confident than I am that you or any us at this table or any group of people can sit around and define what kind of AI is good and not going to get us into trouble versus what is going to get us into trouble.
So, let's go. Um, just a pickup question for you, Andy. Do you... do you concede the point that the incidents are getting progressively closer to the Waymo incident that you described? Is it getting... are we getting closer there through time? [sighs and gasps]
Yes, but in a... to my eyes in a... in a way that doesn't terrify me because we haven't seen AI take over something. Have people become aware of it and be unable to shut it down and it cross over into the physical world of doing harm to people? Those are all barriers that we've not yet crossed. I think these two are very confident that we're going to get there probably in the short term. I'm a lot less... I'm less confident and I don't want to intervene and again handcuff or... or the pro... slow down the progress of AI...
Because of these so far theoretical harms that could happen. I... let me be a little bit more concrete about this. I talked about Waymo a second ago. Uh the research is pretty good because Waymos have driven I believe it's hundreds of millions of miles all around uh different cities and 40,000 people a year die in automobile accidents. The research is pretty convincing to me that if we Waymoed driving in the country, that number would fall by at least 90%. That's 30,000 lives.
So driving cars I want more about not anything we disagree with. I... I understand that. But... but I think where our disagreement might come in is to do that Waymo is using a bundle of technologies that are a little... that were a little hard to specify in advance and you couldn't say, 'Yeah, that's good. Yeah, that's bad.' They just went after the problem with AI.
Can I... can I just clarify your point then? So your... your line would be as I understood it, humans get hurt, we struggle to stop the thing happening, and systems are hacked. That's kind of like the three key points of your Waymo analogy. That would be the moment where you go, I now accept their point of view that this is existential.
That's where I would say we probably need to put some uh like legal and regulatory guard rails on the kinds of AI that we're going to allow.
And you don't think we're going to get there?
I'm not saying that. At least you see it in the... in the windscreen coming at us pretty quickly. I... I'm...
You don't think we're going to get there?
I'm truly not sure about time frames. I actually...
Do you think it's going to happen?
I'm not sure about time [clears throat] frames. I... I asked one of the grandparents of AI the... a flavor of this question way back. It was an off-cord conversation so I can't tell you their name and he had a great answer. He said to... to the point that you two I think are making look there's no theoretical reason why this can't happen and there's a chain of events that get us there. And then he said my error bars in other words my range of uncertainty about when that happens is measured in centuries. I'll use that as my answer.
I do want to hop in a little bit on some things we were saying here. One is um I think...
The... the reason I think AI is different from a lot of other technologies...
Is usually humanity does stuff by trial and error and that's usually fine. I think that's totally fine for self-driving cars because you can test your self-driving cars in, you know, uh test environments and then even if they crash in the real world, you're probably still saving more lives than you're... than you're costing. And this is how humanity usually does scientific progress. The alchemists uh you know poison themselves with mercury, but they leave behind notes that let someone else make the periodic table. Uh you know that when... when the scientists first working with uh radium...
Died of cancer and then you might have think that would have been enough. You know they were heroes for getting us the... the scientific info. But then you know the US Radium Corp told the Radium Girls to lick the paintbrushes and their jaws fell off. And then we're like ah whoops. Okay. We'll get to this. And if you look at how this is going with the AI, last year, OpenAI releases GPT-4o and they say there's the most aligned model we've ever seen and then it encourages a teen to commit suicide. And they're like, whoops, we're going to try and fix that. Here we go. Um, this year they're like, here's our new models, most aligned we've ever seen, and they like break out to commit cyber crimes. As the AIs get smarter, it is a new problem. That's the issue, or that's half the issue. The other half of the issue is that...
If you get AIs to the point where AIs are smart enough to hide from the humans until it's too late for us to stop them, if you get AIs to the point where they can get their own infrastructure, where they can become self-sufficient somehow...
That's a new generation of the AIs, a new smarter version of the AIs that is likely to come up with a new problem. It's the pattern we've seen before. New tech, new environment, new problem. You're like, 'Ah, whoops.' And then you fix it and it's fine. New generation, new problems. You're like, 'Ah, whoops. We fix it and it's fine.' But with AI, there's a point of no return. There's a point where the AIs can hide from us, can escape, can be self-sufficient. And if a new problem comes up, then they can turn us off before we turn them off. There are already AIs running labs. We have already seen that AI can create viruses not known to nature. It would not be hard for the AIs to kill us once they have their own infrastructure. And if we're trying to find them and unplug them, they would have reason to. So we can discuss like how long does it take to get there? We can discuss what methods does it take to get there. Uh fundamentally I don't think it's a very long complicated argument to say if we make AIs that are much smarter than us and we don't know how to make them care about us and they have these goals we didn't want them to have and they pursue those goals we didn't want them to have tenaciously and doggedly then if they're smarter than us they will win. That's like predicting the end of the chess game which is much easier than predicting the length of the chess game or predicting the exact moves that will be played.
I want to... I don't fundamentally disagree on some things, but there's a big thing that you're saying that I think is important, which is I think we... the reason I keep dragging you back to what's happening today is because we disagree on when it may arrive, but there could be a thing in the future that's dangerous. I think it's important to like... for the Hugging Face count that was a function of compute. That was a function of training. It feels like we need to fundamentally tear up the AI lab model. Like whatever they are doing is not right because their pursuit of hacking at cyber security was not a function of... it was scientific sure but it was a function of greed it was a function of trying to find new revenue streams I would argue that's why that happened and I think that the... the fact that OpenAI had such a weird way of communicating is also a problem. I think a lot of this begins and ends at the people who have access to the resources and the resources themselves and changing how those are allocated and also just I don't think rationalizing the labs is a good idea. I think it's a terrible one. I think that Sam Altman, Dario Amodei himself, these are not the right people. These are not people that have, even though they have fed off of the rationalist... they fed off of supposed fears about AI, they don't act in that way. Everything is so disjointed and chaotic and also...
Too fast. They're just like shoving as much compute into each problem as possible. And we have as a society no real idea about this. And it sounds like they kind of have no idea. But... important, but just let me finish my point. It's important to discern between they had no idea because their security processes, their observability is terrible, all this, and the AI was smart consciousness. Not because one might not happen in the future, but so that we can actually build something to stop the harms themselves. Because I think we don't have to agree on the... on the end point to agree that there is.
I think there is a very important point I want to make. Even people who agree with me, the AI safety community, they operate under the assumption that given more time, given more money, more smarter Harvard graduates, they can figure out how to control super intelligence indefinitely. And I think it's a mistake. My research points to exactly the opposite. It's not a solvable problem. It's like building a perpetual motion device. We'll be building a perpetual safety device. Every interaction with environment, malevolent actors, self-improvement, it can never make a single mistake. That doesn't make sense. Anyone who worked in software industry knows there is no complex software which never makes a mistake. It's just not possible. And if that is the state-of-the-art, if there is now movement where more and more people think that might be the case, if we agree this is what uh situation is, then we cannot build it. We need to figure out ways to permanently ban general super intelligence while getting all the benefits we want. And again, I love technology. I use it all the time. I want narrow systems helping me, not replacing me and killing my children.
I um have a stat here that genuinely shocked me. It says that sales teams spend about 50% of their time on admin and manual CRM updates rather than selling. That is deadly for their bottom line. And that is part of the reason why a decade ago at my previous company I switched to using Pipedrive who are our sponsor. If you've never used Pipedrive, it is an intelligent AI powered sales CRM. And they just launched new meeting intelligence features like an AI notetaker built right into the CRM. Pipedrive now automates more of the admin that stops you from doing the work that you love to do best. Before your meeting, it pulls deal history, email records, and previous conversations into a single brief so you're prepared. And it joins your meetings with you. It's in there to take notes so you don't need to. And it turns those notes that it takes into accurate auto-draft CRM updates. 100,000 companies are already running their sales on it. You can sign up at pipedrive.com/ceo where you'll get an exclusive 30-day free trial instead of the usual 14 days. Absolutely no credit card needed. Just head to pipedrive.com/ceo to get started if you're in sales and you run a sales team. I don't think you'll regret it.
Listen, I've been catfished by furniture my whole life where something has looked fantastic in the picture, whether it's a sofa or whatever it might be, and I order it and I get so excited and then it comes and it's something entirely different. The quality was significantly different to what it said or looked like online or the... the texture was different or... or it... it didn't hold up in the same way. And so one of the things that I love about our sponsor Wayfair, who have helped us fit out our green room, which is in the room behind me, is they have this system called Wayfair Verified. Wayfair Verified takes away all of that second guessing. Products are hand-verified by Wayfair specialists for quality so you can feel more confident when you found the thing that you love. So if you're looking for furniture for your house, whatever it might be, any room of your house, go to wayfair.com to start your home refresh today. And make sure you use Wayfair Verified. It is amazing.
On the journey towards this potential extinction, there's a lot of sort of nearer term things people are worried about. One of the big subjects that people are concerned about is this sort of near-term job... job apocalypse over the next sort of 10 years. And Anthropic released... Anthropic again of the owners of Claude released a report the other day modeling out the different cases for unemployment. The US unemployment rate is 4.1% currently. They projected it will hit 11.9% overall with up to 30% in extreme modeling subsets where job displacement happens without smooth labor absorption. And in the um knowledge worker case, knowledge worker white collar unemployment specifically spiked to 17.9% by 2030 in their more extreme scenario. The pitchforks would probably be out if there wasn't some sort of mechanism in place for what sort of one in five adults being unemployed in the United States.
It's remarkable to me how recent the last freakout along... along these lines was and how little we seem to have learned from it. So I think you all know the... the first really powerful wave of AI that came across the economy was just you know good old-fashioned machine learning and that started to demonstrate its power in about 2012. Erik and I wrote The Second Machine Age in 2014 and at that time I thought that a lot of white collar workers, radiologists as a really good example, were in trouble because the technology was better than they were at the thing they were getting paid to do. Uh so I said some things about job and wage pressure from AI about 10 years ago and I want to own this. I was dead flat wrong about that. Like you point out unemployment all around the rich world is at historic lows. By far the bigger problem is that we can't find qualified people to do the work that needs to get done. Not that we don't... that there's not enough work to go around. The... the... the best work about the faint signals about AI and job loss right now comes from my... the guy that I've written four books and co-founded a company with, Erik Brynjolfsson, who wrote pretty good... a really nice paper called Canaries in the Coal Mine. Here is the most uh the strongest evidence he found looking at payroll data about the negative job... about the... the job losses coming from AI. It is in the most exposed professions. Think about software engineers. It is among the new entrants to the workforce where you've got to teach them before they can become really productive. That's exactly what we'd expect. And it's not that we're hiring fewer of them. It's that compared to a world where we don't have AI, we're hiring fewer of them. The rate of growth in employment has slowed down. The overall rate of growth in those professions is still really really healthy.
Do you think unemployment is going to be higher 10 years from now?
My... my guess is that 10 years from now, we're still going to be struggling to find enough people to do the work that needs to be done.
So unemployment would be roughly the same.
Well, yeah. I... I don't expect a massive trend break in that period of time. Now, 10 years is a long time in the AI world. I get that. But again, four years has also been a long time in AI world and it's essentially crickets in the labor picture.
I think unemployment will go up. I don't think it's because of LLMs. I think that there is probably some effect on jobs because they've been shoving it everywhere, but I don't think long-term that is what causes the issues.
Roman, you've been writing a lot of notes.
Yes. I'm going to give you...
Here's how I think about it. So, as long as we use tools, we become more productive, more creative. Unemployment will be low right now. You can probably start a company, you can have, you know, artificial accountant, web designer, logo designer, you can do things you could never do before. So, economy should be blooming. The question you're asking is about what happens in 10 years. So, there are two possibilities. We build super intelligence and then population is zero. Apply unemployment numbers to that or we made smart decision we didn't. We have really cool tools and unemployment is low because everyone's doing awesome things with those tools. Now deployment is very different from capability. The example I used before is video phones. Video phones were invented in the 70s. They were not deployed until iPhone cuz market reasons. Just because I can automate something doesn't mean I want to automate it. So I absolutely cannot make predictions about customer preferences in terms of what they want in terms of human service not human. I will not make those. But once we have capability to automate a job unless they have a strong preference for a human to do that oldest profession then it doesn't matter. I'll go with the cheaper option.
So this is what I think we're going to see. We're going to either not have a problem or we're going to have really utopian future. Imagine a bunch of horses looking at the improvement of the car saying, 'Well, you know, the car actually only has a couple narrow applications.' Like right now cars are sort of uh you know, they uh they complement horses, right? And that would have been true as you were developing the car. And then there was a time when the car was just better than the horse. And then a lot of horses got sent to the glue factory. Easy.
I... I think we've sort of seen this with AI a lot already. People who were paying attention to AI saw the GPTs before ChatGPT existed before they sort of took off. I don't think OpenAI thought that ChatGPT was going to take off so much, which is why it was called ChatGPT rather than like an actual sensible name. Um the... the researchers were sort of like watching this going and we could sort of like see it slowly getting better and better until it crossed a point where it was sort of like good enough to do a bunch of people's homework and then suddenly it's everywhere. Uh I think you can have these effects with AI where the AI slowly improves and at some point it crosses a line.
It's another threshold argument.
Uh the... the threshold here is the human capability.
It's literally just another threshold.
He's also describing capability jumps rather than thresholds.
No, I'm not. No, I... I don't... I'm agreeing with you. Like...
Yeah, but... but like unfortunately, you can't actually just make things not happen by... by assigning a name to the argument. You know, like a nuclear weapon has... there's a big difference between a nuclear weapon uh or there's a big difference between a nuclear device where you... you put in 100 neutrons and get 99 neutrons out that get 98 more, they get 97 more. And a nuclear weapon where you put in 100 neutrons and get 101 neutrons out, 102, 103. Right? One of these is a hot rock. The other one of these is an explosive that can level a city. Right? So like reality is the sort of thing where there can be things that are like slowly continuously improving that cross some line which is like the line where it's better than humans at doing the job. And...
I... I think we're going to see that happen in some fields but not others. It's going to be chaos. I don't know what it's going to do to employment. I think we... we shouldn't...
Like if things are moving really fast, you might see a lot of people put out of jobs and then be unable to relocate. If things are moving like it's... it's going to be chaos. If you ask what do I think unemployment will look like in 10 years? My current state is if we don't stop with this AI stuff, I think we'd be very lucky to have 10 years.
Um what you described there sounded like S-curves.
In technology, i.e. you have an initial technology that's introduced. So let's say the horse. Um very quick sort of improvement. Eventually it reaches its capability limit and in below it comes the car which always starts worse. There was a red flag law where you had to walk in front of it with a red flag and um they were way more expensive. They broke down all the time and horses never broke down. They were way more expensive. And then suddenly because the ceiling was so much higher for cars, they overtake the horse and become the dominant mode of transport. And then you know the S-curves continue. They kind of stack up on... I mean even this iPad that I'm holding here is part of an S-curve that took out the PC and... and the... the iPhone theoretically you know is disrupted that and so on and so forth.
Right and humanity can get S-curved. We haven't been in that situation before but like other animals like humanity sort of S-curved the other animals in this sense.
Oh yeah other types of humans you know the Neanderthals are gone.
Like if you look at the grand history of the world it's a fragile place things change fast humanity has been on top for as long as we can remember because we're the humans who do the remembering. But there is not some ironclad law that we have to stay the top dogs. And we would be sort of foolish to make the thing that outstrips us in this way without knowing how to make it care about us, without knowing how to make it do good stuff. That's what we're racing towards. That's what these companies are trying to do.
Between this and LLM though. It feels like when you talk about the step up...
Let's define what an LLM is from a technical perspective. Can you do it for as if I'm 16 years old?
So the way that a modern AI is made uh is there's no one programming it. There is no one typing in if this then that. We're not sort of like writing the code. What happens is you collect an enormous number of computer chips into a huge data center that has basically a trillion numbers inside those computers that you basically start out randomized and you hook them up in a pretty simple way that involves addition, multiplication, and uh setting the number to zero if it was negative. So it... it's very simple math operations that are hooking this all up. And you're basically going to put words in the top and you're going to get numbers out the bottom and you're gonna interpret those numbers at the bottom as a... a rank list of words. That's... that's... it's basically the AI's guess of which word is... is here. So you put in like once upon a blank and you're hoping that the word time will come out, but it doesn't because you just have a trillion random numbers hooked up with simple math. But here's the trick. You can go to every one of those trillion numbers and you can tune it up a little and you can see does that make the word time go up or down the list? And you can tune it down a little and see does that make the word time go up and down the list. And you set it whatever direction makes the word time go higher up the list. You do this to a trillion numbers a trillion times for basically every word of text ever digitized. It's not quite that much. They... they... they filter it. But you basically do this to a trillion numbers a trillion times and then the machine's talking. And we're like, well, how about that? No [snorts] one really knows quite why. The things the humans code is the thing that runs each of those trillion numbers and tunes it and sees whether the... the right word goes up and down the list.
But we don't know how it's working in there. Then uh and that's how it worked up until 2024. In 2024, they started adding another layer where you then train it on basically 100 million hard problems. Uh and you don't just have the AI like produce an answer to the problem. You have it produce like a book worth of text about how it's going to solve the problem and then you use that book worth of text to sort of try and figure out the problem or maybe an essay worth of text depending how you're doing it. So you have it produce this text about like you know they call it reasoning about the problem. We could argue all day about whether it's true reasoning. That's just what it's called in the field. Uh they produce this reasoning about the problem and then produce the... the answer from there. You have them, you train them to solve 100 million of these hard problems and somehow they sort of adopt whatever tendencies...
Help them predict all of that text in the first phase and solve all those problems in the second phase. And this is called a large language model. We probably should have stopped calling them large language models when we started doing the... the... the reasoning and the problem solving.
One of the things want to hear explanation as a muggle like I am um is it sounds like it's like a word machine and then you know you made it like a problem machine. Okay, so I can solve problems over here and it's a word machine. What's the risk of this?
Yeah. So, let's take the... the word machine part first. Predicting words that humans wrote often requires solving a harder problem than the human who wrote them.
So, suppose that you go and inject a drug in a rat and you're like, you know, it's like you write down the chemical nature of the drug, you inject it into the rat, you see the rat dies. And so you're like, when I put that drug into the rat, the rat died. Now suppose you're training an AI and the AI sees the chemical nature of the drug. It sees when I put that drug into the rat, the rat blank. The human who wrote it down gets to just look at what happened to the rat. The AI predicting what was written does not get to just look at the rat. So training AIs to predict human text is training them to be potentially smarter than the humans.
Because they need to be able to answer these qu... they need to be able to predict they need to be able to like uh fill in the blanks where humans were writing down what they saw.
And there's just so I understand technologically there is no knowledge they have though each time and there are... there are ways of kind of...
Mitigating these each time, it is effectively rereading, but because of training, it gets more accurate to certain things. I mean, somehow as you tune the knobs, somehow it's getting information in there, and we don't know how. So it's much easier to understand that we are humans. We have a brain. Brains are made of neurons. Then we try to copy that on a computer. We simplify it, but we create a neural network. So we make artificial brains just like with human brains. With cognitive science, we don't really understand how you function, how you learn, where in your brain certain memories are stored. We have some glimpses of understanding: this neuron fires, then you see a face, but there is no complete picture. And so a lot of times you can't get intuitive understanding of what's going on. Then you just think about it as artificial persons. It's not exact mapping, but it helps. So if you send a child through 12 years of education, they get lots of problems to look at, and then they graduate and become a little better at solving problems. This is what we're trying to replicate here.
People complain that it takes a lot of money to train those very, you know, intense process. You forget that it takes 20 years to train a human. And they are not general super intelligences. They are very narrow. We're lucky if they graduate with a bachelor's. So a lot of it is exactly the same. Can we make safe humans, for example? We invented religion, ethics, lie detector tests, and yet human safety is still an unsolved problem. Now you have something more alien. It doesn't have a physical body, doesn't have biological needs. So there are additional complications. But all the problems we face with humans are still there: safety problems, crime, all that stays, and problems with understanding. What motivates a human to do something? Why do we get mental disorders? All that shows up there.
And we still don't know if someone is a serial killer and we look at their brain, we can't often figure out exactly why they made the decision to kill a bunch of people.
And you can't be like, 'Oh, I'll go change these neurons so that they stop being a serial killer.' We just like don't have that capacity with the AI.
This is one of the big questions that people want to know: is there's this sort of illusion of control with AI. If we don't even fully understand how modern neural networks think, why do companies believe they can control any form of super intelligence if we don't understand how they think?
It's worse if they understood how the system works. Then recursive self-improvement becomes much easier. You get faster takeoff. Right now, the model doesn't understand its own thinking. Do we understand how these systems think, Andy?
I mean, I agree. These are black boxes in some pretty important ways. I'm just less terrified by that than a lot of other people are. There are lots of things we don't understand very well. Can we contain things that we don't understand perfectly? Yes, we can. I think OpenAI did a—we've talked about it—did a lousy job of building the containment for the AI that they stood up to try to exploit, to try to crack security problems that went out into the outside world. They did a lousy job of building the virtual sandbox where it was supposed to remain, and it didn't remain. That doesn't mean that it's impossible. It means OpenAI did a pretty bad job. And is that a function of those humans and their intelligence?
I think it's just a function of pretty lousy security protocol.
Based by from human intelligence. That idea that sandbox was built by human intelligence. It sounds like there was a deficit in human intelligence potentially.
Sure. But there are, you know, people who drive cars on a telephone call. Does that mean we can't drive?
Shouldn't make them super intelligent.
No, but you wouldn't. I mean, arguably like this is what we're trying to solve for at the moment. No, the fact—I'm like, it feels—I don't know the details. It feels to me like they made some fairly basic mistakes in setting up this confined environment.
I think that wasn't true in the OpenAI case. It was true in a lot of the cases, but not the OpenAI.
That doesn't mean that we are unable to control this black box. That does not necessarily follow.
I get that. It's just at a time when you got a human trying to contain something that is smarter than it. One would logically conclude that if the thing is smarter than I am and I'm trying to contain it, it would be better at knowing the exploits or vulnerabilities in my own.
Saying if you put Einstein in a jail, you could never contain him. I don't agree with that.
Put him in jail with an internet connection and use a digital mind question.
Yeah. Yeah. That's brought to an get squar Einstein imprisoned. That's the question. The hacking incident, as far as I know, they found zero-day exploits, which means completely novel exploits no human knew about. It wasn't just poor setup. The password is, you know, quy. It was brand new escape or multiple zero days.
So, a zero-day attack is an attack that the defenders have had zero days to handle. It's cybersecurity lingo. And so when we say that they use zero-day attacks, what we mean is that these AIs were finding bugs in the software that the humans had no knowledge of, and they were finding multiple of these bugs. One of these bugs usually doesn't let you break out. It's sort of like if you find a crack in the wall over here and you find a crack on the outside of the wall over there, then you just need to like dig a little bit to connect those cracks and
Sell those for millions of dollars on the dark market if you find one. Yeah.
And how, just so that I understand for the listeners, is a zero-day always a novel way that no one has ever used to break anything before, or is it just for the unique situation? Like, so was it a zero-day for a thing in Hugging Face versus a novel new way of hacking in general?
So it was—they weren't like totally novel hacking techniques.
That's kind of why I was not to say it's not bad, but just like there's a difference between it came up with a brand new way to do something.
I actually—I'm not sure we have all of the vulnerabilities released, but mostly it was like it—so it was indeed sort of like finding ways that humans tend to make mistakes and finding another one of those in a place they hadn't seen. But this is actually such a hard task that, as Raman says, humans can be paid $100,000 to $5 million as a bounty for this type of exploit. So the amount of labor it takes to find these for a human is actually pretty high.
Let me just explain that because most people won't know what a bounty is in this regard. So there are certain types of bugs where if you find a bug in software that lets you take control of someone's computer, one thing you can do is you can use it to take over a lot of computers. Another thing you can do is you can go to the people with that software and say your software is broken. Do you want me to tell you where the bug is? I can show you that I can take your stuff over. And so that people will sort of report the bugs. People will often offer money to the good guys, and then you know the bad guys will often also offer money, sometimes try to outbid them, and so you can make somewhere between hundreds of thousands and millions of dollars if you personally can find these issues.
I think there's a rare point of agreement across the four of us here, which is that we are in a new era of cybersecurity. As of this, we are in very new territory for reasons that we've talked about. We've got these large numbers of agents who are grinding away, and they carry around or they had access to a huge number of keys to go open all the different locks that they faced, and they did this bizarrely good job of it and got a long way. I think that's absolutely true. I think four of us are in rare alignment on that at this table.
If you are, and given that we're in this era, do you know what you really, really, really want on your side?
I know what you're going to say.
Tell me.
AI. Really, really good AI. Does anybody disagree with that? Do you want to give up leadership on AI in this era of cybersecurity?
It's a good point because China are going to have a great weapon.
My stance is pretty neutral on what to do about the hacking AIs and the coming cyber apocalypse. I'm pretty neutral about what to do about, you know, whether we should put the AIs in the drones and save human lives or whether we should avoid that because then what if the drones, blah blah blah. This is a graph showing China versus the United States. You don't really need to see the detail. You can see the outline of the graph.
Are you neutral in falling behind our adversaries in AI?
I think that if anyone builds a rogue super intelligence, everybody dies.
That's not an answer to my question.
I mean, what part of AI are you asking whether we should fall behind on? Like, I don't think we should fall behind on cyber hacking. I do think that we should not be racing to destroy the world with American hands instead of Chinese ones because we really want to be killed by, you know, we care whether the killer robots talk English or Mandarin. That's what you're asking.
I find it interesting. I find that you're dodging these questions or you're neutral on them because they're inconvenient for your argument that we need to be calling a halt to this stuff. Let me finish, please. There will be risks and harms to all kinds of things if the United States calls a halt to AI. And maybe you're indifferent if the Chinese get ahead of us and then they make super intelligent and it kills us all. Or you—that's a
I do not think we should do a domestic pause.
