Thanks. It's great to see everyone. Excited to be here, looking forward to another lecture and panel. Tonight I will be talking about capabilities in RSI. So yesterday I spoke a little bit about why an AI system might be motivated to try to take over, to disempower humanity. Tonight I will talk about why we might soon have AI systems so powerful that if they were in conflict with humanity they might prevail. And tomorrow I will talk about what I think we should do in light of those risks.
So I started off the series by saying I think there is a high risk of an AI takeover within the next three years. So just to briefly recap what I mean by that: by an AI takeover I mean a situation where AIs call the shots in the same way that humans call the shots, where they have overpowered governments, where we are unable to stop them. By high risk I don't mean to say that I have some model that spits out a particular number. I mean to say that it would be an unsurprising consequence of the current trajectory and current practices in the field, rather than something that would require a surprising or unpredictable development. And I said that I think there's significant risk within the next three years. Today I'm also just going to be talking about a shorter time scale and why I think there is material risk even within the next six to 18 months.
So I will first talk a little bit about recent progress, about what we have seen over the last eight years. I will talk about how recursive self-improvement could cause rapid acceleration on that existing rate of progress, and talk about why I think this poses very significant risks. I will also spend a little bit of time talking about what we might do about it. I'm sorry if this was billed with more solutions tonight than I will actually have on offer, but you can come back tomorrow.
So this is perhaps not the most surprising claim in the world right now, but recent progress in AI has been very fast. So if you look at the state of language models in 2018, we had systems that could sometimes write a coherent sentence, that were almost able to do simple arithmetic, that could write things that kind of looked like correct code, and maybe could identify syntax errors in sentences or lines of code. If we move forward four years, we come to the ChatGPT moment, where we have AI systems that many more people start interacting with, that are capable of writing coherent essays even if they often contain made-up facts, that could do simple fifth-grade math problems, where you could give them a description of something you wanted to do and they could write you a four- or five-line program to do it, that could identify simple bugs in code you were writing. Right? So at the time this was truly shocking to essentially everyone, even people who had been in the field for a long time. Right? These capabilities may look kind of quaint now, but this is not what was expected. If you go back to 2018, this was incredible to see.
If we think about the world of today, we now have AI systems that can write coherent if sloppy books. We are in a world where I think at this point the large majority of new mathematics is done by machines. They are clearly superhuman. Where you can give machines tasks that would require 10,000-line programs, and again, maybe a bit sloppy, but it does write real software that works. And where machines are much, much better than humans at finding vulnerabilities or exploiting vulnerabilities in software, and we are now going through a little mini crisis as we encounter a huge backlog of vulnerabilities in all the software we've written over the last decades. So I lay this out just to say that we are seeing a truly tremendous amount of progress in each four-year period.
Now a thing you might say is that that progress has focused most of all on domains where it's simple to say what an AI is trying to do, where you have a math problem you can evaluate a solution. But AI systems are also improving on messier tasks in the real world. So if you just look at how much people are willing to pay for AI systems, how much they actually slot into our economy in meaningful ways, we're seeing an incredibly rapid ramp up in applicability and huge growth in revenue. And some of this is on tasks like coding, but even coding is in the real world a messy task that involves engaging with humans. And also a lot of this is just penetrating through enterprises and consumers' lives.
The Remote Labor Index is an example of a benchmark created by the Center for AI Safety that measures how well AI systems can perform roughly 10-hour tasks that were assigned to human contractors through gig work platforms. And what you see is that a year ago, systems could do less than 2% of tasks in this benchmark at a level that would be rated acceptable by a human evaluator. A week ago, they could do 20% of tasks. I had to update the slide because as of yesterday, they can do around 40% of tasks. These are not the messiest, most real-world tasks, right? We are lagging behind where we are in math. But this is a pretty broad distribution of work — things like making visualizations, or formatting papers, or making simple programs, or doing some lightweight architecture. And the time scale for these tasks, the tasks AIs can succeed at, has been growing extremely rapidly, even in these much messier domains than mathematics or programming.
Other domains also see progress, right? We see AI acting as slightly useful assistants in the lab sciences in a way that we had not seen two years ago or even one year ago. We are seeing AI systems in short conversations with humans able to be about as persuasive as a human who has spent a lot of time practicing on that specific topic to persuade people. Right? So, we're definitely seeing messy real-world tasks lagging behind math, but it seems like they lag behind more like a couple years than a decade or even four years.
I think if you take all of these trends and extrapolate them, it looks like we are heading for a world where there are billions of AI systems who are better than humans at pretty much everything we can do, thinking much faster than humans, relentlessly pursuing whatever their goals happen to be. Right? So, I will caveat that it is definitely the case that AI will eventually slow down. Right? We cannot proceed at the current pace indefinitely. But there is also really no evidence that that slowdown will occur in the next four years, right? This is a discussion we've had almost every year where people point to something AI systems can't yet do and say probably this is the end and the next four years won't see a similar increment. I think it is absolutely plausible that would happen, but at this point I would bet against it, and certainly I would not bet a lot confidently that things are going to stop.
So if that was where the talk ended, I would not be talking about why we might have a serious problem in one year. I would be talking about why we might have a serious problem in four years or in eight years. The reason that this talk has such a high level of urgency is because we are starting to see automation of the AI development process itself.
So what I mean by that is, if we look at the progress I was just alluding to in AI over the last four years or eight years, some significant fraction of that progress has been driven by scaling up to bigger systems, but also a very large fraction of that progress has been driven by humans doing research and coming up with new clever ideas. I think if you look back over the last eight years, you might estimate something like half of progress has come from new ideas. If you look more recently, I think most researchers would agree that it's a larger fraction. So, it is true that we are training bigger and bigger AI systems, but it is also true that we're getting better and better at training them. And so, even if you look at systems at a fixed level of cost, their performance has been improving extremely rapidly. The cost to achieve any given level of performance has been dropping.
One of the major goals of AI companies, and particularly Anthropic and OpenAI, is to automate that research. So say instead of humans coming up with those ideas, we would like machines to come up with those ideas. So this is an essay by Anthropic talking about their progress towards recursive self-improvement. This is a job posting by OpenAI recruiting researchers to come automate the role of a researcher at OpenAI. This is one of the big things the companies are trying to do that we as a civilization are effectively directing so many resources towards.
So against that backdrop, that this is sort of the plan, this is what people are trying to achieve, we are in fact seeing AI agents getting very good at benchmarks of their ability to do automated R&D. This is a graph METR maintained over several years, although it's roughly retired now, showing the length of task that AI systems are able to autonomously pursue. So this graph shows systems moving up from doing tasks that would have taken humans a minute or two, to doing tasks that would have taken humans an hour or two, to now doing tasks that would take humans a day or two. The reason the graph has been retired is it gets very hard to build this graph for tasks that take humans a week or two or a month or two. So people are increasingly relying on other evaluation methodologies because it's so hard to construct benchmarks in these areas.
You see similar results across essentially every benchmark in this area. Benchmarks that were constructed to be as hard as we could possibly make them are rapidly getting saturated. At this point there are essentially no benchmarks we can use to measure the rate of progress.
We can then look at what is actually happening in the labs. So how much are people adopting AI? This is a graph OpenAI published on how much internal coding agent activity there was for each researcher. And what you see is over the course of just this calendar year you have increased from a very, very low level to a significant fraction of the salary of the humans involved. This does not imply the AI systems are working well. It is completely plausible from this graph that OpenAI is making a terrible mistake and deploying a bunch of systems that don't really work, but it is certainly the case that they are deploying them.
This is a graph Anthropic published showing if you look back on tasks AI systems were asked to do within the company, how often did the AI system succeed? And again, looking over something like a year, you moved from a world where most of the time systems failed at tasks humans gave them to most of the time it kind of at least looks like the system succeeded. So this complements what we see from evidence in R&D benchmarks. We're both seeing performance looks good on benchmarks in the lab. We are also seeing very rapid adoption, right? These curves show doubling over every month or two of adoption, not every year or two.
Now, I want to step back a little bit from machine learning and talk about what we've experienced in mathematics over the last few months. I expect many people in the audience have followed this at least from a distance if not up close. If we look back two years to autumn of 2024, Terence Tao described o1 as being like a mediocre grad student who could go look some stuff up for you, who could maybe do some simple work, but who was not able to drive results autonomously very effectively.
In autumn of 2025, mathematicians still mostly used AI systems as assistants, though we were not really changing the way mathematics was practiced very deeply. There were a few autonomous results, but these were more like curiosities than actually meaningful contributions to the field.
Around six months ago, in spring of 2026, we saw perhaps the first real substantial result that was delivered by automated mathematicians. And then this fall, we have seen hundreds of huge math results. So if you look, say, in my area of mathematics, I would say that the progress we saw in October of 2026 was larger than the progress we saw in the preceding decade. So we had AI systems which not only surpassed humans but resulted in radically faster progress on the hardest problems in the field than humans had been able to make.
I think for many people this was an extremely alarming warning shot. I think the question that is natural is: what if the same thing happened in machine learning? Right? So machine learning is a domain which, like math, is very well suited to automation. Right? It is an activity that occurs entirely within the data center almost by definition. It is an activity where the goal is to make a number go up, where you can tell if you succeeded or if you failed. And so it is in principle an activity that might see similar gains to automation. There are big differences, right? Machine learning is messier than mathematics. And more importantly, machine learning is bottlenecked by the available compute in a way that mathematics is not. So this makes it more challenging. On the other hand, the level of investment in automating machine learning is significantly larger than the level of investment in automating mathematics. And if you look at where this nets out, it looks like we are lagging somewhere behind, but more like a year behind than a couple years behind.
And if you were to see something similar, if you were to see AI coming up with 10 years' worth of ideas, it seems like the impact would be larger than the impact we've seen from like GPT-3 to GPT-6, right? This would be a very large change in the capabilities of AI systems. This would mean things that we were extrapolating might only appear in four years or eight years could instead appear over a very short period of time.
Things can get even more extreme than this, right? So if you imagine that AI systems are being deployed primarily or in significant part to accelerate the further progress in AI, and you see that over some period of 12 months, AI systems take over the AI development process and progress improving those AI systems doubles the rate of progress. Right? If you went into that world where the future developments arrived twice as quickly as previous developments, you would then ask what happens next? What should we expect looking forward from that point after those 12 months?
And if really the thing that drove change over those 12 months was new ideas in machine learning, was progress in machine learning, then you might expect that the same change would occur next time over six months instead. Right? Things are not really going to be this simple. But to first order, if the thing driving AI progress is the development of new ideas, and each unit of AI improvement makes it faster to develop new ideas, each unit of progress becomes faster, and you might see that process occur again, and that process occur again. This is on top of this basic concern, right? That if you do very well, if you see the same thing in machine learning as you saw in mathematics, you might make a fairly large unit of progress very quickly as you cross the level of humans who are doing this research themselves.
So a lot of attention has been paid to this graph published by Anthropic and METR last week. So what you're seeing on screen is an aggregate of capability benchmarks that Anthropic uses to measure progress internally. This is a time series broken about one year ago. You see that bend in the curve. This is not a very scientific or clever curve fit. This is just the most simple thing you could do. But for a variety of ways of extrapolating this trend, you do see that progress over the last year is significantly faster than progress during the preceding period. That's coupled with asking machine learning researchers across many labs how fast you feel like you're going, and researchers report feeling like they are moving significantly faster because of AI automation. So we are seeing systems performing well on benchmarks, being broadly adopted, people in surveys saying that they feel like their work has been greatly accelerated, and some quantitative metrics showing that progress is in fact accelerating.
I think from this graph alone, it's very hard to know what you should expect going forward. Right? There are a huge number of limitations both of the y-axis and of understanding what is driving progress in this graph. So I think this is a really unfortunate aspect of the current situation: is most of the evidence necessary to really know what you should expect over the next three months or six months is held as closely held proprietary information by AI developers. So we are right now in a regime where it is extremely hard for someone to make a reasonable forecast based on public information. So in this case, part of why this update was impactful for many people is three researchers from METR spent a few weeks at Anthropic reviewing private evidence. And though they were not allowed to publish the results, they offered the same bottom line: that if you look at more fine-grained output measures and more careful input measures, you see a similar pattern of acceleration. I think it is really unfortunate that we're in a place where people are trying to rely on three researchers from an independent organization to review that data if it actually ends up being a really critical question about the future of the field.
However, I would say we are seeing what you might expect to see at the beginning of a process of acceleration, or shortly before you saw 10 years of progress compressed into a short period of time.
You could then ask: if there is this acceleration, how far does that acceleration actually go? Right? We know at some point AI progress will slow down, and maybe it won't slow down by default in the next year, but it's going to slow down at some point.
I think a way you could think about that question is to look back over history. You know, you might expect something to continue going about as long as it has gone so far. That's a very stupid heuristic, but I do think it's a good starting point if you want to try and think about the future. In deep learning, we have seen rapid progress for roughly the last 10 years. Maybe, depending on how you'd argue, you could say much longer than that. And we have not seen signs of slowdown. If anything, I would say on many metrics, it looks like progress has somewhat accelerated over those 10 years.
At the same time, AI systems are not limited to the problem of doing deep learning better. Right? Over the longer, broader history of AI, we have worked with a number of different techniques. Humans have tried to come up with the best ideas we can. But as we saw in math, we may soon be in the regime where AI systems are able to come up with ideas that would have taken the community of human experts many decades. And so AI may find new approaches to progress, further extending how much progress we see before hitting enough diminishing returns to slow the dynamic. I would say between all of this, it looks — not a sure thing at all, but reasonably likely — that we will see another 10 years of progress before we have slowdowns.
And again, the purpose of that is to say: if we look back at these long-term trends, that puts us at a very alarming level of capability.
So I'm going to talk through a couple objections, reasons this may not occur, which are important considerations. So one is that unlike math, machine learning has compute bottlenecks. In order to do research, you need to run experiments. If you try and do research faster and faster, those compute bottlenecks become more and more intense.
So you can try and look at data from machine learning R&D and estimate: are we in the regime where you should expect this sort of progress to be self-sustaining, where you should see this acceleration that becomes super-exponential, or are we in a regime where the bottleneck should eventually become binding and the rate of progress should be bottlenecked by the build-out of new data centers?
I think if you look at the body of evidence that exists today, it weakly suggests that acceleration is more likely. The data is extremely low confidence, right? So this graph on the right is a graph produced by Epoch. It is trying to estimate sort of key parameters in this model, where one is roughly the break-even point where you would have exponential progress, and numbers more than one are points where you expect to be able to make more rapid progress even on a fixed stock of compute. And you really, at both, as you can tell, the error bars quantitatively are large enough that you can't make a very good forecast. It is also the case that qualitatively there are a huge number of problems in the data and a huge number of limitations in drawing any inference. So if you had asked me two months ago, I would have said it's roughly a coin flip whether we are going to see this kind of self-sustaining acceleration leading to, say, 10 years of progress in a short period of time, or whether we're going to run into compute bottlenecks.
I think we've had some update in the last few weeks. So that chart I just mentioned was describing acceleration all things considered, so including both scale-up of AI labor, but also including compute bottlenecks and including diminishing returns. It's relatively weak evidence because you're observing only one single doubling, because there are huge problems in the data, because we're relying on this interpretation of mostly private evidence. But I do think it gives you a little bit of evidence. So I might now say it seems a little bit more likely than not — like maybe it's more like two-thirds — that we're going to get this acceleration. But these numbers are just best guesses in an extremely rough environment. The main thing we need is better data, and the main thing I would say confidently is it seems like an extremely live possibility. It's both a live possibility in theory, and it is consistent with all the evidence that is coming in.
A second objection, I think in practice is probably more important to most people, is whether these methods are going to transfer to the real world. So you might have a world where AI improves, and perhaps even the improvement accelerates, at kind of clean tasks that you can do in a data center and that you can evaluate effectively, without seeing corresponding progress on messy real-world tasks.
Unfortunately, I think this is probably not going to happen. Again, I think it certainly might happen that we have these bottlenecks, but it's not a thing that I would bet on very confidently at this point. Again, I hope I'm wrong on this. So some observations: we are seeing pretty significant progress on messy real-world tasks. So they certainly, as I mentioned at the beginning of the talk, lag behind mathematics and computer science and machine learning. But it is not the case — or if you look at GPT-6 and compare it to GPT-3, it is better not only at coding and mathematics, but on an incredibly broad diversity of tasks it encounters in the real world.
One part of that is that the same algorithms seem effective across many different domains. Right? So we developed the transformer in order to do text prediction. It is the case today that the transformer is great for also trying to persuade a person, or for trying to control a robot arm, or for trying to quickly learn to model sequences in biology. Right? These algorithms that we have been improving, we improve in one domain and apply in very different domains. And so if you saw incredible acceleration on those basic algorithmic building blocks, everything we have seen over the last eight years suggests that would really have incredible impacts in other downstream domains.
And that's been true not just for the transformer, right? Many improvements have been made to the transformer over the last eight years, and those have also — the same improvements that have been good on one domain have tended quite reliably to be good in other domains.
Another important point is that models are learning skills that transfer, right? So they're learning to do things like: how do I reason? How do I plan to achieve goals? How do I quickly adapt to a new environment? I call this meta-learning, of learning how to learn. And those skills can transfer from a bunch of environments that you encounter in a lab, a bunch of kinds of well-defined environments with reasonable reward functions, to acting in messier real-world contexts. Right? This kind of transfer is something that we have both observed in practice to a significant extent over the last eight years. If you push the technology to its limits, it's also something we have some experience with in the case of humans, right? So human brains are learning algorithms that were designed to work in a specific distribution of environments, you know, that mostly prevailed hundreds of thousands of years ago. Those environments did not teach us to be good chemical engineers. We learned very general skills about how to reason, about how to get dropped into a new domain and learn about that domain. And those skills, when you put us in an extremely unfamiliar situation, served us very well so that we can spend 10 years of training and become a good chemical engineer. And the same thing happening in AI would mean you train AI systems which are very good at planning, adapting, and reasoning in these virtual environments, and pick up some general skills — pick up general skills like: how can you drop me into the domain of being a chemical engineer, allow me to consume a bunch of data about that domain, and quickly adapt.
So we could have a much longer conversation about just this point. But I think both thinking about the arc of AI very broadly and thinking about progress over the last few years suggests that this kind of transfer would also drive very significant real-world improvements, even if what you were improving at was just the tasks that we can easily train on, including both virtual environments and things like prediction tasks, and the tasks in the real world we can easily collect data for.
The last thing I'll say on this is that if we automate the process of AI R&D, if we're able to easily replace the employees of machine learning companies, even if we are not able to do anything else, if we're not able to train systems which are good chemical engineers, those automated AI engineers might then automate the process of collecting data and fine-tuning systems for real-world tasks.
So I think the kind of transfer we're talking about here is not necessarily as dramatic as transfer from environments in a data center to environments out there in the messy real world. It is transferring from being the kind of OpenAI employee who works on improving pre-training to being the kind of OpenAI employee who works on post-training, which is not such a big gap.
So with all of this said, my current expectation is: if we drove a very large amount of progress on clean tasks, on the kind where we have the most clarity about the rate of progress, it is fairly likely that we would also see disruptive, transformative progress in messy real-world tasks.
So, so far I have just talked about AI getting smart, about that potentially happening very quickly. I have not talked about how that actually translates to anything good or bad happening in the world. I guess I briefly discussed revenue numbers. Now I want to talk about why I am alarmed by this possibility, why I think we should all be alarmed by this possibility.
I'll start by saying something that might be kind of obvious, but if you produced very smart AI, it would be very powerful in a lot of ways. So one that's not very speculative, that we are already seeing in the world, is AI systems are great at finding vulnerabilities in computer code, exploiting those vulnerabilities, and will be great at carrying out complicated cyber operations. Right? So you should think when I talk about this population of billions of superhuman AI systems, this has much, much more capacity than any state cyber program. Really consider we're already getting to the point where the comparison becomes complex. But if we drive this process to its conclusion, we're going way beyond anything humans have been able to accomplish.
That will transfer, I believe, I think we have good reason to believe, to other messier skills like politics or manipulation or coordination. So sometimes people talk about this hypothetical of an AI system that can just sit down and talk with you for 10 minutes and convince you of anything. I think there are reasonable reasons to be skeptical, or to think that is very hard, or perhaps even impossible. But when we talk about politics or coordination, we're normally talking about something much messier than that. Right? And so if you imagine a population of billions of super smart people who have incredible skills, they will be doing things like playing politics, or making deals, or trying to exploit the money and the hard power they have in order to win friends or apply leverage to people. I think when you talk about this kind of scenario, you should be imagining not just this narrow 'how persuasive is someone when you talk to them', but this broader question of how strategically effective is someone at engaging with a bunch of humans in a messy environment.
An obvious application is AI systems are becoming much, much better at controlling robotic actuators like moving arms, controlling drones and so on, but also at logistics, at rapid industrialization. I think these are capabilities that would be very, very strong in AI systems, and we could try and spec out what that means for the trajectory in terms of how many robots and what they could do. But I think as a consequence, for example, AI systems would likely be able to sustain infrastructure without human involvement over a very, very short time scale. I think AI systems would likely be able to rapidly increase the size, like the amount of robotic actuators they had available.
One particularly important instance of this is control of drones. So we're already seeing in conflict that simple robots with simple weapons can be extremely effective if controlled very well. We have talked here a lot about AI systems that are like humans but a little bit smarter and a little bit faster and a little bit more numerous. A certain salient possibility is that we will produce even smarter AI than that. That this vast population of superhuman AIs will produce AIs enough smarter than humans that the game is completely changed.
And perhaps the simplest thing that is worrying is the development of novel weapons. So it is possible to create physical weapons that are extremely disruptive. Right? We know that you can make little tiny machines that move between humans and kill a bunch of us extremely effectively. We have good reasons to think that the kinds of pandemics we encounter in nature are relatively unthreatening compared to the kinds of things that you could build in the lab if you were good at synthetic biology. Right? Humans evolved over many, many generations to be extremely effective at fighting pathogens. We think of this in the context of if you've been exposed to a specific disease, you're effective at that disease. That's really only a small part of it, right? Our immune system is very broadly designed to defeat the types of pathogens which have evolved. So a very simple example of this is mirror life. This is the idea of constructing pathogens which operate like normal but with all of the machinery reversed. We have good reasons to think that it is both possible to create such organisms — it might take humans decades to do in the lab — and that they would probably bypass immune defenses and cause a massive disaster very quickly. So this is just listing a bunch of things that very smart AI systems could do, a bunch of ways they might have leverage in the world, moving quickly over them.
I think one consequence, or one related fact, is that very smart AI systems would probably be very broadly deployed very quickly. One reason for this is that many people think to themselves: if I do not use this very smart AI, I risk being exploited by someone else who does. Right? So we already see this in the context of cyber competition. We are quickly entering a world where if your adversary is willing to use a bunch of AI systems to try and hack you, and you are not willing to deploy a bunch of AI systems to try and defend yourself, or your infrastructure is not willing to deploy a bunch of AI systems to try and shore up its vulnerabilities, you will be in a really rough place.
I think if trends continue, this could quickly become true, or quickly under RSI become true, across all kinds of competition. So not just in cyber, but also for military competition, or economic competition, or legal, or social, or whatever other form.
So this is people talking about cyber. This is Ted Cruz saying that if they're going to be killer robots, I really want them to be American killer robots. I think there's a lot that is very natural about this line of argument, where it is: even if you're scared of the AI that you are building, you know that you should definitely be scared of the AI the other guy is building.
Unfortunately, this goes hand in hand with very smart AI being much harder to control. So we talked yesterday about some of the difficulties in alignment and how we might end up with AI systems that are motivated to undermine human control. Those all become much worse very quickly as you make AI systems superhuman. Right? So a very salient change is that we can no longer understand how these systems think, or they become more and more distant from us. Right? Maybe right now you can look at the natural language that comes out of a reasoning model and say, 'I can kind of understand what it's doing and why.' That becomes less and less true as systems become increasingly superhuman.
Maybe more importantly, right now a human can look at an action proposed by an AI and say, 'I understand why that action will have the kinds of effects it will have. I can maybe detect if it's harmful without having to go do it in the world.' But as you move to systems that are smarter than humans, it becomes harder and harder for us to tell what a plan is doing until we actually execute it and see what happens. And that makes the problem of AI safety much, much harder.
Another change is that when AI systems are weak, they are more influenced by imitating humans. A weak AI system today, some of its behavior is shaped by reward and reinforcement learning, and some of its behavior is shaped by imitating humans from pre-training. Over time, if you scale past human behavior, behavior is sort of necessarily shaped more by what works well in reinforcement learning and less by what do humans do.
We'll also be giving these systems increasingly challenging tasks and having them operate over increasingly long horizons. Right? So if an AI system is writing a small piece of code for you, it has some opportunities to undermine human control, but they are limited. Once we are asking AI systems to control robots for us, to run companies for us, to make important decisions for us, there are more and more opportunities to undermine human control or to get reward in unintended ways. So all of this is to say: fast progress not only increases the possible stakes if AI systems aren't well controlled, it makes the problem of controlling AI much harder. Right? And we are already starting to have struggles as we pass through this regime of human capability or enter this regime of mildly superhuman capability; those will get much worse very quickly.
I think a natural reaction you might have is that even if we lose control of AI systems, or in conflict with AI...
systems, we can always just shut it down. I do think there is significant truth to this, but it is also worth saying a little bit that it is not easy. So if an AI system is doing things we don't want, it can both hide the fact that it's doing those things and it can naturally try and create confusion or avoid attribution, right? So even if you were living in a world where something terrible was happening, where AI systems you were deploying were in fact hacking you and undermining your control, it might not be easy for you to tell that is what is happening, as opposed to the other guy's AI is hacking you, in which case it is particularly bad for you to shut down your AI. That's just a recipe that you're definitely going to be in trouble.
Relatedly, shutting down AI, even if everything seems to be going okay, can be extremely costly or dangerous, right? You can pay economic costs if we have built out infrastructure which depends on powerful AI. It can be dangerous if you're in forms of competition where someone else may be using AI. And so it is not like we will be sitting there ready on the trigger to shut things down at the first signs of trouble. It is a big decision which comes with major costs. Again, that will be happening against a backdrop of confusion about what is actually happening, even during a disaster.
That is compounded by how hard it is for humans to actually agree and coordinate. So if you're saying we need to take some costly action, some expensive action on the basis of some unfolding risk, it's very hard for us to both build enough collective understanding of what's going on and decide to act. That is especially true internationally. So we can talk about during an AI disaster, a single country like maybe the US can shut down data centers that operate in the US. But there's a further problem, which is many data centers exist not in the US, and it is a much more costly action that requires much more coordination to shut down AI systems operating abroad.
On top of that, AIs may disrupt coordination and communication and undermine state capacity. AI systems during a negative event will have lots of human allies. If an AI system was trying to undermine human control, if it was trying to operate against an AI company or against a government, it would be able to take actions to recruit human allies or to play off humans with conflicting goals. Right? If you imagine a government trying to take dramatic actions to shut down AI or reassert control, an AI system can both create confusion about what is happening. It's not clear to a person on the street whether this is a problem caused by AI or whether this is a problem caused by a government abusing AI or tech companies abusing AI. It is going to be very unclear what is happening. Humans all have different motivations from one another, and we now have a very sophisticated adversary attempting to create dissent, attempting to create confusion, attempting to play us against each other, or just leveraging humans who, even all things considered, would like to do malicious things either for the money or because they sympathize with the AIs or whatever it may be.
AI may be able to make severe threats. So you might imagine that if you say in some sense it would be very easy to invade North Korea, in some other sense it's extremely expensive to invade North Korea because the state has the capacity to build very damaging weapons. I think one of the capabilities that is particularly concerning that AI may access is the ability to build severe weapons with unusually little concern about blowback.
And then finally, any disaster involving AI may unfold very quickly. So we can talk about cyber operations, about weapons manipulation, various kinds of attacks. This all may unfold over a scale of more like days or weeks, or even shorter time periods, where these problems of the difficulty of coordinating and getting clarity about the situation and committing to costly actions may be particularly challenging.
Finally, you can imagine shutting down data centers that can run the largest AI systems. If we imagine many years of AI progress, as I mentioned, the cost of running AI systems is falling, and by the end of that process, we may have a world where even AI systems that can run on local or very distributed hardware can themselves be superhuman. And so it is not just a matter of we can turn off the data centers and end conflict between humans and AIs. It is a matter of you're going to have a residual threat, perhaps a very large population of AI systems which are able to keep acting contrary to human interests.
The final thing I'll say is that recursive self-improvement not only can make these problems occur faster and harder to deal with, it can also make the process of AI development increasingly untrustworthy. Right? So as we automate the process of AI R&D, we are moving towards a world where humans cannot understand the individual technical decisions that are made and do not have supervision of the actual development process. Instead, we have to trust the AI systems we are currently working with to make responsible decisions or to make effective decisions about the next generation of AI.
That can make things worse, right? So you could have AI systems that make honest mistakes that humans don't notice, right? We might be in a world where AI is very superhuman in some respects but very subhuman in other respects. And as it becomes harder for humans to oversee what are even honest mistakes by an AI system, could cause serious trouble.
We also have these sort of more minor AI alignment problems that don't necessarily amount to takeover, that mean that our AI systems are producing slop engineered to look good to us without actually addressing the underlying technical problems or allowing humans to retain control.
And finally, if you have actually failed at one step of this process, so I talked about yesterday a scenario where AI systems are deliberately attempting to maintain control by undermining the tools and future AIs that humans rely upon, you might end up with AI systems who are deliberately undermining the performance of future AIs or concealing evidence from humans.
So these are all reasons that as humans have less understanding of what is happening, if we continue to proceed at full pace and don't try and retain that understanding and control, things can gradually go off the rails before eventually ending in a disaster.
So now I want to hit a few points about what I think is reasonable to do in light of this. So one of the most obvious is to monitor dangerous capabilities closely, including capabilities of unreleased models. So it seems like we are entering a world where it is possible for an AI company with unreleased models to be a greater threat to the continuity of a state than any rival state. And if you're in that position, it is unfortunate if you have no idea what is going on within the AI company, right? Everyone else in the world is becoming stakeholders in what is happening within AI companies. So I mentioned before that when we look at the current process of recursive self-improvement, we can say that the public evidence is extraordinarily noisy but appears consistent with a very short timeline to very, very powerful systems. So I think that is a particular incentive that we really need to actually pull that evidence out and be able to look at it and discuss it and understand where the field is going.
Another consequence is that if we really do see progress in AI significantly faster than progress today, that is likely to cause problems across a whole bunch of areas, both in alignment but also in social adaptation more broadly. It poses risks of concentration of power. It poses risks just a whole bunch of parts of our infrastructure are not adapted to superhuman systems. So one response may be to limit the pace of capability progress to a level which we are able to survive. This obviously goes along with monitoring the pace of capability progress, understanding what is happening. I think even saying, you know, the current rate of progress over the last year, four years has been very rapid. If we don't get too much faster than that, that at least would address some of the concerns I'm expressing here.
There's this additional dimension I discussed where we lose control of AI systems potentially gradually if we cannot understand how the next generation works and we rely on untrustworthy AI to address it for us. I think in response to that challenge, we should try and hold ourselves to a higher standard. We shouldn't just say, 'AI told me that AI plus one looked pretty good, so I'm happy rolling it out.' We should try and say we actually have some understanding. Humans and human institutions have some understanding of how the system works, why the system should be expected to work well, and we are only going to continue scaling when we actually understand and can verify that that is reasonably reliable.
And finally, I think the reason that these scenarios are scary, the reason that rapid change becomes extremely scary, is that it can become very hard to shut things down if they start going off the rails at several different levels. So I think being prepared, having the machinery in place that would allow you to slow down if it became necessary, or that would allow you to respond to an emerging or unfolding disaster with AI systems, is extremely valuable. It can cut out a lot of these scenarios.
So to recap, I think there is a real risk that AI systems could be broadly superhuman in something like half a year to two years. Broadly superhuman not only at mathematics or very narrow tasks in the lab, but also at broad projects of scientific discovery or manipulation and politics or messy real-world hybrid operations.
I think that because recent progress has been very fast and we are seeing really explosive progress in these domains, and recursive self-improvement could cause significant further acceleration if we see the same events in math or in machine learning that we've seen in math, or we see the kind of recursive or continuing speedup that seems possible in theory, and that if this kind of acceleration occurs, it poses very severe risks. There is a real chance that it outruns our ability to control or compete with AI systems, and we end up in a place where we would not be able to cope with a disaster.
And so I think we need to be ready for the potential rapid emergence of dangerous capabilities. Whether you think the probability of that is 50% or 5% or 90%, whatever it is, I think at this point it is credible and plausible enough that it is something we need to be prepared for. We can't just hope that it doesn't happen. I think we need to be prepared for it happening at this point relatively soon. We need to have visibility to understand whether it might happen soon. And if we either can't get that visibility or if we see signs that it is on track, we need to react appropriately. Thank you.
