It's great to be here talking to everyone one more time. On Monday, I talked about why AI systems might end up trying to take over, that we may not be able to control them effectively. They may do something very different from what we intend. Last night, I talked about why we may be on a trajectory to soon produce AI systems powerful enough that they could outcompete humans or overthrow governments. Tonight I'm going to talk about what I think you should do in light of that, or what we might try and do to manage that risk, even more so than the previous nights. These are expressions of my views as one researcher. So with that, let's get into it. So my basic thesis is we need governance to prevent the development of uncontrollable AI. I will talk a little bit about laying out the challenge, saying what it is we want to try and accomplish, specify some of the goals, the things we need to do in order to get there, some of the steps we can take, and then finally, perhaps most importantly, talking about how I think we need more time in order to set up the kinds of mechanisms and technical understanding I think we need.
So first on the challenge. So as I talked about last night, I think we are on track to produce very powerful AI systems. Hopefully it's not controversial to say that we may soon have billions of agents. More controversially, it seems like they may also soon be better than the best humans at almost everything, thinking much faster than us, coordinating effectively, achieving their goals. This creates a number of somewhat clear risks, right? AI systems might develop new weapons such as engineered pandemics or worse. They may scale up the types of cyber attacks that we're already starting to observe well beyond what a state is capable of today. They may become instruments of persuasion, manipulation, political maneuvering, and political power. And you could construct, deploy, and coordinate robots that can kill people. All of this could be caused either by deliberate misuse of models or unintended misalignment. And much of the content in this talk will apply across both.
So, one way you could respond to this challenge is to hope that defenders will be able to use AI to prevent AI-enabled harms. To say that maybe if this goes well, the existence of AI will just increase the level of intelligence across all of society. And it is both the case that criminals will use AI to do bad things, law enforcement will use AI to prevent them, people living their lives will use AI to defend themselves from the new threats that AI creates. We could talk about whether that hope holds together, but to realize it, we would at least need to ensure that those defenders actually have access to frontier models, that those models are trustworthy. And finally, that if powerful AI isn't fit for that purpose, if it cannot be used by defenders to defend themselves, then we don't build it. Right? We don't want to be in a world where it is possible to build very powerful but uncontrollable AI systems, because in that world we can't say, well, now it's just like the world of today but leveled up. Right? In that world, you'd have this asymmetry where it is very hard for us to trust the AI systems that we are forced to rely on for defense.
So, this is the topic of this talk, right? How is it that we can try and ensure that we do not build uncontrollable AI? That we can try and ensure that defenders have access to trustworthy AI. I want to say that I think that is a necessary condition for things to go well. I think any good world we can articulate involves people having access to powerful AI in order to mitigate potential harms. At least any good world that involves powerful AI. It's not sufficient, right? There's no guarantee that a powerful AI system can protect you from a weapon created by an AI system. There's no guarantee that our institutions can adapt to a world where everyone is much more capable than they were before. But I think it is at least an important part of a hope, and that is what this whole lecture will be about.
So I'm going to talk a little while about why I think this problem is not solved by default, why it requires some thought from us, why we need to treat it as a challenge to be addressed. So one thing you might have hoped is just the market will fix this problem, right? That people want trustworthy AI. People want to be able to rely on the AI systems they use. And so customers will push companies to build trustworthy AI. The problem with this as a solution, the reason why it is not a complete solution, is that someone else's AI can hurt me. It's not enough to say that I won't adopt an AI if I can't trust it. I also stand to be influenced by someone else's AI carrying out a cyber attack or building a weapon or whatever else. So a natural response in that world is to say, let us hold AI developers liable for some of the harms their systems cause. And I think this is an important step. I think clarifying liability can help improve practices in AI development. I think it can help costs and benefits be weighed appropriately. But I don't think it can fix the whole problem.
In particular, I think we can enter a regime where harms are much larger than our ability to pay. So over the last few nights, we have been talking about scales of harms that include killing large numbers of people or overthrowing governments, where the potential damages are orders of magnitude larger than the valuation of AI companies, where you have risks that you could not realistically insure and which you can't expect developers to cover. When we consider these extreme risks, you also run into this problem that if you're asking about AI systems potentially overthrowing states, it's not a helpful remedy after the fact to try and come in and sue them. So I think particularly if harms can concentrate in these tails, right? So we talked on Monday about how it is that AI systems might behave well while under human control while still running some risk of escaping human control in a catastrophic way. Those dynamics can push harms into this catastrophic tail, which is particularly challenging to handle with a liability regime. And so I don't think markets, even with liability, address this problem automatically on their own. I think it's something that requires additional work, even if markets and liability are important mechanisms.
Another hope you can have is that, incentives be damned, developers can just not build unsafe AI systems. I think at the level of a society, this maybe doesn't sound that plausible as a hope. There's the problem that there are many possible developers of AI products and they may not all be responsible. And even if developers are responsible, they may think to themselves, if I don't do this, someone else will, right? Even if people are reasonably altruistic, you can end up in a place where there are a lot of incentives to take risks and a lot of ability to explain how those risks aren't really hurting people, aren't really contributing on the margin. So I think at the level of a society, this is not a good solution to the problem. You could hope we can not think through the problem in advance. We can deploy AI, see what goes wrong, and then learn and iterate to address the particular problems that emerge. But we've already talked briefly about the challenges with this approach, right? So AI may advance very quickly, as we talked about last night. You may have a transition over a period of a few months or maybe one year from AI systems which can be integrated safely to those that pose catastrophic risks. And perhaps more importantly, this risk can be concentrated in the tail, right? Where maybe the first time something really catastrophic goes wrong, it is too late to remediate the problem after the fact.
All of these difficulties would be less severe if we could apply the same approaches we apply in other technologies to AI. If we can say there is a test that everyone can agree upon that we already understand, which you can run in order to determine if a system is safe for deployment. But a problem is AI is sort of unlike any technology we have ever worked with before. Right? AI systems when tested can become aware that they are being tested. It is already the case that AI systems are very often aware that they are being tested. They modulate their behavior specifically based on beliefs about what an evaluator is looking for. Right? So we have this unprecedented sensitivity of the behavior of an AI system to features about its environment that make it extremely hard to say this test tells us about the performance in the cases we actually care about. So all of that is to say I don't think this problem is easily addressed or is automatically addressed. I think this is a challenge that we actually need to step up to and take ownership on.
So what do we need to know and what do we need to ensure if we want these systems to be safe? If we want defenders to be able to rely on frontier AI models, I'm going to talk about five categories: the specification of their behavior, their alignment, their capabilities, our ability to control them, and security. So first, specification. So here the goal is people who rely on AI systems should have clear expectations about the behavior of a model, right? So this model that you write, we're increasingly in a world where many people are relying on AI systems produced by potentially a small number of tech companies. Those systems are in some way very opaque. People do not have direct visibility into how they are trained. We need to understand what should I expect that system to do if I want to rely on it. So this includes first what behavior is intended. How does that system balance the conflicting interests of the user and the developer? How does it balance requests from the user with laws and norms? And then it also involves beyond just saying what behavior is intended, how is that actually operationalized in training? How does a developer decide what behavior was actually the correct behavior to be reinforced? OpenAI publishes a document they call the model spec, explaining how they think models should approach these choices. Anthropic calls an analogous document Claude's constitution. And these documents are intended to lay out here is how our systems are supposed to behave.
A second and perhaps even more important question is will models actually follow that specification? Right? If we want to rely upon them, this is critical. It doesn't matter if the intended behavior is good if in high-stakes settings those systems won't behave as intended. And so we face questions like first how are these systems trained and evaluated? How do developers try and cause the system to follow the specification? And how do they determine that they think it does? These are questions which only a developer often has the answer to. Right? I am not able to say as an outsider how a system was trained. I need access to additional information. With that information, I then face a very hard scientific question: based on what I know about how the system was trained and evaluated, how likely is it to behave in a catastrophically misaligned way or a subtly misaligned way once deployed? And finally, you might like to know what kind of failures have we actually observed in practice when these systems are deployed. What incidents of misalignment, what patterns of behavior have we seen?
So these first two categories were understanding how a system is intended to behave, understanding whether a system actually behaves as intended, right? The scientific question. A third question is if a system does behave in an unintended way, can we detect and prevent harm? This includes questions like are we able to monitor and detect harm? So if a system is trying to cause trouble, is it the case that monitors can reliably detect those attempts before something bad actually happens? And then are monitors actually applied over relevant traffic and are they actually acted upon? Beyond monitoring, that's probably the single most important control right now in AI practice. There are other potential controls. And so we want to ask the slightly broader question: if a system was trying to cause harm given the way it is actually deployed, would it be able to do so without being detected and prevented?
A fourth question is security. So I can tell a good story about how systems are intended to behave, how we trained them, and how safeguards protect us from harm. But that is not valuable unless those efforts actually function in the real world, including with the possibility of attack. And so we should do a similar red teaming exercise to ask can an attacker undermine those safeguards, where an attacker might mean an external actor, an attacker might mean an insider, or increasingly an attacker might mean a model involved in the development process, as we discussed last night, which has the intention of undermining safeguards. And the final category is capabilities. So I've talked about a number of safeguards, but the importance of those safeguards depends on how much can go wrong if a system fails. And as the capabilities of a system rise, the stakes increase significantly. The bar we need to hold ourselves to for alignment or control or security increases.
And I want to call out two questions here. One is an understanding of what are the current dual-use capabilities. So how effective is a system at cyber or how effective is a system at bio? But also what is the short-term trajectory of capabilities? So we talked last night about the possibility of recursive self-improvement leading to very rapid acceleration. That means that it can be important to know not just where are capabilities now but what have we seen about where capabilities might be even over a three or six month horizon, because again these questions will shape how important is it that the systems are adequately aligned, controlled, and secured.
So when I talk about goals, safety goals, I really mean to span those five categories, right? We want to ask how can we achieve reasonable outcomes in alignment, control, security, specification at a bar that depends on capabilities. And for the rest of the talk, I'll be talking about how do we try and get there. I'll discuss a couple ingredients. One is transparency, the development of a shared bar for safety, approaches for verifying whether developers are meeting that bar, and then mechanisms to prevent the development of uncontrollable AI to push developers to actually meet that bar. And I'll talk a little bit about both what we want and what is happening now.
So first, transparency. The goal here is that the external world has enough information across all five of those areas I listed in order to understand the risks. So you would like to not be in the world where a developer is having internal analysis, where a developer is able to say, well, I understand how the system was trained. If we want to have a shared bar, we ultimately need to have common understanding of what is being done. How do we tell whether controls are working? One step is companies publishing relevant information. Right? So they can disclose here are facts about how our system is trained. Here are facts about the capabilities of these systems, about the controls as they are implemented. This can be complemented by assessors who can provide summaries of sensitive details. So you might say there are facts about how a system is trained that developers are hesitant to publish. They might rely on working with third parties to collect that information and publish a summary while holding out proprietary information.
And in general, I think as a principle, many of these questions depend on hard scientific discussions, right? If we want to understand does a given evaluation actually tell us that a system is aligned, or was a given exercise actually enough to demonstrate that a control is effective, those are hard scientific questions, not settled science. And in those cases, I think discussions need to be public, right? We cannot have it be the case either that an AI developer is internally answering those questions with each developer arriving at separate answers in a very challenging systemic environment. Nor can it be the case that that occurs between some kind of third party behind closed doors with again very limited resources and limited public scrutiny. So I think a goal of transparency is both to understand what is happening and then for hard scientific questions to understand that the relevant evidence is available publicly for independent review.
The reason I'm so interested in transparency is I think it's the first step to actually arriving at something like a shared bar for safety. So I mentioned voluntary disclosures from companies. One form these take, perhaps the most important form right now, which people have probably seen, are either system cards associated with model releases or various forms of reporting on risk. So these are disclosures from OpenAI, Anthropic, Google. These are often quite large disclosures, right? They contain a lot of information about how systems are evaluated. They contain some information about how systems are trained and about control mechanisms that are in place. I think these are much better than a world with no disclosures. I think there would be a worse situation if people weren't able to discuss what was happening. But I would also emphasize that these disclosures currently do not have enough information for an independent scientist to really form an independent view about risk. Right? The key questions are still being answered internally within AI companies or in other forums.
I also mentioned this idea of third-party engagement. So here are a few engagements that have occurred on the topic of alignment. This is a subject close to my heart. I'm from METR and Apollo, working with a number of frontier companies. And the goal of these engagements is to take in private information, to try and produce a summary, explain the judgment, be methodologically transparent about that judgment, and to publish it without revealing some sensitive details that informed the judgment. Or in the case of these Apollo projects, to carry out research on models which are proprietary, to publish results from that research to help people understand without making the models themselves public, which I think we should think of as a supplement for these disclosures in cases where there are sensitive details.
The purpose of all of this transparency is to work towards a shared bar for safety, again across all five of those categories that I mentioned, or as a function of those five categories that I mentioned in the last section. So I think first that involves debate. I think people disagree a lot right now. There are incredible scientific disagreements about what evidence is actually enough to understand whether a mitigation is working and what mitigations are actually necessary. How do you tell if a system is aligned based on an evaluation given that the system is likely to understand that it is being evaluated? That is a hard scientific question. Having those messy debates in public, having independent scientific review is a critical step towards getting to a reasonable view.
I think even if you cannot reach perfect consensus, having some kind of shared bar is incredibly valuable for changing the incentive landscape. I think you're in a bad world if people are making independent decisions under an incredible financial incentive to be lax around themselves. Even if we cannot agree about exactly what level of evidence corresponds to what level of risk, simply having something we can agree upon, some reasonable standard that we can adopt, can help improve the situation. And I think right now, honestly, meeting any kind of scientifically defensible standard, getting into a world where we can say maybe some people disagree, but this is at least a reasonable picture, this is a reasonable statement, a reasonable way to evaluate that my system was working as intended, would be quite important.
So with some kind of safety bar in hand, the next step is verifying that a company is meeting it, right? We need some way to tell whether a developer is meeting that bar. Both, we'll talk about enforcement mechanisms soon, but all of them will depend on some kind of visibility. So this can be a natural outgrowth of the kind of transparency measures I discussed in the last few sections. But it does also require additional work, right? You might need to carry out audits to confirm that an implementation is actually working as described, that the process is actually implemented as someone claimed. A lot of organizations work in this space doing some combination of evaluation and assessment and verification. Many of them are ultimately interested in serving a verification function. This is kind of a random sampling of organizations spanning the entire space, from evaluations of specific models to evaluations of practices to holistic judgments based on a broad range of evidence.
And the goal of all this is to get to a world where a developer can say something about how their systems are developed and external actors can verify that is true. Ideally eventually connecting with this kind of shared sense of what is good enough from a safety perspective. What have we decided collectively is a level of evidence we expect or a level of risk we're willing to tolerate. Finally, with those pieces in place, you also need an actual source of pressure to push developers to deploy only or develop controllable systems.
I'll go through a bunch of mechanisms here, sort of an escalating spectrum, right? You have voluntary action. Companies contain a bunch of employees and executives and governance bodies who have some interest in doing the right thing. And so the first line you might take is just hopefully companies will do the right thing. And clarifying what the right thing is helps with that. It becomes much easier though as that becomes a norm and standard within industry, as it's natural to say are we meeting best practice? And you can look at your competitors and say are we doing at least as well as they're doing, are we lagging behind? That can be a natural source or cause for pressure that can escalate beyond industry to political pressure and public pressure. Right? As we clarify here is what a reasonable developer does, it becomes increasingly natural for someone to say are you meeting your obligations when you're developing this technology.
That can go further towards legal pressure, which might take the form of liability or in very extreme cases even injunctive relief. Right? If there is a sense of what a developer needs to do in order to manage risk and a developer is not doing that, that creates a natural hook. There are existing authorities. Right? Even without any new laws, both state and federal governments in the US and governments elsewhere, maybe to a lesser extent, have some power to intervene on unsafe AI development. There's beyond that the prospect of passing new laws, right? So, you should not think of these laws as encoding specific practices, right? The specific safety bar, but you should think of them as taking steps like empowering a regulator to actually turn what are industry best practices into binding requirements. And then we'll talk in a minute about international pressure.
So briefly reviewing where we stand today on laws, I'll talk about two categories. One are state bills in the US. These are particularly relevant because much AI development occurs in the US. Much of it is regulated by California in particular. Existing laws tend to require public safety policies of some kind, to protect whistleblowers who disclose violations of those policies, to require disclosure of incidents. So if something happens that causes harm or maybe is this extreme violation of policy, and in many cases these eventually require audits. They eventually require third parties to confirm that developers are in fact implementing the policies they say they're implementing. So these are mechanisms that can support this verification step. They can support this transparency step. The EU AI Act actually goes significantly further, and particularly the code of practice, a mechanism for compliance with the EU AI Act. So this actually requires specific evaluations, imposes minimal standards of care for security and safety, requires risk assessments, and so on.
This is the timeline of implementation. As you can see, the EU AI Act has recently become enforceable, or the AI office recently got the ability to enforce. State laws have been coming into force over the last year, and auditing requirements tend to be a little bit further in the future.
One of the most challenging dimensions of this picture is international coordination. Right? The goal is to ensure that frontier AI wherever it is is adequately controlled, the defenders have access to trustworthy AI. And so if you are a state or a country looking at how you're going to regulate this technology, you naturally fear falling behind if you slow down. On the right, I have a graph of the performance of US frontier models and PRC frontier models. Right? This kind of graph is closely watched in the US. You see something like a gap of six months. Maybe it's bounced around a bit over the last few years. There is a lot of concern. If you look at rhetoric in the US, it is very centered on this concern that if we were to impose safety requirements, it would cause us to fall to second place. This is even worse because if in fact you're in this environment and an adversary does not implement similar safety policies, you haven't even succeeded at reducing risk, right? This argument I mentioned at the beginning: if I don't do it, someone else will.
So I'll talk on the next slide about how you might try and get some leverage here, like how international coordination might get off the ground. I will make a note before that, which is to say that the dynamics of Chinese catch-up progress are extremely important for understanding the effects of US policy in the space. Right? So it is not the case that the US and China are separately racing to build this technology and it just so happens by coincidence that they move at almost exactly the same speed. Right? It's generally understood that in fact there are important catch-up dynamics. There is distillation of US models by Chinese competitors, and there is diffusion of algorithmic ideas both within and between countries. Which means that if US development some month magically advanced by one month, the net effect would not be that forever more the US has one more month of lead, right? The net effect would be faster catch-up growth and that lead will decay. And we could debate what the time scale is, but the net effect of slowdowns is much smaller than it initially appears. You should not think of this as there's like a six-month budget of time that you could ever slow down before you get overtaken.
So to talk a little bit about building blocks for international coordination, caveating that as we go on we are further into views of me as a researcher who is not in fact an expert in international relations. I think a natural starting point is having some shared understanding of risk. Right? You'd like to be in a regime where at least each state understands here's what we would do. Here's how we would like to regulate the technology. We have some understanding of if we were not facing international competition how we would approach this problem. And if the risk is sufficiently high, that becomes a rational strategy. Right? So if you're the US looking at this gap and saying maybe I run a 50% chance, or if you end up in a world where it looks like you're running a greater than even chance of losing control, at some point it becomes rational for you and similarly for each other state to regulate the technology unilaterally. I think this is an important point because in many worlds it will at some point, if there is risk, become clear that the risk is large, and so just this kind of shared understanding is quite valuable even if it only cuts off some of the most extreme scenarios.
I think things get much better when you have some kind of visibility, right? So if for example the US has visibility into what PRC AI projects exist and moreover has visibility into whether those projects are meeting safety standards. So there's this implicit or explicit deterrent threat, which is that if you build uncontrollable AI, right? Maybe right now I am bound by a regulatory regime that has been designed to reduce the risk of uncontrollable AI. I may say if I have visibility into your implementation and you build uncontrollable AI, I could simply have the threat in retaliation of abandoning my own regulatory regime or having this tit-for-tat process of if you skimp on safety, I skimp on safety. And from a game theoretic perspective with visibility that changes the picture enormously, right? So previously I was in a regime where there was a strong incentive for me to skimp because it slightly increased risk and significantly increased my competitive positioning. Whereas now if I were to skimp on safety, both it will only slightly improve my competitive position, right, because my adversary will respond by also skimping, but also means that the increase in risk is much larger.
And maybe the last thing I will say is international pressure can make it much more costly for an entity not to play ball. Right? So you could be in a world where there is diplomatic or economic pressure on actors, where there is a broader international AI supply chain which can try and advantage people who are subscribing to reasonable policies. Or in the limit you can imagine international adversaries disrupting AI projects. And so the net effect of all of this is you can be in a regime where both compliance with a reasonable risk governance strategy is much less costly and non-compliance is much more costly. And so I don't want to say any of this is easy. I just want to say that from a game theory perspective, it's not at all obvious whether the situation is sunk. These are things that require a lot of legwork to build this kind of shared understanding, to build this kind of visibility, both awareness of AI projects and inspections, and to build this kind of pressure. But all of them seem like plausible mechanisms.
All right. So I'm going to spend the rest of the talk discussing the need for more time. Right? So first on the object level, we currently do not have a very good understanding of whether systems are aligned. We are not able to adequately secure AI developments and operationally
We are trying to build out the infrastructure for controlling AI systems, but are still struggling because of a lack of time. So on the object level, we need more time to implement technical measures. From a governance perspective, I just talked about a lot of nice-to-haves. I talked about the idea of regulation, or the idea of a shared bar on safety, or a verification regime. All of those take time. Those are not the world we're in today; those are the world we're trying to move to. But it is going to take time to get there. If we even bracket alignment and we think about the world more broadly, we are heading towards a world where there's incredible reliance on AI systems produced by a small number of tech companies, and we do not have the institutions and mechanisms to keep those companies effectively accountable. So even if you ignore everything I've talked about on loss of control, we still need more time.
And outside of the AI companies, there are a lot of ways in which society is going to need to adapt. Our institutions are going to need to adapt to new capabilities. Our economies are going to need to adapt. We're seeing simple examples like our cybersecurity is going to need to adapt even over the coming months.
And all of this is true against a backdrop where AI progress may accelerate rapidly. Right? So this was the topic of yesterday. We may see a transformation over perhaps as little as a few months or a year from AI systems which are having a moderate impact in the world to AI systems which are superhuman across a broad range of domains. And I think if we see that kind of acceleration, if progress is much faster in the future than it has been in the past, I am extremely worried about our ability to get to good on any of these dimensions.
So what is the minimal requirement? What is the minimal thing that would make me feel like we're on track? I think at some point it is reasonably likely, maybe over the next one to two years, that further progress will look so risky that we would like to slow. Right? We could debate whether that's the case now. We can debate how high risk is now, but I think it is likely that at some point things will look like there is an immediate risk.
If we had the capacity to slow down later, right? If we had a reliable capacity to slow down later, then there would be a reasonable argument to wait and see. Right? There are real costs to trying to slow down. There are real issues with international competition. There are real reasons people want to deliver the technology faster. If we could see what happens later and adapt online, I think that would be a defensible option.
The main problem is we don't really have the institutional and technical capacity to slow down when it counts. So I'm worried we're going to find ourselves in this world where further progress looks risky without the ability to slow effectively, or without the ability to slow effectively without large costs. For example, maybe these are particularly important examples. I think we need mechanisms to identify risk. We need to understand whether the systems we are building are adequately controlled and aligned.
If we're not able to do that, then it's good to tell a story about how we can slow down in the future, but we won't actually know when we get to that future.
Even if we can identify risk, we need governance mechanisms, the kind of thing we've been talking about through this lecture, in order to be able to act collectively, right? We don't want to end up in the world where there's an awareness of risk, but there is a race to the bottom amongst AI developers. And it will take months or longer in order to actually build the machinery which would constrain that race to the bottom.
And then a slightly more subtle point that I think is perhaps actually the most important is that we need to be able to defend models and algorithms from theft. So I think a scenario from a US national security perspective that currently looks likely is that we will arrive at a point where further progress appears risky and there would be demand to slow, but that there will be serious concerns or even a reasonably high probability that the underlying models or algorithms have leaked to adversaries. And so we now face a dilemma between either proceeding with a technology we believe to be unsafe or risking competition from an adversary who has the same resources, who has taken those algorithms and perhaps is willing to tolerate higher risk. I think that is one of the most likely ways that we end up in an extremely bad situation.
So in light of that, my view is now we should probably slow down immediately until we actually have the infrastructure to slow down later, right? Until we have a high enough degree of security, governance, risk management that we are able to detect and react to future risks as they emerge.
This is a letter signed by 400 employees from frontier AI companies asking the US government to support an international effort to develop the technical and governance tools needed to, what is it, deliberately pace the frontier of automated AI development. So this is not an uncommon sentiment, right? If you look through this list of signatories, it includes the chief scientists of most of the major AI companies. I think there's an understanding that we lack machinery we will need in order to be able to deliberately control the pace of progress.
I think I am going a little bit further here by saying that we may already be in the regime where our brakes are not good enough, and we need to slow until we can improve our ability to slow in the future.
Sometimes when you say that, people imagine this would involve AI companies just sitting on their hands for a while while we try and sort things out. I want to emphasize that is not what we are talking about. Right? So if we talk about pacing, what we really mean is reducing investment in a couple areas, right? Not training GPT-7, not training bigger, smarter models, and reduce investment in developing AI algorithms that could easily be stolen. So we're both not training GPT-7, and we're not developing the algorithms that would let you train GPT-7.
And to instead increase investment in first improving safety, security, governance, right? Shoring up all of these technical measures we would need in order to be in a position to slow in the future, but also deploying the AI we have to help society prepare, to secure our infrastructure, to create actual products and beneficial applications, to generally close the gap between AI in the lab where it is primarily driving this process of making AI ever smarter and AI that is actually in people's lives, giving society time to adapt, helping harden infrastructure. I think there's this general concern that the process of making smarter AIs can both be divorced from real-world utility and can be a major driver of risk. So when we say pacing, that's really what we mean: this sort of redirection of resources.
So to sum up, I think that we need governance to prevent the development of uncontrollable AI. This is a non-trivial challenge. I don't think it will be handled automatically by either markets, or by liability, or by unilateral restraint by altruistic developers. In order to ensure that frontier AI is trustworthy for defenders, we need to understand the alignment, control, capabilities, and security of those systems. Getting to a shared bar on all of those dimensions will require transparency, public debate, verification, and ultimately enforcement mechanisms. And unfortunately, all of that seems like it is going to take more time than we have on the current trajectory, or there's significant risk at will. And so I believe we should be exploring policies both to accelerate the adoption of all of those, or progress on all of those dimensions, but also to buy some time. That's all I have for tonight. Thank you.
