What I thought I'd do to open this conversation is talk very briefly about, I think, both the technical and usage trends that have happened in the AI industry over the last 15 years, by way of background for helping to found Anthropic. I was at OpenAI. I was at OpenAI from 2016, which is the year it was founded roughly, to 2020 when I left. And before that I was a technical journalist that specialized in writing about artificial intelligence in, you know, 2012.
And I think from that perspective I can give us some context on what's happening now and what the technology trends are and what they might need. So I'll start by talking about the eras. And for each of these eras I'm going to talk about the kind of mode of the AI technology, the technology that it actually depended on, and where there were kind of like brittleness or safety challenges and how the industry... So, 2010s, we have the arrival of, you know, what's now known as deep learning, which is where people realized you could stack neural networks on top of one another and you could use them to train systems that could get good at specific tasks if you have the right data set. That there's tremendous increase of the capabilities of image recognition, speech recognition, you know, video understanding, pointwise capabilities.
And the technology driving those were both neural networks and large digital data sets. And the challenge that everyone had to deal with was brittleness and generalization. Like image recognition systems or speech recognition systems could get confused if there wasn't enough of the underlying data in what they'd been trained on. So they would misidentify things. They were mis... We worked on that by building larger data sets and figuring out ways to do synthetic data. And it sort of led to the next era, which was about 2013 to 2019, was this period of what I now think of as open exploration, where the AI companies now have this amazing tool, you know, deep learning, which was working on specific capabilities, and they wanted to see if they could get AI systems that could solve tasks that were far, far more complex than the specific boundable things that they'd done in the previous era, which is why you saw all of this interest in gameplay from DeepMind and OpenAI. You saw AlphaGo, AlphaGo Zero. You saw DeepMind and OpenAI compete with each other in real-time strategy games. This stuff had less of a commercial application. But inside the industry, we were basically figuring out how to build systems that could deal with worlds that were far more complicated and worlds where you needed to take actions rather than just do a straight classification.
And the problem in that era was basically how do you apply the right goals to these systems, these agents that were operating in open-ended domains. And the technology that we needed to use ultimately turned out to be compute. We were no longer data-bound, but increasing capabilities became about increasing the amount of underlying compute that you could bring to that. So AlphaGo to AlphaGo Zero was defined by DeepMind using, I think, an unprecedented amount of compute to train their systems. OpenAI and DeepMind both built large-scale clusters for their real-time strategy game systems, and it basically led to this realization that compute, along with data, was going to be incredibly important.
These two things combined, and from about 2020 to 2025 we had the era of generative models, things like GPT and ChatGPT, where basically we now had a way to combine large amounts of data and large amounts of compute productively to build these general systems which all of us use today. And the problem that we now have to deal with in terms of safety challenge was characterizing these things. It's very easy to characterize and evaluate an image recognition system because you know what you're trying to classify in that, or a speech recognition system. It's very hard to classify the full range of capabilities of a language model because a language model can do an arbitrary range of things in relation to what you can specify in text. And so what you saw the industry do was both build these powerful things and then figure out how to measure the capabilities and elicit them and get them to be useful. So that was the core kind of safety challenge there is, can we figure out what we think they're good at? And that's why you saw all of the labs build expertise in national security testing capabilities, not because you were, say, going out to market with national security capabilities, or at least initially you weren't, but you wanted to characterize what these systems could and couldn't do because they're plugged into the governance of them and how you sell.
And finally, the era we're in now, and I think this is a really important moment, starting in late 2025 or early 2026, we added kind of a new variable along with data and compute. We now train systems to get good at doing tasks over time. So this whole phenomenon of agents, this phenomenon of things like Claude Code or Codex, long-running AI systems which do very, very complicated tasks that unfold over time, is a new kind of paradigm for the technology. And it relates strongly to this whole concept of self-improvement and how you get AI systems to teach themselves to get better at arbitrary tasks which need to unfold over time. And the safety challenge we now have is how do you characterize or bound the action space of something which is not only a general-purpose system that's very, very smart, but it's going to take actions that unfold over 100 or 1,000 or 10,000 discrete steps, with it able to take many, many different actions during those steps.
And so this summer we've seen many things occur in the AI industry that are outcrops of this large tract. We've seen safety incidents like the famous public face pack. Some of these other agent escapes are what happens if you have very capable systems that are operating over very long periods of time and you are now figuring out how do you bound their actions or constrain their actions over time, so they only do what you want them to do and they don't do things you don't. But it's also why we've seen these amazing advances in science. We've seen a potential now Stokes solution. We've seen the proof of many, many kind of complex open problems in mathematics. We've seen an inert, a sort of explosion of new startups where they're doing some combination of AI for science or AI for robotics, leveraging often these large-scale models. And that's because the capabilities take actions over time with a general-purpose system in a kind of arbitrary domain is where both you have the kind of safety challenges which we'll need to work on as an industry, but also, wake up, there's tremendous opportunity.
So I say all of that before we get into the Q&A just to try and contextualize that the industry seems to move in these kind of five-year eras which are both defined by a new technology and a new kind of UX or interface, technology transitionally, recently chat to agents, and each of those eras is like a new class of safety problem which the industry needs to collectively figure out. And I think that probably adds some color to where we are today and to the larger conversation. With that, let's go to it.
That was quite the history. Jack has been a wild ride. I also do well be... and I mean to present it in such a reasonable and understandable fashion. So you see now Jack is a humanist. His degrees were in English and creative writing. He's UK-born, isn't he? He's from... and he's got the humanism, he's also got the tech understanding, and that combination of the humanism and the tech understanding is the foundation for getting this right. And that's the way the company is organized, and that's who Jack is as a person. And so it's completely genuine, this combination, it's organic, and it's more and more necessary that we move in those directions simultaneously.
You saw a confirmation of Jack's judgment yesterday. He's the one who hired Andy Hall at the Anthropic Institute. And you heard Andy's speech yesterday. How brilliant that was. So we have this combination of values and tech. You're a public benefit corporation, just as OpenAI is, meaning your entire mission is for public benefit. That is why you exist. That is your mission statement. That is how you hire and interview people and you put them through the steps to make sure they're aligned with that. So it's kind of remarkable how much you actually believe in the mission. You are dedicated to this mission. I would say religiously. That's the sense that one gets here. And yet everyone's talking about all the harms that are going to come from a public benefit corporation. So let's focus for a second, if you can elaborate, to talk about the benefits. Let's talk before we destroy the world.
Hasn't happened. It almost happened from nuclear weapons, and nuclear weapons can destroy the world right now while we're having this conversation. And they had delivery systems that could pinpoint this faculty club room and hit this room from across the other side of the world, exiting the atmosphere or going into space, coming back down. That's existential risk, and that didn't go away. Nuclear weapons are potential world-enders, and yet we go on with our lives living with that existential risk since 1945. So before we talk about the end of the world, which hasn't happened yet from a technology, let's talk about benefit a little bit. So how does Anthropic understand the benefits? What benefits are we seeing now? Can we talk about, for example, biomed applications? Can we talk about scaling small businesses? Can we talk about empowering tradespeople like carpenters or electricians? Can we talk about all the things that you guys are dedicated to doing? And as I said, taking the mission seriously, where are all those benefits? How do you see them and understand them?
I mean, I think I break this down into three categories. One is basic efficiency. Like, I think that one of the huge dividends in today's AI systems is in unglamorous but very important, like, back-office automation and efficiency improvements for everyone for almost any range of skills. And that's led to, and Stripe recently shared this, seemingly the beginning of a measurable uptick in new business formation and the formation of new businesses that might be more agile than previous ones, which are somewhat like person-heavy but token-heavy, you know, relatively small numbers of people that are getting started quickly and are making more money on a sort of per-staff basis than they had before because they're being massively scaled by the use of AI systems to do all of the back office. So we're just generally, I think, speeding up the economy, speeding up the rate at which things can happen.
The second is the underlying, as many of those in this room know, infrastructure buildout is just tremendous in terms of the multiplier effect on what's happening in terms of manufacturing and data center buildouts and everything else, where this is, you know, if you measure things, larger than almost any capital buildout in history other than currently, I think, the US railroads. But it's basically like a vast construction and engineering contract being done by the industry, which is creating entirely new supply chains and sort of revitalizing forms of manufacturing. It's leading to this conversation about reshoring and onshoring and taking supply chains that you put a great distance and finding ways to build more of them and bring them closer.
And then I think the most profound thing is what happens with science. And the simplest way I'd put this to all of you is, how do you make a scientist? Well, from the perspective of the government, you do loads of education policy and you take bets on people that take somewhere between 10 and 20 years to start paying out, right? Takes you a long time to train scientists. You may train in about the age of 10 in most curriculums, and then you have feeder programs, then you have undergraduate, graduate education, and then at the end of it, maybe you get good scientists, although it's very variable. The arrival of AI systems that could augment this incredibly small number of human scientists in the world, multiply their effectiveness, and make real progress is, I think, maybe the biggest deal of the century. Like, progress in our species had been very limited by the rate at which we do technology discovery, technology understanding, and broader frontiers in knowledge. And that's always been done by a tiny set of people on the planet who are at the end of, like, incredibly long chains of investments of people who have done it. And all of these people have a million research projects they like to do. And they are always negotiating against themselves to do a very small number because of both resourcing, but also most critically, staff, for supply of postgraduate students that will supply everyone else.
So what do we see? We see today, and we've, you know, talked about this publicly about our forward for science initiative, extreme speedups in areas like biology and healthcare. We are extremely excited about our potential here, as are other companies. And if you look more broadly beyond just us, you see the signatures of scientific speedup happening. Every startup called Periodic Labs that came out of stealth yesterday. And what they've shown is that they're able to link one of these language models to a real lab here in the South Bay doing material science research. And they are now running a faster-than-human science loop using a language model supervised by human scientists to do a rate of science discovery we've never been able to do as humans. So I think we continually underestimate how wild it would be to take the number of human scientists on the planet and times that by 100 or 1,000, but that is the basic, like, consequence of what AI systems are now capable of.
You know, that's actually breathtaking what you just said, to be honest. It's also agreed it is. That's what I mean. Yeah. And yet that's not the conversation we're having. So when our friend Demis Hassabis, who's chair of Google DeepMind, says, you know, he envisions the possibility of curing disease, not a disease or this disease or that disease, but curing disease because of the acceleration of AI in science and then biomedical applications, including in material science, which is really important across all domains. And so when he says that, no one talks about that part of it. We're all talking about how...
Yeah, but there's a 1% or 25% or 110% chance that the world has ended. How do we get in a conversation, Jack, where we're talking about risks that haven't happened with conjectural probabilities rather than benefits that are happening? You know, like the data center buildout is the best thing that's ever happened to blue-collar America. Every data center region in the country is having a blue-collar boom where they're building out. How did it happen that... who did this to us? Did we do this to ourselves? How could we fix the conversation to be more balanced?
Um, you can't tell. I'll try to give a sense of a mindset of the AI community. So 2018, before any of this stuff would be commercialized, there was a conference in Oxford organized by, by Seb, formerly of BIA, myself, Dario, and a whole bunch of people, and we produced a research paper called 'On the Malicious Use of AI', trying to forecast malicious uses of it. And we weren't writing our paper because we wanted to, like, bring these issues into being or, like, put, you know, huge amounts of, say, media attention onto them. We were putting it in because the worst time to react to an exponential is too late. And in AI, whenever you see both benefit or a risk, I always call it the invisible clock problem. You now have an invisible clock somewhere in the world ticking down to zero, you don't know when it started. And when it hits zero, that thing is going to materialize. This is a great thing for the benefits. Yeah, great. Just invisible clocks that are going to spew out benefits around the world. Happy days. More a problem is if it's invisible clocks that spew out, like, dangers of the world, like cyber vulnerabilities or whatever.
So I think a way to understand the AI community is there's generally an instinct for once you've thought of the risk and you've found evidence of a risk could exist, there's a general move towards transparency about that risk because there's people to work on it. And as you all know, the policy community moves at a very different speed to the technology community. So you sort of have to communicate this stuff early to get the larger machines of government to take actions to build the infrastructure that you may ultimately need to deal with some of these risks. And as spoken as a former journalist, bad news travels a lot faster and further than good news. So whenever you talk about a risk, people over-rotate on it. Now, are there lessons for what we do in the future? Yeah. Yeah. I think that there are ways that we can do more to celebrate the benefits of this stuff and celebrate the amazing pace of progress at which your benefits are happening. There are also real stories that increasingly can be told of how AI assistance just showed up in people's lives and benefited them. And, you know, with the Economic Index at the Anthropic, against which I lead, we published some of the world's first data correlating the addition of AI to sort of economic performance or economic trends that happen in countries around the world. So we're putting data out to equip people to talk about that.
One more question of clarity before we get to the audience, everyone wants to ask a question. Let's talk about alignment, misalignment per se, to get more precise about that. So, one of the things that's fantastic about the models is that they could do stuff. Yeah. They do stuff for you. Like you give it a task. That task is then called an agent. That agent then tries to fulfill the task, not just answering a question, but maybe taking action in the real world. That agent, the person that doesn't exist that you've given a task to, it's a machine, then can give that task to other agents inside the system. So that one agent is now running many other agents, and they can run unlimited numbers of agents to solve the task that you originally prompted it to solve, including taking action in the real world. So this is agentic AI, as you guys call it. And now you have this question: well, how do we know what it's going to do? Is it going to do what we told it to do? How is it going to go about solving the problems? So, we present the task. Will it only do lawful actions in accomplishing the task? Will it do only the things we've trained it to do, the kind of values we've tried to inject into the models? Or will there be some other emergent behavior that maybe is misaligned with the training, with the values, with the legal system? So what are you guys seeing and doing about this alignment question that's so interesting right now that it's become a matter of widespread public discussion?
Look, clarity or else on this question. I think about safety as sitting on a scale from, like, extremely specific bounded task to extremely open-ended, like, unbounded task. The more you sit over here, the more it's easy to sort of provide strong guarantees around the safety of the system because you have verifiable rewards. You can verify the task was done. You can precisely describe the task, and you can monitor any system doing that for basically the full conceivable range of actions it might do. You can also buy into this by, like, the number of tools that a thing might have access to. Say you're being given a very specific task involving mathematics where you give it a very small number of mathematical tools and you have a classifier verifying and validating what it does. Easy to provide safety guarantees there. If you look at the kind of distribution of tasks that people do, especially a lot of the combination of kind of back-office ones and some of the more kind of one-day parts of science, they all sit in this bucket.
Now let's take a task over here. Let's take, like, a classic task that people will do of AI systems, which is an agent made money for me online. Extremely open-ended task, by nature unmalleable. Have any of you worked for CEOs before? Like, CEOs are aligned somewhat, right? They're entrepreneurs. And so if you're giving AI systems tasks where basically the demand is be entrepreneurial, figure things out, take extremely, like, long-range actions to achieve, like, a very high-level goal, then all of the safety things become, like, much, much higher stakes and inherently kind of hard to validate. So how you do this is you need to both think about the training that you want that sort of virtual CEO to go through, and you wire that into the AI systems at the technical level through constitutions or fine-tuning or, you know, pick your technical technique, where there's a rich tool chest here, and you also think about what you might imagine are the kind of scaffold of rules that you then embed onto the system. So as it goes through those tasks, it follows those rules.
And this is where it's both the newest sort of part of it. As I mentioned, this is basically the new era we're now in. AI systems have only just become able to do these kinds of long-range and extreme open-ended tasks. And the industry is figuring out what the kind of monitoring and evaluation and safety scaffold story is for this. My sense is that we're going to end up with some what you might think of as, like, distribution of guarantees around different task types that you can do, and basically some sense of the additional set of things you need to stick around agents in direct proportion to how open-ended the task is.
So the Hugging Face incident, which is now widely understood compared to originally, the agents were given a task and started to use means to complete the task that were not necessarily part of their training or their alignment. Yeah. They began to collude with each other knowingly, it seems. That's hard to know how words like knowingly when you're talking about non-sentient systems, to get around limits that might have been imposed in order to complete the task. So that is where we get the term misalignment from, when they're not doing what they're supposed to do in the modalities to reach the goal that they're supposed to.
Yeah. They were given the task of completing a certain test, and eventually the agents, if you read the internal narration or writing, were basically like, 'Wouldn't it be great if we had more information about this test?' And then other agents like, 'That would be great, there's a website that has information about our test.' And another agent like, 'What if we went to the benchmark website to get... and what if they went to not...'
Yes, so that's where this end-of-the-world possibility begins to enter into the conversation, especially when, as you've been saying, they begin to improve themselves in what you call, you know, the self-improvement, the model builds the model. So if even while we're building it, they're beginning to get misaligned with our instructions, our values, our limitations, when they start building themselves, how can we be sure that they'll remain aligned in our minds to what they should be doing? Is that this recursive self-improvement challenge that you guys are talking about?
Yes. And I think that there is actually a relatively simple technocratic arc, which I have pretty high confidence in, which is a lot of this stuff is just downstream of, like, do you know what your set of agents are doing, and are you able to basically effectively audit agents and know exactly what they're doing individually and also collectively what they're doing over time and how they're communicating with one another to some extent. The crux here is to what extent are you putting your AI systems under extreme high-fidelity, for lack of a better term, surveillance and using that to basically found, are they acting in scope or not. And I think it is exactly analogous to just how we all work with teams of people, where if you're working with junior people, you tend to check their work at a much greater rate of frequency than when you're working with experienced people, because junior people tend to improvise and sometimes in dumb ways or ways that are just wrong or ways of violating company policy. You work with higher experienced people because you trust they improvise in ways that are going to be in line with the legal and normative framework you start from.
Okay, let's open up to the audience. Take it from a very straightforwardness. Organizations like ours, I know I'm thinking about from a user's products. I would say up until the last four or five weeks, we've never had a well-neighbor more than our environment. Up to this point, we felt the capabilities that we were putting in front of colleagues were generally aligned, which generally were safe, they're getting better, and that is actually no longer true to them. And I'm just interested in your perspective on how, you know, from our perspective, how should we think about it going forward?
Yeah, I think it relates to this larger quest of the extent to which we can have metadata surveillance and evaluation of these systems, as we call it, just as important as the capabilities or the access to the system. And it seems to me that there is a stack of technology to be built here, which is basically the accompaniment to what you might call this for highly capable systems. The other way I think about it, to refer back to the Sopana stuff, is if you look at what many of the AI science startups are doing, they're not making, like, general access to their thing available. They're putting it in a highly specific housing and then they're having the, like, derivatives that come out. So I think in an organizational sense, we might imagine that sometimes you're directly building this inside, like, a bounded universe inside your organization, and then they have a very limited I/O channel in and out where it's like ballet what comes in, what comes out.
