Inside Anthropic's Biggest Fear About AI

Models which are as or more capable than all humans are very likely to occur in the next couple of years. Each marginal unit of intelligence is worth exponentially more than the unit that came before. You can keep increasing, potentially doubling, the productive intellectual and physical capacity of humanity in such a way that allows you to truly get to a post-scarcity world.

You guys are the edge of the AI frontier. What are the most exciting things you're working on? AI is going to be the most important technology in the life sciences, certainly of our lifetime and maybe ever. But it also comes with massive risks. We shouldn't be releasing these technologies into the world before we can keep the bad actors from doing a lot of effort.

Tech leaders I know who are more skeptical, who are saying, 'Oh, well, Dario is trying to scare everyone to create more rules to slow things down.' So far, we've slowed ourselves down much more than anyone else. Is there any scenario you'd slow America and let China go ahead? CEO and co-founder. Now I'm here.

Nick Marwell and Schulto Douglas are key technical leaders at Anthropic, the fastest-growing AI company in the world. They're both very optimistic about the future, despite some really serious threats we're facing as a civilization. What are these threats? How do we think about offensive versus defensive dynamics in bio, in cyber, and other key areas? Some people are pretty angry at Anthropic for regulatory capture. What is their view on this? What is Anthropic trying to do? What do they think of the battles with distillation in China? If everyone's able to copy their models, how is Anthropic still going to be worth trillions of dollars? And why? If you're a young person in the workforce, if you're trying to build your career, but you're not in the center of AI, what should you be doing for the next 10 years? An optimistic insider's view on one of the fastest-moving companies in the world at a fascinating time. Excited for you to hear from them. Welcome to American Optimist. Really excited to have today two amazing talents from Anthropic. Nick, you're the key member of the technical staff, help lead RL science, which is reinforcement learning science, and Schulto Douglas, also a tech lead in reinforcement learning. Is that right?

Yep. And how long have you guys been at Anthropic? The company's only how old, though? It's five years or so. Yeah. It's amazing. It's like one of the most important, biggest companies, fastest-growing companies now in the world, and it's only been around for five years. So this is a heady time here in California. You're filming here in Napa today. Thanks for being here, guys.

Let's start with your backgrounds. So where'd you grow up, Schulto? Like, where are you from? Had you studied computer science? Awesome. And Nick, where are you from? Amazing. Well, since you guys have joined Anthropic, I think it has become the biggest of the AI companies. You guys don't release numbers officially, but lots of stuff leaks. The stuff that leaks, just for the audience, it says it's, you know, here in August of 2026. I keep my face very still. Very still. No reactions. You know, it's like supposedly doing over $80 billion of run rate, growing really fast, ahead of OpenAI, ahead of everyone. Like, you guys are the edge of the AI frontier. Like, what are the most exciting things you're working on internally right now that you can tell us about? What's it like to be at that cutting edge?

Well, I think what I can talk about a lot is how the experience of programming has changed, because I think this is the most visceral way that we felt progress. And 18 months ago, I was typing all of my lines of code by hand, basically. Then, you know, a few months into Anthropic, I was guiding a model. I was intervening every few minutes. I'd say what I wanted it to do, write the next function or this kind of thing, but I'd have to check in every few minutes because it would make mistakes. It would go off course and this kind of thing. And where we're at now is I can ask models to do a day or two days of work independently and drive progress basically like a junior team member. And that's just over the last 18 months, we've gone from what is really very much a tool where you have to stay in the loop to something that feels very much like a junior team member.

So, as we're talking right now, they're the junior team members, the AI is working hard for you. Yeah, they are. There's a bunch of them working hard. What kind of stuff are they working on? Could you say at a high level, could you tell us? Awesome. I'll give a slightly different answer here. So I think a lot of model development up to this point has really been about getting models to be able to do tasks that humans are actually already quite good and capable at. And really what we're trying to do is capabilities that bring lots of additional leverage to things humans were sort of already cognitively capable of achieving, right? Building a piece of software that like a team of human developers could also have built in the same period of time. I think what you're starting to see now, and I think some of the most recent results in math, for example, are a good example of this, is people trying to figure out how to use models to push past sort of the frontier of human intelligence and human achievement.

This is like some of the recent math things. Yeah, there have been like a lot of new math proofs that have come out that are sort of things humans have been interested in for a long time and sort of never been able to solve. Some of those, it just seemed like a really smart mathematician realized that the computer is good enough to brute force something that a person couldn't have brute forced. Is you see what I'm saying? Like, are some of them still seem like they're good at specific things that humans are using, or is it really like a whole new theory that they're getting at, or how do you think about that? I think it is not sort of, at least yet, some idea where like they're coming up with ideas that humans would have been incapable of coming up with. But I actually think very little progress or intellectual achievement ever looks like this in many ways. I think, you know, there are of course exceptions, but I think most of the things that society has achieved are really just about sort of connecting the dots and building upon the prior intellectual achievements of things that came before. And I think that what you're seeing when you describe something like putting two ideas that different people had together to find something new is the foundation of like what humans do. And it's sort of the important thing that's different now is it's really putting together sort of new ideas, or sorry, forming new ideas from combining other human ideas that weren't happening before. Whereas I don't think like writing another web app was actually some like new intellectual achievement for humanity.

And so, you know, we've never seen companies scale this fast in Silicon Valley. I think again, not to 10x, 10x, 10x, growing really fast, obviously. We had another company here in Napa last night that I'm on the board of staying over, and it's like a new company every 60 to 90 days. There's so much happening, right? And it's just, I think this is like the experience of a lot of us. I think you're right in the core of like the fastest-growing company. Like, what's it like versus like your life before? Like, where do we go from here? Is it going to keep accelerating? How do we think about this? I think one thing that's been really nice about Anthropic has been that our ambitions have been able to scale now with the acceleration of the company. You know, really felt like 18 months ago we had to be so insanely focused on code and really like proving out that we could be the best in the world at something. And now the mission of the company is so much grander than make good models that are good at code. And as the company has scaled, as the revenue has scaled, it's let us really start putting together the pieces to start pushing on more of that.

Let's talk more about that mission. I was with another friend who's senior in AI, you know, earlier this week, and I understand like he was saying how first you got to be the very best at code and math, but then there's going to be like new things it's going to be the very best at. You can, there's going to be all these other human skills you can get to be A++. Is that the idea, is bring in a bunch of new skills, or like what's the mission from here? Well, the mission from here is always we think that AGI is really achievable in the next couple of years. Explain to our audience what you mean by that. So we think that models which are as or more capable than all humans are very likely to occur in the next couple of years. So it's something that can do all of the things that a human could do on a computer, or once we get sufficiently advanced robotics, all of the things that a human could do in the physical world.

So it's going to be like much better at the podcast. I could just let the model, let the robot jump in. Well, you know, the podcast, you know, let's go to the vineyard. It's more fun. Exactly. And we think this both has huge upsides, right? This, it would allow you to multiply the intellectual labor and physical labor done in the world today. It would allow you to get centuries of progress potentially much, much, much sooner. Yeah. But it also comes with massive risks, and so really the goal of the company is shepherding the world through those risks so that we can get the upsides on the other side.

So let's step back on the AI wave in general, like where we are on this AI wave. Obviously, you were saying in 2020, I think you realized, you said that AGI was going to become possible. I think that's right around GPT-3 came out from OpenAI. I was looking back at some emails because you're doing like a tenure ADC book right now, and I was making jokes even in 2020 about how we're going to replace all the associates. And so it was kind of in the air that, okay, this is starting to become likely. You know, I think we were talking to Dario earlier this year, and by some metrics, some capabilities were like doubling every few months. Like, how do we think about quantifying progress? You talked about the fact you could just use the AI as opposed, you know, a junior employee. Is there any other way to think about like how much better it gets next couple of years, like metrics-wise?

Yeah. I think there's a lot of evals, and these are like basically exams where people test the models. And on these, progress is extremely rapid. We go from being able to do, you know, a great one is, I don't want to lean too much on math as an example. But math is maybe the best in some ways, where we went from 0% on a set of problems assembled by professors across different fields to well over 40, 50, 60% just over the course of last year. And that's an eval called Frontier Math. Do you need something that's falsifiable, right? You need something to iterate on. So could you like have it like talk to a girl on Instagram and like get a certain percent chance that it like gets her to want to go on a date with you? Right. I mean, is innocent enough to ask like that. Train on that. Right. I'm married, but not for me. We haven't trained on that. But that's an example. Other sorts of things like that that'd be probably very useful for like especially some of our friends in San Francisco. Maybe there's two ways the models improve, right? One of them is via training on vast corpora of text from the internet. And then the second form, reinforcement learning, what we both work on, is doing these worked problems and checking whether or not you got the right answer. And so this is what you're talking about, this, which is very easy for math and for computer science. It's pretty hard for romance or poems or other things, right? Yeah. And that's why you've seen math and computer science advance so much faster than everything else. But we actually think it's not that hard to put other problems in this frame.

Yeah. I actually think there's like in some ways a technical explanation for why math and computer science came first. I also think to some degree there's like a social explanation for why this happened, which is that you took a group of people who primarily came from backgrounds in math and computer science, and you asked them to like make intelligence. And of course, they decided that the first things they wanted to do were to like make intelligence in the domains they understood, because they both were most interested in them. And because it's much easier, you know, it's amazing how much time when you're trying to train good models, you actually spend like reading what the model is doing. And trying to understand as a result like how capable it's becoming, what kinds of things you might need to focus on, sort of areas it needs to get better. And it's much easier to do this for domains where you're an expert, you understand what's going on. Do you anthropomorphize it when you think about it? I don't think I personally do. I think there are some people who probably do, and I'm sure some of them work at Anthropic. I mean, our brains are kind of programmed, like there's whole parts of our brains that are set up to think of like the person we're looking at and mirror what we think their emotions are in order to talk to. So we have all these things built in from evolution to like treat an intelligence as another intelligence, but it seems like it'd be hard to avoid doing that when you're working on this day-to-day.

This is maybe one of the bigger debates in terms of how people try and frame these things as tools or as entities. And I really think that entity is a useful frame because with a tool you are using it the entire time. And with these models, what you're really doing is you're taking this thing that is very intelligent and you're releasing it actually out into the world in some respects, where it's going to take actions on your behalf. And so it's often more useful to think about these as an entity for that reason. Are there categories or areas of things we should be having this intelligence do that we're not yet? Like there's things you guys are excited about, are working on, where like we're going to use this a lot more in these other areas. Like what are those areas? I think the big one's biology, right? Yeah. I think it depends if you're asking a question that's about what are models already great at that we think are just underutilized skills versus where are we most excited about where models are going. I think on the latter question, and we can get more into why biology is the thing that I am most excited about over the next say six to twenty-four months in terms of where I think models will really begin to make an impact in the way that they have in software engineering, but not just in the sense of that they'll be widely adopted, but that the sort of societal positives that come out of this will be really diffuse and really, really powerful.

Curing diseases and making all of us healthier and longevity. It seems like there's all sorts of weird things about epigenetics that are tied to longevity that we're kind of just discovering through the Nobel Prizes starting twenty years ago. And all of our friends have companies now in the Bay Area. There's billions of dollars trying to like figure out how to trigger these things and how to make part of your body younger and stuff. So you think AI is going to be critical in advance? I think AI is going to be the most important technology in the life sciences, certainly of our lifetime and maybe ever. In fact, I am confident to the point that I am almost more concerned that if we have the physical and lab infrastructure required to take advantage of this than I am about whether AI will be able to be a useful intellectual. Three years ago, we should have been probably building a lot more compute, for example. We talk about this too much. Should someone be like putting, creating a lot more like lab space for you to use or something right now? I think so. And I think there are a couple of axes on which this is important. I think one axis on which this is important is like literally how much lab space do you have. But I think that there are also a number of sort of metrics around quality of the labs that we have in the US that are clearly lagging behind places like China, really. And I think it is one of the great sort of societal infrastructure projects that people should be thinking about how to go solve is how to get the supply chain and lab complex in America back up to being sort of a gold standard, best in the world. Yeah. This will be the right limiter on biology progress. Yeah.

All right. Well, let's go solve that after the podcast. That sounds like a good one. You get some OSC money from the Department of War for supply chains and build some labs. Just stepping back a little bit on where we are in the wave. What would you do as like a new college grad or someone who's in the workforce? You're not in Anthropic. You're not in the middle of Silicon Valley. You might be a smart person who's ambitious. Like, how should you think about your career given what's happening with you guys in the next few years? So I think that there are a couple of different scenarios we could end up in. And Anthropic, you know, sort of has a responsibility to speak about the risks that it's bringing into the world. And one of them is we're actually quite concerned about unemployment. But I do think it's possible that this technology will take long enough to diffuse into the world that there are a bunch of scenarios in which you have a ten or twenty-year period where there are actually a ton of opportunities to be part of the diffusion of this technology and sort of have that be what defines your career. And a thing I've said to some folks, I mean, I have a brother who's like in college right now and actually thinking about this exact question, is I think that the next decade, if we are in such a scenario, is really going to be one that sort of belongs to the generalists in a lot of ways.

I think a lot of the way you sort of had a great career over the prior, I don't know, twenty or fifty years is you had a sort of set of skills that were very valuable and very marketable. And it was very easy to have a set of skills and depend on other people. Everyone has, they have the power of a thousand-person company at their back, and what it's really going to mean is that the people who are capable of best understanding what are the problems to go out and be solved, what are the things that would make my community better, whether it's from a sort of like commercial standpoint, from a social good standpoint, etc., they'll have all of the tools they need to go affect that change. And so sort of problem selection and things like that will be the defining skill that you said, like what are the problems to be solved? How do I make my community better? One of the kind of fundamental backdrops to my view of the world and Judaism is you're supposed to like pretend you're like repairing the world six days a year and then you pretend you're done on the Sabbath and the day of rest, which I think a lot of our major religions have something similar. And my optimism is that there's always going to be things for each of us to kind of repair and help and fix with the people around us, with the communities around us. Like unless you believe we're going to be in utopia, there's going to be things to do for people, right? I think it's an optimistic framework. Absolutely, yeah. Yeah, there's a lot of work between us and utopia. Yes, exactly. So there's like the whole fundamental, I think like the implication of the religion is it's not ever utopia here until like the Messiah comes or something, and so therefore you have to like just keep fixing things, right? And that's like a positive thing.

I know you guys are optimists. There's lots of positive things. I want to go into some concerns, though, first of all, because this is where a lot of people are very curious about. Obviously, on the coast, I think we're seeing all these really positive things around us in our communities. We all have friends building companies faster than ever before. It's just like this really positive energy. At least for me, when I come out to San Francisco, when I go to New York, it's even things I'm doing in Austin, Texas. But I think a lot of the rest of the country, they feel like there's threats to their business, threats to their livelihood. It's scary. They don't know that you guys aren't just going to like be like some kind of wacky people who are going to like conquer with this or whatever. What are people missing? Like, what is there? Is there a plausible bad path that you worry about? Is the bad path they worry about wrong? Like, how do you think about this problem? I think the concerns are extremely reasonable. There's a couple of big categories that we're worried about. But all of them are ones that we think if we take the right actions over the next couple of years, we can end up with a radically better world in the 2030s. And so those categories are a big one we've mentioned already is unemployment, right? And we should have a long discussion around that and like why we think that's a risk and what we think the right things to do about that. There's also the more immediate near-term concerns of bio and cyber risk that are literally happening like this year right now.

There's all sorts of annoying things there where people would make fun of Anthropic because you couldn't ask it about the mitochondria of a cell was a classic case. And it's like, well, actually, probably you should be able to do that and just maybe just don't help make a bioweapon or something. You should definitely be allowed to do that. But you know, we're sort of putting in place the infrastructure so that we can feel confident that no one can make a bioweapon out of them. Yeah. What's incredibly hard with things like bio and cyber is that these are inherently what's known as dual-use capabilities. So I think cyber is the easiest one to understand. If I prompt Claude and I say, 'Here's a code base. Please find all of the vulnerabilities.' It is just as likely that I am the owner of that code base trying to harden my defenses as it is that I am the attacker trying to find my way in. Yep. And this is what fundamentally makes things like cyber or bio, which has many of the same sort of dual-use characteristics, so hard to police that you're trying to target something and you're in a cell and it's a really good chance, probably like 99.9% chance you're trying to kill it to get rid of cancer or something, but there's some tiny chance you're trying to target something terrible. Right. And so I think we're working on a number of things that are sort of meant to enable us to make sure that good actors get to use these technologies and bad actors are kept. But I think that our general feeling is we shouldn't be releasing these technologies into the world before we can keep the bad actors from doing.

There's definitely people, and it's not even Anthropic. I think they're using like open models from China with terrorist groups, for example, in Africa to figure out how to more easily make certain bombs that they didn't know how to make before. And it's just smart enough to help them because they weren't very smart at that. There's scary use, but at the same time, like I guess my bias is overall with more intelligence, more people can understand the scary uses and use to stop them as well, right? There's a trade-off. And so a big dynamic here that we should talk about is whether an area is offense or defense dominant. Yes. Right. And right now, both cyber and bio are offense dominant, but I think that over the next two years, cyber becomes defense dominant, especially if everyone has access to all intelligence to hack themselves and build it. Exactly. Because you can preemptively try and hack yourselves, build up the defenses, patch all the vulnerabilities, and then eventually we'll end up in a much more cyber-secure world where everyone already has access to intelligence, they figure out how they're going to break in. So therefore, that's what we're doing. That's what we're doing is we're ahead of time. I mean, I think one of my friends helped break in something at the Pentagon and the banks when they asked them and they showed them, but they're using new AI because AI is so good at it. And so now everyone has to do it quickly.

Now, let's step back for a second. I think the offense-defense thing is super interesting, right? This has been a concept for thousands of years in human history. It changes a lot. I'm a big fan of like the very free city-states that used to compete in Europe, and this was a really good thing. And if one city-state was doing something badly, you can go to the other one. And then unfortunately, these kind of jerks came along with cannons. They could just knock down the walls really easily, and then they built these big empires, and you couldn't just leave your city-state anymore because now you're under them regardless. And so there's like different times when things have like moved between offense and defense. And I agree, I think cyber right now is super offense-dominated to the point where like people should be scared. By the way, like anyone listening, if you're not on the cutting edge using AI to challenge your systems, they probably will be broken into by, and it might be a very bad person. So it's like, so right now it's scary. I agree, eventually cyber could become more defense-dominated used correctly. I guess there's a question with AI, like one of the fears for me is like, are there certain hidden things in the world that are just super offense-dominated? So for example, and I don't believe this is the case, but for example, if you can make like a self-replicating nanobot with like enough intelligence that just eats the whole world, and like one genius makes it and then eats the whole world and then we're all dead. And again, I don't think that's going to happen, but I guess as a question, like we have to make sure we're using the intelligence to find out whatever is super offense-dominated sooner. Right.

Right. And actually bio is super offense-dominant until we put a huge amount of work into making the world defense-dominant. There are ways you can make the world defense-dominant against bio, but it's literally like it's going to require hundreds of billions of dollars of infrastructure. It's sort of become, it's not unreasonable to do until you get dramatically advanced capabilities in robotics in like the 2030s to help you build this out. So this is like a very scary period of a sense where like it's currently offense-dominant. We want to make it defense-dominant. I guess the solution is to make sure these things are not too secretive and to maybe show them off and have lots, like my view is you want lots of people using intelligence to challenge everything, not just having like any single group doing secret. We very much agree on that note. So Anthropic is generally like very admired. Also like any company, I compared it to Microsoft in the 90s when you become like the champion, people get jealous and attack you. And like sometimes those attacks are because they're jealous and you have lots of aura and you're dominating. And sometimes there's certain things are legitimate as well. And those attacks, a lot of my very smart friends have impression that there's some kind of regulatory capture strategy is trying to do that's not a good thing and it's a bad thing. Like I'm curious, we could talk more about like the details there, but like how do you react to that? Like you know.

I think there's probably two things that are worth talking about here. I think one is the sort of rise of the regulatory capture narrative tied to the open weights discussion that's been going on. And then there's also probably just the like, if you were trying to do regulatory capture in a very self-serving way, would you have done it the way Anthropic has? I think on the first one, it's not unreasonable sort of if you're not in the details of what's going on to interpret something to the effect of open weights is sort of an important public good technology that we want to keep around. It is in some ways a natural opponent to parts of Anthropic's business. And Anthropic wants to use any regulation it can to sort of push around the people who are developing open weights because historically open weights or open source have been communities that are smaller, less well-funded, and have a hard time navigating the regulatory system. I think that's not what's going on with open weights here. If you look who the people who are pushing and developing open weights models and serving those models are, it is largely the most gigantic corporations in the world. Right, in the US, it's people like Nvidia and Amazon and Microsoft who are like pushing the frontier of open weights forward. And then even with, for example, the Chinese labs, these are extremely well-funded, more often, more and more often now public labs that are pushing this. And so these are companies who are, even though these are open-weight models, these are companies who are very capable of navigating a regulatory environment. And so I'm not particularly concerned about their, you know, having some regulation of models both open and closed being something that slows down the developers of open weights in a way that's disproportionate to the developers of closed weights. I think the other part of this is just like more directly, if you were sort of doing regulatory capture in a way that was meant purely to advantage you, it's very unclear to me that Anthropic would have done the set of things that it's chosen to do. I think that like if one very good example of this, and there are probably others, and maybe Schulto has some other things in mind, is what happened with Fable, right? If you think about Fable or Mythos before that, from a pure economic motivation standpoint, Anthropic had this model that was far and away the best model in the world. Nobody had a model even close to it at the period of time. And Anthropic basically made a first decided to go work with the government to roll it out in a way the government was comfortable with when it could have chosen to take a multi-month-long model lead and probably like put to bed any questions of like could other labs be competitive with the company. And this was just a like decidedly bad economic decision for the company, but one that was sort of aligned with our values about how we think models should be deployed and regulated. And so it's not a strategy of regulatory capture for economic interests. It's a strategy of we think the government has a really important role to play in how critical and possibly dangerous technologies are developed and deployed. And we think it's our responsibility as someone at the frontier to make sure that they're informed of that and that they participate in that process because otherwise who's going to represent.

But it's interesting too because obviously there's like a lot of people in the government. Actually, there's quite a few smart people in this government. There's a lot of people who don't know what they're doing at all as well. And so you'll end up with a silly circumstance where, for example, maybe you can't say it but I can say it, which is that like Anthropic would, from my understanding, would have had more cyber capabilities that people could have used for defense, but then they were nerfed in order for the government to let it go out, which I thought was a mistake. I think that they are leaning into the classic American public-private partnership that has been such an important part. Like, we are not built in a way that the government has to be the most knowledgeable, competent body about every subject in the world. And what we want to be able to do is build a great public-private partnership where the government knows how to engage with Anthropic, with OpenAI, with Meta, like whoever else it may be, to understand and then decide how to regulate.

Some of the tech leaders I know who are more skeptical are saying, just to put it out there, like, oh well, Dario is trying to scare everyone to create more rules to slow things down because he's going to be one of the people who owns that process. Like, what's the answer to them?

So far we've slowed ourselves down much more than anyone else. And in fact, if you take a look at the discussion around Demis' proposal for the sort of closed model providers — OpenAI, DeepMind, Anthropic — to coordinate in a way that they can actually have basically thresholds, like safety thresholds that we have to meet before we can release our models. This is us discussing slowing ourselves down because of our principles.

Now I think also, by the way, we should clarify what our overall stance is on open source. Just what is that? What actually is it? At least personally, we're very supportive of open source. Like, we genuinely believe that it's a very good thing for this to exist in the world. I learned how to, you know, do ML on open source models, right? We also believe that there are a lot of risks and there will be increasing risks in future.

So what we believe is that basically there should be thresholds for what society at large feels they're comfortable releasing. You know, if a model can do cyber attacks, what threshold of capability are you comfortable with? And we believe that closed and open source models should have to meet the same bar. Other than that, people should be able to do what they want.

Is it worth even trying to do that if China's just going to be releasing random stuff that everyone could use anyway? Like, what's the, you know, and similarly, what's the probability we overregulate at a fair and then China doesn't and then just there's a problem anyway and or they're ahead? Like, how do you think about that?

I mean, the ideal world is one where we can actually work with China such that they — I mean, they also don't want cyber attacks on their country. I think they want cyber attacks on us though, is my impression. Maybe they don't want cyber attacks on themselves. That's fair. The ideal world is one where we can work together with them on that. But that's tricky, right? That's like, that is going to be one of the most difficult parts of the next couple years is AI capabilities going to keep racing ahead.

This is going to make a lot of people very uncomfortable for many different reasons — for bio, for cyber, for employment. And as a result you're seeing already, for example, like pushback of stopping the data sets.

And this is one of those worlds where, or one of those like debates where it becomes really tricky to pause unless you can arrange, unless you can sort of coordinate with the other parties and trust the other parties to hold to that.

China's not going to pause though. So right, so is there any scenario you just slow America and let China go ahead? I don't think any scenario that we're comfortable with would involve slowing America and letting China go ahead. Yeah. Which seems to be the only option under our control right now. Yeah. For any slowing scenario, it would have to be fully coordinated in a way that we fully trust every counterparty. Basically, there's no way we'd slow America.

You're surprisingly, relative to other people, like more for open source. It's a good thing it's affected your lives positively. Some of this open source stuff seems to be coming from people stealing it from you, which I would be annoyed by if I were you. And it's called distillation. And a lot of people say, well, why don't they just stop it by being really smart with their AI and watching everything? And so why aren't you able to stop distillation? How do you feel about it? What's happening with that?

Yeah. Well, on the one hand, distillation — let's, there's a couple things to talk about here. One of those is like why are we against distillation? Like, is distillation good or bad? And one of those is why can't you stop it if you think it's bad? Let's just close off the second, which is there's a bit of a cat-and-mouse game, right? Where actually people want a lot of access to their models. They want to be able to inspect the outputs. They want to run on their local computer. They want to, you know, look at the thinking traces. But every additional bit of access you give makes it easier to distill. And so there is this tension between how open your tool and product can be and how easy it is to distill in effect.

And then I think on the first bit, which is like, you know, why should we be unhappy about distillation period. I think the way I often think about it is this, right? You want AI to make progress forwards because we think getting smarter and smarter models are going to do all of these great things for the world. And distillation is not a method that's capable of bringing you past the frontier of current — just copying you. It's a copy. And the problem with this is that getting to the next frontier requires enormous upfront capital investments. We're going to be in regimes — forget about today — eventually we're going to be in regimes where there are 10 billion, 100 billion, trillion dollar training runs. And you can only support that kind of R&D investment if you're able to then make money selling the thing that you invested in. And the problem is if everyone else is able for a fraction of the cost to take your thing, copy it, and put out a copy in a week or two, they will actually be an economically advantaged competitor against you because they don't have this big R&D load. And so if you want progress on the frontier, then you want to protect the sort of IP and rights of people trading these models.

I think people also — just to say one other thing — often have compared this to sort of like drug development, for example. And I actually think AI is very interesting because I think it gets you a lot of — like, protecting against distillation gets you a lot of the good things about how we protect IP in drug development and avoids some of the bad things. So what are the good things? Like, we get more drug development in the US as a result of protecting IP from — obviously I can't invest in drug development if I can't make money on it. Exactly. Some of the bad things are that like costs are extremely high as a result.

What's interesting about AI is unlike in drug development where like you get this protection for like a decade or more — it's 20 or 30 years I think — in AI like intelligence is incredibly deflationary because the shelf life of the frontier is like measured in months. Yeah. And so you imagine that like every 6 months, yes, the frontier is going to be expensive, but everything behind the frontier which felt frontier 6 months ago is going to be made abundantly cheap by the fact that you push the frontier forward more. This is sort of an artifact of how trading goes. So I think it gives you a lot of the good and actually like avoids most of the bad in sort of regimes that we've been in.

And this, I mean, you can very concretely see this in the cost of a given level of intelligence. You can look at heaps of — Evidash is one on Artificial Analysis that shows this exact trend line gets roughly 10x cheaper every year. And it gets cheaper, it gets cheaper after year. So this incredible deflationary extent — and by the way, this is happening without distillation. Without distillation. Just because of the natural progression of the technology. No, of course. And it's just intuitively unfair for someone to be able to copy and would break the whole system if everyone did it. Yeah.

Maybe a follow-up question to you. You'd be like, well, hey, isn't that a crazy business model where like your thing gets obsoleted? Well, you know, in some respects after like definitely a year, if not six months, the frontier seems like it has to keep moving forward for you to win. And the frontier feels like it has to keep moving forward if you win. And I think this is a very important part of like our general worldview which is that actually AI has gone a very tiny amount into the global economy. It's gone, you know, like the sum total of AI revenues somewhere north of 100B, right, into a tens and tens of trillions of dollars economy. So, and that's assuming the economy won't grow dramatically as a consequence of this technology which we expect it will.

And so there is this frontier in many ways you can think of like the level just behind the frontier, like the lagging edge gets commoditized in many ways, but then the leading edge has even more value. There's a huge amount more value at the leading edge. We often talk about this as sort of exponential economic returns to intelligence where each marginal unit of intelligence that you're capable of is worth exponentially more than the unit that came before it.

And this is actually like — we've already seen this occur. So if you look at something like coding, right, like the first unit of intelligence in coding that was sort of like useful at all to people was tab autocomplete in coding. And you moved from this into the next marginal unit of intelligence where you made agentic coding a reality in your terminal with Claude Code. And this was like a 100,000x more valuable capability. And so even though tab autocomplete got like commoditized behind it or something, this is like you can't even really have a company based on tab autocomplete at this point. These like next incremental units of intelligence are just extremely extreme.

And this was very unintuitive to me when I was like thinking about Anthropic a couple years ago because my intuition from like everything else I experience as a human being living in this world is that the exponentials don't go for that long. So I think, okay, well, this is really valuable for a couple years but at some point there's not like an exponential going. In some ways it requires you to be extremely optimistic, right, because you have to believe that like you're going to go from tab completion in coding to agentic coding and but what could be more valuable than agentic coding? Oh, like maybe we can like cure cancer. Right. And even between those it's like you have a full-on drop-in software engineer that's better than any software engineer in the world. And that feels extraordinarily cheap. And you have an executive that helps you all sorts of things.

And so by the time that people often talk about, oh, you know, like won't this just lead to commoditization? By the time that any of those worlds you're even in like a hope of like complete commoditization, the entire economy is radically transformed. Literally the entire economy by that point is being done by AIs. Both intellectual and physical. So yeah, and probably is a much larger economy. And if Anthropic is driving that forward, that's obviously worth trillions of dollars. So, yeah. Exactly.

All right. So, we're all rich. Just kidding. So, there's obviously a lot of concerns. But you're an optimist. So, so, so you're concerned like why are we pushing this ahead so fast? So, remind us again of that. What's going on here? What's the like what are we fighting for? Yeah. What are you fighting for?

So, to come back to like why we're concerned because we're going to get something that's a drop-in for a human, right? It's going to be as good as a human at everything they can do on a computer. It's going to be once we have robots as good as a human at everything in the physical world. But at the same time, that unlocks massive like radical improvements. It means you can literally take technology that would have required the next couple of centuries to develop with our population and compress that into the next decade, couple decades. I'm already seeing this with like aerospace where there's airplane design stuff that's going so much faster that there's probably really advanced planes we would have gotten in like the 2040s that we're probably going to start seeing I think in the next few years. It's really cool. And so you can keep increasing and potentially doubling the productive intellectual and physical capacity of humanity in such a way that allows you to truly get to a post-scarcity world like this is a world where literally everything is reduced to the cost of energy, where building homes is reduced to the cost of energy.

Yeah. The middle class will have giant homes if they want. Giant — you could literally have whatever you want in that world. Yeah. You can go to the mountain and do like a 500,000 foot thing and it's a middle class thing. Exactly. Exactly. That would be a middle class thing in the same way that things we have today are unimaginable to even the kings of like centuries ago, right? Where we've cured every disease where people actually probably we've cured longevity. And so you're in this world where literally the abundance everyone on earth has a level of abundance available only like not even to the richest people in the world today. And that's the world we're fighting for.

But it's a very tricky world because it means that you need to like one, you have to sort of chart this course through to it and two, you have to figure out well how do we even like share the proceeds of that world in such a way that you don't end up with massive inequality, right, that the returns don't just flow to people who happen to have capital in the pre-existing world and instead everyone if they so want to can have that kind of abundance and so there's a big social political question of like how do you share in the benefits. Yeah, there's a lot of mistakes you can make by assuming we're already there though, too, right? Because the dignity of work right now is really and we're not there. We're not even close to there, right? But if we invest in the technology in the right ways, then we've got a shot of getting there. And so we think that like basically let's like draw the trend lines out a little bit from today, right? Over the course of the last four or five years, we've been 2xing or 3xing the amount of compute capacity devoted to AI every year. And an interesting question will be and so I think that's roughly together the hyperscalers are spending about a trillion dollars this year on capex related to AI. A very interesting question will be will that trend line hold, right, and so will it go to $2 trillion next year and then $4 trillion in 2028 and if that trend line broadly holds and you can sort of maybe expand that to encompass the broader robotics industry and this kind of stuff, then that means that in the early 2030s or something like this you start to get to the point where you're actually effectively doubling the GDP of humanity in the early 2030s, which again, a little bit of a ridiculous concept, but requires a lot of things to go right, but it seems like the productivity growth could just start. Productivity go really. I mean, yeah, and it feels like it should even be in the next couple years. I think something we really notice like in the statistics. Kevin Warsh is the new chairman of the Fed. He is I don't know if he's AGI pill, but he's very bullish on AI productivity right now. Very smart friend of ours. Like it feels like we're going to start seeing that a lot more, but then that'd be like the kind of it feels like insane to say, but it could just accelerate much. It just accelerate even more. And I think like one of the interesting charts I saw recently was this chart of employment actually in the Philippines which went up. I was surprised by this. I thought the BPO was going to get really hit there. BPO stands for business process outsourcing. They're doing like the low-end call centers but and yet they're still growing. And yet they're still growing and actually software engineering employment is up as well. And I think a lot of this is because all of a sudden people are dramatically more productive than they were before. And a very interesting trend line would just be like how long does this bump continue where people sort of you're pairing in the same way like there's so much more to do. There's so much more to do. You're using this stuff with this positive sign. I well at least in the short term it seems that way. But I do think it's important that we don't convince ourselves too much that it's just like all going to be fine for it's going to be disruptive volatility right now. Right now, as Shto put it, you're in this it's actually possibly good moment where all your employees are still very important parts of taking advantage of AI. Yes. And as a result, what happened is you got an enabling technology that made every one of them more valuable and so you wanted to hire more of them. But at some point, our belief is that we're going to go from a world where you need a human paired with an AI model to sort of get all the value out of it to a world where you don't. And I think that's the world where we start to get very nervous about the consequences down. This is like with chess for 30 years it was like the best chess player was a human plus the machine and now the machines are just crashing and we think that's just a much shorter time period. Yeah. I think one thing that was a big update for me over the last three years was when I first started the maybe over the last five years when I first started thinking about sort of the rate of AI progress. I believed eventually we would sort of get to a place that looked like where we are today or where I think we'll be in a year, but that it would take longer. And in large part, I thought it would take longer because I thought that scaling the data required to do this was just going to be an extremely extremely time and dollar expensive pursuit. And the thing that happened that even by my sort of lofty expectations at the time of AI progress was unexpected was sort of that the ramp of the revenue and the scale of these businesses of Anthropic and OpenAI was so fast that all of a sudden it became completely plausible for the people developing this technology to throw money at the problem of blasting through much faster what we thought a lot of these bottlenecks might be. The compute ramp is another great example of this. It would have been inconceivable to most people 5 years ago. We would spend a trillion dollars in a year. It was inconceivable to people a year ago, right? It was a year ago. And so it is kind of cool how it worked out how all these giant tech companies always had like hundreds of billions of cash they were hoarding and we kind of we criticized them for it and now they actually were able to use it now right at the right time. It really worked out. It does seem like we are in a timeline where a lot of stuff that actually has is sort of very large interaction term with each other is happening at once some good and some bad right we talk about the move into space for compute it's like kind of interesting that like AI is coming around at roughly the same time that SpaceX is coming into its own very worked out very well it like the timing of that was like really interesting you could imagine if SpaceX had taken 10 years longer to get here that AI timelines might have existed. It's the only things to Elon too. He was like crazy enough to start it like way before anyone else. On the other hand, by the way, you have some much more concerning things that are happening. Like I don't think it's great for the world for all sorts of reasons, including race dynamics, including that AI I think fundamentally is a technology that can benefit authoritarianism more than most other forms of government. That AI is arriving at a time when we're entering one of the like certainly the most bipolar world in my lifetime. Isn't that linked though? Like isn't there something about like social media and like how it uses machine learning and how it uses improving AI that does polarize us? Like is it something about the attention economy? I think when I'm saying bipolar, I'm really referring to sort of like the China US global picture. You don't mean the left right polarization US. No, I mean that's probably another thing that's worth thinking of. But I really do think I would have updated us at least slightly more positively on like the prospects for AI going very well if it was coming into the world say in like the early 2000s when you have this very unipolar more unipolar world it's more safer. Yeah. Yeah. It's like a tragedy worthy of a sci-fi novel that the most valuable resource in the universe is produced in this little island between the two world superpowers. It's very is very it is very quite poetic in some respects. And so I mean overall there's a lot of optimism in terms of where this goes for humanity if we get through this period. It seems to be the high level thing. One other question I want to ask I think a lot of our listeners are wondering about is it's one thing to have like extraordinary wealth but there's also things we care about in our civilization. There's virtues there's families there's like traditional values which a lot of people like base their lives on. There's a dignity of work obviously, but there's all these other things around that which is like our traditions that make us who we are and I think there's some concern that because this is happening in San Francisco where those traditions maybe are not as valued on average that is there some kind of agency of the people in charge where now they're just going to like remake the world and like get rid of my like traditional sense of whether it's Christianity or whether it's like you know monogamy or whatever else like and like are there institutions we can be building to help preserve these things? How do we think about that from the perspective of being the center of San Francisco and changing everything? I think one it's interesting in San Francisco we actually expect a resurgence in importance of these institutions because as work decreases in its primacy to people's lives and as like pure capitalism decreases in its primacy to people's lives because we sort of progressively have more and more and more abundance. we think that the important parts of people's lives are going to be tradition and ritual and providing values to other people and your local community and sort of those bonds. I'm actually yeah I also think like counterintuitively sort of many many sort of intellectual pursuits and the sort of pursuit of figuring out how to become someone who knows how to think about the world and is actually something that perhaps grows in importance just not in a sort of economic sense right I think that in a world where you want for nothing it's sort of the last thing that's left to sort of entertain yourself to feel part of community and all of these things. So you guys are both young and very successful. Let's just say Anthropic keeps working out. You're obviously everyone there is going to have resources and the whole world's got more resources. Like have you thought about building institutions to help with these things? There's something you're passionate about there to create. Yes. So, I mean, Nick, you've got a great example with the early childhood stuff that your dad's working on, but I think it's this is a very big topic of discussion at Anthropic right now. You know, philanthropy is a very important part of Anthropic's culture. Huge amounts I think there's a great blog post on this called like new wave of American philanthropy or something like this. where people talk about how it's the philanthropic outpourings of OpenAI's foundation and the employee base of Anthropic would be like hundreds of Arc Institutes or like tens of like Gates Foundations or something like this. No which and there's all kinds of things that people are thinking of one thing foundation that I've donated to is a program that intends to try and end all viral disease. And so like a lot of people are focusing on things which are health related. One of my colleagues has donated to basically a not-for-profit that's focusing on trying to figure out what the right way to do this like how to handle labor force impacts basically really study them figure out what's going on what should we be actually concerned about what's the real picture what's the data on the ground and what policy should we be advocating for so that everyone makes it through this like transition in a way that they feel supported and they're better off basically on the other side and so yeah I mean I am certainly heavily influenced I think by my father in this regard who spent probably the last 15 or 20 years of his life or at least the last 15 or 20 years focused on education. I just talked about how I think you know education while the things we may want to learn I think will change a lot. It's unclear to me that you know we want to be going out and getting PhDs in highly technical fields 20 years from now. I do think the sort of pursuit of knowledge and the joy of learning and the ability to engage intellectually with one another and frankly to understand the world that we're living in which is going to be a very confusing world for most of us if things play out in sort of the way we've been discussing is only growing in importance. I think that one thing that many people in AI believe that I actually think they get wrong is that AI is just going to fix education. And I think that if you want to understand like if you care about the education of like everybody and not just say like how do I give a really great education to a child with two parents who are in the 1% and who are highly educated themselves. I think that there's two really important things to understand. The first one is your education is a like deeply deeply compounding thing. Yes. And so like a good example of this is the number one predictor of how well a student will do in math is whether they can read at grade level at the end of third grade because the problem is that when you get to the end of third grade this is where you transition from learning to read to reading to learn. Yeah. And so and there are many things in education that are like this especially because children have and many humans have this psychological thing where when they perceive themselves to not be good at something they lose motivation to keep doing it. And the problem, if you believe this, is that there's one thing that at least so far AI seems to be extremely bad at, and it is like computers cannot hold the attention of young kids. In fact, they seem to be an incredibly distracting force. Well, that could be a whole new thing you train them on if you want. You got to figure out the game, although I'm not sure. And so while I think that there are parts of the process of educating children that AI will help in extreme ways in particular sort of making sure we're teaching kids the right thing for them at the right time. I really don't think that sort of the there it's going to be this panacea that we just like get really smart AI and like all of the problems of education going. Well, there's a lot of smart people working. I agree is really critical. So guys, flash forward to August 2028. You're looking back at this podcast. What has to have happen for you to say like, "Wow, that went something went much faster than I expected." Much faster than we expected. We expect things to go pretty fast. Much faster. I mean, I think timelines to get data centers in space is a big one. I think there's like wide error bars on this depending on who you ask. And that's a really big part of it because there's so much more you can do up there. The compute ramp once you have that will be even faster to go faster and that's the main thing we need. Not going to surprise you. Maybe. I broadly expect something like TerraLab to be online by then but TBD on the actual like ramp. And so some question of how much ah no okay I would be interested to see how many humanoid robots we have in homes in 2028. So, I think this is very likely to be like a whole bunch of early deployments, you know, maybe tens of thousands of robots in 28 in homes doing laundry and like some basic cleaning and this kind of stuff. But it could also be a fair bit faster than that. Yeah, we just put out an episode with Generalist. It's like it's pretty amazing how good the world models are getting and it's faster than I expected. Like it seems like that could be that could be the thing that makes everything go a few years faster. Exactly. And then once you get, you know, as I said before, once you get robots, it really compounds like progress goes fast. Yeah. I think there are some other things. I mean, I think a lot of the things here are data related in some ways. Like I think one of the slowest or sort of like the biggest regulators of progress in AI capabilities are how fast can we scale the data in the domains that we care about. And I think that this has already gone faster than I expected for sort of a reason I discussed earlier, which is there's been a lot more money to throw at the problem than I thought there would be because of what lab revenues have become. But I think my expectation is we will be slowed by sort of like some rate at which like humanity can produce this data. And if you told me, for example, that we went on a crusade of like building tons of labs to like produce biological data for training models or something like that and that started today and it was in full swing a year from now, I probably would update my timeline sooner. It's fun. Now we're focused on obviously all these types of data for this very specific task like coding and math and everything else. I wonder you think one day we'll be working on data for like shaping virtuous people or good character or mental health or something like that. I mean obviously it's much more complicated but could there be these other kind of interesting really complex problems the AI solves in the future? Those problems I kind of want humans to solve to be honest. Well could wouldn't we use a tool to solve the problem though? I mean definitely I think AI will help us in those discussions right? But I think that falls into the broad category of data which I expect to be like both quite tricky to scale and extremely dependent on the person and just like just very that comes later than we solve you know robotics you're still somewhat human it sounds like there's still room for us in these I guess like one thing you know would I expect an AI to win a Pulitzer Prize in 28 it's like probably not well what if it didn't we didn't know we were giving it to the AI yeah that's true would I expect it to win a Fields Medal in 28 quite possibly, right? Extremely likely. Nobel Prize also not crazy. Very feels very likely by before the end of the decade. Well, it's exciting to chat with you guys. We definitely have some challenges to work through, but I'm feeling very optimistic about the future of humanity if we get this stuff right. Me, too. And thank you for joining us. Thanks for having us.