Anthropic's Jared Kaplan on the Future of AI Agents l TechCrunch Sessions: AI

Yeah. So I guess when we first developed Claude back in 2022, the motivation for us, we were just a research company, we didn't have a product, was to build a dialogue agent because we figured a tremendous amount of what we do can be encapsulated by talking to someone, asking them to do things, and sort of seeing how that goes. That obviously turned into Claude, which is a chatbot you can go to claude.ai and talk to it. But I think the vision was always that as AI advances and gets better and better, the challenge would be to have AI do tasks that take longer and longer, are more sophisticated. Fundamentally, the way that we get things done isn't just by giving you the answer, but it's by using tools to go out in the world, learn the information you need to do the task, do the task, iterate. If you're a coder, you write code, you run the code, you see if it works. If you're like me, it doesn't work the first time and you have bugs, you see what those bugs are and you fix them. I think the natural growth in AI capabilities and AI utility comes from AI agents that can do more and more for you. A paradigm where you just ask a question, get an answer, and you're done feels very limiting, and it just isn't the focus that we see going forward.

Yeah, I would say that to a large extent, we are trying to help businesses and individuals integrate AI and use AI in the most useful ways possible. I think that having a static platform like that does feel a little bit limiting. That said, I don't think we know where AI is headed, so giving people, giving developers tools to integrate AI and use it in a lot of different ways feels like the way forward. I think Claude Code is a cool example of that. We first built that as an internal tool that engineers and researchers at Anthropic were using to make themselves more productive. Then we shipped it externally. The main use is developers, but I even see people sometimes using Claude Code to do things that are surprising, like almost like an operating system. I think we're all experimenting with what the best way to use AI is, and we're just going to keep experimenting and hopefully keep empowering developers to experiment and build with Claude.

I think the way that I think about a lot of our business is, a lot of it's built on the API. We really want our customers to be as successful as possible. One reason why, as I said, we were playing with Claude Code internally, one reason we decided we'll ship this, we'll make this public, was that it felt like it was at least a demonstration of how far you could push agentic capabilities, and that's something that we were experimenting with and leaning into on the research side. I think our goal isn't to compete with our customers. Our goal is just to make sure that people are exploring the possibilities with AI. I always encourage people to experiment with building applications and uses of AI that don't quite work because we found at Anthropic that since we expected AI to keep getting better and better, and we thought there were safety questions and concerns around that, which is why we're so focused on safety. But because of that trajectory, because AI is getting better so quickly, I think a lot of things that don't quite work with the model right now are going to work in three months or six months, and it's a way of really getting ahead of the curve.

Yeah, so my understanding is that with Windsurf, you can actually bring your own API key and continue to use Claude, right? But it's much more expensive and complicated.

Yeah, it might be more complicated. I mean, I think as you alluded to, we were really quite constrained in terms of supply. We're hoping to greatly increase the availability of tokens, we want to keep them flowing for Claude over the next couple of months, but we really are just trying to enable our customers who are going to sustainably be working with us in the future.

I guess I don't expect it to. I mean, there are many, many companies training their own models for all sorts of different purposes. I expect we'll be working with Cursor for a long time. I hope to be. They've been a great customer, great partner, great tester of new Claude models. If you look at us versus a lot of other AI companies, we're quite invested in our API business. We want people to be able to build with and on top of Claude, and we're definitely not trying to limit that. We're not trying to compete with our customers; we're trying to empower them.

Yeah, so I think scaling is continuing to pay dividends on both the pre-training and RL side. We've seen a lot of advances in the last year or two on RL. The way we train AI models is that we do pre-training, where we teach models to imitate and understand predict the next word in human written text and other data, and then we fine-tune them with RL, both with human feedback and with AI being trained to do things like write code that passes tests. I think both of these show clean scaling laws. The work that I did and many others contributed to maybe five or six years ago now was on the pre-training side and showed really clear empirical trends: if you make AI models bigger, give them more data, more compute, then they get better. I think that's continuing now. There's a question of is there enough compute, is there enough data? For now there is, but I do think that eventually those will be constraints, certainly data. On the RL side, you can go a lot further, and I think RL is much less fully tapped. On the RL side, you're really training models with reinforcement learning to reinforce models doing useful, safe stuff and not making mistakes. That's the kind of thing you're going to want to keep scaling because you want to make AI more useful, not just better at autocomplete. There's a lot of evidence from the last five or six years that you can get clean log-linear scaling with compute in RL, so I think that's a major source of investment, as well as using new techniques like advancements on constitutional AI and more complex agentic environments to train AI to get better and better.

Yeah, I guess the way that I think about it is that for hard problems, reasoning is obviously a great way to get more capability out of the model. You give it more compute at test time and you can get a better response, so I definitely think that's going to be an ingredient. I think that in some ways, agentic capabilities might be even more important. Maybe they're just both important. In the sense that when we do tasks, when we do work, there isn't a lot of the time that we step back and think for an hour about what to do next. I think a lot of what we do is we experiment, we try something, we see what our environment tells us: did we make a mistake? Did we write code that passes tests? Does the paragraph that we wrote make sense or can we make it better? So I tend to think of scaling agency, search, tool use as really important for that as well. But definitely reasoning can allow us to take models that aren't necessarily at the frontier as pre-trained models, but make them better at solving hard problems.

I think it's scaling very rapidly. AI is on and has been on an exponential for a while, and that's continuing. You see that in the news with the fundraises in AI going up and up, because the value unlocked by AI as AI models get smarter is scaling with it. There are more and more use cases, customers can integrate AI more, and that means the feedback loop is even faster. Generally, we're always every year increasing our supply of compute by a significant integer multiple. That's just continuing and continuing. We've just basically started to unlock the capacity on our new Trainium 2 cluster, which is really big, and continues to scale. That's what's going to allow us to unlock more compute for customers and for training the next generation of Claude 5.

Yeah. It was sort of announced at a joint event that Claude is contributing to a lot of Amazon products, including Alexa. Generally, as we were discussing earlier, we're excited about Claude being able to power all kinds of products from many different companies and developers.

I can't comment on ongoing work, but we're always open to working with a lot of different companies and partners. We work with Google as well. Google's great too.

It's a good question. I think it probably depends on all sorts of different factors. For one thing, if you're a startup, you can move extremely quickly and ship and iterate. But if you're a giant company with millions and millions of users, they tend to move a bit more slowly. I think a lot of voice assistants generally aren't just something you're chatting with; it needs to be integrated into some larger ecosystem of tools and other applications. You want to be able to call something that accesses your calendar, maybe you want to be able to turn the lights on and off in your house. Getting really high reliability across tens, hundreds, thousands of different tools and applications that are integrated into these assistants is complicated; it requires work, and the expectations are very high. You don't want to disappoint your customers. Those kinds of integrations are complex, but it goes back to the importance of tool use and agency. Behind the scenes, a voice assistant maybe needs to be an agent, and maybe that's another reason to focus on those kinds of capabilities.

My understanding, if you read the op-ed, the op-ed in many ways is saying positive things about a lot of the work that the administration has done in trying to make AI safer, prevent authoritarian governments from getting access to AI. I think this is really just a question of states' rights. If one state wants to regulate self-driving cars, it makes sense for them to be able to do that. A self-driving car company tests their car in Arizona and then wants to deploy in Michigan, maybe Michigan is worried that car doesn't know how to drive very well in the snow. That kind of use case I think could be blocked. I'm not a policy person, but I think many folks... I don't think Dario is the most high-profile tech CEO to criticize the legislation, but we really want governments to be able to move quickly to respond to AI. AI is changing very rapidly, and states are one of our nation's laboratories for experimenting with different possible forms of engagement with industry and regulation. To enable government to act with the speed it needs to given the speed AI is moving, we think flexibility is great.

Yeah. We have a Responsible Scaling Policy that says we want to think carefully about what kinds of risks there might be from AI in the future. We want to evaluate pragmatically whether AI models really pose those risks, and then put in place various mitigations. Something that you could ask of AI companies is to transparently discuss what risks they're considering, what evaluations they've done, what kind of safety testing they've done before deploying their models, and what kind of mitigations they have in place. At a bare minimum, that would invite the public, regulators, academics, other AI developers to understand what the risks are, what evaluations have happened, and how robust the mitigations are. I don't have a specific policy proposal, but I think that kind of transparency seems like a good first step. Google and OpenAI also have similar kinds of policies, but it would be good sense if that was something that all advanced frontier AI developers were doing. I think it would be reasonable for government to ask us to do that and not let us off the hook.

We're trying to work closely with state governments, the federal government, etc. I'm not on the policy team, but we're generally engaged in helping them advance American interests when it comes to AI.

I can't comment on new, ongoing litigation. I can say that Anthropic puts a lot of effort into carefully obeying requests in robots.txt and industry standard practices.

As I said, people can prevent Claude and Anthropic from scraping their data via robots.txt, and we respect that. More generally, I think AI training is fair use. It's not copying, it's not reproducing data, and that's the basis of why we think it's reasonable to train AI models on publicly available data from the web that developers have not requested we avoid via robots.txt.

...we obey robots.txt. If we've been asked not to scrape data, that's our policy.

Well, as I said, we obey robots.txt. That's our policy.

I don't think anything has changed, and I think this goes back to what you said about transparency. The scenario you mentioned was not a thing that happened in the world; it was from us doing hundreds, thousands, tens of thousands of examples of red teaming. Our team actually calls it 'weird teaming' where we try to find all sorts of weird scenarios where our model might do something problematic so we can flag it and improve it in the future. Then we wrote a 120-page system card where we documented all kinds of these examples in order to encourage others to be transparent, to show what their models are doing, and to do their own testing. I think that in being transparent, it opens us up to more criticism. We do a lot of other kinds of safety testing with respect to our Responsible Scaling Policy as well, and we have mitigations in place there. That may open us up to more criticism, but we're going to keep doing it because we think people should understand how AI models are behaving and be encouraged to do their own tests on other providers as well.

Yeah. I mean, I think AI is getting more and more capable, and so safety risks are becoming more and more salient, more and more real. I think that I'm generally proud of us being transparent, being kind of the first to have something like a Responsible Scaling Policy. I think a lot of our competitors want to do the right thing, but if we do something like that first, it sets a standard that others can experiment with, iterate on, and imitate. I think we'll keep doing that. I'm happy that we've done it so far.

Cool, thank you.