And so I think like these always-on, asynchronous, event-driven agents—that will be a really big productivity unlock. And especially in enterprises, there's so many events that are just triggering, triggering, triggering. And so if you can have agents listening to those and firing off, I think that will be a massive gain.
Welcome to the NVIDIA AI Podcast. I'm Noah Kravitz. Our guest today is Harrison Chase. Harrison is CEO and co-founder of LangChain, one of just the most incredible stories of this whole generative AI era that we're in. As Harrison will get into in a minute, LangChain was founded about three years ago, over a billion downloads. The whole point is to help developers build applications with LLMs now getting into agents and agentic frameworks and all that great stuff. So we're going to get into it in a moment. Harrison, thanks so much for joining the AI Podcast, and welcome.
Thanks for having me. Excited to be here.
So, let's start about three years ago. You started LangChain with this premise of building tools so developers could build apps with LLMs. What did you see back then that either others didn't, or even if they did, you saw it and just thought, this is where things are going. This is where I'm headed.
What got us really interested was seeing the applications that people were building on top of LLMs and the systems they were building around the LLMs in order to power those applications. And those systems had a lot of similarity with each other, even early on. And even early on, we could tell that they would get quite complex over time. And so a lot of what we built is tools to help people build these systems, these agents, which we now call agents, around these LLMs, and figuring out what the common patterns are, and how common this tooling is, and making it really easy for anyone to do so.
And so now you've coined this, I don't want to say this term, but LangChain, you talk about deep agents. So what is a deep agent? And then maybe we can get into talking about the enterprise. And why would an enterprise in particular care about that distinction?
Yeah, so about a year ago, we saw a few really interesting things. So maybe even backing up like, you know, three years ago, great, like LLMs, you want to connect data, you want to connect these other things to them. Fantastic. How do you do that? Turns out, it's really hard. And the best way to do that for different types of agents was actually pretty different. You would build different scaffolding, you would build different workflows around the LLMs. About a year ago, we saw Claude Code come out. We saw Manus come out. We saw Deep Research come out. And under the hood, all of these had the same kind of general architecture. They were simple in some ways, they were now running in a loop calling tools, but then they also had common patterns of connecting to a file system, and having sub-agents, and doing planning. And so about nine months ago, we released, for the first time, Deep Agents, which is a library, and we've been building it ever since. And we've just continued to see this same pattern of giving the LLM more autonomy in this environment for interacting. This is what powers OpenClaw, for example, is this type of harness. And so Deep Agents is really this new type of agent harness that we think is really general purpose and that you can customize to do different things. But it's not like you're reinventing the scaffolding each time. You're just customizing it with prompts or tools. And so it's way easier to get started with and also way more powerful, because it's a simple thing under the hood, and simple is really good. And so Deep Agents is this general purpose agent harness, model-agnostic, open source, that we've been building for a while. And we're starting to see more and more agents build on top of.
So when you're working with customers and the enterprise in particular, and we're getting into these systems that are so powerful, becoming so powerful, in large part because they are autonomous to a larger degree. And as you said, they can do more now. The agents can control the screen and go off and do things with apps and such. What's the... what are your conversations like with enterprise leaders, and what's kind of the feeling around, you know, is it a tension between risk-reward, is it just the excitement for what the systems can do, and so there's trust in building these systems that give agents more leeway? What are those conversations like?
There's a lot of things. So one, not everything needs an autonomous agent. And so one framework we have, LangGraph, is really good for when you actually want to combine some of the autonomy of LLMs with more directed workflows and more control. And so, honestly, when talking with a lot of enterprises about deep agents, some of them are just like, we love LangGraph. LangGraph is better. We're gonna stick with LangGraph. And that's fine with us. We think there are different use cases and different things. But that's definitely kind of like one component that comes into it. Another component that comes into it is definitely just like, okay, great. Like the LLM's doing a bunch, but how do we know what's going on? And so another thing that we work on is LangSmith, which is observability and evals. And that's basically our answer for that. You... there's this really interesting thing about agents compared to software, where agents... the interaction space for agents is way more open ended. You can ask it anything, like text is infinite. If you have a UI, there's a bunch of different buttons you can click, and so it's much more constrained. And then also models are not robust at all—like you know, they're non-deterministic, and then you change one word, and the answer changes completely. So this is why we think observability is really important. And that's a huge thing that enterprises care about, and very related to observability is then evals. Because sure, you can see one thing that happens, you can tell why it goes wrong, but what if you wanna test how it did on like 10 different questions, 100 different questions? And so building up these eval datasets is a big thing we work with folks on.
And so LangSmith is the platform for building agents, as well as observing and evaluating?
Yeah, so the way that we think about the agent development lifecycle is build, test, run, manage. And so the build is all the open source. It's like choose your fighter, choose LangGraph, choose Deep Agents, choose another framework. All of our stuff works kind of modularly. But then this test, run, manage—that's LangSmith. So we've got a bunch of stuff around testing and evaluating these models. We have a deployment platform for deploying these at scale, and then we have observability and other things for managing them.
Let's talk about skill, or maybe you can talk about skills for a minute. When I first started playing with these tools... and I'm, you know, I'm not a developer, I'm just kind of a technical layperson, if you will, right? I love playing with these things. Back in the day when they first came out, maybe it was BabyAGI, whatever it was. I, you know, spun my computer right into the ground in an infinite loop. But when I first discovered skills, it took me... like there was a moment where it sort of took me back, where I was like, wait, I just describe it, and it goes off and it builds. And then, of course, it does, because that's how this all works. But can you talk a little bit about skills and about, kind of, that same idea of giving the agent the autonomy to write the tool and run with it? But how do you keep it in check and keep things secure?
Yeah, skills are a great way to package up knowledge and in other kind of like instruction sets and other tools for an agent to use. And so they started in coding agents, and a skill would involve basically a Markdown file with some instructions and then some scripts that you could run. And one of the things that's kind of become clear over the past few months is like coding agents are very general purpose in a lot of ways. And so this same idea of a skill as a Markdown file and then some scripts to run is really, really interesting. We see a bunch of different types of skills. Some of the skills are purely kind of like informational. So like you want to learn about something, great, go read this Markdown file. Other skills do things. And this is where it starts to get, I think, like really interesting. It could be a Python script that hits a URL. It could be a Python script that runs some GPU-accelerated compute. And so this also ties into the environment aspect. So when we think about agents, we think of a model, a harness, and this is Deep Agents, and then an environment that it runs in—a runtime for it. And so NVIDIA just released OpenShell, which is a secure runtime for it. And then the other thing that's related to the runtime is also like where it runs. Does it run on a Mac mini? Does it run on some GPU-accelerated environment? Does it run in the cloud? Right. And so those three components and being able to pick and choose what you need for different jobs is a big part of kind of like customizing your agents.
Was there a moment, or can I put you on the spot and ask you to think of kind of an aha moment where this, the idea of Deep Agents really clicked and in a use case, and whether it was, you know, something internal at LangChain you were working on, or maybe with a customer, was there kind of an aha moment where you were like, yeah, this is it?
I think so it started just by seeing really the three things of Manus, Deep Research, and Claude Code. This is the same way that LangChain started as well, just going to early meetups, seeing things that people were building, and seeing patterns. And so the first version of Deep Agents, just like the first version of LangChain, I hacked on over a weekend, and it was a weekend project. I'd been talking internally with some folks and being like, oh, like, you know, Claude Code's really interesting. Like, Manus, they've got some similarities. And so it wasn't until I had time to kind of like sit down on a weekend and hack some stuff together that we came up with a few patterns of what these similarities actually were. And then using it, I think the first thing we used it for was a deep research-type thing. And so we gave it access to a bunch of files, and we just put it in this like virtual file system and had to do some research. And it wasn't even really doing RAG. It was just grepping and globbing like a coding agent would over these files. And it worked fantastically well. And so I'd say deep research was the first concrete thing, but really the idea came came from just seeing a pattern and spending a weekend kind of like hacking on it.
You mentioned earlier the importance of, I don't know what the words you use, but auditability, traceability, being able to see how do the agents do what they did. Can you talk a little bit about evaluation-driven development, and how that plays into, and again, in the enterprise, building that trust in what the agents are doing?
Yeah, if you talk about trust in an enterprise, what does that mean? That means that the agent's doing what you want it to do. There's a few different ways that we see people getting that trust. Part of it is observability and traceability and being able to go into an agent run and see exactly what steps it took and exactly what it did. The other part where trust comes in is having these scenarios and running the agent over them and seeing how it performs and evaluating that. This is what we talk about as evaluation-driven development. You come up with these scenarios ahead of time. One common misconception here, by the way, is that you need like 1,000 scenarios for it to be effective. You could start with five. You could start with ten. It really doesn't matter. I think like creating these evals is a really good way to do like product thinking, about what the agent should act. Because this is another thing, like agents can do anything. But they shouldn't do everything. They should do like what you want them to do. And so being able to... being forced to come up with, like, hey, these are 10 questions that we expect the agent to get asked. This is what we think a good response is for each of them. This is what a bad response is for each of them. That's a really good kind of like mental model for kind of like coming up with what these agents should do. And then you can use that to drive all of your changes. So you change a prompt. You can run it against this benchmark. Did it improve? Did it get worse? And then this eval dataset is living over time as well. So as you release it to first like a small set of users, you might see them using it in unexpected ways. And then some of those ways you might be like, okay, maybe they shouldn't be doing that. Let's put some guardrails around it. But other ways you might be like, yeah, that's totally legitimate. We had no idea they would use it. Let's add some data points to our eval dataset. So when we go and change the prompt in the future, we can make sure that it's still good at these use cases.
Are enterprise customers open to kind of rolling with that, you know, oh, we weren't expecting this behavior necessarily, but it's good behavior. And so, you know, is there a sense of kind of experimentation? Obviously, in the AI community and the open source community, it's all about experimenting and sharing, and things are going so fast. Is enterprise embracing that at all?
The best ones do, in limited ways, and with a limited blast radius. They might roll out internally, for example. They might roll it out to a set of like alpha customers. They might roll it out to 1% of users, or something like that. There's definitely way more caution there than there is with gen AI-native startups. But building agents is so iterative, and the importance of this iteration can really not be understated. And so I think the enterprises that are... I think a failure mode for enterprises is you have some idea of an agent. You take three months to craft a bunch of examples. You take another three months to build the agent. You take another three months to get humans to look at everything. But the spaces just move so fast. The whole idea you came up with, there's probably like... there's just a better way to do it at that point. And so I think like you have to kind of ship. You have to learn. You have, this is another thing by the way, that no one likes the answer for, but like you have to, you have to basically redo your agent every nine months at the pace that things have been like with these agent harnesses. If you're using an agent architecture from like a year and a half ago, you should very strongly be considering looking at rewriting on top of an agent harness or something like that.
