Everything Gets Rebuilt: The New AI Agent Stack | Harrison Chase, LangChain

I think two things basically happened. Like the models got better, but then also we started to discover these like primitives of a harness that would really let the models do their best work, and we saw an explosion of people building agents.

I think the harness is the most important thing. The cloud models are great, but the harness is really what made that work.

Thank you for having me. I'm excited to be here.

Yeah. So I think a lot of the ideas behind the agents today were actually present in some of the early days stuff. The difference was the models just didn't work back then. So LangChain came out maybe half a month or a full month before ChatGPT, and one of the main things we added at the start was this idea of running an LLM in a loop and calling tools. And there's this great paper called ReAct which basically said to do exactly that. And you know, it worked for the dataset that they ran it on, which was like Wikipedia question answers, but it didn't work in the real world. And then in March, I think AutoGPT came out, and that was the same thing. It ran in a loop, called tools, gave it a bunch of... It really was like a precursor to OpenClaw in a lot of ways.

And then the way that I would describe the trajectory of agents since then is basically there was this core really simple idea. Just run the LLM in a loop, have it call tools, you know, give it a prompt, give it some instructions, give it a bunch of different tools, but that didn't work really well. So people ended up building scaffolding around the models to make them do things in a more predictable and reliable way. And that's why we at LangChain, we built LangGraph, which was another framework really aimed at that kind of like graph-like workflows and giving more structure. And when you really want like super high reliability, you want to use something like that.

But I think sometime in maybe like November, December with some of the newest cloud models, the models just got really good and you kind of discovered that they could actually just run in a loop. And a lot of this wasn't just the models, it was also the harness around the models. So what I mean by that is if you look at things that came out about a year ago, Claude Code, Manus, Deep Research, they all had the same thing of running the model in a loop, having it call tools, it could write some code, it could read and write files. And so I think two things basically happened. Like the models got better, but then also we started to discover these like primitives of a harness that would really let the models do their best work. And I think over break people basically realized that and we saw an explosion of people building agents for different things using these same core primitives.

So we see a divergence between like two different types of agents out there. One of them is like conversational agents. So these would be like the customer support, customer experience chatbots. These require really low latency. Voice is oftentimes the medium that they interact with. And that's one style of agents that are mostly like conversational. They don't do a ton of tool calling. They'll maybe do like one or two because they can't do too many or it will take too long.

But then we see this other style of agents which Sequoia came up with this name, long horizon agents, and I really like that. They can operate over long horizons. They can do some planning. They can maintain coherence. And yes, a lot of them end up looking like coding agents. And I think there's probably like a few reasons for that. But one, code is really useful. You can use code to do a bunch of different things. You can use it to parse text files. You can use it to do things programmatically, like you want to loop over a hundred different files rather than doing a hundred different tool calls, you can write a script that does that. So code is like really generally useful, but then also the models are trained on code and so all the big model labs have been rolling code and bash and editing files into those models and so that is the stuff that works the best. So I think we see this split of agents, long horizon versus chat, and then yeah, for the long horizon it's basically turned out that coding agents or things that look like coding agents are the stuff that works well.

This is a really good question. I mean, we talk about this much internally because we're debating whether we should build like a different type of agent harness for these types of agents. I think there will kind of be a convergence when there are agents that can reliably like kick off and manage other long horizon agents. So one of the things that we're seeing in coding is that people want this experience of being able to kick off a bunch of other, like do a bunch of work, kick off a bunch of agents, but keep on chatting with like the main agent. And that's very similar to like a conversational agent in some sense, right? Like you've got that constant kind of like back and forth latency TPD, but then you know these voice agents I think will obviously want to do more and more like long running things in the future. And I think the way that you do that is you'd basically have two agents. One that runs in the background and is kicked off by this other kind of like conversational agent. So it could all kind of like converge into this single harness that just supports basically long running async background agents as a tool.

I think the harness is the most important thing. I don't know what will happen, but I think if you like, I think Manus is a great example. Like Manus was an end-user product, but their harness was so good, like that was the secret sauce of what made it work, and it worked with any of the models under the hood. And when you look at Claude Code, like yes, the cloud models are great, but the harness is really what made that work. And Claude Code isn't just a harness though, it's also that UI. So I actually think, so one, I think there is a pretty tight coupling, or there's not that much difference between a harness and a UI on top of it right now at least, and it's still very... But you look at like Codex, it's a coding app, but they also have their own harness. Claude Code, Manus, a lot of the deep research stuff out there, it's this interesting combination of harness and UI. And so I think the harness is really, really important.

And then, yeah, I think one of the interesting things is that a lot of the people building the harnesses also build the model. And so this is one thing that interests me and confused me because I think a very logical argument to make is like, great, okay, we make the harness, we make the model, let's RL the model to be really good at that particular harness. You look at some of the tools that Claude Code uses, it doesn't actually use the tools that are RL'd into the model. So like Anthropic models have some like file editing tools. They have a completely different set of tools in the actual harness. So I don't know really what's going on there. I've asked them a few times and I haven't gotten a straight response, but I don't know what happens, but I do know the harness is really, really important. Like I think this is the thing that matters. And then do you come at it from an end-user application, do you come at it from a model, I don't know.

Is how the model kind of like interacts with its environment is what I would say. So it's the set of tools that it has. And some of these tools can be really specific and I actually wouldn't count those as part of the harness, but some of these tools can interact with a more general environment. So if we think about coding agents, I would say like the file editing tools it has are like part of the harness. I would say the ability to run code is part of the harness. If you take a harness and give it a particular tool for interacting with Slack, I would argue that's you kind of like customizing and building on top of the harness.

And that's how we think most agents should be built. Like we think most agents should be built by taking a harness and giving it some instructions and giving it some kind of like tools. And those tools could be specific tools like a Slack tool or they could be kind of like configurations of tools that are built into the harness. So what I mean by that is most harnesses today have sub agents built in. They have skills built in. And so you could configure them with particular skills, but the fact that those like skill abstractions and sub agent abstractions exist, like I would argue that's part of the harness.

Other things that the harness does is like take advantage of like prompt caching. It does context compression. So when you're getting up to a certain length, it will compress it back. And so these are things that are like pretty general purpose. Like all of these apply across all different types of applications. And so these are things that are general purpose as an application developer, you shouldn't really have to worry about, but you can basically configure them with different prompts, different tools, different skills, different sub agents, and make them yours and make them your own agent that you then expose to your end users.

Yeah, that drives the agent. It kind of like tells it what to do. The way that I think about it sometimes is if you have a standard operating procedure for how a human should do things, like that should influence a lot of what the system prompt is. And so this is loaded up as soon as you start the agent. It's basically loaded up and it tells the agent what to do and it drives it.

It depends how you create the agent. So if we look at like coding agents like Claude Code or something like that, there's a system prompt that's like built into the harness and that tells it how to interact with the generic tools, but then a lot of that prompt is basically augmented by things that you as a user of Claude Code provide. So you provide like a CLAUDE.md file and that's kind of like inserted into the overall system prompt. You provide like skills and sub agents and those are inserted. And so I think in practice what we see is that the system prompt generally is like an amalgamation of a few different things. Some of them are like built into the harness and some of them are built in by whoever is customizing the harness or choosing what to expose to the harness.

Yeah. So there's a few different types of tools. Some tools are basically tools that are built into the harness. So we and a bunch of other harnesses out there have a planning tool that basically creates a plan. And it could actually like write it to file and then let you kind of like edit it over time. It could do nothing. It could just let the agent call the tool. And the reason that's valuable is that then that puts that into the context window of the agent. So it's kind of like giving it a mental scratch pad for it to kind of like think about. So there's different levels of what that planning tool can do. Other tools like...

So most planning tools are a list of tasks to do. Each task has kind of like a description, a status. Those are like the important things and then you can track the status can be like done, like working on it now, or like to do in the future basically. It can of course be whatever you want but those are the most common type of thing that we see.

And then most harnesses don't actually like enforce that you do that plan. It just kind of puts it in there. And it lets it track it. But there's nothing that like splits it up and says, 'Okay, you've created this plan. Now, let's take the first thing and go do that. And then after we're done, let's go to the second thing.' That used to be the case earlier when these LLMs weren't as good. You'd have an explicit planning step and you'd come up with a plan and then you'd go to another agent and that would do the first thing and then you come back. But there's all sorts of edge cases like what if the plan adjusts halfway through? Okay, now I have to add a step where I check should I adjust the plan and it just has become too convoluted. And so now what most things do is they just have that plan in the text file and the main agent can use that to help guide its actions but there's nothing that says I'm explicitly doing this step or I'm explicitly doing another step.

Sub agents are great because they let you basically isolate context. So this main agent is like running in a loop and it's accumulating context over time as it calls tools and interacts with things and that's great because it has all this context but that's also bad because it has all this context and that blows up the context window. And so sub agents are great because what you do is the main agent basically gives it a task, gives it a string and the sub agent spins up with a completely fresh kind of like context window. So it starts from scratch and then it does a bunch of work and then it responds and the main agent just sees the response. So you get this nice isolation between different tasks. The downside is that you have isolation between different tasks.

Because then you need to communicate between the two agents. And so if the communication between the agents is bad, then it won't work. So a very real thing that we see happening sometimes is the main agent will spin up a sub agent. The sub agent will do a bunch of work and like the key stuff will be like halfway through its trajectory and then its final message will be like 'done' and the main agent is like 'what do you mean done? I can't see anything else.' And so that's an example where the sub agent doesn't have good enough instructions. It hasn't been communicated well enough to the sub agent that it needs to communicate its final answer back in its final message. And so communication is the hardest part of life by the way. It's the hardest part of startups, hardest part of relationships, hardest part of working with agents is getting them to communicate. And so sub agents are great, but they do add that extra layer of communication.

It's all on the prompt. It's all on the prompt. Yeah, that's the beauty of these types of agent harnesses. You know like earlier when we were doing things with LangGraph people would be like okay how do I add like a step to make sure that the agent does this before X or how do I enforce that the... for better or worse, and this is why LangGraph still sells a place, I'll get to that later, but like for better or worse the way that you get these things to do anything is you just tell them to do it. And that's great because it's like flexible but that is also not like 100% reliable.

And so we actually see still pretty good adoption and pickup of LangGraph in heavily regulated industries where you want like a ton of control and precision and reliability because as good as these kind of like coding agents are, they are pretty unpredictable in terms of what they do and there's no guarantees on anything. It's why they're so enticing because you just tell them to do things and they do things but there's no guarantee and so that's a downside as well.

And the way that I think about a file system is it basically lets the LLM manage its own context window. So it can decide what to read from files. So rather than, you could imagine an alternate world where you put everything that is in a file, you just dump that into the context window. That would blow it up, right? And so if you let it read files, great. That lets it choose what to pull in. When you let it write to files, that's basically saving it so that if you do compress the context over time, you can return to it and you can read it in the future.

We use file systems to offload large tool call results. And when I say we use, we have an agent harness called Deep Agents. When I talk about our planning and our file system stuff, this is all stuff that we do in Deep Agents. Most other harnesses do similar things, but the one I'm talking about in particular is Deep Agents. So what we do is if you call a tool and it comes back with like 60,000 tokens, we don't show that all to the LLM because that's a ton of tokens. Rather, we actually put that in a file and then say, 'Hey, here are the first like thousand tokens. If you want to read the rest, go read this file.'

We use it for summarization as well. So when you get to a certain context window length and it's about to overflow, what we'll do is we'll run a summarization step, but we'll actually dump all the original messages into the file system. So if it wants to go look things back up, it can. And so we use it in a variety of ways. I would say the overarching theme is it actually lets the LLM manage its own context. And I think the general theme of like these more and more autonomous agents is that they let the LLM do more and more and managing its own context is kind of like that's an increased version of letting it call tools or something like that.

Great question. Great question. It can be anything. The important part is that it's exposed to the LLM as a file system because LLMs are great with working with file systems. And so one of the cool things that we have in Deep Agents that is pretty differentiated is this file system. It could be the real file system on disk or in your Daytona sandbox or anything like that. It could also be a database that just has like a thin layer on top of it that exposes it as a file system. Not everything needs to be a file system. If you have a SQL table, let it write SQL, that's pretty easy for it to do as well. But when you're working with like large amounts of text, even if those are stored as like a row in a SQL database, it's often nice to give it kind of like the interface of a file because that's how LLMs know how to interact with it. So yeah, it could be anything under the hood, database, S3, real file system.

Those are the four that when we launched Deep Agents, and so the story behind launching Deep Agents was we saw Manus, we saw Claude Code, we saw Deep Research, they all had these four things and we were like okay that's pretty common, let's put it into a Python package and make it easy for people to build their own versions of that. So those were the four things at the time. Those are still probably the core things.

Some other things that are frequently used. I mean, bash and executing code is a big one that's not always used because sandboxes like Daytona are still new and so people are still discovering how to run them and how to manage them. And so it's often easier not to do that, but we're seeing more and more want to do that. And so that's where things like sandboxes come in handy. Skills are a new primitive that didn't exist when we launched Deep Agents, but are now very, very, very interesting.

Yeah, skills are great. So, they're basically like a bunch of files. There's usually one kind of like skill.md file, which is a big markdown file that contains instructions on how to do something. And there could be other things in a skill as well. There could be other scripts that it could run, but it's basically these instructions for how to do particular things. And rather than being loaded into the system prompt, they are just like referenced in the system prompt. So you'll tell the agent, hey, you have access to like this code writing skill and you have access to this documentation skill and then if it decides that it needs to use those skills, it will just go basically read those files on demand. People call that kind of like progressive disclosure. You tell the LLM only what it needs to know when it needs to know. It's another way of letting it manage its own context window as well.

So that's a key part that we support in Deep Agents and most harnesses support. I mean other interesting things that we're thinking a bunch about like async sub agents are really interesting. I mentioned this earlier, but I think this is something that most harnesses don't do that well. I think technically Claude Code has support in it, but I don't even know when it triggers it and it's hard to like observe them and manage them, but I think this will become more and more important.