Cool. My name's Harrison, co-founder and CEO of LangChain. I want to talk about evals and harnesses in the context of kind of owning your own intelligence. So, when we talk about intelligence, we're normally talking about agents. What exactly makes up an agent? At LangChain, we think there's kind of like three main parts. There's a harness that orchestrates a model and some context. And if you're talking about owning your intelligence in general, you probably want to own all three parts of these. And so, owning the model, I'm not going to talk too much about. Lin was here from Fireworks and talking about open weight models and owning that. A big part of this is also the ability to switch models. There used to be this concept of kind of being like cloud agnostic and being able to switch clouds back in the day. Same thing exists, but for models. You want to be able to switch to avoid lock-in, but also to just use the best model when it's available. Context, you want to own all the context that your agent uses. Whether that is memory, whether that is semantic knowledge, whether that is previous conversations. These can help personalize and guide the agent as it goes along. And then the last bit is the harness, and that's what I really want to focus on.
So, how do you really own your harness? What does that even mean? What's the main job of a harness? The main job of a harness is to bring context to the model at the right point in time. And so, it does all the orchestration around the fixed context, the dynamic context. It brings it into the context window of the model, shows it something, gets some response, and then does something with that. And so, agents need to do all these different things in order to accomplish their jobs. There's a ton of domain-specific stuff that they need to do as well, but they need to interact with external systems. These external systems, when you interact with them, they emit more context that can get fed back into the agent into the loop. And so, the harness is the thing that really orchestrates all of this together. Agents at their simplest, when everyone talks about agents, what they really talk about is just an LLM running in a loop calling tools. And this is a really simple, but really general architecture. Some request comes in, the LLM makes some generation. That generation may include a tool to call. If it does, you invoke those tools, and you pass that observation back to the LLM. And this is the core architecture behind pretty much every agent out there today.
But they're all different in slightly different ways. And so, on the left here, this is kind of like the base core kind of loop. But there's a bunch of different things that you can do in your particular harness at different stages. And so, this is over here, this is... So, we built LangChain, which is a really, really base minimal harness, and that's LangChain over here. And then this is Deep Agents. Deep Agents is kind of like our model-agnostic and more general-purpose version of Claude Code. And so, it does more things. It connects to file systems. It has skills. It has sub-agents. It's built on top of this really simple harness, but we customize it by using these levers over here. So, you can run particular code snippets before the agent's invoked, before each model call. You can kind of like wrap these model calls, you can wrap the tool calls, and you can customize this core simple loop in a lot of really powerful ways just by using kind of like small what we call kind of like middleware constructs. There's other ways to customize the harness as well, but this has kind of emerged as... there's a concept of hooks and plugins in a lot of the coding agents as well, and that's essentially what they're doing. They're taking this base loop that's running, and they're adding little hooks or plugins at various points to let you customize it.
And so, a lot of the stuff that you can do when you can customize, this is all done by that concept of middleware, by just modifying that core loop. So, the agent's still running in a loop, it's still doing that same simple architecture, but through that you can give it access to a sandbox, you can give it access to a file system, you can give it access to sub-agents, you can give it access to memory, you can have summarizations. So, summarization, if we go back to this thing, summarization would come in before the model. Before the model's invoked, you check if the context is too long, and then you summarize it. And so, that you can add into this core loop through this concept of middleware, same with context offloading, which is a way of basically taking large tool calls and dumping them. That kind of wraps the tool call. And so, the point is there's this really simple kind of like general architecture of an agent. All of these more advanced agent harnesses are basically doing that loop, but adding in a bunch of stuff while still running the core loop. And so, as you think about kind of like building or customizing your own harness, these are the different places that you can insert things into. You can add your own summarization step, you can add your own handling of particular tool calls, and that's one way that you can customize kind of like the agent to your particular domain and the harness to your particular domain.
The other way that you can customize the harnesses that the agent runs in is by having a more explicit kind of like cognitive... So, this used to be the way that a lot of people would build agents in kind of 2023, 2024 because the models weren't good enough to run in a loop. And so, in order to get it to do particular things, you would have these very bespoke cognitive architectures. And so, this one over here is for a deep research example where it would generate some sub-questions, fan them out, and then go and execute them. And then this one over here is for a code review bot. And you can see that there's these very kind of like bespoke steps. A lot of this has gone into the harness now. And by the harness, I mean it's still this core loop. These might be added as particular kind of like modifications to that core loop. But, for a lot of really particular kind of like flows, we do see people still using cognitive architectures like these to really guide it in particular ways. One thing that we recommend to people is to start with a general harness. That's the easiest to get started. It's going to be quickest to time to value. And then as you kind of narrow in on the use case that you want to be excellent at, you can start to add more of these kind of like gates and checks around it to guide it into particular ways.
One question that we get a lot is when to kind of think about building your own harness versus using an off-the-shelf harness. A lot of the off-the-shelf harnesses work with particular models. So, the off-the-shelf harnesses include things like Claude Code or Claude Agent SDK, which works with Anthropic models, Codex, which works with OpenAI models. I think this is a big open question in the industry. My answer generally is the more in distribution you are of what the models are trained on, then the better the off-the-shelf harness will be. As soon as you start to move further and further out of distribution, then you'll probably want to tune your harness in some way. There's different ways to tune the harness as well. So, the models may be in distribution on particular things that you are doing on an out of distribution task. So, what I mean by that is if you think about something like legal AI, which Gabe just talked about, and I think he mentioned how they have their own harness, there are things that are in legal AI that are still in distribution of the main models. So, for example, editing files is something that the main models have all been RL'd on. And they've actually all been RL'd in very particular ways. So, OpenAI and Claude models edit files in different ways in their harnesses, and as a result, their models are actually best at editing files in different ways.
Now, the models themselves are out of distribution on this larger task of legal AI, but they're in distribution on this task of editing files. So, if you think about building a harness that works there, you'll probably want a custom harness, but you'll want it to use the edit file tool that is in distribution for the model that you're using. So, one of the things we do in Deep Agents, for example, so Deep Agents is our customizable harness, we actually have this concept of model profiles, where for things that are in distribution of models like editing files, we basically switch between different edit file implementations depending on which model's being used. And so, I think that's an example of customizing the overall harness when it's out of distribution for a task, but keeping smaller in distribution parts as close to the model layer as possible.
The second big part of what I want to talk about is evals and observability. And so, I think as you're experimenting with all parts of an agent, whether it's the model or the harness or the context, you're going to want to know what's going on inside of this system, and you're going to want to be able to evaluate it. And so, these are useful tools that you can use, again, not just for custom harnesses, but also for custom models. So, there was a great Twitter article that Satya wrote 2 weeks ago, where he talked about a lot of these concepts. And there's three quotes in particular that kind of stood out for me. One, create your private evals because eval defines what good looks like inside the organization. Two, retain ownership of your organization's memory, traces, feedback, though that bold is mine, decisions, and institutional context. And then three, you create your own continuous learning loop hill climbing machine that will allow your AI investments to compound the value of your firm. And so I think these speak to the importance of evals and observability and the learning loop that they power in really owning your intelligence and compounding it. So how exactly do they do that?
So evals. Gabe was here talking about how they built benchmarks for the legal domain. I think every company when they're building a mission-critical agent, they will build benchmarks for that agent. You can use it to define and catch regressions or you can hill climb on that benchmark. Again, either by adjusting the harness or adjusting the model. Things that we see becoming the industry standard for defining these benchmarks is Harbor. Harbor is an open-source eval runner. It's created by the makers of Terminal Bench 2, which is one of the industry standard benchmarks for benchmarking coding agents, and it's become pretty popular for a variety of domains. What it lets you do, so this is Frontier Bench, which is another coding benchmark. You get this nice benchmark and you can compare different agent harnesses, different models, different reasoning efforts, and you can see how all these different harnesses and all these different models do on your task. And so having a benchmark for your task will become really, really important when you're trying to define it.
What exactly is Harbor? It's pretty simple. At a high level, it consists of an agent that you run against the data set. A data set has a bunch of different tasks. Generally, they're run in sandboxes because there are a lot of these different tasks and you might want to parallelize them. And as I talk about in a little bit, each task has its own kind of like environment. So this is what a Harbor task looks like. So, on the right, you can see that it has an environment. This is where you define the environment that the agent runs in. A lot of these longer running, more stateful agents need to interact with their environment. And so, you basically spin up a sandbox, give it its own environment that's defined in a Dockerfile, and run it there. There's then a solution, which is basically a kind of like golden solution that you use to sanity check it, so it's not that interesting. Test is more interesting. This is basically the verifier for the agent run. The test scripts can do anything. They can run code, they can run unit tests, they can run another LLM as a judge, they can run an agent as a judge. You basically define how the agent is scored in this test. And then instruction.md is the prompt that the agent is given. And that's kind of like the core of Harbor. You define these tasks, which are bundled up things that can be run in a sandbox, and then you run a bunch of them against agents. And agents again consist of models and harnesses, and you score how they do.
When you do all of that, what do you get? You get some nice results that you can compare. So, this is LangSmith, the platform that we build for evals and observability. And so, you can see here a bunch of different experiments. We have a great integration with Harbor. You can see the feedback scores. In this case, it's a single reward function. You can also track latency and tokens. So, when you're benchmarking agents, you probably don't just care about accuracy. You also probably care about latency and cost. And so, you'll want to track all of those. And then for a particular experiment that you run, these would be the different tasks that are in a Harbor data set.
Talking a little bit about observability. Observability sounds basic, but I think it's really important and really underrated for agents, actually. So, when agents mess up, they mess up because an LLM call goes wrong. Why might it go wrong? It might go wrong for one of two reasons. One, the model is not good enough. Two, the context that the LLM received isn't good enough. And so, I actually think it's the second one that more often than not causes issues. And so, having really good observability into what is going into the context window of the model and then how that context is accumulated, what steps were run, what tools were run, how does that context get there? All of that is really important for debugging your agent when it goes wrong. So this is one view of observability that we have. This is intended to be a more kind of like user-friendly view where we actually represent it. This is similar to what you might see in kind of like Claude Code. We kind of like hide some of the tool calls so you can see seven tool calls up there. And so we try to make it really easy to kind of like skim through this. Most agent paths these days come in the form of trajectories. Trajectories are basically you can think of them as the list of messages that you see kind of like Claude Code running. So when you run Claude Code or another agent, you type in a human message, it then makes a bunch of tool calls. Those are all messages under the hood and then it responds and then you type in another human message. That's kind of like this message trajectory that is becoming more and more of a central part of these agents. But that's not enough to fully debug it and so we also have this full kind of like trace and you can click into particular things and see exactly what goes on inside the model. And this type of observability is pretty important for knowing what's going on.
Evals and observability really let you set up this data flywheel and compound the intelligence as you start to use the agent, as your users start to use the agent and you start to get feedback. So this is a slide that one of our team members presented at Swyx's AI engineering fair actually around a recipe for continuously improving agents. At a high-level it's really simple. You build an agent, you start running it, you collect lots of traces, you then curate the trace data, and then you run experiments on that data that you create. And so it's really simple, but of course there's a lot of complexity under the hood. So one thing that's really important for this is feedback. Getting feedback either from the environment or from synthetic source. So from the environment, one thing that I think is really underestimated in agent design is actually UX design of how you present the agent to your users. If you present it in a really intelligent way, you can actually end up getting a lot of feedback from them. They may not click thumbs up or thumbs down explicitly. No one's really doing that. But if you design the UX in a clever way, you can get some of that feedback. The other thing you can do is you can start to get synthetic feedback. So you can run what we call kind of like online evaluators over these traces to judge things. So Gabe was talking about an experiment that we did with Harvey where we significantly reduced the cost of some of these LLM as a judge type things. So if you imagine running Opus over every single trace that comes into your system, that's going to rack up a big bill. And so you want a really cheap and fast way of doing this. So we've fine-tuned some SLMs for actually doing this, but you can of course use off-the-shelf models with custom prompting to do it. Or you can just use code if some of the things that you want to test are simple enough.
So this is the full part of it. The curating the trace data, feedback is a big part there. And then the other thing is when you use that data to update what happens, you can update any part of the agent with this kind of like system. So you can update the harness by doing harness engineering. You can update the model by doing fine-tuning on that. You can update the context by doing memory. And so the part that we think most about at LangChain is the harness engineering part of that. And so I want to show a really quick demo of one of the things that we added to help with that. But I think Trajectory is talking next on some fine-tuning that can be done. And so it's a very similar process where you run the agent, get some traces, use that data in some way to improve the system. What is the system? It's these three pieces. Any of these can be updated in some way. And so yeah, this is the full end-to-end flow that you might want to do. One of the things that we think about is how can you automate this as much as possible because this is tricky and takes a lot of time. And so that's one of the things that we've been thinking about for the past few months. I want to do a quick demo of what we call LangSmith Engine, which is basically an agent that sits on top of your traces and does all this work. So if we look at what that work was, you know, you've got these traces, the work from there is curating the traces and running some experiments, suggesting fixes to one of the three things. And as I mentioned, we mostly focus on the harness engineering bit. So in the demo I want to show what this looks like and how it represents that. So hopefully this will work. If not, it's not that big of a deal.
Perfect. Okay. So this is LangSmith. This is a bunch of traces we have coming in. We have this tab called Engine over here. This is an agent. It runs in the background. It creates what we call kind of like issue boards. So this is the part of curating data. It will look for... It will basically under the hood is a coding agent that has access to our LangSmith CLI. The LangSmith CLI, you can filter traces for feedback and other things like that. So we give it a nice big prompt and some sub-agents that help it basically go out and explore this data and identify issues and see what common things are. And then it will create these issues right here. And so here it's created an issue. It gives a description of it. It links to the traces so I can go see some supporting evidence. And then down here, I guess this is very simple changes to the prompts. But here it's updating part of the context in this case. Here it's also updating some instructions. And we can see here that it's adding some code to go into the harness as well. And so this is something we launched in the past few months and I think speaks to this data flywheel, which again is a very simple thing. Run agent, get traces, see patterns, fix. And this is our attempt at automating it. That's all I've got. Happy to take any questions on harnesses or evals.
