Hello, hello everyone. Thank you for coming to this webinar. That was a cool intro video. I didn't know we were doing that. That's fun. My name's Harrison, co-founder and CEO of LangChain, and excited to be talking about the agent development life cycle with everyone today. Before we get started, a few logistical things. There is a place for chat and there's a place for Q&A. And so if you have questions that you want me to answer as we're going through the slides, I will answer a bunch of them at the end. So drop everything in there. I will do the... I'll walk through a presentation for the first 30 or so minutes and then answer questions for the rest of the 15.
This has been recorded and we will put it up on YouTube afterwards as well. Cool. So let me jump into it and figure out how to share my screen. There we go. So yeah, I want to talk today about the agent development life cycle and kind of like a high-level overview and introduction to it.
We've basically seen that over the past year, organizations have figured out how to ship agents. And the ones that have figured out how to do that the best, they iterate systematically. They ship really early and then iterate quickly. And this iteration is done in kind of like a systematic way. And the reason that it's important, or the system that they use, is all around kind of like agent behavior. So the hardest part of building agents and shipping agents is getting them to behave reliably. And so getting something that works, you know, initially locally or in a one-off on a Twitter demo is easy, but like shipping it reliably to 100% of users at scale, the difficulty comes down to like the performance and the behavior of the agent. And so when people are iterating and when people are building these agents, that's a key thing that we see people focus on. And that's what the agent development life cycle is really built around.
There's these components, but all of these components come together into like how can we as developers of agents get the most out of the agents that we're building. And so the different stages of the life cycle that we see, and then I'll walk through all of these, are build, test, deploy, monitor. And then when you do this for a bunch of agents, you want to govern all of them as well.
We obviously build a bunch of tools to help with all stages of this. I will try to refrain from mentioning them until the end. We'll show where they'll fit in, but I really want this to be kind of like education and informative, less on kind of like our technology and more on the process and the solution that we see people getting towards.
So, first, I want to start with build. This is where people get started. You've got an idea for an agent, you want to build it. There's different ways to build agents. And I think this is a big question that you should be asking as you think about how to build the agents that you want. The level of abstraction, the level of complexity that you want.
So, there are a bunch of agent frameworks out there. This is how LangChain got started, obviously, with the LangChain framework. The main value prop of a lot of these is abstractions. Abstractions around model inputs, model outputs, tools, prompts, retrieval. And this provides a standardized way for people to begin building with agents. It makes it really easy. It has all these abstractions. If you're going from one project to another and it uses the same abstractions, it's easier to get onboarded. But you don't always have to use a framework. There are downsides to using a framework. The downsides might be that they obfuscate some of what happens under the hood. This is a lesson we learned early on with LangChain. And we addressed this by making the framework kind of like less heavy and less high-level.
But I remember early on we had this chain and this chain made five LLM calls under the hood. And someone asked, "Hey, like why is this chain taking five times as long as when I call OpenAI directly?" And the reason it was doing those five calls under the hood, but it just wasn't obvious. The person who was using it wasn't aware of that. And that's partially better docs. That's also partially the framework's fault. It doesn't sometimes make that clear. We spent a lot of time to make LangChain more lightweight and less obfuscating of those things, but there are pros and cons to using frameworks.
Runtimes are another version of this. So, runtimes are a little bit more aimed at some of the runtime considerations. So, how do you do state management? How do you do durable execution? How do you have human intervention? These are framework-agnostic concerns, but they have to do with the runtime of the agents and how they run.
And then harnesses are a little bit higher level than frameworks in my mind. Harnesses have been a really hot topic recently, largely because a lot of the coding agents that are out there are basically harnesses. They are coding-specific harnesses, but there are a lot of learnings from those coding-specific harnesses that you can port back into general-purpose harnesses, such as state management, access to a file system as a way to kind of like manage context. So, context management as well, summary and compaction, a really clean human-in-the-loop interface, sub-agent support, skills, all of these are things that are a little bit higher level than a framework in my opinion, and part of the harness.
Just mapping these really quickly onto the LangChain ecosystem. Actually, I'll map it onto the whole open-source ecosystem. So, frameworks, you've got things like LangChain, LlamaIndex, CrewAI, Google ADK, OpenAI Agents SDK, AWS Swarm, things like that. Runtimes, we consider... so LangGraph, we consider that part a runtime and part a framework, but LangGraph is a runtime, Temporal is a runtime, and then harnesses, this is where you have Deep Agents, you have Claude Agents SDK, those are really like the main harnesses that are out there. There are coding agent harnesses, so you could also consider Codex or OpenCode, or Pi. Pi is maybe a good example of a developer harness as well. But there's a really thin line in my mind between kind of like a general-purpose developer harness and then like a coding agent harness, and they look very similar a bunch.
And we have stuff on all our documentation about the differences between these and the differences between the LangChain offerings and other ones out there.
No code I want to explicitly call out. This is really interesting to us. How far can you go in building agents if you don't write code? And I think like one of the things that we've seen is that a lot of the agent definitions are starting to be less and less code-like. So what I mean by that is a key part of the agent is a prompt. A prompt is not code, it's just text. You can represent this as a markdown file, as an agents.md file or something like that. Tools. Tools you can represent with MCP servers. Sure, the MCP servers themselves are code, but connecting to those is just a JSON file or an mcp.json file. Skills. Skills likewise, a lot of kind of like markdown files. They may contain some code in terms of like the skill scripts that you want them to execute. But I think there's a real argument that you can make that building agents can be done with markdown files and other files like that.
And so I'll talk about what we have in that space later on, but I think you see some efforts like Eve from Vercel, to maybe call out something that's not LangChain related, as a really interesting example of representing agents on a file system, and it's a similar approach that we take with kind of like Deep Agents Deploy and Fleet.
So moving on to test. Test has a lot of parallels to kind of like traditional software engineering, but it looks slightly different for agents. So what exactly is testing? Testing's basically running your agent against some inputs and scoring those, and then using the results to decide how well the agent's doing. It's measuring the agent. You typically want to do this to make sure you're not performing badly on cases that you need to perform well on, or that you're improving on a benchmark that you want to improve on. And so these are two slightly different kind of like user journeys. One's more like regression testing and the other's more like benchmark hill climbing or things like that.
What exactly goes into this? First, there's the inputs. What are you running the agent over? These are like expected tasks, how you expect the agent to be used. Known edge cases. You can get these from compiling them from real traces, from dogfooding traces, from generating them synthetically, but these are the inputs. Then you take these inputs and you build up datasets. So one dataset may have 50 different inputs in it. These, in addition to the inputs, you also have outputs or some criteria for how to judge it.
So maybe going to this distinction a little bit. For some tasks, you can have a really clear output that you can imagine. So if you're doing a classification task, the ground truth output you can say, "Hey, this should be true or this should be false." And then you can compare against that. For more open-ended things, you don't have a ground truth, but you have a set of criteria that you want to run against. So maybe taking coding as an example, Terminal Bench 2, which is probably the leading benchmark for coding agents. The way that it works is you get this input task, you run the coding agent, and then to score it, you don't look at the files that are changed, you actually run unit tests against them. And so this is an example of criteria-based judging.
And this starts to get into metrics. And so metrics, you can have correctness, and these can compare against the ground truth. You can do criteria checks, and these can run the criteria checks that are part of the data point. You can define different policies you want to be compliant with, and then putting all of this together, you run the model over the inputs, you score it on these metrics, and then this makes up an experiment. And then with a bunch of experiments, you can compare versions, catch regressions, decide readiness, things like that.
Maybe two things that I want to call out. First, testing, both the dataset and the metrics, oftentimes needs to be application-specific. So thinking critically about what the inputs are, what the outputs should be, how you want to judge it, those are really hard like product questions almost. And so there's some tooling that we can give off the shelf in LangSmith to help make this easier, but you'll still have to build your own datasets, and maybe there's some metrics you can get started with, but you'll probably have to think about defining your own.
And then the second thing that I want to call out is just an example of where I think the space is going here. So I mentioned Terminal Bench 2. Terminal Bench 2 runs on an open-source framework called Harbor. What Harbor does is when it runs all of these different tasks, it actually runs them all in their own sandbox. And so this is really useful for long-running stateful agents. And this is a different paradigm than testing kind of like LLM applications. So testing LLM applications is easier. You basically just call the LLM in a loop. Like it's pretty easy. You maybe need to worry about rate limiting, but it's not that difficult. Testing agents, like each agent can run for, I don't know, 15, 30 minutes. It doesn't just respond, it produces artifacts as well. It interacts with the environment. And so thinking about building these environments is really important, and Harbor is a great way to run these more long-running stateful evals. We have a bunch of integrations with Harbor that are coming out soon.
Maybe going on to the next one, deployment. What does it look like to deploy the agent? So deploying agents obviously has some similarities to deploying software, but there's differences or specific focuses that are really important. So talking a little bit about the runtime, I talked about this earlier, but things like durable execution are really important. If the agent fails halfway through a run, can you just resume from that step or do you need to go back to the beginning? Persistence, memory, like how do you both like thread-level memory as well as longer-term memory that persists across threads? How do you manage that? Streaming, how do you stream things back from the agents? Streaming obviously existed before agents, but agents have made this much more prevalent. And what do you send back? Is it just the tokens or is it also the thinking process or the tool calls and how do you render that? And then human-in-the-loop. Human-in-the-loop, easy to do locally, hard to do when you're running in production at scale. How do you basically like pause the agent and then just wait for it to resume?
Other considerations, sandboxes. Sandboxes are really top of mind right now. A lot of agents, not just coding agents, need to write and execute code. And how do you do that in a safe and trusted way? If it's running code that you've predefined ahead of time, that's basically just calling a tool and that's easy and pretty simple. But what if the agent is writing its own code? That's untrusted code. You need a safe way to do that. And so sandboxes are how you will want to do this.
Virtual file systems are something that we think are also really interesting for agents or just file systems in general. So agents are really good at interacting with the file systems. They know how to read and write context. They've been trained on it. They've been post-trained on it. They're really, really good at it. But sometimes spinning up a whole file system for an agent is a lot of overhead. And so virtual file systems are basically a fancy way of saying keep your context in a database, which is more scalable and you can manage more efficiently, but expose it to the agent as a file system. And it doesn't have to be a database, it could also be S3 or anything like that. It's basically saying store the data wherever you want, and then give it to the agent as a file system and let it use the read and write and glob and grep and ls and these other tools that it already knows how to use really, really well.
Some interesting things here that we've done are using like Box and Notion as virtual file systems, so you can actually use common human scratchpads as virtual file systems for agents. We also built our own. This is Prompt and Context Hub. So Prompt, a little bit unrelated to this, but this existed before agents, but basically storing versioned prompts. Context Hub is basically a way to store context. Context is just files. You can then pretty easily mount this Context Hub as a virtual file system for agents. And so you can store versioned prompts and skills and instructions there.
Going on to monitoring, as the fourth thing here, monitoring is really about showing you what your agent did. And there's different levels of this. So first up is just tracing, and you can see an example of this on the right. And this is the most absolutely basic thing that honestly you should probably have set up even before you go to production. It is just tracing of your agents. What are all the steps that it took? What are the inputs to each of those steps? What are the outputs of those steps? Seeing the different tool calls that are made, the sub-agent tool calls, being able to understand exactly what the input to an LLM is, and then exactly what the output is, super important and necessary for debugging these LLMs and these agents. And again, like most of the time when these agents mess up, they mess up because the LLM messes up, and then it messes up because it doesn't have the right context. And so being able to see the context it has is probably the most important thing that you can do in this. And that's tracing.
But you also want to start collecting feedback on these traces. Because as you're running this agent in production, you might get thousands or millions of traces. How do you know which ones to look at? And so this is where signals like tracking user thumbs up or any feedback that the user gives is really important. This is also where online evals comes in handy. So what online evals means is basically you've got these production traces, can you run some evaluators over those traces and score them on some dimensions and then attach that as feedback. And so, you know, this is a little bit different from evals previously because you don't know what the ground truth should be. So you have to run it in a way where you do it without ground truth.
One example of this, to make this concrete, that we are really interested in for online evals is what we call like perceived error. So I would argue that when you see a transcript of a human talking to an agent, there are clues you can look for to see if the agent messed up. If the user says, for example, "You messed up," that's a good signal that the agent messed up. But also if they say like, "No, you did X, you should have done Y." Or like, "You did this wrong." Or you're doing coding and you see them paste back in the code snippet with an error, that probably means that the user ran that code snippet and there was some error. And so those are all signals of what we call like perceived error. We actually trained a small language model to detect this. And this is an example of an online eval that you can run.
And then dashboards are basically aggregating all of those signals, whether it be latency, whether it be time to first token, whether it be these feedback or online evals, and showing you these in aggregate. So those are obviously an important part of monitoring agents as well.
Lastly, I want to talk about govern. What does it mean to govern agents? I think there's a few different aspects of this. One that's been top of mind for us recently is cost controls. Not just for agents that we build, but agents that we use as well. We're using a ton of coding agents internally, the cost of them is getting pretty out of hand, and so we want to think about how we can view costs and also manage costs.
Tool access, this is another one that we think a bunch about. When agents use tools, what tools do they have access to? Who are they using the tool on behalf of? If an agent uses Notion, is it using it on behalf of me, on behalf of a coworker, on behalf of a service account? What does that look like? And so having proper controls around that is really important. And likewise around audit trails. So what tools did an agent call and who were they calling it on behalf of? This is where some of the observability from the previous section starts to come into play as well.
And so you can see that like governance, yeah, it wraps the whole life cycle. It's not just a different step. Like tracing and evals can easily be involved and are a big part of governance. Human-in-the-loop, we talked about this. This is really important. Like which tools need human approval before they're run. Discoverability of tools and of agents, and then sharing of all these contexts and skills. And so we think all of this goes in governance, which wraps the whole life cycle.
One of the things that's also worth talking about is doing this at scale. So, you know, if you're building a single agent, you probably are figuring this out as you go along. You just have to think about your single agent. You don't have to kind of like think about doing this at scale. When you are part of a larger organization and want every team to be doing... like these are still like emerging best practices. And when you want every team to be following these emerging best practices, it's often helpful to think about how to do this best at scale. And so some things that we've seen be helpful here.
I talked about this earlier, but like frameworks, whether they're on the build side or on the evaluation side, prevent reuse and help people onboard quickly onto new problems. So yeah, using frameworks in the building space is a way of standardizing how different teams build, and so then you can switch between teams or onboard people. And then shared evaluation frameworks, they prevent every team from rebuilding basically the same tests. Again, you'll probably want some, depending on what you're building, you'll want some dataset specificity and maybe even some metric specificity as well. But we do see that there are organizational-level metrics that people always want to test for on every agent that goes out. And so shared evaluation frameworks help with that.
You probably don't want every team thinking about how to roll their own deployment infrastructure. Infrastructure's hard and annoying. And so you probably want some central... a lot of this we see end up being managed by a central platform or central core AI team.
Tracing and feedback and dashboards. This is really important as well. Like building the agent is kind of the fun part, but it's also the fastest part of this whole cycle. Everything else in this cycle, like operating the agent at scale, that's the hardest part and that's what takes the most as well. And so a key part of that is just like observing what the agent's doing, and so giving teams an easy way to have that is really, really important. And then obviously governance becomes really important at scale as well.
I do want to talk a little bit before, for maybe the next five minutes before going into questions, about what we're building at LangChain that fits into this. And so I already see a bunch of questions rolling into the Q&A section, but keep on dropping stuff there. So how does everything that we work on fit into the agent development life cycle? You can see them all mapped out here. So in build, the three things in dark: Deep Agents, LangChain, LangGraph, these are our open-source libraries. Deep Agents is an agent harness. If you're getting started building agents today, this is what I would get started with. Deep Agents builds on top of LangChain. LangChain, as I mentioned, we slimmed down the abstractions. It's a much more minimal agent framework.
If you want to build an agent from scratch, so not an agent harness, but if you want to build your own harness, LangChain is probably the best way to build your own harness. Deep Agents is using our agent harness, and then LangGraph is a runtime that sits beneath all of them. We've also got LangSmith Fleet, which is a no-code way to build agents. So this is aimed at non-developers who want to build agents. It's entirely in the UI. A lot of it is by speaking and talking to the agent in natural language.
LangSmith, so LangSmith is our platform that does the rest of the test, deploy, monitor, govern stuff. So we have datasets and evals and experiments in LangSmith. We have an integration with Harbor, which is the framework I mentioned earlier, in LangSmith. On the deploy side, we have LangSmith Deployments, which takes your agent, spins up an agent server, gives it about 30 to 40 different production-ready endpoints. That includes streaming, A2A, MCP, human-in-the-loop, short-term memory, long-term memory. It spins all that up.
Separately, we also have sandboxes. So sandboxes, these are code sandboxes. You can use them with or without any of our open-source frameworks. That's maybe one thing I should mention as well. All of LangSmith is framework-agnostic. So you can use it with us, with Vercel AI SDK, with CrewAI, with any of that. So sandboxes are a great example of that. They're completely framework-agnostic. And then Context Hub is basically where you can store skills and agent.md files and use them in Deep Agents or just use them in your local coding agent. So it's also framework-agnostic.
Monitoring, this is probably what we're best known for in LangSmith. We trace everything really well. You can run online evals. We trained our own model to detect perceived error. That's still in private beta, but we'll be rolling out over the next few weeks. We've got dashboards. You can collect user feedback. And then on the governance side, the first thing that we launched here, still in private preview, is LLM Gateway as a way with cost controls and cost visibility and data privacy first and foremost in mind.
As we think about this whole life cycle, we really think traces are at the center of all of this. So we think again that the hard part about building agents is getting them to behave. In order to get them to behave, you basically need to observe what they're doing and then tweak them and then measure it. And traces are really that core of how you observe it, how you start to even measure it. And so we think traces are really the core of all of this.
In order to do that, we actually built our own database. And so it's called SmithDB. It is purpose-built for agent traces. My co-founder Ankush, who's much smarter than I am, leads that. And the reason you should care is it makes everything in LangSmith really, really fast. And so storing one trace is easy. Storing billions of traces is hard.
The other really cool thing that we launched recently, and I think I saw some questions in the Q&A, which I'll go to after this, about like auto-improving the agent. We want to help with that. Like, you know, if you think about how humans would auto-improve an agent, they would run the agent, they would look at how it did, they would think, "Oh, it messed up on this. Let me tweak the prompt here." And then let me, you know, rerun it again, and let me create an eval so that I can measure it going forward. And then let me like redeploy it.
And so we built an agent to do that. So that's LangSmith Engine. It's been out for a month now. We think it completely changes how you think about operating these agents because it makes it so much easier. What it does is it basically runs in the background over traces. So it sits on top of a tracing project, runs every 6 hours, and then basically tries to cluster traces into issues that are happening. It will create issues on this issue board. Each issue has a name, a description, and a link to all the traces that are afflicted by this issue.
But then it will also try to fix that issue. So it will suggest code changes. It may suggest prompt changes. And then it will also do the things that you would hopefully do as you fix these issues naturally yourself. It will try to add examples to datasets. It will create online evals to monitor this. And so this is something we're really excited about because we think it drastically lowers the burden of operating these agents at scale. And it's fun to build agents. It's a little bit less fun to constantly be monitoring them and playing whack-a-mole with issues. And so we hope that LangSmith Engine can help with that.
So with that, I'm going to go over to the Q&A section. There are a lot. So let's see. I'm going to start all the way from the bottom. Oh my god, there's a ton. All right, I'm going to try to do this rapid fire. Okay. What do I think about Vercel's eve.dev? Yeah, this is a great example of representing agents as files in a file system basically. Deep Agents is already kind of similar to that. So you can point Deep Agents at agents.md files and skills and basically just instrument it from that. We also have managed Deep Agents which takes that even a step further.
Thank you, Angeline, I think, for putting this on screen. I didn't know how to do that. That's amazing. Yeah, I think there's still a little bit of code involved in Vercel's eve.dev. But I think it's directionally super, super interesting.
To the benchmark for coding agents, Terminal Bench 2. And so yeah, that's the benchmark for coding agents that most people use, and it builds on top of Harbor. So Harbor's the framework that will probably be most useful for building your own benchmarks.
What is being done to keep the token count low inside of framework? Great question. We think about this a lot. So when we think of LangChain, we think about giving people tools to control the token count themselves, but we don't want to have too many off-the-shelf things. We put all of those in Deep Agents. So Deep Agents is where you can see a lot of our best practices for keeping token count low. Specifically, some of the things that we do: if a tool returns a really large response, we basically, like let's say it returns 60,000 tokens, we'll put that in a file. We won't show that all to the agent. And then we'll show the agent like the last thousand tokens and be like, "Here's the last thousand tokens. If you need more, go read from a file." And so we do that.
We do summarization when it reaches some threshold. We also, one other, one last clever thing we do is there's some files that have really long inputs. So if you think about writing to a file, the agent might generate an input that is really, really long. Over time that starts to add up. You can actually remove those inputs because it's already saved in the file. And so what you can do is after some period of time, you can basically remove those inputs. And if the agent really wants to remember what it wrote, it can just go read it in the file.
The thing that I'd say about all three of these is you want to do them cleverly so that you don't necessarily break prompt caching if you don't have to or you don't want to. So prompt caching is basically when if you send the same prompt prefix, so the first same number of tokens to the LLM over and over again, it basically uses a cache for those. And so if you start messing around with the token context, then you don't have that same amount of cache. And if you hit that cache, it's basically cheaper and faster. So you want to try to hit that cache, all things else equal. So we do all these context engineering things in a way where we try to keep that cache relevant.
What is an industry standard pass rate for LLM evaluation in percentage? This is a great question because there is no good answer for this. It's completely application specific. It's really, really hard to give an off-the-shelf answer for this.
Do I foresee a time where other agents will be embedded in this life cycle? For example, agents testing or monitoring other agents, or will human input always be needed? Yeah, great question. LangSmith Engine is our answer for this. So we very strongly believe that other agents will be involved in this life cycle. We'll add more and more capabilities to Engine. At the same time, I do think human input will always be needed. I think the key is figuring out scenarios where you can have an agent doing a lot of the work, but it's still presented as like a first draft that a human comes in and approves. So this is what we do with Issue Board. LangSmith Engine creates issues, but these are just issues on an issue board. If you want to apply a code change, you have to go in and press accept. It's still pretty easy, you just press one button, but you've got this human in the loop accepting it. So, both.
Is there a way to test an agent for new/novel experiences or scenarios? This is a fantastic question and one that we're thinking a lot about. So I have maybe two answers here. One which probably most directly involves this: you can use another agent or an LLM to generate a bunch of scenarios or inputs. The question then becomes like, how do you know if the agent's responses are right or wrong there? So one clever thing that we've done in the open source, it's called LangFuzz. It's from like two years ago, so we should probably update it. But basically what you can do is you can generate two inputs to the agent that are semantically the same. They're asking the same question in just slightly different ways. And then you can run the agent over both of them. And if it gets drastically different results, you know one of them is wrong. You don't know necessarily which one, but you know one of them is wrong. And so you can then use this to identify places where it's messing up.
The other thing that I want to call out here that I just think is new around evals is this concept of like environments. So as you start working with agents that are doing more and more complex things, there's a chance that they start interacting with stateful environments. So what I mean by a stateful environment is something like Linear. Like if you're using Linear or Jira and you're interacting with the tickets there, you may not just be reading tickets, you might be writing tickets or updating tickets. How do you do evals against this stateful environment where if you test against your Linear system that changes over time, and then also like you make changes on one eval run, you know, you want to reset basically to this clean state for the start of another eval run? So we're spending a lot of time talking and working with people on eval environments and creating these synthetic environments to more easily test agents.
Do you have a module to monitor the deployed agents for drift adjustment/changes required in prompt skills references? Yeah, so two things here. Engine will do some of this. So Engine will look for issues and then suggest changes to prompt skills references. If you want to look more at kind of like just like trends over time, we also have a feature called Insights. So Insights basically does hierarchical clustering to categorize how people are using your agents. And this is where you can see drift in topics over time.
All right, let's go on to some more. Just like LangGraph Dev, is there anything for prod servers without using LangSmith as I have my own infrastructure and OpenTelemetry set up? So two answers here. One, you can use all the pieces of LangSmith modularly. So if you want to use just deployments and then not use observability, you totally can. If you want to use just observability and not use deployments, you also totally can.
But then also, we do have some guides, although I don't think an off-the-shelf solution for deploying LangGraph totally open source. But like LangGraph itself, totally open source. All the checkpointing that does the human in the loop and persistence, totally open source. And so if you want to roll your own, you can. We have some guides. We don't have kind of like a first-party offering that is also open source, though.
Oh, I like this question. When you think about agents, are you imagining AI that is performing actual work or glorified chatbot? More often than not, I see businesses talking about AI agents and it's simply a chatbot with RAG. How do we help AI agents grow beyond their definitions, do real work that removes load from humans? This is a fantastic question. I totally agree that a lot of AI right now is chatbots. I absolutely think though that the future is doing work. I think some things are needed. One is just like the right UI/UX. If you ask AI to do work completely autonomously, the bar for doing it is going to be much higher. If you ask it to do a first draft of something, and then the human approves it, then I think you'll see more of these agents doing things. So I think there's some stuff around UX to figure out.
I also think that in terms of actually doing things, I think like ambient-style agents are really interesting here. Basically, can you run agents in the background triggered by events? They're seemingly always on. And I think this will let humans leverage themselves more than just going to a chatbot and asking a question. So I keep an eye out for ambient-style agents here.
How do you run/set up a virtual file system? Any particular way to set it up like a RAG system or other types? Like literally a regular file system? It can be anything. Yeah, that's kind of the fun of it. It could be a regular file system. It could be a database. You can store it anywhere. The key part is just how you expose it. So we have in Deep Agents, we have what we call a backend interface. And so this backend interface exposes, I think, six different methods that are exposed to agents. You just implement those for whatever storage you're using and then you can use it as a virtual file system. And those six different methods are read, write, edit, glob, grep, and ls/list.
What tools do you use for monitoring your agent? LangSmith all the way.
What's the difference between testing and evals? For evals, what do you do with that user feedback? Add it to context directly? So I think testing and evals generally mean the same thing. There are some distinctions. So I think you could argue that regression testing and benchmarking are different ways of doing testing or evals. So regression testing is more like I expect 100% kind of like pass rate, and probably maybe it's testing like simpler single LLM call things. Benchmarking is like I'm at 40% right now, but I want to hill climb to, I don't know, 90%. How do I do that? And then there's online and offline evals. And so maybe some people mean testing to be offline evals and evals to be online evals, but yeah, testing and evals I'd say same thing, but there are different terms within them.
Yeah, this is another good one. How do you evaluate agents that don't have simple success criteria? The success needs to be evaluated by an SME, for example. Yeah, really good question. This is where I think human input will always be valuable. We have a concept in LangSmith called annotation queues. That is where we see people labeling and we see humans labeling and evaluating these agent outputs. Oftentimes what they're doing is defining a criteria so that it's more scalable. So these criteria oftentimes come from SMEs, but hopefully they are scalable and can be run without always having to have a human look at it. If you do always have a human look at it, we have a concept of annotation queues and it can help with that.
Can you show us an end-to-end demo? Probably ran out of time. Although maybe... okay, I'm not logged into LangSmith. So no, but if you reach out, we can have someone on our team absolutely help with that. Let me also stop sharing my screen.
What are your thoughts on agent user analytics? I.e., analyzing what goes on between users and agents to extract business behavior insights. Do you plan to add a similar on top of LangSmith? Yeah, absolutely. I think this layer is really, really interesting. I think online evals get at this. I think Insights in our product get at this. I think a lot of times we want LangSmith to be used to analyze not just errors and latency, but these kind of like fuzzy things.
Will online eval make agent too slow and how far can guardrails be used to help in online eval? Yeah, this is a good question. So online eval runs after the agent runs. So it doesn't affect the agent run at all. Guardrails are very similar to online evals in that they could be testing for the same thing, but they run as the agent is running. So guardrails may make the agent slower, online evals will not make the agent slower. Online evals may cost a lot though because you're potentially running an LLM over all traces. That's why we trained our own small language model because we want it to be cheap.
How do you think we should plan to build a self-improving agent on this framework? This is a great question. This is probably also where I will end off. I think largely because this question perfectly encapsulates kind of like everything that we work on from the build to the monitor to LangSmith Engine.
First of all, this is still very much not easy to do. But we are very interested in this. We put out a video today. It uses LangSmith Engine as memory for agents. So specifically what we do is we take Deep Agents. We're running Deep Agents. We use a virtual file system to say, "Hey, everything in this memories folder for Deep Agents, this is actually stored in LangSmith Context Hub." We then trace everything to LangSmith and then we run Engine in the background. And so Engine will look at all these traces and when it sees things that the agent should remember, it will update the Context Hub. And then the Deep Agents, because it's connected to that Context Hub, it pulls it down all the time.
So Deep Agents plus Context Hub plus LangSmith Engine, those are three pieces of like what one of our best guesses at this like self-improving agent on this whole framework and platform looks like. We just released a YouTube video for it. I think if you go to our YouTube page, it should be the first video there, as well as a blog on Twitter as well. I'm going to go retweet it right after this. And so yeah, this is a great way to end it. I think that encapsulates everything we do.
With that being said, I know there are a bunch of questions here that weren't answered. If you still have questions, feel free to reach out to me on Twitter. I'll try to respond as much as possible there, or shoot us, LangChain the company, an email and we'd be more than happy to help. Thanks for attending. Always love the feedback, so keep it coming. Thank you guys.
