And now, the final act of the day. We've got Harrison and Sam coming up. It's rare that you get two folks of this caliber on the stage at the same time. We managed to do it. So I'm going to present Sam Parti and Harrison Chase. Get on up to this stage and let's give them a big round of applause.
These guys need no introduction.
Way too nice of an introduction.
Do I need this? Do I get a mic?
I think we both have mics.
Oh, I don't think I need this. I don't need it. All right, we're figuring this out live. Um, hello. My name's uh Harrison Chase. I'm the co-founder and CEO of LangChain and I have the clicker here which I'm giving to my friend Sam.
Thank you for announcing that. Hi, I'm Sam Parti. I'm CTO and co-founder of Arcade.dev. Uh, and today we're going to talk about general purpose agents and something that Harrison and I have talked about for a long time, which is why coding agents are actually the foundation for general purpose agents. This might not immediately make sense, but hopefully over the course of the talk it does make more sense.
So what is a coding agent? What does it help you do? Things that we use every single day, right? Claude Code, yada yada. What does it have access to? A file system. It reads and writes files. It is able to over time over its iterations keep and store not only things like memory, but just task execution data in a workspace local to itself. And so in doing so, it's able to keep and store things that as it's working on, it can go back and reference and keep not in its context window, but in something that is a retrievable object. And so this is a paradigm that you can extend not only through just simple search like RAG, but all the way through tool calling. And we'll show over the course of the talk how it extends through a bunch of different paradigms as well. Now, the point of the talk is actually also to get you to a general purpose agent, not just something that works on your laptop for one user, but something that works for an entire organization for all of your users. And so, how do you do that? Well, you start with a coding agent. And that's the primary object of what Harrison's going to talk about today.
So, what are we going to talk about? We're going to talk about two things. First, Harrison's going to talk about the agent harness and why it's important and why this gives you the coding agent paradigm. Why this gives you a basis for what you need for that Claude Code like experience and then I will talk about the tool runtime and what's necessary there to make these two things combine into what is necessary in our view for a general purpose agent for your company. Not only for yourself but for your company, for your friends, for your users, what have you.
These two things that we can say when we put together make a general purpose agent uh have different responsibilities and they might seem like they're really similar. It might seem like Claude Code calls tools and reads and writes files through tools or that it extends its memory through some interface through the tool runtime when actually these two things actually have very different responsibilities and have an incredible responsibility in the foundation of a general agent. So first Harrison's going to talk about that agent harness piece and that is the foundation that gives you that Claude Code like experience and then I will talk about the next piece that brings it out, uh, the general purpose agent as well.
Cool. So agent harness is a fun term to describe the scaffolding around the model that lets it interact with the environment and do things. And so I want to talk a little bit about how we think about agent harnesses at LangChain, how we've thought about building uh our own agent harness called Deep Agents. Um, and then some of the things that we've added into it which are very similar to things that are in other agent harnesses like Claude Code or Codex or things like that.
So, uh, what is an agent? Most people would probably technically define an agent as an LLM running in a loop calling tools. Uh, so, so what is the difference between that and an agent harness? Basically, an agent harness adds more things in. Um, so you can think of it as having more batteries included. And so, what are some of the things that it adds in? One of them very commonly is a planning tool or instructions for doing planning. Um, another uh very common component is file system tools. Uh, so these are for interacting with the file systems. These can be to read and write, um, uh, search files, glob, grep files depending on the harness that you're using. It may have a few of these. It may have a lot of these. That kind of depends um on on what else it has uh in terms of kind of like code execution tools because sometimes you can just execute code to do the same thing. Um, and and and the file system is oftentimes where the agent definition is actually stored. So you can represent an agent as basically an agent.md and skills and an mcp.json. And if you think about all these things that you use in coding agents to define the code to to kind of like guide the coding agents to do things, I would argue that's basically how you can define a custom agent. You can kind of think of your coding agents as custom agents in some way. Um, subagents are another aspect of this that can also kind of like be defined by by files on disk or or or through some other method.
Um, we build a bunch of open source things at LangChain and that's kind of evolved over the years as the whole space has evolved. Um, we started off with LangChain in the middle actually as kind of like an agent framework and then we actually built LangGraph as a lower level kind of like agent runtime. So we think of a framework mostly concerned with abstractions and integrations. Um, we we think of uh LangGraph our runtime as more infrastructure almost. It has durable execution, it has streaming, it has human in the loop, it has persistence. And then recently we started on Deep Agents which we think of as this agent harness and it really has more of of these batteries uh included things.
So what's actually in a harness and I want to talk about this in a little bit more detail. Um, so we talked about file system. So these are the six that we have in Deep Agents. We have uh list, read, write, edit, glob, and grep. Other agent harnesses, so this is pretty heavily based off of Claude Code. So it actually has all of those. Codex has a smaller subset. So it I think just has I think it just has read and write and then it relies on uh using the bash tool for for other options. So you can pick and choose which ones uh you want here. Um, one of the really cool things that we added in Deep Agents that is pretty unique is that we have this concept of what we call kind of like a pluggable backend. So when you run Claude Code, it runs locally against your file system like your real file system that you have. What we have in uh Deep Agents is is that file system. It it can be your real file system or it can be a virtual file system. So we can use a database to store uh quote unquote files. So you store it in a database, but you represent it as a file system to the agent because that's what it knows and that's what it interacts with. This also makes it easy to run uh Deep Agents remotely because you can just plug it into a sandbox or something where the files are stored um uh elsewhere.
Um, we we have a planning tool in Deep Agents. This is a really simple tool. All it does is it asks the LLM to generate a plan. It doesn't even save the plan. It just puts that plan in the in the LLM's context window. It kind of it's like it's taking notes to itself as it goes along. Um, this is pretty heavily based off of Claude Code's planning tool as well. Other agents uh do things where they actually write the plan down to a markdown file. That's obviously a much more persistent way of of keeping it. Um, and uh but but a lot of the planning tool is really just to have the agent think about the plan and put it in the context window so that it informs future generations.
We have sub agents in uh in Deep Agents. They work pretty similar to sub agents in other agent harnesses. The um the really useful thing that sub agents do is they have this context isolation where you give a sub agent a task and it just sees that task. It doesn't see anything previously and all the work it does. The main agent doesn't see any of that. It just sees the final result. And so that's really good for spinning up a bunch of these in parallel and having them just work on things in a very focused way and then they come back and report the main result. Um, fun kind of like side note there, there can absolutely be communication issues here. So, you need the main agent to be specific about what the sub agent should do. And the sub agent needs to respond with the right info. Um, if there's breakdowns in either of those, then this doesn't work as intended. Skills. Uh, so we support skills. Most coding agents support skills. They're pretty cool and a nice way to package up instructions and tools in a way where you can kind of like discover um them as you go along.
Um, some of the things that we do around kind of like context engineering that are built into a harness. And again, this is what I mean by harnesses comes with more batteries included. This is this is kind of beyond the LLM just running in a loop calling tools. One of the things we do is that I think other harnesses do as well is offload large tool results to files. So rather than see like a massive JSON file that's returned from a tool, we'll actually just put that in a file, show you the first hundred lines, say here you go. If you want to see more, you can read the file and then and then it can explore it that way. So this prevents you from kind of like blowing up your context window quickly and and lets the LLM a lot of this is in spirit of giving the LLM more control over managing its own context window. For compaction, uh what we do is when you get to a certain uh point relative to the context window size, we do a compaction step. We save all the original messages into the file system so that if it wants to go look at those later, it absolutely can. We're also thinking about giving it a tool that will let it trigger its own compaction. Um, so again in the spirit of giving the LLM more control over its own context window. Human in the loop is really important. We built Deep Agents on top of uh LangGraph. Um, and so it naturally supports pretty robust human loop controls. And so we have a nice little config where you can specify whether you want tools to interrupt before they are executed or not. And with that, I'm going to hand it over to Sam to talk more about some tool stuff.
Well done. So, human loop is actually a really good jumping off point. Um, because it's one of the points at which the harness needs to communicate with something that's other than the file system of that agent. Um, and so what you just heard from Harrison a lot was how that agent can utilize those local aspects of the computer. Um, from everything from just writing and reading a file to like something like computer use, right? And so these are necessary aspects that the harness provides. And so where does the tool runtime come in after that? What is necessary besides that local aspect of a standard IO type MCP server working with a a harness, right? And really the essence is in the multi-user and authentication authorization and third-party services and that the things your users or you want to be able to use. And so, how do I access my data and secure uh my services securely that aren't on my local computer or that my users want to use with my agent uh that aren't local to their computer.
So, I'll give you an example of this. Uh, let's say you're booking a flight, okay? And you you want to do so with an agent. And I know this might seem not realistic, but this is actually something that an enterprise company of ours does now with agents. Um, so you have to check a calendar. How do you know which calendar, Outlook or Google? How do you sign in as that user? If you're giving this agent to a company, how does that company use their SSO to sign into that agent? Finding flights. Do you have an integration with that? You might have an MCP server, but are you going to surface all of the flights and the flights that are particularly relevant for that particular user? What happens when you log in to Amazon.com? It's your Amazon. It's the one that has the relevant results for you. It has personalized results with your information curtailing those specific results. Why don't agents do that yet? Book the flight. Actually pay for it. Payment mechanisms. Notifying your team on Slack. Once again, OAuth integration, OAuth integration, OAuth integration, OAuth integration. This is the necessary evil that the tool runtime is responsible for is the dirty work of making sure that every system inside of your company, every enterprise that uses this particular one-off OAuth server for their one-off service and this integration here, there and beyond all integrate with your agent properly that's running inside of the harness. This runtime has that responsibility of both that authentication and authorization layer as well as that integration layer. These two can't be separated because now the agent is an intermediary that you can't take out of the equation. It's no longer just a web app with social sign-in you can use a user token for. And why is this? Well, there's two failure patterns we see. One is a service token. So what happens with a service token? Well, it's either got incredibly elevated privileges to the point where no CISO is going to let your product be bought at enterprise or it's got so low privileges your agent's not useful and it's only good for one maybe local user on their laptop. And that could be helpful, right, for that single user. But when you want to deploy an agent firmwide, how are you having that firm's SSO or login making sure that that particular user is using that particular agent? And then even more than that, authorizing them to access their Google. It's not just the front door OAuth that is. It's the authorizing them to access their own Google. And so there's two forms here that that integration can't be separated from now that the agent is an intermediary.
This ends up looking like what we call delegated agent authorization. Essentially, you're not giving it a user token. You're giving it a portion of your user token for as many services as there are. The way we've approached this is that Arcade will hold uh hold one token for the subset of permissions you've authorized for that particular agent at that particular time for that user for that service. So we've covered both of those doors such that the least level of privilege is given to that agent for that particular user in that particular company. And in doing so, it's the same model that you would have had for a website with Okta in the front door and had no authZ. But now an agent has the exact same profile that a CISO has approved for the last 15 years.
And so what does this actually look like in code, which is what I always wish people did in talks when they talked at this kind of level. Uh, what does this look like in code? Well, this is an example of a tool to go to Reddit. And now, it actually isn't the whole tool, okay? It's just basically the function signature, but it is the part of the tool that specifies the exact level of privilege needed by this action that the agent's going to take. And in doing so, the agent now can say, does this user have that privilege for that service in this database? And in doing so, you can now say with certainty that no agent will take any action that an authorized user did not authorize them to take and that you do not have a log of them saying yes to. And human in the loop of the agent harness is a big part of this because how do you surface when the runtime says no to the user? We do this in a Slack agent that a lot of enterprises use. What if it's just... Yeah. So I forgot that I had animations in this slide. It's not it's not the user's token for that service. It's a delegated subset. You don't want to give an agent rm -rf to your Google Drive. That's a bad idea. Giving them delete is almost never a good idea. And so why delegate them that privilege with your user token you've been sticking in your environment variables on your laptop for a standard IO server, which stop doing that by the way. But please, someone posted one on LinkedIn the other day. It's like, man, I can log in as you right now. Um, but the user, it's not the user token. It's a subset. And even if it's just a secret, once again, please don't put this in your environment variables. Instead, put them in a secret store. In the case of Arcade, we've made it so that each function can be labeled with the secret name, and that can be for an org, project, or a user. And so, you have multiple levels of control over where that secret gets applied. Moving on, it's injected. It's not in context. But I'm gonna Well, I probably should have paused there, shouldn't I? Uh, contextual access is another part of this. So, what if you already have an ABAC system? What if you already have some way you do PII? If you already have some way you do named entity recognition and you don't want to replace it, well, what we can do is now take your tools and say pre and post execute, what do you want to have happen? And in doing so hook into your entitlement system or your Kerberos system that's been there for 40 years. It's this legwork that the tool runtime is responsible for and takes care of when you actually want to take your agent and go sell it to an enterprise.
And so in doing so, it gives the agent harness an ability to say, I'm positive that this is that user at this time for this action with this level of privilege to go out into the world and take some action on behalf of that user. Not just for that user, but as that user. It's probably the most important part. Doing work for the user is good. Doing work as the user is much better in many cases because in doing so you're doing their work for them, not a bot, not a service token that's over or underprivileged. You're doing their work as them, which is usually much nicer sending the email as you instead of making you copy paste, for example.
What this ends up looking like is something along the lines of this diagram. I tried to make this as uncomplicated as I could, but obviously it's a pretty big system. So, I will say there's a Helm chart you can read, which for me is much easier. However, you can see some of the things that are part of the runtime here. I mentioned the connection to enterprise systems, but I feel like most people could imagine an MCP server writing a tool that's privileged for their system. What's a little bit harder is actually those layers of the registry access. How do I manage all of these things? Is how do I do this at an enterprise? Um, and that's what you see in the core on the right hand side there.
All of this, of course, connects directly to those harnesses that you see. Put it into LangChain, put it into LangGraph, you're writing it from scratch or take Deep Agents and just hook up the MCP server directly. Put it in a skill, name the MCP servers and it works immediately.
This is what that looks like. You can basically just go inside of Arcade and use our gateway, which essentially takes multiple MCP servers and says, "I want a new MCP server. Give me that tool, that tool, GitHub, Google, Microsoft, this, make an MCP server. Now serve it on the internet securely." That's what a gateway does. And it looks something like that. And then you can take that URL and that slug and plug it right into LangChain and use it immediately or Claude Code or what have you. Deep Agents is pretty great.
So bringing this all together and we'll talk about something we've recently done together to kind of bring this even more together. But these two pieces are distinct in the fact that they're responsible for major pieces of making an agent general and the purpose of that agent general. Hopefully that has come through.
And so talking a little bit about uh what this looks like in a general purpose world. One of the things that we have a really good integration for is what we call Agent Builder in Arcade. So we talked about how this underlying harness which is very similar to coding agent harnesses is is can in fact power general purpose agents. Um, I don't uh I don't really think the interface for general purpose agent is in the terminal. Uh, I think that's um I don't think my mom would use that. I think there's a different interface that other people would and I think you see some ideas with this with with with Manus and Claude Cowork kind of like pushing in this direction of what does a more user-friendly interface for these things look like? And and so that's what Agent Builder is. It's powered by Deep Agents. You can connect to how how many tools does Arcade have? Thousand?
Over 8,000. Okay. You can connect to over 8,000 tools uh through there as well. Um, and uh and and what it is is basically a way to create agents in in a no-code way by just chatting with them. One of the things that's built into to Agent Builder is this memory aspect. The memory is just done by modifying the files on the file system. We talked about how you can represent agents as files on a file system with agent.md and skills and sub agents and mcp.json. And so when you talk to this agent, all it does is it just modifies those files and it it creates and evolves its definition of an agent along the way. So we have a number of very general purpose templates here. Uh, so the where is it? The email assistant is one that I use uh I use daily. I no longer check my email. I check my email.
He actually does use it. I actually yeah I I've been I was I I was using it I've been using it for like a year and a half. It started
It's actually been a while since we did Agent Inbox.
It started off on LangChain and then went to LangGraph and then went to Deep Agent and now it's in Agent Builder and um and and and uh uh uh LinkedIn Recruiter is a good one as well that that we use to kind of like source candidates. We're hiring by the way. Uh nice plug for that. Um
The social media one's fun too.
The social media one's fun too. You can, there's one that someone put out recently to impersonate anyone you want on social media, which may or may not be a great idea, but it is very fun to do.
This is the integration that you see right here. Essentially, it's a very similar page to what you've seen before. This will be coming out soon, but essentially, it'll enable this already really easy to use interface to just access all of the tools in Arcade. Hence, giving all of the Deep Agents that are inside of there with those easy to use templates a ton of different tools to be able to use. So, uh, hopefully most people can see how that'll be really fun. Um, and an easier to use interface like your mom would use.
Yeah. Yeah. We've done integrations for a while, but this is definitely the tightest one. And so, excited to get that. And maybe just wrapping things up with uh three real simple takeaways. Tools really matter. Um, and if you want to use tools in a production-ready way, use Arcade. Like the design around the tools. You guys wrote a really I don't know whether it was a blog post or website but you had like 80 different like tips and tricks for tools.
Tool patterns you can check out online basically just things we've learned about building tools since the past two and a half years.
Um, two, the harness matters. So this is this is what's powering the agent under the hood. Um, you can use LangChain, Deep Agents, or the best part is it's fully open source so you can uh fork it and then modify it as as you wish. We have lots of middleware to hook in and control that. And if you want to use them together and you don't want to code at all, but you want to get the power of these agents, use Agent Builder. Um, and yeah, that's pretty much it.
That's pretty much all. Yeah. Thank you guys for coming.
All right.
Nice. Who's got a question?
We have 15 seconds left. Come on. That was pretty good, right? We timed that up.
Perfect. And funny story, the reason I got Harrison to give this talk is I prompt injected his little Google agent.
I can't believe you said that.
