Horizontal vs Vertical Agents: Who Wins? | Model Routing: Commodity or Increasing in Value?

I really underestimated the demand for inference. Everything just works. This market is so big. There's just so much demand for it.

I'm excited to be here.

We build tools to make it easy to build agents. So, we build developer tools primarily. We have an open source framework which basically connects the model to everything else that you need to build a useful agent. So tools, data, computation, things like that. And then we also have LangSmith. LangSmith is observability and evals. Agents go wrong all the time. How do you know how they go wrong? And how do you measure them so that you can increase their performance over time? That's what LangSmith does.

The hard part about building agents is they don't behave how you want them to behave. So, the models are these like non-deterministic black boxes, right? And they take natural language as input and then they output natural language. And inside the black box, they're also nondeterministic. And so what that means is that the input you get is like infinite in dimension. It can be anything. You put a chatbot in front of someone, they can type anything that they want, right? And so that makes it really hard to know what they're going to type. And then once it goes into this model, and by the way, agents are not just one model call. They're a series of model calls and they're interacting with other tools. And so you have these small black boxes that are then put inside this large system and you get basically these cascading types of errors. So you have this very big kind of like input space. You have this very big output space because two similar inputs can go in and they can arrive at completely different kind of like answers. And the thing that's making all this hard is the LLM itself. Yes, they're very, very smart, but you don't know exactly what they're going to do, and they change answers based on small changes to the input. And so you build this system, but it's really non-deterministic. And so getting this system to behave reliably, like that is the hardest part. Like when we talk about agents 3 years ago, 4 years ago to agents today, they've gotten a lot better. And the main thing that's happened is one, the models have gotten better, and two, we've figured out how to build systems that basically harness these models better. And that second part, the harnessing of these models, is what we focus on.

The main way our business changes as the model gets better is we basically have to erase big parts of our product. Um, so what I mean by that, and Cursor said this really well as well, I think this was one of their actual operating principles as a company, is like erase the product or delete the product, I think is what they called it, where basically like these models get better and things that you had to do 6 months ago you no longer have to do. And so this is very relevant in our open source. So open source is all about basically the harness and the framework for taking these models and helping them do things. And the right way to take these models and help them do things changes every month, 3 months, 6 months, choose your time period, but it changes pretty rapidly.

There's no easy answer on one hand. What are you going to do? Not build? Like that's what I like to say to founders who ask this, like what do I do? Like stuff is changing, um, you know, the models are changing, uh, if I build something now it might not be relevant in 6 months. What's the alternative? You just don't build anything? And so I think you have to make kind of like educated guesses as to what parts are going to be around and what parts aren't going to be around in the future. So for example, early on, the way that we would do tool calling in LangChain was by having the model generate a JSON DSL and then parsing it. And uh, that was something that I think we all thought or knew was like a little bit of a hack at the time, but we had to do it because that was the only way to get it to actually steer things, because you have to get it to do this type of tool calling. And then a few months later, OpenAI came out with tool calling. And I think we recognized pretty early, and this is like a more native version of tool calling that's directly in the model and in the APIs. And I think we recognized pretty early on that that was the future. And so we invested a lot of effort and just completely switched from JSON parsing to this tool calling. Um, and so I think you have to basically recognize what things are short-term hacks and what things are long-term kind of like will always be there. The models will never know your proprietary data. They will never be able to kind of like actually get the output of APIs. You have to take their like API call, pass it to the API, get the output and then continue in the loop. And so I think the system around the model connecting it to other places will always be there. But a lot of the hacks we do around chain of thought or planning or tool calling, things that can just be much more closely integrated into the model or into the model API, will go away. And I think you just have to be quick to recognize that and go on.

I think the reason we need neutral harnesses is that there is a lot of state that you build up either before you build the agent or as part of the agent running. And the state can include things like memory, whether that's short-term memory or long-term memory. Memory also comes in different form factors. It can be from interacting with the agent versus from the agent learning from the environment. And the reason that this state gets built up is the agent interacts. It goes out, it learns, it chats, and it learns things to make it better. I think one of the most interesting things right now is how to make these agents do better over time and learn. And some of that learning and some of the doing better over time will be in the model weights, but a lot of it is outside the model. And a lot of it is in this context. And right now when you build up this context, you really don't want it to be linked to a single kind of like model provider. Um, and the reason for that is that then it becomes linked there. You can't go to other models. Being able to change models is very offensive and defensive. So it's offensive in the fact that you want to be able to take advantage of the best model when it comes out, no matter who it's coming from. It's defensive in the fact that you don't want to get locked in so they can't raise rates, things like that. So you want to be able to switch models. The harness can be very tied to the model. It can also be neutral. The context right now is really intricately linked to the harness. So the harness's job is to basically take the model and interact with all the other things. All the other things includes data and APIs, but also includes memory. And so if you build this memory in a way that is linked to the harness and the harness is linked to the model, then you're kind of stuck. You're kind of...

Yeah. Yeah. You want to basically not be locked in to models. And if memory, if context starts building up in a way that is specific to how one model wants to interact with memory or wants to interact with context, that's very bad for portability. And then that puts you at a disadvantage building your business going forward. We talk with a lot of people about owning their intelligence and what that means, and there's a lot of different versions of that. The version that we think owning your intelligence means, we think there's a few different components of that, but one key component is having kind of like this open and controllable agent. Like you want to be able to adjust all the different pieces. You want to be able to adjust the model. You want to be able to adjust the harness as well because sometimes these model labs are training harnesses to be really good at the things that are in distribution for them, which oftentimes is coding. And coding is great and there's a ton of use cases around coding, but there are a ton of use cases that aren't coding. And maybe they'll write code as part of it, but they're using that as a way to do sales, as a way to do research, as a way to do law, as a way to do these other vertical-specific things. And so when you get out of distribution of what the model labs are training on, you want to have control over what the harness does. And then you also want control over the context as well. So model, the harness itself, and the context make up kind of like this open and controllable agent architecture. And we think that's a big part of what it means to own your intelligence.

If value is derived through verification, I think the people who win are the domain experts. The people who are able to kind of like quantify what that value means, which is very hard. Like building evals is a very hard job. And so I think a lot of the value will accrue to whoever can kind of like write those verifiers, capture that domain knowledge. I'm massively bullish on vertical agents for different domains for this reason because I think the domain experts who are best suited to write those verifiers or build those evals, as we kind of talk about it, they will be able to build harnesses and models and it won't be just the harness, it won't be just the model, it will be some combination of them. So very practically, if you're getting started building a vertical agent, I would highly suggest you start at the harness layer, you do context engineering, you kind of like tweak the harness, and then when you've got users, then you start tweaking the model in post-training. I think it's much higher ROI to start at the context or harness layer first. And so I think we'll have all these different verticals. They'll be able to quantify what good looks like to them in a very domain-specific way, and that'll let them build not only like vertical agents but also vertical agent experiences. Different agents might have different experiences, and so I think like a lot of the value will... I'm massively bullish on vertical agents, and so we want to enable a lot of that.

I think a lot of it is they are post-training open-weight models. Absolutely. Um, I still think that's a valuable thing to do. I don't think there's kind of like anything wrong with that. I think there's different like dimensions that you can go in terms of owning your intelligence. I think the cheapest way is you just control the context. You use an off-the-shelf harness, off-the-shelf model. The next level down is you start tweaking the harness and that gives you more control than just the context, but it's a little bit more expensive to do. The next level down is you post-train an open-weight model. And so you might take an existing open-weight Chinese model and post-train it for your domain. And then the very kind of like bottom layer is they actually train a model from scratch. I don't think many folks are training kind of like a model from scratch. I think most are post-training. I think if you're training a model from scratch, you're probably at Nebius. I think those are one of the few that are doing that. Um, but yeah, the Ramps and the Harveys of the world, they have a ton of talent. They have a ton of data. They have a bunch of usage and are trying to think about really optimizing it. And so they've gotten all the juice they can out of the context, all the juice they can out of the harness, and now they've arrived at basically post-training of these models.

I think what we've seen in regards to post-training is that a lot of the post-training that practically makes sense right now is probably for like simpler tasks that make up a larger harness. So what I mean by that is like when you have an agent harness that's doing interesting work, it's actually very rarely just one model running the loop. That might be the core of it, but there's a bunch of other model calls that happen either as like guardrails or routing or sub-agents or sub-workflows or things like that. And so a lot of the places where we see post-training being efficient now are for simpler tasks like classification or extraction or summarization that might be a tool the agent uses or a guardrail or things like that. I think those types of post-training right now make a ton of sense and will continue going forward. I think the question that is more unanswered is for this core model that's driving the loop, because the main way an agent works is there's an LLM running in a loop calling tools. And so the question is for this model that's driving that loop, will those be kind of like post-trained models or will those be kind of like frontier models? And I think this is where we see more of like a back and forth where like every time a new frontier model comes out, the better they are at just running this kind of like core loop. We do see people like Harvey and Ramp and others kind of like experimenting with post-training models for this core loop. And so I think Harvey recently in the past month or so announced something exactly in that vein. Um, I think that's an open question. I think...

Not to start. Absolutely not to start. But like if we're talking like 5 years in the future, assuming that I'm getting a ton of usage, um, I think it could be a realistic possibility.

Model routing is so interesting. I think it's one of the hottest and maybe most divisive topics right now. Um, so a few thoughts. First, there's a question of where the model routing should even be. Should it be at the harness level or should it be in the gateway level? And this is like getting a little bit into the technical...

The difference is in the harness level it's in kind of like the code of how it runs, and in the gateway it's just an API call. So the benefits of putting it in the gateway level, because it's just an API call, you can drop it into any harness. You could put this routing that's happening at the gateway level into Claude Code, into any harness without having to change the code. The downside, and why I think we're more bullish on routing happening in the harness, as I think Factory and Cognition have also written things about this, is that in the harness you just have more knowledge of what's going on. You have more awareness of the state. When you have an API, it's generally assuming that it's kind of like an OpenAI chat completions API, and so it only takes in a certain amount of state. You have more state in the harness to know, hey, like what's going on with the prompt cache. If you want to try to keep the prompt cache warm and available, you can also change the prompt more easily. So a lot of these models have different prompting strategies. And so if you do the routing at the harness layer, you can more easily change the prompt. And so I think a lot of what we've seen is one, this debate between kind of like gateway and harness and where does routing go. I think we probably believe it belongs in the...

They are in the gateway.

Had them on the show, love Alex.

I think a lot of why people use OpenRouter and Vercel and other gateways is largely for the model access. So I think there's also a difference between a router and a gateway. And so I would probably say most of OpenRouter's usage, and Alex obviously knows the numbers a lot better than I do, but if I had to guess, I would guess a lot of it just comes from being a gateway, not a router. So people specify the model they want up front, and OpenRouter just provides a single API with a single API key for accessing that. In terms of like smart routing, I think that's actually being done like way less than people think. And I think when it's done, it's built into the harnesses by folks like Factory. I think Factory does a tremendous job about this. They've written a number of technical blogs that are very well done in terms of how they do routing. But I think right now I pretty strongly believe that like routing, you have to be harness-aware. You have to know what the task is. You have to know what the harness looks like. So I don't think you'll have routing that works for law and works for finance and works for coding and have it all be the same. It's going to be very domain and very harness specific.

I think when people talk about it at the gateway layer, I think there's maybe like two things that they're talking about. So, one, it makes it way easier to port. So you can take Claude Code. People love Claude Code, right? If you tell people, hey, you can't use Claude Code, you have to use this other harness. Um, and there are other good harnesses out there, but let's pretend for a second that you love Claude Code and you want to let them keep on using that, but you want to do some semblance of kind of like routing to maybe like a DeepSeek or an open source model or something like that, then the only way you can do that is at the gateway layer because you don't control the harness. Side note, this is another argument for open source harnesses that you can control and modify. Um, and then the second thing is when people talk about kind of like routing at the gateway layer, I think they're oftentimes implicitly assuming that there's a domain. So there's kind of like two general domains that you could maybe argue that you can build general routers for. And I actually think OpenRouter has done this to some extent. You could argue coding. Coding is like the most popular domain. That's where all the use is going. Let's build a router that works really good for coding. Maybe we won't be like harness-aware, but we'll be like use-case aware. So we know different models are good for different things in coding and we can put it at the gateway. The other thing you could say is like, hey, we've got this like general chatbot, like we're not trying to be best in the class at law or best in the class at finance, but we're just going to be like a general chatbot across the company. And like we can know well enough that if you say, hi, how are you doing? You don't need like Opus for that. You can use a cheap model. And so I think when people talk about routing at the gateway level, they're implicitly kind of assuming that you're doing either like general-purpose chat or coding and you're not doing like a domain-specific thing. As soon as you get into a domain-specific thing, I really think it has to be at the harness layer. You're not going to use an off-the-shelf gateway router for that.

I think that's exactly right. If you're building a vertical agent, I think routing needs to be in the harness. If you're thinking about a general-purpose horizontal task like just general chat or coding, it could maybe be in the gateway.

I don't think routing is a commodity. I do think people will build it, but I think building it is quite hard even today. And I think part of the reason for that is the models just change so rapidly. So like you can argue that certain models are better at different things, but then a new Opus or a new OpenAI model comes out and it's just better at everything, and so then the value of routing kind of goes down.

I think evals are useful, but I wouldn't trust them fully. And so what I mean by that is I think a lot of the academic evals that are out there that are used when a lot of the models come out are not really that useful anymore. I think all the models that come out are kind of benchmark-maxed against those evals, and so you don't learn a lot. I think internal evals that you have and that you build for your domain absolutely can still be useful. And so I would... I mean, this is the main reason that building evals is such an important part of building a vertical-specific agent. I think you have to have some way of measuring things, and I think you can generally trust those evals. Um, but even then, one of the most important things to do is just look at the data. Just like look at the data and see what it's doing. Don't rely too much on kind of like a single number. I think this is why you want to have traces. You want to look at what's going on. Maybe your evals aren't perfect. Evals are really hard. You can have some sense, but maybe models are doing better, but in just a different way that you didn't know how to capture before. And so I do think, yes, I think you can trust vertical-specific benchmarks, but you have to evolve those as well. And that's one of the main things that we've seen because these models just get better in different ways.

If you're building an agent, I think the value is at what makes that agent good at the domain or vertical that you're working in. So what I mean by that is again, I think there's different ways to control what an agent does. There's the context, there's the harness, and there's the model. Having specific proprietary things for any of those can be an advantage. I think controlling the context is the easiest and the most obvious thing to do when starting, then controlling the harness. And that would be data, that would be memory, um, that would be specific APIs, um, specific tools that you can use. Controlling the harness is like a little bit harder, uh, but still you can still put a lot of your domain knowledge around what should do both routing and things in there. And then the model is the hardest to control but potentially the highest ceiling. The thing that lets you iterate on each of those is evals. So there is a ton of value in these, but there's only value in that you like capture insights that you bring back to the context or bring back to the harness or bring back to the model. And that exact process can look different. So post-training, great for bringing these insights from traces back to the model. For bringing things back to the context or memory, you're probably going to want to run like another agent over those traces and then have that agent output kind of like learnings for memory. And then for bringing these learnings in the traces back to the harness, you're probably going to want to run an agent over those traces and update the code of the harness itself. And we actually have something that does this for people. It's called Engine. It's an agent in LangSmith that runs over traces and tries to update code. And then when you update these things, you're going to need to know that it's actually better. And this is where evals come in handy.

Continual learning, I don't think so much changes how agents work, but I think it changes the systems that you want to have running around the agent. So what I mean by that is I think the core loop for a continual learning agent, the core inner loop maybe to say, will still largely be the same. It will be an LLM running in a loop calling tools, interacting with a model and an environment. I think what continual learning adds is an outer loop. Um, an outer loop that looks at the exhaust of agents and then does things with it. And again, what it does could be different depending on what continual learning means for you. So I think continual learning is also one of those words that doesn't have a super precise definition or maybe... I would define it as a system that gets better over time as it runs. And again, there's different ways it could get better over time. It could update the context, it could update the code of the harness, or it could update the weights of the model. Um, and the outer loop will look roughly the same for all of those. It will look like take traces, get learnings, fix something, measure it to make sure it improved, and repeat. But that exact kind of like changing, again, updating weights of model, very different from updating a text file that is like a memory file.

I actually think the data market is still like pretty early in the grand scheme of things in so much as we're talking about kind of like a context layer for agents. So what I mean by that is like there are a bunch of databases that are... ClickHouse is maybe like a really good example of one that's doing incredibly well recently. Um, I think they're still largely focused on kind of like, they're a database. They're a good old-fashioned database. They're good at storing data and querying data, but it's still in pretty kind of traditional ways. ClickHouse existed pre-AI. When we talk about kind of like a context layer for agents, and again, like there will be a bunch of data produced from like software systems, and great, like I think like existing databases, ClickHouse, Elastic, whatever, are great for that. When we talk about a context layer for agents, I think a key part of what exists there is yes, there's a database, but I actually think there's a system around the database. I think like a context layer for agents will look more similar to like an agent around a database than just a database. And the reason for this is that taking all of these learnings and ingesting them into a database, you need some sort of like intelligence to do that. And I think that intelligence is actually really, really hard. The hardest part about memory is actually not the database at all. When we talked to a number of companies, the way that they do memory is they just store it in a single file. Very, very kind of simple. The hard part is taking all of the observations that happen, whether it's from the user interacting with, let's say it's a traditional SaaS, interacting with the software, or the user interacting with the agent, and distilling that into information that's valuable enough to go into this context layer. That's the hard part in memory right now.

Because it's tricky to get LLMs to do what you want it to do. And this type of meta-reflection, I think, is something that LLMs actually still struggle at a lot, like reflecting on kind of like prompts and improving prompts, reflecting on agent architecture, improving agent architecture. We should build a benchmark for this internally. We do have some benchmarks for this in terms of Engine, which is our agent that's running on things. But I think this type of meta-reflection is still really difficult for agents. I think it's also generally like a hard thing for humans to do as well. Like I don't know if we actually understand, some people probably do, so I shouldn't say this, but I don't know if we understand like what goes on inside our brain. Like I feel like our brains are still kind of like complex things that we don't fully understand. And so if we're trying to now replicate this kind of like learning process in a system, I think there's a bunch of different pieces. And so I think you have to... Yeah, I think models aren't great at kind of like reflection. And I don't think we know what this system should look like. What should the inputs look like? What should the outputs be? What should the harness for this agent be? What should the different workflows be?

Probably. I mean, it seems like they'll get better at everything.

I think the speed of model improvement is something that's probably really tough to say without being inside of a lab. I would say I think there are a lot of tasks for which the model improvements will kind of like flatline in the sense that like it's already good enough. Like I think there are a bunch of tasks where the models are just good enough.

Yeah, coding's getting there. Um, like summarization, routing tasks. Um, there are probably more difficult tasks and or tasks that we don't even know to think about, know to throw a model at. I would probably say like this type of like meta-learning is one of those tasks that the models aren't good at now. I would imagine is very important to the labs in terms of kind of like cracking RSI or things like that. Um, and so I do think the models will get better at that over time, and it's possible that that progress actually looks much faster than previous. Um, I think it's really tough to have an educated opinion without being inside of a lab. There are some tasks where it's saturated. There are other tasks where I think it will probably just increase faster than before.

I mean, I think this memory layer for agents is still massively unsolved. I think one thing about agents that is rare today but will be incredibly common in the next few years is this memory layer. So what I mean by that is if you look at like Instinct or Muse, I think one of the things that they've nailed is this kind of like personalization and learning and seeming like it's something for you, whether it's because it's speaking in the tone or because it's just following up, like that proactiveness. And so I think memory is super, super early. We've been talking with folks about memory for like 2 years and trying to do things in the open source. We launched an alpha and then killed a commercial solution for memory. It's just so early on in memory and it's so domain-specific. What memory looks like for Instinct or Muse is actually pretty different than what it probably looks like for a company or harness, and it's pretty different than what it looks like for legal or finance or things like that. So I think memory is one thing that really makes a difference. You see this make a difference, I think Instinct and Muse have done a great job at this, and so you see how it makes a difference there. Um, but I think it's still very, very early on, and I would expect all products in a few years to have some aspect of memory in it.

I think the fact that memory will have an impact is quite obvious, and that's why you see so many people interested in it and having been interested in it for the past few years. I still think it's non-obvious what the answer is. I still think it's incredibly non-obvious what the answer is. I don't think anyone knows what the answer for memory is.

We killed it because we didn't find product-market fit. I think, um, why didn't we find product-market fit? I think we actually...

tried to be too general. I think memory is so early right now that I think a lot of the solutions that you want to build are actually very like domain or agent specific.

Going back to what the hard part of memory is, I think the hard part of memory is that process for learning from facts about what to remember. And that agent that does that remembering, as I said, I'm massively bullish in kind of like vertical agents. And so, and I think having those agents be good at learning things specific to their domain is a key part of that. And so, I do think these kind of like learning agents, at least to start, probably have to be domain specific. And I don't know if there's like a general like what does the harness for these learning agents look like? It probably looks pretty similar to a lot of the harnesses we have now. It probably looks pretty... we build deep agents. Deep agents is an agent harness. Probably looks similar to that. An LLM running the loop, but it's got specific instructions and specific tools and maybe there's some specific sub-agents that are really good at, you know, navigating existing memory structures or distilling kind of like facts. And so I think you'll have agents for remembering things. I just think they'll be domain specific.

I think my take on the consumer space is that you'll have both. You'll have both horizontal and vertical agents. And the horizontal agents will interact with the vertical agents. Where does it make sense to have a vertical agent versus not?

I think for most apps it probably makes sense to have a vertical agent. And I think the reason that having vertical agents makes sense at all is probably twofold. Maybe more, but two that jump to mind. One, a lot of these apps have knowledge about you and can learn things about how you interact with that app better than a general horizontal agent could. So back to the point about domain specificity, how you interact with music for say like Spotify is very different than how you might interact with Strava, right? And so those apps are actually probably best at learning things about you. And I think the best way to expose those learnings is probably through an agent. The other thing is a lot of these apps have a ton of APIs. And so you could say, hey, you know, Spotify, don't build an agent, just expose a bunch of APIs. I actually think it's a better experience if there is an agent that kind of like wraps those APIs and exposes a single, or maybe multiple, but like probably a single like natural language interface for a horizontal agent to interact with that vertical agent.

When agents are interacting, it makes the number of requests go way up. I think you saw a story about how I think it was Resi that blocked a user's account because they were using Instinct to pull it, I don't know how frequently, but pretty frequently. And so I think, yeah, the infra, there's just way more strain on the infrastructure. How does that percolate down to lower layers? I don't know if I have an opinion on that yet. I think you have to build systems that can be much more bursty, that can scale way broader. I don't know if I have more specifics. Do you think providers will allow agent usage or not?

I think that's exactly right.

I think you could include some of those things in the agent response. So you could do advertising for specific things on Resi or Etsy or whatever and include that in the agent's response, like, hey, I found these three things, you might also want to consider this other thing. That's now a natural language response that goes back to kind of like the main agent. I think also to the point around the strain on the systems, if you have a vertical agent, it's going to be better and more efficient at interacting with the underlying systems than a third-party agent will. So I think it will make it a lot more efficient for them to do that.

Cost is one of the big things that we talk with people about right now. I think there's a few different dimensions to this. One, I think like the frontier models are relatively getting cheaper over time. Astra is like price performance fantastic compared to a lot of things that existed a year ago. Two, there's just way more demand. When you start talking about agents talking to agents or multi-agent systems, there are just so many more tokens that are getting burned and so the costs are going up. We see this in coding primarily. Coding costs have skyrocketed and I think a lot of organizations including ours are now thinking about ways to kind of like pare those costs back. How does this end up getting resolved? I mean if you look at how we're tackling coding agents, I think there's a few hints there. So when we're tackling coding agents, we think of it in three steps. One, we think of basically just like visibility, showing kind of like how much things will cost, how much things have cost to admins. Two is cost control. So you could say something like, hey, you know, go get me a reservation, but don't spend more than $5 like doing it. And then third is actually impacting the harness itself. So I think primarily a lot of harness engineering has been done around making the agent more efficient because that's the... or sorry, a lot of the harness engineering has been done to make the agent just like better, more accurate because that's the biggest blocker. I think as cost becomes more of a concern, we're seeing people care more about things like prompt caching which are really good for kind of like reducing costs. We're seeing model providers actually make this easier by, and this is getting a little bit in the weeds, but they introduced things like where you can change the tools that are available to an agent mid-flight. Previously, this would have busted the cache and caused like a massive increase in cost. Now, you can just add those dynamically and keep the cost the same. So, I think we're seeing more attention to both the model APIs and harnesses in terms of like controlling costs.

Another thing would be latency by the way. So we run developer surveys every year where we ask developers what they care about. For the last two years the number one response, basically double the next two, has been accuracy. So that's still the primary thing. The next two are latency and cost, and security is like a little bit after that, pretty close. When we talk to large enterprises recently, I'd say like cost has risen in terms of concern. I'd still probably say it's below accuracy, but it's probably higher than latency. And then security is, for large enterprises, security is higher, but still probably below accuracy and cost.

I think we spend probably roughly 30% of our engineering salary on coding tools. It's actually gone down the last 2 months. While tokens, the number of tokens that we've used has actually gone up. And so we've done this by doing three steps. First is basically just visibility. We tracked all of our coding agent usage to LangSmith. So we could tell who was spending how much, what they were spending it on. We could do analytics over that. Second is just basic cost controls. We set kind of like a budget and if you hit the budget, you can talk to our VP of engineering and he'll almost always like increase it, but it's just a safeguard against kind of like agents, you know, being used incorrectly or going in a bad loop. And then third would be we control the harness. So one of the things we've done is we actually built our own in-house cloud coding agent. So not an IDE, not a terminal-based agent. We let people use whatever they want for that, but we built our own cloud-based agent, OpenSuite is what we call it. It's actually the only way I code anymore because I'm in Slack all the time and so I just message it from Slack and it kicks off. But in that harness, we've done a bunch of work to control costs. So things like prompt caching, but also things like routing we've built into this harness and I actually think that's like a pretty repeatable three-step process for kind of like getting coding agent spend under control. Just first visibility, second like basic controls, third like try to improve the harness itself.