It's the Harness, Not the Model: David Cramer, CPO of Sentry, on Agents, Expectations vs Reality

Usually the people that are arguing about, say MCP versus CLI, fundamentally don't understand all of the complexity that goes into any of these systems.

It's the first time in history you can build the integration in a way that's great for you and you don't have to go through the painful, painful hoops of like going to build another Jira extension, and I'm going to love every minute of it. So, I just built this thing, published it to GitHub, and I ran it a few times on our codebase. And I'm like, 'Look, I found a bunch of bugs. Like, a bunch of legitimate vulnerabilities.' The 60 vulnerabilities we dug up—these are all verified. 60 vulnerabilities in one section of our codebase.

The amount of money would have been like 30 to 50k for bug bounties if we paid that out. And it was like $1,000 in compute.

Yeah, thanks for having me. David Cramer, started Sentry a long, long time ago. I'm a software engineer. I have the CPO title because it was available, and so I gave it to myself. But I write code actually probably half of the hours in a good week. And so I'm still very in the weeds. I build a little bit here and there for Sentry, but like Warden I built and the MCP server I built. But I unfortunately do not get to spend a lot of time on our core product these days. Yeah.

I think the disconnect is the internet. If you look at people actually at the edge, the frontier, whatever you want to call it this week, everybody looks the same. You have a coding agent in front of you, you're synchronously prompting it to do stuff and maybe you have two or three more running. And it's outputting kind of junk. You're constantly trying to refine it, get it to be good. And that's it. That's the state of the art, despite whatever is popular on the internet. I think a lot of what you see is people—some of it's grift for sure. But there's a lot of imagination and interest and curiosity. Whenever there's anything that's future-leaning, people just jump to the future even if it doesn't exist. And so I think a lot of this is people imagining what could be even if it's technically not possible with the technology. And so I think that's one of the interesting things is like what are LLMs capable of at peak? They might only be capable of what we see today. And then a lot of these imaginary things just don't exist. And I think you see this in Railway, and I think there's kind of an opportunity here, but the challenge is people are trying to fast-forward. And some people will take that risk. I know of the incident but I didn't read their total postmortem so I don't know. What I do know is an agent will kind of do what you tell it and sometimes it'll be somebody else told it to do the thing, but it generally is not going to just make up things. It is just pattern matching at the end of the day from the model. I think you have to give people safety and controls, and security is like—nobody cares about it right now, but it is the most worrying thing I think in this domain and there's going to be a lot more problems coming. And I think you should be cautious depending on the industry you're in and like what you expose the agents or expose your customers to because at the end of the day it's like, well, whose fault in that scenario? Whose fault is it?

Definitely the customer's fault. But it's like you can say it's a customer's fault, which I actually truly believe it is in the scenario, but if I'm a customer and that happens to me even if it's my fault and you didn't protect me from myself, I'm going to blame you. And that's also fair, you should. And so we do stuff like this. We have a Slack bot that I built for work and I spent an inordinate amount of time on how do I not expose anything that shouldn't be exposed, whether it's in shared Slack channels. Thus, it needs to only be able to query data that either everybody in the channel can already see because it's public information in the company, or that the user has specifically asked to see that they have access to, but that should be a rare anomaly. It also needs to not have access to any keys, and if it does do anything on behalf of the user, that needs to be sort of almost revocable so it can't then do more things that it wasn't asked to do. And so you end up with all these patterns that frankly you can't really work past, and I just think people should slow down. And I don't think this is in the way where it's like Anthropic's safety, safety, safety. I think their version is a little bit more religious than it is practical. But I do think people need to slow down. And you kind of got to come back to the reality that software is still hard. Valuable things are hard to accomplish. It's not just you can generate some new software and solve every problem that's in your head tomorrow. It takes a long time to build things still. And so I just think it's people being a little bit too casual with things that I don't think they should be casual about. And they're probably well-meaning, but they don't fit my pattern or my constraints around how to operate. And so, you know, we're very careful about this at Sentry and like we obviously have a lot of data with customers, but you can't really do anything destructive with anything we expose. Which is somewhat intentional. People have actually asked—we have an MCP server, it's like, oh, it'd be nice to be able to delete projects from this. I'm like, no. It's fine. You don't need to be able to do that. You can go to the UI, we don't care. It's such an outlier case. And I think that's the problem—people just have to recognize things versus just pretending the problems will solve themselves. And I think we'll see a lot more of this. And especially with less technical users adopting things and even the technical folks, just people less experienced and potentially not gaining experience as fast as they might once have.

And I think this is Open Claw too. It's imagination. It's what could be, but practically what can't exist right now yet today in a safe way. And so, it's fine. I've got a little robot at home or whatever and it has access to some things and the risk to me is low. It's not fine in an enterprise environment. And again, in my world. And I think memory is probably one of the most interesting domains in the field right now. And I think it is probably one of the most difficult. And you can probably do a good job of implementing it for a single isolated session kind of thing. But if you try to do shared memory, good luck. It's so unbelievably hard to secure that. I spent like a few weeks trying to build one for a personal home chatbot with my wife and it was a disaster of constantly exposing the wrong information to the wrong people. It's just very hard. All these problems are very, very hard. Again, it goes back to—I think hard problems are hard. They haven't really gotten any easier and that's also usually what makes them valuable if you can solve them at all, but you can't just pretend the problems go away or you fast-forward past them or whatever. And so yeah, I think we're still early on this journey for a lot of these things.

Yeah, I think the value—Sentry is a company that's been around a long time. We're an infrastructure company which is actually a good domain to be in. We're much safer than a lot of companies from disruption, but it doesn't mean we're just by default safe. And we think a lot about—the data remains valuable no matter what. It's not like all of a sudden bugs go away or any of this, right? So we're not worried about that, but we think a lot about what value can you now unlock that you couldn't before and how will you consume any of the value. And so Sierra is a little bit of both. One, how you consume it is easier. It's like, well, I have coding agents, I don't need the UI as much and that was part MCP. It's not even Sierra in this regard. MCP, CLI, whatever your flavor is, it's just bring the data to the right location. And for us it was bring bugs to coding harnesses because that's where you fix bugs. So that was already one huge unlock. And it's not a be-all end-all, but quickly what you then find is there's like two problems you hit. One is it goes off the rails. It has the information, it kind of does the thing, but it does it like 50% right—like any coding harness does for any problem right now basically. And then the other problem is it does completely the wrong thing and as a human you can kind of fix that, but it's worse because now you're wasting time. You're wasting a lot of expensive compute at the very least. And so what we did is we look at it—we have a lot of data. That data is hard to use and it's all kind of interconnected by various attributes and stuff. And one common example I always talk about is a lot of people these days they'll use something like their Next.js technology or something. And you've got code that's running on the server and the client that's kind of the same application—it's in the same codebase and it's a little bit harder to reason about. And that's just connected by tracing it across the wire. As a human, you can build things that make this easier to consume, but it's still confusing. You're like, what's the actual bug? What's going on? As a machine it's very easy. It can process lots of information very fast. But historically it's been hard to generalize that kind of stuff. And I think one of the cool things about LLMs is arbitrary information you can throw at it and it can actually do a really good job of pulling information out of it. Pulling signal out. I have a company I invested in that got acquired by one of the big frontier labs and it was like the most absurd paperwork you can imagine. So I immediately upload it to another frontier lab and I'm like, tell me what happened here. I have no idea what my recourse is or what happened to me in this. And it does it in like seconds. And I'm like, this is cool. And it was, I'm pretty sure, accurate. But so with us we have a lot of this data. We can make it more consumable now, which is one big thing, which is: can we truly identify a root cause of problems? The value of that is not super high in abstract unfortunately. Once in a while that is extraordinarily high value and saves you lots and lots of human hours. Most of the time you don't need that much depth. So okay, so that's fine. We saw that. We recognized that. The other thing we recognized as part of that is we can actually make things more consumable, which is actually really important with the market expansion we're seeing happening, where there's just new people trying to build software. Whether they should be or they will be successful is kind of irrelevant, but if you want to capture that, you have to think about how do you make it even friendlier or even easier without dummying it down, because then you risk your core audience. And so what we realized is you can give them all this information that was already pretty curated before. We did a good job of this. Now we can make it even easier to reason about some of these cryptic errors and stuff. And so that's really valuable. But what we think is the opportunity for us—and this is reverse engineering who we are and what we think is plausibly doable in a time period—is you take the long tail of trivial problems in your codebase, like typos, small little errors. They're not that important, but they all take time. You can throw them in a bunch of coding agents today, and you'll probably get an okay outcome. But if we could automate a world where again, even if it's the most trivial of problems that just don't matter that much, but you would still rather they not exist, and we could build you a queue of things that are actually pretty well verified that you would just mash merge, merge, merge, merge, merge—that to me is very high value, because everything is a productivity game. How do I take a thing that either I wanted to do and couldn't do, or that was taking me longer than it needs to, and how do I just do better at that? And so we just repeat that game over and over. And so that's kind of our big bet called self-healing, which is now I guess a trendy topic, which again is not possible really today for a lot of things, let alone full self-healing. But we think we can get to at least what I said, a queue of trivial bugs. We actually have an internal thing right now where in the next, I don't know, 60 to 90 days, we need to ship this where a measurable percentage of customers could experience that reality. Again, not for hard things, for easy things. Maybe one day we'll get to hard things, but there's only so much that's possible anyways today. And that's really what we see for our value add there. But I actually think that's R&D greenfield. I think a lot of the stuff for businesses is: how do you do something that was hard before, do it better now, more efficiently now, or just do more of it, versus let's pretend we're going to reinvent everything as a chatbot or something like that. Nobody really wants a chatbot for all things.

It's great. It's the first time in history you can build the integration in a way that's great for you and you don't have to go through the painful, painful hoops of like I'm going to build another Jira extension and I'm going to love every minute of it and the customer is going to have all these gripes and then I got to go build now a GitHub integration and everything else under the sun which was always a low ROI thing. Now you build this. It's a little bit complicated how you build it but yeah, it was something like you can give your customer something that genuinely is useful and you don't have to integrate it everywhere. It's so good from that regard.

People—there's always hot conversation about these things. People—the usual people that are arguing about say MCP versus CLI fundamentally don't understand all of the complexity that goes into any of these systems. Like our Slack bot for example has MCP built into it. It also has the ability to call CLIs or anything else. How do you secure the CLI? Do you intercept all shell commands? You build a custom shell because you need to intercept network egress and all this stuff. MCP, it's all just baked into a protocol. You've got authentication baked in. It's HTTP. Say what you will—the protocol is a little complicated. It doesn't need to be that complicated, but you take that away, there is at least a tried and true mechanism to make that function because it's just how networks and the internet works with a little bit of excessive complexity on top. Whereas CLIs and all these other systems, they're just so much more free form that you can't rein it in. And again, I just think people misunderstand why people value a lot of this technology because they just don't have the experience. They haven't lived in some of these worlds, or they're just proclaiming stuff that they read on the internet, one of the two. But we get a lot of value out of both. We co-develop both a CLI and an MCP at this point because we're like there's a lot of situational stuff where one makes sense, the other doesn't, and there's different things you can accomplish in each of them.

Yeah. So, Warden is—I would describe it as a code review harness. It's actually just right now a shim on top of Claude SDK, and then it takes a skill as a container and turns that effectively into a coding agent, but not really, but it just tries to make Claude be like a code reviewer, basically. And then it tries to coerce the output into something useful. And I forget why I started this project, but we have—technically it competes with a paid product we have, which is a little contentious internally, but I think what it was is we continue to find security vulnerabilities—it's just more active now. And I'm like, LLMs can definitely do a good job here, but you need it to be shaped in a certain way. And I'm like, we're not going to build this as a product, it makes no sense. So, I just built this thing, published it to GitHub, and I ran it a few times on our codebase and I'm like, look, I found a bunch of bugs. Like legitimate vulnerabilities. The first time it actually was not—there were no high severity ones. There was nothing very, very critical, but they were good like, yeah, we should fix this. But I focused on security mostly to play nice internally. It does anything. It doesn't actually matter. It does anything that looks like code review of any shape or form. But I did that. And then I was experimenting and I'm like, okay, there's got to be a way to get more depth out of this because—my thesis was you can do a form of reinforcement learning, which is take a bunch of prior art, synthesize that into a skill, which is just a container for us and a bunch of references, and it'll create more accuracy because effectively LLMs don't know anything unless you tell them the thing. They can't give you the right answer if you don't give them the right answer, which is why they need search and all these other tools. And so I'm like, well, if I give it more of the right answers, it'll be more likely to find the right problems. And it does. 100% does. But it was kind of tedious to do that. And I was experimenting. You have limitations to the complexity. The more complicated the ask you give it, the less likely it's going to fulfill it. And you see this in any of the tools. And so I'm like, well, what if we just give it lots of different asks that equally have their own depth, like in an umbrella of this kind of thing. And security was my goal. So I'm like, we'll have a—

Whole thing that just tries to find auth bypass, and then we'll have a whole thing that tries to find like remote injection or something like this. It turns out that it did not work at all. Or rather, it performed better than what we were doing, but it was not necessary, I guess is what I'd say.

And what we found was like, I did the same thing. And this was sort of our second iteration of tackling security things with Warden. I did another one where I just thinned it out and built like a more rich context in a skill with a bunch of references that were sort of like, well, here's the JavaScript stuff, or here's the Python stuff. Domain specific, right? Or maybe here's like what authentication bypasses look like in this framework. And I did that, and the accuracy improved even more.

And it turns out I was also accidentally using Sonnet, which was kind of interesting, which is drastically cheaper than Opus, mind you. And we found a ton more vulnerabilities. And that was like the last post I made where it was like, I found two critical auth bypasses in Sentry, for example, that just had gone undiscovered for 5 plus years.

They were not like the place where everybody would go to test when you have pen testers and stuff, because we do this non-stop. And so, all these things slipped through. And I'm like, but how do I get people to be interested? Because we, unfortunately, are a well-known company, but we are never cool. And it's hard to build hype around something. And for me, I'm like, this is very cool what you can accomplish with this. And I think people are sleeping on it, and it's a really important problem.

And so, I'm like, I'm going to go find other active vulnerabilities in open source projects. And I'm also someone who does not want to spend a lot of time doing something. I'm like, how do I get this done and move on? And so, I'm like, I'll spend one day doing this. And so, I went to Home Assistant, and I went to Discourse, like the old forum software. And I did my sort of routine on them, ran it in both. I found like I think it was like 50 findings in Home Assistant. They were not super critical because of what Home Assistant is, but they went and fixed a huge chunk of them really quickly. On Discourse, I don't know if they've shared it publicly yet, but they were definitely issuing some CVEs for the findings. And that was literally me spending like 30 minutes on the problem for them.

And a little bit of money, but not that much. And I'm like, oh, this will get hyped because now people will see it's not just a Sentry thing. And then still mostly nobody cared. And so I think we're still early in this industry where people just have not gotten past the dopamine hit of being able to generate some new app. But there's so many cool things you can do that the way you can approach them is different or more scalable. And you might say, well, tokens are really expensive.

The 60 vulnerabilities we found — these are all verified — 60 vulnerabilities in one section of our code base. The amount of money would have been like 30 to 50k for bug bounties if we paid that out. And it was like $1,000 in compute. And so, one, it's not cheap, but it's pretty cheap in aggregate. And so it just takes these skills, synthesizes it into this thing, and you can run it locally or you can run it on GitHub on like pull requests. And it's kind of like a linter is the way I think about it. That's my analogy.

But it looks like Code Rabbit and Bugbot and Sentry's thing. And it's actually more reliable than a lot of those for the same amount of compute spent. And you control the whole thing.

Mhm. Yeah, and this was kind of the thing. When I told the team I built this, they're like, wait, isn't this competing with Sentry? I'm like, look, if I can build this in two days, anybody can. Then it's like not the easiest thing in the world. And to be fair, it was two days to get it running and then many weeks to polish it.

But the reality is like this is just traditional dev tooling that will be public open source. Ours is licensed under fair source, but it's just stuff that will exist no matter what. And so I'm like, it's fine, we'll put it out there. People will or won't use it, it doesn't matter. And it provides a lot of value to us at the very least. So it's been a good win.

I agree. And you can argue a lot of this the necessity is because we're using agents to generate more code and definitely more stuff slips through. But this isn't just new problems, it's old problems too and it makes it more efficient. And the verification part of code including security is the challenge. That's the only hurdle genuinely speaking. Like, the coding agents maybe they'll get better, they're not great, they're fine. But the verification has been largely unsolved. And so yeah, we have some code review stuff, but it's still non-deterministic and how much compute does it take? And so there's all these problems in there and I do think it's really really important. But every company kind of needs it. And so it was like a no-brainer to me.

And again, I think there's lots of things like this that are really cool unlocks like the Slack bot is just another one of those unlocks. And it's like, yeah, you give it access to data, people can ask the Slack bot questions. They probably weren't going to search the docs in the first place, but they might ping the Slack bot instead. And so it's just another one of those efficiency plays where it's like, yeah, I could spend a lot more time on a security review and pen testers and all this stuff and there's certainly a price to that, but it's all like, well, how do you do this more and ideally more cheaply.