Because we went from my tab complete to instantly we just don't write code anymore. And it's like maybe we should have stopped somewhere in between. So this 100x thing is BS. But the only way you get more done is when you generate junk that you don't need.
This is David Kramer, co-founder of Sentry and one of the few people on a sea level who's actually using AI to ship production software at scale.
But the reality is if I ship something that has massive vulnerabilities in Sentry, that could cause the company to disappear. You want to flex that you can generate all of your code and have hundreds of things going in parallel. I will flex and show you how broken the code is 100% of the time.
If you hang out there, you see like, oh, software engineering as a craft is completely changing, yada yada yada.
There is nobody that is credible that says that LMs are not making it faster for me to build software. Despite what the internet would like to say, I'll sit here all day long on a single patch. This is David Kramer on how to ship real code with AI.
I built a scanner inside of Sentry that identified a ton of security vulnerabilities in our active codebase, which is a little bit borderline hobby project, but it's real world production software that we use every day now. Then I patched some of those security vulnerabilities. I think I have enough experience in the space, plus I'm pretty grounded about my experience. Plus, I do not care at all that I'm willing to share my opinion about what I see.
Okay, so one thing, I'm glad you bring up that security scanner because I read the blog post you published like February or something. I'll link it in the show notes. In one of the first introduction sentences, you said something like we literally have virtually infinite money for AI. Is that something you still stick to? Because I see a lot of companies like Uber recently introducing an AI limit and walking back on statements like that.
I would say it's not infinite but for reasonable use it's fine. It's very easy to waste money. The labs spend money, but it's hard to do something useful and spend $1,000 a week on a coding agent. As soon as it adds up, you realize you're doing things inefficiently. LLMs are not the solution to inefficiency. For example, for a web crawler, the right way is to use an LLM to generate a script when pages change, not to run an LLM on every page. That's much more efficient. Our security scanner cost $1,500 for the first run and found two high-severity vulnerabilities that more than covered the monthly spend.
Well, and that doesn't even include the fact that someone could have exploited that.
Brand damage and all those things. But I do think the money has to come from somewhere. People shouldn't be foolish with it and should have rational expectations. Right now, $1,000 a week is still quite a lot from a budget point of view. But we feel it's okay and expect the baseline to get cheaper over time.
It's funny that you say that in the week where Fable, which is ridiculously expensive, come out, but I get what you're saying. On that, how do you treat AI budget within Sentry on an IC level? It's difficult because some people burn through tokens and some use it moderately. We're nowhere near evaluating ROI. How's that from your perspective?
People ask this a lot. It's too early. We set a budget we thought was reasonable—roughly $15,000 a year per developer, which is about $1,500 a month. Blended across the team, some people are high throughput, some are not. I don't think $1,500 a month is enough right now, but we're okay with it as R&D. It's pretty obvious you have to learn how to use LLMs. We're 200 engineers, so it wasn't that expensive. From an ROI point of view, we just don't care. If I believe in it, that's it. Our CTO came up with the budget, and I said $3 million sounds fine. I see incremental value, maybe a low tens percent boost in output.
You're not expecting 100x developers. How come?
No. I was talking to somebody about this. I have a personality good at context switching, ADHD maybe. I'm impatient and fast. These days, it's annoying because LMs are not making it faster for me to build software despite what the internet says. I'll sit here all day on a single patch. To get one quality piece of code in complicated software, it's not any easier. In some cases, it's harder because I forced myself to stop writing code by hand almost a year ago. So the 100x thing is BS. The only way you get more done is when you generate junk you don't need, like brand new projects that won't add much value.
On that, what is your opinion on Peter Steinberger's way of developing? I have respect for him being on the forefront, but I think no company will spend that kind of money on development efforts.
I think Peter is a sharp guy and certainly insane. You are not generating that output for real world software with liabilities. It's impressive, but it's a chatbot where they've accepted all risk. They basically accepted junk code. There's a type of person that believes LLMs will get better and fix the junk. I don't think that's true. If I ship something with massive vulnerabilities at Sentry, the company could disappear. If you ship something that messes up a chatbot, nothing happens. So it's a science experiment, not worth a seven-figure compute bill.
I get that. At the same time, Anthropic seems to be operating on a similar scale where inference is free for them. So I'm wondering if this is a science experiment or a glimpse of the future. I lean toward science experiment.
It's a science experiment, no shade. I like a lot of people at Anthropic, but the software is broken all the time. You want to flex that you can generate all your code and have hundreds of things in parallel? I will flex and show you how broken the code is 100% of the time. That's not a flex. I want quality software. I stopped using cloud code more than a month ago. People don't see the community sentiment to slow down. Software verification is the hardest problem. We haven't made it better. We're spending inference to fix inference, which is expensive and doesn't actually fix it because it's nondeterministic. A little bit of this is the future, but not to the degree people believe. I think it's mostly a science experiment, living in a bubble.
To be fair, I'm still on the fence whether Anthropic's marketing is next level or pure luck. You mentioned LLMs are not a great solution for a lot of things. Where do you think they are a great solution?
Search is the greatest thing, still underleveraged. The most valuable thing I did this year was build a Slackbot that answers questions by searching information. It's a genuine improvement in behavior and efficiency. Code generation is one of the hardest problems, and somehow we decided it's the most important even if unsolvable. For verification, LLMs are not the right tool. But we get surprised where we can leverage them to augment other mechanisms.
I use them a lot for paralyzing things, jumping around, or sending an LM on a quest to find information. You wrote the Sentry MCP, right? How do you feel about MCP these days?
Yeah, it's great. I'm going to publish something later. There's still nothing that compares. There are flaws—the protocol is a pain, should have been stateless, should have been HTTP. We are where we are. But I care about user experience. MCP provides a plugin with native authentication, reduces friction, makes it secure. It's the first time I've seen a reverse integration where you build a plugin and it works for all future partners. We have many daily active users even with authentication issues. That tells you how useful it is.
Do you think the protocol was published too early as a standard? Some of the problems around OAuth if you have a concrete use case like enterprise software...
The version of OAuth they're using is DCR, which I had never heard of before MCP. It's not super secure in a broader sense. But I don't know if there's a solution. I'm glad it's OAuth because you can implement it natively. The OAuth is less of an issue than the transport. I do agree it was a standard right off the bat, and with money involved, all big companies want to spend time on standards. To MCP's credit, it's making progress. Most standards get stuck, like OpenTelemetry.
For a standard, it's still very fast moving. No one is even aiming to be 100% spec compliant except maybe VS Code.
Or the value of some protocol parts is low. Resources were never used, prompts are dead. Tools are great. Skills are coming but not there yet.
I didn't know that. The UI part is interesting, but I can't see a use case other than visualizing a diagram, which most clients can do anyway.
Here's my analogy for UI. I agree, it's interesting but not practical. Sentry has lots of data like traces that could be rendered visually, but it's not a big deal. Session replays are different if you could embed an interactive component, but you can just click a link. So there's a lot of neat stuff without market demand. We haven't shipped any MCP apps because it doesn't work in most places. I have mixed feelings. There's a bunch that probably won't survive in the spec.
I think that is a good thing, kind of weird for a spec. How do you feel about the talk that software engineering as a craft is completely changing?
When somebody says that, I click their bio and that gives me all the information I need. There is nobody credible that says that. The way we write code is a little different, but software engineering, system design, knowledge—it's not gotten any easier. LMs are not magically going to give you decisions. They can't even write CSS today. A lot of this is people doing marketing poorly. Twitter incentivizes rage bait. I know a lot of people who are tired of these conversations and checking out of Twitter because it's an echo chamber of people not in the industry talking about things that aren't true.
It drains a lot of energy. If you spend too much time in that echo chamber, you lose perspective.
Yes, it's unfortunate. I'll post something with genuine technology interest and get 20 slipshod projects that look like startups. People are out of touch. There's so much noise it's hard to find signal. I can stay grounded in my own opinions, but knowing what's good from others beyond my domain is hard. I rely on reputational value. The mental load is a lot.
I think it's also difficult because you get shitty noise from big companies like Facebook's token leaderboard, or the Jensen Wong quote that an engineer making $500,000 should spend $250,000 a year in tokens. I can easily spend that, but where is the value?
I believe in a moral duty. If you have power, you have responsibility. I have enough influence to feel a responsibility to not feed people BS. A lot of CEOs are out of touch. Some know what they're doing and do it anyway. Jensen Wong is generally grounded, but that quote was absurd. Dario from Anthropic is bad at public comms. He should stop posting. It's turning out okay because they have product market fit, but in any other situation they're burning goodwill. At the start, they targeted developers and said in 6 months developers won't write code. Who thought that was a good idea? I'm sure he means well, but he needs to consider his message.
One thing: I got into the industry 15 years ago, and the easiest way to learn was facing problems head-on and struggling. How do you think people will get into the industry now when they can outsource problems and not go through learning by pain?
I don't know. I think academics are important because they give theory. I'm a high school dropout, but I saw the value later. You still need systems design. There's an impatience amplified by these tools. Are you going to read the output that says this is a bad design and guide it correctly? Probably not. You'll let it do whatever and it will often do it wrong. I don't have an instinct on the solution. It seems complicated.
I don't think saying you shouldn't use LMs while learning is realistic. At the same time, accountability is important—your name is tied to the code.
Yes, there has to be accountability. I don't think we ever get to humans out of the loop because there has to be liability. At minimum, code review and good engineering practices are the same. The problem is everyone wants to go fast. If you practice those things, maybe you learn more through peer review. One thing that's insane is the assumption that writing code was the bottleneck. In every company I've worked, writing code was never the issue. I can write code all day if I didn't have meetings. The whole conversation doesn't make sense.
If you step back and look critically, things don't look like they've changed that much in how we produce software. But there are breaking things. I'm super high throughput, a 10x engineer persona. That's still true with LLMs, but one challenge is learning. I used to build a first version, iterate, and refine. That's hard with LLMs because they're slow. I also refactor as I go, and nobody's found a solution for that with agents. It's subjective. There's a version that refinement becomes about tools on top of agents, another says non-determinism will be a problem. We went from tab complete to instantly not writing code. Maybe we should have stopped somewhere in between.
That's where I'm leaning. LMs are good at generating the necessary characters that make up code, faster than typing, but you still need to read code and maintain systems. One thing: do you think vibe coding will replace no-code solutions?
At the very least, those solutions will change. Not everybody is an engineer. The delta between generating software and using a block system is very different. Will they be able to debug problems? I think generative interfaces are different. The market has expanded—more people doing engineering work who were not engineers before. They may stop at entry-level, but they're building little websites. We see market expansion. Our CFO built an application and it's reasonable. He's a finance guy. So I don't know where the delta is.
An engineer. It kind of makes sense that he'd be capable of doing this. He just didn't know the syntax and he doesn't know systems design to be fair either. But he's definitely more capable than a lot of people would be. And so I don't know what that translates to though over a long period of time.
For what it's worth, I also think vibe coding is very viable if you're just doing something for yourself. Just like you write a tool for yourself, you don't really care about the code quality or something and if it breaks, you just yeet it through a prompt one more time. I'm cool with that and I do that too for some things, but the minute you send customers or friends or family on it, they should maybe know a little bit what you're doing there. One thing you brought up, do you think the distinction between product work or product development, product management, design is somewhat growing closer together with the use of LLMs? Because I can see at least a designer or product manager creating a prototype, not production ready, but at least a prototype which is usually easier to discuss than a user spec. I don't see anything that suggests for example that we're collapsing roles like that an engineer is now also a PM, not in any version that is different than before if you will. Same with design. I do see people having more access to do things especially at a prototype stage. And some people hate this for what it's worth, like our design team hates when I slop together some generative artwork, but it does allow me to get more prototypes out the door and stuff like that. And so I think it is giving us a new set of tools, right? It's like we're still trying to solve the same problems. We've expanded tools. My analogy here, and I'm not an expert on this, so take it for what it is, is once upon a time my mom was an accountant growing up and she started in the era of Lotus Notes, which I know nothing about, but it was originally some accounting software or something. And I only know this story because she stopped being an accountant when she got pregnant and had me as a kid. And there was that gap where they went from Lotus Notes to I don't know if it was Excel, but whatever came after it, right? And it was a big technology change, right? It was a tool chain change, but accounting didn't really change. It was just like there was new software. And at least back then, there wasn't this thing where you could easily pick up the new software. It was like you actually had to learn it. This is just that to me. It's just another set of tools that are going to be used in certain ways, except right now we're pretending it's used in every single way. And it changes. We're sort of, there's a lot of 'we' is not necessarily representative, but a lot of people are jumping to the conclusion that a bunch of jobs no longer exist. They are now LLM. And I'm like I haven't seen a single job. There are real roles that are at risk. I think things like especially with big companies like customer service centers and things like this, those are going to change drastically. Engineering, I don't know. I still need just as many engineers. I need them to be just as capable. Designers, PMs, salespeople, all this stuff. I don't know. I don't see a lot of it changing in terms of people doing the work. Just the way they do the work is probably going to change; some of them will change more than others, I guess, if you will.
I am curious about these things when I think Anthropic has like a proof of concept for claw for lawyers where I'm like that sounds insane to me. Utterly insane.
I mean, we use co-work for legal work essentially. Like we get
