Patrick Collison on Claude Code at Stripe

Well, reliability and security are really important for us. Stripe operates with five and a half nines of reliability, so extreme reliability for our core APIs. But at the same time, we want to be developing and launching new products and adding new features extremely quickly. We want to have continuous deployment of our API. And across financial services, the way things typically work is maybe things get deployed once a month, once a quarter, certain places once a half. And I mean, that provides a kind of local stability, but of course at two enormous costs when you don't get quick feedback from customers, you can't ship things on a regular basis. But then also it does mean that these moments of migration are incredibly fraught and scary because you've this accumulated detritus of months of progress. And so we thought that was totally untenable. In fact, when we're developing things, we want feedback from our customers multiple times a day. And so we really weren't willing to compromise on this. And so in order to achieve five and a half nines of reliability, but with this continuous process of feature evolution development, we thought we really needed to invest in the end-to-end development and quality assurance process. And so the devboxes are just the first part of that, and we can have some instrumentation there and observability and operational characteristics and so forth. And then a whole process of incremental deployment where you first roll it out to, you know, a handful of machines and then 1% of machines and some progressive rollout strategy from there. But I guess the general reason for our conviction, we needed extreme reliability and security, extreme velocity, and I think it's the only way to achieve both.

That's very nice to hear. We obviously were working extremely hard to sustain that. It's been interesting for us in the era of agentic development where obviously there's a question of, well, will AI be a tailwind or a headwind here? And there are obviously a lot of concerns of, well, if people are writing code so much faster and maybe they're not reviewing each line of the code with quite as much scrutiny, lots of speculation as to which way this will cut. I was speaking with one engineer at Stripe this morning, and over the course of H1, he had more than 600 pull requests merged. Every single one was written with AI, and exactly one of those pull requests had to be reverted, which at least suggests that it is possible to have this enormously accelerated development rate and with still empirically quite high reliability. Minions of this thing on top of it, which are a way to orchestrate VMs with prompts from Slack or from a web interface or, in principle, from any tool, you can just ask for some feature to be implemented or some task to be completed. It will create a fully new VM for that and then go and perform all the tasks along the way and then it'll package it up and submit it and build it and run it through our test suite and the whole end-to-end process. We have seen that quality per pull request over the last 18 months has gone up. Now, there's still some challenge because the incidents per unit time has gone up slightly. And so now they're mostly very minor incidents and all of our secondary mechanisms for catching things before they become problematic have meant that our total reliability is essentially unchanged, which we take as quite heartening. And then I think the cool thing is obviously with LLMs and AI and everything else, there's now of course the possibility to build new observability and new instrumentation and new harnesses and new kinds of automated scrutiny that we couldn't build before. And so I feel pretty confident that over the next year or two that AI on net will make Stripe quite a bit more reliable.

It does mostly come back to this idea of relying on invariance and hard barriers rather than things that are somewhat more subjective or discretionary or probabilistic. And so obviously it's nice if the model is aligned and is more likely to write correct code or secure code or whatever. I mean, nothing's ever perfect. You really do want to rely on guarantees.

Yeah. I mean, this is full agreement with all of that. And actually we started seriously investing a lot of this, the apparatus and the tagging and the sort of semantically aware guardrails around our data back in 2017. And it's been a multi-year journey because there's a lot of annotating to do and there's a lot of granular permissioning to do and so forth. And we did all of this because we thought the security guarantees were so important, but you're right that it kind of accidentally put us in a better position when agentic development came along.

Indeed, indeed. Yes. Well, let's go back to the point where I really, I mean, there's kind of a general question in the world about whether AI and LLMs will be on net, offense advantaging or defense advantaging. But my guess is that in equilibrium in a couple of years, this is all going to be substantially defense advantaging.

Every devbox has Claude Code pre-installed. It's the first place people go when just, the median person is going to complete some task. It really has delivered meaningful acceleration. One case that I thought was quite informative here was, we had the idea at the beginning of the year of building a thing called Stripe Projects. And this was kind of inspired by Claude Code because when you're building some project with Claude Code, for anything of any materiality or substance, you invariably need to couple that to some other set of services, right? I want PostHog for logging or I wanna host it on Vercel or I want a database, whatever. Because almost all these companies are Stripe customers already, we thought we could work with them to expose their capabilities in a new way and make it incredibly easy for an agent to instantiate an account with them. We had this idea at the beginning of the year. We decided to go and build it. It was between two and three engineers for about two months, but from first idea to public launch. And that's not only building the internal APIs and services and harnesses and all the things, but also integrating with now about 50 different services and those services all have their own idiosyncrasies and occasional bugs. And I don't think you could have done that with two to three people in eight weeks in the before times.

So I asked one of the engineers involved and his estimate was a bigger team, didn't say exactly much bigger, but a bigger team and six months. Let's just say twice as big a team, I don't know. And I guess that would've been 3x longer. So that's a 6x relative change. Again, maybe that's an underestimate, who knows, in that. Software engineers are famously optimistic when they estimate software projects. So maybe we can say at least a 6x speed up, which I mean Stripe has thousands of software engineers and who knows if every project is being accelerated by that magnitude. I suspect there's a distribution. Some things are massively faster, some things aren't. But again, even if we're really pessimistic if it's only, let's just say it's only a 2x improvement across the entirety of what we do, I mean that's obviously still a preposterously large deal.

Again, with higher quality, at least per pull request.

I think we definitely create more products and try more experiments and we can see this in the numbers. We just had our annual conference Sessions and the number of new products and the number of new features. And again, the quantum of a feature is somewhat subjectively and qualitatively defined, but we try to be reasonably consistent year to year. The number of new products and features at Sessions this year was wildly ahead of any prior year. The engineering organization was somewhat larger, but clearly there's some kind of productivity effect happening there. So we're certainly building more for customers. But the other thing that I find kind of interesting and that cuts against some of the slop concerns is we're undertaking more projects now to improve our architecture and improve our code base. And the line from a friend that we often mention at Stripe is that every code base is now the prompt for another code base. And of course, Jarred Sumner, former Stripe, has demonstrated this with the Bun rewrite. My estimate of the... There could be a question of, with AI and with LLMs, should we expect the quality of code at Stripe to be higher or lower in three years? And I think the answer has to be higher.