You don't have them actually write a prompt because frankly they're not good at writing prompts. That's part of the problem. You just have them click a button to run a prompt that someone else in their team made to generate a great unit test or something like that. Then they're like, 'Holy [expletive], this actually worked pretty well.' And then they can go see what kind of prompt they had to write. So often when we would find people saying, 'Hey, this stuff didn't work well,' we'd say, 'What prompt are you using?' And there they go boneless again. But once we found out what the prompt they were using, it was obvious.
You're listening to the AI Native Dev, brought to you by Tesla. Hi, welcome to the AI Native Dev podcast. I'm really excited today to be chatting with a fantastic guest, Quinn Slack from Sourcegraph. I'm Dion Almaer, I work at Tesla now, and excited to be chatting with you on the podcast. Quinn, welcome. Would you like to introduce yourself?
Yeah, thank you, Dion. I'm happy to be here and talk about code AI and agents and all of that. I'm Quinn, I'm the CEO and co-founder at Sourcegraph, and been a developer all my life. I've felt what it's like to work on massive projects and code. Got some patches to curl, you'll see one of the flags I added. Sent patches to OpenSSL and GnuTLS and all that. And that was these formative days as a coder where I got to feel what it was like to work at a massive codebase and try to understand the damn thing, and it's hard. That was before AI.
Yeah, that's awesome. So let's go back a bit in time just to get a bit of grounding. So Sourcegraph, like we know of Cody now on the AI side, but it started with code search, I think inspired by Google's internal Grok system and the like. Can you talk about the early years a little bit and talk about the code search side of things and the problems you were solving there and still do?
Yeah, seeing those pains in working these big codebases, that's why we started Sourcegraph. And now we've got amazing customers. We have every dev at Stripe and Uber and four of the six top US banks using us, and so on. But back when we formed Sourcegraph, it was because we had felt that pain. It's hard to code, and it's way more repetitive than it should be. We felt that pain ourselves. My co-founder felt that pain at Google, and together we felt that pain inside two big US banks where we were working inside and writing code. So we wanted to go and solve that. We wanted to accelerate human developers. And if you go back to our seed deck, step two is we want to go automate software development. And there is no technology available to really do that in the way that would be magical back when we started Sourcegraph. But we knew that if we could build code search, that would do two things: it would immediately make it so that human developers facing this massive problem—the biggest problem in software development, trying to understand the damn thing—would be able to do so better. And it would also prepare and groom and till the fields of the codebase so that when there was some magical brain that could ingest all the code and do amazing things with it, then all the code would be in one place, we'd have something that all the devs in a company were using. So it felt like a necessary step. And it just so turned out that code search was not just a good first step, it was also something really valuable by itself. And so that's what we built. The coolest moment that I remember in the first two weeks after I started Sourcegraph is we were building Sourcegraph and we had this way for you to see where everybody else is calling this function in the open source world. And I was writing some code and I said, 'Oh, I'm going to use Sourcegraph and see if someone's already done this thing.' And there was some other Go package that somebody had written that did exactly what was going to take me the next two weeks. So two weeks into building it, it had already saved me two weeks. It was like a perpetual motion machine. It was pretty amazing.
That's awesome. So I worked at Google and got to use the Grok system there, so I'm super excited because I think a lot of Google engineers when they would leave, they missed certain tools that they didn't have access to anymore. And so it was awesome to get access to some of those tools thanks to yourselves at Sourcegraph. So it was awesome to see. Then fast forward a little bit when the whole GenAI space blew up again with GPT. If you can think back to then, I'm curious what your initial reactions were. Were you skeptical at first? When did you know that potentially change? And then what happened to trigger you really investing and jumping into something big with building out Cody?
The summer before ChatGPT, we were working with Anthropic and we had access to Claude, which was amazing. It was in our Sourcegraph Slack and we had access to the API. And we found that, hey, this can do amazing things. This can actually explain code, not just autocomplete, which was the only code AI functionality that was available at the time. It could fix code, it could generate unit tests, it could generate whole files of code. And obviously it's gotten a lot better in the two and a half years since then. But that was amazing, and we knew that was the future. We knew that we were so well positioned given that we had all these big customers using us for code search, we had all their code, they were already trying to solve a lot of the same problems: 'Hey, how does this damn thing work?' And they use code search for it. And wouldn't it be amazing if we could also give them AI for it? So when ChatGPT came out, we said, 'Oh yeah, this is like Claude that we have, and this is really cool.' And I even remember I would show my parents and my family, 'Hey, look, it can create song lyrics.' And everyone around us thought, 'Oh, chat, that's nothing new.' But of course it was, and that was a huge moment in the industry. And I think looking back at that time, it's like a BC/AD moment. We obviously put a lot more investment into our AI since then, and we've come a long way. And still literally every week, every other week, our customers are doing something with it that is just surprising and new. And it's never stopped being so exciting. It feels like people are working at 300% ever since ChatGPT came out, and I absolutely love it.
That's awesome. That's so funny that you got to see it the other way around, being early on the Anthropic side, whereas I think most people, it was ChatGPT was their first kind of view into that moment. That's fun, especially now given how fantastic that Claude Sonnet is at writing code, all of the improvement since then. That's really fun that you had an extra early kind of seat at the table to watch this happen. So I'm really curious about the underlying technology and concepts that you had with code search and how that maps to kind of modern RAG, because I see time and time again the same old stick with retrieval where it's just easily chunking up a document, doing some cosine similarity, and there we go, we've done retrieval. But code isn't just text in a document or anything like that. So how do you see code being different than text for the use case of actually really doing a good job of retrieval? And how did code search help there?
Yeah, it's such an interesting topic. There's three groups of people here: there's the academic ivory tower people, there's the enterprise ivory tower people, and then there's the people who actually use the damn thing. And RAG was a term we were doing it, and I was meeting with some more academic people and they're like, 'What do you think of RAG?' I'm like, 'What are you talking about? Retrieval-augmented generation?' I'm like, 'Oh, you mean using search to find code and then present it?' And they're like, 'Yeah, that's RAG.' And I looked and there's all these academics who were doing stuff, and it's just so funny that there was such a divide there. And now it feels like any kind of RAG approach, most of the industry never actually did it that well. 99% of RAG implementations never did any chunking or never measured in a rigorous way, and that's absolutely crazy. We do a lot of rigorous measurement of it. And now it's already past the point where RAG is sufficient. Now most context fetching is done with some agentic—you can say like 'RAGentic'—to figure out, is the first fetch sufficient? What other terms might you need to explore? What other tools you need to use to pull in context? So I think it's already way beyond RAG. And the challenge is to be really rigorous about for what use cases is this kind of context fetching good. But really it's a black box of a lot of different techniques. The code search that we built before all the AI stuff came out, that was incredibly valuable. Just having a code index of all the repos was incredibly valuable. But we've had to add in a lot of new ways to do retrieval and indexing. And it turns out that virtually all of those things, they're also useful for humans just doing code search. We found ourselves, like we hoped it would work out, in a really good position where code search is a great foundation for any AI system that you build on top.
Yeah, that's awesome. So could you walk us through a little bit of the evolution of Cody from the beginning, from code completions, instructions, now hear a lot about chat-oriented programming? And what were the kind of insights along the way as you went through that journey?
Yeah, we started out with the chat. That was the first way that you could use Cody, and that was to fix code, to understand code. Then we built autocomplete, we built edits. And basically it was a new tool that was out there. We were the first to come out with the code chat that at the time really everyone was using autocomplete. And you've got a few kind of competing features in AI. You obviously have the autocomplete, I think everyone uses that. You have chat. With chat, there's a lot of different ways that can be exposed. And I'll talk about some of the things that we do differently that make sense for the enterprise versus for the individual. Then there's edits, where you select some code and you do Option+K and you say, 'Oh, go and do this.' That became really popular. Auto-edits is valuable. I'm coding a ton on a little side project over the last few weeks in the holidays, and I think about 50% of the code that I wrote was from our auto-edits, which is where it's like autocomplete but it can go and suggest edits elsewhere in the file too. You have all these features at a feature level. And if you're an individual dev that's just doing stuff that you can figure out, and maybe consistency and quality is not the most important thing, you're just trying to move fast. But we are totally focused on the enterprise. And I think the enterprise is the most interesting place to be building code AI because they have a massive codebase and they do things where you have a thousand times per day that they're generating a unit test, and they have the kind of scale that you can actually automate. And that is calling for a completely different kind of code AI product than what you'd make if you're just trying to make someone who's building an app from scratch or who's working on a kind of low-stakes personal project really fast. And I love the enterprise because it's where all the real software gets built. All the best devs work at the enterprise, all the best software is built in the enterprise. And if you disagree with that, I would just say, hey, if a dev is so damn good, they're going to build such good software that their company becomes so successful that in a few years it's going to be an enterprise. So the enterprise is where it all happens. And there's so much more potential to make code AI great in the enterprise than what you can do for the individual.
Yeah, that makes sense. It does. I don't know about you, it drives me a little nuts at times to see the kind of Twitter echo chamber of 'Look at this AI generating a little sample app from scratch,' which is a use case, it's good for prototyping and all of those kind of things. But to your point, that's just very different than the 99% case of you're already in an existing codebase, and in the enterprise it's probably massive. And maybe you're in a very large monorepo, maybe they've got lots of different repos. How do you make sure your best practices work between these different worlds? The onboarding experience, like one of the most fun things that I saw talking to a customer of ours in the past was just this new employee, and she just joined and she's got a little bit of impostor syndrome, right? Doesn't know the team yet, and is able to use chat like you're talking about with Cody to just get this understanding of the codebase. And the amount of trust she was able to get from the system, and it's obviously there 24/7, it's not 'Oh man, my TL is time zones away and they're not available right now.' It just utterly changed the onboarding experience, which again, these enterprises have a few developers in there, right? So being able to help with those kind of use cases, that's gold, right?
Yeah, like the quick demos aren't really often showcasing those type of features. There are other things that you've seen in the enterprise from customers that are just like, actually, this is where they're getting a lot of value, and it's not just the autocomplete of this bit of code.
Yeah, we like to look for areas where if you step back, you think first principles, 'Hey, it's crazy how everyone's doing this.' And the RAG situation was one where everyone was talking about RAG but basically nobody was actually chunking in a thoughtful way. No one was measuring how good their chunking is. No one was measuring the effectiveness or accuracy of their pipeline. And for a lot of the companies that were, they had a separate academic team that had a bunch of techniques but none of them were actually getting into production. So it's like measurement but it was totally pointless. And one big area we saw that was like the emperor has no clothes with code AI in the enterprise was the way that developers were using chat across an enterprise. They were all basically—this was a year ago—chicken-typing prompts ad hoc on their own each time. And so if two developers wanted to go make a test or wanted to go add a new UI screen, they would go and they would type, and this developer would do it differently from this developer, and there's no sharing of the prompts. And what a wonder that it actually worked. Our context fetching is pretty damn good, we'd find some files. But hey, obvious opportunity to make it so that if two developers are doing the same thing in an enterprise, use the same damn prompt. And one of our customers, Stripe, has an internal chat tool that's for general purpose chat, and they have talked publicly about this prompt library that they built internally so that Stripes could share prompts for writing good documentation or a good public summary or that Stripe marketing—they do such a good job with their marketing—and hey, why don't we make that for our AI? And we have a prompt library that means that if two developers are generating a unit test or making a new UI screen or any of these kinds of things, they're going to use the same damn prompt. And if a senior dev wants to make sure that everyone on my team who's doing this thing, they do it really well, they can go and make that prompt really good, and they can edit it if it needs to account for something else. That is such a basic fundamental thing, it's so obvious. The prompt is such an important part of the input you get to the AI, and it was crazy to me that still, even now that we've had this out there for more than six months and in massive use by our enterprise customers, that no other code tools have it. We have customers where they're driving more than 80% of their chat usage through prompt library, and that's amazing because the quality is better, the consistency is better, and think of all the time saved from developers not chicken-typing their little prompts on their own. So that's just one of many examples of where in the enterprise you can do that. Because by the way, you cannot do that if you're building an AI tool for individuals, because an individual is not going to put in 20 minutes up front to sit down, write a great prompt for doing something that they only do once a month. They're not going to do that. But in an enterprise, it's someone's job to do that. It's the tech lead or the manager's job to do that, and there's an incentive to put in time upfront because it's amortized over all the times that it's used. So I love the enterprise.
Yeah, that's awesome. You think about being like that someone who's a principal engineer of a large company and they're really struggling with how to spend their time, right? They want to help as many people on the team as they can, but they've got a bunch of code that they need to write as well. And so like, where do they put their time? And so being able to go in there and be like virtually on the shoulder of all of the engineers helping out, not in a creepy way, that impact is just massive, right? It's being able to like you say, spread that across is just absolutely huge. And so all of the value that gets put into the system, being able to spread that across the team, I think is a game changer. And yeah, the other thing that I'm super excited about within the enterprise too, and I'm curious if you've seen this at all, is I saw some companies that totally changed the structure of their engineering teams where they could go back to having smaller teams with these tools and with maybe utilizing another core team that's setting up some of those pieces in particular domains, and then they can move fast again because they're back to being a small team without all of the coordination cost. And so they can do so much more.
I totally think that point about how it changes how a senior developer, tech lead has influence over the team is really interesting. I hadn't thought of this, but it's almost like they're shifting left their work, where instead of waiting until code review, which is like post-micromanagement, which is the most annoying kind of micromanagement—it's not only are these a bunch of annoying comments, but I wish you had told me before I wrote the whole damn rest of the PR. Now they can actually influence how the code is being written. And one next step that we're taking is making it so that every line of code written will run through the standards that have been set for the codebase, so we get that feedback instantly. It's not just being used when you're generating new code, but it's also being used for all the code that you're writing as a human. I think that's going to be a much better kind of mode than waiting for a senior engineer to review something and you got to wait on that. Because I think in code review, every single code review comment should be a failure. It's not that code is bad, you should do it, but it's a failure. That failure is maybe a mean term, but think of it like a factory. Every time there's a defect in a product, in a widget made on the conveyor belt, it's not saying someone [expletive] up, it's saying that's a failure in our process. So every code review comment, it's a failure that the person writing that code did not know that they should have done it somehow different. It's not their fault, but the system has to think, how can we make it so they would not have made that mistake in the future? And that's how you could move so much faster. You can get these dumb repetitive facts out of the human brain and make it so they can think about the higher-level things. So we want to eliminate code review comments, not by saying let's throw out code review—obviously it has a purpose—but how do we make it so it's not as necessary? And that is a really exciting frontier that a lot of our customers are pushing on.
Oh, that's awesome. Yeah, another area that I saw, I'm curious if this resonates too, is just like there's all of these areas that are really painful, like accessibility is an example. I've never seen a company that invests enough in accessibility. It's always a really—maybe it's a really small core team of expertise, maybe it's just like a few people on the team that kind of know it, right? And I was talking to one company where there's one engineer that's just really passionate about accessibility and has a lot of expertise, and he was just like, 'Hey, in code review time, because it hasn't happened earlier, one in 10 reviews I'll be on.' And so basically that means one in 10 PRs get accessibility, right? Whereas being able to put that into the system and having that as part of the best practices for accessibility itself just eliminates that entire problem. And then you have performance, security, all of these like horizontal kind of crosscutting concerns that a lot of devs maybe don't have the full expertise. You can't expect everyone to have that level of it, right? You now can have this ability where you can have the devs run, to your point, really fast, but you've got the kind of guardrails in place by giving the experts in the company the ability to have that influence earlier on.
Exactly. And that's such a shift from someone who's really passionate about accessibility thinking that we need to educate every single human dev on how to do that well, versus how can they make it so that the system has high accessibility. I think there's a bit of a moral valence here. Yes, it's good for every developer to know and care about accessibility on a personal level. My mom went blind in the last five years, and so I've gotten to see how she uses her devices, and it's incredible. Apple does amazing things with accessibility, and I've seen also like cookie banners make websites so hard to navigate. This is personal for me. And I've been using Sourcegraph AI when I'm coding to understand—like for me it was a lot of basic stuff, how to do ARIA and how to do roles on elements, and I didn't know that. I've got some prompts and some techniques that help me. But the moral valence here, I think, is important. We have to change the idea of what it means to be a software developer. It's actually a virtue to not need to know all the low-level stuff. And we've accepted this with so many other evolutions in the role, where you're not writing assembly code, you don't know a lot of the fundamentals. Hell, most programmers don't know how to do some basic CS fundamentals, and look, we've gotten there as an industry. So you got to eliminate the moral valence, and it's okay if developers don't know all about security or performance. The reality is they don't know that today, but you think that it should be their job? Instead, let's figure out how do we make it so the system has great accessibility, great security, great performance, and how can AI sitting on the shoulder of developers actually help there? And again, that's only possible in the enterprise because those things—security, performance, accessibility—mean something different based on the codebase, the stack, the product you're working on, the business requirements. And you're only going to take the time upfront to figure out how can we make a system that enforces those things well in an enterprise.
Yeah, that makes sense. Have you seen anything around helping with dependency management, migrations? Every enterprise I've been in, they're always working on projects to migrate from here to here, which seems like a lot of a waste of time from the business value side of things, but it's really important to stay up to date. And so I think that could be an area too where AI could just massively help with these massive large codebases. A lot of it is legacy stuff. How do you change that whole system? And just like you're saying with code review getting earlier, what if the world is constantly updating and staying up to date, and so you don't get into the situation of needing a project Phoenix to rebuild everything from scratch? Do you see that at all?
