Inside Devin: The AI engineer that's set to write 50% of its company’s code this year | Scott Wu

Our whole team is only like 15 engineers. You know, we use a ton of Devon when we're building Devon. Most folks on the team are definitely working with up to five Devons at once. And so Devon merges like several hundred pull requests into production in the Devon code bases every month.

It's in the neighborhood of a quarter or so.

Honestly, we expect it to be a decent bit more than half.

AI is going to be the biggest technology shift of our lives. Most of the big tech revolutions that we've had over the last 50 years, like the personal computer and the internet and the mobile phone, they all had this big hardware component that was a big part of the distribution. Folks who were building for those industries kind of saw their market grow and grow and grow basically steadily year over year as the number of people with mobile phones increased, right? As the number of people connected to the internet increased. One of the things which is already, I'd say, different in AI is just how explosive the technology can be. There's no weight on hardware distribution. It means that the space is just growing so exponentially.

I think there's going to be way more programmers and way more engineers a few years from now, pretty quickly. The form factor of what it means to be a programmer obviously is going to change. But at the end of the day, of course, the discipline is all about just being able to tell your computer what to do. And so in that lens, I really think that programming is only going to become more and more important as AI gets more powerful.

Thanks so much for having me. Excited to be on.

Absolutely. And so Devon is a fully autonomous software engineer that is going to work on tasks end to end. And so there are a lot of great tools for all parts of the stack of the AI code workflow. What Devon does is it is a full asynchronous workflow. And so you can tag Devon on an issue in Slack. You know, you're talking about an issue and you tag Devon. You can tag Devon in Linear. You can have Devon, and Devon will make pull requests in your GitHub. And so it's very much built to work with engineering teams as your junior engineer.

Yeah. Yeah.

Yeah. Yeah. And it's crazy to think about, by the way, because, you know, a year ago when we did the initial launch, I mean, people didn't really believe that an agent was possible, right? And it was a very, I mean, it was a very different time, you know, it's like start of 2024, you know, things with model capabilities were definitely quite a bit earlier on. Reasoning especially was quite a bit earlier on. And yeah, I mean, in the time since then, it's obviously developed a lot. I think in terms of practical skills, you know, there's some comparisons we make. Sometimes we kind of say, well, when we got started it was kind of like a high school CS student, and then as time went on it became more of like a college intern, and now it's like a junior engineer. But I would say though that those are more like rough guidelines because I really like the phrase jagged intelligence, for example. Because there are obviously certain things that it is much better at than a human. There are certain things that it's much worse at than a human. And I think over the last year we've learned a lot, especially about not just coding agents but agents in general, just like really building out how all of us should be working and interacting with agents as part of our flow. And so a lot of the things that we built, I mean, there was no Slack, there was no GitHub integration, there was no Linear, there was no interactive planning phase working back and forth, there was no way to touch up Devon's code. And so a lot of the features that we built on the product side since then have really been about basically, yeah, figuring out how to make working with Devon and handing off tasks to Devon as smooth of an experience as possible.

I think it's a 50-50 of both. You know, I think the capabilities obviously, you know, have improved a ton and we've seen these get better and get measurably better, but I think the other side of it is everything to do with, yeah, really the product interface and the tools and so on. And I think, you know, I think today folks generally know how to use chatbots and to work with chatbots, right? And that's an interface that people are familiar with. And obviously with agents, it's, you know, it's still like a real curve, I think, to learn how to use them and how to get the most out of them. And so it's really exciting to see a lot of others starting to build and do a lot more in the agent space as well. But I think this is the kind of thing that we're all really figuring out together as a space.

So we work with companies of all stages and sizes. You know, on the smallest end it goes to, you know, startups of just one or two people who are using Devon to build out a lot of their kind of like initial prototype or initial product, all the way up to, you know, big public companies, Fortune 100 companies or public banks or things like that who are using Devon like across their engineering teams. In general, it's, you know, we've seen a huge range of the use cases there, and obviously the kinds of engineering work that you're doing at a one or two person startup is very different from the kind of work that you're doing at a public bank. But throughout it's all been basically, yeah, being that junior buddy of yours that makes you go faster and really multiplies you. I would say, you know, I think it can multiply you as an engineer obviously by just like letting you work with your own team of Devons instead of having to be kind of like fully synchronous on a single task. And then it's also kind of like multiplying your team and multiplying your team's knowledge base because Devon really accumulates a lot of the knowledge from working with every member of your team and is able to bring that into each new session.

The founding team, I mean, most of us have known each other for years and years and years actually. And for almost everyone, this is our first time working together, but we've known each other a long time. And we all actually had our own kind of journeys in AI for the last decade or so. And so for myself, you know, I ran a company before this called Lunch Club, which was an AI for a professional networking product. And I ran that for about five years. And you know, my co-founders, one of my co-founders, Steven, was one of the first engineers at a company called Scale AI, which has obviously grown a lot and done very well. My other co-founder, Walden, was an early engineer at a company called Cursor, which has also obviously grown a lot and done really well. And our whole team kind of like was kind of like that. You know, many of us knew each other from competitive programming and math competitions, but we had stayed very closely in touch, you know, in the decades since then. And we've all kind of all had our own journeys. And so, you know, we had one person who was running teams at Neuro, we had one person who was at Wemo, someone who had their own YC tools startup for machine learning. And we were really excited to build something together. And this was around like late 2023. So about a year and a half ago at this point. And yeah, when we got started, I mean, I think there were a couple things that we felt really strongly about, and one was that reinforcement learning was really working and was going to be the next big paradigm shift in capabilities. You know, back then it was, you know, the initial ChatGPT launch in 2022, and those models to first order were, you know, what we would call imitation learning in AI, right? Which is basically, you know, you have the model read all the text that you can find on the internet and then train it to talk like somebody on the internet would talk, right? And there are obviously a lot more details on top of that, but that's kind of the first order pass of what was really done. And it was amazing, right? I mean, it passed the Turing test. It was able to respond and to have encyclopedic knowledge about a lot of things. And I think this new paradigm which we've gotten into over this last year or year and a half is really high compute RL, which is a very different paradigm, right? Which is basically the ability to go and do work on task and put something together and then be evaluated on whether that was correct or incorrect and use that knowledge to decide what to do and to learn from that. Right? And so, you know, we felt very strongly that that was going to happen. I think for us, code was the natural thing to work on for a couple reasons. One, because, you know, we're all programmer nerds ourselves, and so teaching AI to code is about as cool as it gets for us. But also because, you know, code has this whole automated feedback loop, right? Where you can run the code and that is the kind of automated feedback that really feeds into the RL, which makes these models so great at coding. And then the other thing that we felt very strongly about was that the product experience was going to shift from, you know, what I'll call like text completion to agents, basically, right? And to first order, I would kind of say, you know, there have been a lot of great experiences in text completion. You know, it's been used for marketing, it's been used for customer support, it's been used for education, and in code obviously as well. As you know, the GitHub Copilot was kind of really the dominant product of that initial wave, right? But I think the big shift that we really felt we would see is moving from kind of this text-to-text model to an actual autonomous system that can make decisions, that can interact with the real world, that can take in feedback, that can iterate and take multiple steps to solve problems. And, you know, now we call that agents, but that was what we were really excited about at the time.

Yeah, and there's even, I mean, OpenAI, ChatGPT, Anthropic, and Claude, and Google.

So, you know, when we got started, it wasn't even really a company. I mean, it's more like a project or a hackathon almost. You know, we booked an Airbnb basically for a couple weeks. It was around Thanksgiving time, and just got a bunch of people together who were just excited to hack on some projects and build something cool. And it's funny actually, the first thing that we were building for actually was more like solving these contest programming problems and using an agentic loop to really do better on that. And so obviously if you run your code on test cases you can evaluate, you know, there's a lot of agentic work that you can do there to try and do better. And that we spent some time on that initially. And then, you know, we've kind of gone from... I mean, the story of the whole company for us in some sense has been going from hacker house to hacker house. You know, so after that, you know, we had another hacker house, and that's where kind of some of the initial ideas for Devon came, and really building like a software engineering agent and not just like a coding agent and having it interact with a lot of these tools. But even then there were so many iterations. And, you know, even like the idea of talking to Devon, for example, was like, you know, it was something that we had to come up with, right? Initially it was just like you hand off a task and then it works and then it shows you like this whole finished code, right? And now obviously it's like you can jump in at any time, you can get feedback on the plan, you guys can scope out the task together, you know, when you're working with Devon. And a lot of these things we had to develop obviously, and certainly we've learned a lot about the use cases, the form factor. We've made a lot of big improvements and step function improvements on the capabilities and Devon's ability to use tools and debug and make decisions. And so it's, yeah, it's been a fun journey. I mean, I think like I would say the grounding question for us really is that, which is one that we think about all the time, is really just like what is the future of software engineering, you know, and how should we be working with AI to write code? Because I think at the end of the day, of course, that's what underlines all of the product decisions that we make.

Yeah, so we started in November of 2023, which was in, yeah, just like hackathon mode. We officially made it into a company around the start of 2024, and then our initial launch was in March. And so it was like non-stop. I mean, it's been non-stop for the entire last, you know, 17 months. But, you know, it's getting to the launch and then obviously working with enterprises and developing the product a lot more, building it and getting it to work for a lot of practical use cases, and then getting it, making it fully available self-serve in December of last year. And now we've rolled out 2.0 obviously just a few weeks ago. And so it's been a very busy time for us.

I would say it's a decision we're pretty proud of. I mean, I think there's a lot of different product experiences out there, and I think the thing that really makes Devon unique in what it does is that you can really hand off. And more and more what we've seen honestly is that I think a lot of kind of explaining the Devon experience to folks is really just explaining it as, yeah, this is your junior buddy, you know. And that goes for a lot of the parts of the flow. Where in the onboarding, for example, you know, initially I would say like we've definitely had a lot of users come in and just kind of see the blank screen and not really know, or they'd ask, hey, like I'm going to do this whole big rearchitecture of the whole codebase. And basically, you know, what we've learned over time is to basically get folks to think more like, well, whoa, whoa, you know, let's work on getting the repository set up first. Like, let's make sure we hand Devon a couple one-pointer tasks so it can get familiar with the codebase. You know, let's get it the thing if Devon needs to be able to test the code or run the linter or CI or things like that. Obviously, we want to make sure Devon's got its own virtual machine set up to be able to do that. And similarly, like I think the usage pattern, you know, I think often it wasn't clear. And obviously you can sit and just kind of watch Devon do it action by action and work that way. But we found that the best workflow really as a team building a lot of stuff was to work with multiple Devons and to run them asynchronously and to kick them off and to only jump in basically as you needed to provide feedback or steer the plan or anything like that. And so in many ways I think Devon as a name really is our attempt to kind of capture the soul of that as a product, where it really is, you know, treating it like a bit more of an autonomous entity that you can hand off tasks, that you can work with, that you should be teaching and learning with over time.

You know, by the way, we're all software engineers ourselves. You know, it's like I'm a programmer by training and still a programmer at heart certainly. And, you know, I think the way that we have always thought about it is there's layers of abstraction and there's tools. And one way I would say it at a high level is kind of, you know, I think of AI in general as, yeah, I mean, computers are obviously getting more and more intelligent and are able to do more and more. And, you know, it's possible there may come a day where computers truly do everything that we do and humans are not responsible for any of it. You know, I don't expect that to come particularly soon. But I guess what I would say is, you know, until that point, for as long as we're still part of the equation, one of the most important things to do obviously is, you know, for us as humans is to instruct our computers on what we want and what we want to build and what we want to do, right? And software engineering is, you know, we think of it today obviously as Python and C++ and JavaScript and all these things, but at the end of the day, of course, the discipline is all about just being able to tell your computer what to do. And so in that lens, I really think that programming is, if anything, only going to become more and more important as AI gets more powerful. And I think the thing that's really exciting for us is, yeah, it's like really seeing that kind of iterative transformation. And so you asked what things look like today. And I would say, yeah, it really is like having like a junior buddy or really a team of junior buddies that you can work with, right? And so every engineer on our team, for, you know, we use a ton of Devon when we're building Devon. And so Devon merges like several hundred pull requests into production in the Devon code bases every month, you know, which is, I mean, our whole team is only like 15 engineers. And so it's a pretty sizable fraction of all the code that we write. And the way that we use it is basically, yeah, everyone's got their whole team of Devons. You know, if you're going to be like looking through various issues, if you're going through feature requests, if you're going through bugs, if you're going through new paradigms that you want to build, then it is naturally the case that there's a lot of handoff points where you just say, 'Hey Devon, here's what's going on. Can you please take a pass at this?' Right? And sometimes Devon will be able to do the task 100% autonomously and just makes the PR and then you merge the PR and that's great. Sometimes you want to be able to jump in for the 10 or 20% that really needs your help. Maybe there's a few details with how exactly you want to scope it or how you're architecting this feature, or maybe you want to go and test the front end at the end yourself to make sure it looks exactly the way that you want and give your one or two lines of feedback after that. Right? But a lot of it is really, yeah, is kind of like, yeah, learning to work with Devon to be able to just do more in parallel and build more.

Yeah, I'd have to look, but it's in the neighborhood of a quarter or so of all of our...

Oh, it's, yeah, it's grown a ton. I mean, we've seen it grow exponentially internally ourselves as well. And so it's, you know, it's kind of an interesting one where again it's always both the capabilities and the product interface. And so, you know, I think the intelligence has increased a lot, but the other thing of course is, you know, we've spent a lot of time in figuring out how to build and to really kind of build for an interface where you can get Devon's value on tasks where Devon is able to do the 80 or 90%. So Devon is obviously not, you know, is obviously not perfect and it'll make mistakes and so on. And a lot of the question is basically, yeah, how do you scope out your initial task with Devon and then just kind of set Devon off and have it go and do the things that you want to do? You know, how do you come in at the end and review and give feedback? How do you make sure Devon learns over time? How are you able to kind of just check in as needed and course correct if you want to?

I think by the end of this year, we expect it to be more than half. And I mean, as time goes on, you know, one of the things that we've seen is just it's you're able to do more and more and more work asynchronously, right? And you're able to hand off more and more. You know, I think the soul of programming, the soul of software engineering has really been about through all the eras, you know, not just now but, you know, even when it was assembly, right? And even when it was Pascal, and even when it was punch cards or whatever, you know, I think the soul of it has really been basically just about defining the problem that you're facing and really thinking through exactly what is the solution that you want to build. You know, thinking through the architecture, thinking through the details, and really kind of mapping out in your mind exactly what you want to build basically and what you want to have your computer do. And I think that's, you know, that's what makes software engineering really great. And I think that's like the funnest part of software engineering. I think at the same time that's probably, you know, in the neighborhood of 10% of the average software engineer's time, right? Because 90% of the time is, you know, you've got this Kubernetes error that you've got to debug and you have to see what went wrong and the system crashed or, you know, you left some port open and this is, you know, messing up or, you know, there's a bug report that you have to take care of or you've got to migrate your code or you got to upgrade to a new version or things like that. You know, a lot more kind of like implementation. And one of the ways that we've kind of thought about Devon in building Devon is really allowing engineers to go from bricklayer to architect, so to speak. And a lot of it is, yeah, just like getting to the point where you can do the high-level directing and you can basically specify things exactly how you want. You know, I think it is very much about still having the human in control and having the human able to do the full specification, but just multiplying the magnitude of what you can do and what you can build, you know, in one day or one hour or however long.

Yeah, for sure. I love this question. First of all, the question of, you know, whether you should still learn to code, my answer would be absolutely yes. I think to a large extent, you know, computer science, when you take computer science classes and when you learn these fundamentals, sure, you're learning a little bit about, you know, how a particular language's syntax works or something like that, but honestly, most of what you're learning really is about the ability to logically break down problems for number one. And two, I would say, is just, yeah, the model of a computer and a lot of these decisions and a lot of the abstractions that we've built over time, right? Like what is a database and how should you think about a database? You know, what is a garbage collection system and how do those work and all of these different pieces. And the reason I think that's important is because it's the same with a lot of these other, you know, arguably we've already kind of gone through these phases in programming. And I think this next one is going to be, you know, somewhat faster and somewhat bigger, but in many ways a similar flavor, which is, you know, when you work with Python today, obviously a lot of things are already abstracted away from you. And in some sense, you know, someone from 50 years ago might already call Python, you know, you just get to explain in English what you want and now the computer does it for you, right? And that's great and I think it's really powerful. It's opened it up. I mean, we have far more programmers obviously than we ever have before because of that. But I would say certainly, you know, as you're building your skills as an engineer, it really helps a lot to understand the abstractions and to be able to peel the layers beneath, right? And so, you know, folks will use assembly, for example, if they're really performance optimizing a piece of code, but also, you know, it's in order to build good systems and to understand these things, you certainly want to understand these abstractions of, you know, how does networking work, what is TCP/IP like exactly, or what happens with this Python code when it gets interpreted, or all of these details. And I think similarly, I think we will get to a state where, yeah, you know, with no experience at all, you're going to be able to build some pretty cool stuff and to do some pretty amazing work just by explaining what it is that you want. But I think that, you know, for quite some time, you really want to be able to think precisely about the details, to peel back the abstractions, to be very, very, very precise about what it is that you want to build and how.

For sure. And I think architect, I mean, it's, you know, we already have a term for architect in engineering and I think it is directly the directionally the right term. And I think a lot of it is really, you know, it's, I think one thing to kind of just do a routine implementation and write boilerplate code and things like that. And I would say that, you know, in many ways AI coding has already made us much faster at that, right? But I think a lot of the core questions of understanding very complex systems and working in the context of the whole company and thinking about, you know, the product that you're building or the work that you're doing, understanding, okay, what are the problems that we want to solve, how do we want to solve those problems, what is exactly the solution that we want to build, what are all these key decisions and trade-offs that we're going to be making. And basically I think folks who are able to do that really, really well are just going to be able to leverage themselves more and more. And so if anything, I think there's going to be, I think there's going to be way more product, way more programmers and way more engineers, you know, a few years from now than there are today. And I think, I think pretty quickly the form factor of what it means to be a programmer obviously is going to change and in some sense it already has. But I think there's just going to be so much more for us to build. You know, I think one of the great things folks talk about Jevons paradox all the time. I mean, software is truly the kind of the shining example of Jevons paradox where we have always managed as a society to find more and more things that we want to build software for and build more code for. And I really think there's a lot more out there to do.

Absolutely. Yeah. So Jevons paradox just says that as the price of something goes down, it can still be the case that the total spend on it actually goes up. And so, you know, you can think about this with money, you can think about this with time or resources, but the direct version here is I think as it becomes easier and easier to program and as programming becomes more and more effective, I think we're going to have a lot more programmers. You know, it's, I think in a kind of zero-sum view you might say, well, we're going to be 10 times faster at software engineering and so it means that we're going to need 10 times fewer software engineers, right? But I think in practice, what really is going to happen is actually we're going to build even more than 10 times as much code. And because, you know, all of the work that we do is so capped, obviously on our ability to actually build and execute and iterate, we're going to have so many great ideas out there. We're going to have so many great products out there.

Yeah. So it's very asynchronous, and so obviously you can kick them up and start them up and shut them down basically as you see fit. But most folks on the team are definitely often working with up to five Devins at once, I would say. And it's a nice flow where you think through, all right, what are the five things that we want to get done today? One, two, three, four, five. You have Devin one do number one. You have Devin two do number two, Devin three. And the thing about it is, for what it's worth, I think it's taken us some time to really kind of adjust to it and get to the point where it's really intuitive for us. But I think it's definitely a different experience where you're handing off most things asynchronously, and the goal for each of your tasks is to be there for the parts that really need your expertise, where either you really need to define exactly what it is that you're solving for and what you're building, or maybe some of the more complex parts where you want to steer Devin towards particularly what kinds of changes you want to make. You know, I want the class to be set up this way, and we should go and change all the downstream references to this as well, or whatever. But basically having Devin do the bulk of the work asynchronously with you.

Yeah. So our engineering team today is about 15 people.

Yeah. And for what it's worth, we've already seen this shift ourselves, where in terms of the team, obviously, folks don't spend that much of their time just writing out boilerplate or just kind of doing pure implementation of features, and people get to spend much more of their time focused on really just thinking about the core questions of how do we make Devin better? What is the right interface for Devin? What is the right flow or the right set of features that's really going to make this as great of an experience as possible. And that's how we like things, obviously.

Honestly, I think as a community, all of us as engineers around the world are going to have to think about this and build for this and kind of adapt to these new technologies. But what I would say is, yeah, I think more and more, and especially as capabilities get better, but certainly even in steady state today, I think things are going to shift towards this kind of asynchronous flow. And one of the reasons I would say for that is in the real world you're just capped by real world constraints, right? And I think one way to put it is kind of like, and don't take these numbers exactly, but it's kind of like the first order math of it is, of course, being able to write files or to complete this function or complete this line or things like that. You know, it helps a ton. It's a really great experience, right? There's a lot of parts of building software that obviously are almost not that at all, right? It's, you know, if you have a bug that you're trying to fix, and so you spin up the local server, you click around on your own products on the front end and try to reproduce the bug yourself. You know, once you have the error, you take a look at Datadog and you see what happened and you try to find other errors in the logs. You know, you look at those files and you see what went wrong. You make some edits, maybe you go and rerun the whole process again now that you, you know, to make sure your change looks right, right? And that's a lot of, you know, what it means to be a software engineer, right? And, you know, these are processes that take real time. I think we're going to shift more and more towards this agentic workflow because, you know, that's in some ways it's kind of like the way to really get to the, you know, 200%, 500%, 1,000% gains that we'll be getting to with software engineering over the next few years.

And so the whole process obviously of working with Devin is working asynchronously. And so I thought it'd be cool for us to actually just watch Devin a little bit in action, and then we can go through some other examples of work that Devin's done or things that Devin does for us even on our team, for example. But then we can check back in asynchronously with our Devin after. And so I'll share this real quick. And the key thing I would just emphasize here is a lot of it obviously is really just about thinking about as a software engineer or as engineers ourselves or engineering teams, PMs and so on, what are the things that we would want to build that we would want to hand off. And so we have Devin set up with our own Devin codebase, for example, and so I'll go ahead and kick off a Devin for that. And so I'll just say, hey, at Devin, I'm on with my friend Lenny. Hi, Devin. Can you modify Devin web app to, let's feature your newsletter as part of the Devin website. Let's do it like on the real Devin website.

And so we're going to kick this off. As you can see, Devin gets started instantly and goes ahead and responds. And again, you can work with this asynchronously. You can work with it synchronously as well. For this, we'll just kind of go in a little bit and see exactly what's going on. But as you can see here, Devin's going through files and taking a look through a lot of stuff. And so we can follow here basically as we need to and see what makes sense. You can see Devin's already called out a few particular pieces, right, where there's the sidebar which we have implemented on the front end, and there's pieces there, and we're going to have a new component, and that component is going to link to Lenny's website. That all sounds good. Devin's asking us any questions if there's anything that we have here. Same story here where it's kind of you can let Devin make its own decisions and hand off, or you can go ahead and give some more thoughts, right? Should the button open in a new tab or within the application? I'll say let's open it in a new tab.