Whoa, we got Scott Wu in the Temple of Technology. Welcome to the show, Scott. There he is. What's up, man?
Good to see you guys. How are you?
We're great. How are you?
Good, good, good. Busy couple days for us, but...
Oh, yeah. What's keeping you busy?
Oh, there's always — I mean, there's always so much stuff going on. I feel like there's new products that come out every week. There's new models that come out every week. And then, you know, we obviously have a lot going on with Devin.
Well, we appreciate you taking the time. Could you give us kind of just an overview real quick? Scott and I talked on the phone. I think it was maybe 5:30 or 6 Friday night. And he was like, 'Yeah, everybody will be here until probably...' You know, what did you say? What's your normal? You guys have a pretty intense schedule.
We're usually still around past midnight. Yeah. So, busy schedules for us.
Well, hopefully it's fun. You clearly haven't put yourself out of a job yet. But can you give us the update on kind of like where is Cognition now? Obviously, like massive launch, huge like going direct moment, great launch video, and then you know you kind of went into build mode and we kind of haven't heard that much from the company. So can you give me an overview of where the products are, where the company is, and kind of set some ground rules for where you are?
Yeah, absolutely. So we had our initial announcement about a year ago actually at this point, and from then a lot of it was basically just going through all the frictions of software engineering. You know, the truth is engineering is just so messy in the real world if you think about what has to get done. Obviously, there's some pretty hard logical problem solving type work that has to be done, but there's also a lot of practical stuff. And so, you know, figuring out how to have a clean experience that slots in, figuring out how to work with all these enterprise customer code bases, figuring out the capabilities and working with logging and documentation and tooling and things like that. And so, you know, for a decent chunk of time, our main work actually was with larger enterprises. And so we would work with a lot of these different groups, you know, bigger tech companies, financial services, or things like that. And work with them on a lot of these bigger projects and more repetitive projects and have Devin basically come in and speed up the job for a lot of this stuff. And then in December, we rolled out our general availability. And you know, it's a self-serve plan where you can just put down a credit card and get started and plug in your codebase and have Devin. And we were super, super excited about that obviously because it's really the chance for everyone to just get to try it out and see how it is. And it's been a fun last few months for us. It's been a lot of iteration and a lot of growth and so on. We've had users talking about how Devin's now quickly become the number one committer in their codebase and so on. And so it's very cool to see.
Can you talk to me a little bit about the decisions you made early on on where to kind of put your AI efforts? Like as I understand it, you didn't train a foundation model. You didn't do one of the massive runs. You work with different model providers, but what considerations were made in the early stage and are you still sticking with those or have those evolved over time?
Yeah, absolutely. So we had, you know, I would say two major bets that we really started the company with. And I would say the first one was, you know, that reasoning and this whole RL paradigm was going to work. And this is, you know, this was a pretty controversial thing I think around the end of 2023 because most of what, you know, the first generation of AI was, or the first generation of language models was more like imitation learning, right? It was basically like read all of the internet, you know, read every Reddit post and talk like somebody on the internet, right? And obviously, I mean, it was pretty incredible. You know, we passed the Turing test. ChatGPT was a massive, you know, and for a while all of the work that was being done and all of the products that happened in particular verticals were much more in that style which was basically, you know, I'll call it text completion basically, right? And so, you know, you had products like that in marketing, right? You had a product like that for customer support. And similar things for code where it's kind of like here's the code so far and just predict for me what line comes next, right? I think we had a strong view that, yeah, that this reasoning was really going to work and was going to enable a lot of new use cases. And then the other thing that we felt really strongly about was that the product experience was going to shift from this kind of more Q&A text completion style product to basically an agent, you know. And practically, I think what that means is I honestly think a lot of folks today still think of AI as kind of, you know, a better Google, call it like you have a question and you want to go search it up, you know, you ask your AI and your AI has a better answer which is big obviously, I mean Google's not a small company, right? But I think the actual vision that we're going towards with AI is actually even bigger than that which is basically, you know, your buddy that does your chores for you, right? And the obvious difference between those two is the ability to actually interact with the real world and do real tasks and so on, right? And so, you know, there's a big difference between having a legal Q&A for example versus having a lawyer that can actually go submit things for you, that can go and, you know, interact with all these different surfaces. And the same is obviously the case in code, right? Where it's, you know, this is how we build software, right? It's you write code, you run the code, you see if it worked or not, you run the tests, you look at the documentation, you know, you click around on the product yourself. And this is how we iterate and build, right? And so the ability to do that is a big step function.
How do you think of getting Devin to go from sort of this reactive experience where you talk to Devin like you would, you know, a teammate or employee and say like, hey, can you help me sort of execute on this specific thing, it goes and does it, you know, maybe comes back to get, you know, feedback at different points, to I imagine long-term you want the sort of proactive, sort of like high agency agent? And yeah, how, you know, I'm sure you guys are already experimenting with that kind of thing internally, but, you know, what are your plans along sort of that route?
Yeah, for sure. And so, you know, the way that we kind of think about it is turning all the bricklayers into architects is almost, you know, is how I would describe it. And, you know, I think the really beautiful part of software engineering, I would say, is getting to do this core, you know, problem solving and decision-making around what do you want to build, right? It's like, you know, let me understand the problem, let me understand exactly what's going on, and then let me figure out here's exactly the solution that I want to build for this, right? The thing is you only spend about 10% of your time doing that, you know, as a software engineer in practice you probably spend about 90% of your time, you know, dealing with your Kubernetes and fixing your unit tests and upgrading your thing to the new version and all of this other stuff, you know, fixing these bugs that come up, right? And so a lot of it is the way I would kind of describe it is working with Devin as the implementer that allows you to be a really great architect, right? And so, there's a lot of detail with that because naturally you have to be in, you know, all the same flows that humans use for software engineering if you want to do that, right? And so, you have your logs on Datadog. So, like, you know, Devin's got to go look at those logs, right? You're talking about issues on Slack. So, Devin's got to be in Slack, right? And a lot of it is really kind of figuring out the workflow where it's very natural and very easy that as you're talking about this idea that you have or this new feature that you're trying to build or this bug that you're trying to fix or whatever, you're able to just tag Devin and just say, 'Hey, @Devin, can you take a look at this and make this fix or do this thing?'
Talk about, you know, specifically I can imagine that Devin, because it's so embedded in these different workflows, you know, there's every single day we see, you know, conversation on the timeline from somebody that says, I'm switching from, you know, this code editor to that code editor or that code editor back to this code editor. How do you think about, you know, sort of long-term moats and sort of lock-in as, you know, you're building Devin for yourself but then you're also, you know, trying to build it, you know, you're already a billion dollar company eventually, you know, I'm sure you have much, much bigger ambitions, but how do you think about sort of long-term moats and, you know, around the sort of like software engineering agents?
Sure, yeah. You know, I think in the short and medium term, experiments. There's a lot of different experiences in code and, you know, I think a lot of folks building kind of different product experiences for different verticals or use cases within code. I think in the short and medium term, it's, you know, things are just changing so quickly that the natural thing is just whatever is able to help folks the most is what they're really excited to use, right? And I think there's just, you know, when the pace of development and also the pace of progress in AI is as high as it is, I think that's going to continue to be the case for a while. You know, I think with that said, in the long term, I think these things naturally do kind of converge to a point. And there are a few particular pieces I would say that really kind of cause that. And one of the big ones that I'll just point out is, you know, there really is a lot of, in most spaces, you know, we would call this personalization or something of that sort, but there's a lot of it in code, right? I mean, if you imagine you hire, you know, the smartest software engineer in the world, and you know, you get them to work on your codebase day one, they're not going to know a lot of these little details about here's why we chose to do this architecture, here's what this function does and that function does, here's how you do this database migration or whatever it is, right? So, you know, they have to learn all these things from the ground up as opposed to somebody who's, you know, just as smart but has been at the company for, you know, 10 years and built half the code themselves and, you know, they wrote all these files themselves and so on, right? Then, you know, you get to a point where you really just understand the codebase. You're making the same decisions and trade-offs and you're working with it very tightly. It's the kind of thing where, and we see this already, where it's, you know, as folks start using Devin more and more, one of the really cool things is, you know, you can have one of your engineers ask Devin a question and Devin is drawing on the knowledge that it has from all of the previous sessions it has with everyone on your team, right? And it's able to learn all these things and, you know, someone says, hey, you know, we have to go do this version upgrade and Devin might say, oh yeah, I remember this. I actually just did one just like this like last week for this other part of your guys' codebase. And so let's figure out this one and let's do this, right? And so I think there's a lot of, that's kind of the shape I think of what we're going to see a lot in AI actually over the next couple years where, you know, this is a bit of a hot take, but I would say a lot of the problem solving is actually good enough already. You know, a lot of the reasoning and the problem solving, I mean, AI has been shown to be capable of solving some pretty hard stuff. I think actually what we have left to do is a lot more so just understanding the detail and the complexity of the real world, right? And you think about all these tasks that we spend our time on, you know, there's some amount of problem solving but there's also a lot of just pulling in the relevant context and, you know, understanding what is the common sense, what are the common sense considerations you're working with, and that's a lot of what we spend our time working on with Devin's capabilities.
Can you talk a little bit about, I have this take that we're almost in an AI winter for consumers even though AI feels like it's moving so quickly. The ChatGPT moment was so big because we passed the Turing test that for most average people it's just Google search as you mentioned. But over the last year obviously if you're following this stuff there's been incredible breakthroughs. Inference cost has gone way down, models have gotten bigger, context windows have gotten way bigger. Have there been particular milestones in just AI research and development that have been more transformative to your business than others over the last year?
Sure, sure. Yeah. You know, I think at a high level, you know, I think all the continued progress that we're seeing everywhere in AI actually is primarily due to essentially like reinforcement learning, RL, that's really working. And it is a very different paradigm, right, where we were saying how in the past it was all just imitation learning and it's get as much of the internet as you could possibly scrape up and just train on all of it, you know, and hopefully you get something that knows a lot of stuff, right? Whereas this is kind of, you know, this whole reinforcement learning paradigm is, you know, if you have a clear problem space where you can do the problem 100 times and you know this was right, this was wrong, this is right, this was wrong, you know, it's a little bit more like AlphaGo, you know, this self-play learning but in a language model, right? And I think that's the biggest paradigm and, you know, folks like OpenAI, Anthropic and lots of other, you know, Google X and so on have shown a lot of real progress through this whole RL paradigm. To your point on consumer, you know, I actually think that we'll probably see a resurgence actually in consumer over the next few months or so. And I think one of the big things I think that will actually flip that switch is like a really great consumer agent experience. You know, it's, we talked about this. We haven't had the moment of just chatting with this sort of generalized agent that can just book you a flight and a hotel and it's like super seamless and like that's like what I feel like we've all been waiting for. We've had that preview since Siri 1.0 back a decade ago and it still isn't here and we're still waiting for it, but it feels like now it's actually just a couple months away.
