Let's go back to November 2023. Sonnet 3.5 was out for a few months. We had GPT-4 Codex just starting to finally work. We also had the event of November 17th, the OpenAI and kind of Sam Altman. And it was like it was just a very ripe time. I think we found an email that you had sent to a kind of alum group about, I don't know if it'll come up or not, about the early tenant vote. There it goes. I think this is like the earliest kernel of the thoughts of Cognition. Maybe talk about where your head was at the time and how you were thinking about it.
Cool. Awesome. Well, first of all, thank you for having me. Super excited to be here. It's actually kind of—I didn't realize I said this at 11:59 exactly. So when we had started, we were a bunch of friends who had all known each other for a long time. Many of us have known each other for years and years and years, and we were exploring different ideas in AI, as everyone was. And we were really excited about code, partially because we were coding nerds ourselves, partially because, you know, we just felt that there were a lot of clear breakthroughs to happen. End of 2023 is right around when RL was really starting to work in the labs, where a lot of things started to become possible. And we had this view basically that there was a real window where full multi-step processes, iterative processes were going to really start to work. And obviously now we call those agents, but back then there wasn't really such a name for it and a category for it. And we were talking about this. I remember this. We were in New York, actually. We had lunch and we were talking about, yeah, you know, there's probably an opportunity to start something here. I think there's going to be a lot of interesting products to build and new services and everything. And we had lunch, and then as soon as we got out of that lunch, it was that Friday, November 17. We heard the news that Sam Altman got fired from OpenAI. And so, naturally, we said, 'Okay, well, if there was ever a time to go and start a company, it's probably right now.' And I remember spending a ton of time on this email and saying, 'Okay, we have to get the pieces together and build the critical mass and make this happen.' It's kind of funny reading the email because obviously, I mean, Sam got unfired like two days later, but like, you know, I think by then we were already down the path and we were already doing it. We had the initial team, and I think with this initial group, like everyone was like, 'Hey, let's go build something, you know, let's go do something together.' And so we got a hacker house together. It was in Burlingame, California. We were there for like a week and a half. We were iterating on a ton of ideas. At first it was very theoretical, researchy stuff, like improving problem solving in code. And then we did another hacker house and another hacker house, and eventually it became Devin.
Awesome. And not only was it 11:59, it was on a Friday night as well. So that, you know, says something about how you spend your Fridays. But let's move a bit to Cognition. So I think on the website it's called an applied AI lab, and obviously research and engineering are both very important and critical to what you're doing. Maybe talk a bit about how you think about organizational structure, how you view research and engineering. Maybe talk about, if you're open to it, some of the research areas you're interested in. How do you think of research areas you focus on versus what the labs were focused on? And like, where do you bet on model progress versus what you do in-house?
Yeah, sure. So we think of ourselves as an applied AI lab, specifically distinguished from a foundation lab. And the way that I would kind of describe it at a high level is, I think both companies do research, both companies build products, right? I think there's a question of which one is in service of the other, basically. And I think from a foundation lab, obviously the goal is to just kind of like advance basic research and to kind of, you know, build the state of knowledge, and from there, you know, the products that come out from that are the ones that then get built. For us it's the reverse, which is as an applied AI lab, we are most focused on figuring out how to build something that people will, you know, will use and to deliver a product of value to our customers. And all of the research that we do comes from there. And so naturally, as you can imagine, there are certain trade-offs where, you know, we don't really spend time trying to figure out a new attention mechanism or something like that. We spend a lot more time trying to figure out much more kind of like practical, hairier type problems, like things like, yeah, what is the right way to process and understand a large codebase? You know, what is the set of information that you can pull? How can you, you know, build a fast, responsive agent that can go and, you know, explore a codebase and figure out the right things quickly? You know, how can you teach Devin how to use Datadog really well if it needs to be able to do that to go and debug things like that? And so that's a lot of what we spend our time on. And in practice, I think how that splits between the work that we do and the work that the foundation labs do, which, you know, obviously we partner closely with all the labs as well. A lot of the work that we do, I would basically describe in the category of solving a lot of these very particular practical problems of what being a software engineer entails. And so like one way that I like to say it is, I think there are a lot of labs, foundation labs, that focus on code. We focus on software engineering, which is a little bit different. And so something like, hey, like my Kubernetes is down, like let me go through the logs and figure out what happened and, you know, restart the service and then see if that works, is obviously a lot of what software engineering is, not necessarily just, you know, writing and analyzing like logical code problems in a vacuum.
As a small follow-up to maybe go a little bit deeper. So RL's been popular for I don't know, some time, and really popular again in the past couple months. We also had the Andrej Karpathy talk where he made it was something along the lines of like RL is like horrible but it's the best thing we have. Maybe touch a little bit on that comment. Touch on how you guys use RL internally and just anything you're willing to share.
Yeah, I think horrible but the best thing that we have is actually a pretty good description. I mean, I think the state of things today is basically, you can kind of think of the platonic ideal of RL as the ability to solve any benchmark. It's basically like if you have a really great eval set of here are all the use cases that I have in mind and here's the set of tools that you're allowed to use and here's how we evaluate whether you succeeded or failed at the task, you can just give that to your model like over and over and over. And it's, you know, it's way more finicky and annoying than this in practice, but in theory you can basically just give that to your model and then you wake up the next morning and you have a model that goes and aces that. And at first people kind of thought like, okay, well if you can solve any benchmark then we're done, right? I mean obviously every announcement that people put out is something about like, oh we solved this benchmark, we solved that benchmark, and now we can, you know, get to the point where someday we'll just solve every benchmark. I think in practice there's actually a lot to do from that because specifically RL doesn't really generalize, and you know there are lots of reasons for why that is. But I think to some degree it makes sense. Like we were talking about this, like Kubernetes example, you know, you can learn a lot obviously as a broad software engineer or by, you know, doing math and physics problems or things like that. But there's some amount of going and debugging this Kubernetes deploy that like you have to just have done before, you know, in order to be able to really, really do. And so I think the natural question then for basically all of us in AI is, so you can train a model that aces any benchmark. What is the benchmark that you want to ace, right? And so in practice, what a lot of this work looks like and what it has, you know, started to converge towards is basically figuring out the exact environments, the exact tasks, the evals, how you determine success or failure. And building around that and then creating models that can go and solve that, right? And code is I think one of the first to really get there with RL because you have this very clean feedback loop where instead of having a human have to judge it or whatever, you can just go and run the code.
Awesome. Thank you. And then so next will be on kind of competitive programming. So for anyone who does not know, Scott is one of the world's best competitive programmers. I think the world's best competitive programmer is a guy named Gennady who actually works at Cognition. You did a retweet from Mark Chen, who's the chief researcher at OpenAI, and it was about OpenAI did a full score in the ICPC. They I think scored six in the IOI. They got a gold IMO this year. You made some comment about how difficult that was. Why was that surprising? What comes next? And what is like a non-obvious consequence that follows from all of this?
Yeah. Yeah. I mean, I think it's a good example of the power of RL, which is getting a gold at the IMO is pretty hard. I think it just—I think like maybe the simple way to put it is like most of the things that we do and work on in our day-to-day life are not harder than getting a gold at the IMO. And obviously it doesn't mean, you know, this is I think where it gets to this kind of like RL eval world, which is it obviously does not mean that this model that can get a gold at the IMO itself will just go and do everyone's jobs for them or something like that. But I think what it does mean is it's a very clear proof of concept that we can teach the models some incredibly hard things that basically with the right feedback loops and the right curriculums and the right setup, you can train models that can do some incredibly difficult stuff. And I think the natural conclusion of that I think is not that everything's over or that all these things are gone, but it just means that like as limited as RL is, it's very clear that in the right circumstances, you can teach it to, you know, very powerful and very useful stuff.
Awesome. You've made the comment publicly a few times that you think we have AGI already. There's no shortage of people who talk about AGI, ASI. Maybe touch on why you think that is. What is that moment we're going to look back on? And then as a nuance, maybe it's the same. Do you think of what code AGI means, and is that a thing in your mind, and what is that if it is?
Yeah, I mean, it's a bit of a facetious comment for me. I mean, I think like the way that I would say it is like I think it's very hard to define what AGI is. You know, people sometimes say this thing of like, okay, AGI is when we've like automated 80% of human labor. And then my first thought whenever someone says it is like, doesn't that mean like the tractor was AGI? Because like, you know, at some point more than 80% of all we were doing was just farming, right? Like there are obviously a lot of versions of this, and naturally what happens is humans do the things that the automation cannot do, right? And so like it's kind of like if you use the post definition, like humans should never be doing things that are automated. If you use the pre-definition, we've done this 80% many times. And so it's like very hard to say. And I do think like someone told me this joke which I like, which is, you know, in 2017 if you asked if we had AGI, the answer is no. In 2025 if you asked if we have AGI, the answer is, well, you have to define AGI and it depends, which I think is kind of true. Like I mean, we passed the Turing test, we've solved some incredibly difficult problems, we've like made novel breakthroughs. You know, we all have this thing on our phones that can just talk and think, you know, almost like a human. And so maybe one way to say it is there's actually not going to be a super, super smooth cutoff of what it turns over. And I think we as humans are, you know, we're proud of ourselves and we want to be able to say like we'll always look for the one thing of, oh, but the AI can't do that. So that's not AGI, right? But I think in practice probably the bigger practical shift is when we get to the point where AI just changes all of our day-to-day lives significantly and makes people much more productive or, you know, gives people a much wider option set of what they can do. And that I would argue is happening already in front of us.
