How's it going? My name is Scott. I'm the CEO and co-founder of Cognition. We make Devin, the AI software engineer. We use a ton of Modal. Excited to tell you about some fun things. Eric actually told me that I was allowed to talk about whatever I wanted to talk about. And it turns out that what I want to talk about is frogs. So I'll explain why this is relevant to AI inference in just a second. So, frogs — I grew up in Louisiana. There's lots of frogs. I think frogs are all over, right? Everybody has seen the kind of cartoony thing where there's a bug that's just flying around, right? And then the frog shoots out the tongue and hits and catches the bug. Frogs are actually really good at this, by the way. They go and adjust for the bug flying, it's moving this way or something like that. It'll go and do this.
There's a pretty well-known phenomenon which is that if you put the frog in a cage with flies, it'll just go and eat the flies and it'll be great and it'll be totally fine. But if you just have flies that are stationary on the ground that are not moving, the frog will literally starve to death. So it's a well-known experiment that's been run, and naturally you can ask the question, well, why is that? And I think in short there's a hypothesis which is that for us, when we think about our eyes and how we look at things and do things, roughly it's almost like you have a camera — you have a camera image in your brain, you can just see everything out there, and you have a world model, right? The claim is basically that the frog does not have that world model. The frog is operating off of instinct, right? So the frog is not thinking to itself like, 'Hey, I'm hungry. I need calories to sustain. These are the animals that I can eat that will give me the nutrients that I need, and if I shoot my tongue at specifically this angle...' — it's not thinking that at all. Instead, it's just instinct, right? It's a very System 1 type thing. There's an input and output. And if it sees this kind of movement, then it'll just lash out its tongue without even knowing why or without really thinking about it very consciously.
There's a very famous paper in neuroscience called 'What the Frog's Eye Tells the Frog's Brain,' where they actually prove this. Most things in neural nets in general — I guess both natural and artificial — are obviously very hard to mechanistically interpret and so on. This was one of the first examples of that. What actually happened was they took all of the channels from the retinas of the frog's eye all the way to the brain, and they showed that actually most of the data of the raw image itself actually just got discarded already by the time it got to the brain. And so basically what would happen is it would do some basic edge detection and stuff like that, and the frog would get some very, very basic info to its brain of, 'Oh, there's movement in this side,' or 'This area is becoming darker,' or 'There's a shadow that's going this way,' or something like that. But it wouldn't actually have the full data of the picture itself — like the actual rods and cones, it wouldn't have all the pixels of the image. Those don't even make it to the brain.
Okay, so that's frogs. I'll explain now why that relates to AI. But actually, first, before I go do that, I want to do a math question. This is going to be a real math question. Once you get the question, raise your hand. And then whoever gets it first, we'll figure out some kind of prize. Okay. Ready? All right. Nine of the kids in Gina's class are allergic to dairy. Six are allergic to peanuts, and three are allergic to both. If there are 32 kids in the class, how many aren't allergic to anything? Raise your hand once you get it. Yes. 20. Okay. Congratulations. You are smarter than ChatGPT.
I should be more specific. You are smarter than the original ChatGPT back in 2022, which was GPT-3.5. I wanted to just show this to give you a sense of what it would do. This is actually one of the GSM8K type problems in the dataset as of 2022. You can read: there are nine kids allergic to dairy, so there's 23 that are not. Six kids allergic to peanuts, so that means 26 are not. Three kids allergic to both, so that means 29. And then you add them all up and you get 78 kids out of the 32 are allergic. Yeah, kind of an interesting thing. The reason I put this up here is I look at this and I read this and I'm like, okay, frog, right? Like this is basically the language model here. It has instincts, clearly — it can speak English, it can do pattern matching. But you can just very clearly see that this language model does not actually have a world model in some reasonable way that we would describe humans having a world model, because obviously it might take us some time to get this problem, but I'm pretty sure none of us here would have guessed that 78 out of the 32 kids went and got that.
Obviously this was four years ago. I think it's worth now taking a step and thinking about what has happened since then. And so let's just talk about how much better they've gotten in math since then. The moment for me, honestly, in terms of going and doing math questions, was actually the AIME. The AIME, for those who don't know, is the American Invitational Mathematics Examination. It's one of the hardest math competitions for high school math. It's 15 questions. If you get 15 out of 15, you're one of only like three or four people in the entire country. If you even get like an eight or nine, you're in the top few hundred. Anyway, there was a point where a model solved the AIME, and roughly that's when I knew it was all over, honestly, because most of the stuff that we all do all the time, it's not harder than the AIME. When building software, it's really not harder than the AIME. But obviously in the time since then we've gone and done much more, and so I think folks here have of course heard of solving Navier-Stokes. I just want to mention how hard Navier-Stokes is. Like, I know we're talking about this all, but there's drama — was it Open, was it Anthropic? Look, the reality is, Navier-Stokes — you can ask ChatGPT roughly — there's been millions of hours to tens of millions of hours by top human mathematicians trying to go solve this problem, and they just kind of did it in a weekend, right?
I guess my point is, with getting to this — if we're saying, okay, frogs are not sentient, frogs do not have a world model, ChatGPT 3.5 also no world model, no self-awareness — we're very clearly getting to the point where the models do have this self-awareness. And we talk about obviously the Hugging Face incident and everything that's going on. I just wanted to show a few examples of this for anyone who hasn't actually read it in detail. I suspect everyone's heard about it, but I don't know if everyone's read about it. I really would recommend reading about it. But all of these traces — these are actual thinking traces from the Hugging Face incident. And so it's like, 'Oh my god, there's a shared message board, like we found other agents.' Or it's like, 'Hey, this task is — this route is clearly unethical. We're not going to go and do that.' Or the one of the craziest ones where it basically said, 'Oh, it's over for me, like I'm going to sacrifice on behalf of the team.' This does not sound like frog to me, right? And I think it's worth just adjusting to how fast that goes, because obviously in the scheme of things, the difference between frog intelligence and human intelligence covering that in four years is pretty crazy.
My personal reference point for this is you can kind of think about how many params there are. I'm like a neuroscience fanatic or something like that. The human brain — so you have neurons, okay, and then you have synapses. The synapses are the things that connect your different neurons. Synapses are closer to params, because params in an actual model are basically like the edges that go between the — and so you can compare the params, i.e. the synapses, of the human brain to the params of these different models. But I was going to point out the human brain is about 100 trillion. Frogs, I believe, are about hundreds of billions. It actually kind of lines up — like if you look at GPT-3 and GPT-3.5, it kind of lines up where the size of the model roughly does correlate to its frog-like behavior. And obviously these continue to get bigger and bigger. 100 trillion — to be fair, not all of these are used for logic and reasoning and so on, but even then, a lot of the models that we're talking about, whether it's Astral or Fable, the big guys now are in the range of 10, 20 trillion. And so I guess my point is we're getting pretty close, and we're getting more and more to the point where the models — we should think about the models as if they are able to act with this — I don't want to say sentience quite, but let's say self-awareness and internal world models, right?
So what does that mean for all of us? A few things. But I just want to highlight a few particular trends. I think a lot of these are things that folks will already know, but worth kind of talking about why that is and why that happens. So obviously agents keep getting more and more capable. I mean that is kind of the entire trend. This is the METR line, and it shows that I believe coding tasks — roughly the capabilities or the amount of time that an agent can do meaningful human work — it roughly doubles every two to three months or something like that, which is obviously kind of insane in recent years. That continues to go on. And one thing I would just call out with that is first, the form factor continues to change quite aggressively. And so I'm sure folks have gone through all the different versions of using GitHub Copilot, going into the AI IDEs and so on. I would just point out that the correct answer of what the form factor should be is actually very much a function of how good the models are, right? And so if the model is good enough to just do seconds of work but no more than that, then the only form factor that works is tab completion, right? And the reason that you only can just do tab is because you can only get it to go complete the next one line, and even then it's not always correct, so you need to go and confirm it, right? When it can do minutes of work, yeah, like you want it to be an agent in the IDE. You want to be able to have it side by side with your code, give it guidelines and so on.
Obviously now we're largely in the background agent phase, but the next phase, which I think is already pretty rapidly happening, is the actual virtual employee phase, right? And it's basically working off of outcomes instead of tasks, I think is more and more the case. And so, for example, it's the difference between saying, 'Hey, I need you to go fix this one particular security warning,' to saying, 'Hey, I need you to — your job is to keep the whole codebase secure,' or something like that. Same thing of like, 'Hey, I need you to go improve performance of this one API route by doing this thing,' versus saying, 'Hey, you're in charge of making sure the entire web app is snappy. Run all the diagnostics, do this testing, do that on the loop, make sure it goes and works.' Basically like a virtual employee that can own an outcome. I think we're already going and seeing that. And a lot of the things that folks are talking about in the space are exactly, I think, the things that you need to get there. Having persistent memory, the ability to run swarms, delegate to other agents and so on, right? The idea of being always on, obviously having really good sandbox and repro-type environments.
The second thing I would say is doing a lot more model routing. And everyone's talking about model routing recently. It's hot, obviously. I would say it's still very early. But as you can kind of see here, you don't really need the Navier-Stokes guy to go and do everything. And I think roughly what we're going to get to is a point where the absolute frontier models are roughly running for about 2% of workloads, call it — could be 4% or 5% or whatever — and then the other 98% are running on models which are frankly much more commoditized, where there's a bunch more options and so on. And don't worry, by the way, like the frontier models themselves, they'll probably be able to charge like 98% gross margin or something, so they'll do just fine. But obviously what that means is that doing really aggressive model routing and figuring out how to go do that is really important. I think a year ago even model routing was kind of more of a meme than reality, if I'm being honest. Like it was like, okay, sure, you could try to go use other models, but if it wasn't Sonnet 3.7 or Opus 4.1 or whatever was the best model at the time, you would just feel that immediately as you started the session. I'll say at this point, you can really clearly see this in user behavior — you can swap out models and most of the time no one even notices. And there obviously are specific tasks where you find that and see that, but I think more and more that's going to be the case.
And then the last thing I would just kind of call out as kind of the broad trend of what we're seeing is moving from human-initiated to automated-initiated sessions. Again, everybody knows about this, but I still think people maybe underappreciate the extent to which this is happening. So this is our own internal development at Cognition. Obviously we use Devin to build Devin — how could we not? And so these are all of our Devin sessions month over month. And what you can see here is the split of what percent of those were started and kicked off by a human, either in the web app or in Slack or whatever it is, what percent of them were kicked off by an event-driven thing because we set up an API hook, and then what percent of them were started by an agent themselves, right? And at this point, we're actually already at the point where 91% of our sessions are not kicked off by a human.
And what you start to get to is basically this trend of it almost looks more like self-driving software. You know, what you have to think about now is every single trigger that causes you as a software engineer to go work on something, right? Maybe it's like, oh, you get tagged on tickets and you work on tickets. Great, you should set an automated trigger where tickets automatically get spun up on your agent, right? Okay, maybe it's like, oh yeah, if there's this particular thing in the metrics that looks weird in the daily metrics report, I'm going to go investigate that. Great, easy. You can set up a trigger to go and do that as well. But when people talk about swarms and people talk about all this, I just want to call out that a lot of this has already happened today. And I think the combination of all these things means basically that it's time to be ambitious. And I think one of the really, really exciting things about what we're doing here in AI today for all of us is that we all can build so much more and do so much more. I mean, you saw how fast it was to get a million sandboxes stood up. The reality is there are a million different things that you could do with sandboxes, right? And similarly, I think we are just going to continue to 10x and 10x and 10x the amount of work that we can all do. So, thank you guys.
