Andrej Karpathy: From Vibe Coding to Agentic Engineering

Yeah. Hello. Excited to be here and to kick us off.

Yeah, a mixture of both for sure. Well, first of all, I guess like as many of you, I've been using agentic tools like Claude Code, adjacent things, for a while, maybe over the last year as it came out and it was very good at chunks of code and sometimes it would mess up and you have to edit them and it was kind of helpful. And then I would say December was this clear point where for me I was on a break so I had a bit more time. I think many other people were similar and I just started to notice that with the latest models the chunks just came out fine and then I kept asking for more and it just came out fine and then I can't remember the last time I corrected it. And then I just trusted the system more and more and then I was vibe coding. So it was kind of a, I do think that it was a very stark transition. I think that a lot of people actually, I tried to stress this on Twitter or X because I think a lot of people experienced AI last year as ChatGPT adjacent thing. But you really had to look again and you had to look as of December because things have changed fundamentally and especially on this agentic coherent workflow that really started to actually work. And so I would say that, yeah, it was just that realization that really had me go down the whole rabbit hole of just, you know, infinity side projects. My side projects folder is like extremely full with lots of random things and just vibe coding all the time. So, yeah, that kind of happened in December, I would say, and I was looking at the repercussions of that since.

Right. So yeah, exactly. So software 1.0, I'm writing code, software 2.0, I'm actually programming by creating data sets and training neural networks. So the programming is kind of like arranging data sets and maybe some objectives and neural network architectures. And then what happened is that basically if you train one of these GPT models or LLMs on a sufficiently large set of tasks, implicitly because by training on the internet you have to multitask all the things that are in the data set, these actually become kind of like a programmable computer in a certain sense. So software 3.0 is kind of about your programming now turns to prompting and what's in the context window is your lever over the interpreter that is the LLM that is kind of like interpreting your context and performing computation in the digital information space. So I guess yeah, that's kind of the transition and I think there's a few examples of that really drove it home for me and maybe that might be instructive. So for example when OpenClaw came out, when you want to install OpenClaw you would expect that normally this is a bash script, like a shell script. So run the shell script to install OpenClaw. But the thing is that in order to target lots of different platforms and lots of different types of computers you might run OpenClaw on, these shell scripts usually balloon up and become extremely complex. But the thing is you're still stuck in a software 1.0 universe of wanting to write the code. And actually the OpenClaw installation is a copy paste of a bunch of text that you're supposed to give to your agent. So basically it's a little skill of copy paste this and give it to your agent and it will install OpenClaw. And the reason this is a lot more powerful is you're working now in the software 3.0 paradigm where you don't have to precisely spell out all the individual details of that setup. The agent has its own intelligence that it packages up and then it kind of follows the instructions and it looks at your environment, your computer and it kind of performs intelligent actions to make things work and it debugs things in the loop and it's just like so much more powerful, right? So I think that's a very different kind of way of thinking about it is just like what is the piece of text to copy paste to your agent? That's the programming paradigm. Now I think one more example that comes to mind that is even more extreme than that is when I was building MenuGen. So, MenuGen is this idea where you come to a restaurant, they give you a menu. There's no pictures usually. So, I don't know what any of these things are, usually like 30% of the things I have no idea what they are, 50%. So, I wanted to take a photo of the restaurant menu and to get pictures of what those things might look like in a generic sense. And so I built, I vibe coded this app that basically lets you upload a photo and it does all this stuff and it runs on Vercel and it basically re-renders the menu and it gives you all the items and it gives you a picture that it uses an image generator to basically OCR all the different titles, use the image generator to get pictures of them and then shows it to you. And then I saw the software 3.0 version of this which blew my mind which is literally just take your photo give it to Gemini and say use Nano Banana to overlay the things onto the menu. And Nano Banana basically returned an image that is exactly the picture of the menu that I took but it actually put into the pixels it rendered the different things in the menu and this blew my mind because actually all of my MenuGen is superfluous. It's working in the old paradigm, that app shouldn't exist. And the software 3.0 paradigm is a lot more kind of raw. It just, your neural network is doing more and more of the work and your prompt or context is just the image and the output is an image and there's no need to have any of the app in between. So I think that people have to kind of reframe, not to work in existing paradigm of what things existed and just think about it as a speed up of what exists. It's actually like new things are available now. And going back to your programming question, it's not even, I think that's also an example of working in the old mindset because it's not just about programming and programming becoming faster. This is more general information processing that is automatable now. So it's not just even about code. So previous code worked over kind of structured data, right? And you write code over structured data. But like for example with my LLM knowledge base project, basically you get LLMs to create wikis for your organization or for you in person etc. This is not even a program. This is not something that could exist before because there was no code that would create a knowledge base based on a bunch of facts. But now you can just take these documents and basically recompile them in a different way and reorder them and create something that is new and interesting as a reframing of the data. And so these are new things that weren't possible. And so I think this is something that I keep trying to get back to as to not only what can we do that existed that is faster now but I think there's new opportunities of just things that couldn't be possible before and I almost think that that's more exciting.

Well, going with the example of Menu, I guess, so a lot of this code shouldn't exist and it's just neural network doing most of the work. I do think that the extrapolation looks very weird because you could basically imagine completely neural computers in a certain sense. You feed raw videos, like imagine a device that takes raw videos or audio into basically what's a neural net and uses diffusion to render a UI that is kind of unique for that moment in a certain sense. And I kind of feel like in the early days of computing actually people were a little bit confused as to whether computers would look like calculators or computers would look like neural nets and in the 50s and 60s it was not really obvious which way would go and of course we went down the calculator path and ended up building classical computing and then neural nets are currently running virtualized on existing computers but you could imagine I think that a lot of this will flip and that the neural net becomes kind of like the host process and the CPUs become kind of like the co-processor. So we saw the diagram of, you know, intelligence compute is going to, of neural networks is going to take over and become the dominant spend of flops. So you could imagine something really weird and foreign where neural nets are doing most of the heavy lifting. They're using tool use as this historical appendage for some kinds of deterministic tasks. But what's really running the show is these neural nets that are in a certain way. So you can imagine something extremely foreign as the extrapolation but I think we're going to probably get there sort of piece by piece. And I don't, yeah, that progression is TBD I would say.

Yes. So I spent some time writing about verifiability and basically like traditional computers can easily automate what you can specify in code and kind of this latest round of LLMs can easily automate what you can verify in a certain sense because the way this works is that when frontier labs are training these LLMs, these are giant reinforcement learning environments. So they are given verification rewards and then because of the way that these models are trained they end up basically progressing and creating these jagged entities that really peak in capability in kind of verifiable domains like math and code and adjacent and kind of stagnate and are a little bit rough around the edges when things are not kind of in that space. So I think the reason I wrote about verifiability is I'm trying to understand why these things are so jagged. And some of it has to do with how the labs train the models but I think some of it also has to do with the focus of the labs and what they happen to put into the data distribution. Because some things basically are significantly more valuable in economy and end up creating more environments because the labs wanted to work in those settings. So I think code is a good example of that. There's probably lots of verifiable environments they could think about that happen not to make it into the mix because they're just not that useful to have the capability around. But I think to me the big mystery is, the favorite example for a while was that how many letters are in a strawberry and the models would famously get this wrong and it's an example of jaggedness. The models now patch this I think but the new one is I want to go to a car wash to wash my car and it's 50 meters away. Should I drive or should I walk? And state-of-the-art models today will tell you to walk because it's so close. How is it possible that state-of-the-art Opus 4.7 will simultaneously refactor a 100,000 line codebase or find zero day vulnerabilities and yet tells me to walk to this car wash? This is insane. And to whatever extent these models remain jagged, it's an indication that number one maybe something's slightly off or number two you need to actually be in the loop a little bit and you need to treat them as tools and you do have to kind of stay in touch with what they're doing. And so I think all of my writing long story short about verifiability is just trying to understand why these things are jagged. Is there any pattern to it? And I think it's some kind of a combination of verifiable plus labs care. Maybe one more anecdote that is instructive is from GPT 3.5 to GPT-4 people noticed that chess improved a lot and I think a lot of people thought oh well it's just a progression of the capabilities but actually it's more that, I think this is public information I think I saw it on the internet, a huge amount of chess data made it into the pre-training set and just because it's in a data distribution basically the model improved a lot more than it would just by default. So someone at OpenAI decided to add this data and now you have a capability that just peaked a lot more. And so that's why I'm stressing this dimension of it as we are slightly at the mercy of whatever the labs are doing, whatever they happen to put into the mix. And you have to actually explore this thing that they give you that has no manual. And it works in certain settings, but maybe not in some settings. And you have to kind of explore it a little bit. And if you're in the circuits that were part of the RL, you fly. And if you're in the circuits that are out of the data distribution, you're going to struggle and you have to kind of figure out which circuits you're in in your application. And if you're not in the circuits, then you have to really look at fine-tuning and doing some of your own work because it's not going to necessarily come out of the LLM out of the box.

So I think maybe that comes to the previous question of I do think that verifiability because it, let me think. So verifiability makes something tractable in the current paradigm because you can throw a huge amount of RL at it. So maybe one way to see it is that that remains true even if the labs are not focusing on it directly. So if you are in a verifiable setting where you could create these RL environments or examples then that actually sets you up to potentially do your own fine tuning and you might benefit from that. But that is fundamentally technology that just works. You can pull a lever if you have huge amount of diverse data sets of RL environments etc. You can use your favorite fine-tuning framework and pull the lever and get something that actually works pretty well. So I don't know what the examples of this might be. But I do think there are some very valuable reinforcement learning environments that people could think of that I think are not part of the, yeah, I don't want to give away the answer, but there is one domain that I think is very, oh, okay. Sorry, I don't mean to vape post on the stage, but there are some examples of this.

I do think that ultimately almost everything can be made verifiable to some extent. Some things easier than others. Because even for things like writing or so on, you can imagine having a council of LLM judges and probably get something reasonable out of this kind of an approach. So it's more about what's easy or hard. So I do think that ultimately, yeah, I think everything is automatable.