Martin Casado of a16z on AI Innovation and AGI

Very happy to be here.

Listen, I think like all things, the past is probably the best predictor of the future. My hope is this continues to be a very useful tool to help us with progress and to stave off this universe that seems pretty destined on killing us. So I'm hoping it continues on that trajectory. What is interesting is if you actually look, let's say, the past 80 years of quote-unquote AI, it's been steady progress independent of there being winters and summers. It's been very, very steady progress. There's been progress on economics, some problems solved, and every time we tackle a problem, like everybody's, oh goodness, this is it, we're almost at AGI, and then it just turns out like that's one modality, and then we go on to the next and we say, oh, that wasn't really actually AGI or AI. So like this whole AGI thing has been a moving goalpost for 70 years. And so I would say, listen, there's been this kind of very steady progress. We've gotten good at many things. It's been a very useful tool. I hope it continues to be so. I think it's a very important thing for us to continue to develop and use, but I don't think that there's any step change or it changes the nature of computers or software in ways that we haven't seen before.

So again, I think I'd like to draw from a historical analogy because I think they're very useful in this context because I just feel like we've seen these very often and they all look kind of the same and it's played out. And so when I first joined Stanford to do my PhD, so I did my PhD in computer science and systems, was in 2003, and around then, I don't know, it was 2003 or 2004, Sebastian Thrun had won the DARPA Grand Challenge. So the DARPA Grand Challenge is he drove a van autonomously, fully autonomously for 1,200 miles, and everybody was like, hooray, like AGI is solved, robo-taxi is solved, this is an amazing thing. And 100%, we've hit one of these threshold moments, 100%, like you could actually do things you couldn't do before. It was the start of a new era of vision and perception and self-driving. And now 20 years later, and about a hundred billion dollars invested, a little bit less than a hundred billion dollars invested, the unit economics of self-driving are still three times worse than Uber. And so I am a systems person, meaning all I've been doing for the last 30 years of my life is building computer systems, large distributed systems. This is what I do. And I just know that scale and capabilities, they're not necessarily parametric, so they don't necessarily just go up and to the right. And there's so many examples of this, and we can talk about that. I know that the universe is this very heavy-tailed system, and there's no single solution that tends to reign in its complexity. I know that the economics for these things are very, very hard. Like what gets lost in these conversations is AI has been better than humans for a very long time at many things: handwriting detection, diagnosis for a very long time, game playing for a very long time, mathematics since the creation of the computer, four orders of magnitude better at mathematics than us. And yet none of these things have resulted in kind of generally economically viable solutions for everything. They just for subsets, right? And so for, we can talk about this language. So another way to rephrase what you said about the language stuff is these models are very good at predicting what a mean person would do next, a mean person would do next. So sure, they basically do kernel smoothing over positional embeddings, which is great, which means if you have a corpus of data, it'll give you the average response, which is great, and it's very useful for a number of things. Does that provide economic utility for a broad range of stuff? I mean, I'll tell you, I look at these companies basically full-time, and I have for three years. We probably have the largest portfolio of them, and I don't see that yet. Now, there are areas where they do work, but it's not that, it's not this kind of general knowledge worker that's not actually not working. And the areas where it is working is stuff like the marginal cost of content creation goes to zero. So now we have like amazing new content, or computers are creating an emotional connection with humans that's new and that's exciting and that's amazing. But this idea of this general knowledge worker working, like I haven't seen it yet, and maybe it'll happen, but just like self-driving 20 years ago, like that's just not how the universe tends to play out.

Yeah, so I don't know if we're there kind of unit economic, and I don't know if they're there on an exception basis. I do feel with a lot of these AI things, you enter this area where you get good enough that the economics suggest that you change the environment rather than make the AI general. And I think this is probably best described with agriculture. So we don't have any unit economically viable way to pick strawberries. Like human beings and animals have been foraging food forever. We're very efficient at it. To actually compete with a human on that is very tough. On the other hand, what we can do is we can plant strawberries in certain rows, and we can add these optical cues, et cetera, that kind of get it much closer. And so I definitely think that we're at a place where, for many cases, AI is better than humans at driving, which I think is great, and I think it's a huge positive. I think these are hugely specialized systems. I don't think you can take the one that's good at self-driving and make it, I don't know, be a virtual girlfriend. I think there's a totally different modality. So it's not general, right? It's not general. It's just good at driving, just like something is good at chess. And now I think we're at the long tail of the problem for fully autonomous stuff like Waymo. The unit economics are just very tough, right? They're very tough if you're competing with a human driver. For other ones, I do think that we, if we just start evolving how we make streets and so forth, I think we're pretty close, and I think that's great. I think this is a positive for everybody.

Just a quick digression because this is just so important to this discussion on how the heavy-tailed universe makes this conversation very complicated between human beings, because I don't think people appreciate how heavy-tailed the universe is. So I just want to describe this because it just comes up right now, and it's going to be so relevant to what we're going to be talking going forward, which is there are many things that systems deal with that are heavy-tailed. What heavy-tailed means is if you draw an occurrence at random, just draw one at random, the chances are that it's a very, very rare occurrence. So a very classic case of this is search. And so if you draw a unique search query, say Google, at random, the chances are it's pretty unique, actually, even after all of this time. Now, this is unique searches. So if you take all of the searches that go into Google, say, and then you don't dedupe them, so like you can have a lot of repetitive ones, the vast majority of searches are the same. Let's say 90% of them are common. But if you reduce it to just singular intents, so you don't have any duplications, the majority are exceptions. And that's how the universe tends to be. Like I come from networking, it's a very famously heavy-tailed discipline. A lot of things in the tail. So if you're building a general system, the majority of new things it has to do, not things it has to do, the majority of new things it has to do are exceptions. Now, it's very easy to get tripped up in these conversations because the majority of stuff that you're doing is not new, right? The majority of stuff you're doing is very common. But anytime you get into new areas, then you have to come up with new stuff. It's like self-driving is a very great case of this. Like nobody expected basically a 2D vision problem. Let's be honest, man. Like self-driving is 2D. It's not even 3D. It's like you have streets and you have signs. You don't really worry about the Z dimension, really. And yet a hundred billion dollars in, a hundred billion of investment, we're almost there, but not in an economically positive way. The reason I wanted to make this point now is I think so much of these discussions, as people, I believe, underestimate how heavy-tailed the universe is and how hard it is to make progress. And so we should be working as hard to make progress so that we can do stuff like self-driving and not slow it down because the task is enormously complex.

So let's, let's, okay, so let's move this to here. So as far as I can tell, what LLMs are doing, and there's actually papers that have shown this pretty concretely, is they take a corpus of data, they create positional embeddings, and they basically average over it. And based on that, they can predict what a human being would do or say given a certain situation, right? And they do that basically via averaging. For 80% of the tasks, let's say that's great. This is for 80% as non-unique, right? So you're not duping tasks. But let me give you an analogy. What if I came to you, I'm a startup founder, I'm like, hey, Nathan, I've got this amazing chatbot that for IT help tickets, it answers 90% of the time. And you're like, this is amazing. Like 90% of the work goes away, right? This is what you say. But then you actually do diligence on my company and you realize that 90% of that, 85% of that is just answering password resets. So I was factually correct saying that 90% gets answered. That's the case. But the reality is the situations that I help with are very easy to automate if you wanted to, and they're not that much work. And you still have, from a unique standpoint, you still have to have humans do 50% of the unique things that come in, 50%, right? So that was the state of chatbots for the last 10 years, and we had the exact same discussions. They seem to answer 90%, they answer 85%, they can do these things, we're almost done, but that's not how heavy-tailed distributions work. So let's again, so let's go to this case that you're making. 100%, these LLMs solve problems we couldn't before, and 100%, they're a very useful computer science primitive. We're nowhere close to understanding what these distributions look like, how heavy-tailed they are, and how they can handle that tail, just like we weren't with AV, just like we weren't with original chatbots. And so I just think that without being specific on uniqueness and the distribution and whether we think it's parametric, it's very hard to have these conversations. So we resort to generalities that sound good, but I think from a system standpoint aren't very sensible.

I think it's going to be like self-driving. I think we'll get 80% there with these agentic systems, and then people are going to start changing the environment to make them more effective because human output is also heavy-tailed. And so my best guess, again, I'm going to repeat myself, but I actually think this is how it's going to play out, is right now, anytime we're always at like the beginning of what looks like an exponential, but it's actually a sigmoid. And what we think is like a fat-head system, but ends up being a heavy-tailed system, it always feels like this. Like it feels right now, and everybody's saying all the stuff that they're saying right now, and we're like, it's amazing, you know, everything's going to work out. And then we're going to realize, oh my goodness, the universe is heavy-tailed, this is really hard. These things are good at 80% of stuff, but the 80% of stuff that it's good at is not that hard. And the stuff that I needed to be good at is really hard. And so then we're going to enter an area that we're like, oh boy, we talked about scaling laws, throwing 10 times more compute at something over a period of time is so impossible that what we're going to do is we're going to actually change the way websites work, and we're going to actually have indicators, and we're actually going to have lanes, and then we're going to spend the time actually changing the systems. And so over time, between changing the way websites work and where these agents work, we'll enter a period where these things are more autonomous. But I don't think it's because we've cracked the generality problem. I think it's because we've evolved computer science as a discipline and tech and software as an industry in order to solve the problem.

I think this is a great question. I think maybe this is why I come to this from a different thing. Prior to my PhD, like my life was computational physics. I worked at Lawrence Livermore National Labs. I did large physics simulation. Like I just lived this life. And I think the Occam's razor in all of this is distribution in, distribution out. These things are very good at learning distributions of the training set. And so this is just in this kind of weird, quirky coincidence of this moment. I actually worked on protein folding at IBM Watson Research Center in 1999 on the Blue Gene project. And at that time, there was a belief that we knew enough of the fundamental forces in order to actually from first principles calculate protein folding. That was the thesis, and that we just didn't have enough compute. And so they actually built a whole computer to do this called Blue Gene. And through corporate mechanics, that ended up becoming an entirely new computer because they couldn't fund it. It never really happened, but there was the belief that if you had enough compute, that you could do protein folding and a bunch of other stuff. And so I think it's very reasonable, this is a strict Occam's razor, that if you have enough compute and you have enough data and you're dealing with an axiomatic system that you can reduce to first principles, then you will learn that distribution and you can use it for predictive stuff. And if you want to call predictive emergent, that's fine, but it's just predictive. Like anything is predictive. Like I've worked on simulation codes that were predictive. Could I have predicted the yield of a nuclear weapon? No, I could not have predicted that as a human being. Only a computer can. But it's really just understanding first principles to create these things. I very strongly believe these models are very useful in science where you've got fundamental laws of nature that are being learned that can be predictive. I think the models that are good at that is not going to turn around and solve a different problem most likely because I think you've probably learned one distribution from one domain and that's a very useful tool that we should use.

Now let's go ahead and move that over to language now. So if my Occam's razor is these things learn structure in a data corpus, it's distribution in, distribution out, that means that they're learning structure of the text corpus that they've been fed. And a great example of this, there's a recent paper I saw, I don't know if it got accepted or not, I think it was just on arXiv, but that showed that if you could gzip a text corpus and the compression was good, then the accuracy was good, which suggests almost everything you need to know about this, all it's doing is learning structure in the text. It has nothing to do with underlying meaning. It's just structure that's in the text. So you can understand the distribution of text, you can actually spit out text, but this doesn't say anything about learning fundamental principles of the world from which the text is based. And so it almost seems to be that the magic in the sequence of text is you've got these humans that have spent, say, 3,000 years looking at the universe, and the universe is, again, heavy-tailed, and it's nonlinear, and it's very complex, and it's fractal, and it's similar, and these are notoriously complicated systems to simulate. So then the human brain was like, I'm going to abstract this out as a tree, and I'm going to abstract this out as like this concept. And so we're like these machines that take the universe and abstract it into things that are words, but it's a very lossy representation. You can't describe something and get back the universe. But it turns out because we have those abstractions, they're very predictable. Like we've made the universe more linear, and we've made the universe less heavy-tailed, and we've made the universe less self-similar. And once we've done that, we've added structure that's predictable. So the fact that you can take a corpus of text which a human being has pulled from the universe and then made the universe much simpler and find structure in that is not surprising at all. But to think that somehow then you can go from that to the universe is a step that like just simply has not been demonstrated, and it actually doesn't even stand to reason.

There's structure that's very different than meaning. So we're talking about three things. And this is a great conversation because I think it gets to the heart of it. So the universe is self-similar, it's fractal, meaning no matter what zoom level you look at it, it has the same stochastic properties, right? So like you can spend an entire life studying a cell and a planet, right? That's how much complexity is in the universe. The universe is heavy-tailed, which means the exception is the norm if you dedupe, and it's nonlinear, which means that you can't computationally predict out too far just because we don't know, we don't have closed-form solutions for nonlinear stuff, and it's just a very hard computation problem. Like that's the universe. Now, human beings have had to navigate this crazy universe, and so we've created this amazing engine, which is the brain, and it has reduced this universe to concepts and words and stuff that we use and we talk about. That kind of makes it a little bit predictable, and so at least you and I can communicate about it. But if I tell you like this is a tree, there's a concept tree in my brain, but it's almost an arbitrary distinction that it's a tree. I could talk about branches versus leaves. I could talk about networks of trees. That's actually one tree, like the big aspen grove. I could talk about cells of the tree. It's like this kind of arbitrarily useful distinction. So it has some semblance to the real world, but if I say a tree, it's probably not an accurate relative to how the world is. Is it one tree? Is it a family tree? It's just a useful abstraction. So these models will 100% recover the abstractions that we've put in text because the structure is all there. That's not surprising. Like compression would do that. All it's doing is taking advantage of structure, and that structure is real. But let's say that it's finding a tree. Does a tree actually map to the universe in a meaningful way? It's a human-created concept that has some vague sense to something we all agree on, unless you're a scientist, then you probably disagree with the common understanding. And oh, by the way, my concept of a tree also includes like a toy and a cartoon picture of a tree, which is entirely different, right? And so text is a way that we as humans represent the world that's very different than the actual universe because we find it useful. There's structure there, and these LLMs exploit that structure just like compression would or anything else. And it's very useful for us, but it doesn't necessarily mean that these things can enact on the world, right? These are very different domains.

No, that one I have. I think when dealing with this much compute and this much data, I think the human intuition just totally fails, and we as humans are really bad about thinking in distributions anyway. That's not how we think. We kind of assume the world is parametric, and like, by the way, which is why the text that we create is so well structured and why it can be exploited by LLMs. Like you have to navigate this universe, you have to make the simplifying assumptions, which we do. And so it's good to actually come up with mental frameworks about how these things work. So I'll tell you a few of mine. The first one of mine is, as far as I can tell, these things are explaining structure in whatever data that they're reading, and like we've mentioned before, and it's not clear whether that distribution extends beyond that. And if it does, then you're basically back to simulating the universe, which I've spent a lot of time with. I think that's very true. The second one is the actual mechanisms we're talking about, transformers. The actual mechanism is basically kernel smoothing. It's averaging, which means to me is the further you get out into where the data is rare, the greater the inaccuracies come. And that doesn't mean that for systems that you can actually extrapolate from that, you don't get great results that would be quote-unquote out of distribution. It turns out some systems are linear, or you have enough data, you can map the distribution. So that one is totally fine. Then the third one is this in-context learning one. And I think Vishal Misra, who's a professor at Columbia, did the best work on this, where he actually shows that for in-context learning, where you actually put the context in the prompt and you can move the posterior distribution to get interesting results, he mapped it specifically to Bayesian learning. And it's a beautiful paper. I don't know why more people don't read it. So listen, we know that these things can do some basic Bayesian reasoning, and this is where the prompt is basically the new evidence, which changes the posterior function. So you'll get new stuff there. We know that if you average enough stuff, you'll get new stuff there. It just has to be linearly interpolatable. If it's not linearly interpolatable, you're not going to get new stuff. So none of these things suggest that you're not going to get new stuff. It just puts constraints. We know how Bayesian systems work. We've got 20 years of understanding convergence properties to them. We have work that's specifically mapped ICL to Bayesian learning. So let's just go ahead and use that corpus of work to understand the properties. It doesn't say it's out of distribution or in distribution. It doesn't say that at all. It just bounds what that means. And then we also know the mechanics of the way transformers work, which is this kernel smoothing. I mean, it's more complex than that, and so that can create new things, but it means there's a linear interpolation. And so I think the problem is we have these anecdotal conversations where it's whack-a-mole. It feels almost like the intelligent design discussions, where someone will come up with a new set of evidence of something that there's no way evolution could have created this, and we have to spend all of this time to show actually you can, but we don't know everything, and here's the theory. And I feel like when we have these discussions, they should be a bit more principled as opposed to, I've got this anecdote that seems something's new, because nobody says that you're not going to see new stuff. I mean, it very obviously, if you're doing interpolation, it's new. Very obvious if you're doing Bayesian reasoning, like something's going to be new. And talk more about the distributions and the theory of why we're doing that. But as far as I can tell, that's totally missing. Nobody's come and said, here's my theory of out-of-distribution stuff. Here is my thesis for what is going on functionally to create this new data. And on the other hand, you've got mounting tons of evidence that map these to existing systems that we know that people just seem to not want to follow. And I just feel like, listen, humans love to see things in clouds and complex systems. There's so many facets, and they're so complicated, and they're so huge, and they're so ethereal, and then we see things, and we just do this historically. And we've got these kind of amazing computer elements that are huge, and they surprise us, and they're amazing, but we can map them to formal systems, and we know how they work. And that doesn't mean that they're dangerous or not dangerous. I'm not saying that. I'm just saying that we can actually map them to reasonable systems to have a discussion, and that just seems to be missing. This conversation is a great example. I'm very happy to map these things to formal systems we understand and have that discussion, but it's always this kind of anecdotal whack-a-mole instead, which I just don't know how to answer to every instance of what seems like emergent behavior when, of course, emergent behavior is expected. Anyways, I am definitely a materialist, and so I do believe that in the fullness of time we can probably figure out an arbitrary level of detail how these things work. They're actually way easier to study than life, for example.

Still, so a lot of this reduces to how you view the universe. If you don't view the universe as fractal, self-similar, and if you don't view it as heavy-tailed, and if you don't view it as nonlinear, then you could imagine that. But it is all of those things. We know it is all of those things. And so there's no distribution of data that we know of that's not the universe that will produce something that's predictive of the universe. It's really that simple. That doesn't mean that we can't focus on an area and reproduce that distribution. I could become very good at predicting whatever protein folding. I could get really good at playing chess. But it's distribution in, distribution out. If you notice, the ones that like stuff like recursive self-similarity works, and control loops work, and simulated data works, like synthetic data works, there are these axiomatic areas where the axioms, like, they constrain the search space and you're basically converging on search, and you can get very good at those. I can get much better at you at arithmetic. I can get much better than you at game playing. I can get much better than humans at all of these things, but none of that talks to the fact that can you find the right level of abstraction in a fractal system and can you tackle a heavy-tailed universe where not by occurrence but by uniqueness the complexity is in the tail. Even in this discussion, the use cases that you view tend to be these kind of axiomatic, you know, of course we can do full, like we thought in the late 90s we could do protein folding by fully searching the search space. It's just not surprising to me that we can learn distributions and spit them out. I feel that's a very different statement than saying now we have a model that can navigate the universe in a way that is predictive of all of the complexities of it. Maybe another way to think of this is we've spent 3,000 years doing our best and writing it down, and we can create a model that can learn from all of that and do what the mean human being would have done in the last 3,000 years. But the problem is the stuff that we're doing tomorrow, by uniqueness, a lot of it's going to be new, and that's just how the universe works. And we're going to have to either build a machine that can do that, which we don't know how to do, or we're going to have to do it ourselves and let these machines do the mean task.

I mean, two things. One of them, and the most important one, is we experience the universe and we abstract it into concepts. Like, that's very comp—like, why do you think all the budget is moving to post-training, by the way? What's your thesis of why so much budget right now is moving from compute to post-training?

Yeah, that's about half. And so what I would argue is we're very good—models are very good at taking the output that humans create and being able to reproduce that distribution. They're very good at that. This is why Scale AI is Scale AI and all these companies are so successful. That's not the game. The game is looking at the universe and creating the supervised data. That's the game. And so I think one model to look at this is, again, human beings have been around, let's say, in a capacity for writing things down for 5,000 years. So you've got humans for 5,000 years that have been looking at the universe and doing this thing that models cannot do, which is making a decision like that is a rock and that is a dust and this is a concept and this is a relationship and I'm going to write it down. And then as a group, we're going to synthesize these ideas and work at these. And so we've got this kind of almost kind of Platonic representation in our heads of the universe, and that is very structured. So we did all the hard work. That is hard work to take this untamed universe and reduce it into words and concepts. That's hard. No LLM that I know can do that, not even close. But then once we've done all of that hard work, is it surprising to you that there's structure? We did all this work. Of course there's structure. So you take that structure, you put it into an LLM, it learns the structure and it can spit it back out. So the very specific thing that these LLMs can't do is look at the universe and recreate this kind of structure. And to be super clear, that structure is not the universe. That's very different. Like a rock is a rock is an idea in our head. It's not representative of any single thing. Is a grain of sand a rock? I don't know. Is a boulder a rock? I don't know. These are concepts we've created in order to navigate the universe. They're not the universe. So human beings take the universe and create the concepts and the LLM, and then that's structured because we need structure in order to do anything. The LLMs learn that structure. Feels like the only thing that works is basically you can do exhaustive, quote-unquote, AI, which converges on search to learn distributions, or you do supervised learning, which human beings are doing the hard work, in my opinion, by labeling things and everything else, and then you just learn what the human beings have done. But to take the universe and actually to rain structure and all that complexity, that's what we do. And it would be great if machines can do it. I've never seen any evidence that they can. Yeah, maybe some glimmers of it, but that's just not where we are.

We know enough about the natural sciences to build predictive models when we have for a very long time, right? Like I can simulate a supernova on a computer pretty accurately. I think it's phenomenal that the—it's phenomenal that we have an approach to throw computer at a problem that learning a fundamental law of physics. But again, how is that any different than the fact that we've been modeling physical systems for a very long time, other than the fact that in these cases it allows us to apply more compute at the problem and we can solve problems that we don't have closed-form analytic solutions to? It almost feels to me like an extension of simulation, which would be very different than the claims around general reasoning. It's literally learning laws of physics, which we've been doing since the beginning of compute, right? The creation of computers was for ballistics, right? So the reason that we created computers is because ballistics are nonlinear, right? The trajectory of ballistics are nonlinear. So we had people in rooms that would create logarithm tables by hand. They were called computers. That's where the name came from. AnyC shows up. AnyC does it forwards, it magnitudes about 5,000 times faster than a human being does it, and we have a computer. And this is a perfect example of a nonlinear system, which is a trajectory with gravity being done by a computer. Why is this not just a straightforward extension of exactly that, like we've been doing for the last 80 years?

I agree. Phenomenon. I fully expect that. But how does that not constitute looking at the universe and figuring out what's what? Oh, for sure, for sure. For specialized subdomains, you absolutely can learn distributions 100%, right? Listen, I've implemented Navier-Stokes, like fluid dynamics, where—and this is a turbulent, chaotic system. So we've known how to, like, implement very complex systems with computers for sure, and especially in very specific domains where we can reduce these things to a few fundamental forces or we can reduce these things to mostly linear systems. Like the previous version of this is just all the computational methods where we would kind of take a problem that we know and we'd actually experimentally determine, like just experimentally, we'd say, okay, this material behaves this way under this pressure and this heat and it has these properties, and we take what we learn experimentally and we'd create these models and they would do pretty good at simulation. And I would say this is a very straightforward extension of that, which is you look at how the world works in a constrained situation and then you can predict what would happen in the constrained situation. But just like simulation, remember, we could do this with simulation. We've been able to do this for a very long time, right? So we could experimentally determine, like, how different materials work, and then you can actually simulate a new system based on those. This is how we do most industrial design today anyways. So my question to you is, how is this fundamentally different than that, or not a natural extension than that, where you're giving it a system, you're learning some fundamental properties, you can do something new, but that doesn't mean that you can disobey the laws of physics in predicting stuff that computers can't predict. It's not obvious to me that it can simulate complex nonlinear systems that are chaotic for long periods of time. I think this is just yet another step on this kind of, like, we have computers simulate physical stuff and we're simulating the next thing with the next tool. Does the parallel make sense? Like if you go to, like, in the 80s and 90s, we would literally empirically test physical matter. We didn't know how the physical matter worked. We just empirically test it. We'd have it has this opacity under this heat, it has this tensile strength. We'd use that to build databases of materials, equations of state, we call them. This is how they interop, right? We'd use those when we're doing simulations and we'd simulate what happens when a car explodes, what happens if an airplane runs into a building, like all of these things. None of those instances worked. They were simulations and they were very accurate and none of them came from, like, first principles. They are all empirical, but that has its limits because it didn't solve climate prediction past 15 days. It didn't allow us to simulate life. And so to me, this is just computers being attached to another domain, which, thank goodness, we've come up with a great tool that's going to give us a little bit more insight, but it's a little bit insight is what it is. By the way, we had the same discussions in the simulation. This is why, by the way, we stopped PlayStations from going to the Middle East was exactly this. So again, I worked—I actually worked in the nuclear weapons program at Livermore, so I was very close to the previous version of these discussions. Like, oh my goodness, if Saddam Hussein gets PlayStations, he's going to be able to simulate nuclear weapons, right? Like, just totally misunderstanding that the ability to simulate something is not some runaway process that's going to allow you to recreate a world or anything like that. It's a very specific tool for a very specific situation. But we were here before, whenever it was, 25 years ago.

The other argument is that they have all panned out exactly as you'd expect, and they're going to continue to. And this has been steady progress for 80 years, and it's been a super useful tool. And every time we unlock something, people are like, oh, poo, that wasn't AGI, so that must not have been AI. And they're, like, focused on the wrong goal, and it's a very unspecified goal. And instead, it's like any other computer science primitive where it keeps solving problems, and we should continue to solve problems with it, which is the way that I would view it. It's interesting because people talk about summers and winters and not panning out, like you just did, but it couldn't be further from the truth. It's been a foundational primitive of computer science that we've been using very effectively, and we continue to use more effectively, and I don't see why that won't just continue to be the case.

I'm so glad you reduced it to this. I think this is great, right? And there's two things. There's this notion that language reasoning is general, which a lot of people believe, but you don't seem to be on that kind of kick. So let's put that one aside, and you're more on the, like, learning properties and simulating properties of the universe side, which is totally fine. So what I would do is I would just bring you back to everything we've learned about simulation, which is even when you know all of the properties of a system and you can simulate, you just don't have the computational capacity to simulate nonlinear systems. We don't have the materials or the energy or anything. It's literally like a compute problem. We have codebases that have been around for 20, 30 years that simulate all sorts of crazy stuff, and yet they've got limited utility for exactly this reason. The universe is just so complex that they're useful for a little bit. And so if it turns out—and I'll give you a very specific one that we can bet—listen, if it turns out that these models somehow change that compute tradeoff where it can simulate nonlinear systems in ways that traditional stuff can't, I'm 100% with you. But that's not what they're doing. What they're doing is they're inferring stuff that we haven't been able to infer by looking at data. That doesn't mean that they can be predictive in a way that kind of disobeys our understanding of compute requirements. And like a Ray-Taylor—let me just give you a specific. So Ray-Taylor instability is basically if you have two liquids that are on top of each other with different densities, and that is one Ray-Taylor unstable system. And if you perturb it, like you get these kind of, like, just amazing kind of chaotic turbulent things that happen. We have been looking at this problem for 30 years, and we have no idea how to actually predict what will happen. We just know roughly what they will look like, but we don't know, like, the specifics. And AI systems are way less efficient than an actual, like, code written for simulation. So to think that it can tackle those types of problems, I just don't see any indication. I mean, I view that we have these kind of modalities of compute that we can throw at problems, right? So the most efficient is basically imperative programming, where you give the computer a set of instructions and exactly the rules, right? So that's just normal programming. And then we have another modality, which is declarative programming. And by the way, the first one's the most efficient, right? It just does the rules. The second one is, like, the end state that you want. I know the end state, but you know it very specifically, and then the computer figures out what to do. This is declarative programming. And so to come to the solution, it has to do all of the logical closure stuff. So that requires more compute, right? And we don't even know how to bound that compute. That's why, like, databases, when you run queries, are not bounded. So that's declarative programming. Now we're in AI, where you don't even know what the end state is. You're just like, I have a whole bunch of data, and I want you to find patterns in that data, right? That requires even more compute. So it's even less efficient. So it's another modality of compute. It's one we've been refining for a long time, but it doesn't change the nature of computers. They're still systems, and they're still computers. They still have the same limitations independent of what distribution that they learn. And so if it turns out that these things can, like, simulate systems for a period of time longer than the normal simulation, then I'm with you. I'm like, this is breaking the laws of physics. But until then, it feels to me like simulation where you just don't know all of the rules, but it's learning some of the rules.

If that action requires interacting with a physical world, it has to simulate the physical world. It just does, right? Like, it has to understand, like, dynamics and ballistics. It has to understand what happens if someone throws a rock at it, or if it's, like, in water, or if the weather's—like, I mean, that's how you navigate the physical world. That's really why we created computers, right? It was because these are very hard things to do. And then if it's not that, if it's not interacting with the physical world, then it is interacting in this kind of language domain that we've created. And I agree, it'd be very good at some subset of those things. There's zero indication it'd be good at new things, and that's what we're actually very good at. And again, without actually having a model for all of these things that we understand, like the distributions we understand, the mechanisms, I feel like we just use words, and the words all make sense, but, like, complex systems, we never know convergence and divergence without actually specifying the system. And I feel like for these conversations, we just don't have a system we talk about. And so it's always we live in the world of, like, rectangles and arrows, and somebody takes a rectangle and they have an arrow that goes back to the rectangle, they're like, ah, we've got a virtuous cycle, without actually specifying that if you have diminishing marginal returns, you don't go anywhere, or you're doing the same stuff, or whatever. And so I think this is incumbent on all of us to actually understand the systems we're working with and then come up with these basic views and properties to make sure at least we understand what the convergence properties are. I'm sorry, that was a very muddled thing to say, but I feel that until we talk about specifics, it's very hard to make concrete statements in this.

The regulation one is sticky for me for two reasons. The first one is we don't even have a definition of AI. And so I think it reduces to regulating software. And then for that, I would say we've been regulating software for a very long time, and there's a broad, robust discourse around that. And I think we should make whatever conversations we have part of that broader discussion.

I mean, cars have a lot of—cars are increasingly software products, and airplanes have a lot of software in them, and medical devices have a lot of software in them. So all these regimes, there's tons of testing, there's tons of verification. They are engineered and demonstrated to a standard. But we don't have those like standards really for these AI systems yet. We do. I don't know what the distinction between AI and software—I really don't. Like, I have seen the definition using these regulations. It's so broad that it really could include all non-trivial software. And I don't say this to be a polemic, and I don't say this to be difficult. I'm saying this very clearly. They literally say a system that can, whatever, navigate and change a virtual or physical system. I mean, these are so broad, right? So we're really talking about software. That's what we're really talking about. We have, what, 70 years of history regulating software in many domains, and I think that regulation is very important. I'm not a libertarian. I'm a lifelong liberal, like a very moderate person. I'm just saying this discourse has been around for a very long time, and we should continue. And if there's an area that software is being pushed, an area that we need to have some sort of protections, we should add them to it, right? But that's a very different statement than saying AI is somehow paradigmatically different. There's just like literally zero indication that it is, and then try to somehow regulate a computer science primitive. It's like regulating a database. Let me give you like a specific example because I—so by the way, we went through this during the rise of the web and the internet. And so a lot of my formal studies were in networking, and so I was actually pretty close to these discussions. And at the time, there was this concern that with the rise of the web and the internet, you have actual paradigmatic shift. It's not just software. It's different. And we actually had examples like the Morris worm had taken out a whole bunch of stuff. We were running critical infrastructure on it, and there was this notion of asymmetry, which actually changed the defense posture of the United States. And the notion of asymmetry is the more that you bought on to the technology, the more vulnerable you were. So as opposed to mutually assured destruction, which was our doctrine before, we now have this new thing, which is like, oh my goodness, we're the most vulnerable because we've been adopted. So you had two very compelling reasons to want to regulate this stuff and be afraid of it and do closed source and whatever it was. You had like examples of taking out critical infrastructure, and you had this change in posture. And yet even that, which we have none of those today with AI, none. We don't have examples of like this horror, like there, like literally it took out computers and hospitals, right? We don't have any examples of paradigm shift. And even then, the conclusion after very robust conversation for a long time is you regulate the applications, you regulate the industries, you don't regulate the math and the computer science primitives. All that I would ask at this AI discussions is either A, demonstrate it's paradigmatically different, and if you don't have that, then B, let's absolutely regulate and have regulations. Let's do it as part of the broad, robust discourse we've had over the last 20 years with really smart people in much more dangerous environments. Again, I think that we put all of our fears in AI, but you realize like we're running hospitals on the internet for the first time that were being taken out and it was impacting patients. That's what we were dealing with. And now we're dealing with theoretical threats on bioweapons that don't even exist yet. I just feel like we've all gotten mad and forgotten where we came from and the discussions that we've had. And all I'm trying to do is saying we've been here before. It was actually much worse here. The conclusions, let's draft on all of that work and shared knowledge.

You know, especially you look at base models, right? Base models are totally unpredictable, and nobody really knows. I think one of the reasons that people are putting so much resource into post-training is to try to get control, and it's only sort of working.

No idea. And I guess having been there, like, during these times, and a lot of computer science is building systems, especially distributed systems, that have emergent behaviors that are non-predictable, that are a tool that you use to solve bigger problems. We've got a lot of approaches to doing that we've developed over time. We will develop new ones. And it's just not clear at all to me. And again, like, the internet is my favorite example to hold up to this. Is literally we changed the social structure of humans. They were totally unpredictable. We were putting critical infrastructure on it. We had examples of, like, emergent stuff that was, like, worms weren't possible before the internet by definition. So we actually had, like, provable, not demonstrated, new attack vectors. And yet the conclusion was, listen, we're going to keep this stuff open. It's very important. We're going to regulate the uses of these things. And this was the primary growth driver of the last 20 years. And so either you thought that decision was wrong, but that's a very different discussion. We could have that discussion. We could relitigate those 10 years of discussion. Or you can say, you know, this is the same thing except for, like, a softer form of it because we have no proof that these things are particularly different. I would consider these widely baseless claims. And so why don't we use the same method that we did before?

Yeah, no, I think this is a great question. So I just have to make the point, like, there's no correlation between danger and compute. It's so silly. It's like this made-up correlation. And so I don't think, like, compute laws make any sense at all. I think it's in some ways it just sounds good, but there's just zero correlation. We've been building systems...

for a very long time that have like social and ethical responsibilities, right? And these are ones with user-generated content and user interactions. And companies, as a result, either through regulatory action, which I fully support, or through self-policing, have identified behaviors the software should not do, and they've built a lot of systems around those. And I would say that should absolutely, absolutely should not change. Like, Roblox should protect minors, social networks should protect people from certain types of content. These are the things that we've evolved as an industry as, you know, somewhat of an ethical basis or a principled basis, and some of it is regulatory and some of it self-enforced, and I think that should absolutely continue.

But realize that while we were doing that, we weren't putting compute limits on databases, and we weren't regulating computer science primitives, and we weren't inhibiting innovation of startups. And that's what we're doing now, and that is a paradigm shift, and that is a doctrine shift, and it's really scary. And I am not a political person. I'm very happy building systems. The only reason I've been so vocal is because I find this so dangerous. And once this is done, I'll be very happy to, like, not say a word about it again.

My rule of thumb for general systems is that unless they were purposely trained for malicious behavior, they're general systems.

That's interesting. Like, a lot of actually computer security started with tools that were like scanners that could be used for malicious purposes. This discussion was enormous back then. So remember, back in the early time of the internet, people would come up with Nmap or any of these scanners that would determine vulnerabilities, and some of them would even actually exploit them. Remember Metasploit? And there was all these discussions, how these people are building these things and they're used for malicious purposes. And you know what? We just decided that it's actually the criminals using them — that's who we want to go after, because a lot of good people use them for positive purposes. And oh, by the way, like, the history of, like, cybersecurity is built on open source and these tools. Like, companies like Tenable came out of these tools. So if history is any indication, you want to regulate or you want to definitely criminalize bad people and bad behavior, but not the tooling, because the tooling is very fundamental to defense. So what I'm saying is even more stringent, it's more regulatory-heavy than what we did with the internet. Literally, you could build exploit tools 20 years ago and release them open source, and you were not liable if somebody used that, and that was purposely built for exploiting. I'm saying, like, if somebody's purposely building a model for something illegal, okay, that's bad, right? So that's even more heavy. Like, I'm more of a regulatory hawk on this case, but I don't think general-purpose models that aren't specifically trained for this stuff, the creator is generally liable. Like, that would just basically say software — you're liable for creating software if somebody used it poorly, which I think this would be the biggest kind of inhibitor to innovation and the biggest chill on software we've ever imposed.

Why not the telephone company? Why not the mining company? Generally, you've got to draw a line somewhere, right? The question is where to draw the line. Yeah, exactly. The illegal behavior. General computer science primitives are very useful for solving problems. They may save the human race. To your point, we're going to go ahead and we're going to solve biology and a bunch of other stuff, and that's just another app. There may be, like, whatever, these things solve grand unified field theory and physics goes away as a discipline, and that's another app. And we're just going to keep solving problems, and that's amazing, right? And so we want the primitives to be primitives. If people do illegal behavior based on them, then 100% they should be criminalized. We don't even make guns illegal. We literally don't even hold gun makers liable.

Yeah, but you're talking about hypothetical threats for a very useful primitive that's shown far more good than bad. I feel like we've all gone a little crazy on this one. By the way, this is a dramatic shift in sentiment, and I think I know where the sources came from, but it's not what people think it is. I really think that we're just a little bit off base, unfortunately.

No, I think that's fair.