The following is a conversation with Andrej Karpathy, previously the director of AI at Tesla and before that at OpenAI and Stanford. He is one of the greatest scientists, engineers, and educators in the history of artificial intelligence. And now a quick 50 second mention of a sponsor. Check them out in the description. It's the best way to support this podcast. We got Eight Sleep for naps, BetterHelp for mental health, Fundrise for real estate investing, and Athletic Greens for nutrition. Choose wisely, my friends. And now on to the full ad reads. As always, no ads in the middle. I try to make this interesting, but if you skip them, please still check out our sponsors. I enjoy their stuff. Maybe you will, too.
This episode is sponsored by Eight Sleep and its new Pod 3 mattress. I'm recording this in a hotel. In fact, given some complexities of my life, this is the middle of the night. 4:00 a.m. I'm sitting in an empty hotel room yelling at a microphone. This, my friends, is my life. I do usually feel good about myself at 4:00 a.m., but not with two cups of coffee in me. And the reason I feel good is because I'm going to go to sleep soon and I've accomplished a lot. This is true today except for the sleep soon part because I think I'm going to an airport at some point soon. It doesn't matter. What matters is I'm not even going to sleep here. And that's great because in a hotel, I don't have an Eight Sleep bed that can cool itself. At home, I do, and that's where I'm headed. I'm headed home. Anyway, check it out and get special savings when you go to eightsleep.com/lex.
This episode is also brought to you by BetterHelp, spelled H-E-L-P Help. I'm a huge fan of talk therapy. I think of podcasting as a kind of talk therapy. So, I'm a huge fan of listening to podcasts. In fact, that's how I think of doing a podcast myself. I just get to have front row seats to a thing I love. And it's actually just the process of talking that reveals something about the mind. I think that's what good talk therapy is. It's guided by a professional therapist, it helps you reveal to yourself something about your mind. Just lay it all out on the table. So, yeah, you should definitely use the best method of talk therapy. The best meaning the most accessible. At least to try it, if not to make it a regular part of your life. That's what BetterHelp does. Check them out at betterhelp.com/lex and save on your first month.
This episode is also brought to you by Fundrise, spelled F-U-N-D-R-I-S-E. It's a platform that allows you to invest in private real estate. We live in hard times, folks, for many different reasons, but one of them is financial. And one way to protect yourself in difficult times is diversify your investments. Private real estate is one of the things, I believe, you should diversify into. And when you do, you should use tools that look like they're made in the 21st century, which a lot of investment, even like online investment websites and services, seem to be designed by the same people that designed the original ATMs. That's not the case with Fundrise. Super easy to use, accessible, over 150,000 investors use it. Their team vets and manages all their real estate projects. You can track your portfolio's performance on their website and see updates as properties across the country are acquired, improved, and operated. Anyway, check out Fundrise. It takes just a few minutes to get started at fundrise.com/lex.
This show is brought to you by Athletic Greens and its AG1 drink, which is an all-in-one daily drink to support better health and peak performance. I have to be honest, I completely forgot to bring Athletic Greens with me as I'm traveling now and I miss it. It's not just good for my nutritional base and needs, it's good for my soul. It's part of the sort of daily habit of life and when you don't have that habit, the routine, stuff is off. So, it's good to just put that into your daily routine to make sure that you're getting the vitamins, the nutrition that you need, no matter the dietary, the workload, the athletic endeavors that you partake in. I don't know, it's kind of incredible and yeah, that's what Athletic Greens is for me. They'll give you one month supply of fish oil when you sign up at athleticgreens.com/lex. This is the Lex Fridman podcast. To support it, please check out our sponsors. And now, dear friends, here's Andrej Karpathy.
What is a neural network and why does it seem to do such a surprisingly good job of learning.
What is a neural network? It's a mathematical abstraction of the brain. I would say that's how it was originally developed. At the end of the day, it's a mathematical expression, and it's a fairly simple mathematical expression when you get down to it. It's basically a sequence of matrix multiplies, which are really dot products mathematically, and some nonlinearities thrown in. And so, it's a very simple mathematical expression, and it's got knobs in it.
Many knobs. And these knobs are loosely related to basically the synapses in your brain. They're trainable, they're modifiable. And so, the idea is like we need to find the setting of the knobs that makes the neural net do whatever you want it to do, like classify images and so on. And so, there's not too much mystery I would say in it. You might think that basically don't want to endow it with too much meaning with respect to the brain and how it works. It's really just a complicated mathematical expression with knobs, and those knobs need a proper setting for it to do something desirable.
Yeah, but poetry is just a collection of letters with spaces, but it can make us feel a certain way. In that same way, when you get a large number of knobs together, whether it's inside the brain or inside a computer, they seem to surprise us with their power.
Yeah, I think that's fair. So, basically I'm underselling it by a lot because you definitely do get very surprising emergent behaviors out of these neural nets when they're large enough and trained on complicated enough problems, like say for example the next word prediction in a massive data set from the internet. And then these neural nets take on pretty surprising magical properties. Yeah, I think it's kind of interesting how much you can get out of even very simple mathematical formalism.
When your brain right now is talking, is it doing next word prediction? Or is it doing something more interesting?
Well, it's definitely some kind of a generative model that's a GPT-like and prompted by you.
So, you're giving me a prompt, and I'm kind of like responding to it in a generative way. And by yourself perhaps a little bit. Like are you adding extra prompts from your own memory inside your head?
Or not? Feels like you're referencing some kind of a declarative structure of like memory and so on. And then you're putting that together with your prompt and giving away some answers.
Like how much of what you just said has been said by you before?
Uh nothing basically, right?
No, but if you actually look at all the words you've ever said in your life and you do a search you'll probably said a lot of the same words in the same order before.
Yeah, could be. I mean, I'm using phrases that are common etc. but I'm remixing it into a pretty sort of unique sentence at the end of the day. But you're right, definitely there's like a ton of remixing.
What? You didn't you... [laughter] It's like Magnus Carlsen said, I'm rated 2900 whatever, which is pretty decent. I think you're talking very, you're not giving enough credit to your own nets here. What's your best intuition about this emergent behavior?
I mean, it's kind of interesting because I'm simultaneously underselling them, but I also feel like there's an element to which I'm over, like it's actually kind of incredible that you can get so much emergent magical behavior out of them despite them being so simple mathematically. So, I think those are kind of like two surprising statements that are kind of just juxtaposed together. And I think basically what it is is we are actually fairly good at optimizing these neural nets. And when you give them a hard enough problem, they are forced to learn very interesting solutions in the optimization. And those solutions basically have these emergent properties that are very interesting.
There's wisdom and knowledge in the knobs.
And so what's this representation that's in the knobs? Does it make sense to you intuitively that a large number of knobs can hold a representation that captures some deep wisdom about the data it has looked at.
It's a lot of knobs. And somehow, you know, so speaking concretely, one of the neural nets that people are very excited about right now are GPTs, which are basically just next word prediction networks. So, you consume a sequence of words from the internet and you try to predict the next word. And once you train these on a large enough data set, you can basically prompt these neural nets in arbitrary ways and you can ask them to solve problems and they will. So, you can just tell them, you can make it look like you're trying to solve some kind of a mathematical problem and they will continue with what they think is the solution based on what they've seen on the internet. And very often, those solutions look very remarkably consistent, look correct potentially.
Do you still think about the brain side of it? So, as neural nets is an abstraction, a mathematical abstraction of the brain, do you still draw wisdom from the biological neural networks? Or even the bigger question, so you're a big fan of biology and biological computation, what impressive thing is biology doing to you that computers are not yet? That gap.
I would say I'm definitely, I'm much more hesitant with the analogies to the brain than I think you would see potentially in the field. And I kind of feel like certainly the way neural networks started is everything stemmed from inspiration of the brain by the brain, but at the end of the day, the artifacts that you get after training, they are arrived at by a very different optimization process than the optimization process that gave rise to the brain. And so, I think I kind of think of it as a very complicated alien artifact. It's something different. Sorry, the neural nets that we're training.
They are complicated alien artifact. I do not make analogies to the brain because I think the optimization process that gave rise to it is very different from the brain. So, there was no multi-agent self-play kind of setup and evolution. It was an optimization that is basically what amounts to a compression objective on a massive amount of data.
Okay, so artificial neural networks are doing compression and biological neural networks are not here to survive. And they're not really doing anything, they're an agent in a multi-agent self-play system that's been running for a very very long time.
That said, evolution has found that it is very useful to predict and have a predictive model in the brain. And so, I think our brain utilizes something that looks like that as a part of it. But, it has a lot more, you know, gadgets and gizmos and value functions and ancient nuclei that are all trying to make us survive and reproduce and everything else.
And the whole thing through embryogenesis is built from a single cell. I mean, it's just the code that's inside the DNA.
And it just builds it up like the entire organism.
And like it does it pretty well.
It should not be possible.
So, there's some learning going on. There's some kind of computation going through that building process. I mean, I don't know where, if you were just to look at the entirety of history of life on Earth, where do you think is the most interesting invention? Is it the origin of life itself? Is it just jumping to eukaryotes? Is it mammals? Is it humans themselves? Homo sapiens?
The origin of intelligence or highly complex intelligence? Or is it all just in continuation of the same kind of process?
Mhm. Certainly, I would say it's an extremely remarkable story that I'm only briefly learning about recently. All the way from, actually, like you almost have to start at the formation of Earth and all of its conditions and the entire solar system and how everything is arranged with Jupiter and Moon and the habitable zone and everything. And then you have an active Earth that's turning over material. And then you start with abiogenesis and everything. And so it's all like a pretty remarkable story. I'm not sure that I can pick like a single unique piece of it that I find most interesting. I guess for me as an artificial intelligence researcher it's probably the last piece. We have lots of animals that, you know, are not building technological society, but we do. And it seems to have happened very quickly. It seems to have happened very recently. And something very interesting happened there that I don't fully understand. I almost understand everything else kind of I think intuitively, but I don't understand exactly that part and how quick it was.
Both explanations would be interesting. One is that this is just a continuation of the same kind of process. There's nothing special about humans. That would be deeply understanding that would be very interesting. That we think of ourselves as special, but it was obvious. All it was already written in the code that you would have greater and greater intelligence emerging. And then the other explanation which is something truly special happened. Something like a rare event. Whether it's like crazy rare event like space odyssey. What would it be? See, if you say like the invention of fire.
Or the as Richard Wrangham says the beta males deciding a clever way to kill the alpha males by collaborating and so just optimizing the collaboration. The really the multi-agent aspect of the multi-agent and that really being constrained on resources and trying to survive the collaboration aspect is what created the complex intelligence. But it seems like it's a natural outgrowth of the evolutionary process. Like what could possibly be a magical thing that happened? Like a rare thing that would say that humans are actually, human-level intelligence is actually a really rare thing in the universe.
Yeah, I'm hesitant to say that it is rare by the way, but it definitely seems like it's kind of like a punctuated equilibrium where you have lots of exploration and then you have certain leaps, sparse leaps in between. So of course like origin of life would be one. You know, DNA, sex, eukaryotic life, the endosymbiosis event where the archaeon ate little bacteria, you know, just the whole thing and then of course emergence of consciousness and so on. So it seems like definitely there are sparse events where massive amount of progress was made. But yeah, it's kind of hard to pick one.
So you don't think humans are unique? Let me ask you, how many intelligent alien civilizations do you think are out there? And is their intelligence different or similar to ours?
Yeah. [laughter] I've been preoccupied with this question quite a bit recently. Basically the Fermi paradox and just thinking through, and the reason actually that I am very interested in the origin of life is fundamentally trying to understand how common it is that there are technological societies out there in space. And the more I study it, the more I think that there should be quite a few, quite a lot.
Why haven't we heard from them? Because I agree with you. It feels like I just don't see why what we did here on Earth is so difficult to do.
Yeah, and especially when you get into the details of it. I used to think origin of life was very, it was this magical rare event. But then you read books like for example, Nick Lane, the vital question, life ascending etc. And he really gets in and he really makes you believe that this is not that rare, basic chemistry. You have an active Earth and you have your alkaline vents, and you have lots of alkaline waters mixing with acidic ocean, and you have your proton gradients, and you have the little porous pockets of these alkaline vents that concentrate chemistry. And basically, as he steps through all of these little pieces, you start to understand that actually, this is not that crazy. You could see this happen on other systems. And he really takes you from just a geology to primitive life, and he makes it feel like it's actually pretty plausible. And also like the origin of life was actually fairly fast after formation of Earth. If I'm remembering correctly, just a few hundred million years or something like that after basically when it was possible, life actually arose. So, that makes me feel like that is not the constraint. That is not the limiting variable, and that life should actually be fairly common. And then, you know, where the drop-offs are is very interesting to think about. I currently think that there's no major drop-offs, basically. And so, there should be quite a lot of life. And basically, where that brings me to then is the only way to reconcile the fact that we haven't found anyone, and so on, is that we just can't see them. We can't observe them.
Just a quick brief comment. Nick Lane and a lot of biologists I talked to, they really seem to think that the jump from bacteria to more complex organisms is the hardest jump.
Mhm. The eukaryotic life, basically.
Which I don't, I get it. They're much more knowledgeable than me about like the intricacies of biology, but that seems like crazy. Because how much, how many single-celled organisms are there? And how much time do you have? Surely it's not that difficult. Like in a billion years is not even that long of a time, really. Just all these bacteria under constrained resources battling it out. I'm sure they can invent more complex things. Like I don't understand, it's like how to move from a hello world program to like invent a function or something like that. I don't...
So I don't, yeah, so I'm with you. I just feel like I don't see any, if the origin of life, that would be my intuition. That's the hardest thing. But if that's not the hardest thing because it happens so quickly, then it's got to be everywhere. And yeah, maybe we're just too dumb to see it.
Well, it's just we don't have really good mechanisms for seeing this life. I mean, so I'm not an expert just to preface this, but just from what I've read about it.
[laughter] Who, I want to meet an expert on alien intelligence and how to communicate.
I'm suspicious of our ability to find these intelligences out there and to find these Earth-like radio waves, for example, are terrible. Their power drops off as basically one over R squared. So I remember reading that our current radio waves would not be the ones that we are broadcasting would not be measurable by our devices today. Only like was it like one tenth of a light year away? Like not even basically tiny distance because you really need like a targeted transmission of massive power directed somewhere for this to be picked up on long distances. And so I just think that our ability to measure is not amazing. I think there's probably other civilizations out there. And then the big question is why don't they build Von Neumann probes and why don't they interstellar travel across the entire galaxy? And my current answer is it's probably interstellar travel is really hard. You have the interstellar medium. If you want to move at closer to speed of light, you're going to be encountering bullets along the way because even like tiny hydrogen atoms and little particles of dust basically have massive kinetic energy at those speeds. And so basically you need some kind of shielding. You have all the cosmic radiation. It's just like brutal out there. It's really hard. And so my thinking is maybe interstellar travel is just extremely hard.
And you have millions of years to build hard? It feels like we're not a billion years away from doing that.
Might be that you have to go very slowly potentially as an example through space.
Right. As opposed to close to the speed of light.
I'm suspicious basically of our ability to measure life and I'm suspicious of the ability to just permeate all of space in the galaxy or across galaxies. And that's the only way that I can currently see a way around it.
Yeah, it's kind of mind-blowing to think that there is trillions of intelligent alien civilizations out there kind of slowly traveling through space.
To meet each other. And some of them meet, some of them go to war, some of them collaborate.
Or they're all just independent. They're all just like little pockets. I don't know.
Well, statistically if there's like trillions of them, surely some of them, some of the pockets are close enough to get.
Some of them happen to be close, yeah.
And close enough to see each other. And then once you see something that is definitely complex life, like if we see something, we're probably going to be severely, like intensely aggressively motivated to figure out what the hell that is and try to meet them.
What would be your first instinct to try to like at a generational level meet them or defend against them or what would be your instinct as a president of the United States and a scientist. I don't know which hat you prefer in this question.
Yeah, I think the question it's really hard. I will say like for example for us we have lots of primitive life forms on Earth next to us. We have all kinds of ants and everything else and we share space with them. And we are hesitant to impact on them and to, we're trying to protect them by default because they are amazing, interesting, dynamical systems that took a long time to evolve, and they are interesting and special, and I don't know that you want to destroy that by default. And so, I like complex dynamical systems that took a lot of time to evolve. I think I'd like to, I like to preserve it if I can afford to. And I'd like to think that the same would be true about the galactic resources, and that they would think that we're kind of incredible, interesting story that took time, it took a few billion years to unravel, and you don't want to just destroy it.
I can see two aliens talking about Earth right now and saying, I'm a big fan of complex dynamical systems, so I think it was a value to preserve these, and we're basically our video game they watch or a TV show that they watch.
Yeah, I think you would need like a very good reason, I think, to destroy it. Like why don't we destroy these ant farms and so on? It's because we're not actually really in direct competition with them right now. We do it accidentally and so on, but there's plenty of resources, and so why would you destroy something that is so interesting and precious?
Well, from a scientific perspective, you might probe it. Yeah. You might interact with it lately.
Exactly. You might want to learn something from it, right?
So, I wonder there could be certain physical phenomena that we think is a physical phenomena, but it's actually interacting with us to like poke the finger and see what happens.
I think it should be very interesting to scientists, other alien scientists, what happened here. And you know, it's what we're seeing today is a snapshot. Basically, it's a result of a huge amount of computation of over like billion years or something like that.
So, it could have been initiated by aliens. This could be a computer running a program. Like when, okay, if you had the power to do this, when you, okay, for sure, at least I would, I would pick Earth-like planet that has the conditions based my understanding of the chemistry prerequisites for life, and I would seed it with life and run it.
Right? Like wouldn't you 100% do that and observe it and then protect. I mean that's not just a hell of a good TV show. It's a good scientific experiment. And that I mean it's physical simulation, right? Maybe maybe the evolution is the most like actually running it is the most efficient way to understand computation or to compute stuff.
Or to understand life or you know what life looks like and what branches it can take.
It does make me kind of feel weird that we're part of a science experiment, but maybe it's everything is a science experiment. Does that change anything for us? If we're a science experiment?
Two descendants of apes talking about being inside of a science experiment.
I'm suspicious of this idea of like a deliberate panspermia as you described it, sir. I don't see a divine intervention in some way in the historical record right now. I do feel like the story in these books like Nick Lane's books and so on sort of makes sense and it makes sense how life arose on Earth uniquely. And yeah, I don't need to reach for more exotic explanations right now.
Sure, but NPCs inside a video game don't observe any divine intervention either. We might just be all NPCs running a kind of code.
Maybe eventually they will. Currently NPCs are really dumb, but once they're running GPTs, maybe they will be like hey this is really suspicious. What the hell?
So you famously tweeted it looks like if you bombard Earth with photons for a while, you can emit a roadster. So if like in Hitchhiker's Guide to the Galaxy, we would summarize the story of Earth. So in that book it's mostly harmless. What do you think is all the possible stories like a paragraph long or sentence long that Earth could be summarized as at once it's done its computation. So like all the possible full, if Earth is a book, right?
Probably there has to be an ending. I mean there's going to be an end to Earth and it could end in all kinds of ways. It could end soon. It could end later. What do you think are the possible stories?
Well, definitely there seems to be, yeah, you're sort of, it's pretty incredible that these self-replicating systems will basically arise from the dynamics.
And then they perpetuate themselves and become more complex and eventually become conscious and build a society. And I kind of feel like in some sense it's kind of like a deterministic wave that kind of just happens on any sufficiently well-arranged system like Earth. And so I kind of feel like there's a certain sense of inevitability in it. And it's really beautiful.
And it ends somehow, right? So it's a chemically diverse environment where complex dynamical systems can evolve and become further and further complex, but then there's a certain, what is it? There's certain terminating conditions.
[laughter] Yeah, I don't know what the terminating conditions are, but definitely there's a trend line of something and we're part of that story and like where does it go? So you know, we're famously described often as a biological bootloaders for AIs and that's because humans, I mean, we're an incredible biological system and we're capable of computation and love and so on. But we're extremely inefficient as well. Like we're talking to each other through audio. It's just kind of embarrassing honestly that we're manipulating like seven symbols serially. We're using vocal cords. It's all happening over like multiple seconds.
It's just kind of embarrassing when you step down to the frequencies at which computers operate or are able to operate on. And so basically, it does seem like synthetic intelligences are kind of like the next stage of development. And I don't know where it leads to. Like at some point, I suspect the universe is some kind of a puzzle. And these synthetic AIs will uncover that puzzle and solve it.
And then what happens after, right? Like what, because if you just like fast forward Earth many billions of years, it's like it's quiet and then it's like the turmoil, you see like city lights and stuff like that. And then what happens at the end? Like is it a, is it a calming? Is it an explosion? Is it like Earth open like a giant, because you said emit roadsters, like will it start emitting like a giant number of satellites?
It's some kind of a crazy explosion and we're living, we're like we're stepping through an explosion and we're like living day-to-day and it doesn't look like it, but it's actually if you I saw a very cool animation of Earth and life on Earth and basically nothing happens for a long time and then the last like 2 seconds like basically cities and everything and.
And the lowest orbit just gets cluttered and just the whole thing happens in the last 2 seconds and you're like, this is exploding. This is state of explosion.
So if you play, yeah, yeah, if you play it at normal speed.
It'll just look like an explosion.
It's a firecracker. We're living in a firecracker.
Where it's going to start emitting all kinds of interesting things.
And then the so explosion doesn't, it might actually look like a little explosion with lights and fire and energy emitted, all that kind of stuff, but when you look inside the details of the explosion, there's actual complexity happening where there's like, yeah, human life or some kind of life.
We hope it's not a destructive firecracker. It's kind of like a constructive.
All right, so given that.
A really interesting to think about like what the puzzle of the universe is. Did the creator of the universe give us a message? Like for example, in the book Contact, Carl Sagan, there's a message for humanity for any civilization in the digits in the expansion of pi in base 11 eventually. Which is kind of interesting thought. Maybe we're supposed to be giving a message to our creator. Maybe we're supposed to somehow create some kind of a quantum mechanical system that alerts them to our intelligent presence here. Because if you think about it from their perspective, it's just say like quantum field theory, massive like cellular automaton like thing. And like how do you even notice that we exist? You might not even be able to pick us up in that simulation. And so how do you prove that you exist? That you're intelligent and that you're part of the universe.
So this is like a Turing test for intelligence from Earth.
Like the creators, I mean maybe this is like trying to complete the next word in the sentence. This is a complicated way of that. Like Earth is just, is basically sending a message back.
Yeah, the puzzle is basically like alerting the creator that we exist.
Or maybe the puzzle is just to break out of the system and just, you know, stick it to the creator in some way. Basically. Like if you're playing a video game, you can somehow find an exploit and find a way to execute on the host machine any arbitrary code. There's some, for example, like I believe someone got Mario, a game of Mario to play Pong just by exploiting it and then creating a, basically writing code in and being able to execute arbitrary code in the game. And so maybe we should be, maybe that's the puzzle is that we should find a way to exploit it. So I think like some of these synthetic AIs will eventually find the universe to be some kind of a puzzle and then solve it in some way. And that's kind of like the end game somehow.
Do you often think about it as a simulation? So as the universe being a kind of computation that might have bugs and exploits.
That our physics is essentially.
It's possible that physics has exploits and we should be trying to find them. Arranging some kind of a crazy quantum mechanical system that somehow gives you buffer overflow, somehow gives you rounding error in the floating point.
Yeah, that's right. And like more and more sophisticated exploits. Those are jokes, but that could be actually very close to the truth.
Some way to extract infinite energy. For example, when you train reinforcement learning agents in physical simulations and you ask them to say run quickly on the flat ground, they'll end up doing all kinds of like weird things in part of that optimization, right? They'll get on their back leg and they'll slide across the floor. And it's because the optimization, the reinforcement learning optimization on that agent has figured out a way to extract infinite energy from the friction forces and basically their poor implementation and they found a way to generate infinite energy and just slide across the surface. And it's not what you expected. It's just a, it's sort of like a perverse solution. And so maybe we can find something like that. Maybe we can be that little dog in this physical simulation.
The cracks or escapes the intended consequences of the physics that the universe came up with. We'll figure out some kind of shortcut to some weirdness. And then man, but see the problem with that weirdness is the first person to discover the weirdness, like sliding on the back legs, that's all we're going to do.
Yeah. [laughter] It very quickly becomes everybody does that thing. So, like the paperclip maximizer is a ridiculous idea, but that very well could be what then we'll just, we'll just all switch to that because it's so fun.
Well, no person will discover it, I think by the way. I think it's going to have to be some kind of a super intelligent AGI of a third generation. Like we're building the first generation AGI. It'd be, you know.
Third generation. Yeah, so the bootloader for an AI, that AI will be a bootloader for another AI.
And then there's no way for us to introspect like what that might even.
I think it's very likely that these things, for example like say you have these AGIs, it's very likely that for example they will be completely inert. I like these kinds of sci-fi books sometimes where these things are just completely inert, they don't interact with anything and I find that kind of beautiful because they probably, they've probably figured out the meta meta game of the universe in some way potentially. They're doing something completely beyond our imagination and they don't interact with simple chemical life forms like why would you do that? So I find those kinds of ideas compelling.
What's their source of fun? What are they doing? What's their source of pleasure in the universe. But inert. So can you define what it means inert? So they escape the interaction of physical reality.
As in they would behave in some very strange way to us because they're beyond, they're playing the meta game and the meta game is probably say like arranging quantum mechanical systems in some very weird ways to extract infinite energy, solve the digital expansion of pi to whatever amount, they will build their own like little fusion reactors or something crazy. Like they're doing something beyond comprehension and not understandable to us and actually brilliant under the hood.
What if quantum mechanics itself is the system and we're just thinking it's physics but we're really parasites on, not parasite, we're not really hurting physics. We're just living on this organism and we're like trying to understand it but really it is an organism and with a deep deep intelligence. Maybe physics itself is the organism that's doing this super interesting thing and we're just like one little thing.
Ant sitting on top of it trying to get energy from it.
Kind of like these particles in the wave that I feel like is mostly deterministic and takes universe from some kind of a big bang to some kind of a super intelligent replicator, some kind of a stable point in the universe given these laws of physics.
You don't think as Einstein said God doesn't play dice? So you think it's mostly deterministic. There's no randomness in the thing?
I think it's deterministic. Oh, there's tons of, well, I'm going to be careful with randomness.
Yeah, I don't like random. I think maybe the laws of physics are deterministic. Yeah, I think they're deterministic.
Got really uncomfortable with the question.
Do you have anxiety about whether the universe is random or not? Is this a source.
There's no randomness in that.
It's you say you like Goodwill Hunting. It's not your fault, Andre. It's not your fault, man. So you don't like randomness.
Yeah, I think it's unsettling. I think it's a deterministic system.
I think that things that look random like, say, the collapse of the wave function, etc., I think they're actually deterministic, just entanglement and so on, and some kind of a multiverse theory, something something.
Okay, so why does it feel like we have a free will? Like if I raise this hand, I chose to do this now.
Mhm. Um, that doesn't feel like a deterministic thing. It feels like I'm making a choice.
Okay. So it's all feelings. It's just feelings.
So when our RL agent is making a choice, it's not really making a choice. The choice is all already there.
Yeah, you're interpreting the choice and you're creating a narrative for having made it.
Yeah, and now we're talking about the narrative. It's very meta. Looking back, what is the most beautiful or surprising idea in deep learning or AI in general that you've come across? You've seen this field explode and grow in interesting ways. Just what cool ideas made you sit back and go hm, big or small?
Well, the one that I've been thinking about recently the most probably is the transformer architecture. So basically, neural networks have had a lot of architectures that were trendy that have come and gone for different sensory modalities like for vision, audio, text. You would process them with different looking neural nets. And recently we've seen this convergence towards one architecture, the transformer. And you can feed it video or you can feed it, you know, images or speech or text and it just gobbles it up and it's kind of like a bit of a general purpose computer that is also trainable and very efficient to run in our hardware. And so, this paper came out in 2016, I want to say.
Um, Attention is All You Need.
Attention is All You Need.
You criticize the paper title in retrospect, that it didn't foresee the bigness of the impact that it was going to have.
Were aware of the impact that that paper would go on to have? Probably they weren't. But I think they were aware of some of the motivations and design decisions behind the transformer and they chose not to, I think, expand on it in that way in the paper. And so, I think they had an idea that there was more than just the surface stuff, just like, oh, we're just doing translation and here's a better architecture. You're not just doing translation. This is like a really cool differentiable, optimizable, efficient computer that you've proposed. And maybe they didn't have all of that foresight, but I think it's really interesting.
Isn't it funny, sorry to interrupt, that that title is memable? That they went for such a profound idea, they went with a — I don't think anyone used that kind of title before, right?
Attention is All You Need. Yeah, it's like a meme or something basically.
Yeah. Isn't that funny? That one — maybe if it was a more serious title it wouldn't have the impact.
Honestly, yeah, there is an element of me that honestly agrees with you and prefers it this way.
If it was too grand it would overpromise and then underdeliver potentially. So you want to just meme your way to greatness.
That should be a t-shirt. So you tweeted the transformer is a magnificent neural network architecture because it is a general purpose differentiable computer. It is simultaneously expressive in the forward pass, optimizable via backpropagation gradient descent, and efficient high parallelism compute graph. Can you discuss some of those details — expressive, optimizable, efficient?
You want to have a general purpose computer that you can train on arbitrary problems. Like say the task of next word prediction or detecting if there's a cat in an image or something like that. And you want to train this computer, so you want to set its weights. And I think there's a number of design criteria that sort of overlap in the transformer simultaneously that made it very successful. And I think the authors were kind of deliberately trying to make this a really powerful architecture. So basically it's very powerful in the forward pass because it's able to express very general computation as something that looks like message passing. You have nodes and they all store vectors. And these nodes get to basically look at each other and each others' vectors and they get to communicate and basically nodes get to broadcast, 'Hey, I'm looking for certain things.' And then other nodes get to broadcast, 'Hey, these are the things I have.' Those are the keys and the values.
So it's not just attention.
Yeah, exactly. Transformer is much more than just the attention component. It's got many architectural pieces that went into it. The residual connection, the weights arranged, there's a multi-layer perceptron in there, the weights are stacked and so on. But basically there's a message passing scheme where nodes get to look at each other, decide what's interesting, and then update each other. And so I think when you get to the details of it, I think it's a very expressive function. So it can express lots of different types of algorithms in the forward pass. Not only that, but the way it's designed with the residual connections, layer normalizations, the softmax attention, and everything, it's also optimizable. This is a really big deal because there's lots of computers that are powerful that you can't optimize or they're not easy to optimize using the techniques that we have, which is backpropagation and gradient descent. These are first-order methods, very simple optimizers, really. And so you also need it to be optimizable. And then lastly, you want it to run efficiently on our hardware. Our hardware is a massive throughput machine, like GPUs. They prefer lots of parallelism. So you don't want to do lots of sequential operations. You want to do a lot of operations serially. And the transformer is designed with that in mind as well. And so it's designed for our hardware and is designed to both be very expressive in a forward pass, but also very optimizable in the backward pass.
And you said that the residual connections support a kind of ability to learn short algorithms fast and first and then gradually extend them longer during training. What's the idea of learning short algorithms?
Right. Think of it as — so basically a transformer is a series of blocks, right? And these blocks have attention and a little multi-layer perceptron. And so you go off into a block and you come back to this residual pathway. And then you go off and you come back. And then you have a number of layers arranged sequentially. And so the way to look at it, I think, is because of the residual pathway, in the backward pass, the gradients sort of flow along it uninterrupted because addition distributes the gradient equally to all of its branches. So the gradient from the supervision at the top just floats directly to the first layer. And all the residual connections are arranged so that in the beginning during initialization, they contribute nothing to the residual pathway. So what it kind of looks like is imagine the Transformer's kind of like a Python function, like a def. And you get to do various kinds of lines of code. Say you have a hundred layers deep Transformer. Typically, they would be much shorter, say 20. So, you have 20 lines of code and you can do something in them. And so, think of during the optimization, basically, what it looks like is first you optimize the first line of code, and then the second line of code can kick in, and the third line of code can kick in. And I kind of feel like because of the residual pathway and the dynamics of the optimization, you can sort of learn a very short algorithm that gets the approximate answer, but then the other layers can sort of kick in and start to create a contribution. And at the end of a year optimizing, you have an algorithm that is 20 lines of code. Except these lines of code are very complex because it's an entire block of a Transformer. You can do a lot in there.
What's really interesting is that this Transformer architecture actually has been remarkably resilient. Basically, the Transformer that came out in 2016 is the Transformer you would use today, except you reshuffle some of the layer norms. The layer normalizations have been reshuffled to a pre-norm formulation. And so, it's been remarkably stable, but there's a lot of bells and whistles that people have attached on it and tried to improve it. I do think that basically it's a big step in simultaneously optimizing for lots of properties of a desirable neural network architecture. And I think people have been trying to change it, but it's proven remarkably resilient. But I do think that there should be even better architectures potentially.
But you admire the resilience here. There's something profound about this architecture that leads to resilience. So maybe everything could be turned into a problem that Transformers can solve.
Currently, definitely looks like the Transformer's taking over AI, and you can feed basically arbitrary problems into it, and it's a general differentiable computer, and it's extremely powerful. And this convergence in AI has been really interesting to watch for me personally.
What else do you think could be discovered here about Transformers? Like, what's a surprising thing? Or is it a stable — I wouldn't say stable place. Is there something interesting we might discover about transformers? Like aha moments, maybe has to do with memory, maybe knowledge representation, that kind of stuff.
Definitely that's what guys today are just pushing, like basically right now that advice is: do not touch the transformer.
Yes. So people are scaling up the data sets, making them much much bigger. They're working on the evaluation, making the evaluation much much bigger. And they're basically keeping the architecture unchanged. And that's how we — that's the last five years of progress in AI kind of.
What do you think about one flavor of it, which is language models? Have you been surprised? Has your sort of imagination been captivated by — you mentioned GPT and all the bigger and bigger and bigger language models? And what are the limits of those models, do you think? So just for the task of natural language.
Basically the way GPT is trained, right, is you just download a massive amount of text data from the internet and you try to predict the next word in the sequence, roughly speaking. You're predicting little word chunks, but roughly speaking that's it. And what's been really interesting to watch is basically it's a language model. Language models have actually existed for a very long time. There's papers on language modeling from 2003, even earlier.
Can you explain that case, what a language model is?
Yeah, so language model just basically the rough idea is just predicting the next word in a sequence. Roughly speaking. So there's a paper from, for example, Bengio and the team from 2003, where for the first time they were using a neural network to take say like three or five words and predict the next word. And they're doing this on much smaller data sets and the neural net is not a transformer, it's a multi-layer perceptron. But it's the first time that a neural network has been applied in that setting. But even before neural networks, there were language models. Except they were using n-gram models. So n-gram models are just count-based models. So if you try to take two words and predict the third one, you'll just count up how many times you've seen any two-word combinations and what came next. And what you predict as coming next is just what you've seen the most of in the training set. And so, language modeling has been around for a long time. Neural networks have done language modeling for a long time.
So, really what's new or interesting or exciting is just realizing that when you scale it up with a powerful enough neural network transformer, you have all these emergent properties where basically what happens is if you have a large enough data set of text, you are in the task of predicting the next word, you are multitasking a huge amount of different kinds of problems. You are multitasking understanding of, you know, chemistry, physics, human nature. Lots of things are sort of clustered in that objective. It's a very simple objective, but actually you have to understand a lot about the world to make that prediction.
You just said the U word — understanding. Are you — in terms of chemistry and physics and so on, what do you feel like it's doing? Is it searching for the right context? What is the actual process happening here?
Yes, so basically it gets a thousand words and it's trying to predict the thousand and first. And in order to do that very, very well over the entire data set available in the internet, you actually have to basically kind of understand the context of what's going on in there.
And it's a sufficiently hard problem that if you have a powerful enough computer, like a transformer, you end up with interesting solutions. And you can ask it to do all kinds of things and it shows a lot of emergent properties like in-context learning. That was the big deal with GPT in the original paper when they published it is that you can just sort of prompt it in various ways and ask it to do various things and it will just kind of complete the sentence, but in the process of just completing the sentence, it's actually solving all kinds of really interesting problems that we care about.
Do you think it's doing something like understanding? Like when we use the word understanding for us humans.
I think it's doing some understanding. In its weights it understands I think a lot about the world and it has to in order to predict the next word in the sequence.
So it's trained on the data from the internet. What do you think about this approach in terms of data sets of using data from the internet? Do you think the internet has enough structured data to teach AI about human civilization?
Yes, I think the internet has a huge amount of data. I'm not sure if it's a complete enough set. I don't know that text is enough for having a sufficiently powerful AGI as an outcome.
Of course there is audio and video and images and all that kind of stuff.
Yes, so text by itself I'm a little bit suspicious about. There's a ton of things we don't put in text in writing just because they're obvious to us about how the world works and the physics of it and that things fall. We don't put that stuff in text because why would you? We share that understanding. And so text is a communication medium between humans and it's not an all-encompassing medium of knowledge about the world. But as you pointed out we do have video and we have images and we have audio. And so I think that that definitely helps a lot, but we haven't trained models sufficiently across all these modalities yet. So I think that's what a lot of people are interested in.
But I wonder what that shared understanding of like what we may call common sense has to be learned, inferred in order to complete the sentence correctly. So maybe the fact that it's implied on the internet the model's going to have to learn that not by reading about it, by inferring it in representation. So like common sense, just like we — I don't think we learn common sense. Nobody tells us explicitly. We just figure it all out by interacting with the world.
And so, here's a model of reading about the way people interact with the world and might have to infer that. I wonder. You briefly worked on a project called the World of Bits. Training an RL system to take actions on the internet. Versus just consuming the internet like we talked about. Do you think there's a future for that kind of system interacting with the internet to help the learning?
Yes, I think that's probably the final frontier for a lot of these models because — so, as you mentioned, when I was at OpenAI, I was working on this project World of Bits. And basically it was the idea of giving neural networks access to a keyboard and a mouse. And the idea —
So, basically you perceive the input of the screen pixels. And basically the state of the computer is sort of visualized for human consumption in images of the web browser and stuff like that. And then you give the neural network the ability to press keyboards and use the mouse. And we're trying to get it to, for example, complete bookings and, you know, interact with the user interfaces. And —
What did you learn from that experience? Like, what was some fun stuff? This is super cool idea.
Yeah, I mean, the step between observer to actor is a super fascinating step.
Well, it's the universal interface in the digital realm, I would say. And there's a universal interface in the physical realm, which in my mind is a humanoid form factor kind of thing. We can later talk about Optimus and so on, but I feel like there's kind of a similar philosophy in some way, where the physical world is designed for the human form. And the digital world is designed for the human form of seeing the screen and using keyboard and mouse. And so, it's the universal interface that can basically command the digital infrastructure we've built up for ourselves. And so, it feels like a very powerful interface to command and to build on top of. Now, to your question as to like what I learned from that, it's interesting because the World of Bits was basically too early, I think, at OpenAI at the time. This is around 2015 or so, and the zeitgeist at that time was very different in AI from the zeitgeist today.
At the time, everyone was super excited about reinforcement learning from scratch. This is the time of the Atari paper where neural networks were playing Atari games and beating humans in some cases, AlphaGo and so on. So, everyone was very excited about training neural networks from scratch using reinforcement learning directly. It turns out that reinforcement learning is an extremely inefficient way of training neural networks because you're taking all these actions and all these observations and you get some sparse rewards once in a while. So, you do all this stuff based on all these inputs and once in a while you're like told you did a good thing. You did a bad thing. And it's just an extremely hard problem, you can't learn from that. You can burn a forest and you can sort of brute force through it, and we saw that I think with Go and Dota and so on, and it does work, but it's extremely inefficient and not how you want to approach problems practically speaking. And so, that's the approach that at the time we also took to World of Bits. We would have an agent initialize randomly, so it would keyboard mash and mouse mash and try to make a booking. And it just revealed the insanity of that approach very quickly, where you have to stumble by the correct booking in order to get a reward of you did it correctly, and you're never going to stumble by it by chance at random.
So, even with a simple web interface, there's too many options.
There's just too many options and it's too sparse of a reward signal, and you're starting from scratch at the time, and so you don't know how to read, you don't understand pictures, images, buttons. You don't understand what it means to like make a booking. But now what's happened is it is time to revisit that, and OpenAI is interested in this. Companies like Adept are interested in this and so on. And the idea is coming back because the interface is very powerful, but now you're not training an agent from scratch. You are taking the GPT as an initialization. So, GPT is pre-trained on all of text. And it understands what's a booking. It understands what's a submit. It understands quite a bit more. And so, it already has those representations. They are very powerful and that makes all of the training significantly more efficient and makes the problem tractable.
Should the interaction be with like the way humans see it with the buttons and the language, or should it be with the HTML, JavaScript, and the CSS? What do you think is the better —
So, today all of this interaction is mostly on the level of HTML, CSS, and so on. That's done because of computational constraints. But, I think ultimately everything is designed for human visual consumption. And so, at the end of the day all the additional information is in the layout of the webpage and what's next to you and what's a red background and all this kind of stuff and what it looks like visually. So, I think that's the final frontier is we're taking in pixels and we're giving out keyboard mouse commands. But, I think it's impractical still today.
Do you worry about bots on the internet? Given these ideas, given how exciting they are, do you worry about bots on Twitter being not the stupid bots that we see now with the crypto bots, but the bots that might be out there actually that we don't see, that they're interacting in interesting ways? So, this kind of system feels like it should be able to pass the I'm not a robot click button, whatever. Do you actually understand how that test works? I don't quite — like there's a checkbox or whatever that you click. It's presumably tracking —
Like mouse movement and the timing and so on. Yeah. So, exactly this kind of system we're talking about should be able to pass that. So, yeah, what do you feel about bots that are language models plus have some interactability and are able to tweet and reply and so on. Do you worry about that world?
Yeah, I think it's always been a bit of an arms race between sort of the attack and the defense. So, the attack will get stronger, but the defense will get stronger as well. Our ability to detect that —
How do you defend? How do you detect? How do you know that your Karpathy account on Twitter is human? How do you approach that? Like if people claim, you know, how would you defend yourself in the court of law that I'm a human?
Yeah, at some point I think it might be — I think society will evolve a little bit, like we might start digitally signing some of our correspondence or, you know, things that we create. Right now it's not necessary, but maybe in the future it might be. I do think that we are going towards a world where we share the digital space with AIs.
Yeah. And they will get much better, and they will share our digital realm, and they'll eventually share our physical realm as well. It's much harder. But that's kind of like the world we're going towards, and most of them will be benign and helpful, and some of them will be malicious, and it's going to be an arms race trying to detect them.
So, I mean, the worst isn't the AIs, the worst is the AIs pretending to be human. My — I don't know if it's always malicious. There's obviously a lot of malicious applications, but —
It could also be — you know, if I was an AI, I would try very hard to pretend to be human because we're in a human world. I wouldn't get any respect as an AI. I want to get some love and respect.
I don't think the problem is intractable. People are thinking about the proof of personhood.
And we might start digitally signing our stuff, and we might all end up having like — yeah, basically some solution for proof of personhood. It doesn't seem to me intractable. It's just something that we haven't had to do until now, but I think once the needle really starts to emerge, which is soon, I think people will think about it much more.
So, but that too will be a race because obviously you can probably spoof or fake the proof of personhood.
So, you have to try to figure out how to —
I mean, it's weird that we have like social security numbers and like passports and stuff. It seems like it's harder to fake stuff in the physical space, but in the digital space it just feels like it's going to be very tricky. Very tricky to — because it seems to be pretty low cost fake stuff. What are you going to put an AI in jail for like trying to use a fake personhood proof? I mean, okay, fine, you'll put a lot of AIs in jail, but there'll be more AIs, exponentially more. The cost of creating a bot is very low. Unless there's some kind of way to track accurately like you're not allowed to create any program without tying yourself to that program. Like any program that runs on the internet, you'll be able to trace every single human programmer that was involved with that program.
Right. Yeah, maybe you have to start declaring when — you know, we have to start drawing those boundaries and keeping track of okay, what are digital entities versus human entities and what is the ownership of human entities and digital entities and something like that. I don't know, but I think I'm optimistic that this is possible. And in some sense we're currently in like the worst time of it because all these bots suddenly have become very capable, but we don't have defenses yet built up as a society. But I think that doesn't seem to me intractable. It's just something that we have to deal with.
It seems weird that the Twitter bot — like really crappy Twitter bots are so numerous. So, I presume that the engineers of Twitter are very good. So, it seems like what I would infer from that — it seems like a hard problem. They're probably catching — all right, if I were to sort of steel man the case, it's a hard problem and there's a huge cost to false positive — to removing a post by somebody that's not a bot. That creates a very bad user experience. So, they're very cautious about removing. So, maybe it's — and maybe the bots are really good at learning what gets removed and not such that they can stay ahead of the removal process very quickly.
My impression of it, honestly, is there's a lot of low-hanging fruit. I mean —
That's my impression of it. It's not subtle.
But, yeah, that's my impression as well, but it feels like maybe you're seeing the tip of the iceberg. Maybe the number of bots is in like the trillions and you have to — just it's a constant assault of bots and you —
Yeah, I don't know. I — you have to steel man the case because the bots I'm seeing are pretty like obvious. I could write a few lines of code that catch these bots.
I mean, definitely there's a lot of low-hanging fruit, but I will say I agree that if you are a sophisticated actor, you could probably create a pretty good bot right now, using tools like GPTs. Because it's a language model, you can generate faces that look really quite good now. And you can do this at scale. And so, I think, yeah, it's quite plausible and it's going to be hard to defend.
There was a Google engineer that claimed that LaMDA was sentient. Do you think there's any inkling of truth to what he felt? And more importantly, to me at least, do you think language models will achieve sentience or the illusion of sentience soonish?
Yeah, to me it's a little bit of a canary in a coal mine kind of moment, honestly, a little bit. Because, so this engineer spoke to like a chatbot at Google and became convinced that this bot is sentient.
Yeah, asked it some existential philosophical questions.
It gave like reasonable answers and looked real and so on. So to me — he wasn't sufficiently trying to stress the system, I think, and exposing the truth of it as it is today.
But I think this will be increasingly harder over time. So yeah, I think more and more people will basically become — yeah, I think more and more there'll be more people like that over time as this gets better.
Like forming an emotional connection to an AI.
Yeah, perfectly plausible in my mind. I think these AIs are actually quite good at human connection, human emotion. A ton of text on the internet is about humans and connection and love and so on. So I think they have a very good understanding in some sense of how people speak to each other about this and they're very capable of creating a lot of that kind of text. There's a lot of like sci-fi from the 50s and 60s that imagined AIs in a very different way. They are calculating cold Vulcan-like machines. That's not what we're getting today. We're getting pretty emotional AIs that are actually very competent and capable of generating, you know, plausible sounding text with respect to all these topics.
See, I'm really hopeful about AI systems that are like companions that help you grow, develop as a human being, help you maximize long-term happiness, but I'm also very worried about AI systems that figure out from the internet that humans get attracted to drama. And so these would just be like shit-talking AIs that just constantly — did you hear — like they'll do gossip. They'll try to plant seeds of suspicion to other humans that you love and trust and just kind of mess with people because that's going to get a lot of attention. So drama, maximize drama on the path to maximizing engagement and us humans will feed into that machine and it'll be a giant drama shitstorm. I'm worried about that. So it's the objective function really defines the way that human civilization progresses with AIs in it.
Yeah. I think right now at least today they are not — it's not correct to really think of them as goal-seeking agents that want to do something. They have no long-term memory or anything. It's literally — a good approximation of it is you get a thousand words and you're trying to predict the thousand and first and then you continue feeding it in. And you are free to prompt it in whatever way you want. So in text. So you say, 'Okay, you are a psychologist and you are very good and you love humans. And here's a conversation between you and another human.' Human colon something. You something. And then it just continues the pattern and suddenly you're having a conversation with the fake psychologist who's like trying to help you. And so it's still kind of like in the realm of a tool. It is — people can prompt it in arbitrary ways and it can create really incredible text. But it doesn't have long-term goals over long period of time. It doesn't try to — sorry, it doesn't look that way right now.
Yeah, but you can do short-term goals that have long-term effects. So if my prompting short-term goal is to get Andrej Karpathy to respond to me on Twitter — like I think AI might — that's the goal, but it might figure out that talking to you it'll be the best in a highly sophisticated interesting way. And then you build up a relationship when you respond once and then it like over time it gets to not be sophisticated and just like just talk. And okay, maybe it won't get to Andrej, but it might get to another celebrity. It might get to other big accounts and then it'll just — so with just that simple goal, get them to respond. Maximize the probability of actual response.
Yeah, I mean you could prompt a powerful model like this with its opinion about how to do any possible thing you're interested in. So they will just — they're kind of on track to become these oracles. I sort of think of it that way. They are oracles. Currently it's just text, but they will have calculators, they will have access to Google search, they will have all kinds of gadgets and gizmos, they will be able to operate the internet and find different information. And yeah, in some sense that's kind of like currently what it looks like in terms of the development.
Do you think it'll be an improvement eventually over what Google is for access to human knowledge? Like it'll be a more effective search engine to access human knowledge?
I think there's definite scope in building a better search engine today and I think Google — they have all the tools, all the people, they have everything they need, they have all the puzzle pieces, they have people training transformers at scale, they have all the data. It's just not obvious if they are capable as an organization to innovate on their search engine right now and if they don't someone else will. There's absolute scope for building a significantly better search engine built on these tools.
It's so interesting. A large company where the search — there's already an infrastructure, it works, ads brings out a lot of money, so where structurally inside a company is there motivation to pivot to say we're going to build a new search engine.
Yeah. That's really hard.
So it's usually going to come from a startup, right?
That would be yeah, or some other a more competent organization. So, currently, for example, maybe Bing has another shot at it, you know.
Here we go. Microsoft Edge, as we're talking offline.
I mean, I definitely — it's really interesting because search engines used to be about, okay, here's some query, here's web pages that look like the stuff that you have. But, you could just directly go to answer and then have supporting evidence. And these models, basically, they've read all the text and they've read all the web pages. And so, sometimes when you see yourself going over the search results and sort of getting like a sense of like the average answer to whatever you're interested in, that just directly comes out. You don't have to do that work. So, they're kind of like — yeah, I think they have a way of distilling all that knowledge into like some level of insight, basically.
Do you think of prompting as a kind of teaching and learning? Like this whole process, like another layer. Because maybe that's what humans are. We already have that background model and then the world is prompting you.
Mhm. Yeah, exactly. I think the way we are programming these computers now, like GPTs, is converging to how you program humans. I mean, how do I program humans? Via prompt. I go to people and I prompt them to do things. I prompt them for information. And so, natural language prompt is how we program humans and we're starting to program computers directly in that interface. It's like pretty remarkable, honestly.
So, you've spoken a lot about the idea of software 2.0. All good ideas become quite clichés so quickly, like the terms. It's kind of hilarious. I think Eminem once said that if he gets annoyed by a song he's written very quickly, that means it's going to be a big hit because it's too catchy. But can you describe this idea and how your thinking about it has evolved over the months and years since you coined it.
Yeah. So I had a blog post on software 2.0, I think several years ago now. And the reason I wrote that post is because I kept — I kind of saw something remarkable happening in like software development and how a lot of code was being transitioned to be written not in sort of like C++ and so on, but it's written in the weights of a neural net. Basically just saying that neural nets are taking over the realm of software. And taking more and more and more tasks. And at the time, I think not many people understood this deeply enough that this is a big deal, this is a big transition. Neural networks were seen as one of multiple classification algorithms you might use for your data set problem on Kaggle. Like this is not that. This is a change in how we program computers.
And I saw neural nets as — this is going to take over. The way we program computers is going to change. It's not going to be people writing software in C++ or something like that and directly programming the software. It's going to be accumulating training sets and data sets and crafting these objectives by which you train these neural nets. And at some point there's going to be a compilation process from the data sets and the objective and the architecture specification into the binary, which is really just the neural net weights and the forward pass of the neural net. And then you can deploy that binary. And so I was talking about that sort of transition. And that's what the post is about. And I saw this sort of play out in a lot of fields. You know, autopilot being one of them, but also just simple image classification. People thought originally, you know, in the 80s and so on that they would write the algorithm for detecting a dog in an image. And they had all these ideas about how the brain does it and first we detect corners and then we detect lines and then we stitch them up and they were like really going at it. They were like thinking about how they're going to write the algorithm. And this is not the way you build it. And there was a smooth transition where okay, first we thought we were going to build everything. Then we were building the features, so like HOG features and things like that that detect these little —
Statistical patterns from image patches, and then there was a little bit of learning on top of it like a support vector machine or binary classifier for cat versus dog in images on top of the features. So we wrote the features, but we trained the last layer, sort of the classifier. And then people are like, 'Actually, let's not even design the features because we can't. Honestly, we're not very good at it.' So let's also learn the features and then you end up with basically a convolutional neural net where you're learning most of it. You're just specifying the architecture and the architecture has tons of filling the blanks, which is all the knobs. And you let the optimization write most of it. And so this transition is happening across the industry everywhere. And suddenly we end up with a ton of code that is written in neural net weights. And I was just pointing out that the analogy is actually pretty strong. And we have a lot of developer environments for software 1.0. Like we have IDEs, how you work with code, how you debug code, how do you run code, how do you maintain code, we have GitHub. So I was trying to make those analogies in the new realm. Like what is the GitHub of software 2.0? Turns out that something that looks like Hugging Face right now. [laughter] And so I think some people took it seriously and built cool companies. And many people originally attacked the post. It actually was not well received when I wrote it. And I think maybe it has something to do with the title, but the post was not well received and I think more people have been coming around to it over time.
Yeah, so you were the director of AI at Tesla where I think this idea was really implemented at scale, which is how you have engineering teams doing software 2.0. So can you sort of linger on that idea of — I think we're in the really early stages of everything you just said, which is like GitHub, IDEs. Like how do we build engineering that works in software 2.0 systems? And the data collection and the data annotation, which is all part of that software 2.0. Like what do you think is the task of programming a software 2.0? Is it debugging in the space of hyperparameters or is it also debugging in the space of data?
Yeah. The way by which you program the computer and influence its algorithm is not by writing the commands yourself. You're changing mostly the data set. You're changing the loss functions of like what the neural net is trying to do, how it's trying to predict things. But yeah, basically the data sets and the architectures of the neural net. And so in the case of the Autopilot, a lot of the data sets had to do with, for example, detection of objects and lane line markings and traffic lights and so on. So you accumulate massive data sets of here's an example, here's the desired label. And then here's roughly what the algorithm should look like, and that's a convolutional neural net. So the specification of the architecture is like a hint as to what the algorithm should roughly look like. And then the fill-in-the-blanks process of optimization is the training process. And then you take your neural net that was trained, it gives all the right answers on your data set, and you deploy it.
So there's in that case, perhaps in all machine learning cases, there's a lot of tasks. So is coming up, formulating a task, like for a multi-headed neural network, is formulating a task part of the programming?
How do you break down a problem into a set of tasks?
Yeah. I mean, on the high level, I would say if you look at the software running in the Autopilot, I gave a number of talks on this topic. I would say originally a lot of it was written in software 1.0. Imagine lots of C++. And then gradually there was a tiny neural net that was for example predicting given a single image is there like a traffic light or not or is there a lane line marking or not. And this neural net didn't have too much to do in the scope of the software. It was making tiny predictions on individual images. And then the rest of the system stitched it up. So okay, we don't have just a single camera, we have eight cameras. We actually have eight cameras over time. And so what do you do with these predictions? How do you put them together? How do you do the fusion of all that information? And how do you act on it? All of that was written by humans in C++. And then we decided, okay, we don't actually want to do all of that fusion in the C++ code because we're actually not good enough to write that algorithm. We want the neural nets to write the algorithm. And we want to port all of that software into the 2.0 stack. And so then we actually had neural nets that now take all the eight camera images simultaneously and make predictions for all of that. And actually they don't make predictions in the space of images, they now make predictions directly in 3D. And actually they don't in three dimensions around the car. And now actually we don't manually fuse the predictions over in 3D over time. We don't trust ourselves to write that tracker. So actually we give the neural net the information over time. So it takes these videos now and makes those predictions. And so you're sort of just like putting more and more power into the neural net, more and more processing. And at the end of it the eventual sort of goal is to have most of the software potentially be in the 2.0 land. Because it works significantly better. Humans are just not very good at writing software basically.
So the prediction is happening in this like 4D land, with three-dimensional world over time. How do you do annotation in that world? What have you — so the data annotation whether it's self-supervised or manual by humans is a big part of this software 2.0 world.
Right. I would say by far in the industry, if you're like talking about the industry and what is the technology of what we have available, everything is supervised learning. So you need data sets of input, desired output, and you need lots of it. And there are three properties of it that you need. You need it to be very large. You need it to be accurate, no mistakes, and you need it to be diverse. You don't want to just have a lot of correct examples of one thing. You need to really cover the space of possibility as much as you can. And the more you can cover the space of possible inputs, the better the algorithm will work at the end. Now, once you have really good data sets that you're collecting, curating, and cleaning, you can train your neural net on top of that. So a lot of the work goes into cleaning those data sets. Now, as you pointed out, it could be — the question is, how do you achieve a ton of — if you want to basically predict in 3D, you need data in 3D to back that up. So in this video, we have eight videos coming from all the cameras of the system. And this is what they saw, and this is the truth of what actually was around. There was this car, there was this car, this car. These are the lane line markings. This is the geometry of the road. There's a traffic light in this three-dimensional position. You need the ground truth. And so the big question that the team was solving, of course, is how do you arrive at that ground truth? Because once you have a million of it and it's large, clean, and diverse, then training a neural net on it works extremely well, and you can ship that into the car. And so there's many mechanisms by which we collected that train data. You can always go for a human annotation. You can go for simulation as a source of ground truth. You can also go for what we call the offline tracker that we spoke about at the AI Day and so on, which is basically an automatic reconstruction process for taking those videos and recovering the three-dimensional sort of reality of what was around that car. So basically think of doing like a three-dimensional reconstruction as an offline thing, and then understanding that, okay, there's this 10 seconds of video, this is what we saw, and therefore here's all the lane lines, cars, and so on. And then once you have that annotation, you can train your own nets to imitate it.
And how difficult is the reconstruction, the 3D reconstruction?
It's difficult, but it can be done.
So there's overlap between the cameras and you do the reconstruction and perhaps there's any inaccuracy, so that's caught in the annotation step.
Yes, the nice thing about the annotation is that it is fully offline. You have infinite time. You have a chunk of 1 minute and you're trying to just offline in a supercomputer somewhere figure out where were the positions of all the cars, all the people. And you have your full 1 minute of video from all the angles. And you can run all the neural nets you want and they can be very efficient massive neural nets. There can be neural nets that can't even run in the car later at test time. So they can be even more powerful neural nets than what you can eventually deploy. So you can do anything you want, three-dimensional reconstruction, neural nets, anything you want just to recover that truth. And then you supervise that truth.
What have you learned — you said no mistakes — about humans doing annotation? Because I assume humans are — there's like a range of things they're good at in terms of clicking stuff on screen. Isn't that — how interesting is that you have a problem of designing an annotator where humans are accurate, enjoy it — like what are the metrics, are they efficient, are they productive, all that kind of stuff.
Yeah, so I grew the annotation team at Tesla from basically zero to 1,000 while I was there. That was really interesting. You know, my background as a PhD student researcher, so growing that kind of organization was pretty crazy. But yeah, I think it's extremely interesting and part of the design process very much behind the Autopilot as to where you use humans. Humans are very good at certain kinds of annotations. They're very good for example at two-dimensional annotations of images. They're not good at annotating cars over time in three-dimensional space. Very very hard. And so that's why we were very careful to design the tasks that are easy to do for humans versus things that should be left to the offline tracker. Like maybe the computer will do all the triangulation and 3D reconstruction, but the human will say exactly these pixels of the image are car. Exactly these pixels are human. And so co-designing the data annotation pipeline was very much bread and butter of what I was doing daily.
Do you think there's still a lot of open problems in that space? Just in general annotation where the stuff that machines are good at, machines do, and the humans do what they're good at, and there's maybe some iterative process.
Right. I think to a very large extent we went through a number of iterations and we learned a ton about how to create these data sets. I'm not seeing big open problems. Like originally when I joined I was like really not sure how this would turn out. But by the time I left I was much more secure in actually we sort of understand the philosophy of how to create these data sets and I was pretty comfortable with where that was at the time.
So what are strengths and limitations of cameras for the driving task in your understanding when you formulate the driving task as a vision task with eight cameras? You've seen the entire, you know, most of the history of the computer vision field when it has to do with neural networks. Just if you step back, what are the strengths and limitations of pixels? Of using pixels to drive?
Yeah, pixels I think are a beautiful sensor, I would say. The thing is cameras are very very cheap and they provide a ton of information, ton of bits. So it's an extremely cheap sensor for a ton of bits. And each one of these bits is a constraint on the state of the world. And so you get lots of megapixel images very cheap and it just gives you all these constraints for understanding what's actually out there in the world. So vision is probably the highest bandwidth sensor. It's a very high bandwidth sensor. And I love the pixels. It is a constraint on the world. It's this highly complex, high bandwidth constraint on the state of the world. That's fascinating. Not just that, but again this real importance of — it's the sensor that humans use. Therefore, everything is designed for that sensor. The text, the writing, the flashing signs, everything is designed for vision. And so you just find it everywhere. And so that's why that is the interface you want to be in. Talking again about these universal interfaces. And that's where we actually want to measure the world as well. And then develop software for that sensor.
But there's other constraints on the state of the world that humans use to understand the world. I mean, vision ultimately is the main one, but we're like we're referencing our understanding of human behavior and some common sense physics that could be inferred from vision from a perception perspective, but it feels like we're using some kind of reasoning to predict the world not just the pixels, for how the world evolves over time, etc. So it's not just about the likelihood term coming up from the data itself telling you about what you are observing, but also the prior term of like what are the likely things to see and how do they likely move and so on.
And the question is how complex is the range of possibilities that might happen in the driving task.
Right. That's still — is that — do you still have an open problem of how difficult is driving like philosophically speaking? Like do you — all the time you worked on driving, do you understand how hard driving is?
Yeah, driving is really hard. Because it has to do with the predictions of all these other agents and the theory of mind and you know what they're going to do and are they looking at you or where are they looking, where are they thinking? There's a lot that goes there at the full tail of, you know, the expansion of the nines that we have to be comfortable with it eventually. The final problems are of that form. I don't think those are the problems that are very common. I think eventually they're important but it's like really in the tail end.
In the tail end, the rare edge cases. What from the vision perspective — what are the toughest parts of the vision problem of driving?
Well, basically the sensor is extremely powerful but you still need to process that information. And so going from brightnesses of these special values to hey, here's the three-dimensional world is extremely hard and that's what the neural networks are fundamentally doing. And so the difficulty really is in just doing an extremely good job of engineering the entire pipeline. The entire data engine, having the capacity to train these neural nets, having the ability to evaluate the system and iterate on it. So I would say just doing this in production at scale is like the hard part. It's an execution problem.
So the data engine but also the sort of deployment of the system such that it has low latency performance. So it has to do all these steps.
Yeah, for the neural nets specifically just making sure everything fits into the chip on the car. And you have a finite budget of flops that you can perform and memory bandwidth and other constraints and you have to make sure it flies and you can squeeze in as much compute as you can into that tiny —
What have you learned from that process? Because maybe that's one of the bigger like new things coming from a research background where there's a system that has to run under heavily constrained resources. Has to run really fast. What kind of insights have you learned from that?
Yeah, I'm not sure if there's too many insights. You're trying to create a neural net that will fit in what you have available and you're always trying to optimize it. And we talked a lot about it on the AI Day and basically the triple backflips that the team is doing to make sure it all fits and utilizes the engine. So I think it's extremely good engineering. And then there's also all kinds of little insights peppered in on how to do it properly.
Let's actually zoom out because I don't think we talked about the data engine. The entirety of the layout of this idea that I think is just beautiful with humans in the loop. Can you describe the data engine?
Yeah, the data engine is what I call the almost biological feeling like process by which you perfect the training sets for these neural networks. So because most of the programming now is in the level of these data sets and making sure they're large, diverse, and clean. Basically you have a data set that you think is good. You train your neural net. You deploy it. And then you observe how well it's performing. And you're trying to always increase the quality of your data set. You're trying to catch scenarios basically that are basically rare. And it is in these scenarios that the neural nets will typically struggle in because they weren't told what to do in those rare cases in the data set. But now you can close the loop because if you can now collect all those at scale, you can then feed them back into the reconstruction process I described and reconstruct the truth in those cases and add it to the data set. And so the whole thing ends up being like a staircase of improvement of perfecting your training set. And you have to go through the deployments so that you can mine the parts that are not yet represented well in the data set. So your data set is basically imperfect. It needs to be diverse. It has pockets that are missing. And you need to pad out the pockets. You can sort of think of it that way.
What role do humans play in this? So what's the — this biological system like a human body is made up of cells? What role — like how do you optimize the human system? The multiple engineers collaborating, figuring out what to focus on, what to contribute, which task to optimize in this neural network. Who's in charge of figuring out which task needs more data? Can you speak to the hyperparameters of the human system?
Right, it really just comes down to like extremely good execution from an engineering team who knows what they're doing. They understand intuitively the philosophical insights underlying the data engine and the process by which the system improves. And how to again delegate the strategy of the data collection and how that works. And then just making sure it's all extremely well executed. And that's where most of the work is — it's not even the philosophizing or the research or the ideas of it, it's just extremely good execution. It's so hard when you're dealing with data at that scale.
So your role in the data engine executing well on it is difficult and extremely important. Is there a priority of like a vision board of saying like we really need to get better at stoplights. Like the prioritization of tasks. Is that essentially — and that comes from the data?
That comes to a very large extent to what we are trying to achieve in the product roadmap, what we're trying to — the release we're trying to get out, and the feedback from the QA team where the system is struggling or not. The things we're trying to improve.
And the QA team gives some signal, some information in aggregate about the performance of the system in various conditions.
And then of course of us drive it and we can also see it. It's really nice to work with a system that you can also experience yourself. And you know, it drives you home.
Is there some insight you can draw from your individual experience that you just can't quite get from an aggregate statistical analysis of the data?
It's so weird, right? It's not scientific in a sense because you're just one anecdotal sample.
Yeah, I think there's a ton of — it's a source of truth. It's your interaction with the system. Yeah. And you can see it, you can play with it, you can perturb it, you can get a sense of it, you have an intuition for it. I think numbers just like have a way of — numbers and plots and graphs are, you know, much harder. It hides a lot.
It's like if you train a language model it's a really powerful way is by you interacting with it. Try to build up an intuition.
Yeah, 100%. Yeah. I think like Elon also like he always wanted to drive this system himself. He drives a lot and I want to say almost daily. So he also sees this as a source of truth. You driving the system and it performing and yeah.
So what do you think? Tough questions here. So Tesla last year removed radar from the sensor suite and now just announced that it's going to remove ultrasonic sensors relying solely on vision. So camera only. Does that make the perception problem harder or easier?
I would almost reframe the question in some way. So the thing is basically you would think that additional sensors —
By the way, can I just interrupt? I wonder if the language model will ever do that if you prompt it. Let me reframe your question. [laughter] That would be epic. That's the wrong prompt. Sorry.
Yeah, so it's like a little bit of a wrong question because basically you would think that these sensors are an asset to you. But if you fully consider the entire product in its entirety, these sensors are actually potentially a liability. Because these sensors aren't free. They don't just appear on your car. Suddenly you need have an entire supply chain. You have people procuring it. There can be problems with them. They may need replacement. They are part of the manufacturing process. They can hold back the line in production. You need to source them, you need to maintain them, you have to have teams that write the firmware, all of it. And then you also have to incorporate them, fuse them into the system in some way. And so it actually like bloats a lot of it. And I think Elon is really good at simplify, simplify, best part is no part. And he always tries to throw away things that are not essential because he understands the entropy in organizations. And I think in this case the cost is high and you're not potentially seeing it if you're just a computer vision engineer and I'm just trying to improve my network and you know is it more useful or less useful, how useful is it? And the thing is if once you consider the full cost of a sensor it actually is potentially a liability and you need to be really sure of that it's giving you extremely useful information. In this case we looked at using it or not using it and the delta was not massive and so it's not useful.
Is it also bloat in the data engine like having more sensors is a distraction?
100%. And these sensors, you know, they can change over time. For example you can have one type of say radar, you can have another type of radar, they change over time. Now you suddenly need to worry about it. Now you suddenly have a column in your SQLite telling you oh what sensor type was it and they all have different distributions and then they just — they contribute noise and entropy into everything and they bloat stuff. And also organizationally it's been really fascinating to me that it can be very distracting. If all you want to get to work is vision, all the resources are on it and you're building out a data engine and you're actually making forward progress because that is the sensor with the most bandwidth, the most constraints in the world and you're investing fully into that and you can make that extremely good. If you have only a finite amount of sort of spend of focus across different facets of the system.
And this kind of reminds me of Rich Sutton's bitter lesson. It just seems like simplifying the system in the long run and of course you don't know what the long run — it seems to be always the right solution.
In that case it was for RL but it seems to apply generally across all systems that do computation. So what do you think about the lidar as a crutch debate? The battle between point clouds and pixels.
Yeah, I think this debate is always like slightly confusing to me because it seems like the actual debate should be about like, do you have the fleet or not? That's like the really important thing about whether you can achieve a really good functioning of an AI system at the scale. So, data collection systems. Yeah, do you have the fleet or not is significantly more important than whether you have lidar or not. It's just another sensor. And yeah, I think similar to the radar discussion, basically I don't think it offers extra information. It's extremely costly. It has all kinds of problems. You have to worry about it. You have to calibrate it, etc. It creates bloat and entropy. You have to be really sure that you need this sensor. In this case, I basically don't think you need it. And I think honestly, I will make a stronger statement. I think some of the other companies that are using it are probably going to drop it.
Yeah, so you have to consider the sensor in the full — in considering, can you build a big fleet that collects a lot of data and can you integrate that sensor with that data and that sensor into a data engine that's able to quickly find different parts of the data that then continuously improves whatever the model that you're using.
Yeah, another way to look at it is like vision is necessary in a sense that the world is designed for human visual consumption, so you need vision. It's necessary. And then also it is sufficient because it has all the information that you need for driving and humans obviously use vision to drive. So it's both necessary and sufficient, so you want to focus resources and you have to be really sure if you're going to bring in other sensors. You could add sensors to infinity. At some point, you need to draw the line. And I think in this case, you have to really consider the full cost of any one sensor that you're adopting and do you really need it? And I think the answer in this case is no.
So what do you think about the idea that other companies are forming high resolution maps and constraining heavily the geographic regions in which they operate. Is that approach in your view not going to scale over time to the entirety of the United States?
I think as you mentioned like they pre-map all the environments and they need to refresh the map and they have a perfect centimeter level accuracy map of everywhere they're going to drive. It's crazy. When we're talking about autonomy actually changing the world, we're talking about a deployment on a global scale of autonomous systems for transportation. And if you need to maintain a centimeter accurate map for Earth or like for many cities and keep them updated, it's a huge dependency that you're taking on. Huge dependency. It's a massive massive dependency and now you need to ask yourself, do you really need it? And humans don't need it. Right? So it's very useful to have a low level map of like the connectivity of your road. You know that there's a fork coming up when you drive an environment, you sort of have that high level understanding. It's like a small Google Map and Tesla uses Google Map like similar kind of resolution information in the system but it will not pre-map environments to centimeter level accuracy. It's a crutch, it's a distraction, it costs entropy and it diffuses the team, it dilutes the team and you're not focusing on what's actually necessary which is the computer vision problem.
What did you learn about machine learning, about engineering, about life, about yourself as one human being from working with Elon Musk?
I think the most I've learned is about how to sort of run organizations efficiently and how to create efficient organizations and how to fight entropy in an organization.
So human engineering. In the fight against entropy. What does entropy in an organization look like exactly?
It's process. It's process and inefficiencies. Yeah, meetings. He hates meetings. He keeps telling people to skip meetings if they're not useful. He basically runs the world's biggest startups, I would say. Tesla, SpaceX are the world's biggest startups. Tesla actually is multiple startups. I think it's better to look at it that way. And so I think he's extremely good at that. And he has a very good intuition for streamlining processes, making everything efficient. Best part is no part, simplifying, focusing, and just kind of removing barriers, moving very quickly, making big moves. All this is very startup-y sort of seeming things, but at scale.
So strong drive to simplify. From your perspective, I mean that also probably applies to just designing systems in machine learning and otherwise. Like simplify, simplify. What do you think is the secret to maintaining the startup culture in a company that grows? Is there — can you introspect that?
I do think he needs someone in a powerful position with a big hammer like Elon who's like the cheerleader for that idea and ruthlessly pursues it. If no one has a big enough hammer, everything turns into committees, democracy within the company, process, talking to stakeholders, decision-making, just everything just crumbles. If you have a big person who is also really smart and has a big hammer, things move quickly.
So you said your favorite scene in Interstellar is the intense docking scene with the AI and Cooper talking saying 'Cooper, what are you doing?' 'Docking.' 'It's not possible.' 'No, it's necessary.' Such a good line. By the way, just so many questions there. Why in the AI in that scene presumably is supposed to be able to compute a lot more than the human, it's saying it's not optimal where the human — I mean that's a movie but shouldn't AI know much better than the human? Anyway, what do you think is the value of setting seemingly impossible goals? So like our initial intuition which seems like something that you have taken on that Elon espouses, where the initial intuition of the community might say this is very difficult and then you take it on anyway with a crazy deadline. Just from a human engineering perspective, have you seen the value of that?
I'm going to say that setting impossible goals exactly is not a good idea but I think setting very ambitious goals is a good idea. I think there's what I call sublinear scaling of difficulty which means that 10x problems are not 10x hard. Usually a 10x harder problem is like two or three x harder to execute on. Because if you want to improve a system by 10% it costs some amount of work and if you want to 10x improve the system it doesn't cost, you know, 100x amount of work. And it's because you fundamentally change the approach and if you start with that constraint then some approaches are obviously dumb and not going to work and it forces you to re-evaluate. And I think it's a very interesting way of approaching problem solving.
It requires a weird kind of thinking. It's just going back to your PhD days. It's like how do you think which ideas in the machine learning community are solvable?
It requires — what is that? I mean there's a cliche of first principles thinking but like it requires to basically ignore what the community is saying because the community doesn't — a community in science usually draws lines of what is and isn't possible. And like it's very hard to break out of that without going crazy.
Yep. I mean I think a good example here is, you know, the deep learning revolution in some sense because you could be in computer vision at that time when during the deep learning sort of revolution of 2012 and so on. You could be improving a computer vision stack by 10% or you can just be saying, 'Actually, all this is useless. And how do I do 10x better computer vision?' Well, it's not probably by tuning a HOG feature detector. I need a different approach. I need something that is scalable. Going back to Richard Sutton's and understanding the philosophy of the bitter lesson. And then being like, 'Actually, I need a much more scalable system like a neural network that in principle works.' And then having some deep believers that can actually execute on that mission and make it work. So, that's the 10x solution.
What do you think is the timeline to solve the problem of autonomous driving? This still in part open question.
Yeah. I think the tough thing with timelines of self-driving, obviously, is that no one has created self-driving. Yeah. So it's not like 'What do you think is the timeline to build this bridge?' Well, we've built millions of bridges before. Here's how long that takes. No one has built autonomy. It's not obvious. Some parts turn out to be much easier than others. So it's really hard to forecast. You do your best based on trend lines and so on and based on intuition, but that's why fundamentally it's just really hard to forecast this.
So even still like being inside of it, it's hard to do.
Yes. Some things turn out to be much harder and some things turn out to be much easier.
Do you try to avoid making forecasts? Because like Elon doesn't avoid them, right? And heads of car companies in the past have not avoided it either. Ford and other places have made predictions that we were going to solve level four driving by 2020, 2021, whatever. And now they're all kind of backtracking that prediction. Are you as an AI person, do you feel yourself privately make predictions or do they get in the way of your actual ability to think about a thing?
Yeah, I would say like what's easy to say is that this problem is tractable and that's an easy prediction to make. It's tractable. It's going to work. Yes, it's just really hard. Some things turn out to be harder and some things turn out to be easier. But it definitely feels tractable and it feels like at least the team at Tesla, which is what I saw internally, is definitely on track to that.
How do you form a strong representation that allows you to make a prediction about tractability? So like you are the leader of a lot of humans. You have to kind of say this is actually possible. Like how do you build up that intuition? It doesn't have to be even driving. It could be other tasks. What difficult task did you work on in your life? Classification, achieving certain — ImageNet, certain level of superhuman level performance.
Yeah, expert intuition. It's just intuition. It's belief.
So just like thinking about it long enough, like studying, looking at sample data, like you said, driving. My intuition is really flawed on this. Like I don't have a good intuition about tractability. It could be anything. It could be solvable. Like the driving task could be simplified into something quite trivial. Like the solution to the problem would be quite trivial. And at scale, more and more cars driving perfectly might make the problem much easier. Yes, the more cars you have driving and like people learn how to drive correctly — not correctly, but in a way that's more optimal for a heterogeneous system of autonomous and semi-autonomous and manually driven cars, that could change stuff. Then again, also I've spent a ridiculous number of hours just staring at pedestrians crossing streets thinking about humans and it feels like the way we use our eye contact, it sends really strong signals and there's certain quirks and edge cases of behavior. And of course a lot of the fatalities that happen have to do with drunk driving and both on the pedestrian side and the driver side so there's that problem of driving at night and all that kind of stuff.
So I wonder, you know, it's like the space of possible solutions to autonomous driving includes so many human factor issues that it's almost impossible to predict. There could be super clean, nice solutions.
Yeah. I would say definitely like to use a game analogy, there's some fog of war about you. Definitely also see the frontier of improvement.
And you can measure historically how much you've made progress. And I think for example, at least what I've seen in roughly five years at Tesla, when I joined, it barely kept lane on the highway. I think going up from Palo Alto to SF was like three or four interventions. Anytime the road would do anything geometrically or turn too much, it would just like not work. And so going from that to like a pretty competent system in five years and seeing what happens also under the hood and what the scale at which the team is operating now with respect to data and compute and everything else is just massive progress. So it's climbing a mountain in fog but you're making a lot of progress.
Yeah, the fundamental components of solving the problem seem to be there from the data engine to the compute to the computer on the car to the computer for the training, all that kind of stuff. So you've done — over the years you've been at Tesla, you've done a lot of amazing breakthrough ideas and engineering.
Fog. You're making progress and you see what the next directions are and you're looking at some of the remaining challenges and they're not like — they're not perturbing you and they're not changing your philosophy and you're not contorting yourself. You're like, 'Actually, these are the things that we still need to do.'
All of it. From the data engine to the human side, all of it.
Can you speak to why you chose to leave Tesla?
Basically, as I just described that ran I think over time, during those five years, I've kind of gotten myself into a little bit of a managerial position. Most of my days were, you know, meetings and growing the organization and making decisions about sort of high-level strategic decisions about the team and what it should be working on and so on. And it's kind of like a corporate executive role and I can do it. I think I'm okay at it, but it's not like fundamentally what I enjoy. And so, I think when I joined there was no computer vision team because Tesla was just going from the transition of using Mobileye, a third-party vendor for all of its computer vision, to having to build its computer vision system. So, when I showed up, there were two people training deep neural networks.
And they were training them at a computer at their at their legs like down there was some kind of basic classification task. Yeah, and so I kind of like grew that into what I think is a fairly respectable deep learning team, a massive compute cluster, a very good data annotation organization, and I was very happy with where that was. It became quite autonomous and so I kind of stepped away and I'm very excited to do much more technical things again. Yeah, and kind of like we focus on AGI.
What was this soul-searching like? Cuz you took a little time off, I think. Like what um how many mushrooms did you take? No, I'm just kidding. Uh I mean, what what was going through your mind? The human lifetime is finite. You did a few incredible things. You're one of the best teachers of AI in the world. You're one of the best and I don't mean that, I mean that in the best possible way. You're one of the best tinkerers in the AI world, meaning like understanding the fundamental fundamentals of how something works by building it from scratch and playing with the with the basic intuitions. It's like Einstein, Feynman, were all really good at this kind of stuff. Like small example of a thing to to play with it, to try to understand it. Uh so that and obviously now with with Tesla you helped build a team of machine learning um like engineers and a system that actually accomplishes something in the real world. So given all that, like what what was the source searching like?
Well, it was hard because obviously I love the company a lot and I love I love Elon and I love Tesla. It was hard to leave. I love the team, basically. But yeah, I think actually I would be potentially like interested in revisiting it. Maybe coming back at some point. Working on Optimus, working on AGI at Tesla. I think Tesla's going to do incredible things. It's basically like it's a massive large-scale robotics kind of company with a ton of in-house talent. They're doing really incredible things. And I think humanoid robots are going to be amazing. I think autonomous transportation is going to be amazing. All this is happening at Tesla. So I think it's just a really amazing organization. So being part of it and helping it along I think was very basically I enjoyed that a lot. Yeah, it was basically difficult for those reasons because I love the company. But you know, I'm happy to potentially at some point come back for act two. But I felt like at this stage I built the team, it felt autonomous, and I became a manager, and I wanted to do a lot more technical stuff. I wanted to learn stuff. I wanted to teach stuff. And I just kind of felt like it was a good time for a change of pace a little bit.
What do you think is the best movie sequel of all time, speaking of part two? Cuz like cuz most of them suck.
Movie sequels, yeah. And you tweeted about movies, so just in a tiny tangent. Is there a What's your What what's like a favorite movie sequel?
Are you a fan of Godfather cuz you didn't even tweet or mention Godfather?
Yeah, I don't love that movie. I know it has a huge following.
We're going to edit out the hate towards The Godfather. How dare you disrespect.
Make a strong statement. I don't know why. I don't know why, but I basically don't like any movie before 1995. Something like that.
Didn't you mention Terminator 2?
Okay, okay, that's like a Terminator 2 was a little bit later, 1990.
No, I think Terminator 2 was in the '80s.
And I like Terminator 1 as well. So, okay, so like a few exceptions, but by and large, for some reason, I don't like movies before 1995 or something. They feel very slow. The camera is like zoomed out. It's boring. It's kind of naive. It's kind of weird.
And also, Terminator was very much ahead of its time.
Yes. And The Godfather, there's like no AGI.
That's I mean, but you have Good Will Hunting was one of the movies you mentioned, and that doesn't have any AGI, either. I guess that's mathematics.
Yeah. I guess occasionally I do enjoy movies that don't feature.
Or like Anchorman, that has no That's That doesn't.
I don't understand I'm speaking of AGI, cuz I don't understand why Will Ferrell is so funny.
It doesn't make sense. It doesn't compute. There's just something about him. And he's a singular human, cuz you don't get that many comedies these days, and I wonder if it has to do about the culture or the like the the machine of Hollywood, or does it have to do with just we got lucky with certain people in comedy that came together, cuz he is a singular human.
That was a ridiculous tangent. I apologize. But, you mentioned humanoid robots. So, what do you think about Optimus, about Tesla Bot? Do you think we'll have robots in the factory and and in the home in 10, 20, 30, 40, 50 years?
Yeah. I think it's very hard project. I think it's going to take a while. But, who else is going to build humanoid robots at scale?
And I think it is a very good form factor to go after, because like I mentioned, the the world is designed for humanoid form factor. These things would be able to operate our machines. They would be able to sit down in chairs, drive potentially even drive cars. Basically, the world is designed for humans. That's the form factor you want to invest into and make work over time. I think, you know, there's another school of thought, which is, okay, pick a problem and design a robot to it. But actually designing a robot and getting a whole data engine and everything behind it to work is actually incredibly hard problem. So, it makes sense to go after general interfaces that okay, they are not perfect for any one given task, but they actually have the generality of just with a prompt, with English, able to do something across. And so, I think it makes a lot of sense to go after a general interface in the physical world. And I think it's a very difficult project. I think it's going to take time. But I see no other no other company that can execute on that vision. I think it's going to be amazing. It like basically, physical labor. Like if you think transportation is a large market, try physical labor. It's like insane.
Well, but it's not only just physical labor. To me, the thing that's also exciting is the social robotics. So, the the relationship we'll have on different levels with those robots.
That's why I was really excited to see Optimus. Like um people have criticized me for the excitement. But I've I've worked with a lot of research labs that do humanoid legged robots. Boston Dynamics, Unitree, a lot there's a lot of companies that do legged robots. But that's the the elegance of the movement is a tiny tiny part of the big picture. So, integrating the two big exciting things to me about Tesla doing humanoid or any legged robots is clearly integrating it into the data engine.
So, the the data engine aspect, so the actual intelligence for the perception and the and the control and and the planning and all that kind of stuff, integrating into this huge the fleet that you mentioned, right? Um and then speaking of fleet, the second thing is the mass manufacturer, just knowing culturally driving towards a simple robot that's cheap to produce at scale. And doing that well, having experience to do that well, that changes everything. That's what That's a very different culture and style than Boston Dynamics, who by the way, those those robots are just the way they move. It It's like it'll be a very long time before Tesla could achieve the smoothness of movement, but that's not what it's about. It's It's about It's about the entirety of the system, like we talked about the data engine and the fleet. And that's super exciting. Even the initial sort of models but that too was really surprising. That in a few months you can get a a prototype.
Yep. And the reason that happened very quickly is, as you alluded to, there's a ton of copy-paste from what's happening on the autopilot, a lot. The amount of expertise that like came out of the woodwork at Tesla for building the human robot was incredible to see. Like basically Elon said at one point we're doing this, and then next day basically like all these CAD models started to appear. And people talking about like the supply chain and manufacturing. And people showed up with like screwdrivers and everything like the other day and started to like put together the body. I was like, 'Whoa, like all these people exist at Tesla.' And fundamentally building a car is actually not that different from building a robot. The same And that is true not just for the hardware pieces. And also let's not forget hardware not just for a demo, but manufacturing of that hardware at scale is like a whole different thing. But for software as well, basically this robot currently thinks it's a car.
It's going to have a midlife crisis at some point.
It thinks it's a car. Um some of the earlier demos actually we were talking about potentially doing them outside in the parking lot because that's where all of the computer vision that was like working out of the box.
Instead of like in inside. Um, but all the operating system, everything just copy pastes. Computer vision mostly copy pastes. I mean, you have to retrain the neural nets, but the approach and everything and data engine and offline trackers and the way we go about the occupancy tracker and so on, everything copy pastes. You just need to retrain the neural nets.
And then the planning control, of course, has to change quite a bit. But there's a ton of copy paste from what's happening at Tesla. And so if you were to if you were to go with goal of like, okay, let's build a million human robots and you're not Tesla, that's that's a lot to ask. If you're Tesla, it's actually like it's not it's not that crazy.
And then the then the follow question is then how difficult, just like with driving, how difficult is the manipulation task?
Uh, such that it can have an impact at scale. I think depending on the context, the really nice thing about robotics is that unless you're doing manufacturing and that kind of stuff, is there's more room for error.
Driving is so safety critical and it's so that it and also time critical. Like a robot is allowed to move slower.
Yes. I think it's going to take a long time, but the way you want to structure the development is you need you need to say, okay, it's going to take a long time. How can I set up the product development roadmap so that I'm making revenue along the way? I'm not setting myself up for a zero one loss function where it doesn't work until it works. You don't want to be in that position. You want to make it useful almost immediately and then you want to slowly deploy it and at scale. And you want to set up your data engine, your improvement loop, the telemetry, the evaluation, the harness and everything. And you want to improve the product over time incrementally and you're making revenue along the way. That's extremely important because otherwise you cannot build these large undertakings. Just like don't make sense economically and also from the point of view of the team working on it, they need the dopamine along the way. They're not just going to make a promise about this being useful, this is going to change the world in 10 years when it works. This is not where you want to be. You want to be in a place like I think Autopilot is today where it's offering increased safety and convenience of driving today. People pay for it. People like it. People purchase it. And then you also have the greater mission that you're working towards.
Mhm. And you see that so the dopamine for the team that that was a source of happiness and.
Yes. 100%. You're deploying this. People like it. People drive it. People pay for it. They care about it. There's all these YouTube videos. Your grandma drives it. She gives you feedback. People like it. People engage with it. You engage with it. Huge.
Do people that drive Teslas like recognize you and give you love? Like like hey thanks for the.
for the [laughter] for this nice feature that it's doing.
Yeah, I think the tricky thing is like some people really love you. Some people unfortunately like you're working on something that you think is extremely valuable, useful, etc. Some people do hate you. There's a lot of people who like hate me and the team and what every the whole project. And I think that.
Uh many cases they're not actually.
Yeah. Yeah, that's that's actually makes me sad about humans or the current the ways that humans interact. I think that's actually fixable. I think humans want to be good to each other. I think Twitter and social media is part of the mechanism that actually somehow makes the negativity more viral that it doesn't deserve like disproportionately add of like a viral viral boost to the negativity. But I got I wish people would just get excited about so suppress some of the jealousy.
some of the ego and just get excited for others.
And then there's a karma aspect to that. You get excited for others, they'll get excited for you. Same thing in academia. If you're not careful, there is a like a dynamical system there. If you if you think of in silos and get jealous of somebody else being successful that actually perhaps counterintuitively leads to less productivity of you as a community and you individually. I feel like if you keep celebrating others that actually makes you more successful.
And I think people haven't in depending on the industry haven't quite learned that yet.
Yeah. Some people are also very negative and very vocal. So, they're very prominently featured. But actually, there's a ton of people who are cheerleaders, but they're silent cheerleaders. And when you talk to people just in the world, they will tell you it's amazing, it's great. Especially like people who understand how difficult it is to get the stuff working. Like people who have built products and makers and entrepreneurs. Like making this work and changing something is incredibly hard. Those people are more likely to cheerlead you.
[laughter] Well, one one of the things that makes me sad is some folks in the robotics community don't do the cheerleading and they should. There's cuz they know how difficult it is. Well, they actually sometimes don't know how difficult it is to create a product at scale, right? They actually deploy in the real world. A lot of the development of robots and AI systems done on very specific small benchmarks. Um and as opposed to real world conditions.
Yes. Yeah, I think it's really hard to work on robotics in academic setting.
Or AI systems that apply in the real world. You you've criticized you uh flourished and loved for time the ImageNet, the famed ImageNet data set. And I've recently had some words of criticism that the academic research ML community gives a little too much love still to the ImageNet or like those kinds of benchmarks. Can Can Can you speak to the strengths and weaknesses of data sets used in machine learning research?
Actually, I don't know that I recall a specific instance where I was unhappy or criticizing ImageNet. I think ImageNet has been extremely valuable. It was basically a benchmark that allowed the deep learning community to demonstrate that deep neural networks actually work. It was there's a massive value in that. So, I think ImageNet was useful, but I'm basically it's become a bit of an MNIST at this point. So MNIST is like little 28 by 28 grayscale digits. There's kind of a joke dataset that everyone like just crushes.
There's still papers written on MNIST though, right?
Like strong papers. Like papers that focus on like how do we learn with a small amount of data, that kind of stuff.
I could see that being helpful, but not in sort of like mainline computer vision research anymore, of course.
I think the way I've heard you somewhere, maybe I'm just imagining things, but I think you said like ImageNet was a huge contribution to the community for a long time and now it's time to move past those kinds of.
Well, ImageNet has been crushed. I mean, you know, the error rates are Yeah, we're getting like 90% accuracy in 1000 classification way prediction. And I've seen those images and it's like really high. That's really that's really good. If I remember correctly, the top five error rate is now like 1% or something.
Given your experience with a gigantic real-world dataset, would you like to see benchmarks move in a certain directions that the research community uses?
Unfortunately, I don't think academics currently have the next ImageNet. Uh we've obviously I think we've crushed MNIST. We've basically kind of crushed ImageNet. Uh and there's no next sort of big benchmark that the entire community rallies behind and uses um you know, for further development of these networks.
I yeah, I wonder what it takes for a dataset to captivate the imagination of everybody, like where they all get behind it. That That could also need like a viral like a leader.
Somebody with popularity. I mean, that Yeah, what what did ImageNet take off? Is there Or is it just the accident of history?
The right amount of difficult. It was the right amount of difficult and simple and interesting enough, it just kind of like it was it was the right time for that kind of a dataset.
Question from Reddit. Uh What are your thoughts on the role that synthetic data and game engines will play in the future of neural net model development?
I think um as neural nets converge to humans the value of simulation to neural nets will be similar to value of simulation to humans. So, people use simulation for people use simulation because they can learn something in that kind of a system. Um and without having to actually experience it. Um but are you referring to the simulation we do in our head? Isn't that what you're saying?
Simulation I mean like video games or you know, um other forms of simulation for various professionals.
Well, so let me push back on that cuz maybe there's simulation that we do in our heads. Like simulate if I do this what do I think will happen?
Okay, that's like internal simulation.
Yeah, internal. Isn't that what we're doing? Let's assume it before we act.
Oh, yeah, but that's independent from like the use of simulation in the sense of like computer games or using simulation for training set creation or you know.
Is it independent or is it just loosely correlated? Cuz like isn't that useful to do like counterfactual or like edge case simulation to like you know, what happens if there's a nuclear war? What happens if there's, you know, like the those kinds of things? Is cuz it is a different simulation from like Unreal Engine. That's how I interpreted the question.
Ah, so like simulation of the average case.
Is that What's Unreal Engine? What what what what do you mean by Unreal Engine? So, simulating a world.
Yeah, physics of that world.
Why is that different? Like cuz you also can add behavior to that world.
And you can try all kinds of stuff, right? Like you could throw all kinds of weird things into it. So, Unreal Engine is not just about simulate I mean, I guess it is about simulating the physics of the world. It's also doing something with that.
Yeah, the graphics, the physics, and the agents that you put into the environment and stuff like that.
Yeah, see I think you I feel like you said that it's not that important. I guess for the future of AI development.
Is that Is that correct to interpret it that way?
I think humans use simulators for humans use simulators and they find them useful. And so, computers will use simulators and find them useful.
Okay, so you're saying it's not the I I don't use simulators very often. I play a video game every once in a while, but I don't think I derive any wisdom about my own existence from from those video games. It's a momentary escape from reality versus a source of wisdom about reality. So, I don't So, I think that's a very polite way of saying simulation is not that useful.
Yeah, maybe maybe not. I don't see it as like a fundamental really important part of like training neural nets currently. But I think as neural nets become more and more powerful, I think you will need fewer examples to train additional behaviors.
And simulation is Of course, there's a domain gap in a simulation that's not the real world. It's slightly something different. But with a powerful enough you need the domain gap can be bigger, I think, because neural net will sort of understand that even though it's not the real world, it like has all this high-level structure that I'm supposed to be like learn from.
So, the neural net will actually Yeah, it'll be able to leverage the synthetic data better.
by closing the gap by understanding in which ways this is not real data.
Uh ready to do better questions next time. That was That was a question about I'm just kidding. All right. Um So, is it possible, do you think, speaking of MNIST, to construct neural nets and training processes that require very little data?
So, we've been talking about huge data sets like the internet for training. I mean, one way to say that is like you said, like the querying itself is another level of training, I guess, and that requires little data.
But do you see any value in doing research and kind of going down the direction of can we use very little data to train to construct a knowledge base.
100%. I I just think like at some point you need a massive data set. And then when you pre-train your massive neural net and get something that, you know, is is like a GPT or something, then you're able to be very efficient at training any arbitrary new task.
So a lot of these GPTs, you know, you can do tasks like sentiment analysis or translation or so on just by being prompted with very few examples. Here's the kind of thing I want you to do like here's an input sentence, here's the translation into German. Input sentence, translation to German. Input sentence, blank. And the neural net will complete the translation to German just by looking at sort of the example you've provided. And so that's an example of a very few shot learning in the activations of the neural net instead of the weights of the neural net. And so I think basically just like humans, neural nets will become very data efficient at learning any other new task. But at some point you need a massive data set to pre-train your network.
To get that and we probably we humans have something like that. Do we do we have something like that? Do we have a passive in the background background model constructing thing that just runs all the time in a self-supervised way? We're not conscious of it.
I think humans definitely I mean obviously we have we learn a lot during during our life span. But also we have a ton of hardware that helps us initial at initialization coming from sort of evolution. And so I think that's also a really big a big component. A lot of people in the field I think they just talk about the amounts of like seconds and the you know, that a person has lived pretending that this is a tabula rasa sort of like a zero initialization of a neural net. And it's not. Like you can look at a lot of animals like for example a zebras. Zebras get born and they see and they can run. There's zero trained data in their life span. They can just do that. So somehow I have no idea how evolution has found a way to encode these algorithms and these neural net initializations are extremely good into ATCGs and I have no idea how this works, but apparently it's possible because here's a proof by existence.
There's something magical about going from a single cell to a organism that is born to the first few years of life. I kind of like the idea that the reason we don't remember anything about the first few years of our life is that it's a really painful process. Like it's a very difficult challenging training process.
Like intellectually. Like and maybe Yeah, mean I don't Why don't we remember any of that? There might be some crazy training going on and that that the maybe that's the background model training that is is very painful and so it's best for the system once it's trained not to remember how it's constructed.
I think it's just like the hardware for long-term memory is just not fully developed. Sure. I kind of feel like the first few years of uh of infants is not actually like learning, it's brain maturing.
