The following is a conversation with Ilya Sutskever, co-founder and chief scientist of OpenAI, one of the most cited computer scientists in history with over 165,000 citations, and to me one of the most brilliant and insightful minds ever in the field of deep learning. There are very few people in this world who I would rather talk to and brainstorm with about deep learning, intelligence, and life in general than Ilya. On and off the mic, this was an honor and a pleasure. This conversation was recorded before the outbreak of the pandemic. For everyone feeling the medical, psychological, and financial burden of this crisis, I'm sending love your way. Stay strong, we're in this together, we'll beat this thing. This is the Artificial Intelligence Podcast. If you enjoy it, subscribe on YouTube, review it with five stars on Apple Podcasts, support on Patreon, or simply connect with me on Twitter at Lex Fridman, spelled F-R-I-D-M-A-N. As usual, I'll do a few minutes of ads now and never any ads in the middle that can break the flow of the conversation. I hope that works for you and doesn't hurt the listening experience. This show is presented by Cash App, the number one finance app in the App Store. When you get it, use code LexPodcast. Cash App lets you send money to friends, buy Bitcoin, invest in the stock market with as little as $1. Since Cash App allows you to buy Bitcoin, let me mention that cryptocurrency in the context of the history of money is fascinating. I recommend "Ascent of Money" as a great book on this history, both the book and audiobook. Great debits and credits on ledgers started around 30,000 years ago. The US dollar was created over 200 years ago, and Bitcoin, the first decentralized cryptocurrency, was released just over 10 years ago. So given that history, cryptocurrency is still very much in its early days of development, but it's still aiming to and just might redefine the nature of money. So again, if you get Cash App from the App Store or Google Play and use the code LexPodcast, you get $10, and Cash App will also donate $10 to FIRST, an organization that is helping advance robotics and STEM education for young people around the world. And now here's my conversation with Ilya Sutskever.
You were one of the three authors, with Alex Krizhevsky and Geoff Hinton, of the famed AlexNet paper that is arguably the paper that marked the big catalytic moment that launched the deep learning revolution. At that time, take us back to that time. What was your intuition about neural networks, about the representational power of neural networks? And maybe you could mention how did that evolve over the next few years up to today, over the 10 years?
I can answer that question. At some point in about 2010 or 2011, I connected two facts in my mind. The realization was this: at some point we realize that we can train very large—I shouldn't say very, they're tiny by today's standards—but large and deep neural networks end-to-end with backpropagation. At some point, different people obtained this result. I obtained this result. The first moment in which I realized that deep neural networks are powerful was when James Martens invented the Hessian-free optimizer in 2010, and he trained a 10-layer neural network end-to-end without pre-training from scratch. When that happened, I thought, this is it. Because if you can train a big neural network, a big neural network can represent very complicated functions. Because if you have a neural network with 10 layers, it's as though you allow the human brain to run for some number of milliseconds. Neuron firings are slow, so in maybe 100 milliseconds, your neurons only fire 10 times. So it's also kind of like 10 layers, and in 100 milliseconds you can perfectly recognize any object. So I already had the idea then that we need to train a very big neural network on lots of supervised data, and then it must succeed, because we can find the best neural network. And there's also theory that if you have more data than parameters, you won't overfit. Today we know that actually this theory is very incomplete, and you won't overfit even if you have less data than parameters. But definitely if you have more data than parameters, you won't overfit.
So the fact that neural networks were heavily overparameterized wasn't discouraging to you. You were thinking about the theory that the huge number of parameters is okay, it's going to be okay.
I mean, there was some evidence before that it was okay-ish, but the theory was that if you had a big dataset and a big neural net, it was going to work. The overparameterization just didn't really figure much as a problem. I thought, well, with images you're just going to add some data augmentation, it's going to be okay.
So where was any doubt coming from?
The main doubt was: can we train a big—will we have enough compute to train a big enough neural net with backpropagation? Backpropagation I thought would work. The thing which wasn't clear was whether there would be enough compute to get a very convincing result. Then at some point, Alex Krizhevsky wrote these insanely fast CUDA kernels for training convolutional neural nets. That was bam, let's do this, let's get ImageNet, and it's going to be the greatest thing.
Was your intuition most from empirical results by you and by others, like just actually demonstrating that a piece of program can train a 10-layer neural network? Or was there some pen and paper or marker and whiteboard thinking? Because you just mentioned you connected a 10-layer large neural network to the brain. So in your intuition about neural networks, does the human brain come into play as an intuition builder?
Definitely. I mean, you've got to be precise with these analogies between artificial neural networks and the brain, but there is no question that the brain is a huge source of intuition and inspiration for deep learning researchers, all the way from Rosenblatt in the '60s. If you look at the whole idea of a neural network, it is directly inspired by the brain. You had people like McCulloch and Pitts who were saying, hey, you've got these neurons in the brain, and hey, we recently learned about the computer and automata. Can we use some ideas from the computer and automata to design some kind of computational object that's going to be simple, computational, and kind of like the brain? And they invented the neuron. So they were inspired by it back then. Then you had the convolutional neural network from Fukushima and later Yann LeCun, who said, hey, if you limit the receptive fields of a neural network, it's going to be especially suitable for images, as it turned out to be true. So there were a very small number of examples where analogies to the brain were successful, and I thought, well, probably an artificial neuron is not that different from the brain if it's scaled hard enough, so let's just assume it is and roll with it.
So now we're at a time where deep learning is very successful, so let us squint less and say, let's open our eyes and say, what to you is an interesting difference between the human brain? Now I know you're probably not an expert, neither neuroscientist nor neurobiologist, but loosely speaking, what's the difference between the human brain and artificial neural networks that's interesting to you for the next decade or two?
That's a good question. What is an interesting difference between the brain and our artificial neural networks? I feel like today, artificial neural networks—we all agree that there are certain dimensions in which the human brain vastly outperforms our models, but I also think that there are some ways in which our artificial neural networks have a number of very important advantages over the brain. Looking at the advantages versus disadvantages is a good way to figure out what is the important difference. So the brain uses spikes, which may or may not be important.
That's a really interesting question. Do you think it's important or not? That's one big architectural difference.
It's hard to tell, but my prior is not very high. I can say why. There are people who are interested in spiking neural networks, and basically what they've figured out is that they need to simulate the non-spiking neural networks in spikes, and that's how they're going to make them work. If you don't simulate the non-spiking neural networks in spikes, it's not going to work, because the question is, why should it work? And that connects to questions around backpropagation and questions around deep learning. You've got this giant neural network, why should it work at all? Why should the learning rule work at all? It's not a self-evident question, especially if you were just starting in the field and you read the very early papers. You can say, hey, people are saying let's build neural networks, that's a great idea because the brain is a neural network, so it would be useful to build neural networks. Now let's figure out how to train them. It should be possible to train them probably, but how? And so the big idea is the cost function. That's the big idea. The cost function is a way of measuring the performance of the system according to some measure.
Actually, let me think. Is that a difficult idea to arrive at, and how big of an idea is that, that there's a single cost function?
Let me take a pause. Is supervised learning a difficult concept to come to? I don't know. All concepts are very easy in retrospect.
The reason I asked that, and we'll talk about it, is because are there things that don't necessarily have a cost function? Maybe have many cost functions, or maybe have dynamic cost functions, or maybe a totally different kind of architecture? Because we have to think like that in order to arrive at something new.
The only good examples of things which don't have clear cost functions are GANs. In a GAN, you have a game. So instead of thinking of a cost function where you want to optimize, where you know that you have an algorithm, gradient descent, which will optimize the cost function, and then you can reason about the behavior of your system in terms of what it optimizes, with a GAN you say, I have a game, and I'll reason about the behavior of the system in terms of the equilibria of the game. But it's all about coming up with these mathematical objects that help us reason about the behavior of our system.
That's really interesting. So again, is the GAN... the cost function is emergent from the comparison. I don't know if it has a cost function. I don't know if it's meaningful to talk about the cost function of a GAN. It's kind of like the cost function of biological evolution or the cost function of the economy. You can talk about regions to which it will go towards, but I don't think the cost function analogy is the most useful.
So if evolution doesn't really have a cost function like our mathematical conception of a cost function, then do you think cost functions in deep learning are holding us back?
You just kind of mentioned that cost function is a nice first profound idea. Do you think that's a good idea? Do you think it's an idea we will go past? Self-play starts to touch on that a little bit in reinforcement learning systems.
That's right. Self-play and also ideas around exploration where you're trying to take action that surprises a predictor. I'm a big fan of cost functions. I think cost functions are great and they serve us really well, and I think that whenever we can do things with cost functions, we should. And you know, maybe there is a chance that we will come up with some yet another profound way of looking at things that will involve cost functions in a less central way, but I don't know. I think cost functions are... I would not bet against cost functions.
Is there other things about the brain that pop into your mind that might be different and interesting for us to consider in designing artificial neural networks? So we talked about spiking a little bit.
One thing which may potentially be useful: I think neuroscientists have figured out something about the learning rule of the brain. I'm talking about spike-timing-dependent plasticity, and it would be nice if some people were to study that in simulation.
Wait, sorry, spike-timing-dependent plasticity? Yeah, what's that? STDP?
It's a particular learning rule that uses spike timing to figure out how to determine how to update the synapses. So it's kind of like if the synapse fires into the neuron before the neuron fires, then it strengthens the synapse, and if the synapse fires into the neuron shortly after the neuron fired, then it weakens the synapse. Something along this line. I'm 90% sure it's right, so if I said something wrong here, don't get too angry.
But you sounded brilliant while saying it. But the timing, that's one thing that's missing. The temporal dynamics is not captured. I think that's a fundamental property of the brain, the timing of the signals.
Well, recurrent neural networks... but you think of that as a crude simplified... there's a clock, I guess, in recurrent neural networks. It seems like the brain is the general, continuous version of that, the generalization where all possible timings are possible, and within those timings is contained some information. Do you think recurrent neural networks can capture the same kind of phenomena as the timing that seems to be important for the brain in the firing of neurons?
I think recurrent neural networks are amazing and they can do anything we'd want a system to do right now. Recurrent neural networks have been superseded by Transformers, but maybe one day they'll make a comeback. We'll see.
Let me, on a small tangent, say: do you think they'll be back? So much of the breakthroughs recently that we'll talk about in natural language processing and language modeling have been with Transformers that don't emphasize recurrence. Do you think recurrence will make a comeback?
Well, some kind of recurrence, I think, very likely. Recurrent neural networks as they're typically thought of for processing sequences, I think it's also possible... What is a recurrent neural network generally speaking? I guess what is a recurrent neural network? You have a neural network which maintains a high-dimensional hidden state, and then when an observation arrives, it updates its high-dimensional hidden state through its connections in some way.
Do you think that's what expert systems did? Symbolic AI, the knowledge base growing a knowledge base, is maintaining a hidden state which is its knowledge base and is growing it by sequential processing. Do you think of it more generally in that way, or is it simply the more constrained form of a hidden state with certain kinds of gating units that we think of today with LSTMs?
The hidden state is technically what you describe, the hidden state that goes inside the LSTM or the RNN or something like this. But then what should be contained? If you want to make the expert system analogy, I'm not... I mean, you could say that the knowledge is stored in the connections, and then the short-term processing is done in the hidden state. Yes, could you say that?
So do you think there's a future of building large-scale knowledge bases within the neural networks?
So we're going to pause in that confidence because I want to explore that. But let me zoom back out and ask, back to the history of ImageNet. Neural networks have been around for many decades as you mentioned. What do you think were the key ideas that led to their success at that moment and beyond, the success in the past 10 years?
Okay, so the question is, to make sure I didn't miss anything, the key ideas that led to the success of deep learning over the past 10 years, even though the fundamental thing behind deep learning has been around for much longer.
The key idea about deep learning, or rather the key fact about deep learning before deep learning started to be successful, is that it was underestimated. People who worked in machine learning simply didn't think that neural networks could do much. People didn't believe that large neural networks could be trained. There was a lot of debate going on in machine learning about what are the right methods, and people were arguing because there was no way to get hard facts. By that I mean there were no benchmarks which were truly hard, that if you do really well on them then you can say, look, here is my system. That's when this field becomes a little bit more of an engineering field. So in terms of deep learning, to answer the question directly, the ideas were all there. The thing that was missing was a lot of supervised data and a lot of compute. Once you have a lot of supervised data and a lot of compute, then there is a third thing which is needed as well, and that is conviction. Conviction that if you take the right stuff which already exists and apply it and mix it with a lot of data and a lot of compute, that it will in fact work. So that was the missing piece. You needed the data, you needed the compute which showed up in terms of GPUs, and you needed the conviction to realize that you need to mix them together.
So that's really interesting. I guess the presence of compute and the presence of supervised data allowed the empirical evidence to do the convincing of the majority of the computer science community. So I guess there's a key moment with Jitendra Malik and Alexei Efros, who were very skeptical, and then there's Geoffrey Hinton who was the opposite of skeptical, and there was a convincing moment. I think ImageNet served as that moment.
That's right. And they represented the big pillars of the computer vision community, kind of the wizards got together, and then all of a sudden there was a shift. And it's not enough for the ideas to all be there and the compute to be there; it's for it to convince the cynicism that existed.
That's interesting. The people just didn't believe for a couple of decades.
Well, but it was more than that. When put this way, it sounds like, well, those silly people who didn't believe, what were they missing? But in reality, things were confusing because neural networks really did not work on anything, and they were not the best method on pretty much anything as well. It was pretty rational to say, yeah, this stuff doesn't have any traction. And that's why you need to have these very hard tasks which produce undeniable evidence, and that's how we make progress. And that's why the field is making progress today, because we have these hard benchmarks which represent true progress, and this is why we are able to avoid endless debate.
So incredibly, you've contributed some of the biggest recent ideas in AI, in computer vision, natural language processing, reinforcement learning, sort of everything in between. Maybe not GANs, there may not be a topic you haven't touched, and of course the fundamental science of deep learning. What is the difference to you between vision, language, and reinforcement learning, action as learning problems? And what are the commonalities? Do you see them as all interconnected, or are they fundamentally different domains that require different approaches?
Okay, that's a good question. Machine learning is a field with a lot of unity, a huge amount of unity in fact.
What do you mean by unity? Overlap of ideas?
Overlap of ideas, overlap of principles. In fact, there's only one or two or three principles which are very, very simple, and then they apply in almost the same way to the different modalities, to the different problems. And that's why today, when someone writes a paper on improving optimization of deep learning in vision, it improves the different NLP applications and it improves the different reinforcement learning applications. So I would say that computer vision and NLP are very similar to each other today. They differ in that they have slightly different architectures. We use Transformers in NLP and use convolutional neural networks in vision, but it's also possible that one day this will change and everything will be unified with a single architecture. Because if you go back a few years ago in natural language processing, there were a huge number of architectures for every different tiny problem had its own architecture. Today, there's just one Transformer for all those different tasks. And if you go back in time even more, you had even more fragmentation, and every little problem in AI had its own little sub-specialization and little set of skills, people who would know how to engineer the features. Now it's all been subsumed by deep learning. We have this unification, and so I expect vision to become unified with natural language as well. Or rather, I shouldn't say expect, I think it's possible. I don't want to be too sure because the conv net is very computationally efficient. RL is different. RL does require slightly different techniques because you really do need to take action, you really do need to do something about exploration, your variance is much higher. But I think there is a lot of unity even there, and I would expect for example that at some point there will be some broader unification between RL and supervised learning, where somehow the RL will be making decisions to make the supervised learning go better, and it will be one big black box and you just shovel things into it and it just figures out what to do with whatever you shovel at it.
I mean, reinforcement learning has some aspects of language and vision combined almost. There's elements of a long-term memory that you should be utilizing, and there's elements of a really rich sensory space. So it seems like it's the union of the two or something like that.
I'd say something slightly differently. I'd say reinforcement learning is neither, but it naturally interfaces and integrates with the two of them.
You think action is fundamentally different? So what is interesting about what is unique about policy, of learning to act?
Well, one example is that when you learn to act, you are fundamentally in a non-stationary world because as your actions change, the things you see start changing. You experience the world in a different way, and this is not the case for the more traditional static problem. You have some distribution and you just apply a model to that distribution.
You think it's a fundamentally different problem, or is it just more difficult?
It's a generalization of the problem of understanding. I mean, it's a question of definitions almost. There is a huge amount of commonality for sure. You take gradients, you try to approximate gradients in both cases. In the case of reinforcement learning, you have some tools to reduce the variance of the gradients. You do that. There's lots of commonality. Use the same neural net in both cases, you compute the gradient, you apply Adam in both cases. So I mean, there's lots in common for sure, but there are some small differences which are not completely insignificant. It's really just a matter of your point of view, what frame of reference, how much do you want to zoom in or out as you look at these problems.
Which problem do you think is harder? So people like Noam Chomsky believe that language is fundamental to everything, so it underlies everything. Do you think language understanding is harder than visual scene understanding, or vice versa?
I think that asking if a problem is hard is slightly wrong. I think the question is a little bit wrong, and I want to explain why. So what does it mean for a problem to be hard? The non-interesting, dumb answer to that is there's a benchmark and there's a human-level performance on that benchmark, and how much effort is required to reach the human level on a very good benchmark.
Yeah, like some... I understand what you mean by that.
What I was going to say is that a lot of it depends on... once you solve a problem, it stops being hard, and that's always true. So something is hard or not depends on what the tools can do today. So you know, today, true human-level language understanding and visual perception are hard in the sense that there is no way of solving the problem completely in the next three months. So I agree with that statement. Beyond that, my guess would be as good as yours. I don't know.
So you don't have a fundamental intuition about how hard language understanding is?
I think I changed my mind. I'd say language is probably going to be harder. I mean, it depends on how you define it. If you mean absolute top-notch, 100% language understanding, I go with language. But then if I show you a piece of paper with letters on it, is that vision? You see what I mean? So you have a vision system, you say it's the best human-level vision system. I show you, I open a book and I show you letters. Will it understand how these letters form into words and sentences and meaning? Is this part of the vision problem? Where does vision end and language begin?
Yeah, so Chomsky would say it starts at language. So vision is just a little example of the kind of structure and fundamental hierarchy of ideas that's already represented in our brain somehow, that's represented through language. But where does vision stop and language begin? That's a really interesting question.
So one possibility is that it's impossible to achieve really deep understanding in either images or language without basically using the same kind of system. So you're going to get the other for free.
I think it's pretty likely that yes, if we can get one, our machine learning is probably that good that we can get the other. But it's not 100%, I'm not 100% sure. And also I think a lot of it really does depend on your definitions of perfect vision, because reading is vision, but should it count?
To me, my definition is if a system looked at an image and then a system looked at a piece of text and then told me something about that and I was really impressed...
That's relative. You'll be impressed for half an hour and then you're going to say, well, I mean, all the systems do that. But here's the thing, they don't.
Yeah, but I don't have that with humans. Humans continue to impress me.
Is that true? Well, the ones... okay, so I'm a fan of monogamy, so I like the idea of marrying somebody, being with them for several decades. So I believe in the fact that yes, it's possible to have somebody continuously giving you pleasurable, interesting, witty, new ideas. Friends, yeah. I think so. They continue to surprise you. The surprise is that injection of randomness seems to be a nice source of continued inspiration, the wit, the humor. I think that would be... it's a very subjective test, but I think if you have enough humans in the room, I understand what you mean.
I feel like I misunderstood what you meant by impressing you. I thought you meant to impress you with its intelligence, with how well it understands an image. I thought you meant something like, I'm going to show it a really complicated image and it's going to get it right, and you're going to say, wow, that's really cool. Our systems of January 2020 have not been doing that.
No, I think it all boils down to the reason people click like on stuff on the internet, which is it makes them laugh. So it's humor or wit, or insight. I'm sure we'll get that as well.
So forgive the romanticized question, but looking back, what is the most beautiful or surprising idea in deep learning or AI in general you've come across?
I think the most beautiful thing about deep learning is that it actually works. And I mean it, because you've got these ideas, you've got the little neural network, you've got the backpropagation algorithm, and then you've got some theories as to, you know, this is kind of like the brain, so maybe if you make it large, if you make the neural network large and you train it on a lot of data, then it will do the same function that the brain does. And it turns out to be true. That's crazy. And now we just train these neural networks and you make them larger and they keep getting better. And I find it unbelievable. I find it unbelievable that this whole AI stuff with neural networks works.
Have you built up an intuition of why? Are there little bits and pieces of intuitions, insights of why this whole thing works?
I mean, definitely. We know that optimization... we now have good, we've had lots of empirical, huge amounts of empirical reasons to believe that optimization should work on most problems we care about.
Do you have insights of what... you just said empirical evidence is most of your sort of... empirical evidence kind of convinces you. It's like evolution. It shows you that this evolutionary process seems to be a good way to design organisms that survive in their environment, but it doesn't really get you to the insights of how the whole thing works.
I think it's a good analogy. Physics, you know, you say, hey, let's do some physics calculation and come up with some new physics theory and make some prediction, but then you've got to run the experiment. It's important. So it's a bit the same here, except that maybe sometimes the experiment came before the theory, but it still is the case. You have some data and you come up with some prediction. You say, yeah, let's make a big neural network, let's train it, and it's going to work much better than anything before it, and it will in fact continue to get better as you make it larger. And it turns out to be true. That's amazing when a theory is validated like this. It's not a mathematical theory, it's more of a biological theory almost. So I think there are not terrible analogies between deep learning and biology. I would say it's like the geometric mean of biology and physics. That's deep learning.
The geometric mean of biology and physics. I think I'm going to need a few hours to wrap my head around that.
Because just to find the geometric mean of what biology represents... well, biology, things are really complicated and theories are really hard to have good predictive theory. And in physics, the theories are too good. In physics, people make super precise theories which make these amazing predictions. And in machine learning, we're kind of in between. But it'd be nice if machine learning somehow helped us discover the unification of the two, as opposed to sort of the in-between. But you're right, you're trying to juggle both.
So do you think there are still beautiful and mysterious properties of neural networks that are yet to be discovered?
Definitely. I think that we are still massively underestimating deep learning.
What do you think it will look like? If I knew, I would have done it. But if you look at all the progress in the past 10 years, I would say most of it... there have been a few cases where things that felt like really new ideas showed up, but by and large, every year we thought, okay, deep learning goes this far. Nope, it actually goes further. And then the next year, okay, now this is it, this is peak deep learning, we're really done. Nope, goes further. It just keeps going further each year. So that means that we keep underestimating it. People not understanding it has surprising properties all the time.
Do you think it's getting harder and harder to make progress?
It depends on what we mean. I think the field will continue to make very robust progress for quite a while. I think for individual researchers, especially people who are doing research, it can be harder because there is a very large number of researchers right now. I think that if you have a lot of compute, then you can make a lot of very interesting discoveries, but then you have to deal with the challenge of managing a huge compute cluster to run your experiments. It's a little bit harder.
So I'm asking you all these questions that nobody knows the answer to, but you're one of the smartest people I know, so I'm going to keep asking. So let's imagine all the breakthroughs that happen in the next 30 years in deep learning. Do you think most of those breakthroughs can be done by one person with one computer, sort of in the space of breakthroughs? Do you think compute and large efforts will be necessary?
I mean, I can't be sure. When you say one computer, you mean how large?
Ah, you're clever. I mean one GPU.
I see. I think it's pretty unlikely. I think it's pretty unlikely. I think that the stack of deep learning is starting to be quite deep. If you look at it, you've got all the way from the ideas, the systems to build the datasets, the distributed programming, the building the actual cluster, the GPU programming, putting it all together. So now the stack is getting really deep, and I think it becomes quite hard for a single person to become world-class in every single layer of the stack.
What about what Vladimir Vapnik really insists on, which is taking MNIST and trying to learn from very few examples, being able to learn more efficiently? Do you think there'll be breakthroughs in that space that may not need the huge compute?
I think there will be a large number of breakthroughs in general that will not need a huge amount of compute. So maybe I should clarify that I think that some breakthroughs will require a lot of compute, and I think building systems which actually do things will require a huge amount of compute. That one is pretty obvious. If you want to do X and X requires a huge neural net, you've got to get a huge neural net. But I think there will be lots of room for very important work being done by small groups and individuals.
Can you maybe, on the topic of the science of deep learning, talk about one of the recent papers that you released, the deep double descent, where bigger models and more data hurt? I think it's a really interesting paper. Can you describe the main idea?
Definitely. So what happened is that over the years, some small number of researchers noticed that it is kind of weird that when you make the neural network larger, it works better, and it seems to go in contradiction with statistical ideas. And then some people made an analysis showing that actually you've got this double descent bump. And what we've done was to show that double descent occurs for pretty much all practical deep learning systems.
Can you step back? What's the x-axis and the y-axis of a double descent plot?
Okay, great. So you can take your neural network and you can start increasing its size slowly while keeping your dataset fixed. So if you increase the size of the neural network slowly, and if you don't do early stopping—that's a pretty important detail—then when the neural network is really small, you make it larger, you get a very rapid increase in performance. Then you continue to make it larger, and at some point performance will get worse, and it gets the worst exactly at the point at which it achieves zero training error, precisely zero training loss. And then as you make it larger, it starts to get better again. And it's kind of counterintuitive because you'd expect deep learning phenomena to be monotonic. And it's hard to be sure what it means, but it also occurs in the case of linear classifiers. And the intuition basically boils down to the following: when you have a large dataset and a small model, then small, tiny random... basically, what is overfitting? Overfitting is when your model is somehow very sensitive to the small, random, unimportant stuff in your training dataset. So if you have a small model and you have a big dataset, and there maybe some random thing, you know, some training cases are randomly in the dataset and others may not be there, but the small model is kind of insensitive to this randomness because there is pretty much no uncertainty about the model when the dataset is large.
So at the very basic level, to me it is the most surprising thing that neural networks don't overfit every time very quickly, before ever being able to learn anything, with a huge number of parameters.
So here is one way... let me try to give the explanation and maybe that will work. So you've got a huge neural network, a huge number of parameters. Now let's pretend everything is linear, which it's not, let's just pretend. Then there is this big subspace where the neural network achieves zero error, and SGD is going to find approximately the point with the smallest norm in that subspace. And that can also be proven to be insensitive to the small randomness in the data when the dimensionality is high. But when the dimensionality of the data is equal to the dimensionality of the model, then there is a one-to-one correspondence between all the datasets and the models, so small changes in the dataset actually lead to a lot of changes in the model, and that's why performance gets worse. So this is the best explanation more or less.
So then it would be good for the model to have more parameters, to be bigger than the data?
That's right, but only if you don't early stop. If you introduce early stopping in your regularization, you can make the double descent bump almost completely disappear.
Early stopping is when you train your model and you monitor your test or validation performance. Then if at some point validation performance starts to get worse, you say, okay, let's stop training. We're good enough. So the magic happens after that moment. So you don't want to do the early stopping. If you don't do the early stopping, you get this very pronounced double descent.
Do you have any intuition why this happens?
Well, yeah, so let's see. The intuition is basically this: when the dataset has as many degrees of freedom as the model, then there is a one-to-one correspondence between them, and so small changes to the dataset lead to noticeable changes in the model. So your model is very sensitive to all the randomness, it is unable to discard it. Whereas it turns out that when you have a lot more data than parameters or a lot more parameters than data, the resulting solution will be insensitive to small changes in the dataset. So it's able to discard the small changes, the spurious correlations which you don't want.
Geoff Hinton suggested we need to throw away backpropagation. We already kind of talked about this a little bit, but he suggested that we need to throw away backpropagation and start over. I mean, of course some of that is a little bit wit and humor, but what do you think? What could be an alternative method of training neural networks?
Well, the thing that he said precisely is that to the extent that you can't find backpropagation in the brain, it's worth seeing if we can learn something from how the brain learns. But backpropagation is very useful and we should keep using it. He was saying that once we discover the mechanism of learning in the brain, or any aspect of that mechanism, we should also try to implement that in neural networks. If it turns out that we can't find backpropagation in the brain...
So I guess your answer to that is backpropagation is pretty damn useful, so why are we complaining?
I personally am a big fan of backpropagation. I think it's a great algorithm because it solves an extremely fundamental problem, which is finding a neural circuit subject to some constraints. And I don't see that problem going away. So that's why I think it's pretty unlikely that we'll have anything which is going to be dramatically different. It could happen, but I wouldn't bet on it right now.
So let me ask a sort of big picture question. Do you think neural networks can be made to reason? Why not?
Well, if you look for example at AlphaGo or AlphaZero, the neural network of AlphaZero plays Go, which we all agree is a game that requires reasoning, better than 99.9% of all humans. Just the neural network without the search, just the neural network itself. Doesn't that give us an existence proof that neural networks can reason?
To push back and disagree a little bit, we all agree that Go is reasoning? I think I agree. I don't think it's trivial. Obviously, intelligence is a loose, gray area term. But yes, I think it has some of the same elements of reasoning. Reasoning is almost akin to search. There's a sequential element of stepwise consideration of possibilities and sort of building on top of those possibilities in a sequential manner until you arrive at some insight. Go is kind of like that. And when you have a single neural network doing that without search, it's kind of like that. So there's an existence proof in a particular constrained environment that a process akin to what many people call reasoning exists. But more general kind of reasoning, off the board, there is one other existence proof.
Us humans. Yes, okay, all right. So do you think the architecture that will allow neural networks to reason will look similar to the neural network architectures we have today?
