The following is a conversation with Ilya Sutskever, co-founder and chief scientist of OpenAI, one of the most cited computer scientists in history with over 65,000 citations. This was recorded before the pandemic. This is the Artificial Intelligence podcast. I'll do a few minutes of ads now and never any ads in the middle. This show is presented by Cash App. Use code Lex Podcast. Cash App lets you send money, buy Bitcoin, invest. Cryptocurrency is still in its early days. If you get Cash App and use the code Lex Podcast, you get $10, and Cash App will donate $10 to FIRST. And now here's my conversation with Ilya Sutskever.
You were one of the three authors with Alex Krizhevsky and Geoff Hinton of the famed AlexNet paper that marked the big catalytic moment that launched the deep learning revolution. Take us back to that time. What was your intuition about neural networks, about the representational power of neural networks, and how did that evolve over the next few years up to today?
Yeah, I can answer that question. At some point in about 2010 or 2011, I connected two facts. The realization was that we can train very large deep neural networks end to end with backpropagation. The first moment I realized deep neural networks were powerful was when James Martens invented the Hessian-free optimizer in 2010 and trained a 10-layer neural network from scratch without pre-training. I thought, this is it. If you can train a big neural network, it can represent very complicated functions. A 10-layer network is like the human brain running for 100 milliseconds, enough to recognize any object. I already had the idea that we need to train a very big neural network on lots of supervised data, and it must succeed. There's theory that if you have more data than parameters, you won't overfit. Today we know that theory is incomplete, but if you have more data than parameters, you definitely won't overfit.
So you were thinking about the theory that the number of parameters being huge is okay, it's going to be okay. Where was any doubt coming from? The main doubt was whether we would have enough compute to train a big enough neural net.
With backpropagation, I thought it would work. The thing that wasn't clear was whether there would be enough compute to get a very convincing result. Then Alex Krizhevsky wrote these insanely fast CUDA kernels for training convolutional neural nets, and that was it.
Was your intuition most from empirical results or from pen and paper thinking? You mentioned the brain. Does the human brain come into play as an intuition builder?
Definitely. The brain is a huge source of intuition and inspiration for deep learning researchers, from Rosenblatt in the 60s to Fukushima and Yann LeCun. Analogies to the brain have been successful. So let's assume it is and roll with it.
Now deep learning is very successful. Let's open our eyes. What to you is an interesting difference between the human brain and artificial neural networks for the next decade or two?
That's a good question. The brain uses spikes, which may or may not be important. But artificial neural networks have advantages. The cost function is a big idea. It's a way of measuring performance. The only example of something without a clear cost function is GANs, where you have a game. But I'm a big fan of cost functions. They serve us well. I would not bet against them.
Let me take a pause. Is supervised learning a difficult concept to come to?
I don't know. All concepts are very easy in retrospect.
Yeah, that's true. But are there things that don't have a cost function? GANs have a game. So do you think cost functions are holding us back?
Right. In a GAN, instead of a cost function, you have a game and reason about equilibria. The cost function is emergent. It's like the cost function of biological evolution or the economy. I'm a big fan of cost functions. I think they are great and we should use them whenever we can. I would not bet against them.
Is there other things about the brain that might be different and interesting for us to consider?
One thing is spike-timing-dependent plasticity (STDP). It's a learning rule that uses spike timing to update synapses. If the synapse fires before the neuron fires, it strengthens; if after, it weakens. The timing of signals is a fundamental property of the brain.
Wait, spike-timing-dependent plasticity? What's that?
STDP. It's a learning rule. Something like that. Don't get too angry if I said something wrong.
But the timing is missing. The temporal dynamics is not captured. Recurrent neural networks can capture that? I think RNNs are amazing and can do anything we want. But they've been superseded by transformers. Do you think recurrence will make a comeback?
Well, some kind of recurrence is very likely. RNNs as typically thought of for processing sequences, it's possible.
What is a recurrent neural network to you? Expert systems maintain a hidden state. Do you think of it more generally?
The hidden state is what you describe. Knowledge is stored in the connections, and short-term processing is in the hidden state.
Yes. So do you think there's a future of building large-scale knowledge bases within neural networks?
Let me zoom back out. What were the key ideas that led to the success of deep learning over the past 10 years?
The key idea is that deep learning was underestimated. People didn't believe large neural networks could be trained. The missing pieces were a lot of supervised data and a lot of compute. And conviction that it would work. Once you have data and compute, you need conviction to mix them together.
That's interesting. So the presence of compute and data allowed empirical evidence to convince the community. There was a key moment with Jitendra Malik and Alyosha Efros being skeptical, and Geoff Hinton being the opposite. And ImageNet served as that moment.
Yes. It's not enough for the ideas to be there. They had to convince the cynicism. But it was rational to be skeptical because neural networks didn't work on anything before. That's why hard benchmarks are important.
You've contributed to vision, language, reinforcement learning. What is the difference between those domains? Do you see them as interconnected?
Machine learning has a lot of unity. There are only one or two or three principles that apply in almost the same way to different modalities. Vision and NLP are very similar today. They use different architectures, but that may change. RL is different because of action and exploration, but there is still a lot of unity. I expect vision to become unified with natural language too.
RL seems like the union of vision and language. Do you think action is fundamentally different?
I'd say it's neither, but it interfaces with both. The non-stationarity is unique. But there is huge commonality: gradients, neural nets, etc.
Which problem is harder: language understanding or visual scene understanding?
The question of hardness is slightly wrong. It depends on definitions. But if I had to guess, language is probably harder. If you mean absolute top-notch 100% understanding, I'll go with language.
So you don't have a fundamental intuition? You changed your mind? I'd say language is probably going to be harder.
I think I changed my mind. I'd say language is probably going to be harder.
But if I show you a piece of paper with letters, is that vision or language? Where does vision end and language begin?
That's a really interesting question. One possibility is that it's impossible to achieve deep understanding in either without using the same kind of system. So you get the other for free.
To me, if a system looked at an image and text and told me something impressive, that would be a test. But humans continue to impress me.
I misunderstood what you meant. I thought you meant impress with intelligence. But I'm sure we'll get that as well.
What is the most beautiful or surprising idea in deep learning you've come across?
The most beautiful thing is that it actually works. You have these ideas, backpropagation, and it turns out to be true. That's crazy. I find it unbelievable that this whole AI stuff with neural networks works.
Have you built up an intuition of why?
We have empirical reasons to believe optimization works. It's like evolution or physics. You run experiments. It's amazing when a theory is validated.
You said it's the geometric mean of biology and physics. I need to wrap my head around that.
Biology is complicated, physics has precise theories. Deep learning is in between. It would be nice if it helped unify them.
Do you think there are still beautiful and mysterious properties in neural networks to be discovered?
Definitely. We are still massively underestimating deep learning. It keeps going further each year.
Do you think it's getting harder to make progress?
The field will continue to make robust progress. For individual researchers, it can be harder because there are many researchers. But if you have compute, you can make discoveries.
Let's imagine all breakthroughs in the next 30 years. Do you think most can be done by one person with one computer?
I think it's pretty unlikely. The stack is deep. But there will be many breakthroughs that don't need huge compute.
Can you talk about the deep double descent paper?
Yes. Some researchers noticed that when you make the network larger, it works better, contradicting statistical ideas. We showed that double descent occurs for all practical deep learning systems. If you increase the size of the neural network slowly while keeping the dataset fixed, and you don't do early stopping, then when the network is small, performance increases rapidly, then gets worse at the point of zero training error, and then improves again. It's counterintuitive because you'd expect monotonic behavior. The intuition is that when the dataset has as many degrees of freedom as the model, there is a one-to-one correspondence, so small changes in data lead to large changes in the model. When you have a lot more data or parameters, the solution is insensitive to randomness.
Do you have any intuition why double descent happens?
The intuition is that when the data set has as many degrees of freedom as the model, there is a one-to-one correspondence, so small changes in the data set lead to noticeable changes in the model. When you have a lot more data than parameters or a lot more parameters than data, the solution is insensitive to small changes.
Geoff Hinton suggested we need to throw away backpropagation and start over. What do you think?
He said that to the extent you can't find backprop in the brain, it's worth seeing what we can learn. But backprop is very useful. I'm a big fan of backpropagation. It solves a fundamental problem. I think it's unlikely we'll have something dramatically different.
Can neural networks be made to reason?
Why not? Look at AlphaGo. The neural network plays Go better than 99.9% of humans. That's an existence proof. Humans can reason, so why can't neural networks? I think they are capable of reasoning, but they solve the problem in the easiest way possible.
