Ep. 2 - Five Decades of Neural Networks with Geoffrey Hinton

Thanks for inviting me.

Slightly rocky. So, I went to university. I studied physics, chemistry, and physiology. And in physiology in the last term they were going to teach us how the central nervous system worked and I was very excited and they taught us how action potentials are conducted along an axon which wasn't what I meant by how it worked. And so I switched to philosophy, that was even less useful, and then I switched to psychology which was completely hopeless. And then I became a carpenter. And after I'd been a carpenter for about nine months, I met a real carpenter. And he was so much better than me. I decided it'd be easier to be an academic. So I went to graduate school in Edinburgh with Christopher Longuet-Higgins who had published interesting stuff on using neural nets for memory. Unfortunately, around the time I arrived, Winograd's thesis came out and he switched his allegiance to symbolic AI and gave up on neural nets. And so I spent five years as his graduate student with him trying to persuade me to give up neural nets and he never succeeded. In the end he was very helpful to me. But for a long time there was a lot of argument about how I should really be doing symbolic AI and all this neural net stuff was complete nonsense and everybody else in Edinburgh believed that neural nets were nonsense with actually a couple of exceptions. There was a postdoc called David Wilshaw who'd done associative memory and he'd basically done something quite like Hopfield nets but a long time before Hopfield and Aaron Sloman was a visitor for a while and he was more sympathetic. But basically they all knew it was rubbish and they would explain to me how neural nets can't even do recursion. So because everybody believed in recursion then, I actually figured out how to do true recursion in a neural network and implemented it on a machine with I think by then it had 192 kilobytes of memory and it was only shared by 40 people. But it had a huge disk that had two megabytes. So you never ran out of memory because it used virtual memory. And I actually implemented a little neural net that did true recursion. That is in the recursive call, it used the same neurons and the same connection strengths for the recursive call as it did for the higher level call. Now to do that of course it had to offload all the parameters of the higher level call into some short-term memory onto a stack eventually and I figured out how to implement a stack with associative memory in a neural net. So I had this little neural net running that was doing full recursion in neural nets and that was the first talk I gave and people were very puzzled. They said why would you want to do recursion in a neural net? I mean, it's so easy to do in Pop-2, which was our sort of unfortunate bastard child of Pascal and Lisp. Although I don't think Pascal existed then. So yeah, I keep meaning to go back to that. I never...

Oh yes. I mean to do true recursion you have to use the same neurons and weights for the recursive call. That means you have to have a stack something like a stack to store the parameters of the high level call. That all works if you have fast weights. So that was the first thing I did with fast weights in 1973. I should say fast weights were invented by Jürgen Schmidhuber in 1991.

Oh, no. No. I dropped out again after I finished my thesis. I dropped out and became a teacher in a free school in London. It was voluntary. I was unpaid. They were rough, emotionally disturbed inner city kids. And after a few months of that, I again decided academia might be easier. So I went back to a postdoc with Aaron Sloman in Sussex. Longuet-Higgins had moved from Edinburgh to Sussex and as I was finishing my PhD I got a postdoc with Aaron Sloman. And there there were no proper faculty jobs in Britain. Then there was one job in the whole of Britain which Alan Bundy got. And so I applied for jobs in the states and I got a job as a postdoc in UCSD with Don Norman and Dave Rumelhart and I really got along very well with Dave Rumelhart and that made a huge difference. So I'd moved from a country where it was a sort of small country Britain and there was only room for one ideology and the ideology was symbolic AI and neural nets were just rubbish and I moved to the states where on the west coast, on the east coast it was symbolic AI but on the west coast they were kind of more open and in particular Don Norman and Dave Rumelhart thought neural nets were worth considering. So it was a huge liberation to be in a place where neural nets were regarded as not obvious nonsense. And I went there, I got to meet Terry Sejnowski who I invited to a conference and we've been sort of lifelong friends and collaborators. I got to meet Francis Crick later on who was there. So I was there for a couple of years and then I got a job in Cambridge in the Applied Psychology Research Unit where I was meant to do applied psychology and I was strongly reminded of William James's comment about applied psychology which is to do applied psychology you have to have something to apply. But I actually did some interesting stuff. It was just around the time some workstations were coming out and they had a contract with the British telephone company to help with network management. And network management then was all done by hand and you had information about the loads on various switching centers and the information was on a huge wall that was 20 ft high and worked like you sometimes see at train stations. It was little flaps with white letters on that come down. They sort of rotate around till you get the right flap. And so you could see all these numbers that said how busy each switching station was. And I figured a Sun workstation could do that and would be a lot cheaper. And the resolution wasn't that good then. So I had to figure out if you could display the states of all the switching stations in Britain on the screen of a Sun workstation. And you couldn't type the names, but if you had two letters for each name, you could get two letters there. And I worked on a display with two letter names. And there were a large number of switching stations. There were hundreds. And the question was, could an operator remember which they were? So I actually taught myself to remember all those two letter names. They were in very small type to fit on and I actually got a serious migraine from looking at it too long. That was my interaction with human factors.

Yes. So I wrote a report on it and they said it was a very nice report. Thank you very much. And they weren't going to implement it even though it would have been much more efficient and much easier to update. And I said why not? And they confidentially explained to me that well when people come and visit the network control center or when actually when they visit the headquarters of British Telecom they have to have something to show them like the politicians have to see something and they would always show them this huge wall that displayed the state of all the networks of all the switching stations and they were very impressed by that. And if they got rid of the huge wall and had just some workstations, they were very worried that network management would get less funds from British Telecom. So they were going to keep their huge wall. I learned a lot then about applied research. It's not about whether it works, it's about whether the company likes it.

I went back to San Diego for six months and that's when we worked on the PDP books with Dave Rumelhart and Jay McClelland. I was one of the authors until almost when they were published and at the last minute I dropped out because at that point I decided Boltzmann machines were the future. Boltzmann machines was just a much better idea than backprop and backprop was a silly idea and Boltzmann machines were a much better idea. And there was no point being an author of a book where the main thing was backprop. That was a mistake. But in 1984, I figured out, yeah in 1982 then I applied to CMU and because it was a private university you could just, they didn't have to sort of advertise very widely. And Scott Fahlman was sort of my host. He sort of interacted with him at many workshops and we got along well and he pushed hard to get for me to get them to go there and I had a very funny interview. So I went there on the first day there I gave a talk in computer science and then Scott Fahlman took me out for lunch at a place it might have been called the O. I can't remember what it was called, but it had a motto, which is if you don't get sick, you got a bad one. And I got terribly sick. And the next day I had acute diarrhea. I couldn't eat anything. I was living on coffee and Coca-Cola. I gave a talk in psychology about mental imagery and my theory of mental imagery. And then there was someone at the end who asked a question which I didn't understand to begin with. And then I realized the question he was asking was did I believe the theory of someone called Marcel Just about how you weren't really rotating an image in your mind? You were just looking backwards and forwards between two things. And in my reply I said, 'Oh, I see you mean that silly theory by Marcel Just,' where I didn't realize it was Marcel Just asking the question. And after that I got a request to go and see Nico Habermann. Now, Nico and I were always great friends, even though we were politically extremely different. I was a sort of lefty 1960s radical with long hair and rather disheveled. Nico was a European gentleman who was very nicely dressed, worked with the defense department, set up an institute I wasn't allowed to go to because I was a foreigner. But we got along very well and I think it was because of our initial interview.

Yes. And so in the initial interview he said so we've decided to offer you the position. And I said oh there's something you should know. And he said oh what's that? And I said well I don't actually know any computer science. And he said it's okay here. It's okay. We have people here who do. So I said, 'Okay.' So I said, 'Okay, in that case I accept.' And Nico said, 'Don't you think perhaps we should talk about the salary?' And I said, 'Oh, no. I'm not interested in the salary. You can pay me whatever you like. I'm not doing it for the money.' And he said, 'Well, how does 26,000 sound to you?' I said, 'That sounds fine.' I discovered I was being paid 10,000 less than the next lowest paid professor, 10,000 less. But every year I got a big pay rise and me and Nico got along very well after that because he knew I wasn't doing it for the money.

Yes, with backpropagation. I mean, we didn't invent it. It was invented by several different groups, but we showed that it really worked to learn representations. And as you know, sort of one of the big problems in AI is how do you learn new representations? How do you avoid having to put them all in by hand? And my particular example, which was the family trees example, where you take all the information in some family trees, you convert it into triples of symbols like John has father Mary. And then you train a neural net to predict the last term in a triple given the first two terms. So it's just like the big language models. You're predicting the next word given the context. It's just much simpler. I had 120 total examples of which 104 were training examples and eight were test examples which is a bit less than the trillion examples they have nowadays. But it was the same idea. You convert a symbol into a feature vector. You then have the feature vectors of the context interact via a hidden layer. They then predict the features of the next symbol and from those features you guess what the next symbol should be and you try and maximize the probability of predicting the next symbol and you then backpropagate through the feature interactions and through the process that converts the symbol into features and that way you learn feature vectors to represent the symbols and how these vectors should interact to predict the features of the next symbol. And that's what these big language models do except it's a bit more complicated. The feature interactions are much more complicated. They have many more layers of interactions. So they can disambiguate ambiguous symbols and refine the shade of meaning of things where the meaning depends a lot on the context. But it's basically an extremely simple version of the current large language models. I called it a tiny language model. And that convinced the editors of Nature that we really could learn interesting representations because the vectors I learned for the symbols which were people and relationships, they had six components. And if you use weight decay, you could interpret what all those components were and they were sensible semantic features. They were the nationality of the person and the generation of the person and which branch of the family tree they were in. And so it would learn things like the relationship uncle requires the output person to be one generation older than the input person. And so it would have generations for people. And if the input person was generation two, it would predict that the output person would be generation one. So it was actually only a whole bunch of little rules just probabilistically. And the people interested in rule-based induction got interested in it because they said, 'Oh, we can do that, too.' And it's true, they could do that too with rules that weren't probabilistic. The point about neural nets is they can mimic something that learns discrete rules, but they're also perfectly happy if the rules are just usually true and they use the preponderance of the evidence then, which is much harder to do in logic. And so it was that example which curiously was a little language model that convinced the editors of Nature to publish a paper. I know because I talked to them later, the referees. I talked to the referee, one of the referees later and he said, 'Yeah, it was that example that did it.' And then we were all very excited. We thought we can solve everything. You just have to give it a lot of training data and run backprop and it'll learn all the representations you need and it'll learn to do parallel computation because that time people were very interested in parallel computation but it was quite hard to program and the idea was well this will have all these neurons inside and they'll all be operating in parallel and it'll figure out how to use them so there aren't any problems. At that point people were very interested in races and things like that and you didn't have to worry about any of that. It was all synchronous and they just learn what to do. So we thought we'd solved everything and little did we know we had. It's just we needed more data and more compute.