If you look at my academic history at Cambridge, I switched subjects every year and it didn't make any sense at all. When you look back on it, it was all very useful, but at the time it was just a crazy random walk.
I can relate to that. Isn't it marvelous how indecision in what might be called one's career path can in the end with the passage of time look like requiring sensible background? And in Geoffrey Hinton's case, boy did he flourish once he did eventually find the question that he was really interested in addressing. And it led him to become one of the founding fathers of artificial intelligence. So join me now as we listen to him recount how he explored some of those blind avenues and then eventually found the light at the end of the tunnel. Light which for him has over recent years turned a bit darker.
This is Nobel Prize Conversations and our guest is Geoffrey Hinton, recipient of the 2024 prize in physics. He was awarded for foundational discoveries and inventions that enable machine learning with artificial neural networks. He shared the prize with John Hopfield. Your host is Adam Smith, chief scientific officer at Nobel Prize Outreach. This podcast was produced in cooperation with Filt and Nobel Prize Outreach. Geoffrey Hinton is co-founder and chief scientific advisor of the Vector Institute in Toronto. He talks to Adam about how to best cool your coffee and why we say bite down instead of up. But ultimately they reveal how these mundane musings cut to the core of the question, what makes us human? But first, Geoffrey Hinton talks about growing up with a father who favored bugs over his children.
You come from a very scientific background. There are a lot of scientists in your forebears and your father was a great entomologist. Did that influence you to become a scientist? Did you feel that it was in the blood?
Yes, I felt a lot of pressure to become a scientist. I was expected to become a scientist, but I always enjoyed science so that was okay. When I was little, I wanted to be an entomologist because my father was an entomologist.
He shared with the Lord an inordinate fondness for beetles.
Oh, you read that, right? Yeah, it was beetles he specialized in. And lots of other insects, too. But beetles were his favorite. And among beetles, it was the family Elmidae that was his real favorite.
Did you have to compete with beetles for his favor?
My sister often said that he would have loved us more if we had six legs. And in his office, he had an office at the university that was very high. The ceiling must have been about 16 ft high. And the walls were all lined with shelves and he had a step ladder so he could get at them. And the walls were covered with hundreds of boxes. And each box would have papers, reprints. This was before Xerox machines, right? It would have reprints of papers from journals. And on the box would be the name of the genus of insect that the reprint was about or maybe the family. And so there were hundreds of these boxes. When he died, we sold them to the University of Florida for £10,000. Among those boxes, there was one box that was a bit smaller and was next to the door on a lower shelf. And on that box it said 'not insects'. And that was the rest of his life. And that had things like letters from his children.
And well, at least you were differentiated from the insects. You were something special.
But if you were expected to be a scientist, did that necessarily mean that you wanted to follow expectations or did you want to break away and be something different?
Not until I was late in my teens did I want to break away and do something different. So when I first went to Cambridge, I was doing science and it took a lot of time to do it properly. You went to lectures in the morning, did experiments in the afternoon and then in the evening I would rewrite the notes I've made during the lectures to make them neat and so that I went over it once more and then it would be time to go to bed. And it was just it was like 12-hour days I was doing and after a month I just got totally fed up with it and I left. That was the first time I made a decision on my own.
Golly, that must have gone down badly.
That went down quite badly. Yes, I went back a year later.
But what did you do with the time that you had between first start and second start?
I went to London and got several different menial jobs in order to pay the rent on an apartment. And I read a lot of depressing novels by people like Dostoevsky.
Sounds like independence.
Yes. I decided I wanted to be an architect. Fortunately, before I actually went back the next fall, I spent the summer working in an architect's office and discovered what architects actually do, which isn't as romantic as what you think they do. Let's sketch out airy buildings. What they actually do is decide, are you going to have cheap door handles or cheap flooring? Because there's no way you're going to meet the budget without doing one of those two. And after a day of doing architecture, I went and talked to my tutor and switched back to doing science. But I switched back to doing physics and chemistry and physiology. The first time I've been there, I did physics and chemistry and crystalline state which was a new subject they were teaching because of the success of X-ray crystallography in getting the structure of DNA. I think X-ray crystallography was a big thing then and when I went back I didn't do that. Instead I did physiology and that was the first time I'd done any biology and I found it fascinating.
And had you found your path when once you made the change to physiology?
No. At the end of the physiology course, there was a section which I was really looking forward to about the central nervous system and I thought they would tell us how it worked, which I was very interested in how it worked and instead they told us how the axons of neurons conduct action potentials, how a wave of depolarization goes down the axon. But that didn't exactly tell us how it worked. And I thought they were going to tell us how it worked. And so I got very fed up and I switched to doing philosophy, which is definitely studying how it works. But I thought I'd learn more about the mind. I didn't. I did a year of philosophy and basically developed antibodies to philosophy.
There seems to be a bit of a theme here.
If you look at my academic history at Cambridge, I switched subjects every year and it didn't make any sense at all. When you look back on it, it was all very useful, but at the time it was just a crazy random walk.
So often in life, that's true. You can make sense of it in reverse. But yes. Well, presumably you did quite well in all these things. So they hang on to you and said, 'Okay, no.'
In my first year, I did well in physics, but I knew I couldn't carry on in physics because my math wasn't good enough. Advanced mathematics I found very difficult. I did quite well in physiology and okay in chemistry. Then when I did philosophy, I did just okay. And then I switched to psychology and I didn't really like that at all.
Again, were you looking for a way to get into the mind?
Yes. I thought psychologists would tell us how people worked and instead it was rats in mazes. And there was some stuff that retrospectively is interesting. Signal detection theory, how you distinguish between noise and very faint signals. And there's some interesting mathematics there, but it wasn't exactly what I wanted to know. I wanted to know how people worked.
So, Adam, did Jeffrey Hinton find out how people work?
Don't think anybody's found out how people work yet. I think the brain's just too complicated for us just at the moment. He talks about two things in the conversation that I need to understand better. And that's neural networks and large language models. Can you enlighten me a little bit?
Well, yes, I'll give it a go. The neural network is a step towards understanding how people work in that it's arranging processes in a way that seeks to mimic what happens in the brain. In the brain, you have neurons arranged broadly in layers and hugely interconnected. And in a neural network, you have processes arranged in layers. And there's an input layer where information comes in and there's an output layer where information goes out. And then in between, there are these hidden layers. And in the brain, our understanding is that neurons reinforce their connections with each other when they send electrical signals to each other. So standardly people say neurons that fire together wire together. And the same is happening in a neural network where nodes with inputs that are leading to the right answers further down the stream are reinforced. And you program a neural network with a set of rules and then it turns out that they are so-called adaptive and that they can develop new rules for themselves allowing them to get close to the right answer. So building a synthetic brain then basically. Yes. Exactly. And that's what he was seeking to do and very few people thought he was going to succeed but he did and that led him to become as everyone likes to say the godfather of AI. And then a large language model. Well, that broadly describes the way that these infrastructures are used to input huge complex data sets and then the model is able to recognize and interpret that data set and give you an output which makes sense. As the name large language model implies that refers initially to producing text as we all know from something like ChatGPT but it can also work with visual data or any sorts of complex data and in the case of one of the Nobel prizes awarded last October. Of course, that data was protein folding data which AlphaFold 2 was able to interpret. And what specifically did Jeff Hinton get his Nobel Prize for? For the development of that initial neural network which had introduced concepts of statistical mechanics which allowed his neural network to recognize characteristic elements in sets of data. Apart from sort of creating a digital brain, Geoffrey Hinton himself also seems to have a very interesting mind. Doesn't switch off much.
Yeah, he seems to be constantly on the lookout. Before we started recording, somehow we started talking about religion and I said, 'Oh, let's not go there.' And he said, 'Why not?' So we went there. We chatted about it a little before the recording started and I think it was just indicative of the fact that yeah, not much slips by. So what was his take on religion then?
Yeah, he's not keen on it. He thinks it's all fake news that those who believe are deluded. But then again, there are others who very much believe in an evidence-based view of the world who have space for religion as well. So he doesn't seem to mind to disagree with people.
He certainly doesn't. And I think again it's the characteristic of so many laureates. Not that they necessarily want to hold contrary views, but that they are not fazed by having people criticize their view of things. This conversation about religion is well worth having, but I don't think it's a conversation that can be had by scientists alone. Indeed. Anyway, that mind is constantly operating, even in the dentist chair, it turns out. Let's listen.
At least in Canada and Britain, I don't know about elsewhere in the world, when you go to the dentist and they want to see if your lower teeth fit your upper teeth properly, they say bite down. Now, you don't actually bite down, you bite up. So, why do they say bite down? And they all say it. And I've asked dentists, why do you say bite down instead of bite up? And they don't have an answer. They spend their whole life saying bite down. And it's never occurred to them that, well, you actually bite up. So, why do you say bite down? And oh, you want to know why they say it?
Okay. Well, I was once eating some bony fish. This really is an explanation why they say bite down. I'm with you. Yes. I was once eating some bony fish in a cafeteria with a friend and we were trying to play blindfold chess. Now, we weren't that good at chess and we kept sort of moving bishops through pawns and things. But we were trying to play blindfold chess and I realized you can't play blindfold chess while eating bony fish. And the reason is the sort of spatial processor you use for dealing with where things are in space and how they're relating to each other is the same spatial processor you use both for playing the blindfold chess and for finding the bones in the fish. It's not like you have a separate piece of apparatus for dealing with what's going on inside your mouth and what's going on in this chess game. It's the same spatial processor you're using. So you get a lot of interference. You can't do both at once. It's like trying to hold two conversations at once. So the question is when you use that spatial processor for dealing with things inside the mouth, how does that relate to how you use it for dealing with things outside the mouth? So if for example I'm reaching for objects, there's a kind of me and I'm doing the reaching for the objects. If I'm thinking about what's going on inside my mouth, what's me and where's the sort of center of my frame of reference? So inside your mouth, your tongue is you. The tongue is the bit you can move. I mean, you can move the jaw, too, but the tongue is you. And you sort of feel around with the tip of your tongue. And if you're looking for bones in the fish, you sort of feel for them with the tip of your tongue. So the tongue is you. Now the tongue is attached to the lower jaw. And so if you ask what moves relative to the tongue. Well, what moves relative to the tongue is your upper teeth. When I contract the jaw muscles, so my jaw closes, the tongue doesn't move relative to the lower teeth. There's no danger that you'll bite your tongue with your lower teeth because they're not moving relative to the tongue. The thing that bites your tongue is your upper teeth. And of course, what you need to worry about inside your mouth is biting your tongue. That's the most important thing not to do. You are your tongue and you don't want to get bitten. And these upper teeth are coming down on your tongue. And so, of course, you think of it in terms of biting down. It's all about frames of reference. It's all about frames of reference, which I'm very interested in. And that's my best explanation of why dentists say bite down. But the point of this whole story is there is an explanation of why they say it that I think is pretty good. But nobody ever asks the question. Nobody ever says, 'Wait a minute, the teeth come up, so why do they say bite down?' There's huge numbers of questions that nobody ever asks. So my son, when he was very little, asked a question that very few kids ask. He said, 'Daddy, why do bridges stay up?' And the point is, there's nothing underneath the bridge. And he was just beginning to understand that it is odd the bridges stay up. It's a lovely good question.
All this asking of questions and finding solutions that can slow you up. I suppose most people are rushing through things and stopping to ask is time-consuming.
Yes, it is. As well as being intellectually taxing. I've always envied people who can just read a lot of stuff. If I start reading a scientific paper, I keep getting sidetracked. It takes me a whole day to get through a scientific paper because I'll read a little bit and think, 'Wait a minute, wait a minute.' Or it'll remind me of something else. And so yeah, I get sidetracked a lot by asking questions. And I actually like making up questions just for the fun of trying to figure out the answers. So for example, if you take a coffee cup and you put the coffee in, then you have to go and do something. You know you're coming back in 5 minutes and you want the coffee to be as cool as possible in 5 minutes. Should you put the milk in when you've just put the coffee in or should you leave it and put the milk in when you come back? And the answer is roughly speaking that you should leave it and put the milk in when you come back because the coffee will be hotter if you haven't put the milk in and so it will cool faster. So you'll be losing heat faster if the coffee doesn't have the milk in. Then you put the milk in when you come back and it cools it down some more. But it'll end up coolest if you put the milk in later. But suppose that the coffee cup is conical-shaped because it's losing heat from the surface of the coffee. And so if you put the milk in, you get a bigger surface. So although the coffee overall, if the milk mixes with the coffee, the overall thing's a bit cooler, it's got a bigger surface. So that must mean there's some shape of coffee mug where it doesn't matter whether you put the milk in first or later, it'll cool the same amount. It depends a bit on sort of exactly how much coffee you put in and how much milk you put in. But I think most people don't get sidetracked by wondering things like if we had the right shaped coffee mug, we wouldn't have to worry about whether you put in the milk first or second.
Absolutely not. I think that sounds like a wonderful physics project for some undergraduate to work out what the ideal shape of cup is. But yes, it sounds like the sort of question which really normally elicits the answer, 'Oh, dad. Honestly.'
