So I should say, I was once offered a cross appointment in statistics, and I declined it on the ground that I don't know any statistics. If I was consistent I would decline this one too, but I'm not. So back in the 1980s, I worked on two different learning procedures, and there's something slightly embarrassing about these two procedures. One of them was very interesting, and the other one worked. So I'm gonna start off by talking about the one that worked, which was just a dumb application of the chain rule of differentiation, and I'm gonna talk about that for about 20 minutes or so, because that actually worked, and that underlies current AI, current deep learning and AI, and I'll talk about a couple of the things we did there. I feel I need to do that to be honest about what actually worked. Then I'll talk about the other learning procedure, which is much more interesting, and uses statistical physics, and didn't actually work very well, but it's very interesting. Okay, here goes. So in the history of AI, there were two approaches, going back to the 1950s. There was the logic-inspired approach, where the idea was... No, I can't see the slides. Where the idea was that the essence of intelligence is reasoning and we need to study how reasoning works, and reasoning is done by manipulating symbolic expressions. The other approach was neural nets, where the idea was, the essence of intelligence is learning connection strengths in neural nets, and don't worry about reasoning until you figured out how the brain learns. And Turing and Von Neumann both believed in the second approach, but they both died young. And so, the first approach dominated for 50 years. I'm gonna very quickly explain what a neural network is, in case there's people here who don't know what they are. You have layers of neurons, which are relatively simple devices, in this work. Real neurons are very complicated, but we make gross simplifications. So in that sense, I'm a physicist. I sort of ignore all the details. Physicists like horses to be spherical. And you have multiple layers of these things, you have connections between neurons, and you want to learn the connection strengths, so when you put something in, the right thing comes out. And typically what you put in is the pixels of an image and you'd like to come out the category of the object in the image, a label for the image. So everybody can understand that you could learn a neural network in the following way. Pick one of the connection strengths and change it a little bit, and see if the whole thing works better or worse. That's the sort of mutation method. And that'll definitely work, and it'll take a huge amount of time. 'Cause nowadays, neural networks have a trillion weights. So you'd have to, a trillion different times, pick one of the weights, change it a little bit, see if things work better. And to see if things work better, you need to give it lots of examples. 'Cause it might make it work better for some examples and worse for other examples. So it's hopelessly slow. Back propagation is essentially just doing the same thing but figuring out, for all the weights in parallel, whether increasing them a little bit or decreasing a little bit will improve the performance. So in back propagation, you send a signal forward through the network, you look at the difference between what you got and what you wanted, and then you send a signal backwards. And by doing this and using the chain rule for differentiation, you can figure out, for every connection at the same time, whether you should increase it or decrease it a little bit, and that's what you do. And amazingly, to a lot of peoples' surprise, that works really well. It works really well if you've got a lot of data and a very fast computer. So in 2012, two students in my lab, Ilya Sutskever, who you may have heard of, he just started a company that's producing nothing and is now worth $30 billion. I suggest the dean might like to contact him. So Ilya Sutskever and Alex Krizhevsky was an amazing programmer. They made a system that could recognise objects and images much better than standard computer vision. It got nearly half the error rate of standard computer vision. And that kind of opened the flood gates for neural networks. People were using them already, particularly in speech, but the performance of this network made it clear that they really do work, and they really do work on real images. And everybody in computer vision over the next year or so switched to using neural networks. A couple of years before that, papers on neural networks were just routinely rejected by computer vision conferences, 'cause everybody already knew this stuff was nonsense. And then a few years later, nearly all the papers were about neural networks. So that made a big impact, and at least it was done by students in my lab. So I'll get a lot of credit for that. In fact, I was asked by the computer museum, which now has a display of the AlexNet, I was asked by the computer museum if I could provide a quote for them. And so, the quote that I agreed with Alex and Ilya was... It was Ilya's idea to do it, Alex made it work, and I got the Nobel Prize. Okay, but what about language? So, many people in symbolic AI said, this stuff will never work for language, that's quite different. And if you look on my webpage, you can find quotes from them saying how this will never work for language. You can also find GPT4 explaining to these people what was wrong with their quotes, which is fun. There was a linguist called Chomsky, who you may have heard of, and he's a sort of cult figure. He managed to convince people that language isn't learned. It's pretty obvious that language is learned. In the same way it's obvious that Obama had a bigger crowd. But if you can convince people that language isn't learned, then you've got them. And several generations of linguists are convinced that language is not learned. And Chomsky never had a theory of meaning. His theories are all about syntax. He of course thinks all this stuff is complete nonsense, even though it can have a pretty good argument with him now, but he still thinks it's complete nonsense, it's not really understanding anything. So the linguists thought the idea of a big neural network just using data to learn syntax and semantics was completely ridiculous, it would never work. And for many years, it did never work. But eventually, with big networks, it worked very well. And that's what you see in these large language models. And I want to talk a little bit about the origin of these large language models and what meaning is. So there's two very different theories of the meaning of a word. One theory is that the meaning of a word depends on how it relates to other words. If you have a bunch of sentences with multiple words in, how it's distributed with other words determines its meaning. That's a theory that comes from de Saussure, more than a hundred years ago, and that's what the symbolic AI people basically believe. And to capture meaning, you probably need something like a relational graph. The other theory comes from psychologists, and it's that the meaning of a word is a big bunch of features. So Tuesday has some huge bunch of features, and Wednesday has some huge bunch of features, and they're quite similar bunches of features. Now, those two theories sound very different. But in 1985, in order to convince people that back propagation actually worked, I developed a system, a tiny language model that unified those two theories. So the idea is that you're gonna use features of words to predict the features of the next word in a string of words, and from the features of the next word, you'll predict the word. But the only thing stored within your system will be how to convert a word into features, and how the features of words in a context should interact to predict the feature of the next word. You don't actually store any strings of words. So the symbolic AI people thought you store string of words in your head, or strings of symbols, and you manipulate those with rules to get other strings of symbols, just like in logic. This is a completely different idea of what meaning is. You convert a word into a bunch of features. You know how features interact with features of other words to predict features of the next word. And what meaning is, is taking a string of words and converting those words into the appropriate features. And that may be difficult. You might have a word like may, which could be a month, or a woman's name, or a modal, and you have to use interactions with neighbouring words to decide what features to use for may. But essentially, the large language models that you have now, the way they work is, they take words, actually fragments of words, and they convert those into vectors of features. They then have many layers of refining those vectors of features, by doing interacting to disambiguate things. And then they predict the features of the next word. And from those features, they can predict the next word. And so, they work just like the small language model, except they're much bigger, and the way the features interact is much more sophisticated. So I'm gonna show you, because it's something I did, the tiny language model from years ago, 'cause that's very easy to understand. The big language models, it's hard to understand how they're representing knowledge. The tiny language model, it's easy to see how it's representing knowledge. So I took some relational knowledge, two family trees, and you can convert the information in those family trees into three-word strings, so triples of words, if you use relationships like son, daughter, and nephew, niece and so on. You can have three word strings, like Colin has-father James. And from those strings, if you learn some of the strings, you can now predict other strings. That's called generalisation. The way people in logic would do it is, they would say, you have to have rules that look like this. So you have to have a rule that says, if X has mother Y and Y has husband Z, then X has father Z, or the other way around. And they thought what you should do is search for those rules. Now, that's fine if there's discreet rules and things are always very certain. As soon as you have some exceptions and some uncertainty in the data and errors in the data, searching for discreet rules is much harder. And the idea was, we get a neural network to find those same rules, but by searching a continuous space. There's an interesting thing about continuous and discreet in computer science. For a long time, computer scientists defined themselves as, we're not interested in real numbers. We're only interested in discreet things, like graphs, 'cause we're real computer scientists. And then the mathematicians explained to them that actually, even if you want to do discrete things, using real numbers, using big matrices to understand discrete structures is a good idea. So, the point about neural networks is, they use real numbers and they search in that space, and searches are much easier in that space. So I used a network like this. It has two inputs, a person and a relationship. It converts those into features. Actually only six features for each of them, 'cause it was a tiny computer that goes billions of times slower than the computers we have now. This was in 1985. And then it takes the features of the words, they interact, they predict the features of the next word. And the whole thing is trained by simply back propagating from the output, the error in the output. You send a signal backwards and it tells you how to change the connection strengths that turn symbols into feature vectors and make these feature vectors interact. And it learned interesting features that made sense. So you could see how it worked. So for example, for people, if you think about those two family trees, these were 1950s family trees. There was no divorce, there was no adoption, nothing funny, just very, very straight American families from the 1950s. And so, for example, Italian people would only marry Italian people. And so, if you know the nationality of the input person, you know the nationality of the answer. So it's a very good idea, when you convert a symbol into features, to have nationality be one of the features. It's not explicit in the training data, but it's a strong regularity that Italians only marry Italians. So it learns the nationality feature. It also learns features like generation. And for relationships, it learns features like, the answer should be one generation up from the input. So if you're looking for somebody's uncle, that should be one generation up from them. And because for relationships it knows things about, some relationships require you to be one generation up, and because it's learned to extract the generation of the input person, it can then predict the generation of the output person. And so, the way it gets to the correct output person is by predicting a bunch of features of the output person, like their nationality and their generation, and which branch of the family tree they're in, and from that it can predict the output person. So it actually worked. It didn't have much impact at the time. I thought this was terrific, 'cause I've managed to unify the psychologists' theory of meaning, which is a big bunch of features, with the sort of structuralist theory of meaning by getting the features to produce the sequences, and by using sequences to learn the features, so you could go from sequences to features and from features back to sequences. But the symbolic AI people weren't very interested. But over time, people got more and more interested. And I did get a review that I'm still annoyed about. Every scientist has some review they're really annoyed about. I got a review in 2007 that said, I was talking about turning words into features and having the features interact, and the review said something like, "Hinton has been doing this stuff for many years and nobody's interested. It's time to move on." That was 2007, when it was already beginning to work. Okay, and I'm gonna finish this stuff about back propagation by trying to give you an analogy of how meaning works, how words work to give you meaning. So what Chomsky didn't really understand is, you have to have models of things, and words are a particular way of building models. They're actually a much better way of building models than anything statisticians have, 'cause you can make very sophisticated models with them. So think of it like this. Suppose you want to know where stuff is in 3D, like you want to model the shape of a Porsche and you're not too worried about aerodynamics, so you don't care about the surface. You can make it out of Lego blocks and you can approximate it pretty well out of Lego blocks. Well, words are like Lego blocks, except for some differences. The first difference is, we have say 30,000 different types of Lego block. That's one difference. The second difference is, they're not rigid, they can move about a bit. If I tell you the name of a word, it defines a shape, but the shape has some flexibility to it. It might of course define several alternative shapes, but it's not infinitely variable, I mean, infinitely malleable. The word defines a rough shape. The words then interact with each other to determine the shapes of the individual words, and you can think of that as, the words need to fit together somehow. Now with Lego blocks, you have a little plastic cylinder that goes into a little cylindrical hole and they snap together. The way you do it with words is, every word has a whole bunch of hands on it, and as you change the shape of the word, you change the shape of those hands, and words have to shake hands with other words, and they have to decide which other words to shake hands with. Then they shake hands with other words whose hand shapes match their hand shapes. So, think of getting the meaning of a sentence as you have these words which have these somewhat flexible shapes, and as you change their shapes, their hands that they shake hands with all change shape, and what you have to do is find out how to deform them so they can lock together nicely, and when you do that, when you deform these shapes and they lock together nicely, that is the meaning, it's a way of modelling things, and that is what a meaning is. There's just one other difference, which is that these words are like thousand dimensional instead of three dimensional, but we're not gonna make a big deal out of that. Thousand dimensional spaces, the way you think about them is, you think of a three dimensional space and just say a thousand. I think that must be what physicists do too, but I don't know. Maybe four dimensional spaces they can imagine. When you get up to 11 dimensions, I don't think they can imagine it. Okay, that's it for the first part of the talk. I talked a little bit about back propagation. I talked about two of the things we did on back propagation, a tiny language model and recognising objects and images, and those were both very influential in the development of the use of back propagation, and that's what led to, I mean, they were very influential pieces of work that led to the neural nets we have today, the deep neural nets that do everything. Now, an unfortunate thing about back propagation is, it's hard to see how you could do it in a brain, and I was always interested in how the brain might actually be doing it. That was one of the points of doing this. So if you ask, what's back propagation like as a model of how the brain learns? Well, nobody's found a good way to implement back propagation in something neurally plausible. We can do it for little systems, but as soon as you take a big system, we can't implement in brain-like stuff, anything that works as well as back propagation. The cortical neurons send binary signals to each other. They don't send real values like we do in back propagation. The backward pass of back propagation. So you put in an image, you go forwards, you recognise something. If you know what it is that's there, you can send an error signal backwards. But you have to know what's there, and that's already a problem. And when you send this signal backwards, you're interfering with the pipeline of perception. We do perception in real time. The stuff is always coming in. You don't wanna have to stop and send stuff back. So that's another problem. A problem that's new because only in the last few years have we known this, back propagation works much better than the brain, and that's the reason for thinking it might not be how the brain does things. It works much better than the brain in packing a lot of information into a few connections. So these big language models have about a trillion connections, the biggest ones, and your brain has about a hundred trillion connections. Let's suppose 10 trillion of those are used for knowledge, which seems not unreasonable. We're physicists, so we just do things in orders of magnitude. So at least a 10th of them are gonna be used for knowing stuff. And things like GPT4 know thousands of times more than any one person. They're like a not very good expert at everything. So back propagation is actually much better at squeezing a lot of information into connections than whatever it is we got in the brain, perhaps because the brain's solving a very different problem. Our problem is, we've got lots of connections, like a hundred trillion, but we don't live very long, we only live for about two billion seconds. Luckily it's a little bit more than two billion, but about two billion. So we don't get much experience, but we've got a lot of connections. In these big language models, they have huge amounts of experience and not many connections. So back propagation is good if you have a lot of data and not many connections. Whatever it is our brain's doing is good if you have a lot of connections and not much data, and it's optimised for that, presumably. So this leads to the question, is there maybe a completely different way to get the gradient, a way of getting the gradient of how you should change your weight to make the system work better, that is, the gradient of the error the system makes with respect to the weight, that's just totally different from back propagation, and that means you don't have to keep interrupting pipelines and things? And it turns out there is, and that's where statistical physics comes in. 'Cause I'm gonna use a bit of magic from statistical physics to show you that there's a very simple learning procedure, which is actually very like a procedure recommended by the great Canadian psychologist, Donald Hebb, in around 1949, where he said, neurons that fire together, wire together. So his learning algorithm was, if two neurons are active together, increase the strength of their connection, roughly speaking. That doesn't work. If you just do that, connections only get stronger. He didn't say how connections get weaker. And the answer is, you have to do something else as well, and then that algorithm works. So I'm gonna start off by explaining things called Hopfield nets. Hopfield nets, as you might imagine, were developed by John Hopfield, and they use binary neurons. So rather than having real values now, we're gonna have binary neurons that are either on or off. Hopfield actually used one and minus one, but I'll talk as if he used one and zero. And that fits in with cortical neurons, which in a short period of time either send a spike or don't. Okay, so we've got these binary neurons, and we're gonna have symmetric weights between them, which is completely unrealistic. But we're physicists, so that's okay. So what symmetric weights means is that if, take those two neurons on the top there, when the one on the left fires, it sends an input of minus five to the one on the right, and similarly, when the one on the right fires, it sends an input of minus five to the one on the left. And so, we have weighted connections, the weight's always symmetric in a Hopfield net. And I'll call the global state of the network, that is, an assignment of ones and zeros to all the neurons in the network saying whether they're firing or not firing, I'll call that a configuration. And configurations have energies. If you wanna get rid of a few minus signs, we'll talk about configurations having goodnesses, and the goodness of a configuration is simply, for all pairs of active neurons, just add up the weights on the connections. So for the three neurons that are active there that have ones, you add up the weights, you get four. So the energy's minus four, the goodness is four. That same network has a different... And that's a local minimum. You can't change any neurons without raising the energy or lowering the goodness. But there's a different state of that network which has greater goodness or low energy, which is... Okay, I already said all that. Which is, if you take this same network and you turn on those three units, then the energy of that stage is minus five, the goodness is five, it's a better minimum. So the point is to show that even in a tiny net like this, you can have two different minima. Now, Hopfield realised that means you can use these minima as memories. So his model of what you're gonna do with a net like this is, you're gonna use it to store memories, and then you're gonna get a content-addressable memory. If I give you a little bit of a memory, you can fill in the rest by letting this network settle down. Sorry. I clicked twice. Yes. You can access a memory just from knowing part of the memory. Now, back when he did that in the early 80s, that was before Google, and content-addressable memory was something that seemed quite magical. As soon as Google came along and made big inverted indices, it was easy to do content-addressable memory and we're all used to that now. But this was a way of doing content-addressable memory in a neural network. So Terry Sejnowski and I went to a talk by Hopfield and we figured out, you could use Hopfield networks for something completely different. You could use Hopfield networks for doing perceptual inference, for looking at an image and interpreting what's in the image. And so, the idea is to have two sets of neurons. We'll have input neurons, where you represent what's in the image, I'll call these the visible neurons, And you have hidden neurons that represent an interpretation of what's in the image. And you'd like the energy of the whole system to represent the badness of the interpretation. So by finding low energy states of this whole network, you're finding good interpretations. And so, if you show it an image, what you're doing is, you're fixing the states of the visible units. Let's suppose it's a binary image. So you fix the states of the visible units to represent some binary image, and then you'd like the network to settle down by repeatedly going around updating the states of units to make them go to lower energy states, and you'd like to find a good interpretation of that input. And so, that raises a couple of problems. Oh, here's an example. I might give you an ambiguous line drawing, and I've sort of illustrated two possible interpretations of the Necker cube, one in which it's at a different orientation from the other one. And so, the very same input image can have two very different interpretations, which are more or less equally good. The one on the left's a bit better, 'cause it can sit nicely on the surface and the way the other one has to be glued to the ceiling. But apart from that, they're more or less equally good. And your brain can see both of those. And so, this is a nice example of a system that has two different interpretations of the same image. And it seems nice to model that by two different energy minima of the same system. So how can we make a system that can come up with those interpretations of an image like that? Well, first of all we have to think about, the image is gonna be made of lines. We just use images with straight lines. And we have to think about what a straight line depicts out there in the world. So I want you to imagine that out there in the world you have an edge of an object, and then you have a window between you and the object, and using a marker pen, you draw that edge on the window. So you can see there. Yeah, the laser doesn't work, I don't think. You can see there that I've drawn an edge on the window and there's two lines of sight go through the ends of the edge of the line that I've drawn on the window, and there's many different edges that could've given rise to that line. So that edge, because two lines of sight go through the ends of it, would've given you that line in the image. But so would this edge, or this edge, or this edge. And you can see, there's a whole family of edges that would give you the same line in the image. So you've lost information. And the point about visual perception is, you want to get back that information, and to do that, you need to use some assumptions. So to build this into a neural net, we know just from optics that you can only see one of those edges at a time. And so, we know that line in the image is gonna be one of those edges, but we don't know which one. So we can wire up a network by saying, let's suppose we have neurons that represent 2D lines, and we'll activate a few of those to represent the lines in the image, and we know that for each of those lines, only one of the possible edges is present. So what we'll do is, we'll have all the edges that might have given rise to that line, and the line will try and activate all the edges. Those are the green connections. But also, all those edges will fight one another, 'cause those are all edges that couldn't be seen at the same time, they get in each other's way. So what we really know when we see that line in the image is, one of those edges is present, but we don't know which one. And our neural network will try and capture that by saying, the line excites all those edges and the edges all inhibit each other. Inhibition is very important. The same for another edge in the image, like that. And so, that's got sort of optics into it. But now let's get some assumptions into it. So here's a test I sometimes do to distinguish computer scientists from physicists. If you give a physicist the following problem, X plus Y equals 384, what are x and Y? The physicists will think you don't know much about equations and you can't possibly answer that question, and the computer scientists will tell you, X and Y are 256 and 128, but he doesn't know which is X and which is Y. And that's 'cause in a computer scientist world, you're dealing with powers of two all the time, just 'cause of the way computers work, and so, they're very familiar with numbers like 384, and they know when you see 384 it's probably 128 plus 256, particularly if you program Nvidia GPUs. So we're gonna use the same trick here. We are gonna say, knowing what 3D edge is represented a line in the image is basically impossible, but you can use assumptions about the way the world normally is, and that can help you a lot to come up with a sensible interpretation. So what we're gonna do is, we're gonna say that if you see two lines in the image that join in the image, they probably join in depth in the world. They don't have to. From my point of view, I could have two lines sort of like this, where my eye just happens to line up so they appear to join but they don't actually join, but I'd have to be at a very special viewpoint for that. So it's a very good assumption that if you see two lines joining an image, they correspond to edges that actually join in depth at that point. So we can take edges that join in depth, 3D edges that correspond to different lines but join in depth at the point where the lines join, and we can say, they should support each other because they join in depth. Now, if they join in depth at a right angle, we can say they support each other a whole lot. 'Cause in this little world we're in, we tend to see right angles much more often than other things. That's actually a very bad piece of psychology. But it will be a whole nother lecture to say why it's not true that we prefer right angles. What we actually prefer is edges that line up with the rectangular coordinate system we've imposed on an object. And if you impose a rectangular coordinate system that doesn't line up with two edges, two edges like this and your rectangular coordinate system's like that, you have no idea whether this is a right angle. You're only interested in lining up with a coordinate frame, and you're not really interested in right angles, but anyway. I had to put that in 'cause I don't wanna say false psychology. So now we've got all the things we need for doing interpretation. We have the 2D lines. Those are our inputs here. Those are our visible neurons. We know what edges each 2D line might be, and we know how edges should fit together most plausibly. And so, all we've gotta do now is solve a search problem. What we want to do is take the states of the visible neurons and search for a low energy state configuration of the system. But we have a problem, we might get trapped in a local minimum, and can we do better than just going downhill in energy, updating the neurons so you go downhill in energy and ending up in a local minimum? And it turns out, you can do quite a lot better than that, and you do better than that by using noisy neurons. And it turns out, if you use noisy neurons for doing search, then a piece of magic happens and you get a very simple learning procedure, but I'll come to that. So in a Hopfield net, the energy is actually the energy of configurations of binary things, which are really the corners of a hypercube. But I'll draw it as if it was someone one dimensional thing. So imagine a landscape, and the state of the network is a little ball on the landscape, and as you change the states of neurons, you're moving about on that landscape, and the update rule for Hopfield nets says, go to whichever state lowers the energy. And if you get trapped in a local minimum, you then can't, you're stuck and you can't get into a deeper minimum, like B. And the Hopfield update rule says, if the total input to neuron is positive, turn it on, and if it's negative, turn it off. So it's that square function at the top. But suppose that we make the neurons noisy. Suppose we introduce jitter in the decision rule, or what a physicist would call temperature, then we can have a decision rule set that says that sometimes when it gets total negative input, you turn it on, and sometimes when it gets total positive input, you don't turn it on. Obviously, generally when it's got positive input, you turn it on. But if it's only small positive input, maybe you only turn it on a little more than half the time. So that's that decision rule. And notice that graph is showing you the probability of the neuron turning on. These neurons are still binary neurons. They don't have real values, but their probability of turning on is governed by a decision rule like that. Now, if you use neurons like that, then they can jump to higher energy states. So this ball bearing on this energy surface can jump over that barrier and get to the lower energy state. Now, it might also jump back, but it's less likely to jump back 'cause the energy barrier is higher from that side. So, the way we're gonna interpret a binary image is, we're gonna clamp the binary image on the visible units, and then we're gonna go round updating neurons using this probabilistic decision rule. For a physicist, we're gonna use a temperature of one. And we're gonna pick a hidden neuron, we're gonna compute the total input it gets, and then we are gonna stochastically decide whether to turn it on or off, probably turning it on if it's positive input, but occasionally turning it off, and we just keep doing that with the hidden neurons. And that will do better than just going downhill. That'll typically get us to one of the deeper energy minima. Now, if we do that for long enough, we'll reach a state called thermal equilibrium. And if you're not a physicist, you think thermal equilibrium means things have settled to some particular state, but it doesn't mean that at all. It's what a statistician would call a stationary distribution. It means that, even though the neurons are changing states, the probability of finding the whole system in a particular state has stabilised. And the best way to think about it is, imagine like a gazillion neural networks that are all identical, they all have identical inputs and we start 'em all off in the same state, let's say, but then each neural network makes its own stochastic decisions about how to turn on or turn off hidden units. And you can imagine, if you took all possible binary states of the network, when we start off, they all start off in the same state. So if you did a histogram of how much these different binary states are occupied, all the systems would be in the first state and none would be in any other state. But then you keep running this decision rule and gradually what happens is, systems adopt other configurations. And after you've run it for a while, you'll have a histogram where low energy configurations have lots of systems, and higher energy configurations have less systems, and very high energy configurations have very few systems, and even though any one system will jump between configurations, the histogram will stay fixed, if you have enough systems. That's what thermal equilibrium is. It's when the histogram stays fixed even though individual systems are jumping all over the place. And the nice thing about that fixed histogram is, it has a very nice property, which is that the number of systems, or the probability of being in a particular configuration, only depends on the energy of that configuration relative to the energy of the other configurations. It doesn't depend at all on the history. You have to run it long enough so it's forgotten where it started, and then you get this nice guarantee that the probability of being a state, a configuration just depends on the energy of that configuration relative to the energy of all the other configurations. So, we'll come back to how you use that for learning in a minute, 'cause that's the neat thing about Boltzmann machines. For now, let's go back to this line drawing. What we'd like to do is, we'd like to not have to hand wire this thing. We'd like it to learn all those connection strengths. So we'd like to just give it, we give it some visible units where it represents the line drawing, and we'd like to just give it a pool of hidden units with random connections and have it learn all that structure. Now, it won't actually learn that structure. I put that structure in 'cause it's easy to understand that a system like that might work. It'll learn a much more sophisticated structure, but it'll do the same job. It'll be able to interpret line drawings. And so, the question is, how could you learn all that structure just by looking at a lot of line drawings? And it seems very implausible that you could do that. And it particularly seems very implausible that you could learn. For example, we have to learn the weights between hidden units. That connection that says, join in 3D. It has to learn all those connections. But it's not at all obvious how you would learn all those just by looking at data. And it turns out, there's a very, very simple learning rule. So I already explained thermal equilibrium. Yeah, that's what I already explained, 'cause I got ahead of myself 'cause I couldn't see the slides. Now let's think about, we've got a system, we've already got the weights in. Suppose I put them in by hand. And instead of getting it to interpret a binary image by picking the states for the hidden units that represent the 3D edges, I'm gonna get it to do something else. I'm gonna get it to generate a line drawing, an image of lines. And the way I'm gonna do that is, I'm gonna start off the whole system in some random state, and then I'm gonna go round picking neurons at random and updating them, and just keep doing that, and after a while, I'll reach thermal equilibrium, and then I look to see what the line neurons are doing. And so, we could think of an objective in learning weights.
We could say, if you show me a whole bunch of line drawings of rectangular objects, my objective is to find weights so that if I start the network off in some random state and just go round updating units one at a time, when I look at the hidden unit, when I look at the visible units after it's reached thermal equilibrium, they will look like one of these line drawings I showed you. In other words, it'll have a model of what line drawings look like. Because if I put it in a random state and say, settle to something you find plausible, it'll come up with a line drawing. And finding plausible, for this network, means a low energy state. The lower the energy, the more plausible things are for the network. So if we have the right weights in the network, we have a way of generating data; we also have a way of interpreting data. To generate data, you update all the units and just look to see what the visible units are doing. To interpret data, you fix the visible units and then go round updating the hidden units and look to see the interpretation in the hidden units. So that's generating. The aim of the learning is, when the network's generating, you'd like it to produce stuff that looks like the real stuff. So you can think of generating in this network as dreaming. Francis Crick postulated that when you dream, what's happening is you're getting your neural network to generate stuff. He also postulated that you're getting it to generate stuff in order to get rid of that stuff. So as you'll see, this learning rule exactly fits his speculation. The idea of learning this way, the idea of learning in a generative model that will generate images like the ones you see, the purpose of that may not be to generate images. You do generation during the learning, but your real purpose may be to get at these hidden variables that explain what's going on in the image. Because if you're doing perception, you're much more interested in the 3D edges in the world, not the 2D lines in the image. So the point of doing this generation is so you can do learning, and the point of learning is to get at these variables that are not in the input but are things you made up that explain the input.
Here's our outrageously optimistic approach, which actually works very well in theory and not so well in practice. You start with a neural net that has lots of hidden neurons with just random weights between them, and you have some data, which is the states of the visible neurons. You keep showing it these line drawings, these images, and you want the network to learn for itself all that hidden structure that will interpret images. You'd like to do that with a nice simple learning model. This is where statistical physics meets neural nets, and there's a beautiful synergy there. We're going to have two phases: a wake phase and a sleep phase. In the wake phase, you put an image in, go around updating the hidden units till you settle to thermal equilibrium, and then use the Hebb rule. For every pair of connected units that are on at the same time, both on, increase the connection strength a little bit. That's exactly the rule Donald Hebb proposed, ignoring the temporal aspects: things that fire together, wire together. That's all you do, a very simple rule, nothing like backpropagation. Then for the sleep phase, you don't give it input; you turn off the input and get it just to generate. Every time two units are on together, you decrease the strength of the connection, an anti-Hebb learning rule. These two simple rules — go around updating hidden units when giving input and increase connection strengths between two hidden units or between a visible and hidden unit when they fire together, and then go to sleep, generate images by updating the hidden and visible units until you reach thermal equilibrium — do exactly the right thing in expectation. You have to say in expectation because they're noisy, but on average, they do the right thing.
I'm now going to show you a tiny bit of math, because it is after all a physics lecture. You might have thought at this point... I deliberately kept math out of it so it was available to the general public. The general public can now stop listening for the next three slides. Yes. What that procedure achieves is a maximum likelihood learning procedure. It changes the weights of the model to maximize the likelihood that the model would generate data like the data you saw. Before I give you equations, I want to say why this learning rule is so surprising. Suppose my data just consists of two bits, these two visible units of the two ends of the chain, and suppose the training data just says you either get zero, one or one, zero — those two bits hate each other. When one's on, the other should be off, and vice versa. But their only communication is via this chain of hidden units. I want to know how I should change W1 to make the system work better. Because the two visible units need to turn each other off, the product of all those weights, W1 times W2 times W3 times W4 times W5, has to be negative. If that product's negative, they'll inhibit each other. That means to know how to change W1 to make things work better, you have to know the value of W3. If you do something like backpropagation, you find those values by sending signals backwards through the network. But in a Boltzmann machine, how does W1 know W3? It needs to know something about W3 to know whether it should increase or decrease to make the system work better. The answer is, everything it needs to know about the other weights is conveyed by letting the system settle to thermal equilibrium in these two phases. Now the equation. If I give you an image V, a visible vector, and I ask you, how should I change a weight to make V more probable when the network's generating? The answer is, you get the exact gradient by looking at the difference between two correlations. This is the derivative of the log probability of a particular training vector under the model with respect to one of the weights, and that is the difference between two correlations: the expected value of the product of the states of two neurons I and J when you're clamping V on the visible units, and the expected value of the product of the states of two units when the system is running freely, when it's dreaming. Those have to be the expected values at thermal equilibrium, and then you've got the gradient. This is a much nicer way to get the gradient than backpropagation. You don't need to send a different kind of signal backwards through the network. You get one half of the gradient while you're awake and the other half while you're asleep, and then just change the weight by the difference between the correlations you measure between two units when you're awake and the correlations you measure when you're asleep.
The question is, why is that derivative so simple? The energy of a state of the network is just the product of the two activities times the weight. If you differentiate with respect to the weight, you just get the product of the two activities. At thermal equilibrium, the log probability of a state is a linear function of the energy. If you ask, what's the probability when it's generating that it'll generate a visible vector V? The top line is e to the minus the energy for all possible configurations that include V as a visible vector. Sum over all possible hidden configurations, and that's the probability you'll find V. The bottom line is summed over all possible visible vectors, which is the partition function. To increase PV, you want to make the top line bigger and the bottom line smaller. You make the top line bigger in the wake phase by increasing the strength of connections between two units that are both on, because you settle to thermal equilibrium with V and a hidden configuration that goes with V nicely. The negative phase makes the bottom line smaller, which is why you do unlearning in the negative phase. That's it for the equations. You can start listening again. The process of thermal equilibrium gets all the information you need to get the gradient for a weight into the difference of two correlations in the wake and sleep phases. But there are problems with Boltzmann machines. The main problem is they don't work, because settling to thermal equilibrium in a big system takes a very long time. That was it for Boltzmann machines for 17 years. They take a long time to settle, and the learning signal is noisy, but the engineering problem is that settling to thermal equilibrium is too expensive. From a neuroscience point of view, connections between real neurons are not symmetric, people alternate between wake and sleep at a coarse timescale, and we don't update neurons one at a time. They have two properties: they don't work as engineering, and they're not how the brain works. They have a third property: you can get a Nobel Prize for them.
17 years later, I realized that if you make a restricted form of Boltzmann machines where you only have one layer of hidden units and they don't talk to each other — just visible units and hidden units that don't talk to each other — you can get a very fast learning procedure. If you clamp the visible units and the hidden units don't talk to each other, you settle to thermal equilibrium in one step. You simply visit each hidden unit, all at the same time, update them based on the input from the visible units, and you're at thermal equilibrium in one step. That's the wake phase sorted. For the sleep phase, you still require alternations between updating the visible and hidden units, and it looks like it's going to be slow. But I tried it. I would put data on the visible units, update the hidden units, then update the visible units again — call that a reconstruction. Then activate the hidden units again, and go up and down like that for a long time until thermal equilibrium. But I was interested in how long you had to run it. I ran it for a certain length and the learning worked. I ran it for less time, and it worked. I ran it for less time, and it worked. In the end I just went up and down and up again, and the learning still worked. That was good because the business at thermal equilibrium went away. I made a naive logical error: if you reach thermal equilibrium, the learning will work, but that doesn't imply that if the learning works, you reached thermal equilibrium. The learning can work for other reasons. There's an algorithm called contrastive divergence where you just go up and down up again. You measure the correlation in the first phase and the correlation in the second phase, subtract one from the other, and the thing learns pretty well. It's hard to prove it's doing the right thing because it isn't, but it works in practice. These restricted Boltzmann machines became practical and were used by Netflix for predicting which movies you'd like. A competition offering a million dollars for a 10% improvement over Netflix was won by a team that used restricted Boltzmann machines and matrix factorization. RBMs were useful, but the restricted version threw away most of the power because the hidden units can't talk to each other.
Now you can have multiple layers of these and learn a hierarchy of feature detectors. You train an RBM on data, capture features in the hidden units. Then take the binary states of those hidden units as if they were an image, and learn a second RBM. Then take the hidden states of that and learn a third. There's a variational bound showing that as you add Boltzmann machines, your model gets better — or the bound improves. You can stack them up to get weights for a multi-layer network that extracts features, features of features, and so on. Then put class labels on top. This turned out to be a good way of initializing deep neural networks for object or speech classification. For about four or five years, people initialized nets this way and got them to work much better, especially for speech recognition. Then they discovered other ways of initializing and threw away Boltzmann machines. Think of them as an enzyme: they helped deep learning be born. So there's a very tenuous link between Boltzmann machines and current neural nets. To summarize, they were like an enzyme that helped us make the transition. I still think the idea of using unlearning during sleep may be really useful for understanding how the brain works. If you deprive people of sleep, they go completely crazy. If you deprive them for a week, they go psychotic, and some never recover. Most theories of sleep don't explain why. But if sleep is for unlearning, it does explain: if you use a positive Hebb-type rule without unlearning, the whole system will go crazy.
