03.31.2017-Dr. Geoffrey Hinton Talk - U of Toronto - RBC Entrepreneur Challenge and Vector Institute

Bank ardency and the University Toronto for organizing this event. It's true as you're learning things that some mathematics is very important for your banks but I'm actually very good at explaining the mathematics because I'm no good at mathematics and so I'm able to explain to other people. Can we get the lights back on? There's two ways to get it. Used to be one way to get a computer see what you want and that looks like a program and that meant figuring out how you're doing yourself and then explain it to me you're busy in excruciating detail. Now that's a new way to introduce to do what you want. First I defer their control of computer how to pretend to be a network of neurons with a lot of elements and then to do a particular job you just share it examples. You share examples of what goes in and what you would like to come out. So examples of this table you would like to look at an image and reducer capture the image. Now evil NIR attracts me about 50 years and then get close we couldn't be impressed by the objects in the image. We can add training your lab and starts off knowing nothing and you show the pixels in this image and in light of this country. In fact this is an example for the neural net has ever seen before. It saw this after it was trained and that's the caption alignment. So how does that work? Well I can say wooden you're all those things I'm not just showing examples and wonderful things they can do. So first of all we need to make an ID elimination of an urn. It's a very gross idealization but it's good enough for what we need to do. We say in Iran in Surrey and you're on get some inputs from other neurons. Allocation from the fences and he gives them a group that depends on how big the infants are and how big the weights are on its connections. So what it actually does knowledge you would like a must by people by the weight as it all up it gets a big song to give the big advocates like the graph on the right you get the small summing doesn't give any animal. That's it. That's how your own works one of our idealized humans really are not as much of a propagate it. Then we have to figure out try to put them together to do a computation. So we've put them together in land we have John's at the bottom of the vectors and things like the intensities of pixels. We have Yount are off that are going to learn to be feature detectors. We're not have to define the features. It's going to learn the features and then neurons of the top that make decisions. And to train your system we can either use supervised learning or unsupervised learning. In supervised learning you can write on them that's what works best to present. It's not really more people do people may need your unsupervised learning we just look at the data and figure out what's going on. That's much trickier and we're working on that but we haven't made that work very well yet. So I'm going to give you an idea of how supervised learning works. You put into the data at the bottom for example a picture and let's suppose you want in your lab that can tell the difference in the Catalan drama soon as to our units one of this cast one that says done then this neural network of presence is giving a big output the dog the cat so that's not very good. So you smell good that will work but will be very slow. You picking one of the weights of the network but surely in red and you look at the output with loud wage and then you change the way so we make wait a bit bigger and we notice the athlete changed a bit. If I have got more cat-like and less dog like so that's good because this is a cat so we'll keep that change that we made the metal your might another way like that and we'll try changing that race. Maybe we may need a bit stronger and we were lucky so again Kathlyn Java dog my dancer which is a change. And when you go around in all of a million ways of a network seeing when you can be seen in the house keeping it in advance just like any movie and some going over your ways or maybe a million is a good big Network just to have one image. And then we come to the next image as we do long again but surviving lots of millions of images has been his waves it would have taken tens of thousands of years but the someone will clearly work but it will work in evolutionary time. So you actually do the same thing by just using only with a different weapon and we make it much more efficient. So what we do is instead of terming the ways one at a time and seeing whether it helps we look at the output we can together we compare with the correct output and we send a single bathroom through the net with computing for every single wage independent have what directions changing it so to make yes of all I think we want. And the different spins to algorithms is simply that if you've got a million ways the second album won't be a good name that reduces alias that doesn't really mean much participating on marches on lines explanation but for practically feasible well. So we send back the your back up through the network. I'm not going to go into the man if you're a petition is just changeable the community derivatives and then job flexible ways and they take another cell to go to the waste. And if you keep doing among us it gets me very good. So this algorithm many different people realized is algorithm in the second ages in the mid-1980s computers were fast enough so that we could show you that if you ran two thousand with a Muslim edge would run good representations but if the features in the middle layers would discover meaningful things about the data even though you haven't totally what interest to use. And that was exciting provided in central Pro collaboration intelligence where the representations come from. And in these networks the representations are activities of a whole bunch of feature vectors and vector activities. And so these vector repetition in classic irony of representations of symbolic expressions operational modes of inference here we just have vectors operated on by matrices. And that's an led to a much better it was used for really America mantel chests which taken credit card fraud or interpreting taxpayers but it never work really well. And in particularly these pants with many layers each census you didn't work as well as we hoped. So by the 1990s psychologists were still very instrument but people the machine learning pretty much given up on it because there are other algorithms that were a more reliable as well just as well. It's got slightly better and then at the beginning of this century well earlier of the century researchers in Canada came up with new techniques that an exodus meant to be trained on sir I'd love to go into all the new techniques that I really care about but that's not the main point and sort of going into the darkest history now. The main point is that this algorithm when big datasets and fast computers work really well. So the first big breakthrough of the types of mania industrial consiquences words for speech relief to students of the university older male announcer with a bunch of Ethan glass and they trained it would look at some frames of Croatia's effect for a speech wave and they're traveling at which piece of which folding the person is trying to sell you in the middle MacGregor and if you better than previous technologies that is slightly better. But the point is this is Christian over summer and next week thirty years intense competition by big rooms and that was obvious drug money but this is good away and other students went off to various places. Wow one of those shoes mandy gently went to Google and Google said no government work. Um it's not the ground and he insisted and the Romania worked there very good momentum generators and it worked the since they saw this really worked they looked a lot of engineering into it and by 2012 that would I just answered the first big deployment of the business new technology. And now all the best speech recognition systems you support this out in 2012 two other students in Toronto shows that this stuff works very well for recognizing complicated images. There's a public competition where the best computer vision to the world enters very entries and you think the test date of the thing so you couldn't cheat. And at the bottom there you get the world mini computer vision groups I've been getting back 25 personalities and we got just under 60% errors almost half the overage and that are making impact on communities within a year water be division researchers and switch up until then many things like these neural networks they're crazy there's no way you're going to learn everything in all these legislation you have to widen sensible knowledge about machine when we found they said they were very perfectly they developed move along. Um and this done along whether the world that's very rare in science. So what did you see is over the next three years the rest have got a whole lot better because lots of rooms my working on it and it's now at about five percent on this same data set which is a matching.

If you look at what the data looks like identity to see this that's a close-up of Ishita and at the bottom of the network's getting certain opportunities it is very common and like a cheater but if your other guesses are pretty sensible getting if you know me blade is it thinks that the bullet train even though presented with building in the balance much bigger and there's a passage on platform well but I think you support apparently other guests a reasonable here it gets it wrong it starts getting incisors and if you miss third grade strike that you look at that you can see this your leg needs gospel right you can see why things are flying around but it's not really saintly one but the good news is if error is that a sensible visual learner it clearly sees what things are the planting. Now a few years later essentially the same method was used to solve a major problem game languages when you play chairs you can just get a computer to consider the Blues and they can throw what the in front of my new like a zoo what you might do and search for a dense of like twelve and you can beat any person family. Now no you can't do that because too many moves out and you need intuition you need an intuitive sense of which moves the word considering you also need to be able to look at a ward and say is that good for me and I'll build a two-year-old map that side different from previous systems they go one could look at the board say these are the plots of the places you might move he didn't always agree with the experts as the famous 137 where the experts couldn't stating the book is in such a bad mood and one gave me and another girl that are not reporting just said in homogeneous and those extra ingredients made assistant lips could eat the world champion ago. In fact I think it's only ever please be once I easy darling gave four since then it's been 60 games against Exodus 1. Another example of his someone's Alec Google will respond asleep reading or looking at images of the retina and predicting what stage of economic read of diabetic retinopathy users and this network is now as good as most doctors. The very best doctors are still as available Network and but not a lot and this is going to be very important because for example a lot of people in India have this disease. Rajinikanth mammalogists to look at all these images you get the images instead of the trained people to interpret them but this could not soon as we never consult them. This will be amazing and through love you pretty soon of a cell phone in the next year which are emitted from this system where you could shall pancha your skin to the system and it'll tell you as a cancer and the problem for us is again it's funny patches killing develop unaccountably you're not quite sure you want to go to the doctor because the doctor for the last year for certain cancer it's much better Sam solvent we they weren't laughing.

With the case I'm gonna talk about another kind of neural net a recurrent neural house which is probably the most excitement is now. These were developed in 1997 all that one bowel rhythm was developed by operator to a tumor who were in such a such a fast-paced remotely wonderful idea that nobody could understand what they were talking about. I think you just ignoring it and then in 2007 Alex graves won a trip to the student shows it in really work the gravel and since then it's been taken so. I'm going to be very brief on our current role Network and I'm gonna suppress mr. T James the verification before had letters and this had layers to be related time to no can't your network a sling for neurons I need you one the input John Prine element of each time steps a canoe runs the heat euros and then or a current system for time step to the in between your own comes from the seniors of the previous time step of the current input a time tree and the next time leave misleading your relative aliens and you train them with the same trade young basically after a while I know whatever you want and you send an equation backwards and angles wrong saying please change the weights on the connections so that I'm more likely to get the right angle. So I'm working Lee to students in this was given a necessity certain age a margin up in a bank 2011 we face half a million characters Wikipedia to one of these ends so the let's start off knowing nothing and it's just us to predict the next character in Wikipedia and that allows you to answer the philosophical question. Suppose an intelligent system bring the doesn't like this the Jets or the character sphere would you be you didn't see the world and intentional draining liquids or philosophers it just saw the character soon Wikipedia couldn't understand the meaning of life and most closets will tell you that's not possible but I can give you some empirical danger. So we trained enough and it's just trying to predict the next character after it and it predicts the probability distribution it says eventually in fact I know about having seen this character I think the probability of a space is not point nine and the probability of the disease is one in a billion and so on it has a nice it's probably is about two one. If you want to see what it believes you feel it string characters asking what it takes comes next it gives you something alerted you pick one of those according to those prevented so space is point nine point nine and tiny big space and say okay the next annotation other space and what do you think of next the middle of your prediction and so you can keep running this prediction game I'm picking from the best it makes to see what you're enough. So we gave me that now that maybe dissent or to choose I would have been boring because I think we can medium for sure but it would use a lot of character string and only a small part of this character sharing on Wikipedia just made something up and you can see what it's pretty bizarre and for modern make up and remember it was trained your stuff on the way so it's going to print on one character to travel genuine user . and this is what it says.

And slightly confused I I would argue that is almost got the meaning of life just reading Wikipedia that acts that was of the first thing he says we let it do it ten times and if they surprise you we think the best we should have been what a random that we figment of so using that same technique we can translate languages. What you're going to do is putting words in one language actually fragment two words or it's better than putting words in one language and it should I think the word words in another language. So you train in resultant something which is French so there's anything kinda part of the network that's taking words in one language they're going to the Sakura neural network it's not coming any inference yet and afterwards in that language run into the network you get some final state of the hidden unions and a final state of the hidden units is in fact Lee's vector representation of what this string of words means. The reason we know so vectorization what three words means is because we given that vector to another network that's games they're going to produce this in another language. If I haven't got a network that you can emit any language and just tell it what language any of twelve languages are just tell implement that you like Alice to be in and the very same network will always languages and I think the count of the edges be encouraging really is important is like a lingua franca but it's not a lingering effect. So I call this effect different the key code is network in given that sort and the family doesn't say make you fat for what I think the first word like the approach so maybe it's 40% learn 30 Cal our biggest madam says okay long so at first then you feed that world back Internet it's like the first or whatever which is executives give you problems a nice things personal closest suddenly you might get a different translation but it'll give you a good translation of the new sentence. It's amazing of this world because the network is started with less knowledge whatsoever a bad language it just doesn't have any knowledge about grammar what's going on we just get this string of symbols coming in and the string assistance comes out and enterprise that's a better way to translate than anything else. If I give you go to the website right translation Chinese to Nvidia English Chinese that's what's going on Google switch 2000 think of a whole lot better and it's in the middle of switching all the other languages.

I said that as to get back to my first example the key technically never recognize objects and you use the business in character and you take the last layer of the vision Maps before the a particle one a percent so that's really these pixels in turned into a vector a vector representation in terms of objects and you can ask concepts to a lack of detector instead of the thought that you get language until the Ticos are postponed that's how the caching works. So you can't see this very well but this is a scene of people of micro markets the caption in the database says of people approach to write in an open market the system doesn't give us good account it says a group of people shopping at local market but if the clue to see what's going on when they visit self-releasing already the official captions of that is a young girl asleep on the sofa cuddling a stuffed bear that's the better captures and what resistance is the - the system can see what's going on this big implication she's adopting the proof. Google would really like to be able to return documents to you based on their needs not just based on the words they contain. You could take documents that contain same words and they can say very different things. Wouldn't it be nice if you could say Google give me a document of my climate change the comports to be scientific but actually is a whole bunch of rubbish and not very soon but eventually I think we'll be able to do that. In other words we'll be able to extract the thoughts and sentences and looking to how the thoughts hang together because I just scratched the modeling a sequence of thoughts and when the other thing is report for coherence when they don't really follow ninety-two bad remaining much bigger neural net. What don't you think is true either we'll give any much much bigger than your lens because the great has abundant eyes times as many mention use whoa this is my procreation album is much better now reviews in the brain I have actually no rich because the thing that sound entering all these other languages intuitive look at that on an MRI scan the signs they would be less than 1 pixel in the MRI scan okay so your brains that a local power plant so they she doesn't have a good amount of remembrances I don't like mature alien. I want to finish with a story and this is a story about random other applications. So one season two did the speech recognition for George star so that was a competition so rather late as a station attrition the competition to predict the activity of the molecule an institution has those things that describe the quality like which groups in a non have aeration sorry and when it's hydrophilic hydrophobic or water and that's the limited like chemistry knowledge and and you have to predict will it bind to visit ardent so some biological targeted if it will find there's an income for treating depression. Now you can say fine you can see but that's expensive or you could just try to get a good resistance according to make you to try millions of molecules and some are put a public conditional array and George entity claims using your letter one and after you've won must said well 20,000 cries but part of the special offers in order to get a prize he had so much work to side against the judge said what's Q. So now this is slightly icing because chief I've actually field is called quantitative structure-activity relationship it has a journal it has an annual conference and a lot of people work in that field and Georgian actually does better than anybody could do in Q star without even knowing the name of the field he didn't know what any clean books represented you. So again enjoy he just treated business a little competitive in your website. Um okay I'm done.

[Applause]

Importantly yes I think I'm so responsible my family felt on yarn for too long let other people say go.

Okay he shall prefer to be dissolving intelligence to find the bottom definition if you have to believe you together that it said well you can't play girl because leave this set of intuition it only needs of essential common sense nice play go. So in that sense we saw that the dungeons I think we actually come a long way to absorb intelligence we sort a long way to go there's still major problems like items the need for the compounds of your language the using are going to be overusing time I'm your outline there everybody else needs to believe that Daniel Medical just fine we just needs a few more ideas develop a little bit with a great type attention I actually prefer disagree with everybody else so I think the current neural networks are going to be replaced by some different kind of y'all delicious and the field is already than staying where people believe too much the current technology not they believe actually works it really does work but they believe it in Sega I think it's going to change over time.

I've a project here are the current way we are organizing in your life in your network right now all those neurons are organized in layers there's no functional or gospel book mythology a prophet you are rotting carcasses area to pain neurons a lot of ways for you and paying for danger yes identitaire computer scientist or an engineer having this great learn big Europe that all sort of the problem to begin with we've no other structure Justin's land seems not stretched and if you look at the brains of ranking online and at the brain has been schooled many columns your instructor watching the about your arms box emotional far away it has been fascination it's pretty different from the typical evening I'm strongly the wings that we need to think about all groups of neurons and instead of using the non-linearity we use which simple scalar it's a linear filter full of a scalar full of our skin the non-linearity and you need to think of taking a vector some kind of economic guarantee is a lot more of those of our scale anonymity and people just are looking at this tonight admission I are they believe are not married of the on that self-driving cars and there's a lot of would examine learning in that area so your thoughts on that capability in vacation time using all that publication.

I think it's pretty only advance I mean I did that time thing but it's fine ten years I think realistically there will be a lot of cell phone because it's obviously going to say go multiply so they'll some of the jobs sell and make unnecessary all is given hospital emergency room seeing the kinds those jobs we rather rather not have to have obviously to put some brag about emerging but in the face of the technology like this you can't say let's just not doing it it's going to happen it's going to happen whether we do it or not when you want to do is be we want to be out in front of it. If you look back at automatic television I'm sure the same argument I don't know too much attention what about don't use will put bank teller by the work let anybody now will say they should have been introduced either you can probably remember the holidays when you want to get $20 out of the banking had to wait in line for 10 minutes we don't want that between 10:00 and 4:00.

Multiple

Okay so obviously for the vision systems of the example of deep knowledge and very often not necessary building learning I mean D learning make you part of the system so remember live Israel doesn't lead to the American system but I think they will Twitter you do growing up in addition to one of the general recommencing probably university often uses something learning in some other regions are and as a very impressive system that will within a watery TV camera look at a busy intersection and they give you a planet Lee which is what you need to navigating of we're all cars are a minor going. So this doesn't extremely helpful though but not literally purely about some one place I know it's very helpful if detective destory so Google they have assistive protected veterans the going blind 99% of the time and then the if you Steve learning that when they got in right 99 when I spend time we just need another few knives only be okay.

So part of the point of neural networks as people don't work by logic they work by analogy they to budget when they get to be old and when they're going to Train but really they work by mountain finances question is analogy a half years ago if you ask the biologist what's wrong many biologists the philosophers will say well things are allowing you to have vital force and what's the - Oscar rather what they're dead that's not really the Reformation is the attentive playful really complicated by retrieving a time time again and we be better that you've got it well as is essence you have a lot and I've just handed musical consciousness I mean it's got many different meanings but the sort of primary meaning is you've got these very complicated thicknesses in very complicated conversation and if it's like that it's got consciously I did think I believe forget in business and I think it's not that we won't have consciousness it's just a stock meaning after it apologist analyst to tell Michael torch is just laughable because we have molecular biology we know what's going on at least to the level where we don't need to do some very gently to explain anything and I think consciousnesses have a general letter letter explains complicated mental phenomena for cars.

Yes because you try to work all if you train in your own messages does this all on a translator can I prove anything about how well it's going to your testing can ice cream for example the liquid reliably land the plane on all training samples it will actually reliably like to play in new conditions the honestly you can't prove very tight bands on what you've done and it's question we're very interested in for but these are very complicated nonlinear systems and in three how to prove started keeping you in the countering stuff the assumption seeds may get the proof are way more seizures but the assumptions that apply to my nightly news. So one good example is if you have 60,000 data points you want to predict something a statistician will tell you you should use our letters and significant parameter. So I recently trained in your lab to identify uttering digit there were 60,000 training samples and I use the quarter was billion parameter and it was better than the ones only had on million broke this is completely different rail but in the range that people bravely through ups the tension Alliance faculty and you have about 10 to the 14th parameter so you have to set a hot 2004 actors per second that is not lost on official YouTube and we don't know much about the training but services the beginning he arranged to the battery.

Okay um first of all I'm not coming back to Canada from Google I used to work to do remote Canada on the state tomorrow into Google full time in Canada almost forever camera events but sometimes by collectors. So I have a Leslie like despite what you might read into strong Eileen I saw what you do right through the back I actually thought Google that have a basic research on in Canada so I didn't you Lisbeth web powerful really negotiated to pay much more for our articles part of the deal with those companies will put some basic essential Canada and I don't know how much better research they put ever not much. Google is now sending up to basic research back to Canada what interrupted because of the best in exchange and one in Montreal because of the real Institute but for the sister of the vector in C and that's what we really like would really like the baby research community and the people who do the basic research to pay their taxes in Canada well so a program of man approached alum.

[Applause]

From your lats are quite good commercial section I just learned a lot a few years ago in a suit of just subdued while they're getting better and much better they were a few years ago and I don't see why they should vendor because the people the team leader be deducted school many computer scientists on the autistic spectrum and for for two things are very bad at recognizing racial oppression.

Okay I'm the warden impression exactly but I need to clean your scientists a lot and the whole reason for the leading with in sunlight work with the fact of the brain woman anybody in their life I would not have said you could take this cell and you can take word and you could get from pixels to world we don't have a program in all the stuff in between the only reason to believing that might be possible is that what we do so let me break up does delicious with something but not complicated program. So so your time begins breeding for this and the way I see it we try to build a bridge and why don't you pass empirical classes who are looking at data and try to figure out honestly Michael cute but the other hand in which you have to decided who are looking computation tampering I have might be done a little great like way but so we're trying to build this into the bridge towards website there's other people in our how to build a bridge situation of the most every neuron so it's the inspiration of any of these techniques but just give you one example this is these are actually in your labs is used to say we're not just imagine our activities and went on the connection we're going to wait for the connections that changing many different ice cream when I'm going to change slowly the way to trade much faster the way to change parts or temporary memory the way to turn slowly will have wonderful memory but we won't have to keep all the tech remembers just an activity my direction is inspired by Michael.