Well, good afternoon everyone. Gotta love the buzz in the room here today. Welcome to the Mars Discovery District, this wonderful complex, for this very special Radical AI Founders event co-hosted by the University of Toronto. My name is Merrick Gertler, and it's my great privilege to serve as president of the University of Toronto. Before we begin, I want to acknowledge the land on which the University of Toronto operates. For thousands of years, it has been the traditional land of the Huron-Wendat, the Seneca, and the Mississaugas of the Credit. Today, this meeting place is still the home to many Indigenous people from across Turtle Island, and we are very grateful to have the opportunity to work and to gather on this land.
Well, I am truly delighted to welcome you all to this discussion between Geoffrey Hinton, University Professor Emeritus at the University of Toronto, known to many as the Godfather of Deep Learning, and Fei-Fei Li, the inaugural Sequoia Professor in Computer Science at Stanford University, where she is co-director of the Human-Centered AI Institute. I want to thank Radical Ventures and the other event partners for joining with U of T to create this rare and special opportunity. Thanks in large part to the groundbreaking work of Professor Hinton and his colleagues, the University of Toronto has been at the forefront of the academic AI community for decades. Deep learning is one of the primary breakthroughs propelling the AI boom, and many of its key developments were pioneered by Professor Hinton and his students at U of T. This tradition of excellence, this long tradition, continues into the present. Our faculty, students, and graduates, together with partners at the Vector Institute and at universities around the world, are advancing machine learning and driving innovation.
Later this fall, our faculty, staff, students, and partners will begin moving into phase one of the beautiful new Schwartz Reisman Innovation Campus just across the street. You may have noticed a rather striking building at the corner, with the official opening planned for early next year. This facility will accelerate innovation and discovery by creating Canada's largest university-based innovation hub, made possible by a generous and visionary gift from Heather Reisman and Gerry Schwartz. The innovation campus will be a focal point for AI thought leadership, hosting both the Schwartz Reisman Institute for Technology and Society, led by Professor Gillian Hadfield, and the Vector Institute. It's already clear that artificial intelligence and machine learning are driving innovation and value creation across the economy. They are also transforming research in fields like drug discovery, medical diagnostics, and the search for advanced materials. Of course, at the same time, there are growing concerns over the role that AI will play in shaping humanity's future. So today's conversation clearly addresses a timely and important topic, and I am so pleased that you have all joined us on this momentous occasion. So without further ado, let me now introduce today's moderator, Jordan Jacobs. Jordan is managing partner and co-founder of Radical Ventures, a leading venture capital firm supporting AI-based ventures here in Toronto and around the world. Earlier, he co-founded Layer 6 AI and served as co-CEO prior to its acquisition by TD Bank Group, which he joined as Chief AI Officer. Jordan serves as a director of the Canadian Institute for Advanced Research, and he was among the founders of the Vector Institute, a concept that he dreamed up with Tommy Pizan and Jeff Hinton, Ed Clark, and a few others. So, distinguished guests, please join me in welcoming Jordan Jacobs.
Thanks very much, Merrick. I want to start by thanking a number of people who have helped to make this possible today: University of Toronto and Merrick, Melanie Woodin, Dean of Arts and Science, and a number of partners that have brought this to fruition. So this is the first in our annual four-part series of AI masterclasses that we run at Radical. This is the third year we've done it, and today's the first one of this year. We do it in person and online, so we've got thousands of people watching this online. So if you decide you need to start coughing, maybe head outside. We do that in partnership with the Vector Institute, and thank them very much for their participation and support, with the Alberta Machine Intelligence Institute in Alberta, and with Stanford HAI. Thanks to Fei-Fei. So thank you all of you for being excellent partners. We're hoping that this is going to be a really interesting discussion. This is the first time that Jeff and Fei-Fei, I like to think of as friends, and I get to talk to, but this is the first time they're doing this publicly together. So it's, I think, going to be a really interesting conversation. Let me quickly do some deeper explanation of their background. Jeff is often called the Godfather of Artificial Intelligence. He's won the Turing Award. He is a Professor Emeritus, University of Toronto, co-founder of the Vector Institute, also mentored a lot of the people who have gone on to be leaders in AI globally, including at big companies and many of the top research labs in the world in academia. So when we say Godfather, it really is, there are many kind of children and grandchildren of Jeff who are leading the world in AI, and that all comes back to Toronto. Fei-Fei is the founding director of the Stanford Institute for Human-Centered AI, a professor at Stanford. She's an elected member of the National Academy of Engineering in the US, the National Academy of Medicine, and the American Academy of Arts and Science. During a sabbatical from Stanford in 2017-18, she stepped into her role as a Vice President at Google as Chief Scientist of AI/ML at Google Cloud. There's many, many other things we could say about Fei-Fei, but she also has an amazing number of students who have gone on to be leaders in the field globally. And really importantly, for those of you who haven't heard yet, Fei-Fei has a book coming out in a couple of weeks. It is called, it's coming out on November 7th, it's called 'The Worlds I See: Curiosity, Exploration, and Discovery at the Dawn of AI'. I've read it, it's fantastic. You should all go out and buy it. I'll read you the back cover slip that Jeff wrote because it's much better than what I can say about it. So here's Jeff's description: 'Fei-Fei Li was the first computer vision researcher to truly understand the power of big data, and her work opened the floodgates for deep learning. She delivers an urgent, clear-eyed account of the awesome potential and danger of the AI technology that she helped to unleash, and her call for action and collective responsibility is desperately needed at this pivotal moment in history.' So I urge you all to go and pre-order the book and read it as soon as it comes out. With that, thanks Fei-Fei and Jeff for joining us.
Okay, so I think it's not an exaggeration to say that without these two people, the modern age of AI does not exist, certainly not in the way that it's played out. So let's go back to what I think is the Big Bang moment: AlexNet, ImageNet. Maybe Jeff, do you want to take us through from your perspective that moment, which is 11 years ago now?
Okay, so in 2012, two of my very smart graduate students won a competition, a public competition, and showed that deep neural networks could do much better than the existing technology. Now, this wouldn't have been possible without a big data set that you could train them on. Up to that point, there hadn't been a big data set of labeled images, and Fei-Fei was responsible for that data set. And I'd like to start by asking Fei-Fei whether there were any problems in putting together that data set.
Well, thank you Jeff, and thank you Jordan, and thank you University of Toronto for this. It's really fun to be here. So yes, the data set that Jeff you're mentioning is called ImageNet, and I began building it in 2007 and spent the next three years pretty much with my graduate students building it. And you asked me, was there a problem building it? Where do I even begin? Even at the conception of this project, I was told that it really was a bad idea. I was a young assistant professor. I remember it was my first year actually as assistant professor at Princeton, and for example, a very respected mentor of mine in the field, if you know the academic jargon, these are the people who will be writing my tenure evaluations, actually told me, really out of their good heart, that please don't do this after I told them what this plan is back in 2007. So that would have been Jitendra Malik. The advice was that you might have trouble getting tenure if you do this. And then I also tried to invite other collaborators, and nobody in machine learning or AI wanted to even go close to this project. And of course, no funding.
Right. Just describe ImageNet to us for the people who are not familiar with what it was.
Yeah, so ImageNet was conceived around 2006, 2007, and the reason I conceived ImageNet was actually twofold. One is that, and Jeff, I think we share similar background, I was trained as a scientist. To me, doing science is chasing after north stars. And in the field of AI, especially visual intelligence, for me, object recognition, the ability for computers to recognize there's a table in the picture or there's a chair, is called object recognition, has to be a north star problem in our field. And I feel that we need to really put a dent in this problem. So I want to define that north star problem. That was one aspect of ImageNet. Second aspect of ImageNet was recognizing that machine learning was really going in circles a little bit at that time, that we were making really intricate models without the kind of data to drive the machine learning. Of course, in our jargon, it's really the generalization problem. And I recognized that we really need to hit a reset and rethink about machine learning from a data-driven point of view. So I wanted to go crazy and make a data set that no one has ever seen in terms of its quantity and diversity and everything. So ImageNet, after three years, was a curated data set of internet images that totaled 15 million images across 22,000 concepts, object category concepts. And that was the data set.
Just for comparison, at the same time in Toronto, we were making a data set called CIFAR-10 that had 10 different classes and 60,000 images, and it was a lot of work. It was generously paid for by CIFAR at 5 cents an image.
And so you turned the data set into a competition. Just walk us through a little bit of what that meant, and then we'll kind of fast forward to 2012.
Right. So we made the data set in 2009. We barely made it into a poster in an academic conference, and no one paid attention. So it was a little desperate at that time. And I believe this is the way to go. And we open-sourced it, but even with open source, it wasn't really picking up. So my students and I thought, well, let's get a little more drive off the competition. Let's create a competition to invite the worldwide research community to participate in this problem of object recognition through ImageNet. So we made an ImageNet competition, and the first feedback we got from our friends and colleagues is it's too big. And at that time, you cannot fit it into a hard drive, let alone memory. So we actually created a smaller data set called the ImageNet Challenge data set, which is only 1 million images across 1,000 categories instead of 22,000 categories. And that was unleashed in 2010. I think you guys noticed it in 2011, right?
Yes, yeah. And so in my lab, we already had deep neural networks working quite well for speech recognition. And then Ilya Sutskever said, what we've got really ought to be able to win the ImageNet competition. And he tried to convince me that we should do that. And I said, well, you know, it's an awful lot of data. And he tried to convince his friend Alex Krizhevsky, and Alex wasn't really interested. So Ilya actually pre-processed all the data to put it in just the form Alex needed. He shrunk the size of the images, yes, he shrunk the images a bit, remember, and got it pre-processed just right for Alex. And then Alex eventually agreed to do it. Meanwhile, in Yann LeCun's lab in New York, Yann was desperately trying to get his students and postdocs to work on this data set because he said the first person to apply convolutional nets to this data set, that's going to win. And none of his students were interested; they were all busy doing other things. And so Alex and Ilya got on with it. And we discovered by running on the previous year's competition that we were doing much better than the other techniques. And so we knew we were going to win the 2012 competition. And then there was this political problem, which is, we thought if we show the neural networks this competition, the computer vision people, Jitendra in particular, will say, well, that just shows it's not a very good data set. So we had to get them to agree ahead of time that if we won the competition, we'd have proved that neural networks worked. So I actually called up Jitendra, and we talked about data sets we might run on. And my objective was to get Jitendra to agree that if we could do ImageNet, then neural nets really worked. And after some discussion and him telling me to do other data sets, we eventually agreed, okay, if we could do ImageNet, then we'd have shown neural nets work. Jitendra remembers it as he suggested ImageNet and he was the one who told us to do it, but it was actually a bit the other way around. And we did it, and it was amazing. We got just over half the error rate of the standard techniques, and the standard techniques have been tuned for many years by very good researchers. I remember a standard technique at that time, the previous year, is support vector machine with sparsification, right?
That was... so you guys submitted your competition results, I think it was late August or early September. And I remember either getting a phone call or getting an email late one evening from my students who was running this, because we hold the test data, we were running on the server side. The goal is that we have to process all the entries so that we select the winners. And then by, I think it was beginning of October that year, that computer vision field's international conference, ICCV 2012, was happening in Florence, Italy. We already booked a workshop, annual workshop at the conference. We will be announcing the winner. It's the third year. So a couple of weeks before, we have to process the winning teams. Because it was the third year, and frankly, the previous two years' results didn't excite me, and I was a nursing mother at that time, so I decided not to go to the third year. So I didn't book any tickets. I'm just like, too far for me. And then the results came in. That evening phone call or email, I really don't remember, came in. And I remember, I remember saying to myself, darn it, Jeff, now I have to get a ticket to Italy. Because I knew that was a very significant moment, especially it was on convolutional neural networks, which I learned as a graduate student as a classic, you know, algorithm. And of course, by that time, there was only middle seats in economy class flying from San Francisco to Florence with a one-stop layover. So it was a grueling trip to go to Florence, but I wanted to be there.
Yeah. So, but you didn't come?
No. Well, it was a grueling trip.
But did you know that would be a historical moment?
So who ignored all your advice, right?
Who ignored my email for multiple times, because I was like, Alex, this is so cool, please do this visualization, this visualization. He ignored me. But Yann came, and it was because, for those of you who have attended these academic conferences, workshops tend to book these smaller rooms. We booked a very small room, probably just the middle section here. And I remember Yann had to stand in the back of the room because it was really packed. And Alex eventually showed up, because I was really nervous that he wasn't even going to show up. But as you predicted, at that workshop, ImageNet was being attacked. At that workshop, there were people vocally attacking, this is a bad data set, in the room, in the room, during the presentation, in the room. But not Jitendra, because Jitendra had already agreed that it counted.
Yeah, I don't think Jitendra was in the room, I don't remember. But I remember it was such a strange moment for me, because as a machine learning researcher, I knew history was in the making. Yet ImageNet was being attacked. It was just a very strange, it was an exciting moment. And I had to hop in the middle seat to get back to San Francisco, because then the next morning... So there you mentioned a few people that I want to come back to later. So Ilya, who's founder and chief scientist at OpenAI, and Yann LeCun, who subsequently went on to be head of AI at Facebook, now Meta. And there's a number of other interesting people in the mix. But before we go forward and kind of see what that boom moment created, let's just go back for a little bit. Both of you started in this with kind of a very specific goal in mind that is individual and I think iconoclastic, and you had to persevere through the moments that you just described. But kind of throughout your careers, can you just go back, Jeff, maybe start, give us a background to why did you want to get into AI in the first place?
I did psychology as an undergraduate. I didn't do very well at it, and I decided they were never going to figure out how the mind worked unless they figured out how the brain worked. And so I wanted to figure out how the brain worked, and I wanted to have an actual model that worked. So you can think of understanding the brain as building a bridge. There's experimental data and things you can learn from experimental data, and there's things that will do the computations you want, things that will recognize objects, and they were very different. And I think of it as you want to build this bridge between the data and the competence, the ability to do the task. And I always saw myself as starting at the end of things that work, but trying to make them more and more like the brain, but still work. Other people tried to stay with things justified by empirical data and try and have theories that might work. But we're trying to build that bridge, and not many people were trying to build the bridge. Terry Sejnowski was trying to build a bridge from the other end, and so we got along very well. A lot of people doing, trying to do computer vision just wanted something that worked; they didn't care about the brain. And a lot of people who care about the brain wanted to understand how neurons work and so on, but didn't want to think much about the nature of the computations. And I still see it as we have to build this bridge by getting people who know about the data and people who know about what works to connect. So my aim was always to make things that could do vision, but do vision in the way that people do it.
Okay, so we're going to come back to that because I want to ask you about the most recent developments and how you think that they relate to the brain. Fei-Fei, you... and so Jeff, just to kind of put a framework on where you started, UK to the US to Canada by mid to late 80s, you come to Canada in '87. Along that route, funding and interest in neural nets and the approaches that you're taking kind of goes like this, but I would say mostly like this, went up and down. Fei-Fei, you started your life in a very different place. Can you walk us through a little bit of how you came to AI?
Yeah, so I started my life in China, and when I was 15 years old, my parents and I came to Parsippany, New Jersey. So I became a new immigrant, and where I started was first English as second language classes, because I didn't speak the language, and just working in, you know, laundries and restaurants and so on. But I had a passion for physics. I don't know how it got into my head, and I wanted to go to Princeton because all I knew was Einstein was there. And I got into Princeton. He wasn't there by the time I got into Princeton. You're not that old. Yeah. So, but there was a statue of him. And the one thing I learned in physics, beyond all the math and all that, is really the audacity to ask the craziest questions, like the smallest particles of the atom world, or the boundary of space-time and beginning of universe. And along the way, I discovered brain as a third-year undergrad. Roger Penrose and those books. Yeah, you might have opinions, but at least I've read those books. And it was probably better you didn't. Well, he, you know, it at least got me interested in brain. And by the time I was graduating, I wanted to ask the most audacious question as a scientist. And to me, the absolute most fascinating, audacious question of my generation, that was 2000, was intelligence. So I went to Caltech to get a dual, pretty much a dual PhD in neuroscience with Christof Koch and in AI with Pietro Perona. So I echo what you said about bridge, because that five years allowed me to work on computational neuroscience and look at how the mind works, as well as to work on the computational side and try to build that computer program that can mimic the human brain. So that's my journey. It starts from physics.
Okay, so your journeys intersect at ImageNet 2012. By the way, I met Jeff when I was a graduate student, right? I remember I used to go visit Pietro's lab. Yeah. In fact, he actually offered me a job at Caltech when I was 27.
You would have been my adviser.
No, I would not. Not when I was 27. Okay. Um, okay, so we intersected at ImageNet. For those in the field, everyone knows that ImageNet is this Big Bang moment. And subsequent to that, first the big tech companies come in and basically start buying up your students and you to get them into the companies. I think they were the first ones to realize the potential of this. I'd like to talk about that for a moment, but kind of fast-forwarding, I think it's only now since ChatGPT that the rest of the world is catching up to the power of AI, because finally you can play with it, you can experience it. You know, in the boardroom they can talk about it and then go home and then, you know, their 10-year-old kid has just written a dinosaur essay for fifth grade with ChatGPT. So that kind of transcendent experience of everyone being able to play with it, I think, has been a huge shift. But in the period in between, which is 10 years, there is kind of this explosive growth of AI inside the big tech companies, and everyone else is not really noticing what's going on. Can you just talk us through your own experience, because you experienced a kind of a ground zero post-ImageNet?
It's difficult for us to get into the frame of everybody else not realizing what was going on, because we realized what was going on. So a lot of the universities you'd have thought would be right at the forefront were very slow in picking up on it. So MIT, for example, and Berkeley. I remember going even to talk in Berkeley in, I think, 2013, when already AI was being very successful in computer vision. And afterwards, a graduate student came up to me and he said, I've been here like four years, and this is the first talk I've heard about neural networks. They're really interesting. Well, Stanford probably, probably. But the same with MIT. They were rigidly against having neural nets. And the ImageNet moment started to wear them down, and now they're big proponents of neural nets. But it's hard to imagine now, but around 2010 or 2011, there was the computer vision people, very good computer vision people, were really adamantly against neural nets. They were so against it that, for example, one of the main journals, the IEEE Transactions on Pattern Analysis and Machine Intelligence, PAMI, had a policy not to referee papers on neural nets at one point. Just send them back, don't referee them, it's a waste of time, it shouldn't be in PAMI. And Yann LeCun sent a paper to a conference where he had a neural net that was better at identifying, at doing segmentation of pedestrians than the state of the art, and it was rejected. And one of the reasons it was rejected was one of the referees said this tells us nothing about vision, because they had this view of how computer vision works, which is you study the nature of the problem of vision, you formulate an algorithm that'll solve it, you figure out how to implement that algorithm, and then you publish a paper. And the fact it doesn't work... I have to defend my field, not everybody, not everybody. So there are people who are, but most of them were adamantly against neural nets. And then something remarkable happened after the ImageNet competition, which is they all changed within about a year. All the people who have been the biggest critics of neural nets started doing neural nets, much to our chagrin. And some of them did it better than us. So Zisserman in Oxford, for example, made a better neural net very quickly. But they behaved like scientists ought to behave, which is had this strong belief this stuff was rubbish, because ImageNet we could eventually show that it wasn't, and then they changed. So that was very comforting.
And just to carry it forward, so what you're trying to show is you're trying to label using the neural nets these 15 million images accurately. You've got them all labeled in the background so you can measure it. The error rate when you did it dropped from 26% the year before, I think, to 16% or so?
Okay. And then it subsequently keeps dropping.
15.3 to... I knew, I knew you would remember. Randomization. Jeff doesn't forget things.
And then subsequent years, people are using more powerful neural nets, and it continues to drop to the point where it surpasses... 2015. So there is a Canadian, very smart Canadian, who joined my lab. His name is Andrej Karpathy, and he got bored one summer and said, I want to measure how humans do. So you should go read his blog. So he had all these, like, human doing ImageNet test parties. He had to bribe them with pizza, I think, with my students in the lab. And they got to an accuracy of about 5%.
And that was it. Five or 3.5?
Three, three, 3.5, I think. So humans basically make mistakes about 3% of the time, right? And then I think 2016, I think ResNet passed it, right? It was ResNet, is that year's winning algorithm, passed the human performance. And then ultimately, you had to retire the competition because it was so much better than humans that we had to retire.
Because we ran out of funding.
Okay, all right. A better, a different reason. Still ran out of funding.
Incidentally, that student started life at the University of Toronto before he went to your lab, and then he went to be head of research at Tesla.
Okay, first of all, he came to Stanford to be a PhD student. And yesterday night we were talking, actually, there was a breakthrough dissertation, you know, in the middle of this. And then he became part of the founding team of OpenAI. But then he went to Tesla.
And then he went to Tesla, and then he thought better of it. He's back. But I do want to answer your question of that 10 years. Well, there's a couple of developments along the way. Transformers, right? So the Transformer paper is written, the research done, paper written inside Google. Another Canadian is a co-author there, Aidan Gomez, who's now the CEO and founder, and co-founder of Cohere, who I think was a 20-year-old intern at Google Brain when he co-authored the paper. So there's a tradition of Canadians being involved in these breakthroughs. But Jeff, you were at Google when the paper was written. Was there an awareness inside Google of how important this would be?
I don't think there was. Maybe the authors knew, but it took me several years to realize how important it was. And at Google, people didn't realize how important it was until BERT. So BERT used Transformers, but then became a lot better at a lot of natural language processing benchmarks for a lot of different tasks. And that's when people realized Transformers were special. So 2017, the Transformer paper was published. I also joined Google, and I think you and I actually met on my first week, I think, most of 2017 and 2018 was neural architecture search, right? I think that was Google's bet. And there was a lot of GPUs being used. So it was a different, it was a different bet.
So just to explain that, neural architecture search essentially means this: you get yourself a whole lot of GPUs and you just try out lots of different architectures to see which works best, and you automate that. It's basically automated evolution for neural net architectures, like hyperparameter tuning. And it led to some good, quite big improvements, but nothing like Transformers. And Transformers were a huge improvement for natural language. Neural architecture search was mostly done on ImageNet, yeah. So I'll tell you our experience of Transformers. So we were doing our company Layer 6 at the time. I think we did a pre-, we saw a pre-print of the paper, and we were in the middle of a fundraising and a bunch of acquisition offers. And read the paper, and I mean, not just me, but my partner Tommy, who had studied with you and Max Welling, who came out of your lab. And we thought, this is the next iteration of neural nets. We should sell the company, start a venture fund, and invest in these companies that are going to be using Transformers. So we figured it would take five years to get to be on Google, and then from that moment forward, it would be 10 years for all the software in the world to get replaced or embedded with this technology. We decided, we made that decision five years and two weeks before ChatGPT came out. So I'm glad to see we were good at predicting, but I have to give credit to my co-founders who I thought I understood what the paper was, but they were able to explain it fully.
I should just correct you about, and think, I don't think Tommy ever studied with me. He wanted to come study with me, but a colleague in my department told him if he came to work with me, that would be the end of his career, and he should go do something else. So I don't, yeah, so he took the classes, but he...
And this is my partner who in the late 90s was doing a master's at U of T, and he wanted to go study with Jeff, study neural nets. And his girlfriend, now wife's, father, who was an engineering professor, said, don't do that, neural nets are a dead end. So instead he took the classes, but wrote his thesis in cryptocurrency. So, okay, so but are you still going to talk about the 10 years? Because I think there's something important.
Yeah, so go ahead. So I do think there's something important. The world overlooked this 10 years between ImageNet, AlexNet, and ChatGPT. Most of the world sees this as a tech, tech 10 years, you know, or we see it as a tech 10 years in the big tech. There's things brewing. I mean, it took sequence-to-sequence, Transformer, but things are brewing. But I do think, as for me personally and for the world, it's also a transformation between tech to society. I actually think personally, I grew from a scientist to a humanist in this 10 years. Because having joined Google for that two years in the middle of the Transformer papers, I begin to see the societal implication of this technology. It was post-AlphaGo moment, and very quickly we got to the AlphaFold moment. It was where bias was creeping out, there was privacy issues, and then we're starting to see the beginning of disinformation and misinformation, and then we're starting to see the talks of jobs, within a small circle, not within a big public discourse. It was when I grew personally anxious. I feel, you know, 2018... oh, oh, it was also right after Cambridge Analytica. So that huge implication of technology, not AI per se, but it's algorithm-driven technology on election. That's when I had to make a personal decision of staying at Google or come back to Stanford. And I knew the only reason I would come back to Stanford was starting this Human-Centered AI Institute to really, really understand the human side of this technology. So I think this is a very important 10 years, even though it's kind of not in the eyes of the public. But this technology is starting to really creep into the rest of our lives. And of course, 2022, it's all shown under the daylight how profound this is.
There's an interesting footnote to what happened during that period as well, which is ultimately you and Ilya and Alex joined Google, but before that there was a big Canadian company that had the opportunity to get access to this technology. You know, I've heard this story, but I don't think it's ever been shared publicly. Maybe do you want to share that story for a second?
Okay, so the technology that we were using for ImageNet, we developed it in 2009 for doing speech recognition. We were doing the acoustic modeling bit of speech recognition. So you can take the sound wave and you can make a thing called a spectrogram, which just tells you at each time how much energy there is at each frequency. So you're probably used to seeing the spectrograms. And what you'd like to do is look at a spectrogram and make guesses about which part of which phoneme is being expressed by the middle frame of the spectrogram. And two students, George Dahl and another student who I shared with Gerald Penn, called Abdel-rahman Mohamed, he had a longer name, we all called him Abdo, who was a speech expert, George was a learning expert. Over the summer of 2009, they made a model that was better than what 30 years of speech research had been able to produce, and big, big teams working on speech research. And the model was slightly better, not as big as the ImageNet gap, but it was better. And that model was then ported to IBM and to Microsoft. By George went to Microsoft, and Abdo went to IBM, and those big speech groups started using neural nets then. And I had a third student who'd been working on something else called Navdeep Jaitly. And he wanted to take this speech technology to a big company, but he wanted to stay in Canada for complicated visa reasons. And so we got in touch with BlackBerry RIM, and we said, we've got this new way of doing speech recognition, and it works better than the existing technology, and we'd like a student to come to you over the summer and show you how to use it, and then you can have the best speech recognition in your cell phone. And they said, after some discussions, a fairly senior guy at BlackBerry said, we're not interested. So our attempt to give it to Canadian industry failed. And so then Navdeep took it to Google, and Google were the first to get it into a product. So in 2012, around the same time as we won the ImageNet competition, George and Abdo's speech recognition acoustic model, the acoustic model was in there, was a lot of work making it a good product and making it have low latency and so on, that came out in the Android. And there was a moment when the Android suddenly became as good as Siri at speech recognition, and that was a neural net. And I think for the people high up in the big companies, that was another ingredient. They saw it get this dramatic result for vision, but they also saw that it was already out in a product for speech recognition, was working very well there too. So I think that combination, if it does speech, it does vision, clearly it's going to do everything. We won't see anymore about it. It was a shame that Canadian industry didn't, you know, I think we might have still had BlackBerries if that happened.
All right, we'll leave that one there. I thought it was a story, I've heard this story before, but I thought it was important for the rest of the world to know some of what went on behind the scenes, why this technology didn't stay in Canada even though it was offered. Okay, so let's advance forward. We now have post-Transformers. Google is starting to use this and develop it in a number of different ways. OpenAI, where your former student Ilya had left Google, been a founder of OpenAI with Elon Musk and Sam Altman, Greg Brockman, and a few others. Ilya is the chief scientist. They and Andre, your student, as a co-founder. So they are working together, a very small team, to basically take, well, initially the idea was we're going to build AGI, an artificial general intelligence. Ultimately, the Transformer paper comes out, they start to adopt at some point Transformers, and they start to make extraordinary gains internally that they're not really sharing publicly in what they're able to do in language understanding and a number of other things. They had efforts going on in robotics that spun out, Pieter Abbeel ended up spinning out Covariant, a company we sub-invested in, and other things. But so the language part of it advances and advances and advances. People outside OpenAI don't really know to the extent what's going on. And then ChatGPT comes out November 30th last year, so 10 months ago.
GPT-2 caught the attention of some of us, I think. Actually, I think by the time GPT-2 came out, my colleague Percy Liang, NLP professor at Stanford, I remember he came to me and said, Fei-Fei, I have a whole different realization of how important this technology is. So to the credit of Percy, he immediately asked HAI to set up a center to study this. And I don't know if this is contentious in Toronto, Stanford is the university that coined the term foundation models. And some people call it LLM, large language model, but going beyond language, we call it a foundation model. We created the center of research for foundation models before, I think before 3.5 came out, so before, definitely before ChatGPT.
And just describe what a foundation model is, just for those who are not familiar.
That's actually a great question. Foundation model, some people feel it has to have Transformer in it. I don't know if you... it just has to be a very big, a huge amount of data, pre-trained with huge amount of data. And I think one of the most important things of a foundation model is the generalizability of multiple tasks. You're not training it, for example, machine translation. So in NLP, machine translation is a very important task, but the kind of foundation model like GPT is able to do machine translation, it's able to do conversation, summarization, and blah, blah, blah. So that's a foundation model. And we're seeing that now in multimodality. We're seeing it in vision, in robotics, in video, and so on. So we created that. But you're right, the public sees this in the, what did you say, October 30th, November, I think November 30th. One other very important thing about foundation models, which is for a long time in cognitive science, the general opinion was that these neural nets, if you give them enough training data, they can do complicated things, but they need an awful lot of training data. They need to see thousands of cats. And people are much more statistically efficient, that is, they can learn to do these things on much less data. And people don't say that so much anymore, because what they were really doing was comparing what an MIT undergraduate can learn to do on limited amount of data with what a neural net that starts with random weights can learn to do on a limited amount of data. It's an unfair comparison. And if you want to make a fair comparison, you take a foundation model, that is, a neural net that's been trained on lots and lots of stuff, and then you give it a completely new task, and you ask how much data it needs to learn this completely new task. And that's called few-shot learning, because it doesn't take much. And then you discover these things are statistically efficient, that is, they compare quite favorably with people in how much data they need to learn to do a new task. So the old kind of nativist idea that we come with lots of innate knowledge and that makes us far superior to these things that just learn everything from data, people have pretty much given up on that now, because if you take a foundation model that had no innate knowledge but a lot of experience, and then you give it a new task, it learns pretty efficiently. It doesn't need huge amounts of data.
You know, my PhD is in one-shot learning.
But it's very interesting, even in Bayesian framework, you could pre-train, but it's only the neural network kind of pre-training really can get you this multitask, right?
So, okay, so this basically gets productized in ChatGPT. The world experiences it, which is only 10 months ago, although for some of us it feels like much longer, forever. Because suddenly you have this, you know, you had this Big Bang that happened a long time ago that I think for a long time no one really saw the results of it. Suddenly, I mean, my comparison would be there's planets that have formed and stars that are visible, and, you know, everyone can experience the result of what happened 10 years before and then Transformers, etc. So the world suddenly becomes very excited about what I think feels to a lot of people like magic, something that they can touch and they can experience and gives them back a feedback in whatever way they're asking for it. Whether they're asking, they're putting in text prompts and asking for an image to be created, or video, or text, and asking for more text to come back and answer things that you would never be able to expect, and getting those unexpected answers. So it feels a little bit like magic. My personal view is that, you know, we've always moved the goal line in AI. AI is always the thing that we couldn't do, it's always the magic. And as soon as we get there, then we say that's not AI at all. There's people around that say that's not AI at all. We move the goal line. In this case, what was your reaction when it came out? I know part of your reaction is you quit Google and decided to do different things, but when you first saw it, what did you think?
Well, like Fei-Fei said, GPT-2 made a big impression on us all, and then there was a steady progression. Also, I'd seen things within Google before GPT-4 and GPT-3.5 that were just as good, like PaLM. So that in itself didn't make a big... it was more PaLM that made an impression on me within Google, because PaLM could explain why a joke was funny. And I'd always just used that as a, you know, we'll know that it really gets it when it can explain why a joke is funny. And PaLM could do that, not for every joke, but for a lot of jokes. And so, incidentally, these things are quite good now at explaining why jokes are funny, but they're terrible at telling jokes. And there's a reason, which is they generate text one word at a time. So if you ask them to tell a joke, what they do is they're trying to tell a joke, so they're going to try and tell stuff that sounds like a joke. So they say, you know, a priest and a badger went into a bar, and that sounds a bit like the beginning of a joke. And they keep going, telling stuff that sounds like the beginning of a joke. But then they get to the point where they need the punchline, and of course they haven't thought ahead, they haven't thought what's going to be the punchline, they're just trying to make it sound like they lead into a joke. And then they give you a pathetic, weak punchline, because they have to come up with some punchline. So although they can explain jokes, because they get to see the whole joke before they say anything, they can't tell jokes. But we'll fix that.
Okay, so I was going to ask you if comedian is a job of the future or not. You think it's going to... probably not. All right. So anyway, so what was your reaction to it? Again, you've seen things behind the scenes along the way.
A couple of reactions. My first reaction is, of all people, I thought I knew the power of data, and I was still awed by the power of data. That was a technical reaction. I was like, darn it, I should have made a bigger ImageNet. No, but maybe not. But that was really... you still could... funding is the problem. Yeah. So that was first. Second, when I saw the public awakening moment to AI with ChatGPT, not just the GPT-2 technology moment, I generally thought, thank goodness we've invested in human-centered AI for the past four years. Thank goodness we have built a bridge with the policymakers, with the public sector, with the civil society. We have not done enough, but thank goodness that that conversation had started. We were participating, we were leading some part of it. For example, we as an institute at Stanford were leading the critical National AI Research Cloud bill that is still going through Congress right now. And not right now, actually, Senate, Senate, it's by Kamala, so at least it's moving the Senate. Because we predicted the societal moment for this tech. We don't know where, when it would come, but we knew it would come. And it was just a sense of urgency, honestly. I feel that this is the moment we really have to rise to not only our passion as technologists, but responsibility as humanists.
And so you both, I think the common reaction of you both has been we have to think about both the opportunities of this, but also the negative consequences of it. So for me, there was something I realized and didn't realize till very late, and what got me much more interested in the societal impact was, like Fei-Fei said, the power of data. These big chatbots have seen thousands of times more data than any person could possibly see. And the reason they can do that is because you can make thousands...
You can have many copies of the same model and each copy can look at a different subset of the data. They can get a gradient from that of how to change their parameters and they can then share all those gradients so every copy can benefit from what all the other copies extracted from data. And we can't do that. Suppose you had 10,000 people and they went out and they read 10,000 different books and after they've each read one book all of them know what's in all the books. We could get very smart that way and that's what these things are doing. And so it makes them far superior to us. There's some schooling that we're trying to do that but not in the way. But education is just hopeless. I mean it's hardly worth paying for.
So except University of Toronto and HAI. I've tried to explain to friends that Jeff has a very sarcastic sense of humor and if you spend enough time around it you'll get it. But I'll leave it to you to decide whether that was...
We exchange knowledge roughly speaking. This is something of a simplification but I produce a sentence and you figure out what you have to change in your brain so you might have said that. That is if you trust me. We can do that with these models too. If you want one architecture to know what another architecture knows, which is a completely different architecture, you can't just give it the weights. So you get one to mimic the output of the other, that's called distillation. And that's how we learn from each other but it's very inefficient. It's limited by the bandwidth of a sentence which is a few bits. Whereas if you have these models, these digital agents which have a trillion parameters, each of them looks at different bits of data and then they share the gradients. They're sharing a trillion numbers. So you're comparing an ability to share knowledge that's in trillions of numbers with something that's hundreds of bits. They're just much much better than us at sharing.
So Jeff, I agree with you at the technology level but it sounded like for you that's the moment that got you feeling very negative.
That's the moment I thought we're history.
Yeah, I'm less negative than you and I'll explain later. But I think that's where... actually let's talk about that. Explain why you are optimistic and let's understand why you are more pessimistic.
