Hi, I'm Craig Smith and this is I on AI. This week I speak to Stanford professor Fei-Fei Li, one of the people responsible for the current AI revolution with her creation of ImageNet, the world's first large labeled image data set, which allowed the validation of neural networks. She talked about her early days running New Jersey dry cleaners to finance her Princeton education, her journey into artificial intelligence, and her latest work on ambient intelligence, which promises to transform eldercare. I hope you find the conversation as informative as I did.
I'm Fei-Fei Li. I'm a professor at Stanford Computer Science Department and also the co-director of Stanford's Institute for Human-Centered AI. So I spend my day both doing lots of research with my students and also running this Institute. I'm also very fortunate and proud that I co-founded the national nonprofit AI for All, which is an organization dedicated to improving human representation, diversity, and inclusion in AI by creating high school K-12 education opportunities for underrepresented and underserved populations throughout the country, especially for racial, gender, socioeconomic class, and geographically underrepresented minorities.
Tell us about how you got to Stanford. You came to the States as a little kid and went to school in New Jersey, and then after high school, what happened?
Yeah, so I think I would call myself a typical immigrant story. My parents brought me to this country when I was in my early teens, so I was born in China, but my education, you know, that's very influential to who I am. Started in a small public high school called Parsippany High School in New Jersey. I entered as a freshman in the middle of the school year when I came to this country speaking practically no English at all. And my parents didn't speak English, but they believed in this land of freedom and opportunity, and they brought me here. So throughout my high school, my biggest memory is, you know, how American high schools have these giant textbooks. I remember I had to carry not only the giant textbooks but giant dictionaries to learn the language. But we also worked very hard as a family of immigrants. I started by doing some housecleaning jobs and restaurant jobs, especially working back in the kitchen because my English was not good enough to interact with the customers yet. And eventually, I was very lucky that I got into Princeton University. I was a passionate student of physics, and I think I was very lucky because one of my interviewers was a former physics major, and he shared so much my passion, and I really believe he helped me get into Princeton. So at Princeton, I majored in physics. Another parallel life to my college life was that by the time I got into Princeton, my family and I realized even though I got a lot of scholarships and fellowships from Princeton, it was still expensive. So we borrowed money, especially from my high school math teacher in New Jersey, and opened a very tiny family dry cleaner shop. And I guess I was the de facto CEO at that time. I spoke English and my parents still didn't, and they still don't know. I employed my parents, worked on the weekends. If you know anything about dry cleaning business, it's a weekend business. So I worked on the weekends back in Parsippany at that little dry cleaner shop and then studied during the week. And I just have the best, most grateful memory of my Princeton education because it opened my eyes to the field I really, really love, which is science. I knew since I was a kid I love science, so I started reading about physicists and their lives. And I noticed some of the greatest physicists of the 20th century, like Schrödinger, Einstein, Penrose, they started to be so intrigued by the signs of life and intelligence, not just the signs of the universe and atoms. And that took me on a path, in the middle of my college, to be more interested in the question of intelligence in life rather than pure, hard, atomic physics. So I had a couple of summer internships in various colleges that got me to dabble into neuroscience, and I did a senior thesis in the computer science department at Princeton, all kind of starting to pull me into the orbit of machine intelligence and human intelligence. And to be honest, at that time it was AI winter; nobody calls it artificial intelligence. So after I finished college, I became very, very lucky: I got into Caltech for PhD. I have to say I had a couple of other offers from MIT and Stanford, but Caltech was so beautiful and warm and had a program that allowed me to do interdisciplinary research in both computer vision and human cognitive neuroscience that I really fell in love with Caltech. So I went through Caltech for my PhD and had two PhD advisors: one was a famous German neuroscientist, Christof Koch, and one is a famous Italian AI professor, Pietro Perona. So I studied under both of them and began my career as an AI researcher at Caltech. In the meantime, I was running my family business remotely — my parents were still working back in New Jersey, Parsippany — but my mom's health deteriorated so much that in the middle of my PhD time, I had to sell the dry cleaning shop and bring my parents to California. And since then, I've been taking care of my parents because of their health condition, which I will tell you later has had a profound impact on my understanding of healthcare and the aging population, for COVID-19. But after Caltech, I think one of the most important things to highlight is that for the outside world, the late 20th century and the first decade of the 21st century in AI is a dormant period, what people call AI winter. But as a researcher, it was a vibrant period of ideas, especially the convergence of machine learning techniques, including neural networks, the explosion of data and internet technology, and also the decades of discoveries and breakthroughs in cognitive neuroscience started to converge and give us important North Star questions in AI. And I was very lucky; I was in the lab, or in two labs, that were really focusing on one of the most important North Star questions of AI: the ability that humans have to perceive the complex world of objects and be able to name and recognize tens of thousands of objects in our rich visual world. And my advisors got me really interested in this question, and that was the beginning of my dissertation. And I was also the first generation — I would say one of the very early generation PhD students — who learned the new tool of machine learning, which was a budding sister field to computer vision. Statistical models combined with the power of computing started to show their effectiveness in rethinking these big AI problems. So we used techniques like neural networks, Bayes nets, graphical models, modern inference techniques, SVM, and all that. So after Caltech, I accepted a faculty job at the University of Illinois Urbana-Champaign. It was my first faculty job, and it got me to know the middle of America better as an immigrant who had lived on the coasts of our country, and I really appreciated that year of experience as a young professor. But again, very luckily, Princeton gave me a call during my first year as an assistant professor and said, 'We're looking for a new assistant professor in the Computer Science Department who studies AI.' And when your alma mater calls you, you take the call, right? I was very, very grateful, and I went to Princeton for a couple of visits. But one of those visits was very fateful for starting ImageNet.
It was a serendipitous conversation I had with a linguist who was very kind to just offer to interview me or recruit me, you know, in that process of offering me a job at Princeton. And she heard I was very interested in cracking a Holy Grail problem of computer vision: object recognition, the process of naming objects like cats, dogs, chairs, microwaves, cars, trees, and all that. And then she said, 'Have you heard of a linguistic project called WordNet?' And I had never heard of WordNet before. But with a little bit of research, you realize WordNet was one of the most profound, important linguistic and natural language projects that emerged out of Princeton in the 1990s, which reorganized the entire English lexicon in a taxonomy that is different from typical dictionaries. The lexicons are organized by their relationship, such as what we call the 'is-a' relationship: a German Shepherd would be related to a dog, which would be related to an animal. And that was a large-scale project that really had profound influence in linguistics and computational linguistics. And this researcher mentioned to me that wouldn't it be nice if every node, or every set of words like German Shepherd or tree, in this tens of thousands of entry WordNet dictionary or taxonomy, just had a picture attached to it, so that people who go to WordNet would know what a German Shepherd looks like, or what a panda bear looks like, or what a microwave looks like. And then she said she tried this project with a bunch of undergrads at Princeton and it didn't go very far for a couple of reasons. One is it's not clear it's useful for linguistic research to have a picture attached to the word. Second is that it's really hard: the undergrads had to go through tens of thousands of entries and find the picture, so it just didn't work. But that conversation really was like a spark of light in the darkness. I had been struggling to try to make object recognition work, and suddenly I was thinking there's a role that data can play in a way that we never paid attention to. We've paid so much attention to tuning the parameters of our models, but mathematically we're always running into the problem of overfitting our model without enough data, and lack of generalization. These are kind of jargon words in machine learning, but they point to the mathematical fact that models are hard to fit and one needs lots of good data to drive the model. And that was an insight that wasn't prominent in the field yet. I realized maybe we should try something completely radically new: instead of spending time to tweak the parameters, we should create a large database of pictures of many, many tens of thousands of different kinds of objects and drive the capacity of the models to a whole different state and see how that goes for this important problem of object recognition. So I asked the linguistic researcher, 'Is anybody doing this project?' She said no. She said, 'In fact, we had a name for it and it's called ImageNet, but it's a terminated project.' So I said, 'Would you mind if I started it, but in a completely different way for my computer vision research? But I really like the name, can I inherit the name ImageNet?' She said, 'By all means, just take it.' So that's really the beginning story of ImageNet. It was during my transition into Princeton. So I moved my small, tiny lab to Princeton in 2007 and we began the ImageNet project. And the idea is that we would take 22,000 nouns that are countable and concrete in the WordNet taxonomy, and these nouns are conceptually visual — they're nouns that are not like 'love,' which is harder to visualize, but nouns like 'chair' are so. We wanted to provide hundreds and thousands of pictures from all kinds of sources to drive the diversity and variability of each concept. So if you multiply those numbers together, we're looking at tens of millions of curated pictures. And to do that, we had to download nearly a billion pictures from the internet, and the downloading process itself was very interesting in 2007. And then we needed to find a way to curate them, and we struggled a lot. First, we also went to the undergrads and tried to entice them to label, and that was just impossible; at the hourly rate, trying to hope we could get undergrads to label a billion images, my PhD student then was working with me and did a back-of-the-envelope computation and said, 'I won't graduate for another 19 years if we did this.' So we also tried other ways to get computers to label, but that was actually just philosophically the wrong way to do it, because we're trying to curate a ground truth training data set to improve the computer's ability. If we used anything that was based on the existing ability of computers, we would introduce very low quality, erroneous data. So we had to go to humans. And during early 2007, or maybe in the middle of 2007, another serendipitous hallway conversation changed everything. It was with a master's student who happened to come from Stanford and was at Princeton, and he said to me, 'Have you heard of Amazon Mechanical Turk?' And I said I had never. He said, 'I heard there's a Silicon Valley startup that didn't have enough people to label some data — I forgot what kind of data, some color or wine bottle tags, one of those things — and they used this very new online worker service that Amazon had beta tested. It's a global online worker market; people just post jobs and people worldwide do the jobs.' I remember I was very busy teaching that day, and I went home at night and logged into Amazon Mechanical Turk. That night I knew ImageNet would happen, because I had never seen a platform of that meaning. Now we call it crowd workers that can contribute on a global scale. Fast forward to 2009, we rolled out ImageNet as a research paper in our community. By that time, we had almost 60,000 online workers from more than 150 countries working and contributing to the curation of ImageNet. In 2009, we rolled out ImageNet. In the meantime, we open-sourced it to the research and education community, and we felt strongly that this is a path to our North Star — one of our North Stars. And in order to create this path and invite more researchers worldwide to participate with us, we had to make a challenge. We had to make an international challenge so that we could roll out a benchmark that would test our researcher community. So in 2010, we started what we call the ImageNet Challenge benchmark, which invited worldwide research teams to come and work on the problem of object recognition, and we would release the challenge results annually and have an international workshop to talk about the results. In 2010 and 2011, there were slow progress, but the progress was not significant. And in 2012, I remember the deadline for the challenge was in late summer because we wanted to announce this at the international workshop of our annual computer vision conference, which rotates around the world. But that year it was in Florence, Italy, in September or October, so we needed to process the results in late summer. I got a call from my graduate student at night who said, 'We've got a remarkable result, and we need to check if this is wrong,' because the error rate was just cut in half from last year. And they also made a comment that it used an algorithm that we have known for 30 years, and we didn't realize this algorithm could be this powerful. It turned out to be the entry to the 2012 ImageNet Challenge by Geoff Hinton and his students, and it was the winning entry of the ImageNet Challenge that year. A small story there: I think I was lucky to realize that was a historical moment. So I wasn't going to go to Florence, Italy that year to attend the conference because my child was very small — I was a nursing mom. But I realized it was so significant that I bought a last-minute ticket to Italy. I squeezed in the middle seat, cursing my middle seat the entire trip. I was in the air back and forth for probably 40 hours; I was on the ground for less than 18 hours, just to go there, announce the result, and lead the workshop, because it was just so significant. And I remember at that workshop, Geoff Hinton didn't come, but his student Alex came, and also young Ilya Sutskever came, and a number of very prominent AI and computer vision researchers came. And there was palpable energy in the room. You know, researchers' energy is not everybody clapping; it's really thinking deeply about this result and debating, and some people even expressing skepticism and pointing out the potential pitfalls. So there was a lot of discussion. But in a couple of months, Geoff Hinton published this ImageNet paper and it went viral, and that was the paper that, fast forward eight years later, got them the Turing Award. And that algorithm was the backpropagation algorithm.
Is that right?
Yes, it was a convolutional neural network using backpropagation. It had a couple of modern tweaks, but I think one of the most important changes to that very classic algorithm: one is definitely ImageNet in the data, the other one is GPU. It was the first time that they used two GPUs and got the model to learn onto GPU, because the model is very, very high capacity. It has a huge number of parameters driven by a large amount of data. Traditional CPUs just cannot handle that. So without the GPU and parallel computing, we wouldn't have that result in 2012.
So that started these massive labeled data sets that have been growing all over the world, and algorithms and models are becoming increasingly general in their ability to classify, depending on the kind of model. It seems to me eventually all of that labeled data will flow together into this data sea. And as the models become increasingly able to generalize, they'll have this vast sea of encoded human knowledge, because that's really what it is — that's what labeled data is, encoding human knowledge. Do you see it that way, that there is a future where all of these labeled datasets will come together and algorithms that can generalize will be able to draw on that massive data?
Great question, Craig. It's already starting to happen. One of the latest exciting trends in AI research, especially in the machine learning community, is meta-learning and transfer learning. And when it comes to these ideas of transfer learning, you can see that data from different domains or data from different sources can lend themselves to algorithms that can learn across data and aggregate the kind of what you call knowledge or information. So it's that kind of aggregation and transferring, and meta-learning is all part of the ongoing research of the data sea, as you call it. I do want to call out that even with labels, there's knowledge in data. You know, human babies learn without labels in many scenarios; they learn by trial and error, they learn by rewards, they learn by curiosity. Some of my own ongoing research is in that area. So even the concept of 'do we really need labeled data?' is being challenged, and we're seeing a heterogeneous approach to learning with lots of data these days.
And then moving into the COVID research you mentioned in one of your talks, the need for COVID data repositories. And I had a conversation last week with the Radiological Society of North America, which has a project to aggregate data into a repository. Are there other projects like that, or is that going to be the principal project for North America?
So RSNA, I think it's the organization you're referring to, right? I definitely want to be very clear: I'm definitely not a doctor, and I still feel I'm a student in healthcare research. So I know that there are many, many efforts across medicine and healthcare in data collection and data aggregation. Of course, when it comes to a field like healthcare, we have to make it really clear that there are guardrails and regulatory guidelines when it comes to respecting privacy, personal information, and so on. But in general, researchers in the community — whether radiology, medicine, or genomics — are recognizing the importance of data. I know at Stanford, some of my colleagues are leading a project even called Medical ImageNet. I think that was led by a bunch of radiologists, and I actually look forward to knowing the latest of their progress.
