Yann LeCun: Where Is Artificial Intelligence Heading?

Not good. The people who advocate for the pause are a bit inconsistent in the opinions they express. So in particular Dario Amodei, who is a bit behind this, followed a bit by Sam Altman, Elon Musk, etc. It's partly to wrap themselves in a kind of ethical envelope to show people, you see, we are really responsible. We could just maximize the speed of progress to make money, but we are so responsible that we decided to slow down. So, it's not necessarily very honest, but it stems from an illusion on Dario Amodei's part. You have to remember that in 2019, Dario Amodei, who at the time worked at OpenAI, had said 'We are not going to distribute GPT-2 in open source because you understand it's potentially very dangerous.' So at the time I made fun of them, I really made fun of them. I completely trolled them, actually, because I found it ridiculous. It was an enormous superiority complex, both of their model and of themselves. Six months later, they finally decided to distribute GPT-2 in open source, and then when GPT-3 came out, they didn't release it in open source, not for safety reasons, although they said it was the case, but simply to preserve their technological and commercial advantage.

It was also the time when Dario Amodei left OpenAI to found Anthropic because he thought OpenAI was taking a bit too many liberties with the distribution of intelligent models. But what you have to understand is that the investor who enabled the creation of Anthropic is a foundation called Open Philanthropy, which actually belongs to an intellectual movement in Silicon Valley called effective altruism, in which the people who belong to this movement, which you can liken to a kind of religious sect, are very, very afraid of existential risk and have set out to save humanity from its own foolishness, you could say. It is essentially funded by people who have a lot of money and too much free time. People like Dustin Moskovitz, who is one of the co-founders of Facebook, who left Facebook very early and who has become, in certain ways, paranoid about these existential risks. People like Jaan Tallinn, who is a charming guy, by the way, the creator of Skype, but who also has set out to save humanity from itself. Something that Elon Musk also shares, by the way. That's why he wants to go to Mars. And a few others, Vitalik Buterin and others. So people with a lot of money who, starting from a good intention of saving humanity, are actually doing enormous foolishness.

So the creation of Anthropic perhaps started from a good feeling: make AI that will do no harm. And the way it is used today, Dario Amodei's argument, with which he speaks to several governments around the world, is to say: you understand, AI is really dangerous, it must not be put in everyone's hands. So we must regulate AI to prohibit, in particular, open-source AI, particularly today because it comes from China and we can't trust the Chinese, but especially because AI in everyone's hands will lead to a disaster because of existential risks. So regulate AI, prohibit open source. Of course, don't regulate us because we are intelligent and we know how to manage AI. So it stems from an absolutely unbearable and misplaced superiority complex on one hand, but above all from a desire to capture the market through regulation by convincing governments around the world that AI is inherently dangerous. And it's false.

Yes, yes, I fell into the pot when I was little, but I have always been interested in the problem of intelligence. It's one of the biggest mysteries of science today. How does the brain work? What is intelligence? And then the other fundamental questions: what is the universe made of and how does life work. So there, I made my choice. Being more of an engineer by nature, I told myself that understanding intelligence involved building intelligent machines. In fact, the only way to validate a theory of intelligence is to show that it works if you build a machine based on these principles. So I have always been motivated from a scientific point of view by the discovery of the underlying principles of intelligence, and quite early on realizing that in nature, intelligence is always the result of learning and adaptation. There is no innate intelligence, intelligence is built through experience, through learning. So I always thought that learning was an absolutely essential component of intelligence. And then also, perhaps because I am lazy, I told myself, I am not smart enough to design an intelligent machine. So it's better that it designs itself in a certain way. And learning is a way to allow a machine to build itself through experience.

So, at the time, when I started, when I was finishing my engineering studies, what was trendy, so we are talking about, I finished my engineering studies in '83, I started getting interested in the question a few years before. In '83, what was trendy in artificial intelligence was what was called expert systems and rule-based systems. So AI systems in which engineers, what were called knowledge engineers at the time, would sit down, write rules and facts, and use what are called inference engines to derive new facts from existing facts and rules. And it turned out that this method was applicable to a relatively small number of situations for which it was economically feasible to actually sit down and write rules. We also realized that most intelligent tasks accomplished by humans are not governed by rules or recipes. It's much more fluid than that for, I don't know, medical diagnosis, computer fault diagnosis, things like that. Applications that were deployed. So I told myself at the time, it would be good to have machines capable of learning. I studied the question a bit and it was not at all trendy at the time because learning methods inspired by the brain had been trendy in the late 50s and early 60s but had completely disappeared, at least in the West, in the late 60s, where people realized that the methods developed at the time, in the late 50s and early 60s, were actually very limited in their capacity and would never lead to truly intelligent systems. So people continued to work in these fields but changed the name of the field, no longer claiming to be part of artificial intelligence, but saying we are going to build adaptive filters, pattern recognition systems, and that sounds much more serious. And so these techniques actually went a bit behind the curtain, without the ambitious goal of building intelligent machines.

And so I realized that in the 80s, there was no one, at least in France, but even in the United States, working on using learning to make intelligent machines. I told myself it's quite strange. So I got interested in it. I started working a bit on the question. I realized there was a small community in Japan in particular, and then psychologists in the United States, and later physicists who were interested in it a bit. But I was starting a doctorate or a DEA at the time, which is like the M2 today. And there was no one working on this. So I couldn't find a thesis supervisor. So, I met a certain Maurice Milgram who was very nice and was interested in what was called at the time automata networks. It's what is closest to what we now call neural networks. And he said 'I don't know very well what you are working on but you seem sufficiently, you don't seem stupid so I am willing to sign the papers.' He got into neural networks after I finished my thesis. So I was a bit isolated and it was at the end of my thesis that I met the people, the nascent community of neural networks, in '87.

So it was people like Geoffrey Hinton, with whom I did my postdoc, who is still very present. He retired, he no longer does research but he is still present with a bit of thunderous statements on AI safety, and he too is very afraid of the dangers of AI, in my opinion wrongly. People like Terrence Sejnowski, who is more on the side of theoretical neuroscience. And then a small research group at Bell Labs, Bell Laboratories, which was at the time the research lab of the large telecommunications company AT&T. It's a somewhat mythical laboratory where a large part of modern technologies were invented. The transistor, the laser, solar cells, CCD sensors, etc. Unix, the C and C++ languages. And so after my postdoc at the University of Toronto with Hinton, I was recruited by Bell Labs and that's how I landed in the United States. I had the idea to stay there for two years and then it was in 1988 and so I stayed a bit longer. And it's a bit there that we developed some of the basic techniques that are still used for image recognition. So if you have a car with driving assistance or a medical image analysis system, etc., it uses technologies called convolutional neural networks that I developed at the time, in the late 80s and early 90s at Bell Labs.

So actually it accelerated much earlier than that. There was a dark period between about 1995 or 1996 and let's say 2006, a period of about 10 years in which no one in the machine learning community, so there is a sub-community of artificial intelligence which is machine learning which is now completely dominant, what is called machine learning in English. And this community completely lost interest in what we now call deep learning, which was called neural networks at the time, between '95 and 2006. So actually during that period I worked on other things because there was really no possibility of having any impact on the community, it was the early years of the internet, lots of interesting things to do. I worked on a project called 'Déjà Vu' in French, 'Distributed Document Viewing' on the internet, etc. And then I came back to neural networks in 2001-2002.

I left AT&T. I was briefly at the lab of the Japanese company NEC. And then in 2003 I became a professor at New York University. And so I really restarted a research program on neural networks to show people that it worked. And in this, I was accompanied by Geoffrey Hinton and Yoshua Bengio, who were also interested in the same questions, and we got together to try to, how to say, restart the interest of the research community in these techniques which we knew worked really well but were a bit difficult to make work. It took several years but in 2012, so it took 6-7 years, in 2012, already in 2009, there was already a big resurgence of interest in these methods from the speech recognition community. We started to have speech recognition systems that worked well thanks to deep learning, neural networks. And in a few months, in a year and a half, in 2010-2011, mobile phones could suddenly understand what you were telling them thanks to neural networks. So it was deployed relatively quickly by companies like Google and Apple and others. And in 2012, another revolution that had an even greater impact, which is that by using my methods, convolutional neural networks, Geoffrey Hinton and his university team set a performance record on an image recognition benchmark dataset, classification, so much superior to previous performances that the computer vision community abandoned everything they were doing overnight and started using these new methods which they previously thought were very unpromising. They were extremely skeptical, not to say downright insulting. So there was a complete reversal of the situation in 2013 which attracted, of course, the attention of large companies like Google, Microsoft, IBM, Facebook at the time, and then Chinese companies too who all started activities in the field. So it was at the end of 2013, beginning of 2014 that I created FAIR, the AI research lab of Facebook, and I stayed for about a dozen years.

Yes, and that's why the Turing committee of the ACM gave us the Turing Award because we faced a fairly significant resistance from the research community, we really persisted at the cost of, how to say, the reputation of some of our students who followed us on this path but who had a bit of trouble publishing their articles or gaining some recognition. But suddenly in 2013-2014, they found themselves as being a very small number of people who knew how to make neural networks work. So they were immediately snapped up by the companies working in this field. So two of my former students, for example, Kaiming He and the person who leads the Gemini project at Google DeepMind, Raia Hadsell, and the research director of DeepMind, the vice presidents of research. So these people who were a bit the first students to come out when deep learning was starting, made excellent careers, but indeed it was hard for them during their PhD.

So no, we have the right to be stunned and it is certainly surprising in a certain way that LLMs, whose training method is a bit simplistic actually, work so well. It is indeed a bit surprising. Not as surprising perhaps as some imagine, but still surprising. That is to say, the basic idea that we could build intelligent systems through learning and that this learning is based on prediction is not a new idea, it's a very old idea, particularly in the field of theoretical neuroscience. A whole set of theories that we group under the name predictive coding in English. So this idea that young animals and humans learn to apprehend the world by training themselves to predict. In fact, the brain is a prediction machine. It's not a new concept, but I have been working on the idea of teaching machines to understand the physical world by predicting what will happen in a video for nearly 20 years, let's say it really started about 15 years ago and basically we made very little progress in the first 10 of these 15 years. It's only in the last 5 years that we started to make somewhat significant progress. And it turns out that learning by prediction, when you try to predict natural data like video, is really complicated and difficult to make work, and we are only starting in recent years to make progress there. On the other hand, there is a field where it works really very well and that is text.

That is to say, if we take, if we train a machine, we show it a text, well there are two methods. The first method, we take a text, we remove 20% of the words from the text and we train a large neural network to predict the missing words. Okay? So that's the technology from 2017-2018, about 2019. It works very well to allow machines to do translation, understand text, classify, produce summaries, etc. It works very well. But actually what works better is even simpler and it's a method to which OpenAI contributed, which is not to mask the text. That is to say, we take a text, we remove nothing from the text and we train a neural network to simply reproduce the input text on its output. But we configure this neural network in such a way that to predict a word, it cannot look at the corresponding word in the input. It can only look at the words that are to the left of that word. So implicitly, when we train this neural network to reproduce its input on its output, which looks easy, we implicitly train it to predict the next word regardless of the preceding word sequence. And that is magical. And not only is it magical but it can scale. That is the series of GPT-1, GPT-2, GPT-3, etc. GPT stands for Generative Pre-trained Transformer. And it's this very simple architecture, which we also call decoder-only, which it turns out scales, we can scale it, train much larger networks and integrate implicit knowledge from the text with this method that seems to work as well, actually works much better than the previous method which consisted of masking, removing certain words, which is an older method. So this revolution, which was not immediately recognized as a revolution, dates from 2019-2020 and then GPT-3 in 2022, there were simultaneous projects on exactly the same theme at Meta, at Google. OpenAI was not the only one developing this kind of thing but there was indeed significant progress thanks to this architecture. And then we realized we could fine-tune these architectures, these systems after training them on enormous amounts of text. We can fine-tune them so they follow instructions or answer frequent user questions and that makes them useful systems. And that's ChatGPT and everything that followed.

So well, there are lots of things to say about that. And it will lead us to a long discussion. I will try to break it into pieces. The first thing we must tell ourselves is that these systems, LLMs trained on text, are very limited in their capacity. So I have declared for several years that this type of technology is not a viable path towards intelligent systems with human-level intelligence. Why? For a very simple reason, which is that the real world, the physical world, is much more complicated than text. It seems paradoxical to us as humans because we think human language is the summit of human intelligence but actually not at all. Text is easy. Predicting the word that follows a particular text. You can never predict exactly the word that follows a text but you can predict a probability distribution over all possible words. That's how LLMs work. And we can represent this probability. It's just a list of numbers. Let's say if the dictionary is of size 100,000, it's a series of 100,000 numbers between 0 and 1 whose sum is 1. It's really simple, we know how to do it. And then to predict a text, we do what is called autoregressive prediction. It's not a new concept either. That is to say, we show a text to the system, it predicts the next word. It predicts a probability over all words. We choose a word from this probability distribution, a high-probability word. But it's a bit stochastic. And then we inject this word into the input. That is to say we shift all the words in the system's input and now it sees its own output as part of the input. So now we can ask it to produce the second word. We put that in the input, 3rd, 4th, etc. So that doesn't need a lot of money to do that finally. So where a lot of money is needed is if the neural network in question, we want to train it on enormous amounts of data and especially we want to be able to experiment with several architectures etc. And the problem is that if we want to train it with a lot of data, what it will do is accumulate all the declarative knowledge in that data and for that it needs a lot of memory. So what is expensive in these systems is that you need computers or clusters of computers that are very powerful in numerical calculation because basically what happens at the base in these neural networks is multiplications, additions, comparisons and nothing else. It's really very, very simple. But it's hundreds of billions per inference. And so if we want these systems to accumulate enough declarative knowledge by being trained on text, they need enormous memory. So tens of billions or hundreds of billions or even thousands of billions of parameters. So that takes a lot of memory. Each parameter during learning is stored on 4 bytes or 2 bytes if you want to be a bit more efficient. At inference time it's rather 1 byte but still that's a lot. So that means GPUs with a lot of memory that Nvidia will sell you for an enormous price and a lot of memory traffic. So that drives up memory prices. I'm sure you noticed and it costs a lot. So you need a lot of infrastructure that costs hundreds of billions in capital investment that the big tech companies of Silicon Valley can afford but no one else.

And so that effectively puts other countries, including the Chinese actually, in a complicated situation. The Chinese even more so because the American government is trying to limit Chinese access to the latest technologies. So for the limitations of these LLM-based systems, as I said, predicting the next word in a text is relatively easy. Predicting the pixels that follow in a video is much, much more complicated and we don't really know how to do it yet. Which means we have systems that can pass the bar exam, write dissertations, even prove theorems, write code. So do things that are really based on text. But where is my autonomous car? Where is my domestic robot that can clear the table and load the dishwasher? And let's not even talk about the domestic robot that can repair the bathroom. It's not for tomorrow. And that's the reason for that. We don't even have systems intelligent enough to have a physical world understanding as sophisticated as a 4-year-old's physical world understanding. So there is an enormous gap between language-based intelligence, if you will, or the regurgitation of accumulated declarative knowledge, and intelligence that would be the understanding of the physical world. And we can summarize this in a very simple number, which is that the entirety of all text available on the internet used to pre-train LLMs today is about 20 trillion words. So a text is tokenized, that is to say we cut words into sub-words etc., we transform that into tokens. And 20 trillion words is about 30 trillion tokens. Each token is typically encoded on 3 bytes. So if we do the calculation, it's about 10 to the power of 14 bytes of training which would correspond to a text that would take each of us here about half a million years to read. That's a lot. No human is capable of ingesting and certainly remembering that much information, even in an abstract way.

Compare that to the data that reaches the visual cortex in the first years of life. A 4-year-old child has been awake for a total of about 16,000 hours in their life. And during these 16,000 hours, the information carried by the optic nerve fibers to the visual cortex is about 1 byte per second per optic nerve fiber. We have 2 million optic nerve fibers, 1 million per eye. So it's a flow of about 2 million bytes per second. If we do the calculation over 16,000 hours awake, it's about 10 to the power of 14 bytes. That is to say a 4-year-old child has seen as much information as the largest LLM simply through vision. And as much information as there is in text, the entirety of publicly available text on the internet. So that tells you we will never reach a human-level system by training only on text. There is a whole part of reality that is not present in the text. It is not present in the text.

So then, I can give you the solutions. I will explain a bit the idea around which FAIR is built. And I will take an example that I often use in this kind of conference. If I take a video of this room, I start from the left, I turn the camera and I stop. And then I wait, I give that to a machine trained to predict what happens in a video and I ask this machine, continue the video. Very clearly, the machine will predict that the camera will continue to turn, that we are in a classroom which probably has windows on that side because it sees the projected shadows, etc. But there is absolutely no way it can guess what you all look like. It's not possible. The information is not present. It's impossible-to-predict information. And when we train these systems, generative systems that predict at the pixel level what will happen in a video, it doesn't work. We kill them. So obviously, there are all these video synthesis systems. We have all played with them, they are very impressive, but these systems actually don't really understand the underlying reality of the physical world. So how can we train a system? Simply by prediction, which works very well for text. How to transpose this success to a system that predicts, is trained to predict what happens in a video, knowing that it absolutely cannot predict all the details of what is in the videos. So that leads to a concept that I call JEPA. It's an acronym, it stands for Joint Embedding Predictive Architecture. In French, architecture prédictive à encastrement conjoint. And the idea of this is that the system, while training, must produce an abstract representation of reality in which all the details that are not predictable are eliminated, and then train to predict but in this abstract representation space. And so the whole question, which is actually the original question of deep learning, is how to learn representations of percepts? And we always ask ourselves the same question, it's a question that has obsessed me for 40 years, which is how can an intelligent system during learning learn a good representation of the world, whether for perception or for prediction in video as is the case. And so I arrived at this JEPA concept for which I published a sort of article on my vision of how AI research should go.

It's been more than six years since the article was published in 2022, in which I proposed this architecture. There are about 2,900 articles in the scientific community that mention JEPA. So it has generated quite a bit of interest, but it's still within the research community and doesn't yet interest the AI industry of Silicon Valley. They are all in a trench of improving LLM performance, and none of these companies can afford to deviate too much from the dominant approach because they would risk falling behind. So this benefits us a bit because we can pursue this approach without competition.

Regarding JEPA World Model, what is a world model? Since you mentioned a world model, it would be based on these JEPA stories. It's what would allow an intelligent system to act, to plan actions by predicting the consequences of these actions. The reason we can apprehend a new situation and act correctly is actually a world model we have in our heads that allows us to predict the consequences of our actions. I know that if I push on the base of this glass, it will slide. If I push on the top of the glass, it will probably tip over. If I push on the table with the same force, it won't move. We have an intuition of what happens in the world that we learn in the first months, the first years of life. Initially purely by observation, then by experimentation.

Human babies learn, for example, the notion of gravity, the fact that an object falls when it is not supported, around the age of 9 months. It takes time for humans; it's faster for most animals. So, how do we learn these basic concepts of physical reality? And can we learn them by training to predict? The answer is yes. Now we have the answer, which is that we can train systems to predict what will happen at an abstract level, and we can also train them to predict what will happen as a consequence of an action. So if we have a system that can predict how the world will change given the state of the world at a given instant and an action I imagine taking, can I predict the state of the world at time t+1 that will result from this action? If I have this mental model, I can predict the result of a sequence of actions. I can see if this sequence of actions will accomplish a task I would like to accomplish. And so I can, through search, find a sequence of actions that will accomplish this task. That's planning. We know how to do this as humans, very well. We do it hierarchically, it's more complicated, but we know how to do it very well. Many animals know how to do this too because they have a mental model of reality. Current AI systems don't have one. They have learned many declarative facts about the world, but they don't have an understanding of reality. So that's what we're trying to build at FAIR. What this will cause, for sovereignty reasons, is that it will cause a new revolution in AI. There will be another revolution in AI in the coming years due to the advent of these world models, systems that can plan, systems that have a certain common sense, and therefore systems that will open the door to applications in robotics, for example, or industrial applications. And that's the angle FAIR is pursuing. So, we may have lost the LLM battle in Europe, but we haven't lost the entire AI battle because there will be other revolutions, and we can be at the start of this other revolution.

The other reason we can have contributions to this new revolution is that models based on world models need less memory. That is, they replace the accumulation of declarative knowledge with an understanding of the world. So, for example, you don't need to explicitly remember the fact that when you push on the top of a glass, it tips over. It's an understanding. It's not an isolated fact you remember. You don't need to learn it by heart. Right? So the models we train at FAIR are of this type. In terms of size, they are smaller than LLMs. They don't need as much memory. So we don't need as much computing power to train them, but they have a deeper understanding of reality.

So that gives a path for sovereignty. The second path for sovereignty is open source or open weights. For me, this is a favorite subject for a whole host of reasons and in a whole host of domains, but I think the question is particularly important today because in the present and even more in the future, all the information that will reach us, which will belong to all the citizens of France and Navarre, will come through AI systems, intelligent assistants. If these assistants are proprietary and produced by three or four companies on the West Coast of the United States or in China, it's not good for culture, it's not good for democracy, it's not good for values, it's good for nothing. So we absolutely must avoid this situation where all citizens' information will be controlled by a handful of American or Chinese companies. So sovereignty is absolutely necessary. It's not an option. And for me, the only way to achieve this sovereignty is for countries, particularly those that are neither the United States nor China, but that doesn't mean we exclude them, to ally to produce an open, free, open-source model that would be a repository of all human knowledge and would be trained on all data, not only the data that is publicly available today and with which commercial models are also trained, but also data that is not publicly available, the cultural collections of the National Library in France, and then an enormous amount of knowledge from cultural collections in India, Egypt, Vietnam, Japan, Korea, etc. If all these regions and countries collaborated to train an open model, it would become the repository of all human knowledge and would understand all the languages of the world, all value systems, all interests, etc. It's a bit of a utopian vision, but I started a project called Project Tapestry, which tries to federate regions, countries, universities, computer scientists, and scientists to contribute to the creation of this. We hope to have, we already have the support of a number of governments around the world, particularly India, Japan, Vietnam, and a few others, and we hope soon to have the support of most European countries and other actors.

Okay. We don't have revenue yet, so there's no question of acquiring anything, but for now we're spending a lot of money on GPUs, well, we're calculating. So actually, there isn't really a problem of available data because if we had learning techniques that were as efficient as what we observe in humans and animals, we could train a system that understands reality with 16,000 hours of video. 16,000 hours of video is about 30 minutes of uploads on YouTube. Right? It's very little video. The models we currently train, because our training methods aren't as efficient as what we observe in humans, we train our models with the equivalent of centuries of video. Right? And it's very easy to obtain. It's not a problem of information sources. There are many more videos available than we can handle with the computers we have today. However, there is a type of data, let's say, that humans are exposed to which is a bit harder to obtain. It's data that is observation or state of the world, action, next state. For example, data that would come from a robot where we see the position, perhaps the speed of all the robot's joints, and then we send an action to the robot and observe the result. So that's what would allow building a model of the robot that would allow controlling it intelligently. And that's harder to obtain because you either have to simulate the robot, which is possible but it's not precise when it comes to object manipulation, for example, simulation doesn't work well. It works well, however, if you want the robot to run a footrace, do a high jump, or some kind of kung fu movement. That's why we see all these humanoid robots in China doing all these things. It's really simple, actually. The dynamic models of the robots that need to be learned for that are simple because the interaction of these robots with the real world is actually very, very simple. It's just feet on the ground and that's about it. But when it comes to, for example, manipulating an object, it's much, much more complicated. In fact, there's a big secret in the entire robotics industry today. There are many companies building humanoid robots or others. Many companies also making software to try to make robots intelligent. None of the companies today that build humanoid robots have any idea how to make these robots sufficiently intelligent to be useful. Okay? These robots are completely useless. So, by collecting enormous amounts of data, we can train them to accomplish a task by imitation, but you need to collect enormous amounts of data, perhaps tens of thousands of hours for a fairly narrow task in a factory or something like that. And so it's not really economically feasible. And the domestic robot that clears the table and cleans houses, that's not for tomorrow. So there's a big question, which is how many humanoid robotics companies will go bust before AI has made enough progress to make them useful? We hope at FAIR to play a positive role, naturally.