Yann LeCun: Why LLMs Will Never Reach Human-Level AI — and What Will | JEPA & World Models Explained

Okay, so there's two answers to this. The what's missing is world models, and the second question is what is a world model and how do you build it? So we've seen two revolutions in AI over the last 15 years or so. One is the deep learning revolution, and the second one is of course LLMs, brought about by essentially the GPT architecture, even though there's other architectures that work really, really well. So what is an LLM and a GPT architecture has nothing to do with transformers really. It's this idea that you take an input sequence and you train a system to just reproduce the input sequence on its output. But because its architecture is strictly causal, in other words, it can only look to symbols that are to the left of a particular symbol you ask it to predict, it's implicitly trained to do next token prediction or next symbol prediction. Okay. So this is self-supervised learning in the sense that you're not training the system for any particular task other than capturing the mutual dependencies between symbols in the sequence. And initially the architectures that were used for this were trying to make those predictions in any direction, but it turns out the thing that scales is if you only train it to make temporal prediction in the future, right? So that's a GPT architecture. Now this works for text because text is discrete and there's only a finite number of possible tokens. So you can never predict which token follows a sequence, which word follows a sequence of words. But because there's a finite number of them, you can produce a score for every possibility. And that's basically a probability distribution over all possible tokens in your dictionary of which there is typically 100,000 or something like that. Right? So your system produces a distribution over 100,000 possible outcomes. Now, I've been working for at least 15 years on using the self-supervised training idea for video. So try to train a system to predict what happens next in a video, and you can't do it. You can't do it because if I take a video of this room and I rotate the camera and I stop here and I ask the system to predict what's next in the video, it can predict some things. It will probably predict this is a room and there's an end to that room and there are tables with people sitting. There's absolutely no way in hell it can predict what every one of you looks like or which seats are occupied or what the texture of the carpet is or the reflection on the chandelier. So most of what we observe in the world is completely unpredictable, and when you train a system to, you force it to make those predictions, you can only do it in one way, which is you have to feed it with extra information that allows you to make that prediction. That's called a latent variable, and basically it doesn't work because the latent variable actually contains most of the information about the prediction. Most of what we observe in the world is not predictable. Okay. We can predict the pressure. If we were to close all the doors and windows in this room and we started heating up the air in this room, we could predict the resulting increase in temperature. Or vice versa. If we increase the temperature, we could predict the increase in pressure because PV equals NRT. Okay, but the details of this are impossible to predict. The details are air molecules bumping into each other and creating that pressure and temperature. And there's absolutely no way we can simulate every molecule in this room. There's just too many of them. So this idea that you're going to be able to predict all the details of a piece of data whether it's video or detail dynamics of molecules or even simpler things than that is wrong. You just cannot do it. So how do we do it? You have to use a different architecture that does not attempt to predict every detail of the input, and that's JEPA. That means Joint Embedding Predictive Architecture. And what it means is that you train the system to learn an abstract representation of the observation and make prediction in that abstract representation space. And you train that system all together simultaneously. So the system trains itself to try to find a good representation of the input that represents as much information about the input as possible but yet it's predictable. Okay. So we can predict temperature or pressure as a function of the other in this room. We can't predict individual position of molecules and velocities. And there is tons and tons of examples like this. And physicists have this down to an art. The information they ignore to be able to make predictions is called entropy. Okay. To simulate the flow of air around an airfoil, even though it's air molecules bumping into each other, you don't simulate that. You abstract away that positional molecule and within every cube around the airplane you represent the air by velocity, density and temperature and then you solve Navier-Stokes equations. Now in fact we do this all the time when we do modeling. I could describe everything that takes place in this room in terms of molecules bumping into each other or even in terms of quantum field theory, right? That would be completely impractical because what goes on in this room at this time has more to do with psychology and high-level science rather than quantum field theory or atomic theory, right? So we invent abstractions. We can't simulate quantum fields above a certain size, so we invented particles. We can't simulate particles above a certain number. So we invented atoms and then molecules and in the living world proteins and organelles and cells and organisms and ecosystems and societies, right? And every level in this hierarchy is something that we invented to make predictions ignoring all the details in the level below in the hierarchy. And every level in this hierarchy is a different field of science. A field of science is defined by the abstraction level that we use to make predictions at. Okay. So JEPA is a very simple idea. Do not attempt to reconstruct everything in a signal. Attempt to find an abstract representation that allows you to make prediction. If I dump on you all the data we've ever collected about Jupiter, there's not much you can do with it unless at some point I'm going to ask you where is Jupiter going to be 10 years from now. And turns out to make that prediction you only need to know six numbers, three positions, three velocities and that's it. All the other details that you know about Jupiter are not going to be useful. Doesn't mean you don't want to model them. It just means that they might be very complicated like if you want to model the weather on Jupiter, right? It's like incredibly complicated. It's chaotic and everything. So that's the way the real world is. So, if you are able to train systems like this that are non-generative, they're also non-probabilistic. They can handle uncertainty, but in a funny way. Then you can build AI systems for the real world. Now if you make those systems do predictions but those predictions are action-conditioned so that you have state of the world at time t, action that you imagine taking, state of the world at time t plus one, then you can use that, that's called a world model, and you can use this to plan. You can plan a sequence of actions, an optimal sequence of actions to accomplish a particular task measured by some objective function u in the predicted state of your system. You can build agentic systems that actually work. Everybody is talking about agentic systems. Nobody has any idea how to build them in such a way that they are reliable and frankly I have no idea how to build them with LLMs because LLMs have no ability to predict the consequences of their actions. You need a world model for that. So maybe I'm getting ahead of myself, but the company I'm building is based around this idea of JEPA world models and AI for the real world.

That's an Einstein quote.

I mean I think it's really a fundamental limit of generative approaches attempting to reconstruct every detail of the signal, which I think is just a mistake. And then there is a second limit which is the fact that to make those generative models work you have to discretize the signal. You have to tokenize it, right? You have to turn it into a sequence of discrete symbols. And you can of course attempt to do this for video. It just doesn't work very well. And in fact, we have tons and tons of empirical evidence going back, you know, six, seven years showing that. So this is in the context of computer vision representing images. So you have two ways to train a system in a self-supervised manner, right? Without labeled data to represent images. The first one is you take an image, you corrupt it in some way, mask part of it or blur it or change the color or something, and then you train some big neural net to reconstruct the original image from the corrupted version. Okay, it's a variant of denoising autoencoder. In the context of NLP of text, this is called BERT, right? So the BERT architecture is like that, you take a text, you remove some of the words and then you train some big neural net to predict the words that are missing. And the B stands for bidirectional, it's one of the things I mentioned initially. In the context of images, there's several, vision masked image modeling or masked autoencoder. And those systems kind of work. I mean they produce decent representations of images that you can use in a subsequent supervised task, right? So you train an autoencoder which has an encoder and a decoder and then you use the internal representation. You freeze the encoder and you use the internal representation as input to a supervised network with a few layers that you train for a supervised task and you test the performance. So that gives you an idea of the quality of the representation that was trained self-supervised. Okay. So that's the first approach by reconstruction. Okay. And then there's a second set of approaches by joint embedding. So you take the original image, you corrupt it, you run both of them through encoders and you train the representations to be predictable from each other so that you can predict the representation of full image from the representation of the corrupted or transformed image. So there's a whole bunch of joint embedding methods of this type and the verdict after years of experiments with this is that joint embedding just beats generative methods by reconstruction every time. There's other methods to train autoencoders by reconstruction. There is VAE, there's VQ-VAE, there's sparse autoencoders. There's a whole bunch of methods. None of them work nearly as well as the method that do not attempt to reconstruct. So for me that was a very strong empirical signal that reconstructing natural signals is just a bad idea. And you know I've been living with this idea for 40 years like one of the first things that we tried when we had backpropagation in our hands. You can find this in my 1987 PhD thesis is to train an autoencoder. In fact to train a denoising autoencoder. I trained denoising autoencoders in 1986. And you know it kind of worked but not really. And Jeff Hinton actually also was disappointed by those performances and it turns out the best way to learn representations is to not reconstruct at least for natural signals. So a lot of empirical evidence and then some recent work on using those joint embedding methods with video with extremely good results.

Right. So in 1993, so it goes back a long time, when I was at Bell Labs, a group of people came, they work for a subsidiary of AT&T called NCR, it's no longer a subsidiary of AT&T, they make the ATM machines, right? Cash register, ATM machines, that stuff, right? And also a big check reading for banks. They came to us and they say, well we'd like to be able to verify the signature that people sign when they buy something with a credit card. And we could store the signature on the magnetic strip of the credit card. But we only have 80 bytes available. So can you encode the gist of a signature in 80 bytes? So we can compare two signatures, signatures of signatures in 80 bytes and tell whether it's genuine or not. And I said well I got this idea which we can train what we used to call Siamese neural nets. You give a signature from a person, you run it through a neural net and then you give another signature from the same person, run through the same neural net and then have a cost function that says I want the two representations at the output to be the same. And we of course constructed the representation to have 80 variables that we could code on one byte. So it would fit on 80 bytes. Now the problem with this is that if you just do that the system just completely ignores the input and produces a constant representation. So what we figured at the time is that we had to use what is now called contrastive methods, which is you also show pairs of signatures that you know are different. They're either a genuine signature from a person and then a forgery or just a different signature from a different person. And then you take those two representations coming out of your encoder and you push them away from each other. Okay, so that kind of works. It certainly works for signatures and people in more recent years have managed to make this work for like ImageNet style object recognition but it doesn't scale very well. The representation that you learn from this is relatively kind of low-dimensional if you want. And so I didn't believe this would kind of be the ticket to self-supervised learning until about 5 years ago when a postdoc of mine, Stéphane, and some other work that were related used other techniques to prevent this collapse that were not contrastive methods. They basically consisted in maximizing some measure of information coming out of the encoder. So basically you force the system to basically produce informative outputs and it will prevent it from just ignoring the input while at the same time making sure that the representation for two kind of similar objects end up being nearby. And that turned out to work well and then a variation of it called VICReg. So this was called Barlow Twins and then a variation of it called VICReg that worked extremely well. And then there is a slightly different method based on distillation that we applied to images that worked very very well and in fact beat masked autoencoders and stuff like that. Simultaneously some other group at Meta was working on a method that they call DINO, DINO v1, v2, v3, which is probably the best image encoder that we have at the moment, completely self-supervised. So it was pretty clear that you could prevent collapse in two ways, either contrastive methods or those kind of non-contrastive information maximization methods or with those distillation methods that we don't fully understand but they kind of work really well. And so it clicked in my head that that's the path of the future. That's what we have to work on, but it's complicated, like the theoretical underpinning of this collapse prevention is not completely clear.

Yeah, that's our latest work on forcing the output representation over a batch of samples to be basically an isotropic Gaussian and turns out there are relatively simple ways to do this. This is work with Randall Balestriero who is a former postdoc of mine who's now assistant professor at Brown. So there's been a phase change in the last five years in terms of your view of, okay, we can take this forward. Hence, FAIR Labs right now.

Okay. So, I mean clearly FAIR at least until last year was a great place to do research. But what's happened over the last year is that there's been a refocus of Meta on shorter term goals to basically catch up with the rest of the LLM industry. And as a result, that has deemphasized projects that are a little more ambitious, a little longer term, that sort of upset the current dominant paradigm. So when you have, you know, a whole company that is betting, you know, hundreds of billions on a particular paradigm and have this, you know, crazy chief AI scientist who claims everywhere, we're never going to get to human level AI by just getting LLMs, you make a few non-friends. So Mark Zuckerberg and Andrew Bosworth, the CTO of the company, were both extremely supportive of the project I was working on which internally at Meta we called AMI, Applied Machine Intelligence, Advanced Machine Intelligence. And it's a name that we use for the new company. But, you know, a while back I went to Mark and I told him, you know, politically it's very difficult to kind of push this in the company even though you like this project because LLMs are basically sucking the air out of the room. So that's the first thing. And there is some pressure on my colleagues at FAIR to work on, you know, to help the LLM effort. And second, most of the applications of those JEPA ideas are for AI for the real world like controlling industrial processes, robotics, you know, this kind of stuff and all kinds of applications that basically Meta has no interest in. Meta is entirely focused on connecting people with each other. So the spectrum of application was so large outside Meta that it made sense to leave. Also I realized I could essentially snap my finger and raise half a billion, but I mean turns out we can raise more. And so that's what kind of prompted me to leave, you know, a giant MOAT at FAIR. I had kind of shaped it initially but you know I haven't been director of FAIR for eight years. I kind of came back to being a researcher and so I wasn't managing it and refocus on LLMs and then lots of applications, available money, and then the results we were going to get with JEPA, you know, was starting to work and so it was clear that it was time to kind of push the button and kind of make this real, you know, shift into high gear.