Thank you very much. It's a pleasure to be here. I think I'm coming to India quite regularly these days. Okay. I'm going to tell you about something that is becoming a buzzword in AI world models. And many different people have different meanings for world models. I have my own definition and I will try to explain what it is and why I think about it this way.
Okay? So first of all, we need AI systems. If you want them to understand the real world and use human intelligence, not just when it comes to language or coding or math, but for everything in the world, we need systems that really understand the world at the level of common sense, like children learn how the world works. We need systems that have persistent memory, systems that can plan complex action sequences to achieve an objective, so that they can reason, and systems that are controllable and safe. And those are features that we don't see with current approaches, namely LLMs. And really, the way you can convince yourself of this is that we have systems that can pass the bar exam, win math Olympiads, and do all kinds of amazing feats. But we still don't have a robot that can do what a cat can do. Okay, and so we're not talking about super high-level reasoning here. We're just talking about basic understanding of the physical world and then turning it into control capabilities. And we don't even have, you know, robots that can drive completely autonomously. Still, we are getting closer. But we are getting closer by cheating. We are getting closer. We are getting closer despite the fact that we have millions of hours of training data from people driving cars and we train systems by imitation learning and yet we don't even get close to human performance, while a 17-year-old can learn to drive. You know, 20 hours of practice, maybe a bit longer in India. At least it takes a little time. And so, you know, and a test that a 10-year-old can do for the first time when you ask him, we don't have robots that are capable of doing that. Of course, we can train robots on very narrow tasks by collecting tons and tons of data and training the robot. Imitating humans doesn't work for autonomous driving because, you know, there are safety issues, but it will work for you, you know, filling a dishwasher or something like that. But the point is that, you know, we can do this without collecting tons and tons of data. For the first time, we are facing this situation. You know, how this happens. Maybe one reason is the fact that if we train systems on text, there are only 10 to 14 bytes of publicly available text on the internet. While 10 to 14 bytes match the amount of information obtained in visual context in the first four years of life, and this also corresponds to about 3 minutes of YouTube uploads. So video is 10 to 14 bytes smaller in terms of information context. We will never get anywhere close to human intelligence by training only on text.
Okay. So how do we get there? So what I've been arguing for many years now, actually for the better part of 10 years, more precisely in the last three or four years, is the idea of a model. Or observation from the world gives you some idea of the current state of the world, at least in the environment you can perceive. You can combine it with the content of a memory that contains information about the rest of the states of the world that you don't currently perceive, and then you want a model given a hypothetical action sequence to predict the resulting state of the world. Okay. This is actually a good model. A model of the state of the world at time t. A sequence of actions that you imagine taking. Can you predict the state of the world at time t+1? Whether +1 means anything. Okay, if you have this kind of system, you can predict for an objective task that measures to what extent a task has been completed or not, and you can also feed it for a set of guardrail objectives that guarantees that whatever sequence the world goes through, it will not actually harm people or have harmful effects. This is how you would build a safe AI system. And the way such a system would work is that, you know, looking at the observations, looking at those objective tasks, the system searches by optimization. It searches for a sequence of actions, an optimal sequence of actions that minimizes those cost objectives and satisfies the guardrail constraints. This is the only way I think. First, that you can build a safe system. Yoshua Bengio was talking about safe systems earlier. I completely disagree with him. He says they should have no objectives. On the contrary, I think they should have objectives that we give them and the only thing they can do is to achieve those objectives and nothing else. I think this is the only way to build a controllable system. Besides, designing those guardrail objectives will not be particularly simple. Actually, we will have to train those systems. But I cannot imagine that we can build agent systems. Everyone is talking about agent systems. I don't believe we can build effective agent systems without the ability to predict the consequences of their actions, and LLMs don't do that. We need models. I don't see any other way. You may have some other proposals. But I don't see any alternative at this time.
And really, so the next question we can ask ourselves is how do we train those world models? So obviously, you know, humans and most animals have a world model in their head that allows us to predict the evolution of the world and the consequences of our actions, and we learn, you know, the basic functioning of the world in the first few months of life. So it takes about nine months to learn about gravity and basic concepts. We learn fast. But we learn mostly by observation, a little bit by interaction, mostly by observation. How do we reproduce this in a machine? So a natural idea is to use generative architectures. Okay, so basically, suppose you want to train a system so that it can predict what is going to happen in a video, show the beginning of the video and ask it to predict the rest of the video. This is exactly the same type of self-supervised learning that we use to train LLMs, and I've been trying to make this work for the past 15 years, more actively in the past 10 years, and basically failed for the first 10 years and became more successful in the last few years. 5 years, and what made it more successful in the last 5 years is that the idea of making two models was completely abandoned. Okay? And so it's impossible to train a system to do this because most of the world is completely unpredictable. If you want to train a system to predict at the pixel level, it cannot succeed. It's completely impossible. If I take a video of this room, I start from that side and I pan the camera and I stop here and I ask the system. Can you predict the rest of the video? Is it going to predict that the camera will keep panning and this is a meeting room and people are sitting. There is no way to predict how you look. How each of you looks. It's impossible. The information is just not in the previous part of the video. And so when you train a system to do something that is impossible, it doesn't work very well. Okay? So if you are interested in AI systems that understand the real world, generative approaches should be abandoned. This works for language because language is discrete. It doesn't work for continuous data. What you get is blurry predictions, and we have found some ways to fix it in simple cases, and this is where, you know, video generation systems use generative architectures with latent variables. But really, the real solution lies elsewhere. It's basically to abandon generative architectures and instead use joint embedding predictive architectures, or more precisely, joint embedding predictive architectures, which are shown on the right. And so the left is the joint embedding architecture, and the right is the joint embedding predictive architecture. The difference between joint embedding and joint embedding predictive is that instead of predicting every detail of the variable to be predicted, which is the case, you run Y through an encoder and you only predict the representation of Y, because the representation of Y contains all the information you just cannot predict. Okay? Predicting at an abstract representation level, and this is a crucial insight. This is the JEPA architecture, which some of you may have heard of. Joint Embedding Predictive Architecture. We have a series of papers from four or five years ago. But there are also people who have been working on joint embedding architectures for almost 30 years. And so now we train a system to find an abstract representation of the input signal that allows it to make a prediction, and all those things that you just cannot predict will be removed from those representations.
And you can use this as the basis for a model, which is a version of it that is action-conditioned. Okay? So if you have a segment of video frames showing, for example, a robot doing something, and an action that you know was sent to the robot, then you observe the result. The state of the robot, and you can train your model, which is a model of the state of the world. Once you have such a model, you can use it to plan, which is the architecture I showed earlier, and basically plan a sequence of actions. I explained this in a paper I put online four years ago, and there you know, various longer versions of this discussion, more technical ones, using the ones below, and you know several recent papers on progress along those lines, and what we are building, particularly in the new company I am founding, which is called Advanced Machine Intelligence. And it's a French company. It's consistent. It's based on the idea of building a comprehensive intelligence system based on this idea of world models, planning, objective tasks, etc.
So there are various ways to train those JEPA architectures, and the main issues are to prevent them from collapsing. There is a method based on distillation that works really well. We don't fully understand the theory behind it. Actually, one of the papers trying to analyze it is actually by your co-author Surya Ganguli, who was on the previous panel. But there are some problems with it. It works really well, and some of my former colleagues at Meta built this system DINOv2, which is perhaps the best kind of general-purpose image representation system that you can use for all kinds of applications, and it purely outperforms. The configuration and what you see at the bottom is the sequence of actions planned by this DINO world model system. You know, the configuration of the environment as close as possible to the goal at the top. It has been applied to many different tasks on simulated robots, and we are making rapid progress along those lines, and somehow I cannot switch to the next slide until the video plays. Okay. Let's go. Hopefully, there we go. Okay.
But recently, we have trained quite large-scale video systems based on this distillation architecture, and now looking at recent papers, this is a JEPA system. So this is a JEPA architecture. It is trained to fill in the blanks in video at the representation level. And this is the first one that we have been able to show that it has learned some level of general understanding of the world. So it is able to tell you if something impossible happens in the video. So if there is a video where, you know, a ball is thrown in the air and suddenly the ball disappears or stops in the middle of the trajectory. Thank you. The prediction error inside that model increases because the system tells you. Okay, something impossible just happened. And we have trained the system to plan trajectories for real robots, and it can currently do relatively simple tasks, and then we are making rapid progress there, and hopefully get systems that can do really complex tasks. So I gave you examples in video that the whole approach is really designed to work for more than just video and apply to any sensor signal, any high-dimensional continuous noisy sensor signal, which is an area where LLMs are completely helpless, and we record and break everything.
Include so that we can have questions. So I am basically advocating abandoning generative models in favor of those joint embedding predictive architectures. You can imagine that this statement doesn't make me popular in Silicon Valley. I am arguing for a specific framework for understanding this energy-based model, which I didn't have time to talk about today. Various ways to train those JEPA using the contrastive method, even though I contributed to inventing them. I am actually very skeptical about them now. So the information maximization method is really what I am betting on, and certainly I have been saying for 10 years that reinforcement learning is very inefficient. So this is when you are interested in AI or something like that for human level, and hopefully the company I am building, MA Labs, which is really the objective of building hierarchical JEPA for the real world, universal action-conditioned causal models and using them for reasoning and planning. Thank you very much.
