Thank you so much. It's a real pleasure to be here. I think my last time, my last talk at Dartmouth was eight years ago. I was trying to remember exactly when that was. We'll see if I come again eight years from now.
So I'll talk about the future of AI, largely also a bit about the past or the current state of affairs. And what I think is that true progress in AI will come because of new concepts. We're not done yet.
Despite what you might hear from some of the more optimistic CEOs of Silicon Valley, we're not anywhere close to human-level AI, which some people call AGI. I hate that phrase. I'll tell you why in a minute. And there is this question of what must remain human. I'm not going to answer that question, unless you ask me afterwards.
Okay. So first of all, AI sucks. We certainly made a lot of progress in the last 70 years towards making machines more intelligent and more useful generally. But you know, we have AI systems that can write code, they can pass the bar exam, they can win math olympiads, they can prove theorems that stayed unproven for decades.
But where is our domestic robot? Where are our level five self-driving cars? We don't have those. And so that indicates that while AI can deal with discrete symbols, language, code, mathematics, it still has a very hard time dealing with the real world. The real world is complicated, it's messy, and current AI approaches are basically kind of helpless with it.
So AI really can't deal with high-dimensional continuous data like images, video, audio, etc. It can, of course, a little bit, but it's not the same techniques that are used in AI assistants. Basically, what a lot of AI companies are doing at the moment is that they are kind of bolting all kinds of modules to allow AI systems to solve problems using tools, to perceive the world with vision using vision encoders and various other things, understand speech.
But those are extra modules that are trained separately and bolted onto LLMs, LLMs still being the central part, the centerpiece. We don't really have general learning techniques that allow an AI system to understand high-dimensional, continuous, noisy signals like video, audio, sensor data, etc.
We do have agentic systems, but they need to be trained by imitation, basically reproducing human behavior, and they require a lot of examples or massive simulation environments. And they don't have the ability to anticipate the consequences of their actions, which is really a very bad way to act. If you can't anticipate what's going to happen as a consequence of actions you're taking, you might be doing something really bad without realizing.
So there's a number of questions. Why can't AI systems solve new problems zero-shot without being trained to solve them with lots of examples? Why can't they have common sense? And the answer is the real world is messy. As I was saying, language is simple, surprisingly.
Never mind trying to reach human-level intelligence with AI. We can't even reach the level of understanding the physical world that a cat or a dog can reach with robots. So clearly we're missing something quite big. We don't have AI systems that can do what a 10-year-old can do the first time you ask them, like clearing up the dinner table, for example.
We have billions of hours of training data of cars being driven by people. We should be able to train a neural net to just emulate human behavior for driving. That has been a complete failure. This is called imitation learning. It's been essentially a failure. The most autonomous cars that we have are level four. They're not level five because they can only drive in certain areas, and they use all kinds of heavily engineered systems to be able to do that, and detailed maps and lidar and all kinds of stuff.
So we keep bumping into this paradox called the Moravec paradox, which was formulated by Hans Moravec, a roboticist, in 1988, that things that are easy for humans are difficult for AI and vice versa. And when you say humans, it's also animals.
So perhaps that begs the question of what really is intelligence, right? And we have this impression that the AI assistants that we all use every day are intelligent. They're not that intelligent, except perhaps for code and mathematics. But for the rest, they're not that intelligent. They're very good at retrieving known knowledge. They can accumulate enormous amounts of declarative knowledge and regurgitate it at the right time, but that's not intelligence.
So this quote, which is actually apocryphal — Piaget never actually said this explicitly, but that's what he meant, and a lot of people who wrote about his work kind of formulated it this way — he was a developmental psychologist that studied how children basically become intelligent. So he said, 'Intelligence is not what you know, it's what you do when you don't know.'
And I absolutely love that quote because that really says that you're facing a new situation. What are you going to do? You've never been trained to handle that before. You have to use the full power of your intelligence to figure out what to do. And regurgitating known knowledge is not intelligence.
So the AI systems we have at the moment are of the type that regurgitate knowledge, but they don't really invent new knowledge except in very narrow domains like code and mathematics. So we should measure intelligence not by a collection of skills or by whether a system can answer questions properly, but by how fast a system can learn a new task or a new skill, possibly zero-shot without requiring any prior training.
The other question that we need to figure out is how do humans and animals learn? So this chart indicates at what age in months babies learn basic concepts about the world, really very basic concepts. This was put together by my former colleague from Meta, Emmanuel Dupoux, who is a cognitive scientist.
And there are things that appear pretty early in life, like the ability to distinguish between animate and inanimate objects, that's around two months. Object permanence, around two months. The notion of natural categories: babies know the difference between a table and a chair even though they don't know the names.
But things like gravity, inertia, conservation of momentum — what we could call intuitive physics — that only pops up around nine months. And so it takes a while. It takes a lot of observation of the world. It also takes interaction, but a lot of it is learned mostly by observation, particularly in the first three or four months where babies basically cannot do anything in the world. You can gesticulate, but that's about it.
So perhaps the answer to what is intelligence is the ability to handle new situations that the system has never faced before. And I co-wrote a little piece with some of my younger colleagues that did all the work, really. Judah is a postdoc in my lab now. Ravid was a postdoc before. Adaptation really is kind of the key.
Okay, here's another element. LLMs are trained on human-produced text, of which there is a limited amount. All the publicly available text on the internet represents roughly 2 × 10^13 words, 20 trillion words. You turn those words into tokens, you get roughly 30 thousand trillion tokens. Each token is three bytes, and so you do the arithmetic and you get roughly 10^14 bytes. That's the size of the entire corpus that LLMs are pre-trained on.
Right? They're fine-tuned with way more data, but that's kind of the pre-training phase. Now compare this with — it would take any of us, by the way, about half a million years or so to read through this. Take a human child: a four-year-old has been awake a total of about 16,000 hours, which, by the way, is a very small amount of video — it's about 30 minutes of YouTube uploads.
We have two million optic nerve fibers carrying visual information to the brain, each carrying about one byte per second, roughly. So the data volume that a four-year-old has seen through vision, also through touch, is about 10^14 bytes. And that tells you that we're not going to get to human-level AI by just training on text. LLMs in and of themselves are never going to reach human-level intelligence.
Okay. Then here is another question: what type of inference are we using when we think? Are we producing all the things that we're thinking of by forward propagation through a feed-forward neural net?
The answer is obviously no, because we tend to think longer on problems that are more complex, right? So obviously our reasoning process is something that can take a variable amount of time depending on the complexity of the task, right? We think about things before we act most of the time, except for tasks that we're really familiar with.
Okay, this is not the way LLMs work. LLMs take a sequence of words, propagate it through a bunch of layers of a transformer, and then produce one token. It's purely forward propagation. So if you ask an LLM to answer a question by yes or no, the amount of computation that would be devoted to this answer does not depend on the complexity of the question.
So the trick that people play with LLMs is chain of thought. You get the LLM to produce lots and lots and lots of tokens, basically explaining its process of thinking, and in fact you can explicitly train it to go through particular steps of reasoning, and that's called chain of thought.
But it's really not a good principle by which a system could reason. In AI, if we go back to good old-fashioned AI or even Bayesian models and things like that, inference is not a process of forward propagation through a bunch of layers in a computational system. It's generally a search or an optimization.
Okay, so this is the diagram that you see on the right, where you perceive the world, you get some idea of the state of the world, and then you imagine an action. Let's say if it's an agent or a robot, you imagine an action, and then you check whether this action actually accomplishes the task you want to accomplish through an objective function.
Okay, think of this square or rectangular box, the red box, as a scalar-valued function that takes an input and a proposed output and tells you if this output is a good output for the input. It measures the degree of compatibility between a proposed output and an input. And so the inference process there is the system searches for a proposed action that is optimal, that will minimize this objective function, and then it outputs this action to the world.
So inference by optimization is computationally intrinsically more powerful than through forward propagation. Okay, I have a big caveat to say here, because in principle you could actually reduce any computation to a forward propagation to a two-layer network. There's a very famous theorem by someone who is in the audience — [laughter] — George Cybenko, from 1989, that showed that neural nets, two-layer neural nets, are universal approximators.
So in principle you don't need anything else, but there's a question of efficiency, right? The reason why we train neural nets with multiple layers is because they have practical size. If we only wanted to make them two layers, the size is impractical. So this is kind of a similar type of more careful argument here, which I'm not going to go into labor on, but essentially we have those two models, right? Inference by forward propagation through a neural net, and then inference by optimization.
And inference by optimization would require some way of evaluating what the consequences of a particular action or sequence of actions will be, and then feeding this to an objective that measures to what extent a task has been accomplished.
Now those two types of inference actually exist in humans and animals, and psychologists call them System 1 and System 2. System 1 is tasks that you accomplish without really thinking consciously about them because you're so used to doing them. And System 2 is when you use the full power of your mind to accomplish a task.
So basically we should be working on System 2. How do we get AI systems to do System 2? So I sort of — this is a collection of ideas I've been kind of thinking about and working on for several decades, but they sort of gelled around four years ago, 2022. So I wrote this paper that I posted online, 'A Path Towards Autonomous Machine Intelligence,' that sort of explains my vision for where AI should go over the next 10 years.
I don't know about 20 years, but the next 10 years, 20 years, because there was a series of conferences here at Dartmouth 20 years ago with some of the people who were here in 1956. And so, you know, I don't know what's going to happen over the next 20 years. Perhaps we'll have AI systems that reach human intelligence, but I wouldn't bet on it because it's almost certainly more complicated than we think.
But if we have a chance of reaching that, I think it would have to be based on world models and inference by optimization. So here's the link to the paper. There are other versions of this talk that you can watch, possibly.
Okay, so this sort of new type of architecture I call objective-driven AI, objective-driven architecture, really. Okay, perceive the world — the observation of the world is called X at time t, X_t, run through some perception encoder that you're going to have to train, it gives you an idea of the current state of the world, at least the part that you can observe.
Run this through a world model, which is a trainable function, some sort of neural net, that given the current state of the world and given a proposed action a_t, produces a prediction for the next state of the world, ŝ_t+1, okay, state prediction.
You feed that to an objective function. The objective function tells you the task you set out to accomplish has been accomplished or not. It gives you zero if the task has been accomplished, and some positive number, a function of how far from accomplishing the task you are. And the system works by basically searching for an action or a sequence of actions that minimizes this objective.
Now of course, if you have a world model that can produce the next state of the world, you can apply it multiple times so that you can predict the consequence of a sequence of actions instead of a single one. But conceptually it's the same thing.
I added another component here called the guardrail objectives. You would like to be able to design the system in such a way that nobody gets hurt when the system accomplishes a sequence of actions. If the system has the ability to anticipate the consequences of its actions, then perhaps you can put an objective function on top of that predicted state to tell you whether nobody got hurt during that trajectory, right?
You might have to train this, but at least you have a way of making those systems intrinsically safe by training guardrail objectives that, if they are optimized or satisfied by the optimization process that figures out the sequence of actions, the system will not hurt anyone.
So it could be very simple low-level guardrails, like you have this robot cook in your kitchen and you can have a very low-level rule that says, well, if there are people around and you have a big knife in your hand, don't flail your arms, right? Or you can have more higher-level concepts depending on the type of representation of the state of the world that you have.
So this process of optimizing a sequence of actions according to a model of the system you're controlling to minimize the cost is very classical in optimal control. It's called model predictive control, with a few more ingredients if you do it with receding horizon and everything, but it's not a new concept.
The new concept is we're going to train the world model with neural nets. And there is another new concept, which is hierarchical planning. So this is a problem that until last year, or perhaps two years ago, I would have said is completely unsolved. But at the end of this talk I'm going to show you some evidence that we are on the way to making progress there.
So what is hierarchical planning? Your world model is not a single model in your brain. You need multiple models for different situations, and you need models to be able to predict things that will happen in the next 200 milliseconds, or things that will happen in the next second, in the next minute, the next hour, maybe in the next year.
So if I take an object and I tell you I'm going to open my hand, you know this object is going to fall due to gravity, right? You don't need superpowers to do this. And it's relatively short-term. You can't exactly predict where the thing is going to fall or in which configuration, but you can make some level of prediction, right? You can predict if you send some muscle control to your arm how your arm is going to be 100 milliseconds from now.
If you want to plan a trip, let's say I'm sitting in my NYU office in New York and I want to be in Paris tomorrow, I cannot possibly plan my entire trip in terms of muscle control, which is the low-level actions that I can take with my body. I have to make this planning at a much higher level of abstraction.
I need to be able to, for example, realize that if I want to be in Paris tomorrow, I need to go to the airport and catch a plane. That's a two-step plan. It's very high level. I don't need to know much about the state of the world to be able to tell that I need to get to the airport and catch a plane.
Now I have a sub-goal: getting to the airport. I'm in New York, so I need to go on the street and hail a taxi for the airport. Now I have a sub-sub-goal: beyond the street, I need to go to the elevator, push the button, walk out the building. I have a sub-sub-goal: getting to the elevator. I need to stand up from my chair, pick up my bag, open my door, shut my door, walk, avoiding obstacles.
And again, I cannot plan this in terms of muscle control because there might be obstacles on my route that I haven't foreseen. So, you know, you have to sort of go down in this hierarchy down to a point where an action consists in, for example, standing up from a chair. You don't need to plan this, right? You can just apply it — it's a task that you're so used to doing, you don't have to plan.
But this question of solving the hierarchical planning problem is absolutely fundamental if you want to build really intelligent systems. What distinguishes human intelligence from a lot of animal intelligence is this hierarchy, or the number of levels in this hierarchy. So we're going to have to train hierarchical world models, and I'll tell you in a minute how this is done.
Okay. Now the next question is how are we going to train those world models if we need world models? So one option is to use a generative architecture. Generative AI is what we call AI now, right? All of AI is generative AI. And I'm going to tell you, if you want to train one model, this should not be generative. Okay, doesn't make me very popular, but...
Okay, so generative, right? LLMs are trained to predict the next word in a sequence of words, essentially. Why don't we use the same technique to train a system to understand the real world, train it to predict what's going to happen in video, for example? I've been working on this for the better part of the last 20 years, way before it became fashionable, because I was always convinced that training a system to predict what's going to happen in video will drive the system to understand the underlying nature of the world and perhaps learn things like common sense and intuitive physics.
I failed largely for the first 15 of those 20 years, and about five years ago I realized the reason I failed is because I was trying to use generative models. Okay, I realized this before LLMs became fashionable.
Why? The reason is most of the information in a video, or in a natural signal whatever it is, is completely unpredictable. And so when you train a system to predict all the details in that data, you kind of kill it.
Let me take an example. If I shoot a video of this room, I start from here, I rotate the camera slowly, I stop here and ask the system: predict the rest of the video. It's going to predict the camera is going to continue rotating, there's going to be people sitting in seats, probably going to be a wall at some point.
But there's absolutely no way you can predict what all of you look like, right? That information is just not present in the initial segment of the video. And that's most of the information in the video. That's completely unpredictable.
So you can train a generative model, you know, one of those diffusion models or whatever, to actually make something up. It's going to produce a kind of nice-looking meeting room with, you know, people looking like people, but they're not going to look like you. You know, they're going to be hallucinated people.
So if you want the system to really understand the underlying nature of the world, you don't want to force it to predict things that it simply cannot predict.
And again, you know, going back 10, 15 years, we had projects on trying to do video prediction, and we were getting those like horrible blurry results. You can mitigate the issue with latent variable models like diffusion models, which is what all the video generation systems use, but those systems don't really understand the underlying nature of the world.
So my solution is something called JEPA, that means Joint Embedding Predictive Architecture, and it's shown on the right. So on the left you have a generative architecture: given observation X, run through an encoder, an action that you imagine or intervention, run that through a predictor, and then the system predicts a value for Y. And to train the system you minimize some divergence measure between the observed Y and the predicted Y.
Y can be the future of a video where X is the current observation of the video, but it could be anything else. Y could be two different views of the same scene or whatever.
The JEPA is shown on the right. So here you run both X and Y through encoders, and you do that prediction, but you do it in representation space. Never train the system to attempt to reconstruct every detail of Y. You only train it to predict an abstract representation of Y, hoping that the encoder will eliminate all the parts of Y that are unpredictable. Okay, that's JEPA, nothing more.
So LLMs are of the first type, right? So there's sort of various ways you can — you know, training a generative model is very simple, it's basically like supervised learning. Training one of those joint embedding architectures is a lot harder because of something called the collapse problem.
What does that mean? Let's say we have in the scenario on the left, which is a simplified JEPA, it doesn't have a predictor, we have two views of the same scene, or a corrupted view of the original...
picture. We run those two images through encoder and we tell the system those two views are really the same object, like whatever representation you extract from the image they should be equal because it's really the same thing you're looking at. So if you just use that as the loss function, that system collapses because what it does is that it ignores the inputs and it produces constant representations that are always equal.
Okay, the one on the right is the same. You take an image or video, you corrupt it or you shift it in time if you want, and then you run both through encoders, try to predict the representation of Y from the representation of X. The system can choose to ignore the input, produce constant representations, and now your prediction error is zero all the time, and your system does not do anything useful for you.
So how do you prevent that collapse? So first of all, I want to say this idea that an intelligent system should extract abstract representation of the inputs that ignores a lot of details about the input, I think is absolutely fundamental, which is why I'm telling people to abandon generative models. And the reason is the following: I could in principle explain everything that takes place in this room in terms of, let's say, quantum field theory, right? Could get the wave function of this room or perhaps a big chunk of the universe around us, and then run some gigantic quantum simulation, and it could predict if one minute from now George is going to scratch his nose. All right.
That's a self-fulfilling prophecy. [laughter] Okay, but that would be kind of crazy, right? It would be a huge amount of computation, and also it would be completely impractical because we require some gigantic computing infrastructure that is out of reach, and it would probably not even work because a lot of processes involve sort of quantum entanglement with the rest of the universe. So there's a lot of things that are just not predictable in practice, right? So what do we do? We invent abstractions.
If I give you a box full of air and I tell you I'm going to heat up this box by 10 degrees Kelvin, you can tell because of the law of ideal gas that the pressure inside that box is going to increase proportionally to the temperature in Kelvin. Okay, PV equals NRT. For those of you who have taken a classical physics course recently, you can make that prediction even though the underlying process is molecules of air bumping into each other and onto the walls of the box. But there's no way you can simulate this process at that level of granularity, right? But still at a very abstract level of representation where you represent the state of the box with basically three numbers, you can make predictions.
So the idea that the representation of the state of a system needs to be much more compact than the full state and ignore all the stuff you cannot predict is fundamental to science. So this is why we invent all those abstractions, right? From quantum field theory we invent the concept of particles, and then atoms, and then molecules, and in the living world proteins, organelles, cells, organisms, individuals, societies, ecosystems. Each level in this hierarchy describes the world with fewer details than the level below, but allows us to make longer-term prediction on bigger systems. Right? And that's I think essential to understanding the physical world.
We have to invent abstractions that simplify the world, if you want. And the fact that the universe we live in lends itself to this sounds like a conspiracy to me, or a mystery. I think it's the same mystery that Einstein was talking about when he said the most incomprehensible thing about the world is that the world is comprehensible. I love that quote too.
So every level in this hierarchy ignores some details about the level below, but contains knowledge that is not really derivable from the level below. Chemists know things about chemistry that atomic physicists don't, right? And biochemists certainly know things that chemists don't. And biologists know things that biochemists don't, etc. So models of the world are not going to be world simulators or digital twins. They're going to be sort of abstract versions of it. And we should make them action-conditioned predictors in abstract representation space so that we can do planning with them.
So if we have this type of world model, we can use them for all kinds of stuff. Let's say we have a very complex system whose dynamics is so complex that we can't reduce it to a small number of equations classically, like say a whole turbojet engine. If we have a phenomenological model of the system that tells us given the state of that system at time t and an intervention I imagine taking, give me the state at time t plus one, then we can ultimately control that turbojet to maximize reliability or all kinds of criteria that we are interested in: power consumption, fuel consumption, CO2 emission, whatever.
We can apply this to things like chemical plants or power plants, or to a patient. Let's say we want to have a predictive model of the state of a patient. Okay, a patient is certainly a complex system that we can't completely simulate, so we need to have some phenomenological model of it, which is what physicians do. Or a robot even. It's easy enough to model a robot with a bunch of equations. It's very hard to model the interaction of a robot with the real world. Which is why you see those really impressive humanoid robots from a bunch of Chinese companies doing kung fu and running and everything. There's no interaction with the real world other than touching the ground. As soon as you want them to do something like manipulation or clearing up the dinner table, as I was saying earlier, they don't work.
So this idea of using model-based planning is really old. It goes back to the late 50s in the USSR and early 60s in the West. And the main issue we're going to have to face is how do we prevent those systems from collapsing when we train them? And one trick is you use an objective function that has two terms. One term is minimize the prediction error. So try to predict S Y as well as you can from S X and perhaps the action if you have an action, but at the same time try to make the representation extracted by the encoder as informative as possible about the input. So I represent this by this information content I of S X, and there's a minus sign in front because you want to minimize it. So maximize the information content.
Now here is the problem though. We don't have any good ways of measuring information content. We have bad estimates of information content that are all upper bounds, and that's terrible because we want to maximize information content, and maximizing an upper bound is not a well-principled way of maximizing a quantity. But that's what we have. So that's what we're going to do. So over the last five years or so we've come up with a bunch of different methods to estimate the information content in a differentiable way, so we can maximize it as the cost function of some neural net.
Oh man, I didn't update this slide. A search for 'joint predictive architecture' between double quotes from Google Scholar returns not 1700 papers, 3700. That slide is six months old. It's becoming a very popular topic.
So, how are we going to measure information? Let's say we run a batch of samples through our encoders. What we get on the output is sort of a matrix. For every sample that we run, we get a representation vector. And so we get a matrix, right? When we run a bunch of samples through it, we want that matrix to be maximally informative. That matrix cannot invent information. It's a deterministic function of the input. So how do we make it informative? First of all, we probably want the rows to be different from each other. If the system produces the same representation for every sample, not very informative.
We probably also want the columns to be all different from each other because if they're all the same, that means the system basically is cheating, is not carrying a lot of information. It's producing the same variable multiple times. And we've come up with a number of different criteria to kind of do this. So some of them are sample contrastive that try to repel the rows in that matrix. Those are called contrastive methods. And there's another set that try to repel the columns in that matrix. But as it turns out, the best methods are neither. They're a little different from that.
So there's one in particular that I like called SIGRAG, which I'm going to tell you a little bit about. So SIGRAG belongs to a family of other methods that are based on clustering. So SIGRAG essentially takes each of those samples and projects those representation vectors against many different dimensions, which are random actually, and they change every time. Okay, so take your representation vector, project it against many random directions. Now over the batch, each of those projections is going to be a univariate distribution.
And what SIGRAG is based on is trying to make that empirical marginal distribution as close to a Gaussian as possible. So you take those samples that you see here, those red dots. You project the red dots against several dimensions. You get a marginal distribution which is empirical, and you try to make this empirical distribution as close as possible to a Gaussian. And there are several ways to do this. We use a technique based on Fourier transform, but you can use the distance between cumulative distributions, and that works pretty well.
But there is this fabulous theorem that says if all the marginals of a joint distribution against all dimensions are all Gaussian with variance one, the joint distribution is Gaussian isotropic. So SIGRAG means Sketch Isotropic Gaussian Regularization because it drives the overall distribution on the output of the encoder to be an isotropic Gaussian with variance one, and it works really well. And there is world models that are built on top of this.
So this is LA model. I didn't pick the name. Okay, I have to apologize for this. There's a whole series of models I call something and I have nothing to do with it. Okay, I couldn't talk them out of it. So this is a world model trained using this SIGRAG technique, and it can be applied to simple toy problems of planning like the push-T problem or simple kind of robot control. I'm not going to bore you with details.
There's a theorem which is pretty cool that says if your data actually is a joint Gaussian on the input and then what you're observing, the variable you're observing, the x of t, the x is some nonlinearly warped version of the inputs. So it doesn't look Gaussian anymore at all. You can prove and you can show experimentally that when you train an encoder with SIGRAG, that encoder will recover the underlying joint Gaussian. So that's at least one case where it's doing the right thing.
Now of course if your input generating distribution is not at all a Gaussian and it does not even have the topology of a Gaussian, has like multiple components, it doesn't work anymore. So SIGRAG is not the ultimate answer, but it's pretty cool and it works really well. So there are other methods which I'm not going to go into because I'm running out of time that are based on distillation, but they do produce really good results and they scale up to large scales and dimensions. SIGRAG is still on the way there but not there yet. It's promising.
So there is a series of methods, one called JEPA for example, another one called DINO, which I only peripherally participated in, and they're based on basically having two encoders, but one of the encoders serves as a teacher for the other encoder which is a student, and they kind of share the weights but through this horrible thing called exponential moving average which seems to prevent collapse, and we don't know why, which is why it sucks, but it works. And we published this paper a few years ago, and since then a lot of people have picked up this method to apply it to all kinds of things like medical imaging for example.
So DXA is a particular type of X-ray that can be used to measure osteoporosis and stuff like that. But it turns out if you use JEPA to learn representations of it and then you train a head on top of it supervised to extract other types of information, it works really well. So there's all kinds of information in those X-rays that were completely unsuspected. I'm a co-author somewhere in this paper. I don't deserve it. All the credit should go to the people who have done this amazing work.
So DINO, I'm not going to explain exactly how it works, but it's also a self-supervised learning method using a different criterion for anti-collapse. Another paper with my name on it where I don't deserve, and which works really really well for all kinds of applications. If you want a generic feature extractor for images, you probably should use DINOv3 at the moment.
So can we use those JEPA to train world models and do some planning for robots? And the answer is yes. So this is work from a couple years ago, DINO-WM, where the encoder is a DINO encoder and then the predictor is trained separately. So it's not trained end to end. So we don't have the anti-collapse problem because DINO has been trained beforehand. And what you see here is the planning that is produced by this world model where the target configuration of the world is shown at the top row and the bottom row is the sequence of actions that the robot takes to make the world look like the configuration at the top, starting from the random configuration earlier.
And this can be applied to a number of different toy problems in planning that seem to work pretty well. So that was a bit of an eye-opening, and since then we've been working on sort of various versions of this JEPA idea. V-JEPA 2 and 2.1 are the latest version, still produced by the team that I was associated with at Meta. So this is from last year. And those systems can learn some level of common sense. I'm out of time, so I'm going to skip this a little bit.
But we have, so the latest version has produced world models that work pretty well for various planning tasks that are not particularly complicated and can be solved with more traditional methods, but it's an indication that those things are kind of robust. You can apply the same world model to multiple robot anatomies and it still works, and multiple settings, and it's robust, and you don't need to train it for any particular task because it can plan as long as you have a cost function for it.
And what we're discovering is that the internal representation that those JEPA train on video to basically predict missing parts in a video, the type of representation that those systems produce are really interesting. So for example, if you take an internal representation of one of those V-JEPA and you train it to predict a depth map, it gives you pretty good results. Does a pretty good job at predicting depth. It's never been trained to predict depth.
But because it's been trained with video and to produce missing parts in a video, it has discovered the underlying structure of the world, is that the world is three-dimensional and every pixel you're going to see is going to have a distance. And that's the only logical way to explain how your view of the world changes when you move your head. So you don't have to be born with the notion that the world is three-dimensional. You can learn it very easily. And those systems do, and they can learn the notion of objects and object boundaries and things like this. You don't need to train it to do segmentation. It's just self-supervised.
Okay, the last thing I'm going to mention is this hierarchical world model and hierarchical planning idea. So this is a very recent paper from this month where we trained a multi-level world model to make predictions in simple scenarios. And so the first level predicts a fraction of a second in the future. The level on top has another encoder that learns a more abstract representation and it makes predictions that are kind of four times slower if you want. And the level above another four times slower.
So what happens is that because you force the system at a high level to make long-term prediction, to make those predictions accurate, it has to eliminate a lot of details about the state of the system. So in this case, the videos that I showed to the system are a synthetic video where you have this little kind of ant-like four-legged animal walking around which is controlled by the agent. So it's action-conditioned. And at level one, when you train a decoder from the representation produced by JEPA to reconstruct the image, you get an accurate reconstruction of the ant animal including the positions of the legs.
But if you go to the second level, the position of the legs start getting blurry, and if you go to the third level, the legs basically disappear. It's because if you predict several seconds in the future, you cannot predict the position of the legs of the ant, but you can sort of predict where it's going to be.
So this system can do hierarchical planning. You give it a target. It plans a sequence of high-level actions to get to the target, and then you take the first predicted state as a subgoal for the level below. Then plan a sequence of actions at the second level to get to that subgoal. And then take the first predicted state and use this as a subgoal for the bottom layer. And now plan how you move your legs to get to that state.
And what you see on the right is a curve that shows that if you do this three-level planning, the success rate is much higher for the same amount of computation than a non-hierarchical one. So this is for a four-level system where you see clearly that the representation of the agent in the high levels is much more approximate than at the bottom levels, and the high levels learn sort of a good value function if you know what that is. But anyway, let me go to the final slide.
So if you believe what I've said during this talk, I've told you to abandon generative models, and as I said, that does not make me popular in Silicon Valley. Abandon probabilistic models. I didn't have time to talk about this, but there is sort of a framework in which to explain all of this called energy-based models, which is sort of a more general, weaker form of inference than probabilistic models. I didn't really argue for that, but I'm a bigger fan, you know, those SIGRAG methods, this information maximization. There is a special case called contrastive methods that are fairly popular at the moment, and I'm telling people like it's not the right answer for various reasons.
I've been saying for forever to minimize the use of reinforcement learning because it's so inefficient, and replace it basically by world models and planning. And so these are four of the most popular pillars of machine learning and AI today, and I'm telling people to move away from them, and it's a quixotic quest, but I'm kind of used to it. I've done this before in my career.
So what I did in the last 10 months is I created a company called AMI Labs. So that means Applied Advanced Machine Intelligence. AMI like friend in French. And basically it's AI for the real world, for the physical world, that is based on world models and planning and things like that. And we hope to provide basically new models that are much more widely applicable than LLMs. We're not working on language. We're working on real signals essentially, with lots of applications in the biomedical sciences, healthcare, manufacturing, physical science, all kinds of domains.
So that's it, and the question I want to answer over the next few years is could we use this model to build intelligent systems. Thank you very much.
