This video is sponsored by KiwiCo. More on them later. The startup Physical Intelligence builds some of the most impressive robot brains ever demonstrated. Here's their PI07 model. Peeling a zucchini, folding a pinwheel, and taking out the trash. PI07 is a vision language action or VLA model.
What's your expectation here? Do you think JEPA-based approaches will eventually overtake VLA approaches?
Oh, absolutely. Yeah, VLA are doomed. I mean, they basically don't work really well.
Last time we followed Yann LeCun's path to JEPA, an alternative architecture for building AI models. Like VLA models, JEPA approaches can also control robots. But JEPA's demonstrated capabilities are significantly behind. Here's JEPA taking 60 seconds to move a cup off a platform.
So what makes LeCun so confident here? Are these VLA approaches that look incredibly impressive right now actually doomed? VLA models are in many ways the pinnacle of the current mainstream generative language-driven approach to AI. VLA models are built on top of VLMs, vision language models, and VLMs are in turn built from vision encoders and large language models.
At each level of the VLA stack, there exists an alternative JEPA-based approach with various trade-offs and in some cases impressive advantages. In this video, we'll work our way up this alternative stack. We'll see how a video-based model called V-JEPA 2 compares to the language-supervised encoders that we find in many modern AI systems. From here, we'll tackle vision language models. These include AI assistants like ChatGPT and Claude. Interestingly, we can reframe how these models are trained using a JEPA approach and achieve some impressive results. Finally, we'll zoom out into a full robot control system. This is where Yann's philosophical differences are the most pronounced.
I do not understand how you can even think of building an agentic system without an agentic system having the ability of predicting the consequences of its actions.
Okay. And VLA doesn't do that.
Sure. Right. They do not have world models.
We'll explore exactly how JEPA learns a world model that can be used for robot planning and control and see what advantages this approach might have over VLA approaches.
Modern AI systems have become remarkably good at bringing together vision and language. Chatbots can give highly detailed descriptions of images. And we now can even go the other way, mapping text descriptions to incredibly realistic images and video. Much of this progress can be traced back to a 2021 OpenAI paper and model called CLIP. In part one of this JEPA series, we saw how contrastive learning could be used to train joint embedding architectures by training our encoders to output similar vectors for corrupted and non-corrupted versions of the same image and to output dissimilar vectors for different underlying images.
CLIP works in a similar way, but instead of using corrupted and non-corrupted views of the same image, CLIP instead uses image-caption pairs where images are passed into a vision encoder and captions are passed into a separate text encoder model. From here, the CLIP algorithm maximizes the similarity of the embedding vectors produced by matching image-caption pairs while minimizing the similarity of the embedding vectors produced by non-matching image-caption pairs. For more on CLIP, see the video we did on diffusion models with 3Blue1Brown or chapter nine of the Welch Labs Illustrated Guide to AI.
After training, the CLIP vision and text encoders can be repurposed into a wide range of AI systems. One common application is making large language models multimodal. When you give an AI assistant an image, the image is typically passed into an image encoder model that was most likely trained using a CLIP-like approach. The encoder extracts meaningful information from the image that can then be used by the LLM. This combination of a vision encoder and an LLM is often referred to as a vision language model or VLM.
Now let's consider a JEPA-based alternative to the popular CLIP algorithm. V-JEPA 2 was trained by a team at Meta in 2025 on 1 million hours of video and uses up to 1 billion parameters, making it one of the most ambitious JEPA models trained to date. As we saw last time, in the JEPA architecture, we pass our inputs X and our outputs Y into encoder models which each return embedding vectors or matrices. From here, a separate predictor model predicts the embedding of Y given the embedding of X. The V-JEPA 2 team used a self-supervised training approach where video clips are corrupted by removing patches. The corrupted and uncorrupted video clips are fed into encoder models and the predictor is trained to predict the embeddings of the missing patches.
And the big idea here is that by learning to fill in the missing pieces of videos, our JEPA model will learn how video and by proxy how the world shown in these videos works.
Just like the CLIP image encoder, our V-JEPA 2 model takes in images or videos and returns embedding vectors. Note that natively CLIP only supports images, but is often used to process videos one frame at a time. Now, what would happen if we swapped in the V-JEPA 2 encoder for a CLIP vision encoder in a vision language model?
Yann LeCun's new venture, World Labs, has a line on their landing page that really gets at the heart of LeCun's philosophy: Real intelligence does not start in language. It starts in the world.
While CLIP and V-JEPA both produce trained vision encoders that take in images and video and return embedding vectors, their training objectives are remarkably different. V-JEPA is blissfully unaware of language, exclusively trained to predict the missing parts of video, while CLIP is trained to produce embeddings that match the embeddings of the language descriptions that we give to our images through captions.
So V-JEPA is not aided by or constrained by the language that we've invented to describe the world. The model can learn how to represent concepts like cats however it wants as long as those learned representations help the model fill in the gaps in videos of cats. However, this flexibility raises an important question for applications like the vision language models we're exploring. Will V-JEPA 2 learn representations that our language model can actually use? Will a model trained exclusively on vision be able to interface with a model trained exclusively on language?
The V-JEPA 2 authors go on to show that not only does this work, but that swapping in the V-JEPA 2 encoder achieves state-of-the-art results on a set of video understanding benchmarks. As the authors say, we show that a video encoder pre-trained without language supervision can be aligned with a language model and achieve state-of-the-art performance contrary to conventional wisdom.
These video understanding benchmarks include a range of skills. Here's one example from the TempCompass benchmark where the model is shown a video of a person picking up a pineapple and given multiple choice options about what's happening. Interestingly, in a variant of this question, the video is played in reverse, changing the correct answer. For reference, in our testing, ChatGPT 5.5 gets this question wrong for both forwards and backwards videos, and only some versions of Claude and Gemini get the correct answer. So, V-JEPA 2 shows that remarkably a JEPA-based approach can produce competitive and for some benchmarks state-of-the-art results when used to train the vision portion of vision language models. Now, this is still very much a hybrid approach, applying JEPA to the vision portion of our model while our full VLM still uses standard generative next token prediction objectives on language.
But is it possible to apply the JEPA architecture to our full VLM? In the most widely used VLM architecture, our images or video are passed into our vision encoder and the resulting embedding vectors, sometimes with modifications, are passed into our LLM. Our prompt is tokenized and also passed into our LLM. From here, our LLM directly outputs text one token at a time. Now, let's see if we can map our VLM architecture to a JEPA architecture.
Following the JEPA approach, instead of directly generating output text, we pass our target output text into an encoder model and train a predictor model to predict the embedding of our output text. Aside from this new prediction target, the rest of our standard VLM architecture actually maps pretty cleanly to the JEPA architecture. Both architectures already pass their inputs into encoders. In our standard VLM architecture, our vision embeddings and prompt are passed into our large language model. In our JEPA architecture, our predictor model takes in our embedded images or video. And as we saw last time, we can also pass in additional information into our predictor model. This is known as conditioning. Here we can pass in our prompt directly into our predictor, giving our predictor model access to both vision and text inputs.
So architecturally, the language model in our VLM architecture and the predictor model in our JEPA architecture have very similar jobs and take the same inputs. The key difference here is that our JEPA predictor model's targets are the embeddings of our output text, not the output text itself.
So, how does this JEPA version of a vision language model stack up? Last time we saw that a key advantage of the JEPA architecture was not having to reconstruct full outputs. In theory, the encoder model will extract the salient features of our output while ignoring extraneous details. Yann gave a nice example.
If you train a generative model, you know, to predict what's going to happen in the dash cam video, it will spend most of its resources predicting the random motion of the leaves on the trees bordering the road, and those are things that are essentially not predictable, but they have a lot of pixels, you know, that move around.
A similar argument can be made for the language outputs in VLMs. If we ask a VLM if it's safe to eat a mushroom shown in a picture, there's a variety of ways the model could phrase a correct answer. But our training data likely only includes one phrasing. So if the correct answer according to our training data is 'do not eat this mushroom', but our model instead returns 'this mushroom is not safe to eat', the model will be penalized during training for what is essentially a correct answer. Alternatively, with a JEPA architecture, these phrases are mapped to very similar embedding vectors, abstracting away irrelevant semantic differences in our prediction targets.
In late 2025, a research team at Meta showed that this vision language JEPA architecture, which they called VL-JEPA, produced some impressive efficiency gains. In a controlled experiment where a VLM and VL-JEPA architectures are given the same exact vision encoder and trained using the same data and training configuration, the VL-JEPA architecture learns significantly more quickly, reaching a video classification accuracy of 35% after 5 million training examples compared to an accuracy of just 20% for the traditional VLM architecture. So by learning to predict the embedding of our target text Y instead of Y itself, VL-JEPA is able to learn significantly more efficiently, arguably by abstracting away the irrelevant semantic details of the target training text. This efficiency increase can lead to impressive results, including outperforming significantly larger models on visual question answering benchmarks.
The GQA compositional reasoning benchmark includes tricky visual reasoning questions like figuring out from this image if there is any fruit to the left of the tray the cup is on top of. Impressively on this benchmark, VL-JEPA was able to outperform 7 billion parameter models while using just 1.6 billion parameters.
Now there is an important wrinkle when using VL-JEPA since the model is not generative. It does not by default spit out answers to questions. The team worked around this limitation in a couple of ways. One approach is to pass a given image and question into the model to produce a predicted embedding vector and then pass in all possible answers for a given benchmark into the Y encoder and choose the answer that produces the most similar embedding vector to the predicted embedding vector. This is like giving VL-JEPA multiple choice options to the benchmark questions. Finally, the team also experimented with training text decoders to map VL-JEPA's predicted embeddings to text, allowing VL-JEPA to act like a generative model at inference time.
So the JEPA framework has some really interesting overlap with the vision language models behind AI chat assistants, providing a path to potentially stronger vision encoders like V-JEPA 2 and through architectures like VL-JEPA, an embedding space training objective that allows models to learn more efficiently.
But what about the vision language action models we saw at the beginning of the video? These models effectively turn LLMs into robot brains, taking pre-trained vision language models and training them to output robot control signals given instruction prompts and feeds from the robot's cameras and sensors. Early VLA models had the large language model directly output robot control signals, while more recent implementations, including the PI07 model we saw earlier, use a separate model called an action expert to interface with the language model and output final control signals. Check out the Welch Labs video on VLA to see exactly how these fascinating models work.
Interestingly, VLA models are where we find the strongest contrast with LeCun's JEPA philosophy.
What's your expectation here? Do you think JEPA-based approaches will eventually overtake VLA approaches?
Oh, absolutely. Yeah, VLA are doomed. I mean, they basically don't work really well.
So, what exactly does Yann see as the big issue with VLA and how does JEPA address it? How do JEPA and LLMs compare to human learning? LeCun has an interesting take here, showing with some back-of-the-envelope math that the average four-year-old has actually taken in more bytes of information through their visual cortex than even the largest LLM will see in all of its training text.
If you find yourself thinking about how the children in your life are learning, check out this video's sponsor, KiwiCo. KiwiCo makes hands-on project kits that make learning genuinely fun for kids of all ages. My son is dinosaur obsessed right now. So, this dinosaur dig crate was absolutely perfect. His language is really progressing and it's wild to hear him pronounce these complex dinosaur names.
And assembling these intricate puzzles is great for developing his spatial reasoning. I had to borrow the crate to take these overhead shots and he literally has not stopped asking for it back. My daughter gets a little anxious at the doctor sometimes, and this doctor kit is great for getting her used to all the parts of her checkups. She loves following along with this checklist. As usual, the thoughtfulness and attention to detail are what really set KiwiCo crates apart from many of the toys that we have, gently pulling my kids' playtime in the learning direction. The KiwiCo team really invests in and pays attention to learning outcomes. They recently teamed up with Johns Hopkins on a study of the impacts of using KiwiCo crates in the classroom and found that teachers consistently reported improved student motivation, engagement, and confidence when using KiwiCo crates.
KiwiCo crates make amazing gifts for the kids and families in your life. And they make awesome learning experiences for kids of all ages. Use my code WelchLabs to receive 50% off your first monthly crate for kids three and older and 20% off your first Panda crate for kids under three. Big thanks to KiwiCo for sponsoring this video. Now, back to JEPA.
LeCun's critique of VLA boils down to two main points: the difficulty of scaling behavioral cloning and lack of explicit planning. Let's hear Yann's take on behavioral cloning first.
Oh, absolutely. Yeah, VLA are doomed. I mean, they basically don't work really well. Okay. I mean, the only way to get them to work is to essentially collect tons and tons and tons of examples, you know, or something else, or if it's in the digital world, it's just, you know, people playing with a user interface and whatever. And then just do behavior cloning. And that's only practical for a very small number of applications and for applications where the degree of variability is not too high because those systems basically when they face a slightly new situation are completely helpless. So they're brittle, right?
Human demonstrations are a critical training data source for many VLA implementations including the Physical Intelligence PI models. Training datasets are often captured using sophisticated controllers where the robot mimics the positions of the operator's hands. And Yann's point here is that this approach is simply not scalable. It's impossible to collect human demonstration data for every single variation of every single task we want the robot to perform.
Now, it's important to point out here that VLA models have been shown to generalize to new tasks outside of their training demonstrations. In fact, the breakthrough moment for VLA models back in 2023, where Google's RT-2 VLA moved a Coke can to a picture of Taylor Swift was a breakthrough because the human demonstration data did not have anything to do with Taylor Swift. So, to complete the task, RT-2 had to connect the concept for Taylor Swift that its internal vision language model had learned during pre-training to the actions for moving objects it had learned later from human demonstrations.
Since this breakthrough in 2023, the VLA models have advanced rapidly. The Physical Intelligence team has demonstrated their robots performing a range of tasks not present in their human demonstration data, including taking Tupperware in and out of the microwave, replacing paper towel rolls, and loading and unloading air fryers. Now, of course, ability to generalize is on a sliding scale. While these exact tasks were not in the human demonstration data, similar tasks were. And if we ask a Physical Intelligence powered robot to do something too different from its demonstration data, it will likely fail.
The big question here, the question that Physical Intelligence and many others are working to address is whether or not VLA models will be able to generalize well enough beyond their demonstration data to make reliable and useful robots.
Yann's second big criticism of VLA models is lack of explicit planning. VLA models are trained and deployed end-to-end. At each time step, a new set of camera images and robot joint positions come in and the model is trained to directly output the next set of joint positions. The robot then moves to these new positions. New images are taken and the process is repeated. This is wild when you think about what VLA models can do. In this demonstration from Physical Intelligence, the robot has to do this intricate dance of handing the key back and forth between grippers to get it in just the right position to open the lock. The internal LLM is somehow reasoning about how the key needs to be held and is able to break this outcome down into this repeated shuffling maneuver between grippers to get it just right. The challenge here is that we have limited control of and visibility into this planning process. We're more or less left with a black box that takes in text instructions and camera images and spits out actions.
I do not understand how you can even think of building an agentic system without an agentic system having the ability of predicting the consequences of its actions.
Okay. And VLA doesn't do that.
Sure. Right. VLAs do not have world models. They cannot predict the consequences of their actions beforehand. They just take the action and then deal with it as, you know, as some famous French king said. So if you really want to build reliable agentic systems, they absolutely have to be able to predict the consequences of their actions so that they can plan a sequence of actions to do something. First of all, to fulfill the task that they are being asked to fulfill, but also perhaps to, you know, guarantee some safety guardrails.
And the inference process now becomes a search as opposed to just an autoregressive prediction.
So that's a world model, the whole idea of a world model.
Unlike VLA, LeCun's approach to world models using JEPA does not learn end-to-end and does not learn to imitate humans through behavioral cloning. Instead, the JEPA architecture is used to learn an action-conditioned world model that can then be used to explicitly plan actions.
This is a task called Push-T where a robot is tasked with moving this T-shaped object to a final position marked on the table. The task is a bit trickier than it looks because it's difficult to predict how the T will translate and rotate based on exactly how it's pushed by the robot's end-effector. The robot's actions are limited to effectively 2D joystick controls. We can move the end-effector up, down, left, or right. Let's see how LeCun's world model approach works on a simulated version of Push-T.
Here the brown T is the target position and the blue T is the object that we push around and our control inputs move the yellow end-effector. First we learn a world model using JEPA by taking images and actions recorded from Push-T. At each step we train our predictor to predict the embedding of the next image of the environment given the embedding of the current image and some action taken shown here using arrow keys. Here we're learning from trajectories recorded from humans performing the Push-T task.
This is a similar setup to the behavioral cloning we see with VLA. But the big difference is that the model is not learning to mimic human actions, but instead to predict what will happen next in the world given some action.
Now things get really interesting. Given some initial configuration, we can pass this image into our encoder and get an embedding vector for our starting position. From here we can pass in any action we want into our predictor model and the predictor will return its estimated next state of the world based on our action. Now this prediction is still an embedding vector. So it's hard for us to understand what exactly the model is really predicting here. But for simple environments like Push-T, it turns out that we can train a separate decoder model that will map these predicted embedding vectors back to images of the environment.
And remarkably, when we do this, the results make a ton of sense. If we pass in this starting position and a movement upward, the end-effector in our decoded images moves upward. Here's a movement to the left, to the right, and down. From here, we can chain actions together at each step, passing the predicted new state of the world back into our predictor and passing in our latest action. So our JEPA-trained world model is essentially a learned video game, a learned simulated version of the world that we can use to plan actions and observe their consequences.
Using our prediction loop and decoder, we can compare what happens in our learned world model to the real thing. Here's 18 steps of actions taken in our learned world model and in our real Push-T environment. These match remarkably well. We do see some inconsistencies and drift, but overall our JEPA model has learned the dynamics of our Push-T environment remarkably well. Here's four more comparisons between our learned world model and the real Push-T environment. The top frames show the world model generated rollout passing the output of a predictor back into its input after each step and the bottom frames show the real environment following the same actions. We generally see good agreement, but our learned world model does go off the rails sometimes. In practice, this instability limits how far we can reasonably look into the future when planning using these world models.
The Push-T model implementation we've been experimenting with is from a JEPA implementation called LeWorldModel. LeWorldModel is trained from scratch on Push-T. As we've seen, our model inputs are raw pixels and actions. And remarkably from this data alone, our world model learns the physics of the environment, including the fact that our blue T is rigid and movable, and the complex interaction between our end-effector and the T. Looking inside our JEPA model's learned world like this is fascinating. It's like a learned cartoon sketch of the dynamics of the Push-T world.
From here we can use our world model to explicitly plan a set of actions instead of learning to directly imitate human actions as we would with VLA approaches.
And then if you have this, you can predict the outcome of a sequence of actions and you can, by optimization, you can figure out an optimal sequence of actions to arrive at a particular outcome. Right? This is classical optimal control.
To plan a course of actions, the LeWorldModel team used a very general planning method called the Cross-Entropy Method or CEM. Given a starting image and a goal image, CEM starts with a completely random set of actions. Here's 500 randomly chosen trajectories for our end-effector. From here, we use our world model to select the most promising trajectories. This trajectory bounces around a bit and then bumps into our T. Using our world model, we can predict what would happen if we were to follow this path. Note that the LeWorldModel team groups steps of actions together into groups of five and passes these actions into the predictor all at once. So our first batch of actions moves our end-effector down and to the right. And our world model simulation matches this behavior. From here, we can continue our rollout five steps at a time with each batch passing our embedding space prediction from our previous batch into our predictor along with our latest five actions. After our randomly chosen 25 steps, our world model predicts that our end-effector will rotate our T, not really moving it any closer to its goal state.
To measure how much closer or farther a given trajectory takes us from our goal, we compute the embedding of our goal image and then measure the Euclidean distance between our final predicted embedding vector and the goal embedding vector. From here, we perform the same rollout process for each randomly chosen path and compute the same distance metric for each path. Let's color each path according to its distance in embedding space to our goal image. Here's our best performing path. It looks a bit random, but if we visualize our decoded world model predictions, we see that this path actually bumps into our T twice, pushing it towards our goal.
From here, our top performing 30 trajectories are grouped into an elite set, and the mean and standard deviation of this elite set are used to sample a new set of trajectories. This process is repeated again and again until we're left with a tight set of candidate trajectories and ultimately a final planned path.
And what's really remarkable here is that our planning happens completely in the model's learned embedding space. The score we give each possible path guides the entire planning process and is computed as the distance between the final predicted embedding vector for each path and the goal embedding vector. We can now follow our planned path and see that our end-effector nicely pushes our T towards its goal.
Our resulting system cleanly addresses LeCun's critiques of VLA. It does not learn by imitating humans. So the system does not need to see how a human would solve the task, but can instead find solutions on its own using its world model and an explicit planning process. However, while the architecture of LeWorldModel is elegant and free from these concerns, the performance these models have shown to date is dramatically behind VLA on the Push-T task. LeWorldModel can only reliably plan about five prediction loops in advance, limiting the model to relatively simple manipulations.
When I'm trying to imagine a JEPA-powered robot kind of doing a long-horizon task like cleaning a kitchen for 10 minutes for example, right? I'm in my head, right? It's hard for me to imagine even in embedding space the predictor being able to see 10 minutes into the head moving around a kitchen that seems like longer than I would expect, right? Does that, is that where hierarchical starts to matter? What are your thoughts on long-horizon tasks with JEPA?
Yeah, you have the answer in your question. The answer to this is hierarchical models. Okay, so what's a hierarchical model? It's one where at a low level you make detailed predictions.
But you don't predict long-term because the more detail you preserve about the prediction, the more your prediction is likely to diverge from reality very quickly.
And so you train low levels in the predictor to make short-term predictions with a lot of details, which sometimes you need because you know you need to know exactly what's going to happen when you grab an object, right? You need to grab it exactly the right way and things like this. So you need a lot of information. But then if you want to make longer-term predictions, then you can only do them with fewer details about what you predict, right? And so that, you know, your prediction does not diverge from reality.
