CEOInterviews.AI
Start App
Get an email when Yann LeCun gives a new interview
One email per new interview. Unsubscribe in one click.
Yann LeCun
Co-Founder & Executive Chairman, AMI Labs

Yann LeCun: Special Lecture on AI and World Models

📅 Apr 15, 2026 Al-Khwarizmi Applied Mathematics Webinar 105 MIN 19 VIEWS 73 SEGMENTS · 6 SPEAKERS
Yann LeCun, Executive Chairman, Advanced Machine Intelligence, New York University.

Questions asked in this interview

10
  1. 1:04:06Do you think that there are enough mathematicians dedicating their research to AI, or no, do you think that we need more mathematicians?
  2. 1:11:19Do you see them as replacing digital twins or rather, how to say it, as a higher abstraction layer on top of them for planning, for anomaly or accident anticipation, etc.?
  3. 1:22:18Should humanity be afraid of AI in the future?
  4. 1:26:54So is it closer to a stationary statistic or Markov state or simply an element of the representation space, like in base space representation or something like this?
  5. 1:32:36So if I understand, world models can capture causality and correlation, not only correlation?
  6. 1:39:23And are there internships open, internships in your company this summer or summer internships?
  7. 1:40:14in California that using your JEPA... can you hear me, sorry?
  8. 1:40:32Can you hear me properly now, sorry?
  9. 1:40:35So I was wondering is there anything, later development in biology in your case?
  10. 1:42:36How do you think the role of this geometry in the future?
Moderator 0:00 ↗
Okay. Hello everybody. I would like to convey the apologies of Eric Molin. He was scheduled to be with us today to introduce our special speaker but unfortunately he had a major unforeseen issue. His flight which was initially planned for yesterday was postponed to today and he's probably in the flight right now. It's a great pleasure to welcome Yann LeCun, one of the world's leading pioneers in artificial intelligence and modern machine learning. Yann LeCun was the director of AI research of Meta and silver professor of data science, computer science, neural science and electrical engineering at New York University. He's affiliated with several prestigious New York University centers and institutes including the Center for Data Science, the Courant Institute of Mathematical Sciences, the Center for Neuroscience and the Department of Electrical and Computer Engineering. He received his engineering degree from ESIEE Paris in '83 and earned a PhD in computer science from Université Pierre et Marie Curie in Paris in '87. After a post-doctoral appointment at the University of Toronto, he joined AT&T Bell Laboratories in '88 where he later became head of image processing research department. He joined New York University as a professor in 2003 and later became the founding director of New York University Center for Data Science. In 2013, he was appointed director of AI research at Facebook. Professor LeCun's research has had profound impact on the development of artificial intelligence particularly in deep learning, computer vision, robotics, computational neuroscience. He's widely recognized as one of the creators of convolutional neural networks, a foundational technology behind many of today's AI systems used by major companies such as Meta, Google, Microsoft, IBM and others. These innovations have also led to real world applications in handwriting recognition, banking technologies, image compression, speech recognition and human computer interactions. He has authored more than 180 scientific papers and book chapters and has served in leading editorial and scientific roles across the AI community. Among his many distinctions, he received the 2014 IEEE Neural Network Pioneer Award, recognizing his groundbreaking contributions to neural networks and machine learning. Please join me in warmly welcoming Professor Yann LeCun.
Yann LeCun 3:12 ↗
Thank you very much for the introduction. I should tell you that I left Meta in early January and I started a new company called Advanced Machine Intelligence or AMI Labs. So I'm no longer at Meta. And a lot of what we're doing at AMI Labs I'm going to tell you a bit about today. So, I'm just also realizing that the last time I was in Tunisia was almost exactly 40 years ago. It's been too long, so I should come at some point visit you again. So in 1986, I was still a PhD student. Okay. So I'm going to talk about a concept that is gaining a lot of interest in the AI research community at the moment: world models. And I've been working on this for quite a long time but sort of accelerating the work over the last five years or so and as I told you just recently actually kind of starting a company around this idea which I think will bring about the next revolution in AI. AI is still fairly limited in its capabilities today and we need to enable AI systems with similar capabilities that we observe not just in humans but also in animals. And in fact that's kind of the first statement here is that AI really sucks compared to animals and humans. We have AI systems of course that can write code really well. They can pass the bar exam. They can win math olympiads and you know do all kinds of things that are really useful and impressive but they still can't do very basic tasks in the real world like you know clear up the dinner table, clean the house or learning to drive in 20 hours of practice like any teenager. So AI basically can deal with data at the moment that is represented as a sequence of discrete symbols. At least the type of AI we're talking about, generative AI. But it really doesn't work at all when the data is high dimensional, continuous, and potentially noisy. So things like images, video, audio, sensor inputs, instrument measurements, financial data, the type of generative approach that has worked really well for discrete symbol sequences like language really don't work at all for those things. The other limitation of current AI systems is what has come to be known as agentic systems. So agentic systems are systems that can you know produce actions and current agentic systems are based on LLMs and they are trained using imitation learning. You collect lots of examples lots of data from humans and then you train a system to basically clone the behavior of humans or imitate it. It can be fine-tuned by reinforcement learning through simulation. But the number of examples that is required is enormous and those systems end up being relatively brittle and specialized. The only way I can think of to build agentic systems is systems that have the ability to predict the consequences of their actions before they accomplish the actions. And that's the whole idea of world models. And in fact world models might enable AI systems to solve new problems without being trained explicitly to solve those problems. This is really kind of the essence of intelligence in animals and humans which is that when we face a new situation we can apprehend it and figure out a way to accomplish a task without you know being trained to do that. And then lastly, AI systems currently do not have common sense or understand the real world. There are famous examples of people who actually have a hobby of coming up with examples that totally confuse LLMs even though to us the answer would be obvious. So you know never mind human level intelligence despite the claims by you know some people in Silicon Valley that we're going to get super intelligent systems within a year or two. That's completely untrue because you know never mind humans we can't even emulate the type of physical intelligence that cats and dogs and other animals have. You know any house cat can play highly complex actions. They have a really good understanding of their own physics if you want as well as interaction with the physical world. And then you know if you go to humans you can ask a 10-year-old to clear up the dinner table and fill up the dishwasher and they will be able to do it the first time they do it without you know being trained to do so. So that's you know based on the world model. Same for learning to drive. You know any teenager can learn to drive in a few hours of practice largely without causing any accident. Yet we have millions of hours of recorded data of cars being driven by humans. We could use this to train a machine learning system to basically imitate human drivers and people have done this and it doesn't really work. It's not reliable enough. So that's why we still don't have level five self-driving cars. I mean there is Waymo but it's you know a completely different approach and it's not level five actually it's more like level four. So we keep bumping into this paradox which is actually very well known in AI called the Moravec paradox. Moravec is a roboticist in 1988. He says that you know kind of paradoxical that a lot of complex intellectual tasks like proving theorems, playing chess and things like that seem easy for computers but then the things that we take for granted like just manipulating an object. We're still completely unable to reproduce this with computers with robots.
So we may ask ourselves the question how do humans and animals learn and a lot of learning in animals and humans is basically mostly by observation. There was some hope in the past that we would you know approach human level intelligence through either supervised learning which is what I just mentioned or reinforcement learning. There was a big wave of interest in this about 10 years ago but that turns out to be completely impractical. Reinforcement learning is just too sample inefficient requires too many samples and the same for supervised learning. So what type of learning takes place in humans and animals. And the type of learning has come to be known now as self-supervised learning and in fact is an extremely successful approach to training systems for natural language understanding. So babies learn basic concepts about the world things like the difference between animate and inanimate objects. The fact that the table and the chair are different even though you know babies don't have language they learn those categories spontaneously. Concepts, very simple concepts like object permanence, the fact that when an object is hidden behind another one, it still exists. And then around the age of 9 months, infants can learn basic notions of intuitive physics like gravity, inertia, and things of that type. If you show a scenario like the one showed at the bottom here, where a car is sitting on a platform and you push the car off the platform and it appears to float in the air, a six-month old baby will barely pay attention to it. But a 10-month-old will you know open the eyes and fixate because you know her model of the world is being violated, right? She knows by the age of 10 months that objects are supposed to fall. So you know what is this type of learning? So there's been theories in theoretical neuroscience that this is learned mostly by observation and perhaps by prediction that the brain is kind of a prediction machine and there's a theory called predictive coding where the brain predicts something and then confronts the prediction with reality and sort of adjusts itself to make a better prediction. That's a very abstract concept but the question is how do you reduce this to practice? How do you make it work in practice? I recently co-authored a piece with Emmanuel Dupoux who is a cognitive scientist in Paris and Pieter Abbeel a roboticist at Berkeley and Amazon also a former colleague from Meta about this very question you know what's the difference between the type of learning that we can do with AI and human learning. Then there is the other question of what is intelligence really? So we you know we see people sort of publishing all kinds of benchmarks and they seem to have the hope that all of those benchmarks collectively are some measure of intelligence but all of those benchmarks are designed to test things like LLMs. They're not designed to test things like robots or physical understanding or reality understanding or even common sense in the way we understand it. And it's a wrong way of testing intelligence because intelligence is not a collection of skills. It's an ability to acquire new skills extremely quickly. Either zero shot with no training. You face a new situation, you know what to do. Or with minimal training like driving, okay, in only a few hours. So this ability to acquire new skills is really what defines intelligence or to solve new problems. So I also co-authored a short piece about this with some colleagues and it's more like a position paper if you want. And so you know human intelligence is specialized but we feel it's general because we can acquire new skills very quickly for any problem that we think we can apprehend. But again, human intelligence is very specialized. So the notion that human level intelligence should be called artificial general intelligence makes no sense because human intelligence is nothing like a general intelligence. There's no question we'll reach at some point the point where machines will surpass human intelligence in all domains. You know it already surpasses human intelligence in a number of domains. The question is adaptability and solving new problems. So perhaps the reason why we should not put too much hope in the fact that scaling up large language models and training them with more data is going to reach human intelligence is due to this very simple estimate. LLMs are trained on human produced text right so typically it's about 2 times 10 to the 13 words which is basically all the publicly available text on the internet. When you tokenize this you know tokens are kind of subword units, you get roughly 30 trillion tokens. Each token is about three bytes. So that corresponds to about 10 to the 14 bytes in terms of pre-training data. There's also a fine tuning phase in LLMs that enables AI companies to produce LLMs that can answer questions that people are likely to ask. But the pre-training phase is basically 10 to the 14 bytes. It would take about 400,000 years for a human to read through this at 9 hours a day, 250 words per minute. Now, a human child, a four-year-old is trained from sensory data, vision, touch, audition, in 16,000 wake hours during the first four years, which by the way, the small amount of video that corresponds to about 30 minutes of YouTube uploads. This child gets a huge amount of information to the visual cortex through the optic nerve. Optic nerve has about 2 million optical nerve fibers each carrying about one byte per second. And so do the arithmetic and the volume is about 10 to the 14 bytes. So a 4-year-old has seen the same amount of raw data as the biggest LLM trained on the entire corpus of publicly available text on the internet. So that tells you we're never going to get to human level intelligence by just training on text and other human produced data. This is just not going to happen. We need AI systems to understand sensory data, high bandwidth data from the world, but the world is messy and so that's actually very complicated. So in the end I think the progress towards the next AI revolution will be using self-supervised learning methods which I'll talk about next. But self-supervised learning methods that apply to high-dimensional continuous noisy data coming from sensors like video. And then from this we'll be able to build world models that will power agentic systems. So a world model is something that basically would allow an AI system to predict what is going to be the state of the world when I take this action you know can I predict the consequences of my own actions because if you can predict the consequences of your actions you can plan a sequence of actions to accomplish a particular task. That is why AI systems need models and LLMs don't have that. So agentic systems that understand any environment including physical environments and living things systems that have persistent memory they can perform long chains of reasoning that can plan complex action sequences possibly hierarchically and can accomplish new tasks without prior training or can adapt really quickly. And then finally systems that are controllable and safe. This goes way beyond what LLMs and other generative architectures are capable of. Okay, so let's review what type of inference should a sort of AI system be able to do? And really that asks the question what is thinking and reasoning? So the first type of AI system or even human behavior is what psychologists call system one. Okay, given the current idea of the state of the world, can I predict an action that I should take right now? Okay, in the context of reinforcement learning, that's called a policy, right? Given the state, what action should I take? But humans don't do this exclusively. That's what psychology calls system one. That means if you're accomplishing a task you're very very familiar with, you don't need to think about it. You can do it, you know, subconsciously when you drive and you're an experienced driver. You don't need to think about it. You can talk to someone at the same time and things like that. Okay. However, there is another way of behaving which is system two and that means is planning finding actions that accomplish a particular task using the full power of your mind to anticipate the consequences of your action and therefore plan a sequence of actions to accomplish the task. And that requires world models. Okay. So on the right you have the sort of prediction of actions in the world or tokens if it's an LLM, right? So you have a sequence of discrete tokens and you train a system to predict the next token in a sequence from enormous amounts of data and then you can use that system to do autoregressive prediction. Right? That's the way an LLM works and also VLA which are forms of LLMs that are used to produce actions for agentic systems. So you take a sequence of actions you run it through your LLM your predictor you run through a fixed number of layers of a neural net transformer and then you predict a distribution over all possible tokens in your dictionary there's only a finite number maybe 100 thousand or so. So you produce a score for each of them and that gives you a distribution and then you sample from that distribution and you get a discrete predicted token XT plus one. Then you shift that into the input and you predict the second token. Okay. Now the previously predicted token becomes part of the input. That's just autoregressive prediction. It's a very familiar concept going back decades in statistics and signal processing. So that's one model. Okay. The second model is a very very different one. Let's say you have through perception you have some idea of the current state of the world. Okay. You feed that to a world model together with an imagined action sequence. Okay. So here is an action or sequence of actions I imagine taking. My world model would predict the outcome. It would predict the resulting state of the world. Okay. Then you could feed this predicted state of the world to an objective function. So I use those rounded shapes to represent deterministic functions essentially and those square blocks as cost functions. So it's a function that has one or several inputs and has an implicit scalar output that is like a cost or a penalty. And so this task objective gives you an output of let's say zero if the particular task has been accomplished or a larger number if the task has not been accomplished maybe some sort of distance to accomplishing the task. So then the inference procedure is by optimization the system finds an action sequence that minimizes this objective function. Okay, so this is inference not by forward propagation through a bunch of layers of a neural net but inference by optimization. And this is a much more powerful paradigm because you can reduce any kind of computational problem if you want to an optimization problem. You cannot necessarily reduce it to forward prediction through a fixed number of layers or at least not efficiently. So you know based on this concept I several years ago about four or five years ago I put online a paper that sort of explains a vision for how an AI system could be built around this concept of world model. So the title is A Path Towards Autonomous Machine Intelligence and you can see the URL here at the bottom. This is not on arXiv. I put it on OpenReview and I put it there so that people can make comments and criticize and there's ample examples of that. In fact here is perhaps a better way to access this paper with a QR code. And there is some other version of the talk I'm currently giving you maybe earlier ones maybe emphasizing different aspects.
So let me tell you a little bit about how we go about this and how we build this and what progress we've made since the four or five years since I posted this paper. Okay. So we have this model of an intelligent system that are called objective-driven AI architecture where you observe the world you run it through a perception system that gives you some idea of the initial state of the world some abstract representation of it. You can combine this with memory that indicates what you know about the state of the world you're not currently perceiving. You feed that to your world model and you also feed a hypothesized or imagined action sequence and then the prediction from the world model goes to the task objective which is defined by what task you want to accomplish. Okay, this objective function and there may be other objective functions other additive terms in an objective function that would be kind of safety guardrails if you want. You can think of them either as penalty functions or as constraints that must be satisfied. And then an episode of that system consists in searching through an action sequence through optimization searching through an action sequence that minimizes the task objective subject to the constraints of the safety guardrails. Okay, this looks very much like classical optimal control for those of you who are familiar with this and in fact if you have a world model you can apply it multiple time steps and then feed a sequence of actions to it and then predict apply the guardrail cost to the entire predicted trajectory and then feed the final prediction to the task objective and then by optimization figure out the optimal action sequence that's actually essentially akin to what control theory is called model predictive control MPC and except here our model is not going to be a bunch of equations that we write down it's going to be a trained world model trained from observations and examples. Eventually what we want to do is a system of this type that is hierarchical that does not have a single representation of the state of the world but has a whole hierarchy of multiple abstract representations of the state of the world depending on the task we want to accomplish. So let's say I'm sitting in my office at NYU in New York and I want to be in Paris tomorrow or in Tunis for that matter. I know that at a very high level of abstraction we need to go to the airport and catch a plane. And the reason I need to do this at a high level of abstraction is that my low-level actions are muscle control. You know the output of the human mind always turns into muscle control. And it's you know millisecond by millisecond muscle control. There is no way I can plan my entire trip from New York to Paris in terms of millisecond by millisecond muscle control. It's too complicated. First of all, I also don't have the information. I don't have position of every obstacle on my way, right? Or how long I'm going to have to wait to board the plane or to catch a taxi. Okay, so I place myself at a very high level of abstraction and I can plan at this high level and say, well, I need to go to the airport and catch a plane. So that's a sequence of two actions, but they're very high level actions. Now, getting to the airport defines a new objective function, which is my distance to the airport basically. And I can plan at a lower level to get to the airport. I'm in New York, so I need to go on the street and hail a taxi. Okay, I can do this in New York. Then I have a sub goal which is being on the street and I'm sitting in my office. So I need to, you know, stand up, walk to the elevator, push the button, go down and walk out of the building. And so, you know, I can have a whole hierarchy of levels of abstraction like this. And the predictors at a high level make long-term predictions at a very abstract level with very few details and the lower levels make short-term predictions with more details. And at some point I will get to the level where I can just take an action if the level is I should stand up from my chair. I can do this without thinking about it. I don't even need to use my world model. It's a very familiar subconscious task. Okay. So that's what we need to build. And it's a bit of an unsolved problem. We have some papers on this but we're still it's still very early. Okay. So how do you know what is self-supervised learning? How do we train a world model using self-supervised learning? So the general form of self-supervised learning is at least when it comes to using generative prediction which has been so successful in the context of LLMs is you take an input let's say a piece of text a sequence of words you corrupt it in some ways like you remove some of the words and then you run this corrupted input through an encoder and a decoder and you try to predict the missing parts of the input. Okay, this is sort of generative self-supervised learning in its most general form if you want. This works astonishingly well for language and for any kind of sequence of discrete symbols like DNA proteins and stuff like that. But when you apply it to video, it really doesn't work that well. If you mask a piece of the video and you ask a system to predict the missing piece of the video, it doesn't really work. At least with a generative model because most of the details in a video are completely unpredictable. In fact we've done this experiment multiple times. We started working on this maybe 10 years ago or actually in my case more like 15 years ago and for the longest time we tried to play with generative models and what you get is those blurry predictions. Of course, this is a neural net from 10 years ago, so it would be a bit different now with things like diffusion models, but you still have this problem that you know, you train the system to make one prediction and it can't really figure out among all the possible futures which one will actually occur. So, it predicts the average. And there were other experiments by some of my colleagues at Meta also with sort of very simulated environments. You get this very blurry prediction. And although you can handle the uncertainty in the prediction in a context of text and language because it's discrete tokens you can produce a probability distribution over a finite number of tokens. You cannot do this with video because we don't have efficient ways of usefully representing distributions over the set of all possible video frame sequences. It's completely intractable mathematically. And so we played with things like latent variable models. You know you can play with diffusion models, VAE and things like that. And it still doesn't work. It's not really satisfactory. There's a lot of work on this and a lot of people when they talk about world models they really mean generative world models with latent variables diffusion models essentially. But it's really impossible to do because most of the details in a video are completely unpredictable. Most of the information in the future of a video that you see only the beginning part of are unpredictable and so all of that has to go into a latent variable and that's just a very difficult problem. So here is my solution. My solution is an architecture called JEPA. What that means is Joint Embedding Predictive Architecture. And so compare on the left you have generative architecture that tries to directly predict the part of the signal the input signal that you cannot observe conditioned on the part that you can observe x and perhaps another part which I would qualify as an action or a transformation right so a could be a temporal translation so if x is the beginning of a video a just means wait for half a second and y would be the next segment of half second of video in the context of robotics that would be the action you send to a robot or whatever. Okay, so that's generative architectures. What I'm proposing the JEPA architecture basically says you simply cannot predict all the details of Y. So let's learn an abstract representation of Y and make predictions in that abstract representation space and that abstract representation space will eliminate all the details about the input video that I simply cannot predict that are simply unpredictable. Okay, it's a very simple concept. It sounds very natural when you think about it. But that's what we need to build. And there is a technical issue with this which is that when you train a system of this type to simply minimize the prediction error right when you train a generative model to minimize prediction error it just predicts the next item in your sequence. But when you want to train a system like this and you just minimize the prediction error in representation space it can cheat. It can say well I'm just going to predict a constant representation and then my prediction error is going to be zero all the time. Okay, ignore the input and then make SX and SY just constant. And now your prediction problem is trivial. That's called a collapse. So basically training an architecture of this type consists in preventing collapse. There's a number of different methods to do this. There are methods that are based on distillation. So things like techniques like DINO developed by my former colleagues at Meta in Paris, I-JEPA and V-JEPA developed by my former team at Meta many of whom are now working with us at AMI Labs. And then other methods based on information maximization and I'll mention VICReg. This is the only one I'll really explain a little bit and there is a number of versions of this that my team but other people as well have developed. This whole idea of JEPA actually has become quite popular over the last four or five years since I proposed it. If you search joint embedding predictive architecture on Google Scholar between double quotes you get about 1,300 papers. So there's a whole community really interested in this which is growing. So my money at the moment is on those information maximization methods which basically have some way of measuring the information content in the representation SX and SY and during the training procedure you try to maximize this information content. So train an encoder that extracts as much information about the input as possible but yet simultaneously minimize the prediction error. So the system will find a trade-off. It will try to extract as much information as possible but it will only extract the information that it can actually predict so as to minimize the prediction error and so it will eliminate all the stuff you cannot predict the noise or the things that are just too complicated unpredictable.
So this notion that we need to find abstract representations of the world is very natural not just to build intelligent systems but also in science when we model the world when we as applied mathematicians of course you're all familiar with the concept of writing down a model of reality right let's say you want to simulate the flow of air a fluid around an airplane or something like that Right? You model the state of the air in a little cube by velocity, pressure, density, viscosity perhaps, temperature. And then you solve the Navier-Stokes equations, right? The partial differential equations. This is already an abstraction. The underlying reality is molecules of air bumping into each other and bumping into the airplanes. But no one actually simulates this, right? We always find an abstraction that allows us to make the simulation and makes the simulation more useful. In principle, you know, we could explain everything that takes place between us right now in terms of very elementary physics theory like quantum field theory. It would require an enormous amount of computation and also measuring the wave function of the universe within a cube that contains all of us, right? Which is completely impractical and the amount of computation that would involve making a very short-term prediction is just insane, right? So, what do we do? We invent abstractions, particles, atoms, molecules. So, at the level of molecules, we can start having models, abstract models that give us chemistry essentially, right? Which molecule will stick with which other molecule, things like that. So, you know, physics make fun of chemistry telling them, well, you know, chemistry is just applied physics. Yeah, it's true. But there is a big advantage in actually using a model at the level of chemistry as opposed to ignoring all the underlying details coming from quantum physics. Then in the living world we have concepts like proteins and organelles, cells, organisms, individuals, societies and ecosystems. Every level in this hierarchy allows you to make predictions to build models and make predictions. And the higher level in the hierarchy, the less details those predictions contain. But the longer term predictions you can make about more complex and bigger systems. And every level in this hierarchy ignores some details about the level below. In physics actually this is down to an art. This is called entropy. Right? When you abstract away some property of a physical system and you only describe it collectively the information you ignore you call it entropy. So you know the same way I cannot simulate the air molecules bumping into an airplane. I will not simulate the air molecules inside of a box. Okay. But collectively I can tell that if I heat up the box the pressure will increase right PV equals NRT right ideal gas law. That's a collective law which allows me to make predictions ignoring most of the information about the state of the system. Right? So this notion that we need intelligent systems to find abstract representations of the world to be able to make predictions is absolutely fundamental and it's really the only idea behind JEPA. And what's funny is that I have a hard time convincing people that they should abandon generative models. It seems completely obvious to me but it's very difficult to convince people of this. So a world model which allows you to make predictions should not be a simulator of the world. It should be something that simulates only the part of the world that you think is predictable and relevant. So a world model is not a digital twin. It's not a generative model. It should not be a video generation system. What the world model should be is an action-conditioned predictor in abstract representation space of the next state of the world and preferably differentiable so that the optimization process I was telling you about for planning can be done by gradient-based methods as opposed to combinatorial search. So what can we use a world model for? Let's say we want to optimally control a turbojet engine it's very difficult to write a small bunch of equations that we can solve that explain the holistic behavior of a turbojet engine. Turbojet engine typically has something like a thousand sensors, right? So you get measurements for a thousand sensors. What you'd like is being able to extract an abstract representation of the state of the engine and then maybe an action or intervention you imagine taking. And then observe the outcome the next state of the system through measurements for a period of time. And then at this abstract level predict what the next state of the engine will be as a consequence of the previous state and the action you imagine taking. If you have such a model then you can plan a sequence of actions or interventions or controls that will optimize metrics like fuel consumption, efficiency, longevity, CO2 emission or whatever you want. This is really classical optimal control but with a model that is trained from data as opposed to written in the reductionist approach of writing down a bunch of equations. Okay, I'm not trying to put applied mathematicians out of business here. This is just a new way of building phenomenological models of a complex system which is very difficult to reduce to a classical set of equations. You can apply this of course to more complex systems like say a chemical plant or oil refinery or something like that or any industrial process really or something like a patient which is an extremely complex system. You need to have an abstract representation. You're not going to simulate a patient at the level of biochemical interactions, right? You need something more abstract. But you could imagine having phenomenological models that would allow to treat a patient that has a chronic condition like diabetes for example or even plan a course of action to tell a stem cell to turn into a pancreatic beta cell that produces insulin to take an example. Of course you can use this to control robots. So the sad state of robotics today is that there's a lot of companies building robots including humanoids and they can get them to do really amazing feats like kung fu or whatever dancing except what's difficult is to model the interaction of those robots with the physical world like manipulation or things of that type modeling the real world is complicated and this is what you need JEPA for. So this idea of world models is very old. It goes back to the late 50s early 60s and the whole idea of model predictive control also goes back a long time 1970s. This is not a new concept. This is not invented by me or by Jürgen Schmidhuber or by the machine learning community. It really was invented by the optimal control community it's about as old as I am essentially. And the mathematics behind this is really interesting because it's all classical mechanics and stuff like that. Okay. So how are we going to train those world models and there was a lot of evidence in the past that in the last six seven years that the proper way to do self-supervised learning is using one of those joint embedding architectures and not a generative architecture. So the way you do this is you train a system to learn representations of images. You show it two images that are really different views of the same scene or maybe one is an image, the other one is a corrupted version of that image and you train the system to basically predict the representation of the good image from the bad one if you want or from the transformed one. And once the system has been trained like this which is completely self-supervised, you don't need any labels. You use the representation and the output of the encoder you fix the encoder and you train a simple classifier on top in a supervised manner maybe with just a few samples and you measure the performance. Okay. And what we figured out is that the architectures that perform the best in this context are architectures that are joint embedding that do not attempt to reconstruct. So people have tried reconstruction-based methods like this like autoencoders, variational autoencoders, masked autoencoders which is a special form of denoising autoencoder and diffusion models. The representations you get out of those things are not very good. The ones that are really good are the ones that use embeddings and do not attempt to reconstruct. When you attempt to train a system to reconstruct noisy high-dimensional noisy continuous data, you kill it essentially. So I told you that the main challenge there is to prevent collapse, right? When you train the system here, the system is happy to just ignore the input and produce constant representations. So one idea that a recent idea that we have we have a number of techniques to prevent this collapse but I'm only going to talk about the latest one because I think it's really interesting and it's called VICReg and that means Variance-Invariance-Covariance Regularization and it's basically a way of maximizing information content some measure of information content of the vectors that come out of the encoder and there is a basic theoretical issue there which is unsolvable in my opinion which is that to maximize something what we would need is an exact measurement of that thing or a lower bound on the quantity right so to maximize information content we need a lower bound on information content unfortunately we cannot exactly measure information content because that assumes having an exact estimate of the probability distribution of the points and the only thing we have are discrete samples. So we cannot actually measure information content and the only measures that we have that are approximate are upper bounds. They're not lower bounds. They're upper bounds because they make assumptions about the type of dependencies that you take into account when you do the measurement. Okay. So the technique we arrived at very recently is this technique that consists in given a bunch of data points that come out of your encoder. Here they're represented as Gaussian blobs but really they are data points. If you were able to make that distribution to train the encoder so that the distribution of those points over a batch of samples were an isotropic multivariate Gaussian then all the variables would be independent of each other. The norm the system will not be collapsed to zero because the variance has to be one. Let's say you would constrain the variance to be something and the variables would be independent. So you have like maximal information. So a joint Gaussian is actually the maximum entropy distribution for a given variance. Okay. So how do we do this? That seems completely intractable but the way we do this is using some theorems. And one of those theorems which comes from Blackwell is you project the distribution the empirical distribution along many different axes and so the marginal when you project along a particular axis is going to be a one-dimensional distribution. Okay. And then you say what is the distance between this projected one-dimensional distribution and a Gaussian. And it turns out you can easily compute some sort of distance between this empirical distribution and the Gaussian. In fact the easiest way to do this is not to do it in distribution but in cumulative distribution. Okay. So you compute the cumulative empirical distribution of this one-dimensional projection. You sort the samples according to their value and then you can compute the staircase function that is the cumulative distribution. You can compute the Wasserstein distance essentially the earth mover distance between this distribution and the ideal cumulative distribution of a Gaussian and basically what you get is a gradient for every data point. It tells you how to move it how to change its value. So the cumulative empirical distribution looks more like the Gaussian. Okay. Now, what we do in VICReg is actually different. We compute the characteristic function of the empirical distribution and the Gaussian and we compute the distance in that characteristic function space which is basically Fourier space. So this is a criterion called energy distance. And that's what we use for VICReg. But in fact the more recent versions we're actually kind of reverting to this cumulative distribution idea. So what you see here are various examples of making a bunch of points that come to you in that form. How do you move them so that their distribution becomes Gaussian using various criteria. So this minimizes the covariance the off-diagonal terms of covariance matrix and those use different criteria. Energy is the one that we retained for the VICReg work but again as I said we have some other work that uses the cumulative distribution. Okay so what you get now is a gradient for every point coming out of your encoder and every component of every point coming out of the encoder that tells you how to change the value of that particular component of that particular sample so as to make the overall distribution more multivariate isotropic Gaussian you back propagate the gradient through the encoder and change the weights okay so that's a regularizer and simultaneously you minimize the prediction error and you adjust the parameters of all the neural nets to do that simultaneously. So this works surprisingly well on relatively small scale examples. We still have to do the work of kind of scaling this up to very large neural nets. But more recently we use this technique to build a world model and this works quite well or at least the results are very promising. This is a URL where we show results on this. So this is with collaborators at NYU Lucas Manuelli who's a student at MIT and Dian Sier who is also at MIT myself at NYU and Ronald Balistrieri who's at Brown University is a former postdoc of mine so this is what I just explained except here we have a predictor that is action-conditioned. Action-conditioned on the actions that the robot takes when we train it and this works quite well. We can use this to basically train a system to plan sequences of actions to arrive at particular goals. We can train a decoder to so the system tells us what it imagines in its prediction which seem to work pretty well and it works relatively well on a number of different planning relatively simple planning tasks that are simulated. I'm not going to
to bore you with details. Okay, now let me... so this is what I think is the most promising, but it's not the most accomplished today. What is most accomplished today are the distillation methods to learn representations of images or video. In particular, a technique called I-JEPA, which is a few years old, it's already three years old, where we take one of those JEPA architectures and the way we train it to prevent it from collapsing is something that's a little mysterious. We don't fully understand why it works, but it does work. And the idea comes from a number of different sources, but I think the original idea can be sourced to a paper by Google called 'Bootstrap Your Own Latent'. And the idea is that you still have those two encoders, but the way you minimize the prediction error, you have various normalizations to prevent the system from collapsing. But the other trick to prevent the system from collapsing is to basically not propagate gradient through this part of the system. So this encoder does not get any gradient, and the weights of that encoder are some function of the weights of the encoder on the left, which does get gradients during training. And the way we compute the weights of the encoder on the right is that we take a sequence of previous weight vectors of the encoder on the left during training and we compute an exponential moving average, basically a running average of the past weights of that encoder, and that's what we set the weights of this encoder to. And that somehow prevents collapse. Why? We don't really know, which is why this method is not really satisfactory, but it does work really well and we can scale it up and we get really good results. There are some theoretical papers that attempt to explain why this works. One by Yang LeCun in particular, a former colleague from Meta and his colleagues. They can show that with some hypothesis, if the encoder and the predictor are linear, then you can prove that there are fixed points of these dynamics that are not collapsed, but that's all we know. So it's not entirely satisfactory, but it does work extremely well. Another such distillation method is called DINO. This is a work by our colleagues, former colleagues at Meta in Paris. The latest one being DINOv3. And it's basically the best way of extracting generic features out of images that you can use as image representation for any downstream task you want. This is, you know, the best system in the world for this and it's, you know, open source, extremely popular. It's been applied to a number of different applications including medical imaging and satellite imaging and things of that type.
Okay. So how do we use those JEPA world models to plan? You can train the predictor in an action-conditioned manner. And then once you have this predictor, you can use model predictive control. You can imagine the outcome of a sequence of actions, measure a cost, a distance measure or divergence of some kind which is a distance from the final predicted state to a target state that you want to achieve. And then through optimization, plan a sequence of actions to minimize that cost. Okay, that's objective-driven AI, basically planning. There's a number of papers that use pre-trained encoders for this, either DINO or some others. And this works reasonably well on simple tasks. Let me show you a video. I hope the frame rate is high enough for you to see. So this is the world model trained on top of the DINO representation and what you see on the left is the initial configuration, at the top target configurations, and at the bottom the sequence of actions that the system has inferred using planning to make, basically, turn the initial representation, initial state into the final state. We applied this to a number of different simulated, kind of toyish planning scenarios and, you know, it works reasonably well.
So more recently we trained one of those JEPA architectures end-to-end. So using, you know, training the encoder and the predictor and training a world model based on this, but for video. So you take a video, you corrupt it by masking a big chunk of it and then you train this entire architecture. Some examples will have actions, some others will not. The first phase doesn't have actions. You just train the system to predict and then you use the representation of those short video segments as input to a downstream task like action recognition or something like this. And this works really well. This is probably the best system there is to, you know, represent video in a generic way that can accomplish a variety of tasks. But what it can do is basically those systems acquire some level of common sense. So the V-JEPA, the first incarnation of it, V-JEPA not V-JEPA 2, we put to the test and we try to figure out to what extent the system has learned what is possible and what is not possible in the real world. So you show it a video where something impossible occurs, like, you know, a ball is thrown in the air and follows the usual trajectory but then at some frame it disappears or it turns into a cube or something else that's impossible occurs. And internally to the system you can measure the prediction error. When such an impossible event occurs, the prediction error shoots through the roof and so the system basically knows, is surprised by things that are impossible. And it's the first time that we observe this kind of phenomenon in a system that has spontaneously, through just observation, being trained to predict at the representation level, basically has acquired some level of physical common sense. This is really, really new and interesting.
Okay. So the second phase of training of this V-JEPA system is to train it with action-conditioned videos from robots, real or simulated, and then use that as a predictor to basically plan sequences of actions. And this is a short video that shows an example of planning. So we start from an initial situation. There's a goal target which we run through the encoder that gives us a target value for the state. And then by search and optimization, the system plans for a sequence of actions that will bring the system closer to the goal with the cup being moved. And this is a sequence of actions that were actually planned by the system. We have a slightly more recent version of this now that can do sub-level of hierarchical planning, which I'm not going to explain, and a more recent version of V-JEPA called V-JEPA 2.1. You can get the open source code and the pre-trained models here. Those systems are really good at extracting information from video and they're really good for planning as well. They can do things like depth estimation. So the V-JEPA 2.1 system, if you train a very simple head on top of the representation to predict the depth of every pixel in an image, in a single image, it does a pretty good job at it. And it's kind of interesting because basically the system has learned to represent the 3D world without being told anything about the fact that the world is three-dimensional. It's just being trained on video. But it's learned probably that the best way to explain how our view of the world changes as we move the camera is to basically give a depth to every point in the image. And the system in its representation basically encodes depth implicitly. So that's a really kind of interesting point. It also includes things like boundary of objects and categories and things like that. It does a pretty good job at this. So you train a head to do semantic segmentation and it does a pretty good job. And you can sort of have some graphic color representation, the sort of first few principal components of the high-dimensional representation produced by V-JEPA 2.1 and you see the kind of object segmentation, semantic segmentation you would get out of it.
Let me skip that and go to the conclusion. I'm probably taking a little bit too much time. So I have a number of recommendations for people who are interested in pushing AI to the next level, which is to abandon generative architectures in favor of those joint embedding predictive architectures. That does not make me very popular in Silicon Valley because the entire industry is really investing gigantic amounts of resources in generative AI and LLMs and I'm telling everyone like this is not going to go very far. I'm not saying they're not useful, they're very useful, they should be worked on, but as a path towards human-level intelligence, LLMs are a dead end. So abandon generative models, abandon probabilistic modeling in favor of some other paradigm that I call energy-based models, which I didn't have time to talk about, but it's basically the way to explain how you train all those systems to prevent collapse and things like that. There is kind of a way to understand that. Abandon contrastive methods. I didn't talk much about this because I focused on those information maximization methods which I think are much more promising. Contrastive methods are kind of the old way to train joint embedding architectures and prevent collapse. I've also been vocal about saying for the last 12 years, abandon reinforcement learning or at least minimize its use because it's so inefficient in terms of samples. And so if you're interested in really pushing AI towards human or animal level, you should really not work on that. You should try to work on world models. And I've argued that world models should not make predictions in a generative fashion but should attempt to find abstract representations of the world. And since we started to have good results in those experiments with world models and JEPA, and since most of Silicon Valley is really focused on LLMs, I decided to leave Meta at the end of last year and create a company called AMI, Advanced Machine Intelligence, that really is focusing on developing this thing. We're still doing research. We're open sourcing. We're publishing. We're just getting started. We're hiring too. And basically, you know, solve all the problems in the real world that AI currently really is not so good for. And the hope is that maybe hierarchical JEPA, hierarchical planning will allow us to train sort of universal causal models of any complex phenomenon and be the basis for any intelligent system in the future. And thank you very much.
Moderator 1:03:46 ↗
Thank you very much, Professor LeCun. We can thank Professor LeCun. And now perhaps some questions.
Okay. Thank you.
Okay. I have a first question. The fact that you accepted the invitation in this applied mathematics webinar is, in my sense, the best proof of the importance of this science. Do you think that there are enough mathematicians dedicating their research to AI, or no, do you think that we need more mathematicians?
Yann LeCun 1:04:35 ↗
Yes. So my academic position is with the Courant Institute, which is actually, since September, a school within New York University. It's a school that regroups mathematics, pure and applied, Courant of course is famous for applied mathematics, computer science, and data science. It's a standalone school. It used to be part of the Arts and Science school but now it've a separate school which basically is because of the importance of both mathematics, particularly applied mathematics, computer science, particularly with AI, and data science, which is really applied AI if you want. So NYU has decided that this whole field is so important that it should have a dedicated school separate from, you know, the law school and the business school and the Arts and Science and the engineering school. I've been personally, because being part of Courant, although I'm part of the computer science department, I've had a lot of interactions and contacts and even students from applied math. And I've, for many years, I was on the science advisory board of the Institute for Pure and Applied Mathematics at UCLA. And I've always tried to bring in interest from mathematicians to machine learning and deep learning in particular because the mathematical problems there are, in my opinion, very interesting but also very complicated. And traditionally, people interested in machine learning who have a bit of an interest and background in mathematics have focused in the past on simpler machine learning paradigms, think like kernel methods, support vector machines, Bayesian inference and things like that where the theory is easier to write. Neural nets are messy. And so theorists have reluctantly approached deep learning because it's very complicated. The functions we optimize are non-convex, so you can't prove anything. We use stochastic gradient, which for an optimization person is really, really hard to model the behavior of. All the traditional tools in optimization and criteria that we use to analyze optimization algorithms go out the window because in machine learning we don't care about precise optimization because that creates overfitting. Even statisticians and probabilists are baffled by a phenomenon called double descent. So traditionally every textbook in statistics tells you if you try to fit a function to a bunch of points, you don't want too many parameters because your system is going to overfit. It's going to, if you have 10 points and you use an 11-degree polynomial, your polynomial is going to go through every point exactly. But between the points it's going to go completely wild. So as an approximation for noisy signal, for function approximation, you need models with a small number of parameters. That argument turns out to be completely false empirically initially. And now we need to understand this theoretically. So neural nets are way over-parameterized. The number of parameters in a neural net is enormous compared to the amount of training samples. Yet those systems generalize pretty well. They interpolate quite well. They even extrapolate. And that's a phenomenon, an empirical phenomenon called double descent, where the error on a separate test set of a model is going to go down as you increase the number of training samples. As you increase the power of the model, like the degree of the polynomial or number of parameters in the neural net, then it's going to increase. And this is where overfitting occurs. But then as you increase the power of the model, the number of parameters, it actually decreases again, at least if you regularize it properly. And it may actually go to zero. So this phenomenon is a little baffling. There are a few intuitive explanations for it. I've had one for a long time, but it's not completely understood theoretically. And then there's more specific properties I think where mathematics... so I didn't talk much about energy-based models. There's a lot of really interesting research and mathematics I hope to do there, which is basically, okay, let's say you want to estimate the dependency between two variables, two scalar variables, and I give you a bunch of data points. And the dependency between the two variables X and Y is not a function because there might be multiple values of Y for a given X that are in your samples. How do you capture this dependency between X and Y? So of course you could say, well, I'm going to learn a joint distribution between X and Y, but that may actually be extremely difficult, particularly in high dimension. Estimating distributions in high dimension is basically impossible and intractable. You have to represent it by an energy function. You take the exponential minus and you normalize, and the normalization constant is intractable. It's like the partition function in physics. You don't know how to integrate complex energy functions. So it's basically intractable. Okay, so how about something that's simpler, and that's energy-based models, which consists in, can I learn a contrast function that will give me, which is scalar outputs, and it takes a pair of X and Y, and if the pair of X and Y is in the region of high data density, it gives me a low value, let's say zero, and if I move away from the manifold of data, the value of this scalar function increases. So it's basically a contrast function, an energy function that tells me you are on the data manifold or you are away from it. And that's all I need to capture the dependency between two variables. It's a much better way, actually a much simpler way also of capturing dependencies than probabilistic modeling. There's a lot of theory to do about this. But okay, it's a long answer. I'm sorry.
Moderator 1:10:56 ↗
There is Nasr from ENS Paris-Saclay. Can you... how can we do... can you... I already asked him to unmute. Nasr, please unmute your...
Okay, sorry. It's correct now. I don't know if you can hear me.
Yann LeCun 1:11:18 ↗
Yes. Yes, I can hear.
Perfect. Perfect. Thank you. So, thank you very much for the talk, Professor LeCun. I found the distinction that you made between simple action prediction and the true planning with a world model especially interesting, and also your point that the next wave, if I can say, of AI has to deal with more high-dimensional, noisy data rather than just tokens. My main question is about the deployment in safety-critical industries like space, nuclear. I'm actually a nuclear engineer in Paris, within the R&D team, what we call the DEC, the Digital Excellence Center. So that part resonated a lot with me because our reality is exactly this, actually, what you described: sensor streams, inspection images, maintenance data, etc. So my question is, to make a link with your slides, you argue that world models should not be digital twins and you also take a fairly strong position against generative models like ChatGPT or whatever. But in nuclear physics-based models, explicit simulators, or just how to say it, the uncertainty quantification are not optional. You know, so we are judged not only on performance but on validation and auditability, etc. So how do you see JEPA-style, and I'm not a specialist so excuse me about that, but the action-conditioned world models that you describe, how do you see them interfacing with the first-principles engineering models in domains like nuclear? Do you see them as replacing digital twins or rather, how to say it, as a higher abstraction layer on top of them for planning, for anomaly or accident anticipation, etc.?
Yann LeCun 1:13:25 ↗
Yeah. Yeah. So I mean what we anticipate, and this is something of course we think about a lot in the context of AMI, which by the way is pronounced the French way, AMI. AMI. And we think about that a lot because we think that's basically our first business and customers would be the control of complex systems, such as a power plant, possibly a nuclear power plant. By training a phenomenological model of the system that allows to plan for sequences of intervention for any particular situation. Now you can train this with real data. You can also pre-train it with synthetic data that's obtained from digital twin simulation, but then completed with real data which probably makes it more realistic. And then the system can also kind of fine-tune itself as it goes. It makes a prediction at every time step, so it can then observe the result and say, oh, I need to adjust my model because I see a discrepancy between what I predicted and what actually occurred. I mean this is similar to what people do in model predictive control, procedures like MPC and things like that. So yeah, I mean that's probably our core business is that, you know, before we have reliable models that would be usable for a nuclear power plant control, it might take a while, but yeah, but that's kind of the hope frankly.
Okay. Okay, that's interesting. Okay, thank you. Just to add a little bit of information, in nuclear we are trying to simplify the process of building nuclear power plants, etc. So we have what we call SMRs, I don't know if you have heard about them, which are small modular reactors, a type of, let's say, micro-modular reactors. So with your JEPA-style models, I think there is a potential collaboration that you could benefit from. Yeah, it could be something useful. So thank you again. It was very interesting to follow you. Thank you.
Yann LeCun 1:15:27 ↗
Yeah. Thank you.
Moderator 1:15:35 ↗
Is there another question? There is a question in the chat. I don't see it. Yes. Oh, yes. Muhammad. If you are with us, you can ask them. Ask Dr. LeCun directly. Muhammad.
Muhammad 1:16:00 ↗
Yes. Can you hear me?
Yann LeCun 1:16:02 ↗
Yes, I can hear you. Yes.
Muhammad 1:16:03 ↗
Okay. Yeah. Again, I'm very happy to see you live. My question is a recurring one. It's about how the spark of intelligence occurred suddenly in a system that's only conceived to predict the next word. It's based on language. Yet at the same time we have the impression it's exhibiting a real intelligence at times and it's baffling. I was wondering whether you had an explanation on when and how that happened.
Yann LeCun 1:16:39 ↗
Okay. Sure. There is only two or three domains where LLMs exhibit sort of baffling intelligence, if you want. And those are computer code generation, mathematics, and perhaps law. Okay. So legal text and things like that. And the reason is these are domains, at least for the first two certainly, where the language itself is a kind of substrate for reasoning. When we do mathematics, the simple manipulation of symbols gives us sometimes results that we did not expect, right? So the reasoning can be basically supported by the language itself. And there's a similar phenomenon when you write code where you may have some idea about how to organize the code, but when you start writing it, it kind of crystallizes and the simple fact that you're writing the code sort of makes your concepts clearer in many ways, at least if you're a good computer scientist. So you can imagine that for those two domains, manipulating symbols, which is what LLMs do, can help with reasoning. And in fact, the way those systems do math and code is that they generate lots and lots of different sequences of symbols and then they have another network that kind of selects the best ones. And if it's code generation, you can just test whether the code works and then iterate, which is why code generation takes so long. If it's mathematics, you arrive at a result and you can try to prove it. You can verify if a result is true or not, perhaps with numerical examples or something like that. So those are two domains where there is self-adjustment because you can verify if an answer is correct or not. And the reasoning substrate is the language itself. Most of human reasoning and intelligence has nothing to do with language. It has to do with our experience of the real world, but it's not language-bound or it's not even helped by language. And it's certainly true of animals, they don't have language, at least not to the same extent. So okay, so that's only a partial answer to your question. The type of intelligence that we observe in LLMs is very narrow, is very restricted. And the reason why they seem to be intelligent by answering all the questions we ask them is a bit of an illusion. They basically do retrieval. Those systems are not particularly intelligent, but they are extremely good at accumulating enormous amounts of knowledge and then regurgitating that knowledge at the right time and maybe adapting it in minor ways. But they're not that smart. In fact, they give you really stupid answers sometimes. There is a story that has circulated recently on social networks where someone said, okay, I need to wash my car. I need to get my car washed. And the car wash place is 100 meters from my house. Should I walk? And most LLMs, ChatGPT, the latest one, Claude, all of those, they say no, you should... I mean, yes, you should walk because it's only 100 meters away, not realizing you're not going to have your car with you. The only one that answers correctly is Gemini from Google. But it's very likely that the reason it answers correctly is because in the last few weeks it was fine-tuned with the correct answer. It was tuned to detect that question and answer it correctly. I was made fun of in the past year or two because on an interview with Lex Fridman, a pretty popular podcast, I said, look, LLMs don't have any physical... if you put an object on the table and you push the table, the object will move with the table, right? That's kind of completely obvious physical intuition. But if you ask ChatGPT or one of its successors, it's not going to be able to understand the physical reality behind this and make the right prediction. And of course, six months later, if you ask that question to ChatGPT at that time, it would not answer correctly, but if you ask the next version of ChatGPT, it did answer correctly. And the reason is it's not that it understood physical reality any better, it's that it was explicitly fine-tuned to answer that particular question. Because as soon as the podcast was published, hundreds of people actually asked the question to ChatGPT to verify the answer was incorrect. And so that became part of the training set now, and OpenAI integrated that question in the fine-tuning set. And of course, any particular question that you ask an AI system, you can fine-tune it to answer it correctly, but it's retrieval. It's like those students you have in your class that do rote learning but don't really have a deep understanding of the underlying mathematics.
Muhammad 1:22:06 ↗
All right. Thank you.
Moderator 1:22:10 ↗
If there is no other questions, I have perhaps...
Audience Member 1:22:18 ↗
Yes, I have a question. Thank you, Professor, for this nice lecture. In this lecture you focus especially on human interests in artificial intelligence. My question is in that direction. Should humanity be afraid of AI in the future?
Yann LeCun 1:22:41 ↗
Yeah, it's a very common question. Okay. We should be careful about AI because it's powerful technology with a big impact and we have to make sure the impact is beneficial to all of us. But our society is used to doing this. We do this with every new technology that comes out. We have, particularly if it's consumer-facing, we have regulation to make sure it's safe and things like that, or particularly for things like nuclear power. But we should not be afraid of it for the reasons that a lot of people are claiming we should be afraid of it. The idea somehow that AI is qualitatively different from other technology and is intrinsically dangerous at the research level, that's just completely false. That's just ridiculous. In fact, it's basically a cult that started in Silicon Valley. It's called effective altruism, of people who are claiming to kind of try to make long-term predictions about the future of humanity. And basically they focus on the idea that if a technology is potentially dangerous or powerful, necessarily is dangerous, then we should stop working on it. I mean it's just completely insane. And a lot of people who are working on AI today come from that community. People like Dario come from that community. People like, to some extent, Sam Altman also, although he's moved away from it. And Dario to some extent as well. But Dario Amodei makes claims, very recently in the last two weeks, that Claude is so powerful that we don't want to put it in the hands of everyone because it's too dangerous potentially, it can create cyber attacks. It turns out after investigation it's complete BS, it's just completely false. There's nothing new to cybersecurity capabilities of Claude that you couldn't do with existing tools. It's just that they carefully, it's a strategy, it's a marketing strategy basically, they carefully prompted the system into attacking known software in ways that they knew was vulnerable, and it was completely debunked by the computer security community basically. And so it's a strategy for them. They've been doing this for years, basically lobbying governments to tell them AI is very dangerous, you should regulate AI research, you should ban open source AI research because it's too dangerous potentially, it can lead to humanity extinction. But their strategy is basically to slam the door behind them. They think they're ahead of everyone, they have a huge superiority complex, and basically they're telling governments, regulate everybody else, don't regulate us because we're smart, we know what we're doing. But like the plebs, the unwashed masses, don't know what to do with AI so you should not make it available to them. I mean I think it's incredibly aristocratic or feudal. I think it's insane. There's another point which I think is super important for a lot of regions in the world, including Tunisia and North Africa more generally, but Europe as well, and a lot of places in Asia, which is that we need open source AI. The only way to have any level of sovereignty in AI is to have access to open source, powerful open source AI systems. And so if you are a proprietary AI company like Anthropic, OpenAI, you want to stop that. You have a commercial interest in stopping it.
Moderator 1:26:33 ↗
Okay. Thank you. Professor, can you have enough time for other questions or...
Yann LeCun 1:26:39 ↗
Yeah, sure.
Audience Member 1:26:42 ↗
I have a question. Thank you very much. There is one question from... Yes, I have also one if you have time.
Yann LeCun 1:26:53 ↗
Yes.
Audience Member 1:26:54 ↗
Yes. Thank you, Professor Yann, for this very stimulating talk for me as a mathematician. So I look for the content from the side of mathematics and then particularly in the slide when you speak about action-conditioned world model and causal model. Unfortunately, you say that the model does not predict every detail of the outcome but performs a prediction in an abstract representation space. In my opinion, this calls for more deep mathematical model to go forward, maybe a performant model which predicts all details or more details. So in this slide, I have two questions. First, what is the mathematically precise notion of state in world models? So is it closer to a stationary statistic or Markov state or simply an element of the representation space, like in base space representation or something like this?
Yann LeCun 1:28:10 ↗
Right, it's a very good question. So I have a very general notion of what a state is. There is no single definition for a particular physical system or whatever system you're trying to model. There's no single proper definition for a state. There's multiple levels of representation for state. A particular, a good state is whatever allows you to make the prediction you want to make. So that's my definition of a state. And there is no unique definition for any particular system because, as I said, perhaps the state of a flow of air flowing around an airplane is the position and velocities of all the molecules of air and maybe temperature, for the solid that they move around. But that's a very low-level, very high-dimensional state representation, very detailed, which only allows you to make very short-term predictions at a huge expense. So we find abstractions. As I said, you forget about molecules of air, you represent the state of the air within a small cube or a small volume by velocity, density, temperature perhaps, viscosity, things like that. And then you sort equations. But then there is also a higher level, which would be the same simulation but with a bigger discretization of space. And as you make that bigger and bigger, it's still a state, it still allows you to make predictions, it's just less precise, but perhaps the prediction can be done for longer. So for example, I don't know if any of you practice sailing, but when you sail in a sailboat, you have to figure out how to tune the sail so as to basically maximize the forward force on the sail and maximize the speed that your boat is going at. And essentially the best, the way I do it mentally, is basically to kind of simulate at a very abstract level the flow of air behind the sail. A lot of it is built empirically, but I do have a mental model of how the air behaves in the sense that I know that if I bring in the sail too much with a too high angle of attack, there's going to be turbulences behind it, which I can to some extent correct by bringing in the front, the jib, right in front of the main sail. I know that if I go downwind, there is no way I can have laminar flow on the sail. So the best way to maximize speed, even including downwind, is to not actually go downwind. It's to go slightly off the wind. I know that if I go completely upwind, there is no way I can get any forward force. So I need to stay some number of degrees away. So you have this mental simulator. It's not solving Navier-Stokes equations. It's at a much higher conceptual level, but you do identify when you have turbulences and angle of attacks and stuff like this. This is the kind of model I'm talking about, phenomenological models that you can build from data and from experience, that does not necessarily simulate all the details. And sometimes it's absolutely indispensable, you have to do this. As I said in physics, you cannot simulate the motion of individual molecules in a box full of air or whatever gas. But you can still write PV equals NRT. You know that collectively, if you heat it up, the average velocity of the molecules will accelerate and that will increase the pressure proportionally. So the pressure increases proportionally to the temperature and vice versa, which is what this law gives you. You ignore all the details about the real state. So state is really not a single definition. It's a hierarchy and you choose the hierarchical level as a function of which type of prediction you want to make.
Audience Member 1:32:36 ↗
Yes. So if I understand, world models can capture causality and correlation, not only correlation?
Yann LeCun 1:32:46 ↗
Right. So if you have observation, I mean if you have state, action or intervention, and next state, then you have a causal model. Right. Now causal inference is a little more sophisticated than this because causal inference would be you observe the previous state and the next state but you don't observe the action. So you don't know the cause of what you observed. Now it turns out there's a slide that I skipped where we have a technique to sort of infer latent actions that might explain how the world evolves due to an action you're not observing. To some extent you can observe it. It's possible. But it's not simple. And even humans are not particularly good at this. We attribute, a lot of humans over the course of humanity have attributed hidden causes to phenomena like thunder. You explain this with Jupiter or Zeus or whatever. Or whatever thing happens to you, you have to have an explanation for, a causal explanation for why it's happening to you. And so I'm going to say something slightly offensive, but if humans were so good at causal inference, religion would not exist.
Audience Member 1:34:03 ↗
Thank you very much, Professor. I am really happy to follow your talk. Thank you very much.
Yann LeCun 1:34:10 ↗
Thank you. Thank you.
Audience Member 1:34:11 ↗
Excuse me. I have a short related question to this possible...

32 more exchanges in this transcript

Sign in free to read the rest of this interview. No card required.

Sign in to read the full transcript

Cite this transcript

APA, MLA, BibTeX
APA

LeCun, Y. (2026, April 15). Yann LeCun: Special Lecture on AI and World Models [Interview transcript]. Al-Khwarizmi Applied Mathematics Webinar. CEOInterviews.AI. https://ceointerviews.ai/interview/832741/

MLA

Yann LeCun. "Yann LeCun: Special Lecture on AI and World Models." Al-Khwarizmi Applied Mathematics Webinar, 15 Apr. 2026. Transcript, CEOInterviews.AI, https://ceointerviews.ai/interview/832741/.

BibTeX
@misc{lecun2026_832741,
  author       = {Yann LeCun},
  title        = {Yann LeCun: Special Lecture on AI and World Models},
  howpublished = {Interview transcript, Al-Khwarizmi Applied Mathematics Webinar. CEOInterviews.AI},
  year         = {2026},
  month        = {apr},
  url          = {https://ceointerviews.ai/interview/832741/},
  note         = {Speaker-attributed transcript with timestamps}
}