2026 Lemley Lecture Featuring AI Pioneer Yann LeCun

Thank you so much, Michael. So, one thing you have to realize is that I've become somewhat famous or infamous for bashing a particular I'm going to go back to the title slide. bashing a particular aspect of machine learning called reinforcement learning that Michael is actually one of the pillars of and so he's saying all those nice things about me despite the fact that I've been sort of dissing a lot of the field he's been working on but in a nice way. Yeah. Right and we overlapped in New Jersey Michael and I for a few years when he was at Belcore and I was at Bell Labs they were kind of sister or brother labs I'm not sure how to say that and later when we both joined AT&T Labs we worked on AI although in different locations but in the late 90s AT&T was probably one of the strongest place anywhere to work on AI. So it's not a completely recent phenomenon that industry research on AI is I mean industry is kind of a dominant force in AI research. Okay. So this is a session where I'm supposed to give a short talk. I'm going to attempt to do this. I'm not familiar with the concept but so that we can have a discussion afterwards a fireside chat and question and answers. Okay. So first of all thank you for inviting me. Always a pleasure and I'm going to talk about world models and about the next generation AI systems and start by seeing a slide that you just saw because Michael inadvertently flipped the slide and it's that AI sucks. You know, Michael said sometimes that, you know, I have controversial opinions. That's potentially one of them. So we have AI systems that can write code. They can pass the bar exam. They can win international math olympiads, but where is my domestic robot? Where is my robot that can, you know, clean the house, learn to drive in 20 hours of practice like any teenager? We have systems that can manipulate language and they fool us into thinking they are smart because they manipulate language but in fact they are completely helpless when it comes to the physical world. They can't really handle LLMs in particular cannot really handle high-dimensional continuous data particularly if it's noisy think video or any kind of sensor data images video audio sensor inputs from you know measurements whatever financial data scientific data. And the result is that everybody these days in AI is talking about agentic systems so systems that can produce actions in the world and almost none of those systems at the moment are capable of predicting the outcome or the results the consequences of their actions. It's a very bad way to produce an action to not if you're not able to predict the consequences of it. In fact, it might be dangerous. And so I don't know for the life of me why people believe that they're going to be able to build agentic systems by just training them to imitate humans to accomplish tasks without those systems having the ability to predict the consequences of their actions in advance. And that's basically what world models are all about. The other question is why can't AI systems solve new problems they weren't shot? If you ask them to accomplish a task, they've not been explicitly trained to accomplish, they're not going to be able to do it. So all the robots you see doing amazing things today, they've been trained to imitate humans with enormous amounts of operation data or you see robots doing, you know, kung fu dances or whatever. And that's super simple because the only thing you need to know is the dynamics of the robot. You don't need to understand how they interact with the world. And that's where things become complicated. Why can't AI have common sense and understand the real world? Like not like human, like your cat. So, never mind humans. What about cats and dogs? They can do amazing things. They understand the physical world. They have some level of common sense. Certainly dogs. And you know, robots don't come close to this. As I said, 17 year olds can learn to drive in a few hours of practice. And we have millions of hours of training data of cars being driven by people. We could use that data to clone the behavior of humans trained a neural net to basically behave like a human. Yet, despite that, we don't have level five fully autonomous self-driving cars that are as reliable as humans. We do have autonomous driving cars. They're not level five. They can drive themselves pretty reliably, but they can also call home when they're stuck and their zone of operation is limited and they use all kinds of sensors and they require 15 years of very careful engineering to actually kind of do anything. I'm talking about Waymo, a big success of engineering, but hardly learning to drive in 20 hours of practice. So we're missing something big in terms of learning to get machines to learn as efficiently as humans and animals. And we keep bumping into this paradox that Hans Moravec roboticist 1988 formulated. There are things that are easy for humans and difficult for AI. And for it's they don't seem natural. The things that seem easy for AI are complicated for humans like playing chess. But things we don't even consider intelligent actions like grabbing an object or manipulating one we can't do with AI yet. The real world is messy and the people who think that by training LLMs and making them bigger and training them on more language we're going to reach human intelligence are deluded. Okay. Here's another controversial statement. Okay, the entire there's literally hundreds of billions invested in an industry that basically is counting on the fact that LLM is going to reach human level intelligence. It's complete BS. It's useful, right? I mean, it's not like those hundreds of billions are wasted, but we're not going to get human intelligence by just getting up and it's just not happening. So, how do humans and animals learn? We learn a lot by observation. Of course, quite a bit by interaction as well. And by imitation, by looking at our parents and other humans doing things. But some animals learn without actually the help of other members of their species and certainly not their parents. Think octopus. They never meet their parents. Think cuckoo birds never meet their parents either. So humans and animals learn mental models of the world and in fact cognitive psychologists have tried to measure the type of concepts that humans are young humans infants are capable of learning in the first few months of life. So concepts very basic concepts like object permanence the fact that when an object is hidden behind another one it still exists. We're not born with that. Babies learn this around the age of two to three months. What's kind of interesting is that notions like intuitive physics like conservation of momentum gravity that takes about nine months for babies to learn. So if you show a six months old scenario at the bottom of the slide here where a car is put on a cart and is pushed off the platform and appears to float in the air. A six months old baby will barely pay attention whereas a 10 months old will look at it with big eyes and really fixate because you know his or her world model is being violated. So I wrote kind of a bit of a co-wrote I should say a bit of a vision paper about this with Emmanuel Dupoux who's a cognitive scientist who actually put together that chart and Jitendra Malik who's a computer scientist and roboticist at Berkeley not Amazon. So how do we reproduce this type of learning and what is intelligence really right there's all kinds of things that are being said about what it is to really be intelligent And you would think that with the success of LLM you know LLMs are good sort of interpolating information retrieval machines if they are trained to answer a question they will answer that question appropriately. They can solve certain problems because they have some special mechanisms that are built in when a particular problem is formulated. But they're not very smart. They can accumulate a huge amount of knowledge and the reason why they require a large number of parameters and large memory is because they're based on the accumulation of factual knowledge essentially not understanding not much at least. So intelligence in my opinion is the ability to accomplish new tasks you've never been exposed to and solve new problems without any prior training or with minimal training like learning to drive in 20 hours of practice not millions of hours of examples. So fast adaptation in front of new situation is intelligence and it's not just adaptation it's also creativity and you know the sort of extensions of adaptations if you want. Now the phrase artificial general intelligence or AGI makes absolutely no sense. And the reason is because it's designed to designate human level intelligence but human level intelligence human intelligence is highly specialized. We don't really realize that because all the problems we can apprehend are problems we can apprehend. But it's you know there's no question that we are very highly specialized. Now there's no question either that machines will eventually surpass humans in all domains where humans are intelligent. That's not going to happen next year despite what Elon Musk has been saying for the last 15 years. But it's going to happen next year. And it's not going to happen in two years despite what Dario has been saying. It's not going to happen right away. At the very best we might be convinced that we're on a good path towards human level intelligence, but not yet at human intelligence within five years. But it's going to take a while and it's almost certainly much harder than we think because in the past 70 years of history of AI, it's always been much harder than we thought. There's been about four or five waves and Michael and I have been witness to this over the course of the history of AI of people who said that's it. We've discovered the secret of intelligence. Within 10 years, the most intelligent machine on the planet will be a machine. And that proved to be wrong every single time. It's going to prove to be wrong this time around as well. I wrote a little I mean co-wrote really I didn't really write it. It was mostly written by the co-authors this little piece philosophy piece about you know intelligence and really you know how we should think about human level or superhuman intelligence. Now why is it that human level AI will require real world data sensory inputs as opposed to just language or text? Okay, take LLM which is trained on all the publicly available text on the internet. That's typically something like 10 to the 13 10 to the 14 words. In the case of LLaMA 3, which was a model that my colleagues at Meta trained years ago, it was about 2 10 to the 13 words. You turn this into tokens. Token is a sub word unit. So that's 3 10 to the 13 tokens. The token is three bytes. So you get roughly 10 to the 14 bytes to train your LLM on pre-train. That would take about 400,000 years for any of us here to read through that text if we read nine hours a day at 250 words per minute. Okay, a lot of data. Now take a human child, a four-year-old trained on visual sensory data. 16,000 hours of wait time in the first four years, which by the way is a small amount of video data that would be about 30 minutes of YouTube uploads. We have two million optical nerve fibers going to the visual cortex from the retina carrying each about one byte per second. So that's about 10 to the 14 bytes in the first four years instead of 400,000 years is way more information getting to us through sensory inputs than there will ever be through language. And that tells you we're never going to get to human level AI by just training on language. It's just not going to happen despite what you might hear. Now of course you know video data is much more redundant than language but that's the point is that all the machine learning procedures that we use rely on redundancy. If you try to train a machine learning system with completely random data good luck. So what I've been working on with some of my colleagues over the last five years but really the last 15 years and recently created a company around this idea is basically enabling the next AI revolution which would be AI for the real world using what's called self-supervised learning which is self-supervised learning for high-dimensional continuous noisy data such as images video audio sensors etc. and to produce AI systems that understand any environment or system that they need to control including physical things. So they understand the physical world but also living things. Systems that have persistent memory, systems that can perform long chains of reasoning, can plan complex action sequences, possibly hierarchically, and systems that can accomplish new tasks zero shot without prior training and adapt quickly to new situations. And finally, systems that are controllable and safe. Current AI systems, LLMs, are intrinsically unsafe. It doesn't matter too much because they're not very smart. And you know we learned pretty quickly how to use them. But they can't make they can be made completely reliable. Okay so let me go into the weeds of it. There are two ways to build agentic systems. One is the way LLMs are built. You give them what's called a context or prompt right? they observe something which is a sequence of symbols in the case of LLMs and you train them to predict the next symbol. If those sequences of symbols represent you know configuration of the world and then an action they will attempt to predict the action. There's no reasoning in that. The way you trick an LLM or a token prediction system to reason is that you trick it into producing lots and lots of output tokens. And you hope that because it's going to spend more computation producing many tokens, it's going to turn into some form of reasoning. So you train it to go through steps of reasoning and that's what they call reasoning. But it's not really what reasoning is. Reasoning is a search. When we reason, we search for a solution to a problem. In fact, that's the way classical reasoning in AI has been formulated since the 1950s. I give you a problem. The problem whether a particular hypothesis can be a solution to this problem may be characterized by a function. So, let's say your problem is to find the shortest circuit of a truck, the delivery truck through cities. The traveling salesman problem is a well formulated problem. There's a well formulated set of possible hypothesis which is all the possible orders in which you can go through the cities and then the function you need to minimize is the length of the path. And so you just search for a solution. Now in 1958 Newell and Simon pioneers of AI wrote a program that they very modestly called the general problem solver. And it was basically that if you can characterize a solution to a problem and you have a search space for all the possible solution, the only thing you need to do is search through the solution for one that optimizes that function. They didn't realize yet because computer science was not invented as a field that most problems are exponential or NP complete or NP hard. So but this concept that inference is a search through a set of potential solution for a solution that minimizes a particular objective function that is intrinsically more powerful than autoregressive prediction of tokens computationally. It's intrinsically more powerful. You can reduce any computational problem to an optimization problem. You cannot reduce every computational problem to or at least not efficiently to an autoregressive token discrete token prediction problem. Okay, so we have those two categories of AI systems. My money is on the second one. Okay, the one that performs inference by search. And that plus a couple other ideas that I'm going to tell you about led me to this long-term vision of kind of a cognitive architecture whose centerpiece is a world model. So if you have a world model, what is a world model? First of all, the world model is a system predictive system that given the current state of the world or some representation abstract representation of the current state of the world and given an action that you imagine taking or intervention. Can you predict the next state of the world that will result from this action or intervention? If you have such a world model that predicts what the world is going to be after you take an action, you can use that for planning. you can feed the resulting predicted state to an objective function that measures to what extent a task has been accomplished and by optimization you can search for an action that minimizes this objective according to the prediction of your world model. You can also put guardrail objectives that are constraints that the system has to satisfy in its optimization search that guarantee that whatever actions are being taken nobody will get hurt or the outcome will be good. Okay. So a system built around this blueprint is intrinsically safe because it cannot do anything but satisfy your guardrails according to its world model. Now, of course, the guardrails might be inaccurate. The model can be inaccurate. So, I'm not saying those models are necessarily perfect, but at least by construction, they will not knowingly produce actions that will, you know, produce dangerous results, let's say. Now if you have a world model the classical way to use them in optimal control a very old engineering discipline is that you can run your world model multiple times for you know multiple actions a sequence of actions and then you can plan that sequence of actions to arrive at a particular outcome. This is called MPC model predictive control very classical except classically the model is written by hand you have you want to control an airplane you can write down the equations of the airplane the dynamics of the airplane and that's your world model okay you know the state of the airplane you know the equation you know the control variables and so you can just do that in fact NASA has been using methods of this type to plan the trajectory of rockets since they've had their hands on electronic computers which is going back to the 60s. Now really the way humans and animals perform planning is hierarchical. If I want to plan my return trip from here to New York or in this example, plan a trip from my office at NYU to Paris. I cannot possibly plan my entire trip at the lowest level of actions, which would be muscle control every millisecond. I can't plan my entire trip in terms of millisecond by millisecond muscle control. Not just because it's too long and complicated, but also because I don't have the information. I cannot plan the whole sequence of actions in advance. I don't know what the conditions are going to be when I get down on the street and have to hail a taxi. How long am I going to wait? So all planning is hierarchical. We imagine high level actions like to go to Paris, I know I have to get to the airport and catch a plane. Now, I have a sub goal, which is going to the airport. How do I go to the airport? I'm in New York, so I can take a taxi or take the subway. Let's say I decide to take a taxi. How do I take a taxi? I need to go down to the street and have a taxi because I'm in New York. How do I go down on the street? Well, I need to go to the elevator, push the button, and walk out the building. How do I go to the elevator? I need to stand up from my chair, pick up my bag, open my door, close my door, walk to the elevator, avoid the obstacles, push the button. At some point down in the hierarchy, I'm going to have all the information I need to just act because I have the information. So I can plan low-level actions, but all of the planning we do is hierarchical. Now, here is a big secret in AI. Nobody knows how to do hierarchical planning. This is an idea. It's not been tested. We're working on it but you should also work on it if you are interested in this question. Okay. So how are we going to train those world models? And the natural idea here is everybody is talking about generative models. We should train a generative architecture. All right. Let's say we want to train a system to predict what's going to happen in a video. Like you know LLMs are trained to predict what's going to happen in the text. you show it a piece of text and you ask it to predict the next words in the text, right? And there are forms of this where you take a text and you remove some of the words and you train the system to predict the words that are missing. But LLM really just predicts future words. Why not do the same with video? Okay, take a video, show a system the first half of the video, train it to predict the second half of the video. I tried to do this for 15 years. The first 10 years of those 15 were basically a complete failure. And the reason was I was trying to predict pixels. Now, as it turns out, when you're trying to predict what's happening in a video, you can, as it turns out, if you train a system to predict the next word in a text, you can do a decent job. You can never predict exactly which word will follow a sequence of words, but you can produce a probability distribution of all possible words in a dictionary. There's only a finite number. So it's just a big list of numbers between zero and one that sum to one. We know how to do that. But if you train a system to predict what's going to happen in video, there's an infinite number of plausible futures. And most of the content there, most of the information in video is completely unpredictable. So if I take a video of this room, I point the camera to the left here, I turn the camera slowly, and then I stop. And I ask the system, predict the rest of the video. It's going to predict that the camera probably is going to keep panning. There's absolutely no way it can predict what all of you look like. No way. I mean, it's impossible, right? At a pixel level, it can't predict the texture of the ground or the wood panels. It can't predict there is, you know, some sort of fireplace in the back. Like, okay, most of the information in the world is not predictable, but it's very often partially predictable. there is some abstract representation formulation of the state of the world that is predictable. So I can predict that you know it looks like a pretty full room and so most of the seats are going to be occupied. I can't predict what you look like but you know system might be able to predict there is one person in almost every seat. It's probably going to predict at some point there is a wall right even if you can't predict the details. And we do this all the time. We predict things that are essentially not predictable. But like if I take this object, I put it on my hand and I tell you I'm going to lift my finger, you can tell what's going to happen. Okay? If I repeat the experiment, it might fall in a different direction. So you cannot predict exactly at the pixel level what's going to happen. But at an abstract level, you can tell that this object is not stable and going to fall in one of the two possible directions. So what does that mean? This means the idea that you're going to train a system to predict what happens in a video at the pixel level is simply not going to work because the system is going to spend all of its resources trying to predict things it simply cannot predict. And in fact, we did that experiment. We had a bunch of papers going back 10 years where we tried to train a big neural net. I mean big for the time to predict a few frames in the future of a video and we get those blurry predictions because the system predicts the average of all the plausible futures. So my colleagues at FAIR also did this with you know simulated environments. Was complete failure. eventually figured out a way of making things a little better using latent variables to take care of this the part of the video that is not predictable. But really it was kind of a failure except for very simple videos. So about five six years ago we came up with another idea and that was derived from empirical results with attempting to use self-supervised learning for computer vision systems to understand images. And what we discovered was that the system architectures that worked the best were not generative architectures but were what we now call joint embedding architectures. Okay. So you see the generative architecture on the left. Okay. you show it an X and you ask it to predict a Y potentially conditioned on an action or something you know is happening. So X could be the beginning of video, Y could be the continuation or X could be or Y could be an image and X could be a corrupted version of that image or a different view of the same scene. Okay? And you're supposed to predict Y from X and that doesn't work very well. a bunch of teams at FAIR and other places attempted to train things like variational autoencoders or masked autoencoders to learn representations of images and the results were okay. It was super expensive computationally but it didn't work very well. What actually works is joint embedding. So what is joint embedding? You take both X and Y and you run them both through encoders and you make predictions but you make predictions in representation space. So the encoder applied to X produces a representation of X s_x which does not contain all the information about X and the encoder for Y which may be identical to the encoder for X produces a representation of Y s_y which may not contain all the information about Y and then you train a predictor to predict the representation of Y from the representation of X. This works really well. There's a lot of self-supervised learning procedures for image and video that actually use this type of architecture and they work really well. So there's a big issue though with this type of architecture and the big issue is that the system on the right, the joint embedding architecture can choose to ignore X and Y altogether and minimize the prediction error by just producing constant s_x and s_y. So you have to find a way to prevent the system from collapsing and this is the difficulty of training joint embedding architectures particularly JEPA. So what is a JEPA? It's a joint embedding predictive architecture. It's a system of the type I just mentioned. You have variable y you want to predict variable x that you observe. You run them both through encoders which may or may not be identical. And you predict the representation of y from the representation of x. You train the system by minimizing the prediction error, but you have to also ensure the system doesn't collapse. And one good way to do this is to attempt to maximize the amount of information coming out of the encoders. That sounds great except we don't know how to measure information. We certainly don't know how to measure a lower bound on information content. We have all kinds of ways to measure approximate information and they are all upper bounds. When you want to maximize something, you need a lower bound. You can push up on it and the actual thing will go up. We only have upper bounds. Okay? So, what do we do? We come up with a good upper bound and then we cross our fingers. I'm sorry to say that's all we have. And there's very deep reasons why we can only have upper bounds to information content. we're never going to find lower bounds. Okay. The reason is to measure information, you have to make some hypothesis about the type of dependencies that you allow your variables to have between them. And what that means is that you're going to ignore certain types of complex dependencies, which means that the measurement of information you're going to get is going to be overestimated compared to the actual amount of information. at least if you believe you have objective measures of information which actually I don't believe so it's even more worse than that but there are methods to do this and there is one called VICReg which I'll say a few words about and if you want to know more about VICReg so it's basically a kind of measure of information that's differentiable that you can use to maximize the information content coming out of a neural net or an encoder or any learning machine and this was at least in part invented at Brown and certainly implemented at Brown by Randall Balestriero together with me. Where are you Randall? Right here. Okay. If you have any question about this, talk to him. Okay. He's a faculty new faculty in the CS department. And so I proposed this JEPA architecture several years ago, maybe five years ago roughly. since then there's been something like 1300 papers talking about JEPA on Google Scholar. So this is not just you know Yan's sort of obscure technique now anymore. Okay so this idea that we need a system to learn abstract representations of observations to be able to make prediction is fundamental to intelligence and certainly fundamental to science. We do this all the time in science. In principle, I could explain everything that is taking place in this room here in terms of let's say quantum field theory. Any theoretical physicists here? Okay. One two I mean physicists will tell everyone here you know all of your field just applied physics to some level right because I can reduce everything to physics. But in fact we don't because it would be completely impractical to simulate our brain processes in sufficient details to predict our reactions to this talk by doing quantum field simulation. So we invent abstractions, particles, atoms, molecules in the living world, proteins, organelles, cells, organisms, individuals, societies, ecosystems. And every level of description in this hierarchy allows us to make longer-term prediction than the level below and ignores a lot of details about the level below. That's the essence of being able to understand the world. there's a wonderful quote from Albert Einstein which is the most incomprehensible thing about the world is that the world is comprehensible and it could be because there are those abstractions that we can derive that allow us to predict or explain certain behaviors of the world that would otherwise would be way too complicated for us to understand. So the basic idea of understanding the world and modeling it and be able to make prediction is finding good representations, abstract representations that ignore the details that we cannot predict. That's exactly what JEPA does. And of course that's applicable to you know AI for science and things like this. So world models should not be world simulators. They're approximations. They should not be digital twins. It's a very popular phrase at the moment. they certainly shouldn't be generating models because you don't want them to reproduce every detail of what you observe. They should not be video generation systems. A lot of people currently use the phrase world model to designate video generation system. It's not what I'm talking about. What they should be is action-conditioned predictors in abstract representation space preferably differentiable. Okay. So let's say I have a jet engine. Jet engine has typically a thousand sensors in it. Is there a way I can predict the state of the jet engine from the previous state and the intervention I'm making on it? Let's say it's a chemical plant or a power plant. Let's say it's a patient. Okay, I have a diabetes patient. Is there a way I can treat the patient with a course of treatment that will bring the patient to a good state with low blood sugar or whatever. So the point is not to predict every detail, but to predict enough that is predictable. Ignore the details you cannot predict. Same for a robot. We're not going to have useful robots until we have systems that basically can understand the physical world. And currently we don't. You see robots doing kung fu. It's super simple because the only thing that's involved is the dynamics of the robot itself and that we can write equations for. But the interaction of the robot with an object or manipulating it, we don't know how to do that. This idea of world model is really old. It goes back to the 1950s or 60s, 50s in the USSR, 60s in the US or in the west. The idea that you can use a world model to predict what's going to happen and then you can plan a sequence of actions. As I was saying, it's been used by NASA since the 1960s. I wrote a couple vision papers about where I think AI research will go over the next few years. The one at the top is a long paper relatively easy to read. that I wrote in 2022. And there's sort of various versions of this talk somewhat some slightly more technical than the present one. Yeah, let me go back to this if you want to take a picture. Okay. I'm going to use the last few minutes to talk about this VICReg idea and a model that Randall called LoRA, I have no responsibility in the name, it's kind of a running joke because my boss at Bell Labs, Larry Jackel, whom you know, called nets back when I was at Bell Labs in the late 80s, he called it LeNet. Okay? And the name stuck and now I have to carry this law everywhere. So the idea that Randall and I mostly Randall worked on is the idea that if you want to maximize the amount of information coming out of the encoder you get you pass a bunch of samples through it and you're going to get a bunch of points coming out of it and if you insist that those points form an isotropic Gaussian distribution the variables are going to be independent of each other. Okay, because a joint Gaussian which is isotropic the individual variables are independent right so if you want to maximize information you have to make the variables independent of each other and making the distribution isotropic Gaussian is a good way to do this and it turns out you can do this differentially you can figure out you can project the joint distribution over multiple axes this potentially high dimensional space and when you what you get is a bunch of scalar distributions, one-dimensional distributions and you can differentiably compute the distance between the Gaussian distribution for each of those projections and the empirical distribution and you can move the points around so that they collectively become jointly Gaussian. And it works really beautifully and there's various mathematical tricks to do this. And these are visualizations that Randall generated that show various methods turning so VICReg is the one he used at the right to kind of turn a bunch of original points and move them around so they become sort of jointly Gaussian. And more recently together with Kaiming He who's a postdoc at NYU

Lucas May S who's at MA and Dian Rand and I kind of worked on this thing called L world model. Okay, running joke, which you know we can train a world model to, which is action conditioned and use it for planning for sort of various tasks, you know simple robotics task and everything. This needs to be scaled up but we think we have kind of a good handle. I'm not going to bore you with details because I'm running out of time. There are alternative approaches to this which have so far been scaled up more than this method. One in particular called Japa or VJA or Dinov3 which is done by some of our colleagues at Meta which has lots of applications, probably the best generic image feature extraction system that in existence. But let me skip ahead a little bit, my computer agrees to, and talk about VJ. So this is a video prediction system which is not trained to predict in time but you take a sequence of frames, you suppress some information about the sequence by blocking some parts of it, not just the future but like an entire chunk, and then you train one of the JA architectures to predict the representation of the full video from the representation of the partially masked video with you know various tricks. So it's not using cigreg or any kind of regularization of this type. It's using what's called a distillation method which I'm not going to go into the details of. And you know it's not entirely satisfactory in terms of the property of this algorithm but it scales up and it works really well and what we have at the end training this system on a relatively small amount of data, well a small amount, it's like a 100 years of video which basically is a day of YouTube uploads, which is way more than any LLM has been trained on but that can be trained on like 2,000 GPUs in a few days. So it's not a gigantic. Those systems actually have learned some level of common sense and intuitive physics. You show them a video, maybe a synthetic video where something impossible occurs like someone throws a ball and the ball disappears or it turns into a cube or it stops in the air. The internal prediction error shoots through to the roof when those events occur because the system knows this just cannot possibly happen. So that's really interesting. It's the first time I see this kind of behavior and I'm sorry the video somehow is not available because the video is on the web and I'm not connected. But anyway, let me conclude. So I'm recommending, I'm making a few recommendations here which make me extremely popular among some of my colleagues in machine learning particularly in Silicon Valley. Okay. Abandon generative models in favor of those japa. Abandon probabilistic modeling. So I didn't have time to talk about this thing called energy based model which is basically the way to understand the sort of theoretical paradigm with which to understand all of this. Abandon contrastive methods in favor of regularized method. I didn't talk about this but those information maximization methods are basically a type of training for those energy based models which are called regularized. And I put the last one just for Michael, abandon reinforcement training. I mean really what I mean is not abandon it, just minimize its use because it's so damn inefficient but you have to use it, you don't have a choice right.

No.

So if I count like how much time I spent in academia and industry during my career it's about half and half. The first 12 years, 13 years were in industry at Bell Labs, AT&T labs, NEC labs for 18 months. Then in 2003 I became a professor at NYU full-time for about 10 years and then joined Facebook at the time and was basically half and half between Facebook and NYU for a dozen years and still now I'm still a professor at NYU and I'm also executive chairman of that new company. I don't have any management role because I'm terrible at that. I'm not an operational person. So it's more like you know scientific leadership and strategy but I like that interaction between the two because the two worlds are complementary. They are complementary because the motivation for the people in those two environments are different right. In academia you want intellectual impact. In some parts of industry where you have ambitious research labs like Bell Labs used to be or Fair used to be until recently you also have this ambition of having intellectual impact not just internal to the company you work for but also external. And in industry there's a different set of motivation. So it's okay when you're in academia for example to work on a technique that you know is a dead end because it might give you a paper at the next conference or in a journal and you know you're not going to pursue this but you're going to get that paper out because it's useful for other people to know that you can go that direction and maybe some other person will have the idea of pushing it forward even if you don't. In industry you don't do this if it doesn't move you forward in your kind of main path which is longer term you just don't work on it because it's a waste of time. Okay. So there's the exploratory mode of research which is very much academic. And then there is the thing where you kind of try to make progress on sort of a long-term goal. In the early in my career when I worked on you know convolutional nets and we applied it to character recognition right or handwriting recognition I had no interest in character recognition this is not what I was interested. I was interested in building intelligent machines that had to be done with machine learning because I didn't think I was smart enough to just conceive an intelligent machine right it had to build itself. And then you know what better domain than perception. If you can do perception, you can do a lot of things. And character recognition was pretty much the only task, perceptual task for which you could have data at the time. There was no USB camera or anything. And so that was a logical thing to do and it turned out to be practically useful even though there was not in the plan initially. There was really a very ambitious long-term plan. So, pick a long-term very ambitious plan and work your way back and you're going to make progress. Useful things are going to come out of that.

Well, I'm merely a professor, so I don't know anything about education. But whatever I mean I get this question pretty often either from students or sometimes from their parents. You know what should I study? Like AI systems are going to be doing everything for us, right? Maybe I don't need to go to college. That's not true. In fact the trend that we're seeing at the moment, it's very difficult to identify this particularly in the US but the trend that we've been seeing over the last decade is that there is more demand for more advanced degrees. Certainly that's true in computer science, there's more demand in industry for people with PhDs in AI in particular but not just than there was just 10, 15 years ago. It's true in the US. It's much more striking in other countries that didn't have this tradition. A lot of countries in the world where doing a PhD was seen as a complete waste of time unless your purpose was to become a professor. And this is still true in a lot of domains. But in STEM, there's a lot of demand from industry for people who have PhDs because why is that? It's because more and more the health of the economy and the growth of the economy relies on technological innovation and technological innovation relies on scientific breakthroughs and those come about by people who do research. So research is becoming more and more crucial to economic expansion. What has been one of the main factors of the economic success of the US since World War II is a very very strong research ecosystem, you know, with universities, public research. Michael Litman can tell you more about this because he was at NSF until recently, which I'm not happy to say the US is currently completely destroying, burning to the ground, which is insane. So there's more demand for more advanced degrees and regardless of whether AI is going to help us or not. What is that going to change to the way we work? All of us instead of doing a lot of low-level tasks that AI can do, we're going to be their boss, right? We're going to tell them what to do. It's like, you know, all of us even at a junior level will be the manager of a team of intelligent agents kind of doing work for us. We're just going to be managers essentially but a good manager knows the low-level technology. So that's the first thing. Second thing is so study more not less. The second thing is what should you study? Now technological progress is accelerating. It's not a new phenomenon. It's accelerating but now it's accelerating in a sense that the time constants are very short. So you're almost certainly going to have to change job during your career because a new technology is going to come in that is going to make your old job completely different and obsolete. You're going to have to learn something new. So what you should do in college and school and grad school is learn to learn or perhaps invent new things yourself. Which is what I was saying with the previous point. So learn to learn. How does that translate into what you need to study, study things that have a long shelf life and that are very fundamental? The joke I usually say, which is not a joke despite the fact that we're April 1st. If you had a choice between, let's say you're studying computer science and the choice between taking a course on mobile app programming and quantum mechanics, take quantum mechanics. You might think it's completely useless to computer scientists, but it's not true. Turns out like all the mathematics that is used for Bayesian inference is the same as statistical physics. So if you want to be able to learn quickly new things, learn the basics, learn fundamental things.

So causality, okay the bad news is that in physics causality is not a well established concept because all micro physics is time reversible. So causality does not exist in the first place but at least until you start talking about thermodynamics or collective phenomena which seem to have a time interaction. The I think an important point is there are a few, a small number of systems for which you can write down a small number of equations that are more or less solvable that will allow you to predict the behavior of the system. So you want to predict the behavior of a small molecule interacting with another molecule. You do a simulation what's called density functional theory basically models the density of electrons around a molecule and figures out like how they stick to each other. It's very expensive. You can only do it with small molecules unless you have gigantic clusters of GPUs. You can't do it to predict for example the conformation of a protein. It's just too complicated. Right? So for a protein you need to have a specific model that has been trained from data to produce the right result. And of course those models incorporate some very deep knowledge about the nature of the problem like you know AlphaFold and ESMFold have some pretty clear way of representing the conformation of the protein to be able to make the prediction. So there's a lot of knowledge that is in there but it's a very sort of high level conceptual knowledge. It's not so very very detailed to the point that you can reduce to a small number of equations. You have to train the model to make the prediction. Most complex phenomena that we observe are of that nature where you cannot go from the microscopic to the mesoscopic because the underlying processes are so complicated. For example, you might be able to write down a small number of equations that govern the behavior of a neuron in the brain, but that doesn't mean you understand the brain at the system level. Certainly not the learning mechanisms, right? So this idea that you might be able with AI systems to learn a phenomenological model of a complex system that you cannot reduce to a small number of equations in the traditional reductionist approach. I think it's very powerful. I'm sure there's a bunch of material physicists here. There's something called a magic angle. You take a monolayer sheet of graphene. Okay, single monolayer bunch of carbon atoms around hexagonal structure. You put another sheet of the same stuff on top of it. You twist it by one degree, 1.1 degree and it becomes superconductor. Why? We don't have like reductionist explanation for this. There are theories. There's a lot of people interested in developing phenomenological models of material properties that may help us discover new types of batteries or new ways to. There's some colleagues at Meta that has this really cool project where they're doing sort of DFT simulation and this kind of stuff using the supercomputers that Meta has access to to generate training data for a system to predict what is the interaction between a molecule of water with a particular material whatever it is. With the idea that if you might use this model to design a particular material that would facilitate the separation of hydrogen from oxygen in water you might have solved the energy storage problem. The reason why we can't cover a small desert with solar panels to produce electricity is that we need to store the energy somewhere. And separating hydrogen from oxygen and water is a really good way to do that. If you can make it efficient, you can only make it efficient if you have catalysis. You're a chemist. I'm not, you know, catalyst that can do this. And basically AI can help us perhaps discover some new ones. That's a really interesting problem. Another problem they're working on is you want to make displays for smart glasses so you can display information. You need optical materials with a very high refractive index. There's one that we know silicon carbide is too expensive to manufacture. So what if we can discover a new one that has a high refractive index? And they actually kind of made a few proposals for that.

So the kind of stuff we're working on at first are going to be behind the curtain. They're going to be applied to industrial processes. Situations in industry where you have tons and tons of sensor measurements and you want to optimally control a process whether it's manufacturing, a chemical process, you know refinery, a power plant or maybe some production of a drug of some kind. So that's going to be you know business to business B2B not sort of in the face of consumers. Eventually all intelligent systems that users will want to use particularly wearable assistants are going to be of the type that I'm describing and used by everyone. So there is a future in which all of us will be sort of wearing devices like smart glasses and things like that which will have AI assistants that we can talk to in various ways either by speech or through what's called EMG interfaces, electromyogram interface. So basically a bracelet that you wear that allows you to point and click and even type or handwrite and with a display and we're going to have those AI assistants with us at all time. We're going to be like, you know, a captain of industry or a politician walking around with a staff of helpers, except those helpers might eventually be smarter than us. I mean, it's already the case, right? It's clear that, you know, the staff of politicians are way smarter than they are, right? At least many politicians in many countries.

So it's called, it's pronounced AMI, Amy. Sorry. It's a French, I mean it's a global company but the headquarters are in France and so we pronounce it the French way but no it's a very good question and I think the option of being closed or open is kind of by model you can't do it halfway you have to be very open or completely closed right. You have to be, you know, OpenAI went from one to the other. Google has always been somewhere in the middle. FAIR, the research part of, you know, fundamental research part of AI research at Meta is slowly going from essentially fully open to not so open. And Anthropic has always been completely opaque right. So what we plan to do with AMI is I'm a firm believer in the idea that good ideas can come from anywhere and they come from the interactions between people working on different assumptions with different motivations in different environments. And what we've seen at FAIR, at least until recently, is that the best way to do exploratory research, including in a university environment, is to host resident PhD students and collaborate with universities. Because you know I was telling you that there are exploratory projects that may or may not have something to follow and there is sort of the main line, the mainstream of where you want to make progress. Permanent staff in an industry research lab tend to work on this mainstream. If you want to do exploratory it's very hard to do this with permanent staff because that's taking a risk for them. And so you do this with postdocs and PhD students. And so that makes you in a completely different mode. Now we have empirical data on this. In 2015, I created the Parisian branch of FAIR, FAIR Paris and shortly thereafter Google DeepMind opened a lab in Paris as well. They didn't have a lab in Paris. They have a Google lab in Paris but not DeepMind. FAIR Paris has been an enormous success for internal success for the company. A lot of things have been produced at FAIR Paris that have a huge impact on the company. One of them is LLaMA. LLaMA one, the first version of LLaMA, the open source LLM was produced by a small team of 12 people in Paris. Okay, it's not American technology. It's actually French technology paid by American dollars. And then it was picked up by the rest of the company and developed into two and three and etc. At any one time until recently FAIR Paris had 40 resident PhD students. A PhD in France is three years because you do a masters before. So FAIR Paris graduates 12 PhD students per year in AI that completely jumpstarted the Parisian ecosystem of AI. Two of the co-founders, two of the three co-founders of Mistral which is the big French AI company did their PhD in residence at FAIR and worked on LLaMA one and then left and started Mistral. And you know I can give you like a dozen companies of this type which were created or helped by people who did their PhD there. What happened was that also we were very integrated with the academic research community in France and just profited from that because of the openness. You compare that with Google DeepMind in Paris, it was not nearly as much of a success because DeepMind is much more secretive. They don't like to have students who kind of see what they're doing internally and kind of take that everywhere else afterwards. And so they could not have because of IP issues they could not have resident PhD students. As a consequence their impact on the ecosystem is much much much smaller. So you know you want to be in that mode where you progress quickly because you collaborate and that's the mode we're going to be at least at the fundamental research level. Now, of course, there's going to be part of the company of AMI that's going to be more about, you know, technology transfer and product development and that doesn't, it's not helpful if it's open.