Ashok Elluswamy: Building Foundational Models for Robotics at Tesla

Hey everyone, my name is Ashok Elluswamy. I work at Tesla. I've been there for the last 12 years and I lead the Tesla AI team. Thanks for having me today. I'd like to present about the things that the team has been working on, what is the mission of the team, and what is the upcoming roadmap of the team's work.

To start out, Tesla's mission is producing amazing abundance for the entire world. Tesla has been historically known for producing vehicles, but the more important thing is that these vehicles can drive themselves. All the vehicles have all the necessary sensors and compute to do autonomous driving. Here you can see our robotaxi service operating in Austin. There's no one inside the car that's driving the car. This is just a passenger hailing a ride and then the car just drives them around all of Austin. This is publicly available as of earlier this month. Previously we used to have safety monitors to just supervise FSD, but now that's gone and then people can just get in the car, they already have the address punched in so they can hit start trip and then the car just takes them wherever they want. And these are public roads with dense traffic and everything, and this is all driven by cameras and AI.

And the future of the company is not just vehicles. We also have working on humanoid robots that sort of go to the next level of functionality. The Tesla vehicles are a low-cost scalable solution to transport and humanoid robots are a low-cost scalable solution to automate all physical work. The reason for humanoid robots is that they're backwards compatible. You can send humanoid robots to any mission and just like how humans can go in and solve problems in the real world, it uses the same interface. So it's backwards compatible. So you don't need to build any new infrastructure to welcome these robots into our world. They are already ready to go and make a big impact.

In addition to having the robot access service, we also have deployed full self-driving to all the Tesla owners who have purchased this. And these are video clips of people who have engaged FSD getting saved by the software itself from sort of tricky situations. And here for example there's a bus coming and the vehicle automatically reverses to make way for them, which just shows a lot of intelligent behavior. And this is in every Tesla with hardware three and four.

And this is all the safety improvements also quantitatively shown. Here we are showing the miles before a collision happens in the entire vehicle fleet. And the blue chart that's much higher than the rest is actually when you use full self-driving to drive those miles. And then if you just use the traditional driving with active safety, you can get some safety improvements compared to not having any active safety. It's very clear that using self-driving to drive you around as opposed to manually driving the car definitely makes you safer over billions of miles of study. And this is not just on the highway or something like that. Even highway, off highway, in both cases it is at least 2x better than the baseline of driving manually. So if you have a Tesla you should definitely use self-driving. I personally use it all the time.

So today we are going to cover what makes the software work, what are the challenges in building such a system, and how do we evaluate the safety of this. First of all, Tesla uses an end-to-end driving system to create the self-driving software. You might think that in the modern era obviously you should have end-to-end systems, but this is not contemporarily agreed upon. For example, many competitive solutions use modular approaches that have separate perception stacks, planning, prediction stacks and so on. And Tesla has foregone all of those systems to have a single end-to-end neural network that takes in raw sensor inputs, which is predominantly just camera videos from the eight cameras in the car, but also things like the navigation instructions, kinematic states like the vehicle speed, steering etc., audio, to just come up with how the car should behave and what is the next action in terms of steering or jerk or something like that.

And the reason for this is that I guess I can go over some of the reasons. First of all, codifying everything in rules-based or having modular systems, they have very leaky abstractions. It's going to be hard to communicate every single detail, the uncertainty that the raw measurements can have to the lower level systems. And all of these are tightly coupled. You can't clearly separate the concerns that traditionally we have been taught in good software engineering practices. You want to separate the concerns, make it modular, but that's just very hard to do in real world robotics. You want the information to flow densely.

A few examples here. Typically going into the oncoming lane is quite bad, you would not want to go into the oncoming side. There could be oncoming vehicles at high speed, you don't want to go in there. But then there is a water puddle here, so you can add a separate rule that says avoid puddles. But then now you have to solve this tiny trolley problem of should I go over this puddle or should I go into the oncoming side. And it depends on not just these two conditions, it also depends on whether there's an oncoming car or not. But just because the oncoming car is not there doesn't mean you can go there because you might not have the visibility for various different reasons. So you got to holistically consider the entire scene to come up with the decisions. You can't just decide based on independent rules or independent logical elements here.

Like here's one more example. Here's this chicken that wants to cross the road on the left side here. The car actually on FSD patiently waits for the chickens to cross on the left side. And if you're paying attention, there's one last chicken. There's a straggler chicken there. It still waits for it.

Waits for all the chicken to cross and then autonomously proceeds.

Yeah, they want to test FSD. And on the right side, there's a similar scenario. There's a bunch of geese on the road. And there's no one inside the car here. This is using the smart summon software. Then the geese, they're not crossing. They're just standing there and the car just backs up and then goes around them. So you have to see these subtle cues of how the legs of these birds are moving to know that they're either intending to cross or not intending to cross and do the appropriate action. Like if you had explicit perception, prediction and so on, what are we going to do, have a chicken leg detector and then predict how the chicken's going to move and then take action. I just think it's too complicated. In an end-to-end system, all of this information can flow from the pixels directly to control.

It also has deterministic latency. For driving you have to produce actions in real time. You can't take a very long time to process regardless of how long we think the world is changing around the car. So the latency is quite important and with neural networks it's way easier to control the latency. You get a lot of determinism to take actions. It's very easy to validate in that kind of system sense. And overall it's on the right side of the bitter lesson. We're betting on scaling with the new network size, data, more training compute, more rewards and so on as opposed to hand-engineered processes.

So then okay, it sounds so great, then why doesn't everyone adopt this? Why isn't this just commonplace? Why is Tesla or a couple of other companies uniquely doing this compared to the rest of the autonomous driving industry? First I would say is the curse of dimensionality. In order to make the car, it has eight cameras and they are high resolution cameras. This is 5 megapixels and they run at pretty high frame rate and then you need a reasonable size history to make decisions. If you come to a stop sign you need to remember who came first to the stop sign, who needs to go and so on. So you need a lot of context to solve this self-driving problem and pretty much any robotics problem. I would say even this history is not necessarily sufficient. You might in theory want infinite history to solve robotics in general, but say we can get away with a 30-second history. And even then if you just compute the raw amount of bytes that's streaming into the network, it's roughly, it depends on how you tokenize it, you can think of it as two billion tokens coming into the network. And then in the end you have to produce two actions, roughly what is the steering angle for the next time step and what is the acceleration or brake for the next time step. So you're compressing down this ridiculous amount of information, like two billion tokens input and it produces two actions. This would be a trivial job if you just randomly took two bytes and piped it out, but you have to take the two correct actions. And so you have to causally understand what amongst this giant stream of bytes and bits is the reason why I should take these two correct actions that for example the human took or whatever optimized the reward. Learning that correct causal mapping is very difficult because the network can learn all kinds of spurious correlations. You don't want to learn spurious correlations, you want to learn the correct correlation that I'm braking right now because the vehicle in front of me has the turn signal on or something like that.

Luckily the Tesla fleet is quite large and you can collect a significant amount of data from the Tesla fleet. To rough order of magnitude, the entire fleet can produce 500 years of driving data every single day. But obviously most data is boring, most people are just driving on the highway minding their own business. It's not interesting. And it'd be a tremendous waste of resources to literally collect all this data and then filter afterwards. So what we do is instead just identify what is interesting data. There's a lot of work that goes into identifying what is interesting. But once you identify the interesting data, then you know, okay, now there's a school bus here and that's more interesting or there's a fire engine crossing and that's more interesting and so on. And you can source this uniquely interesting data from this ocean of mundane data. Then having a large fleet allows you to very quickly collect interesting data that's very rare for other people to collect.

Here's an example of what that could look like. Just like really hard to collect this data otherwise. You can't stage these kind of things because there's a lot of actors involved, a ton of traffic, they can be hazardous. And these kind of issues, they're happening every day but very rarely, but across the entire fleet you can definitely collect a lot of them. And if you collect a lot of this data and then train a system then you can get extremely good proactive safety and also generalization to rare events.

Here's an example where self-driving software is driving here and this person in front of us sort of skids out into the barrier here, but the self-driving software realized this quite early and then braked automatically and then avoided what could have been one more collision here. And what I like to emphasize here is how early it reacts to this stuff. You can see the velocity here of the vehicle, it starts braking already at this time step as seen by this planned velocity. The lead vehicle in front of us hasn't even crashed here. It's just going from this lane to this lane but it already realizes that something is wrong here. Something is odd. So it already starts applying a pretty significant amount of brake. People might be a little bit shook because it anticipates something bad is going to happen many seconds in advance of the actual event happening. So you need the predictive intelligence of okay the yaw rate of this other vehicle is more so than necessary for a lane change, maybe they're going to crash, what if they crash and come into our path, hence we must brake right now. And this kind of third order intelligence needs to be done in real time while processing, there could be these vehicles coming here, there's so many other bytes here, it needs to focus on the correct things to make this conclusion to hit the brakes in a confident manner. And that's what all this data is required for, not for driving around the block here.

So it's an end-to-end system, does it mean that you can't understand what it's doing? How do you debug if there's a failure? Even though it's an end-to-end system there's all kinds of probes you can attach to the system. And you can study whether it fully understands the world around it. For example, in addition to just getting the next action, it's just like reasoning where you can look at the reasoning traces and then know whether the reasoning was consistent or the reasoning was correct with respect to the action it took. And reasoning here might not be just text-based reasoning as traditional LLMs do. The reasoning here can use geometric reasoning, like where are these vehicles, where are the other obstacles, how are they going to move forward. But not all reasoning needs to be produced at test time. But at training time you can have all this reasoning attached to it and then at test time you can detach it and then only use it for either debugging or the reasoning can be implicit. You don't have to explicitly manifest the reasoning.

One form of such reasoning is 3D reasoning like I mentioned. So here for example we can use Gaussian splatting as a representation for the 3D world around the vehicle. One thing we do well is that traditional Gaussian splatting which is used to reconstruct 3D worlds usually looks very good when you're close to the training views of the Gaussian splatting, but when you are off the manifold of the training views and you deviate and go to novel views that are far away, the generalization fails very poorly. So if you took a video clip from the vehicle and you run traditional Gaussian splatting, you get the video on the leftmost side and the middle video here is Tesla's own generative Gaussian splatting which has very good generalization even beyond the training views and you can see that the semantics are quite correct on which type of vehicle it is and so on. And all of this happens way faster than traditional Gaussian splatting. Traditional Gaussian splatting can take for example 30 minutes or even more sometimes versus Tesla's own system can run on the order of hundreds of milliseconds, doesn't require any kind of initialization, it's all neural network based and this is part of the same neural network that produces the control action so it understands geometry and can produce and explain the 3D geometry around the vehicle along with the action.

Here's one more demonstration of this. It takes in the videos at the top as the network takes on the top as an input. I can entirely generate a 3D world that's interactable. You can move around and you can see what the network thinks the world around it looks like not just in the camera's point of view but in a 3D view. So this really makes it understand, this basically helps with also learning the correct correlations. It knows that these pixels in the image space or video belong to this vehicle and so on.

In addition to geometric reasoning, you can also do just text-based reasoning. For example, here there's some road closure. You can reason out in a mix of language and also pointing to things on the videos. You can say okay the detour sign here I must turn left. So it has this comprehensive understanding of the world around it. Obviously producing all of this at inference time is going to be quite challenging because you can't produce all these many tokens in a very short amount of time because the car has to produce actions in real time. But a lot of this can be implicit. You don't have to explicitly manifest it or you can do it offline during training time to just distill all of this knowledge into the neural network.

The last challenge I would say is evaluation, which I personally think is the hardest of the three challenges because self-driving is a long-tail problem. You want to verify that even the tail cases work very well. And this is a closed-loop system. So developing evaluations for closed-loop requires a good simulator or either testing in the real world. But testing in real world for all long-tail situations is extremely difficult. So what we did was we trained a world simulator neural network. This is trained on data where you have paired state and action pairs which is very easy to collect and then you just invert it, as in given video and action it produces the next state, which is given current video and current steering or axle pedal actions it produces what the next video frames is going to look like.

Then once you have this you can connect this to the policy neural network that takes the actions. So the world simulator produces the next video frame and the other sensor measurements. This is consumed by the policy neural network that then takes the next action and then they just go in round robin fashion simulating the world. And these two neural networks are trained differently. So the world neural network can use privileged information that the policy does not have access to and that is how they can be verified independently.

So here's an example of video that is generated by the neural network. The neural network produces the eight camera videos at 36 fps and 5 megapixel resolution. This is a very long generation. This video is about a minute long. But all the pixels here are generated by the neural network. And you can see how the vehicle's moving around the van and everything is consistent across different cameras as the other objects move around.

Yeah, as I mentioned, every camera, every pixel here is fully generated. There is no real pixels here, even though it looks quite realistic.

The reason for doing this again is for evaluation purposes. So what we want to do is we add some historical issues. We want to replay them on newer policy models to verify that have they solved the issue. For example on the left side here there was a historical issue where the vehicle was driving a bit close to the pedestrian and then the human driver intervened on him and then now we have a newer neural network that should have improved this and then we try to replay this on the same video clip and then verify that now have we improved on the performance compared to the original neural network. So on the right side you can see the same scenario has been reproduced, at least the important details. For example the stop sign is missing but that's not the important detail here. The important detail was a pedestrian and then the dog that they were walking and that is reproduced accurately and then now you see that the new policy deviates correctly away from the pedestrian.

You can also inject newer issues that were not originally found. So you can take any clip here, this is driving straight, and then you can modify the video to inject adversarial scenes. You can take the same scenario and then just inject different motion. This lets the simulation system be used in novel ways and then test corner cases that might not even be explored at the fleet scale.

And if you reduce the test time compute significantly, you can also just make this work in a real-time basis. Here you can see that these are all generated pixels. They're driving in a fully synthetic world using just some game engines like steering and pedal control and you can just drive around in real time and all the pixels here are generated. Obviously, it's running at real time, so it requires some reduced test time compute. But it's still pretty reasonable for driving in real time.

And all of this isn't just needed for self-driving. The end-to-end driving neural network is not just a driving neural network. It's actually a foundational neural network for robotics. The same for the simulation neural network too. Both of them are trained on common data across all of the robots. Here you can see that the same video generation network also generalizes to generating indoor scenes for Optimus to walk around. These are also controllable. Here you can take a single video clip and then convert it to be an action-based one and you can take the action to go straight or turn left or turn right and then you can see that the video generation properly reflects the actions. So what I explained earlier today is not just for the vehicle but also for humanoid robots or in fact any robots for that matter.

You also see it work for manipulation. It can, this is again action-conditioned, where it can open a drawer and then you can see that the video sort of reflects opening a drawer or picking up an object or something like that.

Okay, the benefit of using generative world models or neural network-based world models is that yeah, it can basically, everything is green, everything is great, you should definitely do it. Just you have to spend a lot of money, but otherwise it's great.

So yeah, what is the team working on? Again, like I said, bringing in amazing abundance to this world in the form of widespread self-driving cars. We're going to have Cybercabs. The vehicles that you see here, they are designed for autonomy. They don't have any steering wheel or axle pedal or brake pedals. It's meant for full self-driving only. It's coming later this year. This will have the lowest cost of transportation even beating public transport while delivering a premium point-to-point transportation experience for everyone. We'll also have Optimus robots in production. And together they should significantly reduce the cost of doing things in the real world. Thus improving productivity and value for the entire world. And if you're interested in working on such problems, please come join the team. There's a lot of interesting problems still. While it works quite well, there's still a lot of other challenges that are technically interesting and challenging and very meaningful to this world. And with that, thank you everyone.

Yeah. It's like, how did you get here today?

With a... Okay. All right. But many people drove here with their own eyes. And then, you know, obviously you can walk around this building with your own eyes. It's so obvious that you can solve this with cameras. For me at least. Like why wouldn't you solve with cameras? It's like 2026. It should be solved with cameras just like how every other human and animal lives around this world. It's just very simple sensors. The self-driving problem is thought of as a sensor problem. It's actually not a sensor problem. It's an AI problem. You need to understand the world. You need to understand what other people are going to do. And the cameras have enough information already. It's just the problem of extracting the information which is an AI problem. The sensor solution was developed back in 2008 or whatever during DARPA days when there wasn't enough intelligence around back then to extract this information. And that's why you need to depend on all these other sensors back in the day. But nowadays intelligence has grown tremendously. You can obviously extract this information. And once you have the AI, you don't need the other sensors. You just need the information which the cameras do provide.

Yeah. Cool. Yeah, we do.

Yeah, we can have a model or we can also have explicit collision checks.

Not... you can't see across the corner, right? So I think those would be like cherry on the cake. Humans are able to navigate the real world with this. We want to navigate the real world with the same infrastructure and tools that humans have. Not to say that we can't do better than them, but we shouldn't use those other things as a crutch to make the solution work. We should solve the core problem and then you can add other infrastructure to make it even better. And the team is just focused on solving the core problem right now.

That is the frequency at which we issue the control commands to the car. It means that you issue a control command roughly every 27 milliseconds, you send what the car should be doing. Yeah.

Yeah, currently Tesla vehicles do have, you know, like the Grok app, you can chat with it. We have our own audio model that's not meant for natural language interaction right now, more for understanding the noises around the car like sirens and emergency vehicles and such. But in the future yeah, we will have fully integrated voice control commands that can control the vehicle. It's just a bit too early for that. It opens up an entire area of testing that we got to do. For example, you shouldn't be able to tell the car to crash and then it shouldn't crash it. There's a lot of safety that needs to be done to protect against adversary attacks using voice. And it's just not worth the trouble right now.

Yeah, that's one of the many things. I listed like four or five things, but yeah, it's like some 3D understanding using Gaussian splats or other forms of 3D understanding.

For example, like the 3D locations of vehicles, their shapes, their future predictions, a lot of such things. Yeah.