Tesla - Ashok Elluswamy: Building Foundational Models for Robotics at Tesla

Hey everyone, thanks for the invite. I'm happy to be here talking about the model that we are building. Uh, Tesla this green water.

Yeah, just kind of see this maybe and now slides.

Uh, it's not a great presentation technical difficulty. So we start. My name is Ashok Swami. I lead the AI team at Tesla. We are building foundational models for robotics. The mission of Tesla is amazing abundance and we want to have this happen through the creation of general purpose robots. What we mean by that is creating robots that can operate autonomously in a wide range of embodiment. On the left you can see our Tesla vehicles driving autonomously without anyone inside the car. We have robot taxi service that people can publicly use. On the right side you can see Optimus the humanoid robot of Tesla. This is our generation 2.5. We are working on the next generations. And on the bottom right you can see digital Optimus which is the digital form of Optimus as in it's just like a human can use a computer by just looking at the screen. You can use a computer by looking at the screen and taking actions such as commanding the mouse and the keyboard. It's kind of like a digital bot. So we see all of these things as just different forms of robots that take in high dimensional sensor data and then produce low dimensional actions out and they all can have long tail cases. They can all be safety critical. So the model has to be quite smart to work in the real world.

Today I focus a bit more on the self-driving stack but then a lot of this also transfers to the other forms of robots that I talked about earlier. The unsupervised robot taxi service from Tesla is already operating in three cities across US in Austin, Houston, Dallas. Here's an example in Houston. This is a random video from X where an unsupervised robot taxi passenger was shooting video of it currently stopping for a school bus here. He's waiting for all the kids to get out and then proceeds on its own and stuff. And this is operating autonomously. And anyone in the cities can try it out. In addition, we also ship self-driving software to 1.3 million vehicles globally. It's available across US for quite a bit now. Canada, Mexico, etc. Recently we started shipping in Europe. We were able to convince regulatory authorities that it is extremely safe and with that we have started expanding in Netherlands, Estonia, Lithuania and a lot more countries to come soon as we work with the respective governments to get approval. It's also shipped to part of China, South Korea, Australia and basically what our goal is to bring self-driving for the entire world in a scalable and affordable fashion for everyone in the world. You can see some examples of the same stack generalizing. Here's the self-driving software driving in Netherlands. I mean you can see that the conditions are not that easy. You have to go around this construction zone. It's a bridge here. You know, there's a pretty narrow bridge and you want to make sure that your speed is appropriate. For example, here goes up the bridge, a bicycle is coming on the oncoming side. It immediately recognizes it, moves to the right side and then proceeds. So it's able to handle a lot of tricky cases across a wide range of geographies. I especially love our Chinese customers. They really put us through the difficult domains here. Yeah. It's not as there's a lot of room for maneuvering here. But the car is able to handle it and there's a dead end but I think you fall off a cliff if you go farther. There's some sheep that want to cross. So the car backs up to them. You can drive in dense fog. No problem.

Yeah. Yeah, I just keep watching these videos for relaxation. Skip ahead a little bit. Yeah, here there's an oncoming car. There's no space to pass. The system recognizes it, backs up on its own, and then it keeps backing up for quite a while to find a spot where either car can pass. I'm going to keep playing the video until they pass. But reversing for a very long time on a tricky mountain road can be tricky for humans. Eventually there's a gap, it recognizes that it can pass them and proceed here and it tries to make a very sharp turn, like back up on its own and then makes progress. So as you can see this stack can be extremely general and the interesting fact is that when we shipped this initial software we had literally zero China data. This is pure generalization from the model. So what we're building here is basically understanding the essence of driving. It's not just obstacle avoidance in a traditional sense but what is driving mean truly and capture it and put it in a box and ship it to cars. And you can see this quantitatively also. Over 10 billion miles of driving, when you drive on FSD you are 2x safer than driving on manual even with active safety that protects. The reason why this is possible is because FSD proactively does the right thing. Humans, you know, if you look at the long tail of humans, they are dangerous drivers. They do horrible things for no good reason, maybe for fun. But in FSD it won't put you in such a situation. So even though this looks very impressive, this is actually somewhat easy to achieve because the computer is never distracted, it always focuses, it doesn't argue with its spouse, what have you. So it's very easy to be safe when a computer is driving the car. And you see this not just on highways or something. You can see it on city roads and highways. It's unequivocally better than humans driving the car. If you have a Tesla, you should definitely use FSD. If you don't have a Tesla, you should definitely buy a Tesla. That's the right thing to do for your health and safety. Also good for the company, but that's not why I'm saying this.

Okay, so it's all impressive. How does this look? Basically we have a giant neural network that takes in all the raw sensor data, camera videos, maps and navigation. It can also take in custom prompts, the kinematics of the robot, audio, like whatever sensor data. Just like for the robot, the stacked information, just like high dimensional sensor data, throw them all into this one giant neural network pot, mix it up with a lot of data, and it produces the next action. All of this runs on Tesla's own computer that we have built. We have this series of chips, A3, A4 and so on. A4 is the one that's widely available in all the cars. And there are two of them on every car. The reason for two of them is because they have redundant sensor data input and redundant power supplies. Communication everything is redundant. So in case of even long tail hardware failures, if one of the SOCs goes down, then the other SOC can take over and drive the car safely. So we have a copy of neural networks on each of them and they sort of cross check each other and when both are functioning we use both of them to provide higher levels of intelligence. So then you might ask, why should we do end to end driving? Every other autonomous driving company seems to be doing this modular approach. Why are you taking this sort of insane approach to solving this problem? And I think I already sort of answered this question because the first two videos that I showed sort of demonstrate the extreme generalization that can come out of us. If we had a modular approach, then I think the system performance, even though it sounds nice on paper and it's easier to debug for developers, the overall ceiling is quite low. Some of the reasons why this is the case: first of all, it's very hard to code what humans want. You might think okay they want smooth driving, but even something like comfort or something like that is extremely subjective depending on the situation, depending on the person. There's a lot of nuance that goes into it. The interface between perception, prediction, planning, etc. Again, for the developer it offers nice boundaries, but for the system it's better to let the information flow continuously until the last decision is made. You don't want to have artificial bottlenecks. All of these are tightly coupled systems. Errors in one thing should be understood by the other, and it's very hard to put probability distributions over this even though we can technically do it, but I don't think in practice it works very well. Overall, with a system like this, your latency is deterministic. You can deal well with long tail situations, works well with data, and looking back, this would be the correct bet to make.

Couple examples of why this is the case. Again, here's a giant puddle. You know, if you have some modular or rule-based system, you might say that okay, shouldn't go into the oncoming lane usually. But in this case, there is no great visibility. You can see very far. There's no oncoming vehicle coming. So the value judgment between going over this puddle versus going into the oncoming lane, the going over the oncoming sorry, oncoming vehicle wins over going into the puddle. And basically self-driving is filled with these tiny problems. People sort of joke that oh yeah, this problem is not real, this is some made-up professor thing, it's not real in practical life. But actually the opposite is true. Self-driving and I think in general robotics is filled with tiny problems where you have to make value judgment calls between different options you can take, and it's very hard to explicitly code those values. Here's another example of that. On the left side here, you'll see that a bunch of chickens want to cross the road. The car recognizes that the chickens want to cross the road and waits for them. So, patiently waiting. This chicken seem to have crossed, but then there's one straggler chicken that's like, you know, maybe lazy or something. It's like taking its time. The car understood the intent of this bird and waited for it and then proceeded on its own when the chickens had crossed the road. On the other side you have a bunch of geese just idling on the road there, chilling in the puddle. And in this case there's no one inside the car. This is our smart summon feature and the car again recognizes this and then just backs up and goes around. And again this is emerging behavior from just training on a lot of data. Imagine you being a developer and you have an ontology for what does the bird want to do and trying to code that up. It's basically impossible and people shouldn't be doing it. It's torture. So yeah, so all sounds great. Then why doesn't everyone do it? There's a bunch of challenges. I'll go into why.

First is the curse of dimensionality. As I mentioned, we throw in all of the high dimensional sensor data into the system. This is an illustration. This is not the exact configuration but you imagine if you add seven cameras to get 360 vision, reasonably high frame rate, 5 megapixels, and just assume 30 seconds of history is sufficient to drive, a bunch of other inputs, and in the end you have to produce roughly two tokens of output which is what is the next steering and what is the next acceleration. That is the only two outputs. So you have to compress down roughly two billion tokens of input down to two tokens. And this would be a trivial problem if you could just take any two bytes, isn't it? Or something. But that's not the case. You want to learn the correct causal mapping of these two billion tokens to two tokens. It's like you don't want to be deluded that it's the tree branches that are shaking that's the reason why I should brake, or you know, we shouldn't form incorrect correlations. We want to form the correct correlation, in fact the correct causal reason for why the vehicle must take those two actions and not any other action. And it's an incredibly challenging problem because of the dimensionality of the data. You can compare to state-of-the-art language models. They have context roughly, I don't know, a million, couple million context length, going to two billion context maybe, and running them on consumer hardware essentially is going to be extremely challenging. This is not running on a data center. This is running in the car live. So you don't have a lot of compute. You don't have a lot of power on these robots. So you want to be mindful of the compute and the power that you're using. And that is what makes this problem extremely challenging.

There's a few ways we can solve this. One is that Tesla has this huge advantage where we can get the data from the entire fleet and use that to train these neural networks. There's a lot of ways how we can get interesting data. I don't want to bore you with all the details but if you do this correctly you can mine a lot of this kind of corner case data that you won't find normally. And it's hard to just practice this kind of situations because it's sort of commentary in state space. So you want to find these natural distribution of states in the wild and sort of keep growing your database of these long-tail situations and keep evaluating across them. And if you do all of this correctly you'll get extreme proactive safety. Here's an example where the vehicle in front of us is driving in some slippery conditions and spins out and hits a barrier. And I think what is interesting is that the self-driving software first of all handled it on its own. There was no intervention here. It's still driving and handling it. But the actual interesting part is when does it start responding to this? Let me pause the video and sort of scrub it myself. Right. The vehicle is starting to brake a little bit perhaps. And then like here it's starting to lose control. The lead vehicle is starting to lose control. And in this frame you can notice that the velocity plot here, the ego vehicle, the Tesla, is already starting to slow down for this. And this is, you know, you would think this is an ordinary situation. It requires extreme intelligence to know that this vehicle is spinning out of control and it's going to crash into the barrier and come into our way and hence we must slow down. So this is not simple collision avoidance behavior because you predict that this vehicle is moving away from your path, not towards your path. So that's why these things are hard to code and can only emerge if we have a lot of these training samples. Another example of this, this is from our unsupervised robot taxi fleet. No one inside the car again as you can see. These bicyclists are having fun and suddenly they fall into the path and the car autonomously swerves around them and then safely comes back into the lane and proceeds. And they are fine if you look at the backup camera. So these kind of things again emerge because of this large data advantage and training on this large data in a single large model.

You might wonder, okay, if you have an end to end system, how do you know that it's always going to work? These examples are great, but what is the proof? How are you going to validate this? Just because it's end to end doesn't mean that there's nothing else that you can use to introspect the system. You can predict a lot more things to understand how the system performs, including a lot of the classical things like vehicle detection, traffic lights, road boundaries, and natural language can be used also to prompt these ones. And so it sort of looks like this. It is still end to end, just there's a lot of other outputs that it also produces that can be used to debug the system and understand what it's thinking. And you can sort of think of these as reasoning traces that help produce the correct action. And if the action is incorrect, then you can trace it down to what in the reasoning failed to understand the system. One form of reasoning is 3D reasoning. So here on the left side is the traditional Gaussian splatting. But if you took videos from the vehicle's driven path and then build a Gaussian splat and pan around, initially it might look nice, but then as you go into novel views, the generalization sort of breaks down and looks distorted and not very nice. Whereas Tesla's generative Gaussian pattern where we can have feed forward models produce these Gaussians, they are extremely good. They are geometrically correct. They are very fast. There's no explicit initialization required. Overall it's much nicer to work with. And these are all again produced purely from the camera views of the car. So in the top row you see the video camera views from the car and then the bottom is the reconstruction of the scene. And you can pause any time and then go into the 3D view, spin around, and go into extreme detail as to the reconstruction. And you can imagine that this sort of reasoning and spatial understanding would be very helpful in making right decisions. When the car has to park here and if it got too close to this white car, you can debug, okay, didn't understand the geometry correctly, so that in case it takes the wrong action, then you can fix it up in reinforcement learning or something. You can also use natural language. Here for example, the road is blocked and the model interactively reasons about this entire scene and then concludes that it cannot proceed straight and needs to change its navigation route to go in a different direction. And again, all of this is helpful to take the correct decision. Because if the model did not understand this and started to go around these cones and go straight, that way might be very bad. But then if you have the reasoning trace, then you can check, did it understand that this road is actually blocked? That's how you can make guarantees around the model's performance in long-tail situations.

The last and the most difficult problem in self-driving is evaluation. Because in self-driving, as everyone probably knows at this point, it's a long-tail problem. You can drive for one year be perfectly safe and then have one issue and that might stop a program. So it's extremely important to validate the system's performance in the long term. And that is the reason why we want this end to end model, because now you can add all this data up front and then make sure the model has enough capacity to sort of push on all of this simultaneously and not have this whack-a-mole problem where if you fix one thing, something else breaks, and you go back and fix that. And even if you do, you need to have good evaluation to catch this and fix it up before it goes into production. In order to do this evaluation, we have built this world simulation neural network. And fortunately, it's the same dataset that's required to build this as the policy neural network. So you just invert the inputs and the outputs. Previously for the policy network, you took the high dimensional state as input and produce low dimensional actions as output. But in this case, you want to take the previous state and the current actions and produce the next state, and it can be trained on the same datasets. And once you have this, you basically run them in a loop. Your policy produces new actions and based on that the world simulation produces new states, and then you just keep going back and forth and roll this out. And this is an extremely powerful and general tool that's applicable again for any robot. And this is extremely useful. Here's an example of the video that it generates. These are eight cameras from the system that I mentioned earlier, and these are all synthetically generated videos. You can see it's multi-view consistent as the vehicles are passing us on the left side. It goes from the front camera to the side camera to the rear camera, and all of this is extremely consistent and natural. It looks extremely realistic. And that is what is required to accurately reproduce real world scenes. And once you have this, you can run new policies on old examples that were failing. For example, in this case, there was a pedestrian walking their dog at night, and I think we got a bit close to the pedestrian and the previous software got close and the person intervened. And now we can take the same video clip, replay on newer driving software, and verify that now it does the correct thing. And you can see that it goes wider now. So this gives you signal that you're improving your driving policy. You can also take the original scene. So in this case, just a vanilla highway scene, and then control the objects in the scene to create novel scenes. So you can make the object on the right side cut in front of you and the rest of the scene looks the same. So in this way you can synthetically inject extremely rare situations and verify beyond the data manifold that you have and extrapolate to newer, more difficult data points.

And this not only works for self-driving, the same thing also works for the humanoid robot Optimus. Here these are all again synthetically created videos of the robot walking inside the factory. And it's also action controllable. So here you can take some video of a robot or person walking straight and then rerender them for different actions. You can turn right, turn left, and observe the consequence of your actions. And this is extremely useful because then you can validate your software. You can train on this to close the reinforcement learning loop. So it's extremely useful. So what's next? What we're building is the Cybercab. These are two-seater vehicles that are built purely for the purpose of autonomy and robot taxi services in the US and probably in the entire world. Most trips, 80% or above, 40-90% I think, are single passenger trips. And you don't want to be hauling a large vehicle and wasting all this energy driving around in large vehicles. So these vehicles are built to be extremely energy efficient yet they offer a premium point-to-point experience. So you can get in the car, it probably already knows your destination, you can smoothly and safely drive you there at an extremely affordable price, probably even lower than public transport. So again, you're getting premium experience. It's extremely smooth, safe, and affordable. And we believe it's going to be the future of transportation. You're going to be seeing these Cybercabs all over the US, all over the world, solving for all of transportation. That is our goal. And I think the stack that we described earlier is going to scale very nicely to these Cybercabs. It's going to work very well and it's starting to be manufactured this year and the ramp will continue. Yeah. So that's my talk. Tesla is all in on real world AI and robotics. We really want to deliver to the customers and make the amazing abundance vision come to life. There's still a lot of work. I showed a lot of success cases. There's still a lot of improvements that we have to make. This is a great place to join if you are a researcher or a student. We have a very small team, you have access to a lot of resources. I think it's very fun. I've been there for like 12 years at this point. Yeah, come join us. Let's make this awesome robot. Thank you.