Executive Officer, Tesla, Inc.
A Peek into Tesla’s Autonomous Future: Core Tech Revealed by VP Ashok Elluswamy at ICCV25 WDFM-AD
👉 Checkout our blog for the highlights of the talk: / a-peek-into-teslas-autonomous-future-core-...
At the ICCV 2025 WDFM-AD Workshop (https://wdfm-ad.github.io/iccv25/) in Honolulu, Ashok Elluswamy, VP of AI at Tesla, delivered a keynote titled “Building an Autonomous Future.”
For the first time, Tesla offered a structured, technical breakdown of its end-to-end foundation model stack—the AI backbone powering Full Self-Driving (FSD) and the Robotaxi.
About Ashok Elluswamy
Ashok Elluswamy, Vice President of AI at Tesla, has described the company’s mission as “producing amazing abundance for the entire world” through autonomous vehicles and humanoid robots. In early 2026, he stated that Tesla’s robotaxi service is operating in Austin, Texas, with no safety driver, and that the company aims to scale the service to the entire fleet. Elluswamy said that Tesla’s Full Self-Driving (FSD) system is an end-to-end neural network that uses only cameras and no radar, lidar, or HD maps, and that the same foundational model is being adapted for the Optimus humanoid robot. He predicted that by 2035 all cars manufactured will be self-driving and that humanoid robots will be able to interact in natural language. Elluswamy has emphasized that Tesla’s approach relies on large-scale data and compute, including the custom Dojo training hardware, to achieve safety and smooth driving. He stated that the company’s end-to-end system can be probed for reasoning traces and 3D understanding, and that it has enabled features such as emergency braking for crossing vehicles. Elluswamy described Tesla as “all in on robotics” and said the company is focused on producing intelligent robots for physical work, with the goal of creating a car that “doesn’t need to crash ever.”
Transcript (37 segments)
A
Ashok Elluswamy0:02
All right, I guess let's get started. My name is Ashok, I lead the Tesla AI team here at Tesla. I'm so excited to be here. I couldn't be in person, it's a bunch of other company work here, but nonetheless excited to share our recent progress. This is the work of the Tesla AI team over the last several months, so I'm talking on their behalf here.
I'd like to start with some of the recent accomplishments that the team has achieved. Earlier this year, around June July, we launched our robotaxi service where if someone was in Austin or in the SF Bay area, they can hail a robotaxi service from Tesla. And then in Austin, below 40 miles per hour, you can get a car without anyone inside the passenger seat. It is just driving using cameras and neural networks and adapting to real world conditions.
We also did this thing where we delivered the first customer purchasable vehicle from the Tesla factory in Austin directly to a customer's home somewhere else in Austin. We covered about 20-30 minutes of driving across highway, city roads, parking lots, etc. to deliver the car. You can see the car driving itself. And this is a production vehicle out of the factory using production cameras, the same computer that's in every car and so on.
Every Tesla that's manufactured in the US also delivers itself from the manufacturing line all the way to the loading docks that's like a couple miles away. So this year we made a lot of strides in autonomy and we continue to scale the technology to be even more robust and resilient so that we can go to even more locations, have no one inside the car and so on. Today I'd like to touch on some of the technologies that help power this self-driving.
So first of all, we switched to having a single large end-to-end neural network that can take in pixels and other sensor data as an input and then just produce the next action as an output. This is what's driving the car. It no longer is doing explicit perception of like vehicles, road boundaries or things like those. They can be implicit and they can be trained as auxiliary things, but then the entire thing, the raw video streams go in and then the actual actions to take are produced by the neural network. And this has been so for the last few years and this has been extremely good.
The reason being, we used to work on an explicit modeler approach because it was so much easier to debug and so on. But what we found out was that codifying human values is really difficult. For example, in order to brake for an obstacle or go around, you could go as fast as possible and then brake a bit later, or brake earlier for more smooth driving. And this really comes down to alignment of preference to human values as opposed to there's one objective value of like how much braking and when to apply. You don't want to brake too early either because then it'd be too slow, and you don't want to brake too late either then it's too uncomfortable. And this varies from situation to situation. The amount at low speeds versus high speeds is very different. So it's really hard to codify what humans actually want and you can't write it in code, at least not very well.
And also the interface between traditional perception and planning is very ill-defined. It can be very lossy and lose critical information where you might need to propagate the uncertainties through the entire system. That's really hard. Going to an end-to-end neural network also allows for your compute to be homogeneous. Because if you have traditional branching compute you can have non-deterministic amounts of latencies and in a real-time system you don't want to have the kind of latencies, you want to have predictable deterministic latencies. And such a single neural network doing the entire thing offers that. And overall this is what we believe as the right path to solving robotics as opposed to this kind of modeler brittle systems that were developed earlier.
Here I have to give a few examples of why this is hard. For example, this vehicle is driving on a bi-directional road and then to avoid this puddle you got to go to the oncoming side. If you have to write this using explicit cost functions, it's going to be really hard because typically you don't want to go into the oncoming side. But then what is the pain of driving through a puddle versus the risk of going through an oncoming lane is hard to write in explicit code. But here obviously if you look at the scene it's pretty obvious that you can go here. You have a lot of visibility, there's no oncoming vehicle coming and you should definitely go into the oncoming side.
It's also hard to understand intent. For example here on the left side there's a bunch of chickens crossing the road. I'm not sure if the video is clear for you guys but there's a bunch of chickens crossing the road. This is driving on self-driving software and it's just waiting for the last chicken to cross here. Especially waiting for the chicken to cross the road and then the car proceeds on its own. This is all just driving on its own. In this case, the car was smart enough to understand that the chicken was actually crossing the road and was patient enough even though there was no collision risk. It was just the right thing to do.
Yeah. In this case, the model understands that the geese are actually not crossing, they're just stationary and decided to reverse and then go around them. So this is something it's very hard to write in explicit code what is the right thing to do here. But looking at the scene end-to-end and for humans too it's pretty obvious what was the right thing and that is why end-to-end is way better than a modeler approach.
That said, it's not that easy to develop such a system, it's pretty difficult in fact. And I'd like to touch on three different aspects of it that make it really difficult. First is the curse of dimensionality. What I mean by that is that the input context length is quite enormous. Especially our cars have seven or eight cameras depending on the vehicle. Each produces 5 megapixel camera streams running at a high frame rate. And even if you want to have like 30-second context or something like that along with the other necessary inputs like the routes and then the vehicle speed and other kinematic data, you can easily be more than billions of tokens, like two billion tokens for a 30-second context window. And if you naively just use some kind of ViT giant transformer, it's just going to be a really difficult task of mapping two billion tokens down to roughly two tokens which is what is the next steering and acceleration that the car should take.
And thankfully Tesla has a huge advantage here, which is the Tesla fleet is quite large so it basically has access to the Niagara Falls of data and it's up to Tesla to use this data to solve this problem. You don't want obviously spurious correlations from the two billion tokens down to the two tokens. You want the right correlations as to why those two output tokens must be the right tokens. So what we do is we refine this 500 years of driving which obviously is more data than we could ever store on our clusters to the essential amount of data which covers the overall spectrum of driving. There's different ways we can collect this data based on explicit triggers or small neural networks targeting specific scenarios, or like constantly evaluating what the model predicted versus what actually happened and things like those. We can use such tricks to extract just the right amount of data and the right quality and the kind of data that we need for our training.
Here are some examples of what that kind of corner case data might look like. These are examples in manual driving, people are just still driving and then once every now and then they encounter such rare scenarios and we can have triggers that catch these. And then this is very very useful data to train on and this is not normally accessible. You can't stage this that easily because it requires the stationary scene, the speed of all the vehicles in state space, this is really hard to achieve. And Tesla has a unique advantage here that it can tap into the entire fleet to get this data.
Once we get this data, you can use this to train our end-to-end model like I mentioned before. And it just generalizes to extreme scenarios like these ones. And not just reacting at the last second but proactively safe. I would like to show a demonstration of how that manifests. This is again a self-driving system in operation here just on a highway. I'll let you watch the video for a second.
Yeah, here the car in front of us spun out and then hit the barrier and the self-driving system pulled over safely. But what is quite interesting is that if you watch the video again, like pause where the ego car brakes, if you notice this car here it's spinning out of control and it's going to hit the barrier and it's going to bounce back into our lane. It requires so much intelligence to know that this is not a first order collision here because at this frame the Tesla already determined that this vehicle, there's something wrong here, and started applying the brakes. It did not wait for the car to hit the barrier and then bounce back for its velocity to change or something like that. This is a second order effect that it needs to model and it could have also attributed this to a lane change of this vehicle but it did not. It understood because it was requesting roughly 4 m/s squared of braking which is not a light amount of braking. And this is only possible if you have a ton of data and cover all these corner cases. And that is how you can provide both safe and also a smooth ride because a less intelligent system would wait until the direction has changed or actually some bad event has happened but it requires a lot of intelligence to understand that something is going wrong way before it actually goes wrong. And that is what you get with an end-to-end system with a ton of data.
Secondly, so it's an end-to-end system. How do you guys debug? How do you actually develop this thing? If something goes wrong, how would you know? Just because it's an end-to-end system does not mean that it cannot predict anything else. The same model can be prompted to predict arbitrary things including 3D occupancy, other objects, traffic lights, traffic signs, the road boundaries. Even just in plain language, you can ask it why it made some decisions, does it understand the scene and so on. All of this helps interpret the model's understanding and also sort of gives some guarantees around the safety of the entire system.
So in practice it sort of looks like this where you can take in arbitrary sensor data and prompt it to produce arbitrary things but in the end the only thing that actually matters in the car is the control actions that it produces. Everything else is auxiliary but they can be quite helpful for predicting the correct control actions.
One specific task that I would like to go over is the Gaussian splatting which in the last few years has been quite prominent in the field. What I'm showing here on the left side is the traditional Gaussian splatting. I know it looks really bad and it's not like taking some bad example and showing the problem. Typically vehicles drive in sort of a linear fashion going forward. And there's not a ton of baseline. And because of this, if you just took the same camera views of the vehicle's motion and ran traditional Gaussian splatting, you know, NeRF Studio or what have you, with the same camera views, the views close to the train views look great. But then if you go to novel views that are far away from the train views, it breaks down quite a bit.
But what we have in the middle column here is Tesla's variant of Gaussian splatting where the same model can produce Gaussians. It generally is much better with the same limited camera views that the left side has. It can also produce semantics as shown in the third column. And the cool thing is that it can run ridiculously fast compared to traditional Gaussian splatting which can take tens of minutes. It also requires steps like COLMAP or something else to produce the camera poses. It requires initializations from these kind of explicit solvers to do a good job.
So you can see that this can update to scenes quite rapidly. If you use the same traditional splatting a lot of novel views might look blurry or like fuzzed out but as I'm rotating here in 3D space a lot of the structure remains intact. And such interpretable representations can be used to debug the system, you know like if it's going faster or slower you can easily observe okay is it safely avoiding some obstacle or not.
Like I mentioned earlier, we can also use natural language to interact with the model. The same model can sort of point to things, explain why it made some decision. We shouldn't need all of this to drive the car in real time. But then if you do need it, you can always think longer and produce reasoning tokens to then produce the right action that is consistent with the reasoning of the entire thing. And again it can be the same model that's running in the car but based on the situation can use it or can directly produce the control actions if it's not required to reason in such detail because if you keep reasoning in detail for every single thing then it's going to take too much latency. But then wherever is needed it could reason longer to produce the right answer.
The last piece that I would like to touch on is the evaluation which actually is the most difficult of these three problems I mentioned. If you just train a bunch of data from drivers and then train models and just like scale it, your open-loop performance might look amazing but then it may not translate well to how the car actually drives. There's a lot of reasons for this. I don't want to get into the details there but then it is a very important problem that requires a lot of attention to get good performance out of the system. It also requires thorough evaluation sets. It's not just if you randomly sample data from your fleet, most of it is going to be boring highway driving, you don't want to just be evaluating there. That's where the early data engine approach that I mentioned comes in handy where we can build a balanced, fully covering eval set which is extremely important but very tedious work.
One great thing we figured out was that we can use the cheap to collect state-action pairs inverted and then basically build a world simulator where given past state and then actions it can synthesize newer states conditioned on the actions. This is quite easy to collect because you can get it for free. You don't need optimal driving or something like that. Any kind of driving is good enough for this kind of simulator because it needs to simulate the edge cases. Then once you have that, you can connect that with our policy neural network that's driving the car. And then they can both run in sequence in a loop to sort of simulate the world.
So here's an example of our rollout through a learned neural network simulator. I'll let you watch the video for a second here, but the entire thing is generated video. There's eight cameras. The top row is all the front cameras of the car driving forward. The middle ones are the left and the right side cameras and the bottom row is the rear-facing cameras.
All eight cameras are generated simultaneously by a single neural network and takes action as an input. So you can steer the network. I'll show you in the next couple of slides how that can work. But yeah, I'm usually blown away by the consistency of generation across different cameras. Even the vehicle rims are consistent, the traffic lights. And this is over like a minute, minute and a half long generation of eight 5-megapixel video streams.
In addition, you can take past issues, you know, like take a year-old issue and then you can rerun the latest neural networks to see how they would perform. On the left side here was the original failure where it was maybe a bit too close to the pedestrian. And then on the right side we are evaluating a newer neural network that's driving the car. And you can see that the policy starts offsetting way earlier as soon as the pedestrian emerges from behind the car. And it's super useful for such kind of evaluation because you don't want to just re-evaluate on newer mileage. You just want to, if you have a database of older issues, you just want to replay them and verify that you are doing a good job on these past issues.
You can also synthetically create new issues. In this case the original video on the left side, that vehicle was just going forward on its own lane. But then on the right side we can make the vehicle cut across our path and then inject this kind of adversarial event to test the system's corner cases. And you can see that the rest of the scene remains consistent. All the other vehicles are moving the same way just that this one vehicle cuts across and that's how you can make lots of synthetic data sets to verify the corner case performance of a self-driving system.
If you are fine with reducing the test-time compute a little bit, you can get close to real-time rendering performance. This is the same model that's rendering the eight cameras with lower test-time compute and you can see that you can actually drive this in real time. The FPS is high enough even though it's generating eight 5-megapixel streams in parallel.
So obviously you can see that it's responding to the user's driving commands. They can steer and brake and then sort of navigate the world just as if they're driving in the real world. But the entire thing is neural network generated video streams.
I think the driver is trying to do something adversarial by going on the curb and then just demonstrating that the generalization of the simulator is quite good.
Yeah, this entire video is 6 minutes long and you can just have consistent video generation for a very long time and responding to control commands. You can hence imagine how this tool can be quite powerful both for evaluation purposes but also for closed-loop reinforcement learning where you can just let the car drive and then verify that it doesn't collide with anything for a very long time.
So yeah, what's next? We want to scale the service to be much much better and then unlock the entire fleet of Tesla vehicles to be fully autonomous. This is the Cybercab which is our next-gen vehicle which only has like two seats and designed for the purpose of robotaxi. It's going to have the lowest cost of transportation across even public transportation. And it's all powered by the same neural networks that you saw earlier.
We find that the approach that we took is quite scalable. It scales across different vehicle platforms, different spatial locations, different weather conditions and overall offers a very safe, comfortable and fast ride to the users.
There's one more thing which is it's not just scaling self-driving to all the vehicles but also scaling it to other forms of robots. The humanoid robots too. Tesla builds humanoid robots, we call them Optimus. And the same technology that we developed here for self-driving transfers most seamlessly to other forms of robots too. I could show one example here, the video generation also works for Optimus. So this is Optimus navigating around the Tesla factory and all of these are generated videos and you can see that they are all quite consistent.
You can also just like how we did for the car, action-condition it. So here the top left is the robot just going straight. That was the ground truth. Then you can have different actions like going left or right or some other direction and then it can correctly generate the pixels for the action. And it's the same neural network with just some more data added from Optimus, it generalizes to other robot form factors too.
So yeah, in conclusion, Tesla is all in on robotics. The entire company is just focused on producing intelligent, useful, large-scale robots for helping everyone in the world. And if you are working on AI, robotics, computer vision etc., I'd highly encourage working at Tesla because it has extreme real-world impact. It's fully in on solving robotics. The team is super pumped. I think it's a great place and this is the best place to work on AI right now.
U
Unknown24:25
Thank you, Ashok.