Executive Officer, Tesla, Inc.
[CVPR'22 WAD] Keynote - Ashok Elluswamy, Tesla
Talk given at the Workshop on Autonomous Driving at CVPR 2022. 2022-06-20.
About Ashok Elluswamy
Ashok Elluswamy, Vice President of AI at Tesla, has described the company’s mission as “producing amazing abundance for the entire world” through autonomous vehicles and humanoid robots. In early 2026, he stated that Tesla’s robotaxi service is operating in Austin, Texas, with no safety driver, and that the company aims to scale the service to the entire fleet. Elluswamy said that Tesla’s Full Self-Driving (FSD) system is an end-to-end neural network that uses only cameras and no radar, lidar, or HD maps, and that the same foundational model is being adapted for the Optimus humanoid robot. He predicted that by 2035 all cars manufactured will be self-driving and that humanoid robots will be able to interact in natural language. Elluswamy has emphasized that Tesla’s approach relies on large-scale data and compute, including the custom Dojo training hardware, to achieve safety and smooth driving. He stated that the company’s end-to-end system can be probed for reasoning traces and 3D understanding, and that it has enabled features such as emergency braking for crossing vehicles. Elluswamy described Tesla as “all in on robotics” and said the company is focused on producing intelligent robots for physical work, with the goal of creating a car that “doesn’t need to crash ever.”
Transcript (49 segments)
A
Ashok Elluswamy0:05
Hey everyone, my name is Ashok Elluswamy. I work in the Autopilot team at Tesla. Today I would like to talk about some of the recent work the team has done. Before I dive into the talk, I'd like to give some of the basics of Autopilot.
For those who do not know, every Tesla that has been manufactured in the last few years has Basic Autopilot. Basic Autopilot allows the vehicle to stay in the lane, follow vehicles, slow down for curves, etc. In addition to Basic Autopilot, every Tesla also comes with standard safety features. These are features such as emergency braking and steering to prevent many types of collisions.
In addition to these basic features, about a million Teslas since 2019 have more advanced navigation on the highway. This stack would be able to perform lane changes by checking the blind spot, take exits off the highway, handle clover leaves, then merge back into highways.
And finally, around 100,000 Teslas now have the Full Self-Driving Beta software. This is our most advanced driving software yet and is capable of handling driving all the way from the parking lot through city streets into the highway. So this software, for example, would make the car be able to stop at traffic lights and stop signs, properly yield for vehicles that are crossing, and make protected and unprotected left and right turns, go around parked cars and other obstacles. And this is again in the hands of 100,000 Tesla customers.
Here I'm showing a video recorded by one of our customers driving around in dense San Francisco neighborhoods. All of the things that you're seeing here, such as the road boundaries, the lane lines, vehicles including their position, orientation, velocity, all of it is produced by running algorithms and neural networks on our on-board computer that consumes eight 1.2-megapixel cameras. These cameras produce image streams at 36 frames per second and in total give full 360-degree coverage around the car.
We have an inbuilt 144-teraop compute platform on every car that runs these neural networks. Again, everything that you are seeing here is based on pure online perception. There are no usage of radar, lidar, ultrasonics, or other sensors in the making of this. In fact, we don't even use HD maps, which are typically used in other autonomous driving solutions. All of this is based on the live camera video stream that produces this output, and the planning stack is able to navigate such complex scenes with just this raw live input.
Today I'd like to talk about something that's actually not shown in the visualization here. That is how we handle general obstacles. When we started out building out our more advanced stack, we wanted to come up with some way to represent general obstacles. And we started out with this image space segmentation method, which is pretty much standard for doing this. So here, every pixel in the image space is labeled as whether it's drivable or not drivable. And then the hope is that the planning stack can then use this to navigate the scene.
But there are a few issues with this approach. First of all, these predictions of whether some pixel is drivable or not is in the image space, in UV values, or whether some pixel is a drivable pixel or not. But then in order for the car to navigate the world in 3D, it needs to have the predictions in 3D space so that you can build physical models of interaction and handle the driving task. But going from image space to 3D space when done in this way of pixel-by-pixel segmentation can produce unnecessary artifacts or unnecessary noise in the system. For example, the few pixels at the horizon of the image can have a huge influence depending upon how we transfer this image space into 3D space. And we do not want the planning stack to have such noisy drivable space. And this is actually not a fundamental limitation; this is just a limitation of this representation.
This kind of representation also does not provide the full 3D structure of the scene. So it's hard to reason about overhanging obstacles or walls or other things that can occlude the scene.
A different way to model general obstacles is to use dense depth. In this task, you can make the network predict depth on a per-pixel basis, so every pixel produces some depth value. But even though these kinds of depth maps look really pretty when visualized in the color space, when you use the depth to unproject the rays into 3D points and visualize those 3D point clouds, while they look decent close by, as the distance increases they are inconsistent and very hard to consume downstream. Depth, for example, can be inconsistent locally, so walls would not be straight lines; it could be waving and curving, etc.
This also has the same issue that things close to the horizon are represented by very few points, that makes it hard for the planning stack to write reasonable logic around avoiding obstacles. These depth maps are produced on the image plane on a per-camera basis. This also makes it hard to produce a single consistent 3D space around the car.
Since the depth maps are typically modeled as regression targets, it's hard to predict through occlusions, and also at the boundaries they can, due to the properties of networks, smoothly change from, for example, a vehicle to the background. And this causes unnecessary artifacts in the 3D space.
Our solution to this problem is what we internally refer to as Occupancy Networks. Here's an example prediction from our Occupancy Network. I'll pause for a moment here for you to look at the predictions.
These Occupancy Networks take in all the eight camera streams of input and produce a single volumetric occupancy around the car. So every voxel or every location around the car, the network produces whether that voxel is occupied or not. In fact, it produces a probability of that voxel or the 3D location being occupied or not.
Like I mentioned, it takes in all the 8 cameras and produces a single volumetric output, so there is no stitching of independent predictions to produce this. The network does all of the internal sensor fusion to produce a single consistent output space. These networks all produce both static occupancy, like walls and trees and things like those, and also dynamic occupancy of things that are from moving objects, such as vehicles for the most part, but sometimes there can also be other obstacles that are moving, such as debris on the road.
The output space is directly in 3D. We can predict things through occlusions, so you can predict the presence of a curb even though a car might occlude it. Finally, this approach is extremely memory and compute efficient. Even though on first sight it might not appear so because it produces dense 3D occupancy, it might appear that the volume might be too large, but in the end it turns out to be much more memory and compute efficient because it allocates resolution where it really matters.
Like I mentioned earlier, dense depth or drivable space in the image is very low resolution far away and very high resolution close by. But in the Occupancy Networks, you get almost uniform resolution throughout the volume that is relevant for driving, and that makes it extremely efficient. The networks that I'm showing you, for example, run within 10 milliseconds on our compute platform. This enables the network to be run at 100 Hz, which is much faster than what the camera even produces the images at.
So how is this done? Let's take a brief look into the architecture. Here I have shown a couple of camera streams: the fisheye camera, which is a front-facing wide-angle camera, and the left pillar camera, which is a left-side facing camera. But it does take all the cameras as input. We first normalize to remove any vehicle-specific configuration, such as intrinsic calibration or distortion or things like those.
Once we have normalized these images, these images are then fed into state-of-the-art image backbone architectures. So here, for example, we use RegNets and BiFPNs to extract the image features. But this can be swapped with the state-of-the-art architecture from even, for example, CVPR 2022. These backbones now produce high-dimensional features in the image space. But we want the occupancy to be in 3D, right? How do we get this?
We use query-based attention to produce these 3D occupancy features. This is in fact similar to how a paper of the same name, Occupancy Networks, came out a few years ago, which again queries 3D points for whether the 3D point is occupied or not. So here it's of similar nature where we are taking 3D positional encoding, then mapping that into fixed queries. These fixed queries then attend to every image space feature. We also have positional embedding in the image space. So now these 3D queries attend to image space queries of all the image streams and then produce 3D occupancy features.
Since these are high-dimensional features, it's hard to do this directly at every point in space. So what we do is produce these high-dimensional features at a lower resolution and then use typical up-sample techniques like deconvs or what have you to produce denser, high-resolution occupancy.
An interesting fact is that when we started out this project, we initially were only targeting to do static objects such as walls or trees, because we have lots of different neural networks running in the car that deal with different kinds of obstacles. So we had continued to have a network that primarily handles moving objects such as vehicles, pedestrians, bikers, etc. And these networks also produce the full kinematics of the vehicles, such as depth, velocity, acceleration, jerk, etc. So we have this moving object network, and we thought, okay, we do not need to deal with moving objects here. We just need to deal with static objects such as trees, walls, etc.
But it turns out that having explicitly defined ontology can be pretty tricky to produce. So here in this example, this is from our own internal testing as well, where there's a pickup truck that looks like a fence. When the vehicle was moving and on the highway, we obviously knew it was a moving object and it was produced as a truck. But then when the vehicle stopped for multiple minutes at a traffic light, we said, okay, it does look like a fence, maybe it's not a moving object. And the moving object network decided that it's not a moving object. But obviously we do not want to drive into this vehicle; we want to account for it and drive around it or slow down appropriately.
This can get even trickier with pedestrians. On the left here are false pedestrians, where these can be like statues or mannequins dressed up like real people, but those are by definition false pedestrians. And on the right side you can see some true pedestrians, like a person in this Halloween costume standing on the side, or this crouched-up pedestrian who could look like a pile of trash from far away because of how the posture is right now.
This is obviously an impossible fight to fight, and we do not want to fight this fight. We just want to avoid the obstacles and hit neither moving nor stationary obstacles or anything for that matter.
So our solution to this problem is to produce both moving and static obstacles in the same framework. This way, things cannot slip through the cracks in between moving and stationary. In fact, what we realized was that there is no such thing as stationary obstacles. Any object, when enough force is applied, will move, and hence it's a moving object. So this distinction doesn't even make sense fundamentally. Anyway, to deal with this problem, we have introduced dynamic objects also in the same framework. So now the Occupancy Networks produce everything that's occupied in the scene; it doesn't distinguish between these things. In the occupancy, we can have additional semantic classes that can help with the control strategy later on, but in terms of pure occupancy, it does not distinguish between what is the reason something is occupied. It gives instantaneous occupancy.
But that is not sufficient to drive, because if you just have instantaneous occupancy, say you're following a vehicle on the highway, we don't want to assume that the occupied voxels are at zero velocity and then just slow down because you want to slow down to avoid these stationary obstacles. What we instead want to know is future occupancy at different times: how would the occupancy change? This allows us to know that the vehicle in front of us will move away by the time the ego vehicle reaches that location. Hence, we predict, in addition to occupancy, we also predict occupancy flow. This flow can just be the first derivative of the occupancy over time, or can also be higher derivatives that can give more precise control. And in order to produce this occupancy flow, we take in multiple time steps as input. So we take all the different occupancy features from some buffer of time, align all of them into a single coordinate frame. We use the same up-sample techniques to produce both the occupancy and the occupancy flow. And this provides robust protection against all kinds of obstacles.
Here you can see some examples of this combined Occupancy Network's occupancy flow. The colors of the vehicles represent the direction of the velocity vectors. Here, red means the vehicles are moving along their path, and green means they're moving towards their path. You can even see that there's a trash bin that actually flies out in front of the car; we also get that.
Here's another example of a strange-looking vehicle that's crossing our path. Our regular moving object network did not know the type of this object, didn't know it was a sedan or SUV, etc. But for the purposes of control, it does not actually matter. And this Occupancy Network provides this nice protection against these kinds of classification dilemmas from the network. Regardless of what's causing the obstacles, it will say that this volume of space is occupied and it's moving at this rate.
These kinds of vehicles can also have strange protrusions that are hard to model using traditional techniques. Typically, people reach for either cuboids or other polygonal shapes to represent moving objects because for the most part that is sufficient. But then the object can have arbitrary extrusions like so. And with this approach of occupancy, we get all of these extrusions for free without requiring complicated mesh-like topologies.
Finally, we can also use this geometric information to reason about occlusion. When the vehicle is making a protected or unprotected turn, it needs to reason about both the things that it sees, like vehicles that are currently visible, but also vehicles that it cannot or might not see. Here, for example, this vehicle is making an unprotected right turn. There is a crossing road in front of it, and there can be potential vehicles that are occluded by the trees and the signposts on the left side here. So the car knows that it cannot see the road from these occlusions, and it also knows why it cannot see them. It knows that it's occluded by either a tree or by a different moving object. And there are different control policies that the car can come up with to deal with this and remove this occlusion. So for example, for a stationary object, it can crawl forward a few centimeters or meters before encroaching into the crossing road and check for when it becomes visible. And since we have the full 3D obstacles, we can also reason about at what speed would we hit this, at what distance would we hit this, and then have very nice smooth control to creep up and look for these occluded objects. So the Occupancy Networks help in lots of different ways to help improve the control stack.
Zooming out a bit, the team thinks of Occupancy Networks as an extension to Neural Radiance Fields. As many of you know, Neural Radiance Fields have taken over computer vision research to a large extent over the last couple of years. There's a lot of interesting work happening in this field, and this works really well for us because in our mind this is the same approach. Neural Radiance Fields try to reconstruct the scene from multi-view images, essentially. But the way typically it's done is it is reconstructing the scene for a single scene or a single set of images from a single location.
So to start out with, we can take any trip from the fleet. We have a really good calibration and trajectory estimation stack, so you can use that to produce accurate camera poses across time and then run state-of-the-art NeRF models to produce really good 3D reconstruction by differentiably rendering images from the 3D scene. The original NeRF paper used a single neural network to represent the full 3D scene, but then there are other works that are more recent, such as Plenoxels, that use voxel-based representations to do this. We can also extend these to have both voxels, like voxels of tiny MLPs or other continuous representations, to interpolate the probability to produce these differentiably rendered images.
There is one problem with running NeRFs in the wild from real-world images, and that's because there's a ton of image artifacts that we observe in real-world imagery, such as the sun glare here on the left side, or dirt or dust on the windshield can produce really bad artifacts due to diffraction of the light, or a raindrop can further distort light travel to produce really ghosting artifacts.
One way to be robust against this is to use higher-level descriptors that are somewhat invariant to the local lighting artifacts, such as glare, because RGB can be extremely noisy. But adding descriptors on top of RGB can give a layer of semantic protection against changing RGB values.
So our goal then is to use this kind of supervision for Occupancy Networks. And if you think about it, we can take this differentiable rendering architecture of NeRFs and then stick it as a loss function at the output of these Occupancy Networks. Since these Occupancy Networks need to produce the occupancy in few-shot, we cannot run the full NeRF optimization in the car. But we can have a reduced optimization that's running in the car all the time in the background, making sure that the occupancy that it produces is explaining all the sensor observations that the car receives at runtime. In addition, this can also be stacked on top at training time to produce good supervision for these networks. So instead of directly supervising the occupancy, which we also do, in addition to that we can also supervise by just differentiably rendering different sensor data of held-out images, for example. This supervision can supervise both the occupancy and also occupancy flow by having some kind of temporal constraints of motion.
So that's everything on the side of producing obstacles in a general way. Now that we have general obstacle reduction networks, the next step is to avoid all collisions. Autopilot already has a lot of safety features. Here I'm showing a particular mode of failure of humans where they accidentally press the accelerator pedal instead of the brake pedal. So here, for example, these people are pressing the accelerator pedal thinking that they're pressing the brake pedal, but the car realizes that they are doing this and are heading towards a collision and automatically cuts out the acceleration, presses the brake to prevent the humans from colliding. In the previous case there was a person, but in this case this driver would have launched the car into a river, and the Autopilot saved them.
One final video here where this person again is trying to park but then misapplies the pedal and then would have crashed into the storefront and maybe grandma here as well. While we already save a lot of collisions, there are still collisions that humans get into that the Autopilot system could prevent them from doing so. Here's an example of that where this person again misapplied the accelerator pedal. The Autopilot system saved them from their forward collision, but then they changed gears and then went backwards at full speed and hit the garage. While it's incredibly sad, I'm glad that no one was harmed in this accident, but it's kind of senseless to collide when we have a great system that can detect general obstacles.
So one might wonder, now that you have obstacles, it must be trivial to avoid collisions. While it's tractable, it requires a little bit more thought into doing this correctly. I say this because building a safe system, if safety is the only thing that matters, is quite easy. But in order for a self-driving car to be actually useful, it needs to be safe but also comfortable and reasonably fast in transporting the person from A to B.
To illustrate this, if you take a hypothetical example where this imaginary car has infinite jerk at its disposal, so it goes at full speed, at highway speed, say 65 miles per hour, and then just before crashing, a few centimeters before crashing, it applies tremendous amount of braking in, say, infinite jerk and axle to brake, and it would obviously instantaneously stop before hitting the obstacle. While this can be deemed as safe, first of all, it's not practical because you don't have cars that have infinite braking. But even if it had infinite braking, we don't want to use that unnecessarily. Even typical, like our Teslas, for example, have 10 or 11 meters per second squared of braking, but we don't want to use all of that unnecessarily. Secondly, the car can drive extremely slow. Say, for example, the car is crawling at one or two miles per hour; it can hit the brake at any time in all conditions very easily. Just that it's practically really annoying to drive so slow. That's why I say it takes a little bit more intelligence to do driving in a safe, comfortable, and reasonably fast manner.
If you merely approach this driving problem using purely search-based techniques, for example, this compute can actually take a long time because you again want to brake early on with low enough jerk. So you need to predict that the collision is avoidable or unavoidable many seconds before the collision to hit the brakes in a smooth manner to safely and smoothly avoid a collision. The search space of such things can be quite large and can take many seconds or minutes to produce a good solution. Running online in the car, there is not enough time to do such compute. So what we do instead is we try to approximate these with neural networks. Especially with the recent advent of implicit fields, we are able to tap into the same work to produce implicit fields that encode obstacle avoidance.
So what we do is we take the occupancy from the previous networks, we encode that occupancy into a super-compressed MLP, essentially, where this MLP is an implicit representation of whether a collision is avoidable or not from any particular query state. And this collision avoidance gives some guarantees for collision avoidance for some time horizon. For example, we can ask, is a collision avoidable within two seconds or four seconds or some time horizon? And the query can be some high-dimensional vector state. Here I'm showing the position, orientation, velocity, the lateral and longitudinal accelerations of the car. And based on this query state, the network can output the probability of whether a collision is avoidable or not. This gives extremely fast lookup on the order of few microseconds to produce this approximate probability of whether a collision is avoidable or not.
In order to illustrate this, let's look at this example where you're seeing a top-down view. Black pixels here are obstacles, the gray pixels here are just the road surface, and the white pixels are road paints, just lane lines. In this top view in 3D space, you can place the car at any pixel location and simulate whether a collision is avoidable or not. If you imagine the car to be a single point mass, and the collision avoidance horizon to be immediate, so whether the current time is in collision or not, this would really just be the location of obstacles. But the point is that the car is not a point mass; it has some shape, it has a rectangular shape, and it can rotate as well.
So here, if we show whether the shape of, when we convolve the shape with the obstacles, we will know instantaneously whether the car is in a collision state or not. So here you can see that as the car rotates, the collision field is changing. The green color output means that it's in a safe spot where there is no collision, and the red pixels mean that the car, when put in this configuration in that pixel location, is in a collision. So as the car rotates, you can see that the narrow corridors kind of close down because in that configuration the car would be in collision, but when it's aligned, it opens up and becomes more green, meaning that the car would not be in collision. This is a trivial task to compute, and we do not need neural networks for this task, obviously.
But when we extend the time horizon to, for example, two seconds, now the car can perform some actions to avoid a collision. So you need to have a search over this search space to know whether a collision is avoidable or not. In this example, the vehicle is at a fixed velocity of 13 meters per second, that's longitudinal speed, and they are changing their heading. And you can see that as the heading changes, things quickly either go into collision or collision is avoidable. They can also vary the velocity at a fixed heading. So here the car is facing straight, and then we are just changing the velocity to be increasing or decreasing, and you can see that different regions open and close down based on the velocity. So if you're at a low velocity, the car can get really close to a curb and then adjust course.
You can obviously do this jointly as well. Here we are changing both the velocity and the heading of the car. You can see how, based on the configuration parameters, the network correctly thinks that different locations can be, either the car can be in different locations where the collision is either avoidable or unavoidable. Finally, you can combine all of this to roll out some trajectory that avoids collision and makes progress. Here we are simulating an inattentive, distracted driver who just presses the accelerator and doesn't do any steering, and the car comes in and intervenes whenever necessary to either steer the car or brake as necessary to avoid the collision.
The car starts out here in this rollout, and then since it's heading towards this wall on the right side, it immediately hits the brakes and also steers hard on the left side. Once it's aligned there, and then it knows that it's on trajectory to collide again, so it turns right side and then orients itself inside this narrow corridor. But the interesting thing is that once it's inside this narrow corridor, even though the obstacles are close to it, it does not freak out. It says that, okay, it's actually fine, we can go fast here because there is no risk of collision or a collision is easily avoidable. Doing this again naively would have taken multiple minutes to come up with a solution, but then doing this using networks enables us to quickly query for whether a collision is avoidable or not and then take actions that prevent collisions from happening.
When you simulate this in closed loop, you can also see that the car is able to avoid collisions. Here the regular Autopilot system is not running; this is just the collision avoidance system running on top of a simulated human driver who is not paying attention. This driver is just pressing the accelerator and then not touching the steering wheel. And then as they are getting close to a collision, the collision avoidance system kicks in and then turns the wheel or presses the brake on its own to prevent the car from colliding.
Yeah, so in summary, we showed how we use multi-camera video inputs to produce dense occupancy and occupancy flow. We also briefly saw how you can use large multi-view constraints from the fleet to supervise this in addition to our auto-labels for vision. Once we have this great occupancy, we can plumb this into other neural networks that produce an efficient collision avoidance field. And if we do all of these steps correctly, we can produce a car that doesn't need to crash ever. Maybe one missing piece is more engineers and scientists to work on this problem with us to enable to build this great technology. Thank you.