[CVPR'23 WAD] Keynote - Ashok Elluswamy, Tesla

Okay, thank you so much for the introduction. Hi everyone, my name is Ashok Elluswamy. I'm a contestant on the autopilot team. Hopefully you're able to hear my voice and see my screen and video and things. Please let me know if that's not the case.

Today I would like to present our work on what we think is going to be the foundation model for autonomy and robotics. This is not just a work of myself; I'm representing a large team of talented engineers in our team. Let's get started.

Our team has shipped the full self-driving beta software to everyone who has purchased it in the United States and Canada. There's roughly 400,000 vehicles and today they have driven up to 50 million miles on FSD. I think the cool thing about this is that this is a scalable self-driving stack that you can take the car anywhere in the US, turn it on, put in a destination, and the car would attempt to navigate to this destination, handling all of the intersections, stopping at traffic lights, interacting with other objects. And all of this is driven primarily by the eight cameras that are on the car that are giving a full 360-degree coverage around the car.

The reason that works well is because our stack is based on a really modern machine learning-based stack where a lot of the components of the self-driving stack are just folded into your neural networks. And I would say this is different than the more traditional approach to self-driving which uses localization maps, a lot of radar, ultrasonics, etc. to fuse together. Instead, this is primarily being driven by just cameras. And if you have a Tesla yourself, you can obviously buy the car and experience it. Otherwise, you just take my word or look at some videos, but it works quite well and we are in the process of making it even better.

I shared about these occupancy networks that are one of the more important pieces in our stack. I would consider this as one of the foundational model tasks because this is very general. This is a very general task and doesn't have any specific ontology, or like it is robust to ontology errors. It really just predicts whether some voxel in 3D space is occupied or not and the probability of that, and can represent arbitrary scenes. There is no labeling or ontology design required and it's quite general and can apply anywhere. In addition to just the occupancy, we also predict the flow of voxels in the future that kind of like gives arbitrary motion as well. And everything runs in real time. This is quite similar to NeRF in general, but unlike NeRF for multi-view reconstruction which is usually done for a single scene, we predict the occupancy based on the eight cameras in real time. So the video is streaming and then we just predict for all of space around the car on whether this box was occupied or not, as opposed to doing this offline post-processing step.

So the architecture looks, you know, with a lot of it looks very complicated, but then it's actually not that complicated in the end. Videos from multiple cameras stream in and you can choose whatever backbone you want, you know, RegNets, what are the latest bits, you can throw anything in there. And then everything comes together in a large Transformer block that does sort of a spatial attention to build up features and also does temporal attention, with some geometry thrown in there to form some features that can then be upsampled into the actual predictions. It's quite straightforward even though the diagram looks a bit complicated.

In the same architecture and the modeling can be used not just for occupancy but for other tasks that are needed for driving. Obviously lanes and roads are very important for driving tasks, but I'd say lanes are quite obnoxious to predict. The reason is, first of all, lanes are higher dimensional objects unlike, you know, it's definitely not like one lane or two lanes, you know, high dimensional and then they have like a graph structure. Like objects for the most part, say vehicles, they're self-contained, they're just local, whereas lanes can span the entire road. You can see multiple miles of lanes in your view and they can fork and merge and cause all kinds of trouble in the modeling. They also have large uncertainty. Sometimes you might not be able to view the lanes because they're occluded or it's nighttime, only part of the lane is visible. And it's not just that, sometimes even if everything is visible, even humans cannot agree on whether something that you're looking at is two lanes or one lane for instance. So there's a ton of uncertainty in what are lanes. And then it's not sufficient to just predict them as some kind of raster, it's very hard to use downstream then. So it's better to predict them as some kind of vector representation, you know, like polylines, splines, polynomials, etc. to help with ease of use. And all of this needs to happen within tens of milliseconds in real time. Like I said, it's a very difficult problem to predict lanes in real time in the real world.

Nonetheless, we use state-of-the-art generative modeling techniques. In this case, we use autoregressive Transformers, quite similar to I would say GPT in terms of how we model the lanes. So we can predict the, you know, you can tokenize the lanes and predict them one token at a time. Unlike language which is mostly linear, we have to predict the full graph structure. Hence we come back, predict, you know, where is the working point, what is the merging point, etc. And everything is done end-to-end using neural networks with little to no post-processing required after this.

Another important task for self-driving is obviously moving objects, you know, vehicles, trucks, pedestrians, whatever. And it's not sufficient to just detect them, you need to have their full kinematic state and also predict their shape information, their futures, etc. All of these models, the models that I described earlier, even the lanes one and objects one, are in some ways multi-modal models in the sense that they take in not just camera video streams, they also take in other inputs such as in this case, ego's own kinematics. So ego's velocity, acceleration, jerk, etc. all goes in. We also provide in the navigation instructions to the lanes to kind of guide us where to, like which lane to use, etc. So everything is done within the network. That's why I say it's like a modern machine learning stack where instead of doing this in post-processing, we just try to combine everything and then do perception sort of end-to-end so to speak.

So here you can see predictions of these models. The lanes that you see here, the vehicles that you see here are all just predicted by these networks with a lot of post-processing. There is no tracking or anything like that in the things that you're seeing here. So overall, I would say that it's quite stable. The green spines that are coming out of these vehicles are just their forecasted future. It's kind of like a standard task at this point I would say, but it all works quite nicely and in real time in the car.

It doesn't have to stop with just perception too. Once we have all of these percepts, like lanes, occupancy, objects, and even a few more like traffic controls and other things, you can do the entire motion planning also using just a network. I'm not going into too many details on how we do that, but essentially it can just be thought of as one more task instead of it being a separate thing.

So how is all of this possible? And I think it's because we have built this sophisticated auto-labeling pipeline that gives us data from the entire fleet. Millions of video clips across the entire world can be tapped. On the left side what you're seeing is an example of multi-trip reconstruction where we choose some location, multiple Tesla vehicles driving through the location approach their video clips and other kinematic data to us. We bring everything together and we construct the entire 3D scene. So the lines that you look, you know, there is a cyan colored one, there's also a few other colored ones, those are all different cars doing different trips through the world and it's actually all very well aligned. Let me see if we can play it again. Yeah, the pink line and the cyan line that you see there, those are like different trips of different cars driving around and everything is just aligned very nicely. And this multi-trip reconstruction has enabled us to get all the lanes, road lines, everything directly from the fleet in the millions, anywhere on Earth essentially.

Once you have this base structure of trajectories, calibration from all these cameras, so you can really do a lot of cool things to reconstruct the entire scene. I'm not sure if the video plays quite nicely, but on my own screen it looks very smooth where you can see the ground surface, it's reconstructed quite nicely. There's like no artifacts such as double vision or blurring, things are crisp, look geometrically correct. This is a hybrid approach to NeRF and general 3D reconstruction. Sometimes a NeRF, even though the rendered visuals might look very nice, the underlying geometry might be very fuzzy and cloudy. So we have a hybrid approach which works quite nicely. And you can see here, the barriers, the vehicles, even trucks, etc. are reconstructed pretty accurately.

Once we have these reconstructions, we then run even more neural networks just offline to produce the labels that we want. Like I had mentioned earlier, for lanes we need some kind of vector representation to make it very easy to use. So instead of just using the raster directly, we have offline neural networks that run on top of it and then produce the vector representation that can be then used as labels for the online stack. Similar to the lanes, like once you have the lanes and the roads reconstructed, you can also auto-label traffic lights. Here you're seeing traffic lights auto-labeled by our system without any human inputs. And these are multi-view consistent. We try to create, again, we can predict their shape, color, relevancy. You can see these wide traffic lights on the side, they also reproject correctly into all the camera views. And it's because we have this really good auto-labeling system that calibrates everything jointly and it's pixel-perfect in 3D space.

So yeah, all of these predictions together give us super-sport understanding of the world from cameras. And I would already try to call this sort of a foundation model that can be used in a lot of different places. And these predictions really help FSD drive in any place. You don't have to geo-restrict it. You can even sometimes construct a new road, turn it on, and it would work quite nicely there. In addition, they also help manual driving, obviously, because humans are not perfect drivers. So they need some help every now and then. In this case, on the left side, the ego driver for some reason blew past the stop sign and was almost about to crash into this red car. And our system rejected this and then braked automatically. And similar on the right side, driving straight and then someone just comes in and cuts us off. It's quite dangerous, but then the system applied the brake quite early. The reason why this is different than what you would think that AEB systems have been there since the 1980s, what is new about this? I think Tesla is the first company to ship this emergency braking for crossing vehicles to the best of my knowledge. And the reason that crossing objects are harder, unlike say vehicles that are in your own lane, is because for crossing objects you need to know whether they're going to stop in time or not. What is the stop line, do they have traffic lights, and if they were to turn, which lanes would they turn into, etc. There's a ton of work that needs to happen to understand where the crossing object should go and is likely to go. Do they have room to stop, etc. It's not as simple as just detecting a vehicle and having their velocities and things like those. Yeah, so like I said, I believe Tesla is the first company to ship crossing AEB and it's already in customers' hands the last several months.

But is a foundation model really just a bunch of these tasks concatenated together, or can there be more to it? We think there can be. These tasks, like in the occupancy for example, while it's quite general, some things are harder to represent even in that space. I'll probably go into more details shortly. Because that's why we are working on learning a more general world model that can really just represent arbitrary things. So in this case, what we do is we have a neural network that can be conditioned on the past or other things to predict the future. And obviously everyone has wanted to work on this forever, and I think with the recent generative models like Transformers, diffusion, etc., we finally have started it. What you're seeing here is purely generated video sequences. Given the past videos, the network predicts some samples from the future, hopefully the most likely sample. And you can see that it is being predicted not just for one camera, but it predicts all the eight cameras around the car jointly. And see how the car colors are consistent across the cameras, the motion of objects is consistent in 3D, even though we have not explicitly asked it to do anything in 3D or baked in any 3D priors. This is just the network understanding depth and motion on its own without us informing it of so. And since this is all just predicting future RGB values, the ontology is quite general. You can throw any video clip from driving or from YouTube or from your own phone, anything can be used to train this general dynamics model of the world.

Additionally, it can also be action-conditioned. I'll bring to a few examples. So here on the left side, the car is driving in the lane and we're asking it, okay, just keep in this lane and keep driving. And then, like I said earlier, the car is able to, or the model is able to predict all of the geometry flow very nicely and understands 3D. On the right here, we're asking it to change lanes to the right side. Maybe we'll go back and play it again. So on the left, it's just going straight and we ask it to go straight, the model goes straight. And then on the right side, we ask it to make a lane change and it makes a lane change. And the past context is the same for both of these outputs. So given the same past, and when we ask it for different futures, the model is able to produce or imagine different futures. This is super powerful because now you have essentially a neural network simulator that can simulate different futures based on different actions. And unlike a traditional game simulator, this is way more powerful because it can represent things that are very hard to describe in an explicit system. I'll show a few more examples, but it is super powerful. And also the motion, the intention, and then the natural behavior of other objects such as vehicles is very hard to represent explicitly, but in this world it's very easy to represent. It doesn't have to stop with just RGB. You can obviously do this kind of future prediction task not just in RGB but also in semantic segmentation or reconstruction into 3D spaces where you can imagine future 3D scenes entirely based on just the past and then your action prompting, or even without prompting, you can predict different futures. This is, I personally, I'm amazed by how well this works and it's a very exciting future that we are working on here.

Yeah, here's some examples, but I think something like this is going to be needed to represent what's happening in the scene. Like there's a lot of smoke coming in, all of the pictures, like there's paper flying everywhere. That's going to be tough for even occupancy where, okay, you have paper flying everywhere, the soccer bench is occupancy, but then how do you know it's paper? What do you know the material properties of it? There's like smoke, obviously you can drive through it, but it is occupying space and light does not transmit through. There's a lot of nuances to driving and we have to really solve all of these problems to build a general driving stack that can drive anywhere in the world and be human-like, might be fast, efficient at all speeds, yet very safe. And I think we're working on the right recipe for building this.

And obviously, training all these models takes a ton of compute and that's why Tesla is aiming to become a world leader in compute. Dojo is our training hardware that we have custom built at Tesla that is starting production next month essentially. And with that, we think we are on the way to become one of the top compute platforms in the entire world. And we also think that in order to train these foundation models for vision, we need a lot of compute. And not just compute for training this one model, but compute to try a lot of different experiments to see which models actually work well. And that's why it's super exciting for us to be in the spot where compute is going to be abundant and it's just going to be mounted by ideas of engineers.

And in addition to this, this is not just being built for the car but also for the robot. We already have the occupancy network, for example, and a few of the networks all shared between the car and the robot. And it actually works quite well and generalizes across these platforms. And we want to extend it to all the tasks that we have. Lanes and vehicles, for example, should not be specific to cars. If the robot say happens to walk to the road and looks around, it should understand roads and vehicles and how vehicles move, etc. All of this will just be built for both the platforms and any other future robotics platform that would also need this.

That's basically it. To summarize again, we're building these super cool foundational models for vision that really understands everything. And if you generalize across cars and robots, this can be trained on tons of diverse data from the fleet and tons of compute. Yeah, so I'm really excited for the next 12 to 18 months of what's going to happen here. Thank you.

Happy to answer any questions. We have actually quite some time for questions, so if you have a question, you can come forward. Otherwise, I can start with one from the Zoom chat. And it's actually about the 3D reconstruction with your hybrid NeRF approach. And the question there is, how can you actually deal with moving objects in this formulation?

We can really just track moving objects. There's a lot of cues that the object is moving with respect to the scene and we can also extract the information to segment moving things separately. Obviously, for things that we know can usually move, it's very easy, like vehicles, trucks, etc. Those are very easy because we already know that these objects can move. But arbitrary things such as flying trash bins or whatever, you can use a lot of other signals in the data itself to help do it.

I'd say it's like work in progress. I don't think we have to over-fixate on the current architecture. I think the main message is that this kind of future prediction tasks are starting to work now and that gives both a simulator where we can roll out different outcomes but also learn representations just by making this prediction. I think that is the key message.

Sure. Like you mentioned, lanes can fork and merge and do other things and there's a graph structure to them. And you can predict them simultaneously, but it's like hard to, like it is, we want to model a distribution or high dimensional space, right? Because if there's uncertainty on whether it's like one lane or two lanes, you don't want to predict like one and a half lanes. You know, like if we want either predict one lane or we want to predict two lanes, so we want to sample from the distribution. There's some distribution that, okay, maybe 60% probability it's like one lane and 40% probability it's two lanes. If you predicted without a proper generative model which gives nice samples, you will get sort of like smear and then everything will look blurry and it's very hard to then use downstream. So that's why it's useful to do something like this to get a nice sample.

Yeah, again, these are like hyperparameters. Anyone can try with different voxel sizes. It's really a trade-off between memory and compute and what you're willing to do. So these are not set in stone. It's not, you know, whatever the application needs. Even for between the robot and the car, you can configure different sizes and they have different needs. So it is really configurable. And optionally can be made queryable, as in you can get like arbitrary position by just having the network, gamut small MLPs or something like that where you can query to arbitrary position. Again, it just comes at compute, so it's up to the design of the downstream task to determine what is the base size you need, density or not, and you get away with some simple formulations. It's really up to, there's a large design space here.

Yeah, I think the same should apply for a humanoid robot in the sense that you can prompt it to arbitrary things. You can ask it to pick up a cup or walk to some door or something, and then the model should be able to imagine what picking up a cup would look like or walking to a door would look like. And I don't see why it should be any different between the car and the robot. So, you know, it's because it's all pixels and the shapes are all the same. So it's just going to be a video that looks like the hand is reaching to a cup or the robot, if it's the first-person view, then it's the first-person view of the robot walking to the door. I think it should work.

Yeah, the occupancy is already used for a lot of collision avoidance. Sometimes when there's like some strange vehicles, say like there's construction vehicles that have their like lever arms going like this and then can pick up things, and it's very hard to model using say cuboids or something like that. And you can obviously represent it with meshes, but then they become these high-dimensional objects and distribution over them is pretty hard. And the occupancy is a slightly, I would say, low-level or like a dumber task where you can just predict whether something is present and where it is moving, and that is already enough to avoid collisions. And we make sure that the planning system respects collision avoidance from the occupancy.

One thing I can say is that it doesn't ship to customers yet and we are still optimizing it. Maybe later this year, I don't know.

Sure, yeah, I think these are not HD maps, so the alignment is not super critical in some sense. You can have 20 meters of error or like 50 meters of error and it's still fine. It's really just what you would look at at your phone or something when you use navigation. Your Google Maps or whatever map application you use might give you some route and represent this, and that is enough to kind of tell the network wherever you want to go. So it's really like low-definition maps and it is interesting to design that way so that we don't take a large dependence on the maps. It's really just to roughly guide which roads we should take, which lanes to pick, and things like those.