Tesla AI Day 2021

Okay, hi everyone. Welcome. My name is Andre, and I lead the vision team here at Tesla Autopilot. And I'm incredibly excited to be here to kick off this section, giving you a technical deep dive into the Autopilot stack and showing you all the under-the-hood components that go into making the car drive all by itself. So we're going to start off with the vision component here. Now, in the vision component, what we're trying to do is we're trying to design a neural network that processes the raw information, which in our case is the eight cameras that are positioned around the vehicle. And they send us images, and we need to process that in real time into what we call the vector space. And this is a three-dimensional representation of everything you need for driving. So this is the three-dimensional positions of lines, edges, curbs, traffic signs, traffic lights, cars, their positions, orientations, depth, velocities, and so on. So here I'm showing a video of... actually, hold on. Apologies. So here I'm showing the video of the raw inputs that come into the stack, and then the neural net processes that into the vector space. And you are seeing parts of that vector space rendered in the instrument cluster on the car. Now, what I find kind of fascinating about this is that we are effectively building a synthetic animal from the ground up. So the car can be thought of as an animal. It moves around, it senses the environment, and you know, acts autonomously and intelligently. And we are building all the components from scratch in-house. So we are building, of course, all of the mechanical components of the body, the nervous system, which is all the electrical components, and for our purposes, the brain of the Autopilot, and specifically for this section, the synthetic visual cortex. Now, the biological visual cortex actually has quite intricate structure and a number of areas that organize the information flow of this brain. And so, in particular, in your visual cortices, the light hits the retina, it goes through the LGN, all the way to the back of your visual cortex, goes through areas V1, V2, V4, the IT, the ventral and dorsal streams, and the information is organized in a certain layout. And so, when we are designing the visual cortex of the car, we also want to design the neural network architecture of how the information flows in the system. So the processing starts in the beginning when light hits our artificial retina, and we are going to process this information with neural networks. Now, I'm going to roughly organize this section chronologically, so starting off with some of the neural networks and what they looked like roughly four years ago when I joined the team, and how they have developed over time. So roughly four years ago, the car was mostly driving in a single lane going forward on the highway, and so it had to keep lane and it had to keep distance away from the car in front of us. And at that time, all of processing was only on individual image level. So a single image has to be analyzed by a neural net and make little pieces of the vector space, process that into little pieces of the vector space. So this processing took the following shape: we take a 1280 by 960 input, and this is 12-bit integers streaming in at roughly 36 hertz. Now, we're going to process that with the neural network. So instantiate a feature extractor backbone. In this case, we use residual neural networks. So we have a stem and a number of residual blocks connected in series. Now, the specific class of ResNets that we use are RegNets, because we like this... RegNets offer a very nice design space for neural networks because they allow you to very nicely trade off latency and accuracy. Now, these RegNets give us as an output a number of features at different resolutions and different scales. So in particular, on the very bottom of this feature hierarchy, we have very high-resolution information with very low channel counts, and all the way at the top, we have low spatial, low resolution especially, but high channel counts. So on the bottom, we have a lot of neurons that are really scrutinizing the detail of the image, and on the top, we have neurons that can see most of the image and a lot of that context, have a lot of that scene context. We then like to process this with feature pyramid networks. In our case, we like to use BiFPNs, and they get the multiple scales to talk to each other effectively and share a lot of information. So for example, if you're a neuron all the way down in the network and you're looking at a small patch and you're not sure this is a car or not, it definitely helps to know from the top layers that hey, you are actually in the vanishing point of this highway, and so that helps you disambiguate that this is probably a car. After a BiFPN and a feature fusion across scales, we then go into task-specific heads. So for example, if you are doing object detection, we have a one-stage YOLO-like object detector here, where we initialize a raster and there's a binary bit per position telling you whether or not there's a car there. And then in addition to that, if there is, here's a bunch of other attributes you might be interested in, so the x, y, width, height, offset, or any of the other attributes like what type of a car is this, and so on. So this is for the detection by itself. Now, very quickly we discovered that we don't just want to detect cars, we want to do a large number of tasks. So for example, we want to do traffic light recognition and detection, lane prediction, and so on. So very quickly we converged on this kind of architectural layout where there's a common shared backbone and then branches off into a number of heads. So we call these therefore HydraNets, and these are the heads of the hydra. Now, this architectural layout has a number of benefits. So number one, because of the feature sharing, we can amortize the forward pass inference in the car at test time, and so this is very efficient to run. Because if we had to have a backbone for every single task, that would be a lot of backbones in the car. Number two, this decouples all of the tasks, so we can individually work on every one task in isolation. And for example, we can upgrade any of the data sets or change some of the architecture of the head and so on, and you are not impacting any of the other tasks, and so we don't have to revalidate all the other tasks, which can be expensive. And number three, because there's this bottleneck here and features, what we do fairly often is that we actually cache these features to disk. And when we are doing these fine-tuning workflows, we only fine-tune from the cached features up and only fine-tune the heads. So most often in terms of our training workflows, we will do an end-to-end training run once in a while where we train everything jointly, then we cache the features at the multi-scale feature level, and then we fine-tune off of that for a while, and then end-to-end train once again, and so on. So here's the kinds of predictions that we were obtaining, I would say several years ago now, from one of these HydraNets. So again, we are processing individual images. There we go. We are processing just individual image, and we're making a large number of predictions about these images. So for example, here you can see predictions of the stop signs, the stop lines, the lines, the edges, the cars, the traffic lights, the curbs here, whether or not the car is parked, all of the static objects like trash cans, cones, and so on. And everything here is coming out of the net here, in this case, out of the HydraNet. So that was all fine and great, but as we worked towards FSD, we quickly found that this is not enough. So where this first started to break was when we started to work on Smart Summon. Here I am showing some of the predictions of only the curb detection task, and I'm showing it now for every one of the cameras. So we'd like to wind our way around the parking lot to find the person who is summoning the car. Now, the problem is that you can't just directly drive on image space predictions. You actually need to cast them out and form some kind of a vector space around you. So we attempted to do this using C++ and developed what we call the occupancy tracker at the time. So here we see that the curb detections from the images are being stitched up across camera scenes, camera boundaries, and over time. Now, there are two major problems, I would say, with this setup. Number one, we very quickly discovered that tuning the occupancy tracker and all of its hyperparameters was extremely complicated. You don't want to do this explicitly by hand in C++. You want this to be inside the neural network and train that end-to-end. Number two, we very quickly discovered that the image space is not the correct output space. You don't want to make predictions in image space, you really want to make it directly in the vector space. So here's a way of illustrating the issue. So here I'm showing on the first row the predictions of our curbs and our lines in red and blue, and they look great in the image, but once you cast them out into the vector space, things start to look really terrible, and we are not going to be able to drive on this. So you see how the predictions are quite bad in vector space, and the reason for this fundamentally is because you need to have an extremely accurate depth per pixel in order to actually do this projection. And so you can imagine just how high of a bar it is to predict that depth so accurately in every single pixel of the image. And also, if there's any occluded area where you'd like to make predictions, you will not be able to because it's not an image space concept in that case. So we very quickly realized... the other problems with this, by the way, is also for object detection. If you are only making predictions per camera, then sometimes you will encounter cases like this where a single car actually spans five of the eight cameras. And so if you are making individual predictions, then no single camera sees all of the car, and so obviously you're not going to be able to do a very good job of predicting that whole car, and it's going to be incredibly difficult to fuse these measurements. So we have this intuition that what we'd like to do instead is we'd like to take all of the images and simultaneously feed them into a single neural net and directly output in vector space. Now, this is very easily said, much more difficult to actually achieve, but roughly we want to lay out a neural net in this way where we process every single image with a backbone, and then we want to somehow fuse them, and we want to re-represent the features from image space features to directly some kind of vector space features, and then go into the decoding of the head. Now, so there are two problems with this. Problem number one, how do you actually create the neural network components that do this transformation, and you have to make it differentiable so that end-to-end training is possible. And number two, if you want vector space predictions from your neural net, you need vector space-based data sets. So just labeling images and so on is not going to get you there. You need vector space labels. We're going to talk a lot more about problem number two later in the talk. For now, I want to focus on the neural network architectures. So I'm going to deep dive into problem number one. So here's the rough problem, right? We're trying to have this bird's-eye view prediction instead of image space predictions. So for example, let's focus on a single pixel in the output space in yellow, and this pixel is trying to decide, am I part of a curb or not, as an example. And now, where should the support for this kind of a prediction come from in the image space? Well, we know roughly how the cameras are positioned and their extrinsics and intrinsics, so we can roughly project this point into the camera images. And you know, the evidence for whether or not this is a curb may come from somewhere here in the images. The problem is that this projection is really hard to actually get correct because it is a function of the road surface. The road surface could be sloping up or sloping down, or also there could be other data-dependent issues. For example, there could be occlusion due to a car. So if there's a car occluding this viewport, this part of the image, then actually you may want to pay attention to a different part of the image, not the part where it projects. And so because this is data-dependent, it's really hard to have a fixed transformation for this component. So in order to solve this issue, we use a transformer to represent this space. And this transformer uses multi-headed self-attention and blocks of it. In this case, actually, we can get away with even a single block doing a lot of this work. And effectively, what this does is you initialize a raster of the size of the output space that you would like, and you tile it with positional encodings with sines and cosines in the output space. And then these get encoded with an MLP into a set of query vectors. And then all of the images and their features also emit their own keys and values. And then the queries, keys, and values feed into the multi-headed self-attention. And so effectively, what's happening is that every single image piece is broadcasting in its key what it is, what is it a part of. So hey, I'm part of a pillar in roughly this location, and I'm seeing this kind of stuff, and that's in the key. And then every query is something along the lines of, hey, I'm a pixel in the output space at this position, and I'm looking for features of this type. Then the keys and the queries interact multiplicatively, and then the values get pulled accordingly. And so this re-represents the space, and we find this to be very effective for this transformation. So if you do all of the engineering correctly, this again is very easily said, difficult to do. You do all of the engineering correctly. There's one more... there's no problem actually before... I'm not sure what's up with the slides. So one more thing you have to be careful with some of the details here when you are trying to get this to work. So in particular, all of our cars are slightly cockeyed in a slightly different way. And so if you're doing this transformation from image space to the output space, you really need to know what your camera calibration is, and you need to feed that in somehow into the neural net. And so you could definitely just like concatenate the camera calibrations of all of the images and somehow feed them in with an MLP. But actually, we found that we can do much better by transforming all of the images into a synthetic virtual camera using a special rectification transform. So this is what that would look like. We insert a new layer right above the image rectification layer. It's a function of camera calibration, and it translates all of the images into a virtual common camera. So if you were to average up a lot of repeater images, for example, which faced at the back, you would, without doing this, you would get a kind of a blur. But after doing the rectification transformation, you see that the back mirror gets really crisp. So once you do this, this improves the performance quite a bit. So here are some of the results. So on the left, we are seeing what we had before, and on the right, we're now seeing significantly improved predictions coming directly out of the neural net. This is a multi-camera network predicting directly in vector space, and you can see that it's basically night and day. You can actually drive on this, and this took some time and some engineering and incredible work from the AI team to actually get this to work and deploy and make it efficient in the car. This also improved a lot of our object detection. So for example, here in this video, I'm showing single-camera predictions in orange and multi-camera predictions in blue. And basically, if you can't predict these cars if you are only seeing a tiny sliver of a car, so your detections are not going to be very good and their positions are not going to be good. But a multi-camera network does not have an issue. Here's another video from a more nominal sort of situation, and we see that as these cars in this tight space cross camera boundaries, there's a lot of jank that enters into the predictions, and basically the whole setup just doesn't make sense, especially for very large vehicles like this one. And we can see that the multi-camera networks struggle significantly less with these kinds of predictions. Okay, so at this point we have multi-camera networks and they're giving predictions directly in vector space, but we are still operating at every single instant in time completely independently. So very quickly we discovered that there's a large number of predictions we want to make that actually require the video context, and we need to somehow figure out how to feed this into the net. So in particular, is this car parked or not? Is it moving? How fast is it moving? Is it still there even though it's temporarily occluded? Or for example, if I'm trying to predict the road geometry ahead, it's very helpful to know of the signs or the road markings that I saw 50 meters ago. So we try to develop... we try to insert video modules into our neural network architecture, and this is kind of one of the solutions that we've converged on. So we have the multi-scale features as we had them from before, and what we are going to now insert is a feature queue module that is going to cache some of these features over time, and then a video module that is going to fuse this information temporally, and then we're going to continue into the heads that do the decoding. Now, I'm going to go into both of these blocks one by one. Also, in addition, notice here that we are also feeding in the kinematics. This is basically the velocity and the acceleration that's telling us about how the car is moving. So not only are we going to keep track of what we're seeing from all the cameras, but also how the car has traveled. So here's the feature queue and the rough layout of it. We are basically concatenating these features over time and the kinematics of how the car has moved, and the positional encodings, and that's being concatenated, encoded, and stored in a feature queue. And that's going to be consumed by a video module. Now, there's a few details here again to get right. So in particular, with respect to the pop and push mechanisms and when do you push and how, and especially when do you push, basically. So here's a cartoon diagram illustrating some of the challenges here. There's going to be the ego car coming from the bottom and coming up to this intersection here, and then traffic is going to start crossing in front of us, and it's going to temporarily start occluding some of the cars ahead, and then we're going to be stuck at this intersection for a while and just waiting our turn. This is something that happens all the time and is a cartoon representation of some of the challenges here. So number one, with respect to the feature queue and when we want to push into a queue, obviously we'd like to have some kind of a time-based queue where, for example, we enter the features into the queue say every 27 milliseconds. And so if a car gets temporarily occluded, then the neural network now has the power to be able to look and reference the memory in time and learn the association that hey, even though this thing looks occluded right now, there's a record of it in my previous features, and I can use this to still make a detection. So that's kind of like the more obvious one. But the one that we also discovered is necessary in our case is, for example, suppose you're trying to make predictions about the road surface and the road geometry ahead, and you're trying to predict that I'm in a turning lane and the lane next to us is going straight. Then it's really necessary to know about the line markings and the signs, and sometimes they occur a long time ago. And so if you only have a time-based queue, you may forget the features while you're waiting at your red light. So in addition to a time-based queue, we also have a space-based queue. So we push every time the car travels a certain fixed distance. So some of these details actually can matter quite a bit. And so in this case, we have a time-based queue and a space-based queue to cache our features, and that continues into the video module. Now, for the video module, we looked at a number of possibilities of how to fuse this information temporally. So we looked at three-dimensional convolutions, transformers, axial transformers in an effort to try to make them more efficient, recurrent neural networks over a large number of flavors. But the one that we actually like quite a bit as well, and I want to spend some time on, is a spatial recurrent neural network video module. And so what we're doing here is, because of the structure of the problem, we're driving on two-dimensional surfaces, we can actually organize the hidden state into a two-dimensional lattice. And then as the car is driving around, we update only the parts that are near the car and where the car has visibility. So as the car is driving around, we are using the kinematics to integrate the position of the car in the hidden features grid, and we are only updating the RNN at the points that are nearby us, sort of. So here's an example of what that looks like. Here, what I'm going to show you is the car driving around, and we're looking at the hidden state of this RNN. And these are different channels in the hidden state. So you can see that this is after optimization and training this neural net, you can see that some of the channels are keeping track of different aspects of the road, like for example, the centers of the road, the edges, the lines, the road surface, and so on. Here's another cool video of this. So this is looking at the mean of the first 10 channels in the hidden state for different traversals of different intersections. And all I want you to see basically is that there's cool activity as the recurrent neural network is keeping track of what's happening at any point in time. And you can imagine that we've now given the power to the neural network to actually selectively read and write to this memory. So for example, if there's a car right next to us and is occluding some parts of the road, then now the network has the ability to not write to those locations. But when the car goes away and we have a really good view, then the recurrent neural net can say, okay, we have very clear visibility, we definitely want to write information about what's in that part of space. Here's a few predictions that show what this looks like. So here we are making predictions about the road boundaries in red, intersection areas in blue, road centers, and so on. So we're only showing a few of the predictions here just to keep the visualization clean. And yeah, this is done by the spatial RNN, and this is only showing a single clip, a single traversal, but you can imagine there could be multiple trips through here, and basically a number of cars, a number of clips could be collaborating to build this map basically, and effectively an HD map, except it's not in a space of explicit items, it's in a space of features of a recurrent neural network, which is kind of cool. I haven't seen that before. The video networks also improved our object detection quite a bit. So in this example, I want to show you a case where there are two cars over there, and one car is going to drive by and occlude them briefly. So look at what's happening with the single frame and the video predictions as the cars pass in front of us. Yeah, so that makes a lot of sense. So a quick playthrough through what's happening: when both of them are in view, the predictions are roughly equivalent, and you are seeing multiple orange boxes because they're coming from different cameras. When they are occluded, the single-frame networks drop the detection, but the video module remembers it, and we can persist the cars. And then when they are only partially occluded, the single-frame network is forced to make its best guess about what it's seeing, and it's forced to make a prediction, and it makes a really terrible prediction. But the video module knows that there's only a partial... that you know, it has the information and knows that this is not a very easily visible part right now, and doesn't actually take that into account. We also saw significant improvements in our ability to estimate depth, and of course, especially velocity. So here I'm showing a clip from our remove-the-radar push where we are seeing the radar depth and velocity in green, and we were trying to match or even surpass, of course, the signal just from video networks alone. And what you're seeing here is in orange, we are seeing single-frame performance, and in blue, we are seeing again video modules. And so you see that the quality of depth is much higher, and for velocity, the orange signal, of course, you can't get velocity out of a single-frame network, so we just differentiate depth to get that. But the video module actually is basically right on top of the radar signal, and so we found that this worked extremely well for us. So here's putting everything together. This is what our architecture roughly looks like today. So we have raw images feeding on the bottom. They go through a rectification layer to correct for camera calibration and put everything into a common virtual camera. We pass them through RegNets, residual networks, to process them into a number of features at different scales. We fuse the multi-scale information with BiFPN. This goes through a transformer module to re-represent it into the vector space in the output space. This feeds into a feature queue in time or space that gets processed by a video module like the spatial RNN, and then continues into the branching structure of the HydraNet with trunks and heads for all the different tasks. And so that's the architecture, roughly what it looks like today. And on the right, you are seeing some of its predictions sort of visualized both in a top-down vector space and also in images. So definitely, this architecture has definitely complexified from just a very simple image-based single network about three or four years ago, and continues to evolve. It's definitely quite impressive. Now, there's still opportunities for improvements that the team is actively working on. For example, you'll notice that our fusion of time and space is fairly late in neural network terms. So maybe we can actually do earlier fusion of space or time and do, for example, cost volumes or optical flow-like networks on the bottom. Or for example, our outputs are dense rasters, and it's actually pretty expensive to post-process some of these dense rasters in the car, and of course, we are under very strict latency requirements, so this is not ideal. We actually are looking into all kinds of ways of predicting just the sparse structure of the road, maybe like, you know, point by point or in some other fashion that doesn't require expensive post-processing. But this basically is how you achieve a very nice vector space. And now I believe Ashok is going to talk about how we can run planning and control on top of it.

All right, thanks Ashok. Hey everyone, I'm Milan. I'm responsible for the integration of our neural networks in the car and for most of our neural network training and evaluation infrastructure. And so tonight, I'd just like to start by giving you some perspective into the amount of compute that's needed to power this type of data generation factory. And so in the specific context of the push we went through as a team here a few months ago to get rid of the dependency on the radar sensor for the pilot, we generated over 10 billion labels across two and a half million clips. And so to do that, we had to scale our huge offline neural networks and our simulation engine across thousands of GPUs and just a little bit shy of 20,000 CPU cores. On top of that, we also included over 2,000 actual Autopilot Full Self-Driving computers in the loop with our simulation engine. And that's our smallest compute cluster.

So I'd like to give you some idea of what it takes to take our neural networks and move them in the car. And so the two main constraints that we're working on there here are mostly latency and frame rate, which are very important for safety, but also to get proper estimates of acceleration and velocity of our surroundings. And so the meat of the problem really is around the AI compiler that we write and extend here within the group that essentially maps the compute operations for a PyTorch model to a set of dedicated and accelerated pieces of hardware. And we do that while figuring out a schedule that's optimized for throughput while working under severe SRAM constraints. And so, by the way, we're not doing that just on one engine but across two engines on the Autopilot computer. And the way we use those engines here at Tesla is such that at any given time, only one of them will actually output control commands to the vehicle, while the other one is used as an extension of compute. But those roles are interchangeable both at the hardware and software level.

So how do we interact quickly together as a group through these AI development cycles? Well, first, we have been scaling our capacity to evaluate our software neural network dramatically over the past few years, and today we're running over a million evaluations per week on any code change that the team is producing. And those evaluations run on over 3,000 actual Full Self-Driving computers that are hooked up together in a dedicated cluster. And so on top of this, we've been developing really cool debugging tools. And so here is a video of one of our tools which is helping developers iterate through the development of neural networks and comparing live the outputs from different revisions of the same neural network model as we're iterating live through video clips. And so last but not least, we've been scaling our neural network training compute dramatically over the past few years, and today we're barely shy of 10,000 GPUs, which just to give you some sense in terms of number of GPUs, is more than the top five publicly known supercomputers in the world. But that's not enough, and so I'd like to invite Ganesh to talk about the next steps.

Thank you, Milan. My name is Ganesh, and I lead Project Dojo. It's an honor to present this project on behalf of the multidisciplinary Tesla team that is working on this project. As you saw from Milan, there's an insatiable demand for speed as well as capacity for neural network training. And Elon prefetched this, and a few years back, he asked us to design a super-fast training computer, and that's how we started Project Dojo. Our goal is to achieve best AI training performance and support all these larger, more complex models that Andre's team is dreaming of, and be power-efficient and cost-effective at the same time. So we thought about how to build this, and we came up with a distributed compute architecture. After all, all the training computers out there are distributed computers in one form or the other. They have compute elements in the box out here connected with some kind of network. In this case, it's a two-dimensional network, but it could be any different network. CPU, GPU, accelerators, all of them have compute, little memory, and network. But one thing which is a common trend amongst this is it's easy to scale the compute, it's very difficult to scale up bandwidth, and extremely difficult to reduce latencies. And you'll see how our design point catered to that, how our philosophy addressed these aspects of traditional limits.

For Dojo, we envisioned a large compute plane filled with very robust compute elements, backed with a large pool of memory, and interconnected with very high bandwidth and low latency fabric in a 2D mesh format. And onto this, for extreme scale, big neural networks will be partitioned and mapped to extract different parallelism: model, graph, data parallelism. And then a neural compiler of ours will exploit spatial and temporal locality such that it can reduce communication footprint to local zones and reduce global communication. And if we do that, our bandwidth utilization can keep scaling with the plane of compute that we desire out here. We wanted to attack this all the way top to the bottom of the stack and remove any bottlenecks at any of these levels. And let's start this journey in an inside-out fashion, starting with the chip. As I described, chips have compute elements. Our smallest entity or scale is called a training node. And the choice of this node is very important to ensure seamless scaling. If you go too small, it will run fast, but the overheads of synchronization and software will dominate. If you pick it too big, it will have complexities in implementation in the real hardware and ultimately run into memory bottleneck issues. Because we wanted to address latency and bandwidth as our primary optimization point, let's see how we went about doing this. What we did was we picked the farthest distance a signal could traverse in a very high clock cycle, in this case, two gigahertz plus, and we drew a box around it. This is the smallest latency that a signal can traverse one cycle at a very high frequency. And then we filled up the box with wires to the brink. This is the highest bandwidth you can feed the box with. And then we added machine learning compute underneath, and then a large pool of SRAM, and last but not the least, a programmable core to control. And this gave us our high-performance training node.

What this is is a 64-bit superscalar CPU optimized around matrix multiply units and vector compute. It supports floating point 32, bfloat 16, and a new format, CFP8, configurable FP8. And it is backed by one and a quarter megabyte of fast ECC-protected SRAM and the low-latency, high-bandwidth fabric that we designed. This might be our smallest entity of scale, but it packs a big punch: more than one teraflop of compute in our smallest entity of scale. So let's look at the architecture of this. The computer architects out here may recognize this; this is a pretty capable architecture. As soon as you see this, it is a superscalar in-order CPU with four-wide vector and two-wide vector, two-wide, four-wide scalar and two-wide vector pipes. We call it in-order, although the vector and the scalar pipes can go out of order, but for the purists out there, we still call it in-order. And it also has four-way multi-threading. This increases utilization because we could do compute and data transfers simultaneously. And our custom ISA, which is the instruction set architecture, is fully optimized for machine learning workloads. It has features like transpose, gather, link traversals, broadcast, just to name a few. And even in the physical realm, we made it extremely modular such that we could start arranging these training nodes in any direction and start forming the compute plane that we envisioned.

When we click together 354 of these training nodes, we get our compute array. It's capable of delivering petaflops of machine learning compute, and of course, the high-bandwidth fabric that interconnects these. And around this compute array, we surrounded it with high-speed, low-power serializers, 576 of them, to enable us to have extreme I/O bandwidth coming out of this chip. Just to give you a comparison point, this is more than two times the bandwidth coming out of the state-of-the-art networking switch chips which are out there today. And network switch chips are supposed to be the gold standards for I/O bandwidth. If we put all of it together, we get a training-optimized chip, our D1 chip. This chip is manufactured in 7-nanometer technology. It packs 50 billion transistors in a miserly 645 millimeter square. One thing you'll notice, 100% of the area out here is going towards machine learning training and bandwidth. There is no dark silicon, there is no legacy support. This is a pure machine learning machine. And this is the D1 chip in a flip-chip BGA package. This was entirely designed by the Tesla team internally, all the way from the architecture to GDS out and package. This chip is like a GPU-level compute with a CPU-level flexibility and twice the network chip-level I/O bandwidth. If I were to plot the I/O bandwidth on the vertical scale versus teraflops of compute that is available in the state-of-the-art machine learning chips out there, including some of the startups, you can easily see why our design point excels beyond power.

Now that we had this fundamental physical building block, how to design the system around it? Let's see. Since D1 chips can seamlessly connect without any glue to each other, we just started putting them together. We just put 500,000 training nodes together to form our compute plane. This is 1,500 D1 chips seamlessly connected to each other. And then we add Dojo Interface Processors on each end. This is the host bridge to typical hosts in the data centers. It's connected with PCIe Gen4 on one side, with a high-bandwidth fabric to our compute plane. The interface processors provide not only the host bridge but high-bandwidth DRAM shared memory for the compute plane. In addition, the interface processors can also allow us to have a higher-radix network connection. In order to achieve this compute plane, we had to come up with a new way of integrating these chips together. And this is what we call as a training tile. This is the unit of scale for our system. This is a groundbreaking integration of 25 known-good D1 dies onto a fan-out wafer process, tightly integrated such that it preserves the bandwidth between them. The maximum bandwidth is preserved there. And in addition, we generated a connector, a high-bandwidth, high-density connector that preserves the bandwidth coming out of this training tile. And this tile gives us nine petaflops of compute with a massive I/O bandwidth coming out of it. This perhaps is the biggest organic MCM in the chip industry, multi-chip module. It was not easy to design this. There were no tools that existed. All the tools were croaking. Even our compute cluster couldn't handle it. Our engineers came up with different ways of solving this. They created new methods to make this a reality.

Now that we had our compute plane tile with high-bandwidth I/Os, we had to feed it with power. And here we came up with a new way of feeding power vertically. We created a custom voltage regulator module that could be reflowed directly onto this fan-out wafer. So what did we do out here is we got chip package and we brought PCB-level technology of reflow on fan-out wafer technology. This is a lot of integration already out here, but we didn't stop here. We integrated the entire electrical, thermal, and mechanical pieces out here to form our training tile, fully integrated, interfacing with a 52-volt DC input. It's unprecedented. This is an amazing piece of engineering. Our compute plane is completely orthogonal to power supply and cooling. That makes high-bandwidth compute planes possible. What it is is a nine-petaflop training tile. This becomes our unit of scale for our system. And this is real.

I can't believe I'm holding nine petaflops out here. And in fact, last week we got our first functional training tile, and on a limited, limited cooling benchtop setup, we got some networks running. And I was told Andre doesn't believe that we could run networks till we could run one of his creations. This is MiniGPT-2 running on Dojo. Do you believe it?

Next up, how to form a compute cluster out of it. By now, you must have realized our modularity story is pretty strong. We just put together some tiles. We just tile together tiles. A two-by-three tile in a tray makes our training matrix, and two trays in a cabinet give 100 petaflops of compute. Did we stop here? No. We just integrated seamlessly. We broke the cabinet walls. We integrated these tiles seamlessly all the way through, preserving the bandwidth. There is no bandwidth drop out here. There is no bandwidth clips. All the tiles are seamlessly connected with the same bandwidth. And with this, we have an ExaPod. This is one exaflop of compute in 10 cabinets. It's more than a million training nodes that you saw. We paid meticulous attention to that training node, and there are one million nodes out here with uniform bandwidth.

Not just the hardware, the software aspects are so important to ensure scaling. And not every job requires a huge cluster, so we planned for it right from the get-go. Our compute plane can be subdivided, can be partitioned into units called Dojo Processing Unit, a DPU. A DPU consists of one or more D1 chips. It also has our interface processor and one or more hosts. And this can be scaled up or down as per the needs of any algorithm, any network running on it. What does the user have to do? They have to change their scripts minimally. And this is because of our strong compiler suite. It takes care of fine-grained parallelism and mapping the problems of mapping the neural networks very efficiently onto our compute plane. Our compiler uses multiple techniques to extract parallelism. It can transform the networks to achieve not only fine-grained parallelism using data, model, graph parallelism techniques, it also can do optimizations to reduce memory footprints. One thing, because of our high-bandwidth nature of the fabric that is enabled out here, is model parallelism could not have been extended to the same level as what we can. It was limited to chip boundaries. Now we can, because of our high bandwidth, we can extend it to training tiles and beyond. Thus, large networks can be efficiently mapped here at low batch sizes and extract utilization and new levels of performance. In addition, our compiler is capable of handling high-level dynamic control flows like loops, if-then-else, etc. And our compiler engine is just part of our entire software suite. The stack consists of an extension to PyTorch that ensures the same user-level interfaces that ML scientists are used to. And our compiler generates code on the fly such that it could be reused for subsequent execution. It has an LLVM backend that generates the binary for the hardware, and this ensures we can create optimized code for the hardware without relying on even a single line of handwritten kernel. Our driver stack takes care of the multi-host, multi-partitioning that you saw a few slides back. And then we also have profilers and debuggers in our software stack.

So with all this, we integrated in a vertical fashion. We broke the traditional barriers to scaling, and that's how we got modularity up and down the stack to add to new levels of performance. To sum it all, this is what it will be: it will be the fastest AI training computer, 4x the performance at the same cost, 1.3x better performance per watt, that is, energy saving, and 5x smaller footprint. This will be the Dojo computer.

We are not done. We are assembling our first cabinets pretty soon, and we have a whole next-generation plan already. We are thinking about 10x more with different aspects that we can do, all the way from silicon to the system. Again, we will have this journey again. We're recruiting heavily for all of these areas. Thank you very much. And next up, Elon will update us on what's beyond our vehicle fleet for AI.

Now, unlike Dojo, obviously that was not real. So Dojo is real. The Tesla Bot will be real. But basically, if you think about what we're doing right now with the cars, Tesla is arguably the world's biggest robotics company because our cars are like semi-sentient robots on wheels. And with the Full Self-Driving computer, essentially the inference engine on the car, which will keep evolving obviously, and Dojo, and all the neural nets recognizing the world, understanding how to navigate through the world, it kind of makes sense to put that onto a humanoid form. They're also quite good at sensors and batteries and actuators. So we think we'll probably have a prototype sometime next year that basically looks like this. And it's intended to be friendly, of course, and navigate through a world built for humans, and eliminate dangerous, repetitive, and boring tasks. We're setting it such that it is, at a mechanical level, at a physical level, you can run away from it and most likely overpower it. So hopefully that doesn't ever happen, but you never know. So it's, it'll be a, you know, a light, a light, yeah, five miles an hour, you can run faster than that, you'd be fine. So yes, it's around 5.8. Has sort of a screen where the head is for useful information, but otherwise basically got the Autopilot system in it. So it's got cameras, got eight cameras, and Full Self-Driving computer, and making use of all of the same tools that we use in the car. So, I mean, things I think that are really hard about having a useful humanoid robot is it cannot navigate through the world without being explicitly trained, I mean, without explicit, like, line-by-line instructions. Can you talk to it and say, you know, please pick up that bolt and attach it to the car with that wrench, and it should be able to do that. It should be able to, you know, please go to the store and get me the following groceries, that kind of thing. So, yeah, I think we can do that. And, yeah, this I think will be quite, quite profound because if you say like, what is the economy? It is, at the foundation, it is labor. So what happens when there is, you know, no shortage of labor? That's why I think long-term that there will need to be universal basic income. Yeah. But, but not right now because this robot doesn't work, so we just need a minute. So, yeah, but I think essentially in the future, physical work will be a choice. If you want to do it, you can, but you won't need to do it. And, yeah, I think it obviously has profound implications for the economy because given that the economy at its foundational level is labor, I mean, capital is capital equipment, it's just distilled labor, then is there any actual limit to the economy? Maybe not. So, yeah, join our team and help build this. All right, so I think we'll have everyone come back on the stage and you guys can ask questions if you'd like.

All right, cool. You guys brought me back on. I just like stage. Try to get the camera angle from there, but not from the side. All right, cool. So we'll probably turn the lights back on. And, yeah, great. So we're happy to answer any questions you have about anything on the software, hardware side, where things are going. And yeah, fire away. Um, we have, because the lights are like interrogation lights, so we actually cannot see. Ah, there we go, great. All right, cool.

Um, well, I mean, it is fundamentally extremely expensive to create the system, so somehow that has to be paid for. I'm not sure how to pay for it if it's fully open-sourced. Um, yeah, unless people want to work for free. But, but I should say that this is, if other car companies want to license it and use it in their cars, that would be cool. This is not intended to be just limited to Tesla cars.

Have we solved the problem? Not yet. Are we confident we will solve the problem? Yes. We have demonstrated networks on prototype hardware. Now we have models, performance models showing the scaling. The difficulty is, as you said, how do we keep the localities? If we can do enough model parallel, enough data parallel to keep most of the things local, we just keep scaling. We have to fit the parameters in our working set in our SRAM that we have, and we flow through the pipe. There's plenty of opportunities. Sorry, as we get further scale, for further processor nodes, have more local memory, memory trade also bandwidth, we can do more things. But as we see it now, the applications that Tesla has, we see a clear path. And our modularity story means we can have different ratios, different aspects created out of it. I mean, this is something that we chose for our applications internally.

Yeah, I guess the proof's in the pudding. Um, so we should have Dojo operational next year. Um, and, um, I think we'll obviously use it for, uh, training, video training. It's, I mean, fundamentally this is about, like, um, the, the...

The primary application initially is we've got vast amounts of video and how we train vast amounts of video as efficiently as possible, and also shorten the amount of time. If you're trying to train to a task, innovation is how many iterations and what is the average progress between each iteration. If you can reduce the time between iterations, the rate of improvement is much better. If it takes a couple days for a model to train versus a couple hours, that's a big deal. The asset test here, and what I've told the Dojo team, is it's successful if the software team wants to turn off the GPU cluster. But if they want to keep the GPU cluster on, it's not successful.

Hi, right over here. I love the presentation, thank you for getting us out here. Loved everything, especially the simulation part of the presentation. It looked very realistic. Are there any plans to maybe expand simulation to other parts of the company in any way?

Hi, I'm Ian Glow, I manage the Autopilot simulation team. As we go down the path to full self-driving, we're going to have to simulate more and more of the vehicle. Currently we're simulating vehicle dynamics, but we're going to need BMS, we're going to need the MCU, we're going to need every single part of the vehicle integrated. That actually makes the Autopilot simulator really useful for places outside of Autopilot. We want to expand eventually to being a universal simulation platform. But before that, we're going to be spinning up a lot of Optimus support, and then a little bit further down the line, we have some rough ideas on potentially how to get the simulation infrastructure and some of the cool things we've built into the hands of people outside of the company. Optimus is the code name for the Tesla Bot.

Hi, this is Ali Jahanian. Thank you for the great presentation and putting all of these cool things together. For a while I have been thinking that the car is already a robot, so why not a humanity robot? I'm so happy that today you mentioned that you are going to build such a thing. This can give opportunities for putting multi-modality together. For instance, in the example you showed with a dog and passengers running together, language and symbolic processing can really help for visualizing that. I was wondering if I could hear a little more about putting modalities together, including language and vision, because I have been working with MiniGPTs and Andre put out there, and I didn't hear much about other modalities going into the car or at least in the simulation. Is there any comment you could tell us?

Well, driving is fundamentally almost entirely vision neural nets. It's running on a biological vision neural net, and what we're doing here is a silicon camera neural net. There is some amount of audio. You want to hear if there's emergency vehicles, or converse with the people in the car. If somebody's yelling something at the car, the car needs to understand what that is. All things that are necessary for it to be fully autonomous.

Hi, thank you for all the great work that you've shown. My question is for the team. The data shown seems to be predominantly from the United States that the FSD computer is being trained on. But as it gets rolled out to different countries which have their own road systems and challenges, how do you think that it's going to scale? I'm assuming a ground-up approach is not a very viable solution, so how does it transfer to different countries?

We actually do train using data from probably like 50 different countries. But we have to pick one country, and since we're located here, we pick the US. We get a lot of questions like, 'Why not even Canada?' Well, because the roads are a little different in Canada, different enough. When trying to solve a hard problem, you want to say, 'Let's not add additional complexity right now. Let's just solve it for the US and then we'll extrapolate to the rest of the world.' But we do use video from all around the world.

Yeah, I think a lot of what we are building is very country-agnostic. Fundamentally, all the computer vision components and so on don't care too much about country-specific features. Different countries have roads and they have curbs and they have cars, and everything we're building is fairly general for that.

Yeah, and the prime directive is don't crash, right? And that's true for every country. Yes, this is the prime directive. Even right now the car is pretty good at not crashing. Whatever it is, don't hit it. Even if it's a UFO that crash-landed on the highway, still don't hit it. You should not need to recognize it in order to not hit it. So that's very important.

I want to ask that when you do the photometric process, multiview geometry, how much of an error do you see? Is it like one millimeter, one centimeter? What is the difference between the synthetically created geometry to the actual geometry?

Do you merge with different kinds of modalities to bring down that error?

Yeah, I mean, I think we want to aim for the car to be positioned accurately to the sort of centimeter level. Something on that order. Obviously it will depend on distance. Close by things can be much more accurate than farther away things, and they would matter less because the car doesn't have to make decisions much farther away. As it comes close, it will become more and more accurate.

A lot of questions. Thanks everybody. My question has to do with AI and manufacturing. It's been a while since we've heard about the Alien Dreadnought concept. Is the humanoid behind you guys brought out of the production health timeline, saying that humans are underrated in that process?

Sometimes something that I say is taken to too much of an extreme. There are parts of the Tesla system that are almost completely automated, and then there are some parts that are almost completely manual. If you were to walk through the whole production system, you would see a very wide range from fully automatic to almost completely manual. But the vast majority, most of it, is already automated. With some of the design architecture changes, like going to large aluminum high-pressure die-cast components, we can take the entire rear third of the car and cast it as a single piece. Now we're going to do that for the front third of the car as a single piece. So the body line drops by like 60 to 70 percent in size. The robot is not prompted specifically by manufacturing needs. It's just that we're obviously making the pieces that are needed for a useful humanoid robot. So I guess we probably should make it, and if we don't, someone else would. So I guess we should make it and make sure it's safe. I should say, volume manufacturing is extremely difficult and underrated, and we've gotten pretty good at that. It's also important for that humanoid robot, like how do you make the humanoid robot not be super expensive.

Hi, thank you for the presentation. My question will be about scaling of Dojo, and in particular, how do you scale the compute nodes in terms of thermals and power delivery? Because there is only so much heat that you can dispense and only so much power that you can bring to a cluster rack. How do you want to scale it, and how do you plan to scale it in multiple data centers?

Hi, I'm Bill, one of the Dojo engineers. From a thermal standpoint and power standpoint, we've designed it very modular. What you saw on the compute tile, that will cool the entire tile. Once we hook it up, it is liquid cooled on both the top and the bottom side. It doesn't need anything else. When we talk about clicking these together, once we click it to power and once we click it to cooling, it will be fully powered and fully cooled. And all of that is less than a cubic foot.

Yeah. Tesla has a lot of expertise in power electronics and in cooling. We took the power electronics expertise from the vehicle powertrain and the advanced cooling that we developed for the power electronics and for the vehicle and applied that to the supercomputer. Because as you point out, getting heat out is extremely important. It's really heat-limited. It's funny that at the compute level it's operating at less than a volt, which is a very low voltage. There's a lot of amps, so therefore a lot of heat. I squared R is what really bites you on the ass.

Hi, my question is also similarly a question of scaling. It seems like a natural consequence of using significantly faster training hardware is that you'd be either training models over a lot more data, or you'd be training a lot more complex models which would be potentially significantly more expensive to run at inference time on the cars. I was wondering if there was a plan to also apply Dojo as something that you'd be using on the self-driving cars, and if so, do you foresee additional challenges there?

Yeah, I mean, Dojo's first application will be consuming video data for training that would then be run in the inference engine on the car. That is an important test to see if it actually is better than a GPU cluster or not. But then beyond that, it's basically a generalized neural net training computer. It's very much optimized to be a neural net. CPUs and GPUs, they're not designed specifically for training neural nets. We've been able to make GPUs especially very efficient for training neural nets, but that was never their design intent. GPUs are still essentially running neural net training in emulation mode. With Dojo, we're saying, let's just ASIC the whole thing. Let's just have this thing that's built for one purpose, and that is neural net training. Any system that is designed for a specific purpose will be better than one that is designed for a general purpose.

Hey, I had a question here. You described two separate systems, one for vision and therefore planner and control. Does Dojo allow you to train networks that cross that boundary? And second, if you were able to train such networks, would you have the onboard compute capability in the FSD system to be able to run that under your tight latency constraints?

Yep, certainly. End-to-end is the guiding principle behind a lot of the network developments. Over time in the stack, neural networks have taken on more and more functionality. We want everything to be trained end-to-end because we see that that works best. But we are building it incrementally. Right now the interface there is vector space and we are consuming it in the planner, but nothing really fundamentally prevents you from actually taking features and eventually fine-tuning end-to-end. I think that's definitely where this is headed.

Okay, I think I had a question very similar to what he was asking about. It seems like a lot of neural nets are around computer vision and kind of traditional planning. You had model predictive control in solving convex optimization problems very quickly. I'd wonder if there's a compute architecture that's more suited for convex optimization or the model predictive control solutions very quickly.

I mean, I do think that as long as you've got surround video neural nets for understanding what's going on and can convert those surround video into vector space, then you basically have a video game. If you're in Grand Theft Auto, whatever, you can make the cars drive around and pedestrians walk around without crashing. You don't have to have a neural net for control and planning. But it's probably ultimately better. I think you can probably get to, in fact I'm sure you can get to much safer than human with control and planning primarily in C++ with perception vision in neural nets.

Hi, my question is, we've seen other companies for example use reinforcement learning and machine learning to optimize power consumption in data centers and all kinds of other internal processes. My question is, is Tesla using machine learning within its manufacturing design or other engineering processes?

I discourage use of machine learning because it's really difficult. Unless you basically have to use machine learning, don't do it. It's usually a red flag when somebody says, 'We want to use machine learning to solve this task.' I'm like, that sounds like... 99.9 percent of the time you do not need it. You reach for machine learning when you need to, but I've not found it to be a convenient, easy thing to do. It's a super hard thing to do. That may change if you've got a humanoid robot that can understand normal instructions. But generally, minimize use of machine learning in the factory.

Hi, based on your videos from the simulator, it looked like a combination of graphical and neural approaches. I'm curious what the set of underlying techniques are for your simulator, and specifically for neural rendering, if you can share.

Yeah, so we're doing at the bottom of the stack just traditional game techniques, just rasterization, real-time, very similar to what you'd see in GTA. On top of that, we're doing real-time ray tracing. And then those results were really hot off the press. We had that little asterisk at the bottom that that was from last night. We're going into the neural rendering space. We're trying out a bunch of different things. We want to get to the point where the neural rendering is the cherry on the top that pushes it to the point where the models will never be able to overfit on our simulator. Currently we're doing things similar to photorealism enhancement. There's a recent paper, 'Enhancing Photorealism Enhancement,' but we can do a lot more than what they could do in that paper because we have way more labeled data, way more compute, and we have a lot more control over environments. We also have a lot of people who can help us make this run at real time. We're going to try whatever we can do to get to the point where we can train everything just with the simulator if we had to, but we will never have to because we have so much real-world data that no one else has. It's just to fill in the little gaps in the real world.

Yeah, I mean, the simulator is very helpful when there's these rare cases, like collision avoidance right before an accident. Ironically, the better our cars become at avoiding accidents, the fewer accidents there are, so then our training set is small. So then we have to make them crash in the simulation. It's like, okay, minimize potential injury to pedestrians and people in the car. You have five meters, you're traveling at 20 meters per second, what actions would minimize probability of injury? We can run that in simulation. Cars driving down the wrong side of the highway, that kind of thing happens occasionally but not that often.

For your humanoid context, I'm wondering if you've decided on what use cases you're going to start with and what the grand challenges are in that context to make this viable.

Well, I think for the humanoid, for the Tesla Bot, Optimus, it's basically going to start with just dealing with work that is boring, repetitive, and dangerous. Basically, what is the work that people would least like to do.

Hi, so quick question about your simulations. Obviously they're not perfect right now, so are you using any sort of domain adaptation techniques to basically bridge the gap between your simulated data and your actual real-world data? Because I imagine it's kind of dangerous to just deploy models which are solely trained on simulated data. So maybe some sort of explicit domain adaptation or something, is that going on anywhere in your pipeline?

So currently, we're producing the videos straight out of the simulator, the full clips of kinematics and everything, and then we're just immediately training on them. But it's not the entire dataset, it's just a small targeted segment, and we only are evaluating based on real-world video. We're paying a lot of attention to make sure we don't ever overfit. If we have to start doing fancier things, we will, but currently we're not having an issue with it overfitting on the simulator. We will as we scale up the data, and that's what we're hoping to use neural rendering to bridge that gap, to push that even further out. We've already done things where we're using the same networks in the car but retrain it to detect sim versus real to drive our art decisions. That's actually helped prevent some of these things as well.

Yeah, just to emphasize that overwhelmingly the dataset is the real video from the cars on the actual roads. Nothing's weirder or has more corner cases than reality. It gets really strange out there. But then if we find a few examples of something very odd, and there are some very odd pictures we've seen, then in order to train it effectively, we want to create simulations, say a thousand simulations that are variants of that quirky thing that we saw, to fill in some important gaps and make the system better. Really all of this is about over time just reducing the probability of a crash or an injury. It's called the march of nines. How do you get to 99.999999% safe? Each nine is an order of magnitude difficulty increase.

Thanks so much for the presentation. I was curious about the Tesla Bot. Specifically, I'm wondering if there are any specific applications that you think the humanoid form factor lends itself to. And then secondary, because of its human form factor, is emotion or companionship at all thought about on the product roadmap at all?

Well, we certainly hope this does not feature in a dystopian sci-fi movie. But really at this point, we're saying maybe this robot can just, we're trying to be as literal as possible, can it do boring, dangerous, repetitive jobs that people don't want to do? Once you can have it do that, then maybe you can do other things too. But that's the thing that we really want to have. It could be your buddy too. Buy one and have it be your friend or whatever. I'm sure that people will think of some very creative uses.

So firstly, thanks for the really incredible presentation. My question is on the AI side. One thing we've been seeing is that with some of these language modeling AIs, we've seen that scaling has just had incredible impacts in their capabilities and what they're able to do. So I was wondering whether you're seeing similar kinds of effects of scaling in your neural networks in your applications.

Yeah, the auto-labeling is an extremely important part of this whole situation. Without the auto-labeling, I think we would not be able to solve the self-driving problem. It's kind of a funny form of distillation where you're using these very massive models plus the structure of the problem to do this reconstruction, and then you distill that into neural networks that you deploy to the car. But we basically have a lot of neural networks and a lot of tasks that are never intended to go into the car.

I think one of the things we're seeing is that the car's predictive ability is eerily good. It's really getting better than human in terms of predicting, like you say, predict what this road will look like when it's out of sight, like around the bend, and it predicts the road with very high accuracy. Predict pedestrians or cyclists where you just see a little corner of the bicycle and a little bit through the windows of the bus, and its ability to predict things is going to be much better than humans, really way, way beyond.

I have a question about the design of the Tesla Bot specifically. How important is it to maintain that humanoid form? To build hands with five fingers that also respects the weight limits could be quite challenging. You might have to use cable-driven and then that also causes all kinds of issues.

I mean, this is just going to be bot version one and we'll see. It needs to be able to do things that people do and be a generalized humanoid robot. You could potentially give it two fingers and a thumb or something like that. For now, we'll give it five fingers and see if that works out okay. It probably will. It doesn't need to have incredible grip strength, but it needs to be able to work with tools and carry a bag, that kind of thing.

All right, thanks a lot for the presentation. An old professor of mine told me that the thing he disliked a lot about his Tesla was that the Autopilot UX didn't really inspire much confidence in the system, especially when objects are spinning and classifications are flickering. I was wondering, even if you have a good self-driving system, how are you working on convincing Tesla owners, other road users, and the general public that your system is safe and reliable?

Well, I think that's... the cars, a while back, cars used to spin. They don't spin anymore. If you've seen the FSD Beta videos, they're pretty solid. And they will be getting more solid as we add more and more data and train these multi-camera networks. These are pretty decent actually, just a few months old, and it's really improving. It's not a done product. We've never... we can clearly see how this is just going to be perfect for space, because why not? All the information is in the videos. It should produce given lots of data and good architectures. This is just an intermediate point in the timeline. It's clearly headed to way better than human, without question.

Hi here. I was wondering if you could talk a little bit about the short to medium-term economics of the bot. I understand the long-term vision of replacing physical labor, but I also think repetitive, dangerous, and boring tasks tend to not be so highly compensated. I just don't see how to reproduce, start with a supercar and then break into the lower end of the market. How do you do that for a humanoid robot?

Well, I guess you'll just have to see.

Hello, hi. I was curious to know how the car AI prioritizes occupant safety versus pedestrian safety, and what thought process goes into deciding how to make this into the AI.

Well, I mean, the thing to appreciate is that from the computer standpoint, everything is moving slowly. To a human, things are moving fast. To the computer, they are not moving fast. I think this is in reality somewhat of a false dichotomy. Not that it will never happen, but it will be very rare. If you think about it the other direction, like rendering with full ray tracing, neural net enhanced graphics on something like Cyberpunk or any advanced video game, doing 60 frames a second, perfectly rendered, how long would it take a person to even render one frame without any mistakes? It would take like a month just to render one frame out of 60 in a second in a video game. Computers are fast and humans are slow. For example, on the rocket side, you cannot steer the rocket to orbit. We actually hooked up a joystick to see if anyone could steer the rocket to orbit, but you need to react at roughly six, seven hertz. People can't do it. Not even close. That's pretty low. We're talking more like even for 30 hertz type of thing.

Hi, over here. With Hardware 3, there's been lots of speculation that with larger nets it's hitting the limits of what it can provide. How much headroom has the extended compute modes provided? At what point would Hardware 4 be required, if at all?

Well, I'm confident that Hardware 3 or the Full Self-Driving Computer 1 will be able to achieve full self-driving at a safety level much greater than a human, probably at least two or three hundred percent better than a human. Then obviously there will be a future Hardware 4 or Full Self-Driving Computer 2, which we'll probably introduce with the Cybertruck, so maybe in about a year or so. That will be about four times more capable roughly. But it's really just going to be like, can we take it from say for argument's sake 300% safer than a person to 1000% safer? There are people on the road with varying driving abilities, but we still let people drive. You don't have to be the world's best driver to be on the road. So as we see... yeah.

Are you worried at all since you don't have any depth sensors on the car that people might try adversarial attacks, like printed out photos or something to try to trick the RGB neural network?

Yeah, like pull some Wile E. Coyote stuff, paint the tunnel on the wall. We haven't really seen much of that. For sure, right now if you had a t-shirt with a stop sign on it, which I actually have a t-shirt with a stop sign on it, and then you flash the car, it will stop. I proved that. But we can obviously, as we see these adversarial attacks, we can train the cars to notice that, well, it's actually a person wearing a t-shirt with a stop sign on it, so it's probably not a real stop sign.

Hi, my question is about the prediction and the planning. I'm curious, how do you incorporate uncertainty into your planning algorithms? Do you just basically assume, you mentioned that you run the autopilot for all the other cars on the road, do you assume that they're all going to follow those rules, or are you accounting for the possibility that they might be bad drivers, for example?

I should say, before we introduce something into the fleet, we will run it in shadow mode. We'll see what this neural net, for example, would have done in this particular situation. Effectively, the drivers are training the net. If the neural net would have controlled and veered right, but the person actually went left, it's like, oh, there's a difference. Why was there that difference?

It should be aspirationally that the car should be, the less it knows, the slower it goes. Which is not true at some point, but now... yeah, we've... yeah.

Thanks for the presentation. I am curious, I appreciate the fact that the FSD is improving, but if you have the ability to improve one component along the AI stack to present it today, whether it is simulation, data collection, planning, control, etc., which one in your opinion is going to have the biggest impact for the performance of the full self-driving system?

I mean, in the short term, it's arguably we need all of the nets to be surround video. We still have some legacy, this is very short term, obviously we're fixing it fast, but there's still some nets that are not using surround video. I think ideally that all use surround video.

Yeah, very... yeah, I think a lot of puzzle pieces are there for success. We just need more strong people to also just help us make it work.

Yeah, that is the actual bottleneck, I would say. That is one of the reasons that we are putting on this event. Exactly what Andrej said, there's just a tremendous amount of work to do to make it work. That's why we need talented people to join in and solve the problem.

Thank you for the great presentation, lots of my questions answered. But one thing is, imagine that now you have a large amount of data, even unnecessary. How do you consider that there's a forgetting problem in neural networks? How are you considering those aspects? And also, are you considering online learning or continuous learning, so that maybe each driver can have their version of self-driving software?

So let's take a few more questions here, and then we'll call it a day. As our models have become more and more capable, and I guess you're deploying these models into the real world, one thing that's possible is for AI to become more misaligned with what humans desire. Is that something that you guys are worried about as you guys deploy more and more robots? Or do you guys say, we'll solve that problem when we get there?

Yeah, I think that we should be worried about AI. What we're trying to do here is narrow AI, pretty narrow, like just make the car drive better than a human, and then have the humanoid robot be able to do basic stuff. At the point at which you sort of start to get superhuman intelligence, yeah, I don't know, all bets are off. But that'll probably happen. What we're trying to do here at Tesla is make useful AI that people love and is unequivocally good. That's our aim.

Okay, maybe one more question. Hi, my question is about the camera sensor. In the beginning of the talk you had mentioned about building a synthetic animal. If you think about it, a camera is a very poor approximation of a human eye, and a human eye does a lot more than take a sequence of frames. Have you looked into these days, there are cameras like event cameras. Have you looked into them, or are you looking into a more flexible camera design, or building your own camera, for example?

Well, with Hardware 4, we will have a next-generation camera. But I have to say that the current cameras, we have not reached the limit of the current cameras. I'm confident we can achieve full self-driving with much higher safety than humans with the current cameras and current compute hardware. But are we good to be 1000% better rather than 300% better? We'll see continued evolution on our levels and pursue that goal. I think in the future, people will look back and say, 'Wow, I can't believe we had to drive these cars ourselves.' Self-driving cars will just be normal, like self-driving elevators. Elevators used to have elevator operators. There's someone there with a big relay switch operating the elevator. Every now and then they'd get tired or make a mistake and shear somebody in half. So now we made elevators automatic. You just go and press the button and you can be in a 100-story skyscraper and don't really worry about it. Just go and press a button and the elevator takes you where you want to go. But it used to be that all elevators were operated manually. It'll be the same thing for cars. All cars will be automatic and electric, obviously. There will still be some gasoline cars and some manual cars, just like there are still some horses. All right, well thanks everyone for coming and I hope you enjoyed the presentation and thank you for the great questions.