Raquel Urtasun Uber

Hey, thank you very much. I'm really, really excited to be here today talking about self-driving cars. And what is a better way to actually finish an EIS summit than talking about self-driving? So my goal today is for you, when you go out of the room, to really understand everything and be an expert in self-driving. So let's see how I get there.

If you think about building self-driving vehicles at scale, there are three things that you really need to build. On one side, you need to actually build the hardware. You need to be able to get vehicles that are able to have rather a computer than a steering wheel, etc., because there is not going to be anyone that is going to be able to take over the vehicle. So you require new vehicles; they are not the standard vehicles that you have out there. On top of this, you need to build the brain of the self-driving car, which basically has to, given the output of the sensors, understand everything that is going on around in the world. And the third piece, which is typically actually forgotten, is that you need a network. The model of transportation is changing from having an ownership of cars to actually having fleets of cars where you can, with an app, basically call them. And for that, you need a way to route them over your network.

If you think about the vehicles out there, this is one example of the fleet of self-driving vehicles we have at Uber. They're basically like super cool robots. They have a whole bunch of sensors. They have this LiDAR spinning on the top. They have a whole bunch of cameras as well as radar, etc. And basically, they see around 360 degrees around the vehicle. On top of this, we have custom compute that is able to run a lot of AI stuff inside such that in real time you can do very complex inferences and reasoning about the world in front of you.

So in this video I'm showing, so you get a little bit of an understanding what these sensors actually see. You can see cameras, so they open up the cameras which basically are high-definition cameras seeing, as I say, 360. And on the other panel you basically see, seen from the top, what the LiDAR actually outputs: this point cloud of three-dimensional points that gives you a lot of information about the geometry and where things are in 3D space. You can notice that both sensors are actually very complementary. The cameras give you a lot of semantics and LiDAR gives you a lot of 3D information.

So if you think about how the industry is actually trying to tackle this problem, I'm going to tell you a little bit of how this is done with a few visuals. So you have the output of the sensors, in this case the LiDAR as you see here again being seen from the top. The first thing that we all do is that we localize the vehicle with precision of a few centimeters. And the reason that we do this is that we want to use our high-definition maps that contain a lot of very useful information encoded about what is static in the scene: where the lanes are, the traffic lights, the rules in this particular intersection, where the crosswalks, etc.

Once you have the vehicle localized, one of the things that is interesting is that this allows you to see much farther than what the sensors see. So the next thing that you typically do is that you take the output of the sensors and you say, okay, I want to understand where everyone is around me and what I see in front of me. I'm sorry, I ran all over it. And I'm showcasing this here with 3D boxes for vehicles and circles for pedestrians. And we need to do this in three-dimensional space. But now, understanding the present is not sufficient. We need to understand also the past and how the past has evolved into the present, because we need to understand how the different actors are potentially moving in the scene. And the past of the present is even not sufficient. We actually need to hallucinate how these things are going to evolve in the next few seconds, right? Because in order to plan a safe maneuver, we need to know: is this pedestrian going to cross in front of me? Is that a new walker? Is this vehicle going to potentially have a lane change or cut in front of me with high probability?

So we hallucinate probabilistically where all these things can be, and then basically the self-driving car can focus in an area around the vehicle and say, I want to plan a safe maneuver. Given the past, the present, and the future, then basically you do motion planning. You decide where the vehicle should go. And typically there is a difference between what you plan to do and what the vehicle executes. The reason is friction. The vehicle cannot execute all possible maneuvers. There is, you know, you might not have the right vehicle dynamics, etc. So you basically need the control in order to push back the vehicle into what you intended to do in the first place.

Okay, so if you look at this, it's actually a fairly complicated system, right? You need to be able to do localization, you need to be able to do perception, you need to do prediction to the future, motion planning, control, etc. And as a consequence, what happens is that you have hundreds of engineers that are actually developing this technology. And typically you have these engineers that are working in different topics and they don't necessarily talk to each other a lot. Let's say I am a perception engineer, right? I'm trying to get the best new neural net for what I'm trying to do, say the technique detecting vehicles. And then this is going to affect the whole system, but maybe somebody that's working in motion planning has a hard time to understand what are the consequences of that change that I actually did in my cell. This is one of the reasons why progress in the past few years has been relatively slow in this field.

More problems that we see here is that there is typically a very tiny interface between these models, and if you made a mistake, basically you cannot recover from it. Maybe you pass the vehicle state and that's about it. On top of this, these models do not necessarily share computation, and you have very restricted time to do all these inferences in real time, right? So you make compromises into what are the type of sophistication of the algorithms that you actually put in every one of these boxes over here.

Furthermore, for those of you that do AI, what happens is that we train the systems from the bottom up, our objective. So this is an example of our vehicle driving around some streets in North America. It's real footage. So what you see here is that as the vehicle is driving, there are some actors that are extremely important to get right, right? Because we're going to potentially be in close proximity. However, there are some actors that are much less important because maybe they are on the opposite side and they have already passed us a long time ago, right? However, typically we train the systems to maximize average precision, which is saying I want to get everything right. And that's actually using a lot of the power of the networks to try to get very precise with things that actually don't necessarily matter for the planning task that you're trying to solve.

All right, so what are we building at Uber is actually different from what other people in the industry are building. So what we're trying to do is to build a single AI system that is able to solve this task and at the same time produce interpretable representations of the world. And interpretability is extremely important, right? For many different reasons. First, it gives you model introspection. You can actually understand what the system is failing, so you can actually improve the system. Furthermore, if you get to a bad situation, you can explain what the self-driving car decided to do something in the first place. What is important in what we are building is, since it's a single AI system, it can actually be trained with the push of a button. You don't need engineering to engineer, saying, hey, I'm making this change, you know what, let's see how it affects your system. Instead, the system adapts itself all at once.

Importantly, the system is actually safer in the sense that we're going to share computation between all the modules, and as a consequence, we can actually have more sophisticated processing. The system is actually doing reasoning. We are able to incorporate prior knowledge and make sure that the system satisfies the rules of the road, for example, and avoids collision at all cost. But it's also important is that it's a much simpler system, and so we can make progress much faster as we go.

So in the remainder of the talk, I'm just going to show you videos of the different technology that the lab has actually been building in the past two and a half years since I joined Uber. So just relax, sit in your chair, and eat some popcorn. So what you see over here is some of the... at the beginning what we did was to revamp the technology for computer vision. So I'm showing you here the object detection where the goal is to place 3D boxes for all the actors of interest. And this needs to work at all weather conditions or times of the day, etc. And what you can see here is a system that in a few milliseconds can actually estimate actors that are very, very far away, many hundreds of meters away.

However, if you do just the bounding box estimation, it's actually not so efficient for self-driving, and it's actually not even safe. Why do I say this? Because maybe your detector decided to not place a bounding box despite the fact that there is evidence that maybe there is something in front of you. So we need to build systems that are able to estimate and explain every single piece of evidence, right? This is a system that is much safer. So in image space, the alternative is to do something where what you're interested in doing is for every pixel you want to explain what it is. So you want to say which category it belongs to, and for countable objects you want to basically delineate the boundary of the different objects.

However, I mentioned before that we need to reason in 3D, right? So these are nice computer vision solutions, but this actually is not going to bring you to self-driving. Instead, what we need to do is to be able to use all the sensors and be able to estimate in three-dimensional space where all these actors are. So in here I'm showing you one of the latest and greatest algorithms that we have at Uber to do perception using cameras as well as LiDAR. And you can see projections into the image as well as LiDAR space for you to see how it's consistently able to estimate what everything is using both sensors, and also in a few milliseconds onboard.

The same idea of explaining all the piece of evidence can also be used here, right? So this is the concept of semantic segmentation. Well, here for every pixel what you want to do is to explain the category of interest, right? Or whether it's an unknown object, and you want to estimate where the object is. And this is what I'm showing here on the video. Now, I was mentioning before that at Uber we are trying to build a single AI system that is able to do all these things. And so far I have shown you, yes, perception algorithms, right? So yes, it's a single perception algorithm that can do a whole bunch of things and use many sensors, but this is just one step towards that goal.

So let me show you a series of things that we have done in order to get to this one single AI system that can actually do perception, prediction, and motion planning, which is going to come in a few slides. All right, so the first thing that we looked at was: can we actually estimate where the vehicles are? Can we actually track them over time within the same network? And can we predict into the future where they're going to be going? And this is the system I'm showing you here. And what you see is for every 3D box there is some points emanating from it. This is the prediction into the next few seconds: what is this vehicle going to do? Okay, so what you can see here is that we were able to do all this task and much faster than if you use traditional algorithms, and much more sophisticated in terms of the processing with neural networks.

Now you can go one step further and say, well, it's not yet about predicting the future trajectory of vehicles. I also want to understand their intention. Is this vehicle going to cut in front of me? Is it going to change lanes, etc.? So the next iteration of the system, what we did was to be able to do the detection, prediction, as well as understanding the intent in terms of what in the future these actors are going to be doing. I want you to focus on the video on the bottom where for every 3D box we color-coded the actual intention. So the more red it is, the higher the probability of arriving to a stop. The more purple it is, the higher the probability of being parked. And it's important in self-driving that you make a difference between parked and stopped, because what you're going to be doing as a self-driving car is going to change depending on the semantics. In yellow you see is higher the probability just to continue driving. They also represent whether you're going to take a turn or whether you're going to actually change lanes or continue straight. Okay, so you see here that the system is able to probabilistically, in this case, estimate all these different things.

Now, it turns out that as humans, right, for us we understand intention is important. So we actually build something on normal vehicles such that we can tell each other what our intention is in the first place. This is turn signals, for example, right? So what we built here is also systems that are able to, given videos, understand which turn signal is on and use this in order to better predict what is your intention. Okay, so what I'm showing you here is a system that takes video as input and is able to say what is the right, left, or both turn signals are on. And the panel on the right, what you see is what the network is thinking when it's trying to do this task. So the only supervision we give is which turn signal is on in the training set. And here, as you see in different videos, the system is really learning into which areas potentially could contain turn signals. And this is interesting because different makes and models of vehicles have very different systems for turn signals, and it's able to generalize very well to vehicles it has never seen before.

All right, so as I mentioned before that we wanted to go to a single AI system, and this is what I'm going to show you next. And this is the first of a kind, it's the only interpretable motion planner that is able to go from raw sensors to do perception, prediction, and motion planning jointly, okay, with a single AI system. This can be trained simply by imitating how a human actually drives, and a few bounding boxes. So the video is going to be fairly complicated, so pay attention here. So we have a bird's eye view where we see the different vehicles, we see their trajectories into the future. And the other thing that we see is that for the ego car, which is in red, which is the self-driving car, I'm going to plot the cost function of the areas that is very low cost to actually go at a particular point in time. Okay, so this is to show you what the vehicle is thinking into what are the potential areas where it will go as it's driving. Okay, so it's learning with very little supervision to have multiple predictions. Right, in this case I can continue straight, I can take a right, I can change the lanes. And it was in the case before, right? In this case it's trying to maybe incorporate to the left or continue straight. It's understanding we shouldn't go, I should adjust velocities, etc. So it's doing very sophisticated level of reasoning without needing to incorporate all this prior knowledge, which is typically what you do with motion planners. With the standard motion planners, it takes months to build one capability. Here, yes, by imitating from humans doing this task, you can actually learn all these complex behaviors.

Now, I mentioned before that one of the aspects of self-driving is that we typically use high-definition maps. So these maps contain two sources of information. On one side, they contain complex three-dimensional representations of the world. And this is what I'm showing here: is how we can build these maps using LiDAR as well as cameras. And what we have done here is extracted automatically all the different objects that we believe are not going to be there next time we drive. For example, vehicles, right? If you have parked vehicles, you shouldn't have them in the map because next time you drive, probably they're gone. So we can build these sophisticated three-dimensional maps automatically, and we can use this for localization. And on top of this, what we can actually do is build semantics of the environment. So for each high-definition map, we estimate where the lanes are, where the crosswalks are, what are the different rules and intersections, etc.

So it was estimated by a third party that just building high-definition maps of the US will cost one billion dollars. Let me pass that: one billion mapping once. The world is constantly changing, right? So my team is pioneering building automation methods to be able to build these maps with a fraction of the cost that you will see in the industry. So let me show you some examples of how we can do that. So what you see over here is what we can do is, in this case, I'm going to showcase on highways. We drive the vehicle only once, and we can basically create a system that automatically is able to estimate where all the lane boundaries are as well as the topology of the highways. I'm focusing here in complex cases where these changes of topology, where you have forks and merges, which are difficult for these systems. And what is important here is that this technology is built to be friendly to human in the loop, so that a human can actually correct, say if the machine makes a mistake, can correct one dot, and then the machinery corrects afterwards. And this is the way that you can go from a one-billion-dollar cost to something that is much, much cheaper.

And it's important because at Uber we really want to build self-driving cars at scale. But it's not just about building offline maps for safety. We need to also be mapping on the fly, because maybe sometimes it might happen that we lose, suppose we are not localized, right? So we still need to be able to drive safely or at least to a safe stop. And also maps can change, the world can change with respect to the maps. So we need to be able to say that it's a map change. So in this case I'm showing you technology that we actually exploit to drive autonomously on highways without requiring to have an HD map at all. In this case we can drive hundreds of kilometers without intervention with just this technology. Now we envision this to be not necessarily the way that we drive. This is the failsafe if our map is not up to date or localization system actually is not working.

Now in terms of mapping, actually life is quite complicated because there is a lot of construction. These are actually taken during my commute from home to work. I work every day, 20 minutes. And in Toronto we say that there are two seasons: it is construction season and there is winter. Okay, so it's amazing the amount of things that you see out there. So we have built a technology that actually runs on our fleet, where the idea here is that we want to detect all these construction elements, we want to estimate very precisely where they are in 3D, such that we can actually route and move appropriately. What's interesting is that on average we see every one of these little cones at least 150 meters from the vehicle, which means that we can plan a safe maneuver with a lot of headway, right? And actually do a safe lane change, for example.

The last thing I want to talk about is that in order to test this technology and to be able to understand where your technology might not be ready, it's very important to use simulation. In the industry, when you see simulation, what they do is that they simulate where bounding boxes are and basically they test motion planning. But if you do this, it's great for testing motion planning, but it's not great because you're not testing the whole system, right? So you might be overestimating the performance of your system because you're not modeling how your perception and prediction system might actually fail. So instead, it's very important to actually be able to do sensor simulation such that you can actually test the whole perception stack.

So what we do at Uber is that we use the millions of miles that we have driven in order to create how the world looks like. We can create these assets of actually real vehicles that we've seen as we drive in the world, and then we can compose arbitrary scenarios to test at the scale of billions of scenarios our system. Okay, and we make the actors to be also intelligent. This is what you are seeing here, where we can actually simulate how traffic works and how actors react to each other. Once we have the world simulated, basically we can simulate how these sensors will see this particular scenario. On the left-hand side you see what the real LiDAR saw, and on the right-hand side what our simulation can be. And it's really difficult to see the difference between the real world and the simulation. Importantly, the neural nets cannot even tell the difference between the real world and the simulation. This is fantastic, right? Because I don't need to do domain adaptation. I can just use a simulator to train and test my system, which is really what you want to be in the first place.

And that's basically what I'm showing you here. So this is the simulated scene where the sensor will see, right? The LiDAR is much more responsive than the real scene. And on the bottom is our systems that are trained in the real world that when we test them on simulation, basically they perform up to 1% difference only. You can do the same thing for images. And I'm going to ask you the question: which one of this is actually fake? So it's the one on the middle left. Okay, so we can do the same thing, right? We can take the logs and we can remove and add actors so that we can actually create very realistic simulations to again train and test our systems.

Okay, so yes, to summarize: there is many benefits of self-driving in the world. There is 1.3 million people that die every year on the road. There is 20 to 50 million people that actually get severely damaged and disabled due to traffic accidents. As medicine gets better, the population is getting older, so there is more and more need to provide mobility. And we want to reduce the cost of transportation such that everybody can actually access the service. It's very important that we can build this technology, right? And that's fantastic. But if every one of you goes and basically buys a car, then what's going to happen is that we're going to collapse the planet, right? Instead, we should be using a combination of public transportation, car sharing, or ride sharing, so that we actually have a better place for our kids.

And I'm going to just finish by showing you a video of what, you know, soon the world might look like. So you take your self-driving car, it brings you to one particular building. You take the elevator. Then you go to your next modality of transportation, right? So if you think about it, two-dimensional space is very constrained. The three-dimensional world is very open. So at Uber we have a big program on this: flying cars, okay, which is very, very exciting. So this might sound like science fiction to you, but it turns out this is much closer than you think. Thank you.