Today, what I'm going to focus on is giving more highlights on top of what we discussed in these papers or things. Trying to give you some sort of what are these things that you think about or what are things that are important when you go from doing academic research to actually controlling a real robot or a car. And I think I wanted to start by saying that very few of the physical AI and robotics systems are secure. Right? And centralized systems are the ones where you don't really want to see these systems get stolen. And this is imminent. It's not something that we could really see yet. But the centralized vehicles are everywhere. Over the next couple of years, it's really going to explode. Now, when you think about again moving from the lab to moving to a real system, there are a few properties that are tremendously important to be taken into account when you're actually designing your autonomous system. The first thing is that it needs to generalize. You're never going to observe the known only. You're going to see things that are very, very far, very out of distribution. And when building an autonomous system, you want to build it in a way that it handles the dynamic situations no matter what. Right? Because in physical reality, if it wasn't a successful action, it could be a disaster. Efficiency is also something that we need to think deeply about because you need to do reasoning in real time on the vehicle in a fraction of a second. And it's also very important when you think about these systems of the sustainability of being able to do it without burning the planet. The last thing that is also tremendously important is you need to build a system that is safe. But yes, you can say it is not enough. And that is actually surprising people, right? But what you need to do with centralized systems is you need to be able to prove that the system is safe before you actually deploy it in real time. Right? You need to make sure that you know what the risk is effectively putting in the system.
Now, if you look at the evolution of the industry over the last, I would say two decades, what the first generation of technology was all about is to prove that indeed you can build a system that can be close to others. Okay? And that's what EV1 was all about. And so those of you that are not so familiar with this, this is about, can you build a system where you take the massive problem of self-driving and then you divide it into a gazillion number of different modules. You build a very large team and then you kind of iterate by going on the road driving miles and then you iterate fast. That type of technology really proved that self-driving is possible. What it cannot really do is scale. Right? So, we need to go beyond this to something that is much more realistic. And here I think that we will lose the race.
The industry has moved towards an idea that is AI-centric. By this idea, we can build a single AI system that will be able to do all the tasks necessary for driving. So, think of this as the polar opposite of what we've seen with EV 1.0. Then we're coming from the Silicon Robotics mindset with an AI system. Now, the type of technology that you typically see out there is a black box architecture. And you can actually build really good level two plus products with this type of technology where you still have a human on board that can take over in the event that something is wrong. Now, when you think about that, that human backup is not there anymore. Things are much more complex. And over the years people haven't realized necessarily that between the two models and four there's some massive difference in terms of the performance of the system, the safety of the system at a given time. One of the things that we found was important for regulating for OEMs is that the values that the system should be able to wish for the system. By the way, back to what I said before, before putting on the road, you should know that. It's also very capital intensive typically for the safety here, but in the last architectures where you need billions of miles or something like that. The players that are involved in the logical simulation is... And the other thing with this type of architectures is that they are actually really good at interpolating between training data. They are actually not great in the same as the real world at some times. In the United States, you know, well and United States is okay, you can close again instead of having a validity. So since the creation of the while we have others that are, the idea that we had was that you can actually build an architecture that is built from the ground up for the vehicle. And such that that architecture is fully verifiable. Right? And so it's still a single layer system that you can train and run. But it's something that you can actually incorporate in this within that architecture in a fraction of a second.
And the other big thesis when we started working was that we, no matter how big your fleet is, you will never have enough data. And also, the system, by acting, you can't do that in the real world situation. So you have to think about big data sets. So the second kind of thesis was that if we can build a simulator that's as realistic as the real world, then suddenly we have the key to be able to really train the systems and also to be able to prove the safety of our self-driving systems. So these were the two big bets. And what this has enabled us to do is to build a super safe system that is very efficient, that is very scalable and it goes across different form factors, that is very easy to integrate with vehicle platforms, which is something that you need to think about when really bringing technology to the real world. And can also be adapted to use any sensors, any feedback that is available in the market.
Let me give you just a little bit more kind of details of the architecture that we built. It's something that looks like this, where the end-to-end architecture is actually composed of, we can think of two big models. There is a first model that is taking the data from the sensors, so this is lidar, camera, radars, any sensors, any means, right? And as well as high-definition maps, when you have them, when they're correct. We save all these estimates or build abstractions of the physical environment and also vision. These abstractions are at very different levels. Some of them are explicit, some of them are implicit. But really gives you a very rich set of representations that then the reasoning model is going to query in order to make decisions. And this reasoning model, what is very important is that it is not going to be a chain of thought type of architecture because the virtual reasoning is difficult to do in the real-time constraints that you have in self-driving. All right, so if this is the main system that actually is thinking and you need to do this in a few milliseconds, then we pioneered a novel way we can do parallel processing. And we're able to give you super fast thinking and reasoning together with the real-time constraints so that you can actually be driving a car. Okay, so this is in a nutshell the type of architecture that we give.
Today if you come to Texas, you can actually see a lot of big rigs on the road doing commercial operations with many of the big shippers, Fortune 500 companies, etc. And what I'm showing you here in the video is a few very difficult situations and things that you can typically observe in Texas. And one of the things that you will observe is that even for trucks, it's not just about driving on the highway, you also need to drive around surface streets, etc. And you see over here something that you see a lot in Texas where drivers are crazy. You see this guy cutting across the grass in order to incorporate into the family. Turns out that in that same or even less, walking around the street, it's right. Turns out that happens a lot, much more than you think, and you need to be able to react. Now this is a tractor trailer, typically 80,000 lbs just for context, okay? Around 40,000 kg. Cool. So as you see over here, one of the things that is important with trucks is you need to see super super far, you need to be very very proactive, and you need to be able to understand that you are an articulated object that is really really big, so you have to have a good self understanding of the road and be able to self assess to reason really far in the future as well. So that was just some interesting challenges when you think about building autonomous systems.
Now thanks to this reasoning model, what we can do is actually not just drive in what typically has been done in the industry which is a hub-to-hub model where you have these hubs next to the highway, you drive autonomously between them and then humans need to do from the hubs right to the end customer. Instead of doing things like this, which is very expensive, it's not something that you can actually, it's not very good from the customer satisfaction, etc. With this technology, you can drive from the customer to wherever you want. Now, we can build this really exciting autonomous system, right? But what is very important as well is what is your product deployed in the real world? Retrofitting trucks is something that you can do for development, but it's not something that is safe where you can actually really bring a product to market. So, that's where it's very important that you partner with the OEMs, you actually get redundant platforms that are built from the ground up, that are fully validated, where you actually can put your autonomous system.
Let's talk a little bit about simulation. Okay. So, as you many of you know, simulation has been at the core of the technology that Waymo has actually developed. And what you see over here is some of the latest in terms of our simulation system. So, I'm going to give you a glance in terms of what our world model build, and how to think of it as how you know, what are the things that are very important when doing research to really push the boundary of simulation. In terms of how we design our simulation system, it was also with a similar philosophy of how we build the autonomous system, which is this idea that it's going to be modular, right? Such that you can actually plug and play with things that will allow you to generalize across different vehicle platforms. For example, if you're doing different types of trucks, robotaxis, where as you see over here, we really mimic how this works in the real world, where your self-driving vehicle is observing, right? So, it's sensing with all the sensors, lidar, camera, radar, right? And then that information is passed to the autonomy system under test, where there are latency constraints that come from the computer that actually runs on the vehicle. And this is very important because depending on how powerful that computer is, your reaction time will be very different. So, the way that you can receive from the real world is different. Then there is vehicle dynamics, which is there's a difference between what you tend to execute and what the vehicle actually does. This is because of friction, because of gear changes, all sorts of different properties of the real world. First of all, driving squats turns out that this is actually super complex compared to typical cars. And then there is all the world that acts upon, right? Not only can you not be the only one driving, but everybody's actually trying to go to their destination. Right? So, we build this modular approach where we can use the best development in next generation for all these pieces. Where now instead of being real data for all these things, real actors, real vehicle dynamics, or real latency is going to be an AI system that's actually doing any one of those things. Okay? And what this allows you is you can go to different form factors, different things, with a lot of different configurations. And that's amazing, right? Because you don't need to rebuild the simulator, retrain the simulator, or anything like that. And the last thing on simulation that often times is not talked about is the fact that it's not just about simulating things that we think are not as important. It's important that you understand when you're testing a system on a simulator, how good is this system? Where is it having trouble? So building a critic that is able to build out those real metrics for you is also tremendously important.
All right. So, let's go a little bit deeper on the simulator, right? And let's start with how do you build what to simulate in the first place? Our philosophy on how to do this was about, we are going to have a very comprehensive way of generating the scenarios and depending on what you try to do, you're going to use plug and play any of these bodies. As a user, you can prompt the system, right? You can prompt with text or whatever it is, maybe structure or the code if that's what you're interested in. So that's one way of generating scenarios. You can also import anything that you have ever seen in the real world directly from raw data to the simulator. You can generate, right? This is generative AI simulators, you can generate any situation and we can do it in a way also that is necessary for that autonomous system that is under test. All right, so these are all these different ways of doing this, but what you can do is combine all these types of scenario generation at the same time. So you have a very robust way of really from the data of the scenarios and basically generate anything that any person or machine, right, we don't really care what kind of scenario it is. The other thing that is important with simulation is that it's not just about building a single behavioral model. Why is this? Because you're going to use this for testing an autonomous system and it's very important that you are able to generalize and test no matter what humans are going to do in the physical world. All right, so for this we created really a zoo of actor models that enable us to really test comprehensively the system as well. And there are many types of tests and things that you can do as well. We build many generations of these actor models and what you see over here is something that you can see at CVPR this descending, which is the latest greatest, which is this idea, this is a pretty interesting mathematical paper where is this idea that you can build efficient equivalent transformers whether it's for prediction, planning, or simulation in this case, which is not just beautiful math, but also you can actually do inference in these models in a much faster way while having the same equivalent properties than doing things like processing all of these encodings without having the quadratic constraints. Important to consider the probability of them. Now you can generate a scenario, you can see what all the actors are doing. That's not enough. Why? We need to understand what the world looks like. And what we invested a lot since day one was this idea that, well, we can observe the world. So we don't need to reinvent everything, right? So if we can actually build a clone of the world in a way that we can compose and change that world, then we have an amazing distribution. Think about a single vehicle that actually defines... If you can have all that, the content that you can place on the ground, in different situations, in different time of the day, in different appearance, conditions, etc., then suddenly you will have a massive, massive library that then you can generate on top by permutations of things. So this is something that we invested since day one and has been tremendously empowering to both training the autonomous system as well as to do a lot of the full system level testing for self-driving.
Now just giving you some insights in terms of, from the UniSim that was the first version that made public a couple of years ago, from that we focused on a few of the characteristics that are actually really important for building simulators for deploying several vehicles. And this is controllability, right? This is something that is a must-have. So we build different generations of this. One of the things that is actually pretty challenging is when you're trying to create digital twins from the vehicle that are actually moving. You have pretty sparse views. And you're not going to observe many of, often times only a small part of the vehicle or the object or the thing that you're trying to simulate. So it's very important that how can you use generative models, for example, to create full and complete scenes from partial observations that are in the wild with difficult properties like motion blur and other things. So going beyond what is an academic benchmark to how can I now create something where I can simulate the world with all these properties? Modeling dynamic scenes, right? As we have and some of them also with pedestrians and things that are deformable. We also build many generations of these. So it's pretty challenging in general. But we're pretty happy with what we have today. But it's important for simulation when you're actually, everything that goes into simulation is a workflow. The speed that you can actually render is also tremendously important. So we spend also many cycles in terms of how can you do this in a fraction of a second on what typically our services do. And you want to do this at scale. It's not just about, you know, it seems or anything like this, we want to actually create an entire world and any permutation of the world. So imagine about that. And the last bit is that often times you want to make sure that when you create a digital twin, not only you reason about things that are moving, but also sometimes you need to reason about how are you going to put together 20 plus sensors when creating these twins and you can actually do calibration in the loop when you're planning to use, which is very, very interesting. And basically, it allows you to do calibration with everybody targets, which is a massive unlock for lenders. Well, yes, a small thing. I really don't want to focus on a specific papers or anything like this, but just to have a few things to show you. But for example, we can increase the efficiency to create like cameras of the three frames per second, and later on the three cameras frames per second. Which really allows us to simulate very, very quickly. Now, the digital twins are very exciting, but they're also brittle. It's not the end of the story. You know, we've seen them to work tremendously well when the views that you're trying to generate afterwards are relatively close to the trajectories that you took when actually creating the twins in the first place. Right? And that's a problem, because we're going to have close-loop testing of autonomous systems that might do something that is totally different from what the actual recording was in the first place. Okay? So, this is a must-have that you can actually really extrapolate from the current views, given that input was also or that data was really sparse in the first place. When you look at the industry or sorry, when you look at academia, etc., this is a very, very hard task. So definitely something to spend cycles for those of you that are looking for that. Now, everybody's probably thinking, 'Oh yeah, yeah, yeah, but we can use synthetic fonts.' Of course, of course, we can use 4D models, we can use neural radiance fields, it's a thing these days, and we've seen impressive visual quality, right? And they have for example, mission on support. It's not as much controllability as we like, but it's pretty decent, I would say option. However, there are some additional challenges, right? There are lots of especially temporal inconsistencies. We can't have this. We're going to use this to assess the safety of our self-driving vehicles. So we need to be able to simulate something that is really high fidelity. And there are hallucinations as well. Okay?
Question, fake or true? Any guesses? Come on.
