Well, thank you very much for the kind introduction. We were just discussing, I'm very excited because I haven't been in a conference in three years. This is what being a CEO does to you, and I'm really, really, really excited to be with the community I really care a lot about. What I wanted to do today is really give you a peek in terms of what are the really important problems that you need to solve when you're not just trying to do core research, but actually deploy a product in the real world that's going to have tremendous impact when you think about the transportation of the future, which actually starts today.
Just to frame everything, what is very clear is that we are in the moment of a physical AI revolution. We've seen the digital revolution, and now it's coming to the physical world. Over the next 5 to 10 years, robots are going to be everywhere. What is also clear is that the first application where really scale is going to happen is self-driving. It's by far the most mature of all the physical AI domains that you see out there.
Now, when you think about being able to build autonomy systems for the real world, there are, I would say, three main characteristics that are very important to think about when designing what an autonomy system should look like. The first one is generalization. Generalization cannot be an afterthought. What that means is that you're going to see things on the road as you deploy these systems that you have never seen before. They're going to be very out of distribution no matter how many miles they actually drive, and the consequences of not being able to handle those can be catastrophic. So when you actually design your system, it should be such that generalization is a principle that is ingrained into the AI systems that you actually build.
Another thing that is very important is efficiency. For me, efficiency comes in two different flavors, or two important differentiators. We build systems that have to react in a fraction of a second in real time, so whatever you do has to be very efficient—the compute on board, but also the way that you build this technology should also be very efficient. That's very important for the sustainability of the world that we live in today. Think about the future, think about how much energy you need. That's an important principle of what we do at Waabi. The last bit is also very important: safety is not enough. Oftentimes people get confused when I say this sentence, and the reason I say this is that not only should the system be safe, you need to be able to prove that the system is safe. That's actually harder than just building a safe system, which in itself is pretty hard.
I think it's important to also acknowledge what we have seen in terms of generations of technology in the industry. Over the last 20 years, there has been quite a bit of progress since the DARPA Grand Challenge. There is a generation of technology which we call AV 1.0, these highly engineered systems that have gotten us to the point from something that was just a challenge to actual deployment in the real world. A lot of good came out of this generation. Now, the problem with these kind of engineered systems is that they're pretty brittle. There is no way that you can come up with or program by hand what the system should think about, and they are very capital-intensive because you need a very large fleet, you need to drive miles in the real world, and you need many thousands of engineers that work on this gazillion number of models that these traditional stacks have. All of this makes it not scalable. But it was very important to prove that you can actually launch a self-driving product.
The problem happens after, which is now: launching a product in a small domain is not enough; you need to be able to do this at scale. That's what we are all up to. What we've seen over the last couple of years is a switch towards what we call AV 2.0. Many of you are familiar with this concept, which is the polar opposite. AV 1.0 is built by roboticists, which is divide and conquer. AV 2.0 is all about how you would build it as an AI engineer or scientist, which is a single AI system that can do everything necessary for driving. We've seen amazing products out there for Level 2+ systems where there is still a human on board. In this black-box type of end-to-end AI architecture, they are not really suited for Level 4, where there is no more human on board, let alone Level 5 everywhere. The reason for this is that they are very difficult to impossible to verify that they're going to do the right thing, and remember, we're talking about provably safe. They're also very capital-intensive in terms of how much data you need to collect as well as the size of your data centers and the chips that you need in order to train these systems. The same as large language models, they are very good at interpolation, but they are really not good at extrapolation. This generalization capability is not something that really comes from this type of system.
We need something else. At Waabi, what we did from day one when starting the company five years ago was to build an architecture for an end-to-end system that is built so that you have all the characteristics that you need for Level 4, meaning it's fully verifiable while being full end-to-end, and I will show you in a second what this looks like. The other big bet that we had was that you will never have a real-world fleet that is big enough. We just saw Ashok showing 10 billion miles. That's a lot, but they still need more—how much do you really need? Our thesis five years ago was that if you can actually build a simulator that is as realistic as the real world where you can test and train your system in closed loop, you actually have the solution to the equation of really building these systems at scale.
The reason I say this is because you want to test as you're actually driving, and you want to test and train in safety-critical situations, unavoidable accidents, and these things that you can't really do in the real world because it's not ethical. These two components actually allow us to build a technology that is safe, and we can prove safety; that is very, very efficient in terms of time to market, in terms of the capital the company needs, in terms of the compute on board that the system needs; that is truly scalable, not only across geographies, but across form factors and any sensor type that you might want to put in the autonomous system. You can permute, change things around, and the same model can actually drive that. That's a characteristic that is amazing when you think about bringing this to any form of transportation and beyond. It's also very easy to integrate into an OEM platform—this is very important—and adaptable to different parts. Let's look a little bit under the hood in terms of what we build.
In this next generation, sort of AV 2.0—we didn't want to be cheesy by calling it AV 3.0, so call it whatever you want—the idea is that you have sensors and you might have high-definition maps. What this system needs to do through this inference system is that it's forced to interpret and create abstractions of what's happening in the world, and it uses these abstractions in order to reason about what are all the possible actions that the vehicle can actually do, what are the consequences of every one of these actions, and then chooses the best maneuver in a fraction of a second. That is a very, very different type of reasoning than chain-of-thought and all the things that you have seen. What is very exciting in this technology is that it's actually parallel reasoning. In chain-of-thought, how many iterations do you need to do and how much time do you have becomes a problem. Instead, with this, you can actually cover a much larger space of possibilities, truly understand consequences, and build abstractions that allow you to generalize. It is very, very different from anything that you've seen or you will see in other companies out there.
This type of technology really allows us to drive our big rigs, these beautiful trucks that you're going to be able to see from Friday on. One of our trucks will be at the exhibition. It allows us to drive not only on highways, which is typically what you see with long-haul trucks, but also in all forms of generalized surface streets. This is actually pretty difficult with a Class 8 big rig truck that is 80,000 pounds because it's an articulated big object with lots of complicated pieces. However, with this technology, for the first time in the industry, we can actually do so. We can handle tight turns; we can handle all sorts of situations. You see here some of the crazy Texas drivers that go through grass to avoid a queue in order to merge onto the highway. All these situations and many more our trucks need to actually handle.
What is great about this type of technology and the ability to drive in all sorts of surface streets is that it actually matters for the product that you're trying to bring to the market. Why is that? The industry in long-haul trucking is going with this model of hub-to-hub, which is: I have a hub close to the highway, between hubs we go autonomously, and from the hub to the end customer a human will do that job. The problem with this is that, again, you're adding a lot of friction and you're adding massive cost on both sides, so the economics don't work that well. The customers don't want to hear about this; nobody wants to pay for that drayage at the end. That model actually will not allow you to really deploy at scale. Instead, with this technology where you can drive in more urban environments, etc., for the first time it allows you to bring a product to market where it really drives the freight the same as humans do today. When you think about moving from academia to industry, this is very important to think about: is your product something that people want to use?
The other thing that is also very important that oftentimes is not necessarily talked about is that being vertically integrated with the OEM—in this case, Volvo partnered on the OEM platform—is tremendously important. Why? This is a special type of truck. It's not just a truck that you buy off the lot and then you retrofit. This is a truck that has all the redundancies for steering, braking, etc. This is important so that you can build a truly safe, scalable solution. It is the only way that you can have a fully validated approach on the roads today.
Now we have the great autonomy system, we have the OEM platform. What is the last piece that we need? It was the data engine that I talked about. I'm going to talk a little bit about simulation in the next few minutes. One of the things that we discovered very early—it was the thesis of when I started the company—is that simulation is a must. What is very important is that you have a simulation that can generate anything and is very realistic. These are examples of the latest simulations that we can actually build.
The way that we build this simulation system is also with a similar philosophy to how we build autonomy, which is a fully controllable, interpretable simulator that mimics how any self-driving vehicle drives in the real world, but now it's in simulation. What you have is the world around you; you're going to sense the environment typically as you're driving with all your sensors—LiDAR, radar, cameras, etc. You're going to execute the autonomy stack. Your compute is going to have some latency in terms of it's not instantaneous to give you the answer, so that's actually very important for reaction time. There is an action that you want to execute, but actually the truck is going to do something slightly different. Why? Because there is friction, there are potentially things like gear changes, where physics in trucks are actually pretty interesting compared to a passenger car. Basically, what you will observe is a little bit different from what you intended. All the actors and folks around you, all the traffic participants, will actually be driving or moving around at the same time in real time. That's how the real world works. We built a simulator that actually mimics exactly this. What is really great is that it is plug-and-play, so if you have a new platform or new sensor, you just need to change the configuration and everything works for you. It's very modular and composable, versus needing to fine-tune a model or do this or do that.
The other thing that is very important is that having a great simulator is not enough; you also need to have the right metrics. You run a gazillion simulations—how do you know whether your system solves those scenarios versus struggles? You need automatic metrics that are really informative in terms of what the system is able to do and what it doesn't do well, and also a way using this critic to train the system. This is like your reward or loss function. That really enables you to continue progressing and be able to create the most informative examples when you think about adversarial generation while doing training.
In this simulator, it's actually very interesting the way that you can generate scenarios. It's not just one way. As a user, if you're interested in simulating a particular situation, you can do it with text prompts, you can do it programmatically, or you can do it from any scenario or thing that we have seen in the real world—we can clone it and create simulations and variations out of it. You can run it in full generative mode so that you can generate from whatever distribution you're interested in, or you can run adversarial simulation so that for that autonomy stack you try to really understand what are the issues. It is important that you have all these pieces and components within the same simulation system so that you can do all the testing, training, etc., that you're interested in.
Oftentimes people try to have the perfect or the most comprehensive actor model. Actually, when you're using a simulator for training and testing, what is very important is that you have a zoo of actor models that are also controllable, so that you have the distracted driver, you have drivers that are out of distribution, etc. Why? Because you need to make sure that the autonomy system can handle all those situations, not just nominal conditions. Just being able to handle or sample from the likelihood of the real world is not enough. This is very interesting in terms of how you go from a research paper to a very comprehensive way of testing. You can actually see one of our latest and greatest actor models here at CVPR. If you like math like Shai, you can look and see some interesting ways to create an efficient invariant transformer without the quadratic nature of positional embeddings. Definitely go check it out if you're interested in having accurate models or prediction systems that are very efficient.
Now, okay, we can actually create scenarios, we can actually have actor models, but what does the world look like? How do I encode that in the simulator? Over the past few years, we have built many generations of digital twins. The thesis was that if I just drive around or use any data, if I can clone that, I can actually create massive content from which to create variations and permutations of that. That's actually a way to really quickly and automatically create simulations grounded in the real world. Then you can create variations, make everything reactive, and have a closed-loop simulator. UniSim was the first version of this that we actually showcased publicly.
Since then, we have created many variations of different things. At a very high level, I just want to give you some examples of the things that we look at that are important for simulation: controllability, an absolute must. By this, I don't mean text; real controllability is very important for some of these use cases like safety, completeness. Typically, when you are actually collecting the world with a sedan vehicle, oftentimes you only see a little bit of the object, but you're interested in simulating that from many perspectives. How do you complete that world? Those are some of the methodologies that we developed: handling dynamics, which traditionally has been difficult, and also deformable things. We built a bunch of things in that direction. Being able to render for all the sensors—radar, camera, LiDAR—super efficiently is another very important component.
You need to build this at scale. It's not about 10 scenes. As I mentioned before, I want to clone the world and create all permutations possible, and even generate more things that I have never seen. You need to be able to do this much faster than typical approaches. You can also use this digital twinning technology to even do calibration automatically. That's actually really important because we don't need targets and things anymore that are really hard when you're trying to deploy this technology in the real world. Just to show you a few examples: we can render, for example, cameras at 30 frames per second, LiDAR at 300 frames per second. I'm not going to go into the details of all these things, but this is just an example of some of the technology if you're interested.
Now you can think, are digital twins enough? Unfortunately, no. The reason why is that if you deviate from the original trajectory significantly, they're going to have difficulties and a lot of artifacts. This is 3DGS in this case, one favorite technology that people use, and you see the artifacts are massive. We need a better way to do this, and this is where generative models come to your rescue. This is a pretty exciting technology, impressive visual quality. You see some of the conditional support—maybe not as much as we would like, but at least there's some stuff over there. Now, there are also some challenges with this technology: you have temporal inconsistencies, you have hallucinations, so you need to build technology that doesn't have these problems as well. Real or fake? Sorry for those of you who haven't been in my—I see one or two that were in my previous talk. Any thoughts?
