Hi everyone, welcome back to another episode of AI Pro's differentiate understanding. This is your host Grace Shia. Look where I am. The back of a car. Doesn't look that exciting, does it? Let me flip this around. Look at that. I've got no driver. I'm in the back seat of a Pony AI robo taxi. And today joining me is James Pong, the co-founder and CEO of the leading robo taxi company. It's expanded its footprint across the globe in Asia, in Europe, in the Middle East, but obviously today we're in its leading home market, China, and in Shenzhen, where it has a couple hundred fleets deployed in the streets already. Hi James, thank you so much for sitting down with me. I'm really excited to be having this conversation with you. So you left Baidu in 2016 to found Pony.ai when many in Silicon Valley were saying self-driving cars were only three years away, but obviously that wasn't the case. So you know, what did you believe then that this consensus was getting wrong, and tell us about your journey from 2016 until now.
Yeah sure, we were founded in 2016 about 10 years ago. But even at that time, I didn't believe that autonomous driving could be solved in three to five years. From a technical point of view, because even back then, 10 years ago, even a demo for autonomous driving was already very hard, and let alone there's complexity involved in the autonomous driving industry that involves regulation, user acceptance, and the readiness of the ecosystem. So because of sheer complexity, even then my prediction was it would take at least a decade for this to be a real application. It turned out that my prediction was about right. Now, 10 years down the road, we actually have fully driverless commercial applications in many cities. Of course, it's just the beginning of the long journey for autonomous driving, but at least now we have real commercial applications. So, I think people, like any new industry, were super optimistic for the short term, but they underestimated the potential for the long term. So, I think autonomous driving is definitely one of those industries.
And what really drove you to want to actually work on this technology in the future mobility?
I think the motivation was twofold. One is that the potential, both commercially and also societal benefits, for autonomous driving is so huge. Think about everyone needs some sort of mobility, and autonomous driving is much safer than a human driver. So it has huge societal benefit of saving people's lives. So essentially, it's just such a great industry to work on, although back then, 10 years ago, it was very unclear when this could be done. The other reason is, of course, because of the sheer technical challenge of autonomous driving. In my previous jobs, I worked on different areas: software, hardware, large-scale distributed systems, AI, and whatnot. But none of the things I worked on is as complex as autonomous driving, which is a field that involves hardware, software, hardware-software integration, and many other things. There's AI, there's real-time system, there's also large-scale AI training, and all that. So from a sheer technical point of view, it's such an amazing and challenging thing to work on. So I think those two reasons propelled me to start the company.
There's definitely a lot to unpack there. I think later on we can definitely double-click on the hardware-software integration as well as, you know, the safety concern there. You know, if you say autonomous driving is much safer than humans, for sure safer than me driving. I know that. But some may argue otherwise. So let's talk about that later. But first, I want to ask you about something that was quite interesting. During 2022-23, there was a bit of a public reckoning I think within the industry, right? A lot of peers folded during that time. People decided to pull out of this sector. Some people worried that, you know, autonomous driving would really become a reality, but you guys charged ahead and you really believed in your vision. Tell us about that period and how maybe that changed your vision or your growth mentality.
No, I think 2022-23 were a period of time where the autonomous driving industry had evolved for roughly 10 years. I think that was the time of reckoning. That's the time where the haves and have-nots have really diverged. So I think that's exactly the time we, as a company, have seen tremendous progress. At the end of 2022, beginning of 2023, that was the time we actually finally had the first fully driverless commercial applications and operations on the road. So because we made such progress, both from a technical and also from a regulatory point of view, we made the progress and we think we're finally seeing the glimpse of hope, then of course we charged ahead. I think a lot of the other companies who weren't able to, either from a technical point of view or from a pure capital-raising point of view or from a regulatory approval point of view, weren't able to have fully driverless applications, then they faded away. So it's sort of like, well, everyone is in school, there's no big difference, but after graduation, then there are haves and have-nots. So I think that was the time of division.
Yeah. So speaking of milestones, I want to kind of go back into history a little bit. So in 2021, Pony.ai had the third highest number of miles driven behind Waymo and Cruise. In 2022, Pony.ai became the first autonomous driving company to get a taxi license in China. 2023, Pony.ai was licensed to operate robo taxis in Guangzhou, etc. And your expansion continues. So it kind of follows what you just said. There was good momentum behind you guys. Now today marks Pony.ai's 10th year officially. You kind of talked about how you guys have grown, but what was one or two of the biggest milestones that you're really proud of looking back now and that you think have really set the tone for your company now as you are really spanning globally?
Yeah, I think in my view, the biggest milestone I actually have already mentioned is the end of 2022, beginning of 2023, where we were granted the fully driverless commercial license in both Beijing and Guangzhou, and we started having the operation to the general public. Actually, it was in January, mid-January in 2023, that I was the first rider in our commercial robo taxi operations in Beijing. Surprisingly, it was exactly on that day, it was snowing in Beijing, and I was in the vehicle by myself. That was the moment where I actually saw our vehicles were able to drive by itself, and because of the snowing, it was also a very challenging scenario, and we were actually not being suspended for operation. We continued to operate, and I was in there. That was a moment finally it feels like a dream come true. Finally, not just because our technology is ready, but also because we actually got the approval from the government to have the license to operate. So it's like all the seven-plus years of efforts finally paid off. To me, that felt like the, as Neil Armstrong said, small step for a person but a giant leap for the human race. Although I wouldn't call it as big as the Apollo, but to me, it felt like it's finally from zero to one. So I think that was a deciding moment or defining moment for Pony.ai.
Oh, that's a personal pivotal moment. I love how you visualized it because I could just imagine how chaotic the roads were in Beijing, and also it could be quite romantic when Beijing is snowing because it's such a beautiful city. Okay, but let's talk about what is a robo taxi and how the public actually even felt about it when it first rolled out. Before we start recording, I was even telling you I was like, 'Hey, look, I get a bit scared when I see Waymos on the roads or Pony.ai when there's no one driving behind the wheel.' Now, that's because I'm not used to it. You said, 'Oh, yeah, it's okay. People get used to it eventually, right?' But let's look back at 2022 when it first was deployed to the public. How was the public's reaction?
I think because it was a gradual process. In the operational domains, in the operational zone that we had, we used to have a safety operator behind the wheel, although the driver actually didn't touch the wheel or push the pedal, but people gradually got used to it. Actually, at the very beginning when we were just deployed in Guangzhou, in those days, if you look at the picture of our first and second generation of autonomous driving vehicles, you still see those spinning lidars on the top, and they were very much visible. People were curious, but gradually, people are just getting it as business as usual. And from a rider's point of view, the experience of a robo taxi is exactly like a typical taxi. The only difference is there's no driver inside the vehicle. The way you get the vehicle, the way you get in and get out, is exactly the same. And also the other traffic participants like pedestrians and cyclists, they get used to it. So I think it just takes time. It's just like the first cell phone came out, the first real smartphone came out, people were very curious, and now nobody cares. So I think it just takes time.
It normalizes eventually, right?
Yeah. And I think we've really had a few years of consumer education done by quite a few of the players, including yourselves. Okay, well, let's talk about the technical side of things. For outsiders, people might not understand the nuances between L2 and L4, and increasingly we're getting closer to L5, supposedly. Are we? So help us understand your thinking on that. How do you structure your own team, your products, working on different technology? Who gets held accountable for the actions in an L2 vehicle versus an L4 vehicle? And then finally, are we going to be able to complete a route anywhere we want with an autonomous vehicle? It's a big broad question, but I'll throw it to you.
Yeah. So the definition of the level of automation for vehicles were actually defined about 20 years ago. Of course, the industry evolved quite a bit. I don't think that the level zero to level five might be the right way of defining what the level of automation is. So in my opinion, actually there are two different products. One is we call the driver assist systems, ADAS. The other is fully driverless. So in a broad sense, I think there are two categories. There are definitely two different products. The biggest difference is not on the technical side, but rather as you mentioned, on the regulatory side, is who is first in line for the responsibility if there is ever an accident. I think for any ADAS system, any driver assistance system, it's always the driver behind the wheel that's responsible, regardless of whether he or she is looking at the road or has hands on the wheel. Whereas for the fully driverless systems, it's the system that's first in line. And because of that requirements, think about if there's a driver behind the wheel, it sort of serves as a safety net. So the system does not need to be bulletproof. It's probably, as long as it can handle 99% of the cases, it's probably good enough. Whereas for fully driverless, it has to be dealing with all the edge cases, all the extreme cases, and have fallback systems. We can get into those details later, but essentially in my opinion, there are two different products. Of course, for the driver assistance systems, there are different levels: you can be on highway, or only keeping in lane, or they can even be able to handle some of the automation in the urban environment. And for the fully driverless, of course, as you mentioned, there's L4, L5 in a traditional definition. L4 means in certain areas it can be fully driverless, L5 is everywhere. But I think it's never a clear division. Essentially, you can think of it as how we drive. We start with an area, but then we gradually improve, eventually we'll be everywhere. So I think that's just a gradual process instead of a clear division.
So actually, I want to double-click on what you just said. So help me understand what is the gap between L4 to L5 right now? And right now, Pony.ai is at L4, right? The robo taxis. Is that correct how I'm understanding this?
Um, no. I wouldn't call them a gap. I think it's a different product definition because they serve different purposes. I think most people view this as a process of evolution, from L0 to L2, L3, L4, but it's actually a wrong way of looking at it. As I already mentioned, because the clear definition, the clear difference is that who's first in line with responsibility. That's decided by regulatory actually, by product definition and by regulatory as well. So because of that, it's essentially two different products. As the product is getting more and more mature, getting more powerful, in my personal view, the division of two different products is getting wider instead of narrower.
Okay. Then I will actually push on this. What is the real bottleneck right now for companies like you to deploy at a faster scale or into going into more cities quicker?
I think that's the reason. I wouldn't say it's a single blocker or a single bottleneck that prevented us to grow faster. I think it's because of the sheer complexity of autonomous driving and what entails to ensure safety. There's regulatory, there's technical things, and also because this is such a brand new system, we need manufacturing capacity, we need deployment, we need to get all the operational things ready, like all the garage space and whatnot, and also user acceptance, user education, as we already mentioned. I think all those take time.
And I believe also you have different partnerships with different managers of your local fleets. And that kind of know-how also takes time for them to understand to transfer over, right? For them to manage robo taxi fleets versus human fleets.
Absolutely. Absolutely. It takes time.
Um, so I want to bring it back to technical. You publicly claimed before that you use the least amount of compute footprint to reach L4. I thought that was quite interesting. Help us understand how you achieve that and how the model on the vehicle versus the large model you train in the labs actually work together.
I think all the AI systems more or less take the same approach. You have data on the back end, on the data center side, you train a large model where you essentially try to get all the cases to be learned. In our case, we use the world model, which you can think of as a simulated city where we train the virtual driver and let it drive on all different kinds of roads and learn the driving ability. Then what is condensed as a model from all the learning, we deploy on the vehicle. In the traditional AI sense, I call it edge computing. You put on the edge, put on the devices, put on the car, where it's a much smaller model. In a human sense, it's like we learn everything, then when we go to a test, we don't need everything, we just need to be able to handle the test. So that's typically the training. The backend system needs a lot of computing, but on the actual usage side, you don't need that much computing power.
So when the car is running on its own, it's actually only using the model on edge essentially.
I see. Okay. So let's talk about AI systems because AI systems for language, images, code have improved dramatically. There's also obviously a lot of hype right now around world models, but what you've been describing actually has been something that's not been coined world models for a decade, over a decade. What has generative AI done for you guys or how has it changed how you view your own AI system? Do you think the word 'world models' does your system justice in that sense?
Yes. It's a little bit different and they're also related. Again, use human as analogy. It's actually quite similar to how we think. Think about the large language model, how we process image, how we process knowledge, is sort of related to our memory and our logical areas of the brain. Whereas when we drive, it's not just the memory and our knowledge, it's also how we react, how we action, and all that. So the example is one is related to how we learn a new skill, that's a language model, whereas for driving, it's like how we learn to ride a bike. It's actually different types of brain, different types of skill sets. That's why it's different. It's not the same AI because that's how humans deal with different skills for knowledge.
So Gen AI has not affected you, but how do you view the idea of now calling the physical AI world models because you guys have been doing this for more than a decade? That's kind of my question, I guess.
Yeah, that's what I'm trying to get to. For language model, it's related with knowledge, with language, with logic. That means you need to be a very large model because the thing is, if you don't know a historical event, there's no way you know it. So you have to have all the knowledge of a human ever created in your model for it to be powerful. So that's why large language models require a lot of computing power and memory and everything. While for driving, it's like how we learn to ride a bike. We don't need to have a PhD degree to learn how to ride a bike. Rather, it requires a lot of practice and training. That's what the world model is. It's essentially a model where the virtual driver can start learning by itself to learn how to interact with other cars, cyclists, pedestrians, and then learn the driving skill out of that. So it's a bit related to large language model but it's quite different because it's related with action, with manipulation, with collision avoidance. So that's the key for the world model.
So tell us about how you simulate these systems. How do you leverage simulation systems for these edge cases then?
Yeah. So essentially, that's how we learn to drive. There are several key factors for the world model. One is it needs to be very real. So we call it fidelity. It needs to have high fidelity, meaning it resembles the real world. Second is that everything that moves in the world model, meaning cars, pedestrians, needs to be smart, meaning that because driving is an interactive process, our action will affect everybody else around us. So they need to react accordingly. So it's more like an interactive gaming where we react with everything else. So that's the second challenge: all the interaction needs to be smart, needs to be intelligent. And the third challenge is how do we evaluate what is good driving? You can avoid collision, is that a great driving? No, not enough, because there is a passenger inside. Comfort is important, efficiency is important from A to B. We want to use the minimum amount of time. So essentially, it's a multi-metric evaluation system to see what is good driving. So there are three key challenges for the world model, and we certainly put all our effort into developing the world model related to those three areas.
But human drivers can be so emotional, right? You can be scared or even rage driving, or we can be communicating sometimes with obvious signs. We're looking at each other, we communicate with eye contact, hand gestures. How do you train your fleets to understand human behavior right now? Because obviously human drivers are still the majority of drivers on the roads today. And in your case, I actually think, yeah, you're right. Like at one point, maybe removing all the human drivers will make it even safer, especially removing drivers like myself, as I say it again. But how do you actually help these cars understand all these non-obvious signals? Not someone quite directly crashing into you, someone forgetting to turn on the turn signal, someone at a stop sign looking at you and waving you to go, like all the nuances.
Absolutely. See, that's why the first thing is how we become a better driver. Essentially, it's a continuous learning process. The first thing is you avoid collision. You become a cautious driver. Then gradually you start learning all the signs, all the non-verbal cues and hand gestures. So that's exactly the case for us as well. The earlier model of our system is just driving and trying to avoid collision. Then gradually we put a lot of new things, new recognitions, new perception models into our system where we start recognizing, for example, the hand gestures, especially all the policemen's typical police gestures: stop, go, and all those things. And then we start recognizing, for example, on the road, small obstacles on the road. So it's sort of how we learn. We start getting the big picture first, and then we start learning all the nitty-gritty details down the road and put them to enhance our system. And regarding the second point, you'll see why when all the cars are autonomous driving by themselves, it will be easier to drive. Yes or no? Because the thing, especially in China on the roads, the biggest challenge is not the other vehicles, it's in a lot of cases cyclists and pedestrians. And we can't make them to be autonomous driving. So I think by having the ability to recognize pedestrians, recognize their intention, their sign, and giving you a specific example: on a crosswalk, the way pedestrians look and how they pay attention. For example, if they want to directly cross, they typically look straight, but if they are looking back, that means they will more likely not cross the crosswalk. So we actually take those cues to decide whether we let them go or we go straight ahead. So a lot of those details need to be put into the system to make it safer and at the same time to be efficient.
And the judgment is made on the spot through the camera.
Yeah. Everything else. Yeah. Camera, lidar, we take all the sensor input and then we make the comprehensive decision based on the input.
Definitely, China has less predictable driving conditions, especially given the amount of pedestrians, cyclists, motorcyclists we just talked about. So if you can drive safely there, I bet you can drive safely anywhere. But jokes aside, it's really interesting because you know, we talk about as your fleet grows, you accumulate more and more world data, real world data. Is that kind of data eventually becoming an advantage and a serious edge for incumbent fleets and a structural barrier that makes it very hard for new entrants to compete? Then
Data is important but data is not everything. So how we understand that is, let me give you an example. Think about how we learn math. You can think of the data as the practice sets. Of course you need to do enough practice to have good knowledge about the subject, but that doesn't mean that if you have the problem sets of the whole world you become a math expert. So that's exactly the same case. We need enough data sets to know what the real world driving condition looks like, but we don't need everything because once we know enough, we can always generate enough knowledge about the driving. So in a way, I think the driving data is important, but it's not everything. So that's exactly how you view this.
So, as we speak of this, how do you view the whole landscape right now? Who would you say are the biggest competitors globally and how do you view the different markets playing out?
Yeah, that's a very complex question to answer. I think first and foremost, let me get some premises on this. First, the whole mobility industry, especially related to autonomous driving, is very large. It certainly has enough room for several players. Second, it's still at a fairly early stage for fully driverless. I don't think the landscape is already divided and set. Given those two premises, I think currently when I look at the players, I have to judge their current deployment. Everybody can say they will have thousands or hundreds of thousands of vehicles on the road, but actually given the current situation, I use the metric of having fully driverless commercial operation as a baseline. Given that as a factor, I think in the US we are definitely leading the way because Waymo already has four or five thousand vehicles on the road, four thousand plus. And then of course there are some other players trying to play catch-up, like Zoox, Cruise, maybe Tesla as well. So I would say in the US is way more leading the way. There are three to five players trying to play catch-up. In a global sense, from a technical point of view, China's players are certainly on par with the US players. But from the total cost or the economical sense of a vehicle, for example, our vehicles are four or five times cheaper than Waymo's vehicle. So in the global market such as Europe, such as Middle East, I think we will have a huge edge compared with the US players. Certainly the whole landscape is still evolving, but especially in the global markets, you'll see we will definitely not be playing catch-up but taking a leading position.
And you've been an advocate for hardware optimization, software optimization, battery solution optimization. Is that the strategy behind being able to have a vehicle that's four to five times cheaper than Waymo? Or where is the edge? How are you building these comparable vehicles at a relatively cheaper cost?
Yeah, I think as you mentioned, you definitely mentioned the most important factor to have a much cheaper price on the vehicles, which is optimization on hardware, software, and everything else. I think another reason of course is because the whole ecosystem related to autonomous driving is relatively mature and the scale is larger, so the price is cheaper. For example, the vehicle itself, the sensors, they are relatively cheaper in China than anywhere else because of the ecosystem, because of the scale. So that plays an important role as well.
