Pony.ai’s Founder and CEO James Peng on What It Takes to Scale Robotaxis

Yeah sure, we were founded in 2016 about 10 years ago. But even at that time, I didn't believe that autonomous driving could be solved in three to five years. From a technical point of view, because even back then, 10 years ago, even a demo for autonomous driving was already very hard, and let alone there's complexity involved in the autonomous driving industry that involves regulation, user acceptance, and the readiness of the ecosystem. So because of sheer complexity, even then my prediction was it would take at least a decade for this to be a real application. It turned out that my prediction was about right. Now, 10 years down the road, we actually have fully driverless commercial applications in many cities. Of course, it's just the beginning of the long journey for autonomous driving, but at least now we have real commercial applications. So, I think people, like any new industry, were super optimistic for the short term, but they underestimated the potential for the long term. So, I think autonomous driving is definitely one of those industries.

I think the motivation was twofold. One is that the potential, both commercially and also societal benefits, for autonomous driving is so huge. Think about everyone needs some sort of mobility, and autonomous driving is much safer than a human driver. So it has huge societal benefit of saving people's lives. So essentially, it's just such a great industry to work on, although back then, 10 years ago, it was very unclear when this could be done. The other reason is, of course, because of the sheer technical challenge of autonomous driving. In my previous jobs, I worked on different areas: software, hardware, large-scale distributed systems, AI, and whatnot. But none of the things I worked on is as complex as autonomous driving, which is a field that involves hardware, software, hardware-software integration, and many other things. There's AI, there's real-time system, there's also large-scale AI training, and all that. So from a sheer technical point of view, it's such an amazing and challenging thing to work on. So I think those two reasons propelled me to start the company.

No, I think 2022-23 were a period of time where the autonomous driving industry had evolved for roughly 10 years. I think that was the time of reckoning. That's the time where the haves and have-nots have really diverged. So I think that's exactly the time we, as a company, have seen tremendous progress. At the end of 2022, beginning of 2023, that was the time we actually finally had the first fully driverless commercial applications and operations on the road. So because we made such progress, both from a technical and also from a regulatory point of view, we made the progress and we think we're finally seeing the glimpse of hope, then of course we charged ahead. I think a lot of the other companies who weren't able to, either from a technical point of view or from a pure capital-raising point of view or from a regulatory approval point of view, weren't able to have fully driverless applications, then they faded away. So it's sort of like, well, everyone is in school, there's no big difference, but after graduation, then there are haves and have-nots. So I think that was the time of division.

Yeah, I think in my view, the biggest milestone I actually have already mentioned is the end of 2022, beginning of 2023, where we were granted the fully driverless commercial license in both Beijing and Guangzhou, and we started having the operation to the general public. Actually, it was in January, mid-January in 2023, that I was the first rider in our commercial robo taxi operations in Beijing. Surprisingly, it was exactly on that day, it was snowing in Beijing, and I was in the vehicle by myself. That was the moment where I actually saw our vehicles were able to drive by itself, and because of the snowing, it was also a very challenging scenario, and we were actually not being suspended for operation. We continued to operate, and I was in there. That was a moment finally it feels like a dream come true. Finally, not just because our technology is ready, but also because we actually got the approval from the government to have the license to operate. So it's like all the seven-plus years of efforts finally paid off. To me, that felt like the, as Neil Armstrong said, small step for a person but a giant leap for the human race. Although I wouldn't call it as big as the Apollo, but to me, it felt like it's finally from zero to one. So I think that was a deciding moment or defining moment for Pony.ai.

I think because it was a gradual process. In the operational domains, in the operational zone that we had, we used to have a safety operator behind the wheel, although the driver actually didn't touch the wheel or push the pedal, but people gradually got used to it. Actually, at the very beginning when we were just deployed in Guangzhou, in those days, if you look at the picture of our first and second generation of autonomous driving vehicles, you still see those spinning lidars on the top, and they were very much visible. People were curious, but gradually, people are just getting it as business as usual. And from a rider's point of view, the experience of a robo taxi is exactly like a typical taxi. The only difference is there's no driver inside the vehicle. The way you get the vehicle, the way you get in and get out, is exactly the same. And also the other traffic participants like pedestrians and cyclists, they get used to it. So I think it just takes time. It's just like the first cell phone came out, the first real smartphone came out, people were very curious, and now nobody cares. So I think it just takes time.

Absolutely.

Yeah. So the definition of the level of automation for vehicles were actually defined about 20 years ago. Of course, the industry evolved quite a bit. I don't think that the level zero to level five might be the right way of defining what the level of automation is. So in my opinion, actually there are two different products. One is we call the driver assist systems, ADAS. The other is fully driverless. So in a broad sense, I think there are two categories. There are definitely two different products. The biggest difference is not on the technical side, but rather as you mentioned, on the regulatory side, is who is first in line for the responsibility if there is ever an accident. I think for any ADAS system, any driver assistance system, it's always the driver behind the wheel that's responsible, regardless of whether he or she is looking at the road or has hands on the wheel. Whereas for the fully driverless systems, it's the system that's first in line. And because of that requirements, think about if there's a driver behind the wheel, it sort of serves as a safety net. So the system does not need to be bulletproof. It's probably, as long as it can handle 99% of the cases, it's probably good enough. Whereas for fully driverless, it has to be dealing with all the edge cases, all the extreme cases, and have fallback systems. We can get into those details later, but essentially in my opinion, there are two different products. Of course, for the driver assistance systems, there are different levels: you can be on highway, or only keeping in lane, or they can even be able to handle some of the automation in the urban environment. And for the fully driverless, of course, as you mentioned, there's L4, L5 in a traditional definition. L4 means in certain areas it can be fully driverless, L5 is everywhere. But I think it's never a clear division. Essentially, you can think of it as how we drive. We start with an area, but then we gradually improve, eventually we'll be everywhere. So I think that's just a gradual process instead of a clear division.

Um, no. I wouldn't call them a gap. I think it's a different product definition because they serve different purposes. I think most people view this as a process of evolution, from L0 to L2, L3, L4, but it's actually a wrong way of looking at it. As I already mentioned, because the clear definition, the clear difference is that who's first in line with responsibility. That's decided by regulatory actually, by product definition and by regulatory as well. So because of that, it's essentially two different products. As the product is getting more and more mature, getting more powerful, in my personal view, the division of two different products is getting wider instead of narrower.

I think that's the reason. I wouldn't say it's a single blocker or a single bottleneck that prevented us to grow faster. I think it's because of the sheer complexity of autonomous driving and what entails to ensure safety. There's regulatory, there's technical things, and also because this is such a brand new system, we need manufacturing capacity, we need deployment, we need to get all the operational things ready, like all the garage space and whatnot, and also user acceptance, user education, as we already mentioned. I think all those take time.

Absolutely. Absolutely. It takes time.

I think all the AI systems more or less take the same approach. You have data on the back end, on the data center side, you train a large model where you essentially try to get all the cases to be learned. In our case, we use the world model, which you can think of as a simulated city where we train the virtual driver and let it drive on all different kinds of roads and learn the driving ability. Then what is condensed as a model from all the learning, we deploy on the vehicle. In the traditional AI sense, I call it edge computing. You put on the edge, put on the devices, put on the car, where it's a much smaller model. In a human sense, it's like we learn everything, then when we go to a test, we don't need everything, we just need to be able to handle the test. So that's typically the training. The backend system needs a lot of computing, but on the actual usage side, you don't need that much computing power.

Absolutely.

Yes. It's a little bit different and they're also related. Again, use human as analogy. It's actually quite similar to how we think. Think about the large language model, how we process image, how we process knowledge, is sort of related to our memory and our logical areas of the brain. Whereas when we drive, it's not just the memory and our knowledge, it's also how we react, how we action, and all that. So the example is one is related to how we learn a new skill, that's a language model, whereas for driving, it's like how we learn to ride a bike. It's actually different types of brain, different types of skill sets. That's why it's different. It's not the same AI because that's how humans deal with different skills for knowledge.

Yeah, that's what I'm trying to get to. For language model, it's related with knowledge, with language, with logic. That means you need to be a very large model because the thing is, if you don't know a historical event, there's no way you know it. So you have to have all the knowledge of a human ever created in your model for it to be powerful. So that's why large language models require a lot of computing power and memory and everything. While for driving, it's like how we learn to ride a bike. We don't need to have a PhD degree to learn how to ride a bike. Rather, it requires a lot of practice and training. That's what the world model is. It's essentially a model where the virtual driver can start learning by itself to learn how to interact with other cars, cyclists, pedestrians, and then learn the driving skill out of that. So it's a bit related to large language model but it's quite different because it's related with action, with manipulation, with collision avoidance. So that's the key for the world model.

Yeah. So essentially, that's how we learn to drive. There are several key factors for the world model. One is it needs to be very real. So we call it fidelity. It needs to have high fidelity, meaning it resembles the real world. Second is that everything that moves in the world model, meaning cars, pedestrians, needs to be smart, meaning that because driving is an interactive process, our action will affect everybody else around us. So they need to react accordingly. So it's more like an interactive gaming where we react with everything else. So that's the second challenge: all the interaction needs to be smart, needs to be intelligent. And the third challenge is how do we evaluate what is good driving? You can avoid collision, is that a great driving? No, not enough, because there is a passenger inside. Comfort is important, efficiency is important from A to B. We want to use the minimum amount of time. So essentially, it's a multi-metric evaluation system to see what is good driving. So there are three key challenges for the world model, and we certainly put all our effort into developing the world model related to those three areas.

Absolutely. See, that's why the first thing is how we become a better driver. Essentially, it's a continuous learning process. The first thing is you avoid collision. You become a cautious driver. Then gradually you start learning all the signs, all the non-verbal cues and hand gestures. So that's exactly the case for us as well. The earlier model of our system is just driving and trying to avoid collision. Then gradually we put a lot of new things, new recognitions, new perception models into our system where we start recognizing, for example, the hand gestures, especially all the policemen's typical police gestures: stop, go, and all those things. And then we start recognizing, for example, on the road, small obstacles on the road. So it's sort of how we learn. We start getting the big picture first, and then we start learning all the nitty-gritty details down the road and put them to enhance our system. And regarding the second point, you'll see why when all the cars are autonomous driving by themselves, it will be easier to drive. Yes or no? Because the thing, especially in China on the roads, the biggest challenge is not the other vehicles, it's in a lot of cases cyclists and pedestrians. And we can't make them to be autonomous driving. So I think by having the ability to recognize pedestrians, recognize their intention, their sign, and giving you a specific example: on a crosswalk, the way pedestrians look and how they pay attention. For example, if they want to directly cross, they typically look straight, but if they are looking back, that means they will more likely not cross the crosswalk. So we actually take those cues to decide whether we let them go or we go straight ahead. So a lot of those details need to be put into the system to make it safer and at the same time to be efficient.

Yeah. Everything else. Yeah. Camera, lidar, we take all the sensor input and then we make the comprehensive decision based on the input.

Data is important but data is not everything. So how we understand that is, let me give you an example. Think about how we learn math. You can think of the data as the practice sets. Of course you need to do enough practice to have good knowledge about the subject, but that doesn't mean that if you have the problem sets of the whole world you become a math expert. So that's exactly the same case. We need enough data sets to know what the real world driving condition looks like, but we don't need everything because once we know enough, we can always generate enough knowledge about the driving. So in a way, I think the driving data is important, but it's not everything. So that's exactly how you view this.

Yeah, that's a very complex question to answer. I think first and foremost, let me get some premises on this. First, the whole mobility industry, especially related to autonomous driving, is very large. It certainly has enough room for several players. Second, it's still at a fairly early stage for fully driverless. I don't think the landscape is already divided and set. Given those two premises, I think currently when I look at the players, I have to judge their current deployment. Everybody can say they will have thousands or hundreds of thousands of vehicles on the road, but actually given the current situation, I use the metric of having fully driverless commercial operation as a baseline. Given that as a factor, I think in the US we are definitely leading the way because Waymo already has four or five thousand vehicles on the road, four thousand plus. And then of course there are some other players trying to play catch-up, like Zoox, Cruise, maybe Tesla as well. So I would say in the US is way more leading the way. There are three to five players trying to play catch-up. In a global sense, from a technical point of view, China's players are certainly on par with the US players. But from the total cost or the economical sense of a vehicle, for example, our vehicles are four or five times cheaper than Waymo's vehicle. So in the global market such as Europe, such as Middle East, I think we will have a huge edge compared with the US players. Certainly the whole landscape is still evolving, but especially in the global markets, you'll see we will definitely not be playing catch-up but taking a leading position.

Yeah, I think as you mentioned, you definitely mentioned the most important factor to have a much cheaper price on the vehicles, which is optimization on hardware, software, and everything else. I think another reason of course is because the whole ecosystem related to autonomous driving is relatively mature and the scale is larger, so the price is cheaper. For example, the vehicle itself, the sensors, they are relatively cheaper in China than anywhere else because of the ecosystem, because of the scale. So that plays an important role as well.

You're absolutely right. I think autonomous driving is probably the first large application of physical AI, and all the others, humanoids, robots, and everything else, probably will have real application down the road. For us, we view autonomous driving as our bread and butter. Of course, as I mentioned, this is still early stage. I think we still have a lot of mileage to go. For all the other physical AI applications, we don't have any specific plans yet, but I think they are definitely interrelated. We may enter them down the road depending on whether we need it or not. My judgment is that for physical AI, it will probably follow a similar trend as autonomous driving. It might take another decade for it to mature. For us, it's more about whether we have real applications for it. Giving you a specific example, even for our fleet, once we go to hundreds of thousands, millions of vehicles, how we maintain those vehicles, how we do the charging, cleaning, servicing, they may use robotic applications. So my view is that we will not probably do robotic action just for the sake of doing it, but we may do the related applications when we see the real applications.

Yeah, I think it's the same thing as autonomous driving 10 years ago.

Yeah, I think the mobility demand everywhere is the same. There's a strong demand across the globe. But we have to focus on the most important markets first. I think eventually we'll go everywhere because that's in our model. We have autonomous mobility everywhere. That's our ambition when we started 10 years ago. But for our first launch, we have several criteria. One is related to regulatory. It needs to have a relatively accommodating regulatory environment. Second, it needs to be a relatively mature mobility market. In a more obvious sense, the local taxi fares need to be relatively high because our pricing anchor point is always a human-driven taxi. So that price needs to be relatively okay. The third criteria is that we need a good local player to partner with because a lot of other things like regulatory and backend services need to be handled by the local partners. So judging from those three categories, I think Europe, Middle East, Southeast Asia, Japan, South Korea, Australia maybe, those will be the potential markets for the initial launch. Of course, those are already big enough in terms of number of countries. So we'll pick and choose some to start with.

Yeah, maybe I'll take one step back first. Think about what is a robo-taxi industry. The type of players, I'll divide them into four categories. One category is for user acquisition. Those are ride-hailing applications like Uber, Lyft, and Didi alike. The second is the vehicle. You need a car, so how you manufacture the car. The third is the driver. And the fourth is all the back-end services: cleaning, charging, servicing, insurance, and everything else. For us, our main job is creating a virtual driver, making a really safe, efficient driver. So that's definitely what we do. And all the other three categories, we might have partners or we might do ourselves. So depending on the market, depending on what strong local players exist, we might pick and choose players who handle one or two or three of the other things. For example, we work with ride-hailing platforms for user acquisition. We work with some of the back-end services that provide parking space, cleaning, charging our cars. We also have OEM partners that work on the cars. So that's how we view the partnership landscape.

Great question. That really depends on the different partnerships and different regulatory environments. In some markets, it's the local player who is first in line with managing the fleet. That means in those cases, we manufacture the cars with OEM.

It's like selling hardware. But we will of course have engineers handling the driving because we are in charge of the driving. So all the driving-related work will be done by us. But then the user acquisition, the cleaning, servicing, charging will be done by the local partner. So those are one case. But in some markets, we actually ship the vehicle and we apply licenses by ourselves. The vehicle still owns our own book. But those are rare cases. Our preferred model is to have the local partner that handles most of the logistics, and we will be the tech providers. We will essentially have the virtual drivers handling the driving, and everything else is done by the local partner.

I think in most cases, in my view, they probably will be partners instead of competitors. It's very different because they are mostly in the hardware business. Very few of them will be in the robo-taxi business because it's quite different.

Yes, as I already mentioned, think about our business. All our technology is that we are creating a safe virtual driver. Virtual driver is our core. As a driver, you should be able to drive all different types of vehicles. The two biggest applications for a driver are transportation of human beings and transportation of goods. That's related to robo-taxi and trucks. Within the logistics industry, there are actually three categories. One is for the long haul, typically done by heavy trucks, 18-wheelers. Then there is the in-city network, which is light-duty trucks. And then there is the last mile, typically handled by much smaller vehicles. Our main focus is on the long haul and the intra-city transportation. On the last mile, we are providers of domain controllers, but that's not an area we are working on. So think about it: we are creating a driver that should drive different types of vehicles. That's how we view trucks versus robo-taxi.

No, they will eventually be everywhere as well.

We start with ports. We start with some dedicated routes, for example, from a mine field to the local distribution center, those 30 to 50 mile routes.

The reason we start with those applications is mostly because of regulatory reasons. The ports and the dedicated routes are typically semi-private roads, so it's much easier to get regulatory approval. And of course we are working on long-haul trucks as well. We already have a fleet of heavy-duty trucks doing real goods transfers on highways, but still with a safety driver, and eventually will be fully driverless as well.

Again, I think for a robo-taxi commercial business to be a reality, three important factors: technology, regulatory approval, and user acceptance. I think within the last three to four years, we have gained a lot of experience on all three categories. The reason we are confident to deploy in 20 cities is because we have a clear vision on regulatory approval. There are a lot of cities globally, both in China and in some global cities, that are actually starting to come out with regulations supporting fully driverless commercial applications. And we also have partners that want them. So I think all the important factors are falling into place, which gives us confidence.

Yeah, actually driving is a hard job. Driving in a stuffy vehicle for 10 to 12 hours a day is a really tough job. And because autonomous driving itself is a highly regulated industry, the pace of our rollout is determined by the number of licenses. Also, a lot of the drivers are not young. The younger generation actually doesn't want to be drivers. So I think especially in a lot of global markets, we actually come in to fill the gap for the labor shortage for drivers. And it will not change human-driven vehicles overnight. It will be a gradual process. So that's the development of cities and human society. It takes time. It becomes gradually a norm, and then the drivers can find other jobs. Even we actually absorb a lot of jobs, for example, for remote assistance, maintenance, which are much safer and much less strenuous job conditions. So I think society as a whole always has a way to absorb jobs.

Yeah, it's not just replacing. Look at the truckers. The average age is 45 plus in North America. In China, they are 40 plus as well. So a lot of the younger generation say they don't want that type of job. And we are coming not only to replace, but actually to fill the void for that job shortage.

I think we probably covered a lot of topics.

To the city? Of course there were some challenges from a technical point of view. Two challenges initially. One is there are a lot of roundabouts. There were not many roundabouts in China. So although we could handle a lot of other very complex situations like heavy storms and all that, roundabouts we had some but not really trained that much. So we actually had to retrain a bit on roundabouts. The second is the trams. There were just a lot of trams in Zagreb, and their behavior is different from cars. So we needed a little more training to get used to it. But it's like how we drive when we go to a new city. We might not drive perfectly initially, but then we learn and adapt. Once we have a good learning system set up, we can quickly learn. That's exactly our experience in Zagreb, Croatia. Two things we had to learn: roundabouts and trams, because those are not something you typically see on the roads in China. For those new situations, it's like how we learn driving. When we go to a new city, we probably know 95 to 98% of situations, but some scenarios we didn't encounter previously. Then we learn and adapt. So that's exactly the case for us in Croatia. After three to four months of learning, training, and retraining, we were able to handle those cases like roundabouts and trams really well. And because there are a lot of roundabouts in other cities in Europe, and they have different rules for roundabouts, some have the cars outside the roundabout having right of way, some have the vehicles inside having right of way. But we can adapt once we have the system set up. So as I mentioned, the most important characteristic of our system is not how powerful it is, but how adaptive and how easy to learn it is, so that we were able to adapt.

Autonomous driving industry, I think people put too much focus on technology and probably underestimated the complexity of robo-taxi as a business. Of course, technology is the most important. If you can't drive safely, you won't have a business. But once you have the safest driving, you still have to, as a business, involve a lot of other things. For example, how you deploy a fleet, how you make the pick-up and drop-off easy for the user, how you handle all the edge cases of complaints from passengers, how you make charging, servicing, cleaning efficient. For example, electricity fares during the day fluctuate. If you charge at the low fare, you can save a lot of cost. Then how do you manage your fleet? Because if you have the low fare for electricity but the demand from passengers is really high, how do you make a decision? So essentially, it's a lot more optimization involved than just the driving itself. I think a lot of people underestimate the complexity of managing a fleet of autonomous vehicles. We as a company have put a lot of emphasis and taken a lot of efforts in optimizing everything. So that's why I think those will be a very strong competitive edge down the road.

Exactly. Exactly. Absolutely.

I think I'll take the one related to the answer of my first question. Again, people always put too much emphasis or give too much credit to zero to one and think less about one to ten. Give a lot of examples. People always think an invention is so hard, but putting an invention to be a scaled application is equally hard or a lot harder. Because what the scale involves: cost optimization, user education, regulatory approval, making things a lot easier to use. So many examples like this. Rockets put in the sky, they say it's so hard. But having the rockets always be able to safely take off and recycle, that's extremely hard. So I think related to autonomy, we certainly crossed zero to one. I think we crossed one to five maybe. But from five to ten, ten to one hundred, I think there will still be a lot of challenges ahead.

Thank you for having me.