Tim Kentley-Klay: Self-Driving Cars for Everyone [Entire Talk]

Can you give us a little story of how you got here? Maybe each of you can give us a little bit of time to tell your story of how you ended up here, sitting here on this stage.

Right.

I love this idea of bringing the horse intelligence to the car. I'm gonna look at my car differently. So Jesse, you've been thinking about this a long time. What brought you to this stage?

Yeah, so I got to spend a bunch of time at Stanford. I actually specifically came to Stanford, in large part, because Stanford started working on self-driving cars. And it seemed to me that of all the application of computer science and AI, self-driving cars had, by far, the most potential to impact society in a whole variety of positive ways. Economically, environmentally, and socially. And it was the first time that we'd sort of gotten some level of actual interaction between autonomous vehicles and other dynamic objects in the world. Which was really exciting. I got to work on a really small team and be part of that. And I kind of stayed in academia for a bit, instead of going to Google or one of the car companies. Because, although I really—

Well, a little bit of tennis, some photography. But I really wanted it turn into a product, but I wasn't really convinced that the car companies saw the future. Like how is this technology really gonna actually make it to the market? And Google is doing super cool stuff. I have a lot of friends there and they were doing amazing technology. But also, it just didn't seem like they quite knew what was the product going to be, right? And I didn't know either. So, I was sort of like, 'Well I'll just keep working on this research.' And then I met Tim and he started telling me about this idea for a ground-up autonomous vehicle that was unlike anything anyone else was talking about. And so, my first reaction was, 'Well this is probably nuts, right?' Because it's so different. I spent an hour with Tim, and I was like, 'Well, actually sounds pretty amazing.' And it seemed like it probably was the future of transportation in cities. So, we spent a few months getting to know each other. And I got really excited because I felt like I could take the work that some of my team and I had been doing at Stanford, and actually start to apply it to what I thought would in fact be the full realization of that technology. So, we got started, yeah about almost three years ago now.

So, okay. Paint a picture of what the future of autonomous vehicles are. So, okay. Think about five years from now. Hopefully you'll be launched by then. Ten years, maybe even twenty years.

That's the idea. They're actually quite intelligent. They have a lot of computing power, a lot of sensors. And I actually hope that they'll be a little smarter than horses. Somewhere in between horses and people.

Well.

They'll certainly be a lot more attentive. So, what are the trade-offs that you're considering right now as you're developing this technology, right? The hope is to launch when? When are we gonna see these on the road?

Wow.

So, I want to paint a picture. I had a chance to chat with you at another time and so I have this image in my mind of these vehicles that you're creating. And I understand that they're not like anything that we currently see on the road, right? Because if no one's ever, ever, gonna drive them, they don't need a steering wheel. They don't need a dashboard. They don't need windows. They don't need mirrors. How do you design a product like that from the ground up, when you really are not comparing it to something that already exists?

Super interesting. So, can you paint a picture of what this user experience will be when this thing comes up and picks me up at the Ferry Building?

Okay. Now, this is really important because you actually are in stealth mode, right? I'm gonna guess it's not because you don't know. It's because you just don't want to tell us. Are you choosing not to tell us what this experience is going to be like?

Well, so this is really important, right? A, your competitors aren't seeing it, but also, your potential customers aren't seeing it and able to give feedback, right? So how do you get feedback? And how do you set up the ability to pivot, right? To make some changes if you're going in the wrong direction. If you're not actually out there sharing what you're doing with the world.

So, Jesse as the CTO, and with all of this deep technical knowledge, how much are you actually having to build yourself versus buying off the shelf? Is this something that okay, all the technology actually already exists, and it's just piecing it together in an interesting way? Or are you actually reinventing the wheel? Haha. Or inventing the wheel.

There's a big gap between autonomous sort of features, that you can get on a car today, and then true Level 5 mobility, which means you get in a vehicle and it takes you where you want to go. And you really don't have to do anything. In fact, you can't even do anything other than say where you want to go. Or, 'Hey, I want to get out of the vehicle.' From a startup's perspective, one of the really exciting things is that we get to do the hardware and the software and the service jointly and holistically. I think people sometimes don't appreciate how big of an advantage that is. If you look at what Apple did with the iPhone, they didn't just write a new operating system and put it on a Motorola Razr or Blackberry. And they also didn't make a new hardware device that was running Windows Mobile Phone from 2006. They could have done either of those things, but I don't think it would have made nearly the impact that it did. And so, one of the fun things that we get to do but also the very difficult things is we get to think holistically about where do we put sensors? What type of sensors do we need? How much computing power do we need? It's probably more sensors and more computers than you're going to be able to buy on a car in the next couple of years. Because our business model is very different. We don't actually sell these vehicles to customers. We own and operate this fleet and then you can use them the way you use Uber and Lyft. They'll just hopefully be much better and even cheaper to operate 'cause you're not paying the driver. So, we do need to develop our own software, our own AI, our own algorithms. We're not using, you know, Mobileye or anybody else's AI technology. It's completely developed in-house at Zoox. When it comes to computers and sensors we, as much possible, buy things off the shelf or work with technology partners to integrate them into that design that cars today are not built for. Cars today are not built to drive a hundred thousand miles a year, for four years. And they're also not designed to be necessarily safe when a whole bunch of components might fail. The good news is we don't have to reinvent, for example, the entire concept of an electric car. The way Tesla had to do. What Tesla did 10 years ago, from a vehicle engineering perspective, was quite amazing. Very, very difficult. And there was really no supply chain for EV technology then. In the last decade, Tesla, and now a bunch of other companies, make electric cars in quite some scale. So if we're looking for batteries, and motors, and steering columns, and these sort of things, sometimes we might need to innovate a little bit with the supplier but for the most part, it's about building a new architecture but not reinventing every single hardware component in the vehicle.

And environment. What are the things that keep you up at night?

Yeah.

So, even though you're not telling us what you did, you've certainly told investors and they have given you a lot of money. You have been able to raise a tremendous amount for the vision that you've sold.

That really is a moonshot. Who do you see as your primary competitors? Is it folks like Google? Or is it folks who are doing something totally radical that you might not even know about?

That's an interesting question. It's actually a little hard to say at this point, right? Because nobody has Level 5 technology on the market, right?

Can you do us a favor and describe what that is? What Level 5 is?

Yeah, absolutely. So, the SAE came up with this series of autonomy sort of rankings. Level 1 is cruise control, basic stuff. Level 2 is you can do sort of adaptive cruise control and lane keeping. All right, so sort of like the Tesla autopilot system that you can get today. Level 3 means sort of a fully autonomous vehicle, but you still have to be sitting behind a steering wheel, paying attention. Because at any point in time, it might decide it doesn't really know what it's doing and you have to take over. But other than that, it can get through intersections, so maybe it's fully autonomous but only during the daytime. Or fully autonomous, but only in this particular chunk of the city, right? And then Level 5 means it's sort of fully autonomous, and it can drive anywhere. And pretty much anything a human can drive, it can drive. So of course that is the goal. That's what we're trying to get to at Zoox.

So, I'm curious though. It's really different if all the cars on the road are autonomous, and they all have the same rules, and the same etiquette. But when you have a car, an autonomous vehicle on the road, that is not being controlled at all by a person, and you also have human controlled cars, there's got to be some mismatch. I heard the other day about the concept of bullying autonomous vehicles, right? The people who know that, well they know the autonomous car is gonna be very polite, so you can cut it off.

Does it change color? Does it send out signals? Does it do something to say, 'Hey, I'm going from my friendly mode to my fierce mode?'

All of that? Okay. So, it really does have a personality. Okay. I'm not gonna ask some rude questions about what it does. Okay. When it gets mad. So, what is the business model, here? You eluded to the fact that this will be a fleet that will be out there. Is this a fleet that you'll control? You said it would be like Uber. Would Uber buy these and use them? Is this something or something that you're gonna be managing centrally?

They're just going to keep evolving.

Is gonna be pretty obsolete, right? If you look at the cell phone you had 10 years ago. It certainly wasn't an iPhone 'cause they actually didn't have iPhones 10 years ago. Although I'll only be able to say that for another month. But, it's kind of crazy, this idea that you buy a car and you buy a 50 thousand dollar car, or a Tesla 80, 100 thousand dollar car, right? And you know it has several thousand dollars of computers and sensors in it, and it's super cool, right? It's by the far, the best system on the market. But, in five years, 10 years, it's not gonna be very useful anymore. That's one of the things that we're excited about, is that if we own and operate our fleets, we can continuously upgrade the technology as it changes every year or two, which would be very frustrating if you bought a car and then a couple years later you're like oh not so much.

Is this dependent upon any changes in the ambient infrastructure?

The downside, the consequences of this technology? And how are you, or are you doing anything to try to mitigate that? So what are the things you're concerned about and how are you thinking about that?

1976.

Do you anticipate this essentially obliterating the traditional car industries? Will people still have a car?

Yeah, how about a pogo stick. I don't know. But, okay. Let's get some burning questions. And I'm gonna focus on students first, okay? Great. Back in the corner. Back corner.

I have two questions. First, when you're in the car, if you have an accident or if you have somebody coming, you have the human who just stops, who has reactions. Your car has two modes. A human being has more than two modes when he drives. How do you prevent an accident?

How do you prevent it because when there's two human beings behind wheels, they have emotions, reactions. A robot has no emotion.

Well, so the good news is that 96% of accidents are caused by human error, right?

Well, the good news, well I mean-- (laughing) It's good news for autonomous technology, right? And actually to add onto the previous question, we're talking about what is the dark side. The fact is 40 thousand Americans are dying every year in car accidents. More than a million are getting injured. And then globally, you have over a million people dying. So, when that's your starting point, right? There's a lot of ways to improve that technology. So you might have radar, you might have lidar. These are technologies that can see where things are in 3-D space. And so when you combine this 3-D reconstruction that you can do, with the semantic understanding that you get from your cameras, you actually do understand what's going on around you, and you can react to it much, much faster than a human can. So, in those situations where somebody else is doing something strange, somebody's making a weird turn, or they're speeding, or a person is jumping out and they're not paying attention. This type of technology will be able to react significantly, significantly faster than a human would. And it also won't make stupid mistakes the same way people do, because it's not texting or drunk, right? So, although there are very, very rare circumstances where some of this emotional stuff might come into play. I think we can get rid of probably around 95% of accidents, an order of magnitude improvement before we get to that level of AI.

Great. Back there. Yes.

Hi. So, the system that you're building from the vehicle, to the sort of app platform, has a lot of complicated pieces. If there was one technology that you would most want to see mature, if it could just get like 10 thousand percent better, what would you pick?

So the question is, this is a very large multi-dimensional problem. If there was one technology that you could put the gas on and really have a vast improvement, what would be your dream?

That's a good one. There's a whole bunch.

No, one.

Yeah, number one, I would say if somebody could make extremely good solid state lidar that could see out to 150 meters in all directions. It shoots out the beam of light, and then it measures how long it takes to bounce back. And then, you know the speed of light, so you can figure out how far away things are, right? And so what people have traditionally done today, is they've used a technology called mechanical spinning lidar. And it's these little things that go like this. And they shoot a few million points per second into the environment and then measure how long they take to get back. And you get what's called a 3-D point plot. It's a really good way of knowing where things are in 3-D space. The technology is still fairly expensive and it's not automotive grade yet. So, there are a whole bunch of people working on this, but it hasn't quite materialized yet. So that would be pretty cool.

Great. Lidar. Over there, back there. Yup.

So, if the car is gonna be completely autonomous, we have to deal with the problem of security of the car. How will you deal with cybersecurity threats?

So, how will you deal with cybersecurity threats?

And I think there will be a series of laws, I mean there already are laws, but I think there will be even more laws that make it extremely undesirable to mess with this stuff. But from a technical perspective, there's sort of two things you can do. The obvious one is just to have extremely good encryption, right? And just have a top tier security team, as well as you can actually pay hackers to try to break into your system, and then they find vulnerabilities and you fix them. So that's the sort of obvious thing. I think the less obvious thing is the way you actually build the architecture. So that these vehicles will be on a network, and able to do some communication, but you want to make sure that it's actually physically impossible to command them to do dangerous things. So when we build the sort of network and communication infrastructure, we're making sure that even if somebody were to try to instruct the vehicle to do something really dumb, it would just simply refuse to do that. The internal software would be designed to not be able to obey any type of a dangerous command, even if a hacker were to intercept it. So, that would be how it would handle it. But it's really obviously, an important topic and it's one we take very seriously.

Right there.

Yeah, I have a question about the autonomous driving strategy ecosystem. So, actually I think there are quite a few autonomous driving startups been founded in the past two years. And actually, it's quite a complicated thing to do, like when it's driving, it needs time, talent, and a lot of money, and resources. So Zoox's quite lucky, you had extreme talent, and you are well funded. But what do you think about those other small startups that started, just starting to do autonomous driving? What's their future, or what's their chance? Or is it a good chance to do that anymore?

Yeah, it's literally now like every two weeks there's a new self-driven car startup. Which is fun, right? Obviously, not everybody's gonna pull something off. I think one of the reasons there's a lot of excitement is that there's also a lot of acquisitions in this space. I think that's partly driving it. But, you know, if you take a step back, like Tim said, ground transportation is a 10 trillion dollar market. So, for all the money that VCs are putting into self-driving technology, you'll see that there was a pretty good return on investment there.

Good.

Are gonna try to jump into this space?

Well, I mean they are. I mean it's this is a real Black Swan event. Who knows who wins? I'm as interested as anyone else to see what happens.

We'll wait for the movie.

Cool. Okay.

Directly to that point about training the AI, getting to the level of reliability you need. It's been widely rumored that Zoox is using three reconstructions of the real world as video game simulations to train the AI.

Widely rumored?

So there are all these-- Oh, go ahead.

So tell us about your thoughts on the strengths and limits of using simulation. Something that WeMo says publicly on their websites that they use simulations to do training. It was reported by Bloomberg that WeMo, and Ford, and others are using simulation.

So the question is, are you using simulations to train these vehicles?

Yes, we are. It's actually really exciting because there's two things you can get out of simulations. So, we've labeled a whole bunch of data. But there's only so many cars you can see before you kind of learn what a car is, right? If you've labeled 10 million examples of cars in images, it's pretty obvious what's a car and what's not a car, right? And so, one of the reasons why companies like WeMo drive so many miles, or one of the things that Tesla has going for it, is they have these big fleets, they're getting lots of miles. It's not that they want to see 7 billion cars instead of 2 billion cars. It's they want to see what happens in these corner spaces, right? What happens when something goes super, super wrong? And they want to get examples of that. The interesting thing is not a very efficient way to get corner cases, right? I think WeMo's driven a few million miles so far, and they've seen, I think, they've publicly mentioned a few dozen examples of really weird stuff that they've seen. Really recreating 3-D models, and sort of dynamic models of cities, where you can generate synthetic lidar data, synthetic camera data, synthetic radar data. And you actually run your entire vehicle's AI driving software in that virtual world, and you can see how all of the software interacts. Because usually when something goes wrong, it's not just one thing failing. It's usually, there's some weird object and some sensor failed, and this other thing happened. It's usually two or three things interacting that caused the problem. And so, we can simulate a lot of those things happening at the same time, and then we can see how the vehicle would react. And then we can try to change the software and make it better. So we believe very heavily in simulation combined with lots, and lots of real world data. And I think, putting those together, you can build something really safe. And even if you did, it might not even reveal something. And so, you really need to have a very advanced simulator that's able to run a lot of regression testing, a lot of structure tests, a lot of virtual tests, to give you the confidence that the change that you're making is performing to the system that you can roll out.

Back there.

Who?

The two of you. One-- With the red sweater first and then the next one.

Okay. So, from what it sounds like you're saying what Zoox is planning in confidence have, like a line, basically building a line of taxi vehicles, autonomous system, and for the current moment a full simulation. Almost kind of a full game in that it has a full cityscape, as an advantage to do this? Especially as you mentioned, other competitors are focusing on more specific things. And yeah, how would you--

So the question is, this is a very complicated process, what make you, what's given you an advantage in this very complicated problem?

So, I'm curious. Maybe you could tell us the types of engineers you have working there, because that'll give us the sense of the range of types of problems you're trying to solve.

Great. Super.

Can I talk?

Could you stand up so we can hear you?

Could you talk about a defining moment that has shaped who you are today, whether it be professionally or tempermentally?

Great, I love that question. A defining moment in your career that's shaped who you are today.

Well, I want to just say, for Tim, this is interesting, right? You came from a very, very, very different background--

And did you feel like you had to prove something? Or just was, did you come away thinking you were a creative thinker?

Did you go back and talk to your second grade teacher?

Ah ha. Okay, well. We know a lot of those in this room, too. So, Jesse a defining moment.

Hmm that's a tough one. I think one that stands out for me is the DARPA Grand Challenge. They weren't gonna give us the digital map file until the morning of the race. But then they decided the night before, they'd give everybody the map file just to make sure there's no surprises, right? But we had frozen our code several days before and we weren't touching anything. And they gave us the map file and it turned out that it had about 20 times more of these sort of digital waypoints than we thought it would have. 'Cause every file they'd given us previously, if there was a straight road, there would just be one waypoint at the beginning and one at the end, and that was it. And all of a sudden, this file had every meter there was a waypoint. And so we tried to load it and it was just too slow and the entire thing didn't work. It was like 11:00pm and the race was the next morning. And we're like, 'Well this is not good.' So, it was actually just a really fun three hours because we had to come up with a whole new set of algorithms.

Sounds really fun.

It was super fun. We had to come up with a whole new algorithm. 'Okay, I think yours makes sense. Let's go try it.' So I coded it up, and it seemed like it worked, and then we ran it the next day. And it actually worked and then we got to watch our robot car drive for six hours and not screw up at all and come back to the end. To me, that was really exciting, 'cause I think before that race, it was like, 'Is any of this stuff even gonna work, or is it just too hard?' And then you see it do those six hours, and you see it come back, and you're like-- That was the first time where I felt like, obviously there was tons of work to do, but it was the first time where I felt like, 'Hey, this is really possible and it's coming at some point in the future.'

That's great. Well, so let's then talk about the business for a few minutes. You guys are co-founders. You come from really different backgrounds, different areas of expertise. How do you work together as co-founders?

'Yeah, I don't feel strongly about that.' I'm like, 'Cool.' Or sometimes he'll say, 'I feel quite strongly about that,' and we'll talk about it. And sometimes it's, 'I feel very strongly about this is not the right thing to do.' And so, it's kind of nice there's this sort of scale. And, when we both feel very, and it's not often that we actually feel very strongly that this is the wrong thing to do with each other. So the first thing I think you need is mutual respect, so you can listen to the person. And then I think when you do get to that situation, where you both feel very strongly about something, you really need to go back to first principles and decompose the problem and why it is. And fortunately for me, Jesse's one of the most rational people that I've ever met. And so, that's a good way to process it. And sometimes, sometimes the answer is, 'Well, let's just go and explore that thing, and if we're wrong, it's not the end of the world.' Or, let's go and get some data to see how that works.

We call that creative friction.

So I have a thousand more questions, but I'm gonna just ask one more. Because we just are gonna run out of time. And I would love to know what really motivates each of you? This is a really big, hard, problem. Something with a very long deadline. Long, sort of vision into the future.

That they don't necessarily understand, but they find to be amazing and they find to make their life better in some meaningful way. I love solving technological problems and coming up with creative solutions to them. Not for the sake of just writing a paper. I've done that. It's fun. I like going to conferences. But at the end of the day, when you can actually create something that somebody can use, and they don't understand how it works but they can tell that it's amazing, that's super rewarding for me. And that's actually why I wanted to work with Tim on Zoox, because I knew that at the end of the day, there will be other companies that figure out autonomous technology, right? We're not going to be the only one that ever figures this stuff out, right? And at the end of the day, the customers getting into these vehicles, are going to care a lot about the product, and the experience, and what is it like, and how do they work? And of course it has to work and be safe, and I love, obviously, the technology element of that.

I'm trying. (laughing) And so yeah, for me it's that you're connecting people and places. But for me, it's also, you know I have a background in animation. And what we're really doing here is animating matter. And we're animating in a way that it becomes independent from us. Silicon Valley was founded on computation. Silicon Valley, right? In a way, that computation now has been plugged into sensors and algorithms that can make sense magically of what's happening around it. And that's going down to electromechanical actuation. And that feedback loop can drive a device through the environment and that's robotics. And so, I find that fascinating as an animator and a creative person, is to figure out spatially and geometrically how to solve that product. 'And then explore the universe.' 'Cause ultimately, I think that's where intelligent robots go.

Wow. Thank you so much for animating us. (applauding)