TIM KENTLEY-KLAY: Real world continuous learning is the future of Robotics

Hey everybody, Anna, thank you for the introduction. It's great to be with you all today even though virtually. I'd like to thank the folks at Reinforce for convening this conference on the future of AI. On that theme, I'm going to present my conjecture as to why I think real-world continuous learning is the future of robotics. I'll do this in the context of my work and my new venture, Hyper, and the work that it's currently undertaking. At Hyper, we are a next-generation autonomous mobility robotics company, and we're on a mission to build next-gen mobility experiences. We're a crew of ambitious, creative, and technical thinkers acting to form atoms and AI into exquisite mobility products that people simply love. Here's a sneak peek at the underside of an extremely high-performance robot we are currently developing. Now, we're not going to be revealing that today. Today I'm going to be focusing on the AI we're developing to enable such a system. We believe the optimal way to build AI is to develop robotic systems that master their environment by interacting directly with it. As Anna said, our mantra is robots that learn as they move. Why do we think that? Well, one of the best ways to understand the future is to look into our past. It was around 550 million years ago on this planet that something super interesting happened. We went through what is known as the Cambrian explosion. Previously, for the last four billion years, not a lot had been happening. We had single-celled microbial life that didn't really move; it was just a gooey mat on the sea floor. And in around 20 million years, around 550 million years ago, this Cambrian explosion gave rise to modern animals—animals that grew brains, neural networks, exoskeletons, and could move around, forage for food, and had modern eyes. So what was the catalyst that enabled this explosion? Well, one thing these microbes were doing was generating oxygen through the process of photosynthesis, and the oxygen levels in the water hit a tipping point where it became fuel to allow more complex life to evolve. And we think that this planet is about to go through a similar catalyst moment. This time it's not oxygen; it's computation. We think computation is the new oxygen. If you look at what's been happening in robotics in the last sort of five or ten years, we've had some really cool breakthroughs in deep learning. For example, if you look at vision systems, it was around 2012 with AlexNet that a neural network could look at an image—a still image of 320 pixels wide—and say semantically, you know, this is a dog or this is a bottle of sake. That was considered a breakthrough at the time. And what's happened over the next sort of eight or nine years is we've gone from one still image to being able to process semantics at 4K resolution at 60 frames per second across multiple cameras. And that's really enabled by computation. We're creating silicon chips, TPUs, that are optimized for processing vast amounts of linear algebra. Now looking forward, we think that there's about to be an explosion in what's called reinforcement learning, which is a different branch of machine learning. A deep neural network really only works as a pattern recognition engine. You input a bunch of patterns and it will tell you semantically this is what I think I'm seeing. It's trained in an offline way. Reinforcement learning is a really different architecture within machine learning. With reinforcement learning, you create an agent—whether it's virtual or a robot in the physical environment—and you let it perform an action in that environment, and the sensors read in the consequence of that action. Then its systems, its neural networks, evaluate based on the action I performed: was it positive or negative? And I'm actually going to tune my own systems to then hopefully bias myself to be more positive. So it's a process of iteration, and it's actually a learning system. And it's going to be sample efficient, so it's going to require breakthroughs in compute to let these systems scale into the marketplace. But we think now is the time to do that. If we look at adjacent verticals to autonomous mobility in the AI community, we could look to DeepMind's AlphaStar, which plays a strategy game, StarCraft. This algorithm, by interacting with the game, first it was bootstrapped by imitation learning looking at human players play, but to get better than them it had to do what's called self-play. So the algorithm adversarially played itself and was able to get to superhuman performance using the architectures of reinforcement learning. This is super interesting work because StarCraft is a highly strategic game; it's also a game of limited information, unlike chess. And we think with Hyper, we can take these algorithms and apply them to the area of autonomous mobility, because one of the hardest things in autonomous mobility is the strategy piece, the game theoretic piece, prediction, understanding: who am I, what's happening around me, if I act, what is the consequence of that? This is the key thing. And we think by creating an agent that builds models of the environment, it can become very proficient. So hopefully you can see this graph. It's a little bit detailed for a slide, but I just want to take a moment to take you guys through it because I think it's super interesting. On the left going up we have scalability—you know, we want to build systems at scale so they can have impact. On the right along the bottom we have complexity increasing. And then we have a purple line which is dipping down, which is what's called an expert system, and we have a green line which is dipping up, which is called a learning system. And what we found is when you have a naive environment, a simple environment, it's very quick to use classical code to develop a system that has a very high proficiency. But as the environment becomes more complex—and autonomous driving is complex, it's a limited information environment, it requires high levels of abstract reasoning, and there's lots of cross-correlations between agents from a pedestrian to a traffic light to another vehicle—it becomes very complex very quickly. And what we've found is these expert systems, as complexity increases, they become brittle and they require more and more hand engineering. And this is kind of like a losing game, and this is part of the reason why we haven't seen self-driving cars arguably scaling in the marketplace just yet. Whereas learning systems, we find they're very slow at the start, but through iteration, through computation, through more data, they eventually become very, very proficient. And the more complex it becomes, the better they become. And so what we want to do at Hyper is build an end-to-end learning system. I also put three little calendar years on the top of this graph. On the left we have 2005, on the right we have 2025, in the middle we have 2015. People are always asking me, you know, Tim, when are self-driving cars coming? I was promised it at 2020. Well, if we go back to 2005, we had the DARPA Grand Challenge, and all the vehicles that entered that challenge were classically engineered systems; there was no machine learning on them at all. Around 2015, a lot of self-driving car companies were creating hybrid systems. So they have classical code, probabilistic planners, lidar, localization, but they're starting to fuse in some deep learning. So they're using deep learning for semantics to understand what the vision system's seeing; they're maybe using some models as well for prediction. But these are hybrid systems, and as you can see in that graph, they're kind of in this halfway house between these two different worlds, and they haven't really scaled into the marketplace. So we think in 2025, as we get computation, as we have decosted hardware and all that sort of stuff, we can really get to proficient self-driving by using end-to-end learning systems. And if you look at the overall time scale, 2005 to 2025, this is 20 years. You know, most humans when they're born, it takes 20 years to become a proficient driver. So this technology is coming, and it's coming really fast. It's arguably coming in one human generation. But once we have an expert driver's AI system, we're never going to look back. So let's take a little bit of a deeper dive into the current frameworks that self-driving car companies or autonomous vehicle companies are developing. This is a typical setup: you drive manually your vehicle with your sensors on it through an environment, you log that data, and then you take it offline, you compress that data into a 2D map, and then you annotate that with splines to tell you where the road boundaries are, where the pedestrians crossed geometrically, where the traffic light is. And then you load that map onto your self-driving car, it drives back through the environment, and using those same sensors, it scans the environment and says, okay, given that this is what I'm seeing, I can cross-reference that with my map; this is where I am on my map. Now for us at Hyper, when you create a system like that, what you're really doing is lobotomizing your robot. It's pretty gross. The vehicle has no ability to learn as it drives; it's not building models of itself and its environment. And so if you give it more data, it's not going to continue to learn. It's a hand-engineered system that is brittle and doesn't scale, in our view. And it's also kind of boring. If we look at the software frameworks that these companies are using, this is a typical architecture: you get some new software, you write some new code, you then take that software build and you put it through what's called a logged scenario. This is a prior recording of a vehicle driving through the environment, so you can take your new agent and inject it back into that log file and get it to play forward. That can work if you don't change too much of your software stack, but if you change too much, that log file is a static recording. So if your vehicle in a multi-agent scenario diverges, it breaks straight away. So it has very limited efficacy. Then you can go into simulation. People develop a lot of time, spend a lot of time developing simulators, downtowns, Unreal Engine, all the sensors, that kind of stuff. It creates a really complex setup, takes a lot of engineering effort, and there's always a delta between your simulator and the real world. And that delta is really complex because when you're trying to validate a system, you have to try and validate that delta, and it just creates kind of this downward spiral that's quite negative. You get through the simulation phase, you then compile your software stack, you load it onto the vehicle, you then go and test it on a track. If the testing on the track doesn't work, you actually, like snakes and ladders, have to go all the way back to the start and go through that whole process again. If you get through track testing, you can then go and do field testing. When you do field testing, the vehicle is driving through the environment, but it's just running inference; it's not a learning system. And you have safety drivers annotate where the system malfunctioned. They then take those metrics and they have a meeting offline with software engineers, and they try and analyze where the system is working, where it is not working, what do we need to do to improve it, how do we tune the knobs. Then you go through that whole process again. It's a very convoluted process; it takes like hundreds of thousands of software engineers, very, very complex. This is the Hyper framework: we have a robot, it passes its sensor data over a high-speed network like 5G to a reinforcement learning neural network in the cloud. The network in the cloud takes in that data structure, it has distributed compute in the cloud, it can break it apart, perform depth estimation, motion estimation, and then it can build dynamics models of itself and its environment. And then crucially, the neural network in the cloud actually gives actuator output back over the network to the vehicle. So what we've done here is we've actually given a neural network agency with a real vehicle in the real environment. This gets rid of all the other steps that people are doing. We don't have a model offline and online; the software is running online over the air in real time. And because we're giving the network agency over the vehicle, we've in a sense turned the real world into the neural network simulator, but we don't have the gap of the simulator. So it means that the neural network in a multi-agent environment moves a robot and someone else reacts to it; it's getting real data. So this obviously produces some challenges around safety and how you do this, and we'll talk about that a little bit later. But it's a very elegant, very straightforward architecture where the whole system is wired to be live and continue learning. When a gazelle is born, 20 minutes later it's running around. One of the questions we ask at Hyper is: what would the set of algorithms be to put on a robot that after 20 minutes of driving itself around, it's built a model of itself and can drive proficiently? We actually think we can do that. And this is quite interesting work because what we're really trying to do is understand the abstract algorithmic structures that you can put on any piece of hardware, and it should learn the model of that hardware and learn how to control it, and it should do it quickly. So at Hyper, we use simulator as a prototyping tool. We're also experimenting with remote control cars, electric go-karts, cars, and even a race car. But we actually have the same algorithms running on all those different hardware types. And we're going to be super excited later this year to run the experiments and see if that can actually be true—that our agent, because it's an abstract structure, can adapt to all these different pieces of hardware. So let's have a look at some of the development work that we've been doing at Hyper. Again, we use the simulator as a prototyping tool. We inject our agent into it; it can have an arbitrary physics model, arbitrary vehicle. It doesn't really matter about how accurate that model is because it's going to learn it, even if it's naive or complex. So here's our end-to-end learning system. We've gone to a racetrack for our first testing. We chose a racetrack because it's a controlled environment; it is hardware, it may do crazy things, so we don't have people around, we don't have a multi-agent environment. And we give the neural network three camera views: we have a forward camera, we have a local overhead camera, and then we have a global camera that shows the entire track. We feed those camera views into a convolutional neural network, and those neural networks distill the views into a latent space. We also put the IMU data from the vehicle into the latent space, so that's things like what steering, what the braking is doing, what the GPS location is. And then this is the interesting part: once we've created that latent space, we then have three policy frameworks that start to interpret that data structure. So the first thing that we have is a dynamics framework. The dynamics function starts to build a model of how the vehicle is moving through the environment, a model of itself. Then we have a policy framework. The policy framework starts to build a model of what are the rules of the system I'm in. So crucially, we don't actually tell our network what the rules are. What we do is we create a reward system. So in the case of the racetrack, we give a reward of a positive one if you increase your average speed over time, and we give a reward of negative one if you drive off the track. And so the policy framework, through interacting with the environment, is actually developed dynamically; it's not given as a prior. And the interesting part is once the network has built a dynamics model of itself and it's built a policy framework of the environment, it then has a value function. It says, okay, given this is how I can move, given this is the rules of the structure that I've inferred, what's the optimal way to move through it to maximize my reward? And so this is the structure that we set up at Hyper. So here's some development work we've done in the simulator. The company is 11 months old, so we apologize that this is a little bit basic visually, but what's happening here is actually really important. So we have our three cameras; you can see in the bottom left there's a little V that represents our vehicle, and we've got a basic physics engine running in the simulator. Now at this point, the network, none of the weights have been set; we've just set some hyperparameters. And so the network is randomly pushing the accelerator, pushing the brake, and turning the wheel. So it's wobbling; it has no real notion of the track or the policy, and you can see it drives off the track very quickly, and that's the end of the training episode. After around 500 iterations, where we're perturbing this network, we're using population-based training, so agents that get further along the track we select them, respawn them. When we get to 500 training episodes, you can see our agent is starting to learn that driving off the track is bad. So you can see here it sort of hits the edge, hits the edge, heads back, it gets to its first corner, and it can't handle the corner and drives off the track, and that's the end of the training episode. After around a thousand training episodes, we actually get to the point where the agent can drive completely around the racetrack without driving off it, but you can see it's not driving particularly well. And we have here one of our latest sims; we added sectors to the track where it gets rewards for going through each sector, and we've sped this up. We got to the point where with this physics model, our agent could drive around the track without driving off it at reasonable speed. And we also were excited to see what we're looking for, which is emergent behavior: the agent's actually cutting corners through the chicane, so it's picking approximate racing lines as part of its optimization. So we're quite happy with this little proof of concept we did. It showed us that by creating a reinforcement learning agent that's interacting with the environment, it can build a model of itself, the policy, and how to move through it. So the next step is to say, okay, let's try and deploy this onto real hardware. So we've got a Smart car, mainly because I thought it'd be funny to make the Smart car actually smart. They're also pretty cheap, so if we crash them, it's not too bad. Our hardware and electrical engineering team, which is actually one person, hacked into the vehicle, reverse engineered the CAN bus code, we took actuator control of the vehicle. The compute we're running on the vehicle is super minimal. We really love elegance at Hyper—elegance in terms of minimal hardware sets, minimal software development, algorithmic sets as well. These are very important to creating systems that can deal with complexity and can scale in the marketplace. So in the trunk of the car, we have a laptop that's actually not doing anything to do with self-driving; that's actually just to log in to the Jetson computer, which is running around 30 watts of power. And then the yellow brick that you can see there, that's actually just the remote kill switch hardware, so we can remotely stop the vehicle if we want to. So it's a very simple setup. We also have GPS, and that system will take the telemetry of the vehicle, the location of the vehicle, and broadcast it over a network to our neural network in the cloud. We're also being thoughtful about how we're doing this. You know, you can't just let hardware and a neural network go and explore in the real world; it's going to hurt itself or someone else will crash pretty quickly. So we've created an app we call the Hyper Grid, and this app interfaces with our hardware and the neural network and lets us run experiments. So in this case here, we're defining what we call a play space. So we can load in Google Maps in 2D to the location that we're testing, and then we can map out a polygon on Google Maps, and then those polygon points are converted into GPS locations, and that becomes the boundary space in which the vehicle is allowed to explore. If it drives outside that, it'll trigger an e-stop. We've also built other structures in as well, like the notion of a drive envelope. You know, when we're learning to drive a vehicle, we don't start out on a 1000cc motorbike; we start out on a tricycle as a kid, right? And then we go to a bicycle with training wheels, and then we take the training wheels off, and then you might go from a bicycle to a moped to a motorbike to a high-performance motorbike. And really, with these learning systems, you need to take exactly the same framework to bring them up safely in the environment. If you're purely in sim, go for it; you can just put high performance on there and let it crash over a thousand training steps for episodes. But with real hardware, that gets too expensive too quickly. So we need to be thoughtful about how we make it proficient without damaging our hardware. So we created this notion of a drive envelope, which mimics sort of trike to bike. So we can limit the speed, we can limit the rate of change of the steering, we can limit the braking authority, and create a dampened vehicle and say, okay, network, go and learn baby steps within these parameters. And once you've mastered that, we'll give you a little bit more actuator authority and give you a reward for exploring that new space, so you can become proficient in that as well. And so what we're looking for is a technique where you can actually bring this hardware up safely in the environment. You can see at the top here we also have some data; we call it robot spots, meaning that the robot is online on the track, the cloud is up, meaning our neural network's plugged in, the network's up. When we're at the track, we actually use a Wi-Fi mesh network, which gives us a latency of around two to four milliseconds round trip from the vehicle to the cloud and back again. So even though we're doing the compute off-board, it's super low latency and super fast. So here's our setup: we have our Smart car on the track broadcasting over a Wi-Fi mesh network to our neural network in the cloud. So we're off. So what we're in the mode here is an exploration mode. So we have our play space; we've said, okay, here's a rectangle in which you can drive, and the vehicle's going for a random walk. So it's only allowed to go up to 12 miles per hour or 20 kilometers per hour, but it's actually just randomly moving the steering wheel, pushing the brakes, pushing the accelerator. And the dynamics model is actually trying to predict based on these actuator inputs at the next time step where would I be geometrically in the environment, and the GPS gives it the ground truth of where it would be the next time step. So it might predict, you know, I'm going to be here, and it ends up here, but through iteration over time, it actually builds a model of dynamics. And this is very similar to how we would do it: we get in a vehicle, we drive around, and we sort of get a sense for how the vehicle moves. And we think this is pretty exciting work because typical AV companies would have a team of say 10 to 20 control systems engineers trying to control the vehicle. We're doing it with just a few hundred lines of code as a learning system, which is pretty interesting. So once the vehicle is calibrated—and it's also an online calibration system as well, so as the vehicle continues to drive, the brakes get hot, the tires wear, it will continuously update its model of its dynamics—we then go into an exploitation mode. So it may be a little bit hard to see, but we have some traffic cones out here; we have a basic racetrack that we set up. The drive envelope is the same as when we're in exploration mode; it can only go 12 miles per hour. And so you can see here, even though the vehicle has a dynamics model, it's not sure how to plan with that dynamics model. So it's now learning the policy of the road network. When it drives off the track, that's negative, and so it's learning to stay on the track. So you can see here when it drives off, okay, we stopped the vehicle straight away, and it learned that that's a bad thing to do. Now what we found is after about 20 minutes of driving, we can get to this level of proficiency. So the vehicle actually drives around this track, not super well, it's quite wide and it can go so fast, you can see it's actually going to do a whole loop of this track as an end-to-end learning system. So it's learned its own dynamics, it's learned the policy of the track—staying on is good, driving off it is bad—and then its value function is saying, okay, given that that's what I'm trying to do, this is how I should drive. And so we're pretty excited that sort of 11 months into the company, we have this very simple proof of concept of a car that has essentially learned itself how to drive. And whether it's a track or a lane on a freeway or a dirt track, we think these algorithms can scale to more complex environments. And then we just have a last shot here which we kind of liked. You know, when you're a roboticist, one thing you love seeing is emergent behavior and your vehicle doing the right thing. And so you can see as it comes around this corner here, it's about to go off the track, and it taps the brake lights to cut back in. So we love those little moments. So in summary, at Hyper, we have a new pipeline for a new paradigm: robots that are cloud-connected for continual learning. We also have new paradigms for safety, things like setting up drive envelopes for safe performance exploration. If we were in an urban driving setting, we have a safety driver behind the steering wheel. There are other techniques that we're going to develop as well. This is an area of pioneering thinking about how you do this, how you let real hardware explore safely in the environment, but we're excited to go figure that out. And then you could also ask about the validation piece. So, hey Tim, this learning system, how do you ever know it's safe enough to be autonomous, right? How do you turn this into a product? Well, one interesting point is that it doesn't matter whether you have an end-to-end learning system or you have an expert system with some machine learning in it; you're actually in the same bucket. You're actually in the same problem space that both of those systems are too complex to be validated in traditional ways, say an aerospace where you do peer review code or something like that. It's not going to work with these techniques. And so the only way you can really validate these systems is having a fleet and driving a lot of miles and stochastically sampling the environment and having data to show that you're statistically safe. This is the way to do it with a learning system, and it's actually the way that everyone is going to have to do it. And so at Hyper, we are going to build a robo taxi; it's going to be super beautiful. That's how we're going to validate these systems at scale. There's some other interesting work you can do as well. You know, when the neural networks are doing a model of themselves and their environment, it means they can also clone that model of themselves and create another virtual agent and then train adversarially against each other. And this can be some interesting ways you can also validate your system, using multiple neural networks to be adversarial against each other to find areas where they may break. So there's some interesting work happening in that area as well. Finally, I just want to share this quote from Sergey Levine. I think he's an awesome assistant professor at Berkeley, very passionate about reinforcement learning and robotics. And last week in an article, he said this: 'I think real-world tasks present the greatest challenge but also the greatest opportunity for reinforcement learning.' You know, what we're doing is hard, but we think it's going to scale. We actually think this is the way all robotic systems will learn in the future. They'll essentially be edge computers connected to the cloud that do distributed learning and can do it safely in the environment. So Hyper is all about robots that learn as they move. Again, not showing it today, but here's a sneak peek of something we're building which is super exciting and super fast. Thank you very much.

Yeah, that's a really good question, straight to a super technical one. I like it. So we're actually using a soft actor-critic algorithm at the moment, which is working quite well for us. We actually have some more advanced stuff than what we shared today, which is pretty interesting. We're not quite ready to show that yet, but we've actually got pretty high performance in simulation going around racetracks at this point. But we're going to move beyond that algorithm to one of our own that we're building, which has our own structures about how we think about it. And hopefully later this year, we'll put out a paper on how that system works.

Sorry, implement what? Absolutely, moral compass? So adding a moral compass?

Yeah, I mean, I think there's been a lot of questions about the trolley problem; it's been a well-covered area. Firstly, I think it's a bit of an academic question; it doesn't really happen too much in the real world that you have to sort of make a trade-off between one scenario or another. The road network is a policy structure; there are legal structures as well. So the reality is that if you're in your lane and there's an object in front of you, you're going to decelerate as quickly as you can, and that's going to be the position that you're going to take. And so I'm not sure that that scenario actually presents too much in the real world. The other thing is the algorithm will try and minimize delta V or velocity, so you want to take energy out of the system, and so it would choose the maximum amount of path that it had to come to a stop rather than a smaller one. So inherently, it will avoid obstacles and bad things from happening. Yeah, I mean, that's really one of the key motivations behind Hyper. You know, what's the hardest part of self-driving? I don't think it's perception; I don't think it's planning. I think it's actually prediction, which is an aspect of both of those elements. But it's really in a multi-agent environment having a tactical understanding of who am I, what's happening around me, and what should I do. Because if you get that prediction right, you have a very big business and a great product; if you get it wrong, you're out of business and off the road very quickly. And that's why we're interested in using reinforcement learning, because its superpower in a way is prediction, and it becomes very good at prediction by building models. It's these internal models working in a constellation that allow it to do very good prediction. Whereas if you use other frameworks, like we're saying, you just lobotomize your robot by doing linear localization; it has no sense of itself and what's happening. And so we think to really solve the problem, you need to go really deep and build really complex models where the system actually is understanding what's happening around itself to then make the right decisions.

Yeah, it's quite technical. Maybe in one of the chat groups after the call, we could go into some of that. It'd be good to get some of the software team to answer those questions as well. Would probably be better for him.

Sorry, it's a little bit hard to hear. How would we prevent the agent from learning the exact lane in which he's going? How would we prevent it, or how would we let it learn to be in the right lane? I guess my interpretation is that if you don't have lanes available, how do you prevent that? Actually, what the agent is doing is just relying on the lanes, because then maybe there won't be lanes. Yeah, so I mean, this is really one of the interesting parts, right? These agents dealing with ambiguity. Again, in simulation you don't get that, but in the real world you definitely do. And certainly driving around San Francisco and Alameda where we are, there's a lot of roads without good lane markings. I think if you are testing in say Singapore where everything is manicured, you could really rely on the lanes, and it would probably be a lot easier in some ways. And so again, the way our system will work is the perception system will say, here's free space in front of the vehicle, and again through iteration, it'll learn to generalize whether those lanes are there or not, what is the right thing to do. And what we're really looking to do is create an expert driver. So rather than trying to say what's every corner case, what's every long tail issue, let's go and get data or try and train on this specifically, you're actually trying to create an agent that is good at making trade-offs with ambiguity. And having lanes or not is a good example. And so just like we work as an expert driver, we don't go and test against every scenario, but when we do come across scenarios, we're good at generalizing enough to make the right trade-offs and make the right decision, which again comes back to the prediction piece. And so this is some of the more tricky stuff we have to do, but we think we're developing in the right direction to solve some of these problems. And so it's an example of exactly where we think our system will do well, is in ambiguity where it can generalize across these areas because you can make the right set of trade-offs.

Yeah, really good question. I mean, if you think about it, even the computers on the vehicle, you can still have a cable come loose; it's the same situation as the network going down in the cloud. And so whether you have either scenario, you need to have a strategy for dealing with that, and these are pretty solved issues; they're not too complicated. So whether we had the AI computer in the cloud or the AI computer on the vehicle, if either one of those was to crash or lose power for example, you have a lower-level computer on the vehicle that has an instruction set of this is how you safely stop. So the vehicle would go into a no-go trajectory and it would decelerate on the last good trajectory that it received to stop the vehicle safely. That said, I mean, these outages happen very, very infrequently, and so it's okay to stop the vehicle if it happens occasionally; it's not really a problem. But like I was saying, you do need to be thoughtful about how you engineer those systems. Definitely what's not going to happen is we lose connectivity and the vehicle just disappears or doesn't know what to do; it'll have a strategy. The other thing I'd point out is you always need to look at development, temporarily, what stage of development am I at. If you're first bringing up the vehicle, it's going to be doing a lot of training in the cloud and using a lot of cloud power, so it's highly dependent on that network. Whereas once it's trained and proficient, and the weights of that model are pushed locally to the vehicle, it will then run in inference locally on the vehicle, and it doesn't require all the data and all that to send to the cloud and back, because that's also quite expensive potentially to send that amount of data. And so you also need to look at what part of development we're doing, and when the system is more proficient, it'll use the cloud less. What's interesting is we've kind of hybridized our system so it can run locally in inference, but if it comes across a feature it's not sure about, it can instantly send that data to the cloud to start learning on it, and then once it's learned on it, it can then go back into its own model. So again, it's a real-time distributed learning system, which is pretty exciting.

Yeah, that's a really good question. We get this one a little bit: okay, you can drive on a racetrack, but can you drive on a surface street? We'll be demonstrating that later this year, but we're very sure that the abstract principles behind driving on a track or staying in your lane are the same. And this is what's kind of exciting about what we're doing. In the same way that the same agent can adapt to different hardware, the same agent on that hardware should adapt to different environments that it drives through. We'll prove that later this year that we think we can get that working.

Yeah, really good question. I think there's some really interesting work happening with imitation learning. What we're interested in with a commercial product is robustness. So with imitation learning, if you shoot footage of a car driving down a freeway and you then play that back to a neural network, it can learn pretty quickly that okay, when I see this straight data structure, I should stay in the lane. That's actually not too hard. The problem is that when the vehicle is perturbed off nominal—so for whatever reasons it goes out of its lane, another vehicle swerves or something like this—it then doesn't know what to do because it hasn't seen that data structure before, and so it becomes brittle. And so imitation learning, I think if you're trying to use that as a development tool to get to an end state of autonomy, I don't know that it's going to work because it's not going to work off nominal. I think where it can work is as part of your training pipeline. So as we were saying before, reinforcement learning is computationally dense, and so one of the ways you can initialize your system to bootstrap it is letting it observe drivers drive first, which will set a certain set of weights and biases within the system. But once you have that happening, you then need to let it use self-play and give it direct authority over the environment so it can perturb itself, go off nominal. We can even do scripted scenarios where we put in virtual assets it has to react to, to create that robustness. And so we think it's a good step, but we don't think it's a full solution. Imitation learning might give some cool demo videos that look pretty impressive, but we don't know that they're fully robust. One of the things we sort of found with racing on the track is our vehicle can actually drive off the track, but it has a notion of how to drive back on the track, and that's the kind of robustness that we're interested in creating.

Yeah, good question. And you know, people should be asking these questions because this is a new sort of framework of how to do this sort of cloud-based federated learning with robotic hardware in the loop. Again, you need to look at what the temporal state of your vehicle is in its development pipeline. When we're learning to drive a car, we often start on a quiet street; we don't go straight into the center of the city. It's the same thing here. So when our system is first learning to drive, we're going to do it in a really thoughtful way in a controlled environment. But once it's hit certain levels of proficiency, we can then put it into more complex environments; we'll have safety drivers behind the wheel for example. But the goal isn't to always be sending all data off the vehicle to the cloud and doing actuator control in the cloud and sending it back to the vehicle. That's really for the training phase. Once it's at good proficiency, that model can run locally on the vehicle. To run a model that's trained doesn't require a lot of compute, as I said before. And so we don't see that as being a dependency in the end state of the system. But that said, the system, if it does come across new data, can instantly send that to the cloud to start running on that novel data.

Yeah, I think Tesla is definitely going to solve full autonomy, and I think they're going to do it at scale, and I think they're going to do it faster than anyone else because Tesla can do that and put it in their fleet, which is already at scale. Whereas for someone else to create a fleet and get to that scale, even if they had autonomy, would take a reasonable amount of time. So we think they're going to get there. Part of it is they do have access to a lot of data, but again, that data is static; it's a recorded file. And so the key differentiator that Hyper is doing is we're not doing static recorded files for imitation learning or for offline supervised training on vision. We're giving the actual vehicle real agency in the environment. And we're very curious to see if that means we can get to the same level of proficiency as Tesla or more with actually less data and less training time. So it's going to be an interesting experiment to run.