The equation, I think, for starting a robotic business has changed and will continue to change at an accelerating pace because the upfront cost is not that high anymore. Everyone's sort of spending a lot of time in the digital world and it feels like, you know, now is the time to start thinking about the world of atoms. You literally just gave people the playbook for how to build a vertical robotics company. This has really been our mission from the start: is to create that Cambrian explosion. It still, like, blows my mind. I didn't know if this would exist even in my entire lifetime.
Welcome back to another episode of the Lightcone. Today we have a very special guest, Karol Hausman. He's one of the co-founders of Physical Intelligence, which we think might be the robotics AI lab that brings about the GPT-1 moment for all of robotics. Karol, thank you for joining us.
Pleasure to be here. I've been a longtime admirer of YC, and our mission is to build a model that can control any robot to do any task that it is physically capable of, and to do so at such a high level of performance that it's going to be useful to people in all walks of life. And so GPT-1 for robotics, you know, what is it? You know, is the ChatGPT moment for robotics? Our perspective here is that we want to build a model that's really intelligence. We want to build a platform that allows us to externalize that intelligence to the rest of the world and allow them to use it to build very interesting applications in all sorts of verticals in robotics. And we think that it's going to be more like a peeling an onion analogy, where you start from a really strong base model that has all sorts of common sense knowledge and already works to some extent on your robot. You have then a mixed autonomy system, very similar, for example, to an autonomous driving car today. And then you actually deploy that system to do a real job. That system might make mistakes, it's okay. And then over time, by actually exposing the system to the complexity and the edge cases of the real world, that system gets incrementally, even just slightly better over time, every day. And, you know, one day you wake up and you certainly have a system that is just fully autonomous and just provides tremendous value.
Might be helpful to give the audience a bit of a mini history lesson on why robotics is so hard. And there's been a lot of breakthroughs in the last two years. And I mean, just to simplify, the robotics problem is three pillars: semantics, which I think we got a lot of unlocks with language models that somehow we ported into robotics. Then you have the planning, and then the last thing is control, which needs to be done in real time and interact with an environment that changes. Walk us through the seminal papers that a lot of the team at PI Robotics published that gave you the inkling that the GPT-1 moment is near, and that started in 2024.
Yeah. So the dream to build general-purpose robots, like robots, has been a longtime dream, I think, in humanity. Like, you know, we're not the first to say that our mission is to build a model that can work on any robot. And we're really fortunate to be in this moment in time in history where we feel that it's possible. To kind of walk back a little bit, a few years before, there was, I think, the first is SayCan, which to me was the first demonstration of language models and how you can bring all of the common sense knowledge in language models into robotics, and therefore that significantly kind of reduces the need to collect robot-specific data. So, for example, if you have a task of, oh, I want to go to the YC office to record a podcast, you know, what are the steps I need to take? You can ask a language model, you know, just show me the steps and show me the plan, and that worked incredibly well. And then the way kind of language models infiltrate, if you will, in robotics is it starts at the planning level, at the semantic level, and then, but there's still the control problem. You know, at the end of the day, you still need a mechanism to convert the plan into low-level actions that can actually actuate the robot. And that brings us to PaLM-E, and that brings us to RT-2, which stands for Robotic Transformer 2. And what these two works really show is that if you start from a vision-language model that is really powerful and you kind of use robotic data to adapt this model to speak robot language, if you will, then you see a lot of transfer from the kind of knowledge that exists in the language, in the vision-language model, down to the low-level actions. Like, one of my favorite examples when we did the RT-2 project was you can have a picture of a celebrity on the table, you have a picture of Taylor Swift, you have a picture of the Queen of England, and you can ask the robot, you know, pick up the Coke can and move it to Taylor Swift, even though the concept of Taylor Swift just doesn't exist in the robot data at all, and that worked. You can do other examples such as kind of spatial reasoning that doesn't exist in the robot data at all. Like, for example, move the dinosaurs next to the, like, red car, and these are always just completely unseen objects in robot data. And so that was RT-2, and that was PaLM-E. Now, RT-2 and PaLM-E are single-embodiment exercises.
Just for the audience, single embodiment meaning it worked for a very specific robot. It worked for a very specific robot. In robotics, you can ask the question, how do you scale? Especially, how do you scale data collections? And one of the insights that we had back then was, you know, maybe the data from one robot is not that different from another robot's. Anyway, if you have enough robots in your training data, maybe what the model learns isn't to control one specific robot. What the model learns is something that's more abstract, which is how do I kind of learn a general notion of what it means to control any particular robotic platform, and therefore I will be better at controlling any particular platform. And that brings us to what we call Open X-Embodiment and Robotic Transformer X. That was a big paper because it was the first that showed potential scaling laws that apply to robotics, because now you could start training all these models across multiple kinds of hardware, not just one, which has never been done in robotics ever before. Because from all the research labs, they would all train with a very specific set of sensors, actuators, and motors, and it was all very finicky with that particular hardware, right?
Yeah. One of the really interesting results from Open X-Embodiment, and let me provide the context here, is that you can take, let's say, 10 different robot platforms, collect data from them, train a policy, and really optimize the policy to work well on that platform. So let's say, you know, you have that, you have 10 different platforms, 10 different policies, and now if you simply take the data and absorb it into a model that is high-capacity enough to really absorb that data, and you can compare, you have this generalist, right, that learned to control how to the 10 different robots, you can compare it to the specialist that has been optimized to work well on a particular embodiment. How does it compare? And the interesting result from Open X is it was 50% better.
And that was really surprising because in robotics, it's hard enough to get your model to work on one particular robot platform. And one of the reasons why I say that we're really fortunate to be in this moment in time in robotics is because Open X was really only possible because of the support that we received from the robotics community. It was a huge collaboration across the robotics community. And the reason why that's really important is there is this joke in, like, robotics grad school that, you know, if you want to add two years to your PhD, just work on a new robot platform.
You know, by that logic, if you want to have 10 robot platforms, that's 20 years, like...
Why is that? It takes like a year or two to just get the platform up and running to even collect the data.
Yeah. Is it fair to say that the dataset that was created from X-Embodiment is similar to the scale of an impact that ImageNet did for vision? Because it was huge, and it was the first large dataset across multiple hardware, huge collaboration, and...
I still think that ImageNet was more impactful in the vision community, and the reason for that is a few. The first is that ImageNet also allowed for reproducible evaluation, right? Open X as an effort was more about making data available for kind of people to use, and evaluation is a really difficult problem in robotics that Open X did not solve. And the second is I think Open X is a drop in the bucket at this point in the robotics community. If you measure in the kind of the scale and the volume and the diversity of data that the community is collecting, I think Open X at this point is a drop in the bucket.
I mean, I guess we started talking about sort of GPT-1, but even GPT-1, you know, that was sort of this moment where you can prove, you know, Alec Radford figured out that there was a neuron based on a very specific input and output. And then that allowed the scaling laws to sort of take hold. The biggest problem in robotics I've heard is basically exactly what we've been talking about, is like, it's the data problem. You know, language you could bootstrap off of, like, you know, the sum total of what you could get off the internet, which is actually quite a lot. Can you give us, like, a sense for, like, scale? Is it like petabytes? Like, you know, what do you think is necessary as an input to, you know, the true GPT-1 of robotics?
Yeah. So the data scarcity problem in robotics, there's a few ways to look at it. The first way is that it's really two problems in disguise. There is the data generation problem, and there's the data capture problem. And the difference is that the data capture is that there might already be lots of robotic data that is being generated, but there's just never been really an incentive to capture it, to make it easy for digestion in training. And that's one of the goals that Open X was trying to solve, which is if you have robotic data, it's a really good idea to capture it and make it possible to train on. The second way to look at it is that robotics is very different from language models. There is not an internet of robotic data that you can use. And so you see this kind of very operationally heavy effort to collect data. And there's the question, is it going to scale? Well, the way that I look at it is, let's take the US GDP, $24 trillion US. Let's say if we actually solve robotics, a model that can control any robot to do any task, napkin math, maybe contribute 10% to US GDP. Well, that's already a massive number. And I think that promise is one of the reasons that warrants the investment into data collections in robotics. And the third way to look at it is we're very focused on cross-embodiment. And cross-embodiment, there is the data collection aspect of it as well, which is to really make sure that your model and your organizations and infrastructure are set up to consume data from many different sources of robots. And that actually allows you to scale easier. For example, if I were to contrast our approach compared to, let's say, a company that has a particular hardware platform that they optimize for and they scale, it's not an approach that has really allowed people to scale, because it's just much harder to figure out how do you manufacture, like, a thousand units of something for now compared to making sure that you yourself are ready to absorb data from, like, a thousand different types of robots that are already in there in the community.
I mean, it's a crazy problem, isn't it? I mean, the hardware itself, even within the same design of embodiment, if there's a hardware run that goes awry, or like one of the servos is slightly different, like you see it in the data, right? And then how do you control for that?
Yeah, so I think we were doing kind of like an inventory of robots in the company. We were so shocked to find that there are no robot, no two robot platforms that are the same. And if you ask people in the robotics community, sometimes there's debate about multi-robot versus single robot. And the argument is that, you know, single robot is simpler to scale. And actually, that's not how it plays out in practice. Like how it plays out in practice is even if you have a single robot that you're optimizing for, over time that platform is going to drift. You know, maybe you want to make hardware change or you have software change. You end up in a situation where it's much harder for you to reuse old data because, you know, in machine learning, if you want to generalize from a distribution, you would like many samples from that distribution. And if you just have one robot platform that has a major change every three months, maybe you have a few data points from that distribution. Whereas if you start from the hypothesis that if you have many robot platforms in your fleet, your model is going to learn something more abstract, which is how do I control a robot, not any particular robot, then the model will be able to ingest data from, you know, a slightly different robot better.
And actually, we're starting to see emergent properties in this kind of robot large foundation model. That's good news.
Where you start to see like interesting transfer between different data sources. For example, today it's possible to perform tasks zero-shot. Zero-shot meaning you don't collect any data, and these are the tasks that last year might have required like hundreds and hundreds of hours.
What are some examples?
Do we have any videos we can see that like show?
So, you know, I might get some flak when I come back because this is not published result. Hopefully this will come out soon. So, you know, I want to reserve the excitement for that. I'm kind of like building up the excitement a little bit. So hopefully this will come out soon. These are not simple tasks. These are like actually difficult tasks that just last year required like hundreds of hours of data collections.
You hear here on Lightcone first that there's some emergent properties that are going to come out of PI.
Can you give us a sense of like the flavor of the tasks?
It's really easy to fool yourself, and so we wanted to test across like a few different tasks of different flavor. A task that requires precision, a task that requires reasoning with multiple objects in the scene. It all seems to have this property. That's really nice. So it does seem like that's something that's kind of a more general property that emerged rather than we just, you know, got lucky and suddenly the model started working on one particular task.
Could you help us understand where we are now in terms of like what's working and how well it's working? Like we're not quite at the ChatGPT moment yet. Like where are we? And I think you brought some videos that you were going to show us to like help everybody visualize what the current state-of-the-art actually looks like.
I think where we are is, I think if you have a task where it's okay for the robot to make a mistake, and it's possible for you to set up a mixed autonomy system where you have a person that takes over when the robot makes a mistake and provides corrections, it is possible to get to a level of performance where it starts to make sense to think about scaling robot deployment. And the example that I specifically want to highlight here is this blog post that we did with Weave and Ultra. And, you know, it's great that these are both YC companies. I want to provide a little bit of context here first. The context is that PI is a primarily research organization. We want to focus on building the best model, but we also want to not be tunnel vision. We want to make sure that the model that we build is actually going to be useful and actually perform tasks that people in society care about. And one of the really good ways for us to do so is to partner really closely with companies that want to get robots out there today. And the way that these relationships work is that we treat each other like we're on the same team, very free flow of information. And we designed a system that tries to get the best possible performance for the tasks that these companies care about. So let me talk about Weave. What you're seeing in this video is a system that we built together folding really diverse items of laundry in a real laundromat. In the mission, you can see, you know, people walking outside. And why this task is difficult is because there's just infinite possibility of observation space. Like, you know, clothing are deformable, and no two items of clothing here are the same. And these are also unseen. You know, these are not like clothing items that are seen in the training data.
Yeah, I love this team. They are some of the most cracked people out of Apple I've ever met. Gary was the partner for Weave. Maybe I want to like explain like what Weave is and what their like what their like company is.
Yeah, I mean, they're actually, you know, shipping their first robots into the home. We sort of talked about it as, you know, being able to do household tasks like this. And I think they were very inspired by Physical Intelligence's first demos with laundry folding. So it's actually a total trip to hear about it. You know, a bit a year ago we were talking about them doing it, and then now to see them do it working hand-in-hand with you is really awesome. I think this is a great example of like, you know, you need the model smarts, you need the data collection, and then the hardware and the sort of system integration all working together is just hard to nail. So.
Yeah, and to get back to your question about why robotics is hard, it's really it is a really hard system problem. Like you need everything to work well and work well together to get this result. And like Weave is such an incredible team for us to work with to get this result. And it actually didn't even take us that long to get this result. It was roughly, well, we set a goal and maybe it was like two weeks afterwards where we got a model that was good enough at performing this task.
It still like blows my mind to see a robot actually folding laundry because I remember until basically until ChatGPT I didn't know if this would exist even in my entire lifetime because like folding laundry, I mean, it's always been like the Turing test for robotics because there's no way to like deterministically program a system the way that you did like pre-AI to do this because the space is like so infinite and like we've shown that it's possible for us to do like basically everyone can do this like robots will be able to do everything it's only a matter of like improving it from here.
There was a funny story where when we first published Pi Zero, people thought of us as the laundry company because the demo was just focused on laundry. And actually picking home tasks, especially tasks that have to do with deformable objects, it's a very intentional choice on our end. We're not just after the home. We really want to make it broadly applicable. But picking home tasks for us to start with has a few benefits. Like one is relatable. You know, you can see the laundry folding demo and you can kind of like grok how this is going to be useful and you can get a sense of why it's hard. And the second is that it's really easy to set up to test generalization.
You can talk about Ultra, which is your company, Jared, a demo of it.
Yeah, this is Ultra. The thing that I love about this video is you see, you know, it's bright outside and you see this is 4x speed and it's 100 minutes. If I scroll to the end, the sun has set.
Oh, wow.
That was one of the big problems in robotics where it would be so sensitive to the environment and lighting and mess up the vision system, the semantics and part of it.
Yeah. And the interesting thing here is that it is possible to get to the level of autonomy that the robot is just performing the task. This is autonomy at scale. Like this is ready to be scaled. Karol, because this task is less familiar than laundry folding, do you want to explain what the robot is doing here and what Ultra is like doing as a company?
Ultra is a company that wants to make it really easy to adapt robots to, you know, new tasks. And right now they're focusing on logistics space, which is really important because, you know, there's lots of labor shortage in logistics. And the task that we focus on together here is, you know, if you order an item from Amazon, you sometimes get this soft pouch that item gets shipped from. And the task here is you have a tray of these items here and the robot is supposed to pick one of them at a time and place it inside this pouch. The machine would then close it and then pick up the pouch and put it on the left here to be ready for shipping. Now, this task is hard because there are many different types of objects that can be in this tray. And the opening here is actually very narrow. So you see this interesting example of the robot kind of nudging the item to go into the pouch. And that's really hard. Like that requires very good understanding of the scene and like very precise motion to nudge the object into the pouch. The other thing that's hard about this task is the level of autonomy that's required. Like this is running for an entire day. There is still human intervention, I want to say, in this like full day operation. But the level of intervention is actually quite minimal.
This is not just like some like demo station, right? This is actually recorded in an actual e-commerce warehouse where they're actually shipping real products to real customers. This isn't just like a lab.
This is packaging real customer, real order for customer to be shipped out in a real warehouse. So this is real operations.
So I think this is really cool because I think when people think about robots, they tend to think of the consumer use cases like Weave because that's, you know, what we're familiar with in our daily life. What I find really interesting is that there's like a million applications like this Ultra thing that you wouldn't think of as obviously like, oh, who packs the like soft pouch of things that you get from like Amazon? Well, there's some person like who does that and this is like a job that we can now build a robot to do. The interesting thing about the approach is that you're converting it from a very difficult engineering problem into an operation problem of how do I identify the use case and how do I collect the right data, which is in some sense more scalable because you can build the system that allows you to collect data for many different tasks. So, you know, it's now a problem of how do I scale data collection rather than, you know, for every new task, how do I design a really difficult engineering system to solve it.
YC Startup School is back. We're hand-selecting the most promising builders in the world and flying them out to San Francisco for July 25th and 26th to discuss the cutting edge of tech. Apply now for a spot. Okay, back to the video. I think one thing that the audience may not know is that you have a very unique technical insight that in the past robotics folks would have kind of gasped and be shocked because robots need to run in real time. A lot of times all of the compute runs on device, but you guys have done something very different. Can you tell us more about that so that this works in real time with large models and really well?
So the context here is that, you know, we talked to many companies that would like to deploy robots and one of the first questions we get is what compute unit should we get on the robot? You know, it's expensive, it's going to increase the BOM cost and they're worried that it's going to go out of fashion very quickly because the model changes, the model gets bigger. How do I make sure that the hardware that I'm going to commit to today is going to be viable for, you know, a couple of years? It's a very difficult question. People are often really surprised when I tell them that almost all of the robot evaluation that we run at PI today, including the really complicated demos that we have shown, making coffee, folding laundry, mobile robots navigating around, the model is actually hosted in the cloud. And, you know, this is not like a cloud isn't a server in the office. It's a real cloud. The model is hosted in a data center somewhere and within this high-frequency control loop that is controlling the robot, the robot is actually querying an API endpoint that hosts the model, sending it images and language commands and getting back actions that then execute it directly on the robot. And this is surprising because of precisely the reason that you mentioned, you know, how do you actually make it work? This is why it's really important for PI to couple system, hardware, and model development.
System, hardware, and model development.
and research like very tightly together because like it allows us to solve for this problem. So for example, one of the insights that we have here is that you can actually bury the inference time within the robot control loop because you know if I'm a robot, I have enough action for me to execute for the next 100 milliseconds. Like there's no reason for me to wait until I finish executing that action to ask my model for a different action. You know, I can do it as fast as inference essentially. Um, and so you know maybe when I only have 50 milliseconds of action left I can ask for the next sets of action and when the current 50 milliseconds is over like I have something that's ready for me to continue with you know my next 100 milliseconds. Um so that's one of the insights. The other uh kind of algorithmic improvement um we refer to them as real-time chunking design inference in such a way that you know there's going to be a delay in how long it takes to query the model on the cloud. Basically like the problem here if I get uh a little bit more technical is an action chunk is a sequence of action that I can execute on the robot. So you know it's not just one action. And if I have an action chunk that I can execute for 100 milliseconds and 50 milliseconds in I want to predict another action chunk and I'm going to transition to that new action chunk after my current 50 milliseconds is over. How do I make sure the two are consistent? Like you know how do I make sure that if I'm moving this way the next action chunk is going to continue me to allow me to continue to be smoothly moving this way.
You can precompute. Yeah, you can precompute and like that's one of the algorithmic improvements that we've made to make inference using model hosted in the cloud possible.
I studied computer engineering so I'm not really an algorithms person but when it comes to systems like that like pipelining like get me all over that. That sounds great. That's so interesting.
I mean this simplifies it's kind of it's a brilliant choice because it simplifies so much of the system for the robots. You don't need all these clunky I don't know people have two operating systems at sometimes for for robots embedded arts and then the regular one and all these complex giant compute and power and this is what the initial versions of Waymo used to run basically a server on the trunk and you can't afford to do that with general day robotics which is brilliant that you figure out how to do it.
Yeah, you don't have to. I mean you can do things some of it there obviously has to be some compute there but a lot of the compute can happen elsewhere and then is there there must be a video like this this thing that we're looking at in the top left like how much of that is sort of like video feed back how much of it is like local processed I mean is there any compute locally on this robot or is it just like a dumb like video camera that streams data to the cloud
for this I am not 100% sure but I am inclined to believe that it's just a dumb computer. Like for this specific video, um I don't remember, but I'm just 100% confident that we can make this work with a dumb computer and the robot. And the one other interesting thing about our collaboration with Weven Ultra is one, I've never seen the robot in person.
Um two is I have very little idea about how the robot actually works.
