We are building the next frontier of AI, which is what we call spatial intelligence.
As Synex, we are developing what we call a real-to-sim-to-real pipeline. We can replace all the data, all the evaluation we need in the real environment by using the data that can be generated at a scalable way in our digital world.
Think about human intelligence. We do a lot of simulation in our head. Why? There's a very important role simulation plays that real world data doesn't play, which is counterfactual reasoning.
What we are building is a consistent world, consistent both over space, over time, over different viewpoints, and over different types of interactions. My north star is I want the robot to work.
The world we live in can be a multiverse that we create technology to allow people, builders, developers to act within different spaces.
Do you believe we'll ever be able to build robots that have the power efficiency of a human being? How far away are we from this? Is this like 5 years or is this like never?
All right. Well, it's great to have you both here. So, Fei-Fei, for the listeners that may not have the background, maybe you can give an overview of what World Labs does.
Yeah. Well, World Labs is a two-year-old startup. I think we should just recognize it's a frontier model lab. We are building the next frontier of AI, which is what we call spatial intelligence. And spatial intelligence is about creating AI that has the ability to generate, understand, reason with, and interact with spaces, whether it's physical or virtual. And of course, a means to an end towards spatial intelligence is building large world models. And that's what World Labs is mostly focused on.
Yeah. So you've been saying this since the very beginning, which is, you know, the machine's ability to perceive and reason about spaces and act on spaces. But I always had the assumption that the acting on spaces was some like long-term future thing. But now you're acquiring a robotics company, and so maybe talk a little bit about the timeliness of this and the intentions.
Yeah. So first of all, it doesn't just take robotics to act within spaces or to interact, right? I mean, look at the creative field, whether it's VFX or gaming or design. Many use cases, you can create and act within virtual spaces. World Labs' thesis has always been that the world we live in can be a multiverse that we create technology to allow people, builders, developers to act within different spaces.
Having said that, the ability to act within the physical space is one of the most exciting and most profoundly important capabilities of the future AI world. So robotics is very much that. So World Labs has always believed that robotics is an important application as well as use case of spatial intelligence and world modeling. So by joining forces with, inviting Synex's team to World Labs is part of our long-term vision and mission. We've always committed to that.
Amazing. So, Yunzhu, you're the co-founder of Synex. So, maybe provide everyone with a quick overview of your background and what Synex does.
Yeah. So, I'm Yunzhu. So, I'm currently co-founder of Synex and also assistant professor at Columbia University.
So, my research started from my PhD at MIT and then postdoc with Fei-Fei really at Stanford University.
It is.
Throughout my career, my goal has been very simple: trying to help the robots better perceive and interact with the physical world. So I'm a very practical person. I want my robot to work in the real physical environments.
So for Synex, the unique opportunity we see is that there has been a lot of bottlenecks that we see faced by the developments of general purpose robots, especially around training and also around evaluations. So as we are developing what we call a real-to-sim-to-real pipeline, we want to map the real environments into the digital world that has the best alignments with the real environments. By alignments we mean that whatever happens in the digital world is also going to happen in the real environments.
Such that we can replace all the data, all the evaluation we need in the real environment by using the data that can be generated at a scalable way in our digital world. So that is how everything started in Synex. We put together a very, very strong and best teams around robotics, robot learning, and also simulation and rendering, trying to build this real-to-sim-to-real stack to solve some of the key bottlenecks.
It's amazing that you two work together.
Yeah. And there is a funny story here because you would think because we worked together, he was my amazing postdoc, we've been talking about this and World Labs integration for a long time. It's actually not true. They came into World Labs as a customer.
When we released the first version of our generative model called Marble last winter, around November, December,
No kidding. As a customer.
Yes. And I didn't even know what it was. And then I realized this is Vindra's company. I called Vindra. I'm like, 'Wow, this is your company.' And then we realized there's so much synergy.
Maybe Fei-Fei could just quickly describe what Marble is.
Yeah, Marble is the code name for the base model that World Labs has been training and iterating on. The fundamental capability right now of Marble that is publicly released is to take a prompt. It can be an image, it can be a few images, or a text, and turn that into a geometrically consistent world that can be represented in 3D geometry, whether it's Gaussian splat or mesh. Really what Synex's team is doing is trying to solve this extremely difficult problem in robotics, which is the lack of data.
The lack of data in training, the lack of data in evaluation. This is very, very different from language models where data is abundant on the internet.
And we know that in order for robotics to work, we have to somehow unlock the power of scaling law. But where does that come from? This is something that is a profound problem that everybody's battling with in robotics.
It'd actually be great to talk about this. You have put together a very, very talented team. So like, to what extent is there overlap, to what extent is this an extension? Maybe talk a little bit about that.
It's actually the TL;DR is very complimentary and with a shared mission. So I'm one of the three technical co-founders. The other two are Changxi, another Columbia professor who has been a world-class technologist in simulation. And Changxi has his background in also VFX. He worked at Weta, he worked at Tencent, he's been an entrepreneur. Then there's Sunan, who is a phenomenal engineering leader who was also in a startup that was acquired by Amazon many years ago. So he worked in many different tech stacks in the computer vision field in Amazon. So when we started talking more seriously, I recognized that a couple of things that Synex has from a talent point of view is extremely complimentary to World Labs. One is obviously incredible thought leadership and just technical prowess in robotics, right? So from really from hardware, full stack robotics, and even when he was my postdoc at Stanford, at that time you already had your faculty offer, so you were there only for one year. I wanted you for more than one year, but he had to go become a, have the real job. So he was a full stack researcher in robotics from modeling to hardware. And of course, Vindra and his students at Synex was that pool of talent World Labs hasn't had yet. Then on the Changxi side is just incredible simulation capability, right? He's such a senior researcher and technologist in simulation, and what World Labs is doing is very much interfacing the world of simulation. So I think what they don't have obviously is on the generative model side as well as the computer vision 3D reconstruction side, we're also very strong at World Labs. So that's a technology that Synex needs. So together these two sides come together and make it much more complete.
Fei-Fei's motivation in this, as like this is an extension and a compliment to get into robotics. You know, having been in your situation, which is deciding when to sell a company, it would be great to hear from you on like how you think about joining World Labs and kind of the fit there and like why you made the decision to do it.
Yeah. So at the very beginning we were deciding, okay, we want to just keep going, but after chatting with Fei-Fei, after seeing all the synergies that are happening in the middle, it just makes perfect sense for the forces to join each other. So in any sense as Synex, what we have been doing is real-to-sim-to-real is to do this reconstruction of the environment. So we capture the appearance of the environment, geometry of the environments, and also the dynamics of the environment, meaning how the environment is going to change when you apply actions. So this tensor reconstruction right now is still a little bit on the heavier side, and what World Labs right now has been doing involves a lot of profound capabilities around sparse reconstruction and generations. So we see a lot of opportunities of leveraging like Marble and other like capabilities as World Labs in order to do very efficient reconstructions and modeling of the environments.
So can we expect a foundation model for robotics from World Labs?
World Labs is building a foundation model. As you know, Martin, we're building a base model, and as the technology has been evolving, some of the most exciting base models are omni models, right? They take multimodal input, they have multimodal outputs. And what is a foundation model for robotics? It's very likely going to involve actions. It's very likely going to involve the output of actions in addition to the state of the world, and we're definitely not ruling this out.
So for example, for the foundation models, it essentially needs to be a multimodal model. So it has to take into account frame, text, image, depth, and different kinds of modalities, and action is a very, very important part of that modality. So if you think about frame actions as inputs, that is essentially a forward simulator that is going to predict how the environment is going to change when you apply a specific action. When the action is output, this is essentially a policy model that is trying to predict given a specific goal, like what should be the action you take in the real environment to get you closer to that goal. So this kind of omni models actually can benefit a lot and actually provide huge amount of values for the robotics communities in trying to understand how to model the environments and at the same time how to act in the environments. And this can also act as a backbone for you to fine-tune into specific robotic applications to making sure it's really live up to the reliability and efficiency that expected by the clients.
You know, if Yunzhu, if you don't mind a kind of a lay investor question, I see a lot of robotics companies, and a very popular approach right now for the robotics companies that come in is like, we'll use a video model, you know, and like, you know, that's the predominant method. Where this is, you know, 3D and simulation, it's a very different approach. And so maybe you could contrast, you know, this popular approach of just using video only versus kind of what the ambition here is.
Yeah, so in order to create worlds where the robot can learn, the worlds, as I mentioned, is to capture the essential structure of the problem. And one of the very important necessary requirements for those worlds will be consistency. So that is where I actually see there's very, very strong synergies with Marble, because what we are building is a consistent world, consistent both over space, over time, over different viewpoints, and over different types of interactions. And Marble, the generated worlds from Marble, also provides an infrastructure, a component of that entire world that we believe is necessary for the robot to learn. Imagine if a robot pushes an object forwards, the object just magically disappears, which has been a problem of many of the existing video prediction models. This won't provide good enough signal for the robot to know like what is the right thing to do. But obviously right now there has been a lot of investigation on building better and better and stronger and stronger video models. So we actually see a way where some of the infrastructure we build can provide as an initial momentum and to going through this data flywheel of going from this more simulation-driven models into like a robot policy models which going to do the execution in the real environment, collecting new data. The data will come back in where the model doesn't necessarily have to be physics only or learning only, but somewhere in the middle which be able to capture the essential structure of the problem, but at the same time be able to scale and become better and better as you accumulate more data.
You know, I've worked now very closely with Fei-Fei for a while, and you've always had this north star which has driven this, and you know, you've articulated variously as kind of 3D and in a number of other ways. And I'm just wondering for you, is there also a similar philosophical north star, or you're more the pragmatic, like I am, like build the system, like do the thing.
My north star is to make robots work.
In the real environment. I'm a very practical person. I want the robot to work. One interesting thing that's actually coming from my collaborations with Fei-Fei during my postdoc, we are building this kind of benchmark. We actually send out surveys asking the general public what they want their robots to do for them.
Among the thousand tasks we collected, one-third of the tasks are about cleaning.
People just don't like to do those like dirty tasks.
And those are the scenarios we really want to making sure we have robotic solutions to deal with.
One thing I really like about Synex, Martin, especially continuing your question, there's a lot of robotics companies building models and all that. One thing I truly like about Synex is Yunzhu and his co-founders have such an incredibly pragmatic approach to robotics. They especially, they come from academia, right? Sunan doesn't, but Vindra and Changxi come from academia, but their first instinct is work with design partners and customers in real industry, whether it's labs at industry labs or warehouses or electronics assembly. That is such a refreshing, actually a refreshing way of approaching robotics, and that really made me very excited to work with them.
Maybe this is for you, but I'll just be, this is personal curiosity, which is it seems to me that for robotics you have to be pretty exact. I mean, not perfect, but pretty close. But for the creative use cases which World Labs has done a lot of, you kind of don't need to because, you know, I mean, you know, even sometimes like being wrong is stylistic or intentional or whatever. And so from a technical perspective, what is the challenge here for reconciling these two things, or do they never get reconciled, like there always be two points in the design space?
So they will be like reconciled in the long term, of course, and modeling of the environments doesn't have to be perfect. The model doesn't have to be perfect in robotics.
And by the way, is there, again, this is pure curiosity, but is there like a bit more formal way to say that, like what does that mean not to be perfect? It has to be pretty close.
So let me put it this way. For example, models over the developments of all different kinds of robotic applications has been a very important cornerstone. If you look at all the existing robotic applications like planes, drones, Roomba, or even for quadruped robots, bipedal robots, model has been the way for them to actually work and be able to transfer from simulation to the real environment.
But if you look at those locomotion robots like quadruped robots, bipedal robots, they can walk on snows, they can walk on bushes. But you don't need to have a simulator. They can simulate all the bushes and snows like very precisely. You need to have a simulation that captures the essential structure of the problem and do a whole different kind of randomizations inside the digital environment. So that is what we're aiming for. So basically with Synex and together with World Labs, we're trying to investigate like what is the level of fidelity we need to model the massive worlds besides the robots, such that we'll be able to transfer the robotic systems trained in the simulated environment, digital worlds, back into the real scenarios.
As an investor, I've heard other researchers say, like Sergey Levine say, simulation will always eventually deviate from the physical world and real world data collection is absolutely critical. And so maybe talk a little bit about like the viability of this approach where simulation is a cornerstone as opposed to some other approach.
So they don't contradict with each other. So if you think about the simulation, simulation is essentially trying to predict how the environment is going to change when you apply the actions, and this is essentially a model of the worlds. It doesn't necessarily have to be pure physics. It can be a combination between both physics and also learning. We are collecting real world data. We will be using those real world data. It's just at different stages of this data flywheel. Maybe at the very beginning we have stronger emphasis on, we have more physics to making sure we have the right consistency and right structure for us to learn the worlds, for us to train the robot policies. But as we accumulate more and more data, both through data collection and also through the collaboration with our clients, we'll have the data that will be moving more towards more learning-based like modeling of the environments. So this kind of transition and also this kind of data flow is really an enabling factor of both getting the best of both physics and geometry and consistency as well as all the power and magic from the data and compute.
I want to add to this and be slightly philosophical here, is there isn't a binary choice between simulation or no simulation. All this comes in together to make robotics work. Think about human intelligence. We do a lot of simulation in our head. You know why? There's a very important role simulation plays that real world data doesn't play, which is counterfactual reasoning, is that you play out events that haven't happened or cannot happen or you don't have enough data to make it happen in real world. And while you play it out, you learn how to act in it. Humans do this all the time. We probably don't, you know, we just, I know you were at the World Cup.
Congratulations to Spain winning. I'm sure in the planning of every game there is simulation, whether it's digital or on the whiteboard or whatever. That simulation, the role simulation plays is counterfactual reasoning. And that's really important in robotics because we just do not have, cannot possibly have enough real world data for that. Here's a real life example, the industry of self-driving cars. Waymo has officially said they use billions of hours of simulation, and actually Waymo is more simulation-heavy than just real world data heavy. So these are real examples, and as you know, Martin, autonomous cars are the simplest kind of robots.
So clearly simulation plays a huge role in robotic learning.
I also want to add to that. So like there are, if you put things more specific, simulation can provide two levels of benefits. The first one is reliability and the second one is efficiency. So for reliability, if you're thinking about a robotic system working reliably in the real environments, you need data to provide systematic coverage of all the state space and the variations that robots might encounter. That's how you can learn how that is robust. So with simulation, you can do systematic randomizations and control and the variations of lighting, frictions, geometries, object types, and also all different kinds of physical parameters to making sure you have sufficient coverage of the state space. So this is what can give the robotic systems reliability. And second is about efficiency. So right now many people are doing teleoperation, and if you look at many of the teleoperation devices, imagining all the actual skeletons you are using, you're actually collecting the data at a speed that is actually slower than human actually doing the task.
But for many of our clients, human speed to them is not good enough. They want faster than human speeds.
So for the robot to like move faster, it's not as simple as just drive the robot faster because the gravity doesn't change. But in simulation, you can do systematic speed up of the robot's behaviors to train the robot such that it considers all the dynamics changes of the environments. So this is what can give like our clients for them efficiency. So both for the reliability and efficiency, though there are some kind of like very unique like values where simulation can provide.
You've talked about the technology and the platform, what it does. Maybe talk about the specific use cases people use it for.
There are essentially like two specific use cases, especially around both training and also around evaluations.
Starting from the evaluations.
So evaluation is something like people often overlooked in the robotics, but if you are tuning like robotic models, you have to know how well it works, and that is the only source of information for you to iterate.
Yeah. By the way, a lot of, every AI person really understands what evals are and uses it all the time. Non-AI people, it often means something a little different. So maybe it's even worth just describing specifically what you mean by evaluation.
