on the path to spatial intelligence, generating pixels that are truly spatially contextualized and grounded. That is the very hard step that Atlas has taken.
We know LLMs are built on next token prediction. We've seen video models as being built on next frame prediction. Atlas is really new view prediction.
This is the real place where AI can actually unlock a ton of value for people and their process. We're seeing like 50, 100x reduction. There was a famous shot in the first Matrix movie where Neo is like falling down. Exactly. They had hundreds of cameras viewing that angle on a green screen. On Atlas, we can do this with just three cameras. No studio capture, no green screen, no expensive calibration.
No one has ever seen this result.
When you set out to do this, did you know it was going to work?
I was pretty sure. Each time we made the model bigger and each time we trained it for longer, it got significantly better.
Does that mean we're going to get 4D video? We like go walk around.
So big day yesterday. You launched a new Frontier model which got an amazing reception which is still coming in. I think maybe a good way to structure this conversation, let's just talk about exactly what that was and then we'll go back to history and work our way back up. So maybe Justin, do you want to talk about what was launched yesterday, why it's significant?
Yeah. So Atlas is our new next generation world model. It has three basic things. It can generate, reconstruct, and simulate the world. So within that, there's a couple different major capabilities. It has really good camera condition generation. So you can input an image together with a camera trajectory and steer the model and have it generate, you know, video frames along any perspective you want. It's really good at sparse 3D reconstruction. You can input one or multiple up to 100 frames that are views of the real world and use those to reconstruct the real world. And that reconstruction can take the case either of a novel video flying through the space or an explicit 3D reconstruction of the space. Then finally it can be used for simulation. And for this we show off, you know, these awesome bullet time videos which got a lot of attention online and then also robotic simulation.
What's a bullet time video?
A bullet time video, this comes from the Matrix. You know, there was a famous shot in the first Matrix movie where Neo is like falling down. Exactly. So, and then remember in that famous shot he's like falling down. It's in slow motion and the camera flies all the way around. So that's the way that they did that shot is they had a ring of like hundreds of cameras. So then like he fell over in the studio. They had hundreds of cameras viewing that angle on a green screen and then they used those hundreds and hundreds of cameras to make that famous shot in the Matrix. But now with Atlas, we can do this with just three cameras. So like no studio capture, no green screen, no expensive calibration. We can literally stick like three cameras on three iPhones on tripods. Use these to take sort of a video of something happening, like someone shooting a basket, someone dropping a strawberry into a bowl of milk and then from those three like iPhone videos, we can then reframe the shot and imagine like freeze time, have the camera fly in like as the milk is splashing up and get these amazing frozen time views. And we can do this with just a couple cameras.
Can you maybe what is the simplest description of what Atlas does? Like what goes in and what comes out?
Yeah. So one of the really core principles of Atlas, like the most fundamental thing is it does new view prediction. And this is a really fundamental primitive that we think is super exciting, a super new primitive for base models that no one's ever done before. Right? So we know LLMs are built on next token prediction. We've seen video models as being built on next frame prediction. Atlas is really new view prediction. Right? That given some number of views of a scene or a description of a scene, those go into what we call a spatial context that describes implicitly what is the world that we want to talk about, then you can point a virtual camera at an arbitrary point in space and time and Atlas will understand what that world is supposed to look like from that position in space and time.
Ben, you know, with a bajillion video models out there all claiming to be world models and all claiming to have novel views, can you maybe tease apart kind of more concretely how this is different from like the myriad models that have come before.
Mhm. Yeah, I think what Justin was saying about the spatial context aspect is super important here. So there's many video models, a lot of video models actually got their claim to fame from their single image input or their start to last frame interpolation. Now, we're starting to see models that can do this kind of omni reference thing with, you know, 20, 30, 50 images. But what's key with Atlas is that it actually has a kind of like spatially grounded meaning to every frame you put into it. So, it's not just an image that the model's going to interpret whatever way it wants or you can kind of try to argue with it in the text prompting and get it to do something specific. With Atlas, every image actually has an associated three-dimensional camera pose. And that means that you can perform this task of reconstruction with an extremely high degree of precision. Right? So, if we had four views of this room, one at each corner, you can put those into the model and then get an exact replication of everything you see in this room. And it's not going to guess what's in the other corner, like a relationship between things. It's just going to reproduce exactly what you give it. And you can also do that in a kind of creative or imaginative sense, too. If you take two photos from different, you know, AI generations or real world locations, you can actually position and stage those to build these kind of intentionally directed flythroughs that are really governed by exactly the precise place that you put the content you want and where the camera's going to look and travel. Which is very different, I think, than the kind of like more slot machine effect you get of having to retry generations over and over with just that kind of higher level of text control you get with video models.
Is this just kind of an obvious, you know, scaled up version of a traditional video model or is it a new architecture?
I think it's a pretty new thing for a couple different reasons. One that like we talk about is it does both generation and reconstruction jointly in the same model. Like Ben was saying, this thing can take a couple views of this room and then reconstruct everything in this room exactly as you see it. And historically, reconstruction has been its own subfield in computer vision with its own specialized task, its own specialized models. And generation is what all the text to video models are really good at, like what all the big diffusion models we've seen the last couple years. And those are great for creative applications. I want to imagine something that's never been there before. But now with Atlas, for the first time, we're putting these two different parts of visual intelligence together in one model. So, it can do both 3D reconstruction and generation together in one architecture. So, to do that, we had to make a couple changes. One is we had to make it multimodal from the start. So, this thing natively works on text. It works on images. It works on videos. It also works on camera poses as a native input to the model, which I don't think anyone's ever done at the pre-training phase before. And it uses 3D as a native modality that it works on. So, this thing from the beginning was designed to be natively multimodal in a way that no one else I think.
Sorry, I just I don't know the space super well. By 3D, is this like depth or models or like what does that mean?
Yeah. So the formulation we used so far is depth maps. Right. So right now you can have a when you have a frame that has a virtual camera telling its position in 3D space. That camera position and camera parameters are a native input to the model. And then what attached to that camera position, you can have both like RGB telling you what does that position in space look like and you can have a depth map that tells you what is the spatial structure of that position in 3D space. So then you know text, image, video, 3D cameras are these modalities that this thing all does jointly in a multimodal way.
I want to add something because I think what Justin just said is actually so important and also what Ben said that it's underappreciated. It's the first time we have a unification of pixel generation and pixel reconstruction in the world of computer vision. This field has been around for more than half a century. Sitting here having been in this field for decades, I cannot tell you how many PhD thesis have been written on the problem of reconstruction or novel view synthesis and also our field traditionally have multiple tracks. You go to a computer vision conference, you have the pixel generation track, you have some recognition track and you have 3D reconstruction track. This is an elegant model that combines or unifies the problem of reconstruction and generation by anchoring on viewpoints and the viewpoint estimation and that's just incredibly powerful.
Can you maybe well can we take a step back and then maybe you just fill something out. So when you started the company I remember you saying you know you want to tackle spatial intelligence right and you know now we have this new model and so it feel I mean like as a lay person feels very general to me you've got next token prediction and this is next new view prediction.
New view prediction right and so like you can get one view set of views and you get a new view.
Can maybe you pencil out like how this is a significant step to this general problem of spatial intelligence maybe by like starting to describe what spatial intelligence is.
Well spatial intelligence eventually must enable us to both generate what the space is, reason within it, and being able to edit and interact within it. Now we can argue is it 3D or 4D ultimately it's 4D with the time dimension but even just 3D these are the fundamental tasks that one has to do or spatial intelligence has to enable and then we talk about with that you can render, you can simulate, and you can plan actions but to do that a fundamental problem to solve is to understand the geometry and structure and the physics of the space.
Yeah. And I do believe Atlas is a significant step forward because now with every single frame you have a you can generate and estimate an important piece of information which is the viewpoint, the camera pose. And that is the most critical information one needs about the geometry of the space. And that can lead to all the emergent behaviors we see in the downstream of the model which we showed in the blog. So, on the path to spatial intelligence, generating pixels is definitely an early step which we have seen with what you call gazillions of models. But generating pixels that are truly spatially contextualized and grounded is absolutely another major step and that is the very hard step that Atlas has taken.
We definitely have you know we can just keep going here right like there is the fourth dimension of time which will bring in dynamics and there is more higher fidelity simulation and delineation of the space. So this is part of the road map of spatial intelligence.
Great. Yeah. I mean and I definitely want to like dig into like where this is going, but first maybe let's talk about getting here. How long has World Labs been in existence?
Two and a half. Yeah. And so you've actually released models before. So why didn't you just jump right to Atlas?
Great question. It's so magic, right? Yeah. Justin's team needs a lot of chips. Yeah, you need a lot of GPUs to actually scale this thing up. So one of the Well, last year we released our Marble world model and that was the first kind of big major world model that we put out. That powers our current Marble product and Marble is really cool. Marble can take images, it can take videos, it can take text prompts and use these to generate 3D worlds. But one of the biggest differences between Marble and Atlas is exactly what is that output modality. So Marble was really focused on Gaussian splats as an output representation. So whatever you're inputting it's going to output a 3D world represented as a Gaussian splat and Gaussian splats are really useful right they're really nice they're easy to render they can render efficiently on mobile devices on VR devices they can interoperate with other with game engines with simulation engines so there's a lot of nice things about Gaussian splats but you know I think that was kind of a bottleneck in the previous Marble model so what we did with Atlas is redesign the thing a little bit and we realized that we need to bifurcate these modalities earlier and actually have these modalities working in a more unified way in the model. So now with Atlas the fundamental primitive is not like could generate a Gaussian splat world. The fundamental primitive is as we said new view prediction and that can generate RGB frames that can generate 3D and we can use those to generate beautiful Gaussian splat worlds when you need them. But we don't need to bottleneck our outputs through the Gaussian splats when we don't need to. And that actually took a lot of you know blood, sweat and tears to understand like what are all the pros and cons of these different representations.
Yeah, that's one part of it. The other part is you got to like climb the scaling ladder right you got to like work your way up and like do smaller experiments, do smaller models like to build your conviction on what's going to work and what's going to scale. And there's you know if you could instantly know the right thing that's going to scale you know you should just do that. But when we started the company the world was a very different place like there was no scaling law of spatial intelligence right so like when we started the company like the world was in a very different place the tech was in a very different place we had a lot of ambitions about where we wanted it to go but it took a couple iterations for us to hit upon this formulation that we thought is actually like this is the one that can scale out.
Ben, you know being the creator of NeRF and doing a lot of 3D and reconstruction so it's not so obvious to me that like if you have multiple views that you actually end up with a 3D thing, but like you've kind of like made a career of ending up with a 3D thing. So maybe talk a little bit about like kind of that step.
Yeah. Yeah. I mean, as you said, I've spent many many years of my career, the vast majority of my career actually working on producing 3D things from images. And this actually something we talked a lot early on in the company even of like is this going to be the approach that produces 3D, right? Are we going to synthesize multiple views and then build 3D out of that? Are we going to try to go direct to 3D? Like there's been a lot of uncertainty in the field around like which of those approaches kind of will win out or will kind of reap the best advantages earlier on. But I did have a lot of conviction just from seeing the kind of power of what I would almost call the brute force scaling in a very very very small baby scale not like real model scaling but the scaling of dense reconstruction that we had seen happening over the past three years before. So basically we put
Why is dense reconstruction dense?
Yeah, dense because I know we're going to talk about sparse and I want to make sure that people understand what is dense and what is sparse.
Yeah, so I think this has actually even on the kind of like business and commercial side I think been one of the challenges of productizing 3D reconstruction technology like at a fundamental level, right? People kind of don't in a casual sense like you think I took three photos of this object or I took six photos of this room like I look at the photos I can understand in my mind like how those piece together I can kind of fill in the gaps and get it but there's just never been really any kind of reconciliation between those like really data-driven priors and then the kind of brute force dense reconstruction which it actually is much more akin to almost like scientific or medical imaging what we did in dense reconstruction, right? You basically have to say every single thing I want to appear in this reconstruction, I need at least three or four views of it. And if you think about that, like even just in this room, right? There's like under the microphone, under the table, between every different crack and crevice and the plant leaves, right? To actually truly get a picture that covers every one of those spots, it's this like very tedious and exhaustive effort to walk around the room. I think you've all seen me running around various places like capturing them. It takes, you know, for someone who's well trained per se, like it can take minutes. But if you hand a casual consumer or even some like kind of professional trying to do this for the first time, an average like cell phone camera or capture device, like it's going to take them probably an hour. I've seen someone for their first time trying to scan a multi-room environment spend like two hours walking through it and get enough coverage. And that's just this like very very exhaustive and tedious loop. And so yeah, when we say dense, we really mean dense. It's like this room I want like so many photos, right? I want like 100, 200, 300 photos of this room to capture it. And what we're trying to do is bring that down to like three, right? We're saying like 50, 100x reduction. And then that's at that scale where it just completely like flips that calculus on its head of like what type of captures you reconstruct. You can go back to existing imagery you have. You can go to stuff you find on the internet and even build scenes out of that. You can go to casual videos and like kind of unearth a lot of footage in the past we would never have treated as reconstructible and go back and like bring it to life as 3D potentially. This is something we've been playing around with a lot with Atlas, right? Like taking old clips like I've taken a bunch of my own old captures that never worked before and then put them through the system and then kind of seen a reconstruction for the first time or taken my old captures and thrown away 95% of the photos I took. And you know imagine angles that I never would have gotten from a traditional kind of like NeRF or splat type reconstruction.
One thing that's underappreciated on the website of the demos is the Stanford demo where Ben showed anywhere between 3 to 25 images. You can reconstruct that entire Stanford quad. But the thing is we had to show it from an aerial view. But every single input image is Ben standing on the ground taking a picture from the ground. So everything you see are generated but according to the laws of reconstruction and this is really magical.
And this is where like generation and reconstruction need to interplay in a really fundamental way to solve this problem because under the classic kind of reconstruction stuff that Ben was talking about like the reason you need so many views is because I need like multiple images and I need to triangulate this point in 3D space and see it from multiple viewpoints. So that means like that's required in the traditional version. And on the flip side, anything that wasn't captured in these views, like any pixel that was not visible in one of the input views will be a hole in the 3D reconstruction because fundamentally like if a thing wasn't visible in the input views, you know, you need to imagine it to fill in the gaps. And that's fundamentally a generative process. So, even in this room, even if we set Ben loose with a DSLR and like let him like capture like hundreds of views of this room, even the world expert on doing these dense captures is still going to miss some spots. Like he's not going to get like underneath all of the microphones or underneath all the tables or like in between all the chair lights, you're always going to miss something no matter how many views you get. So that's where you need generation as another mechanism in the model because you're never going to get everything. So you need to have some generative capacity for the model to imagine oh based on what I'm seeing then like first triangulate what I can see but then fill in the gaps of the stuff that inevitably was not captured.
Yeah. And there's something like super cool about this that LLMs have really understood this for a long time, right? There was kind of almost these like context wars. Like the first couple years, right? It was like, oh, we got to 128 to 256 to like 512, we got a million, right? And everyone kind of understands now at a pretty tangible level the value of, you know, you crank your context length to high when you're using your coding model. It's a hard problem. Like everyone has a feel for that, but like no one has pushed that at all on the image and video model side in the same kind of like principled way. Like no one's out there trying to like put an hour-long video through and do a needle in a haystack retrieval of like a frame at the 37 minute mark. Whereas with reconstruction generation, you actually have the same exact thing of like reconstruction is just like generation with a really long context and you put a lot of stuff in it, right? Like that's the way to actually build this continuum where you kind of bridge between those two things. And like Atlas like being able to like this is something we can never do with Marble. Marble had this kind of fundamental blocker of like you couldn't really jam more than honestly like a couple images in. Atlas, I can go and I can actually take like a 64 image capture and do like a fly through of an entire house and everything is grounded by being, you know, seen or like almost seen or like slightly extrapolated from what's not there. But you're just getting these, you know, I'm taking captures I did with 2,000 images of a multi-room house and taking it down to like 30, 40 inputs and the fly through looks like basically the same. And this is just like totally inconceivable before. And it's all enabled by building this gracefully scaling kind of context window that you can dump stuff into.
And so the way to think about it is like the sparseness are the pictures that you physically took and then Atlas as a model creates the rest of the views and then you use classic reconstruction techniques. Is that roughly the way to think about it or?
In some sense? Yeah. Yeah. I mean that's the beauty of Atlas. It's like you can take however many inputs you have down to like a single view and then you can almost use Atlas as this rendering engine to produce anything else you want. Right. You can navigate a virtual camera.
Yeah, exactly. You can just say like, okay, I have a picture here. I want the picture there, there, there. You can make a couple of those. Then you can stage a dense fly through. You can do this in sequence because it's an autoregressive model. It's up to you, right? To kind of pick and choose what you add interactively into the context as you generate.
I mean, the thing that just blows my mind is listen, I just have a very simple mental model. I have four pictures and then I've got to like have the model extrapolate between them and then it has to fit when you're in control. Like it's got to be 3D because like and I always think of these diffusion models as like being visually great but not accurate. And so like and I don't even know if there's a question here but like how is it like the room fits like how is it that it's 3D consistent? Is it just lots of data or
Yeah. I mean it's partially a belief in the scaling hypothesis, right? Like, you know, did you—by the way, but I have to ask, when you set out to do this, did you know it was going to work?
That's so—so were you sure? I think three of us have total conviction about the scaling law, that I think we do. I do think the exact architecture choices and data mixtures is where the devil's in the details. I, you know, have watched Justin and his team going from 'we really don't know how long this is going to take' to 'oh, maybe side of life' to 'wow, this is going to work.' So, no one has done it, but I think the hypothesis—one hypothesis is scaling law hypothesis, the other one is next viewpoint prediction. We had conviction of these two things from early on.
So I think I was very convicted that it was going to work. I was not sure it was going to work this well at this fast, right? Like I thought there's a chance that we do this, maybe it's not clear that like the first cycle of pre-training a new model with a new architecture and a new paradigm, like the first cycle of that working is insane. So I thought there was a chance in which we might have had to do a couple more turns of that pre-training cycle before we got to the level of quality we wanted.
Is it—are we kind of like at the end of like the scaling for this architectural approach? We need another breakthrough or is there—
No, no, we're at the beginning.
Really? Without changing the architecture?
I think we're basically at the beginning. I think we're basically at the beginning and we're basically limited by compute at this point, right? Like data is very important as Fei likes to point out, but like everything has a bottleneck and I think the main bottleneck on continuing to scale this thing is actually training compute. Right? Like during development we trained a sequence of models—we read about this in the blog post a little bit—but we trained a couple models, the like the first couple rungs of the scaling ladder, and each time we made the model bigger and each time we trained it for longer, each time we put it on more chips, like it got significantly better.
And the model size that we—like the model that we showed in the blog post is obviously the biggest and best one that we trained. But the thing that was limiting it was not the scale or the data or anything like that. It was literally like we had a deadline of when we wanted to release this thing and therefore we backed up what we could afford to train by that deadline. But here is a little bit of an insider story, right? Like Justin and team are training from the smaller and slightly bigger, you know, having these roadmaps. And then there was one day in summer, early summer, that it's not even the current Atlas model size, it's a smaller model. And then Ben, Justin, Ben feed it into, you know, the viewpoint generation. And remember that famous table, the garden table for NeRF paper and many papers that—