Hi everyone and happy Render Con. I'm really sorry that I can't be there in person with you all. Last year was fantastic and I'm sure that you're all having a great time. It is the most exciting time in history. And today I'm going to be talking about how to build a holodeck. Where are we with this technology and where's it coming? The theme of the conference is how do we take the amazing advances and bring them all together. So I thought I'd give a bit of insight into where we are, where we're going, and when we're going to reach that 'oh my god' point because this is what it feels like every day. Reality is like, oh, funny robot doing monkey tricks, and you're like, there's monkeys everywhere typing on their typewriters creating Shakespeare. I think that we're going to see even an acceleration from here. We go from dumb human to Einstein. And Einstein is something that you'll see releases on in the next couple of days from us. And it really is just such an exciting and slightly terrifying time.
To bring it back, I'm best known for running Stability AI where we created Stable Diffusion. So this is where we were just under four years ago. The first release of Stable Diffusion. It wasn't quite there, but it was the first really consistent model as we took an image model and we combined it with a text encoder and then made it available to everyone open source and free so you could run it on your own laptop. And there was such an explosion of innovation that it led to where we are today.
How did it actually work though? This is one of those things that I think isn't really covered a lot. The way that models like Stable Diffusion work is by something called a diffusion process. It's in the name. As opposed to language models that guess the next word by training on long strings of words and seeing, well, probably you're going to say this, then that, then that next. With diffusion models, what you do is you take an original image and then you add noise to destroy it down into its minimum representation.
You then have a text encoder that says this is what it is. This is a cat with its ears, etc. and you figure out how to rebuild by removing noise to get the generated image adding on extra things. In that way, you create this manifold of representations. And that's the real knowledge of what these models are. Except for unlike databases, you can have billions, trillions of words and images in just a few gigabytes because it learns the process and procedures.
When we've got that base mapping, we can then fine-tune it. So, this is how DreamBooth worked back in the day. I think you remember like take yourself and turn into an astronaut or gladiator or something. You took the input and the output and you adjust that latent space as it's known to point to your own personalization on there. So, this is you as a fluffy dog. Then you could take your dog and do Vincent Van Gogh, Pierre-Auguste Renoir, Leonardo da Vinci, and more. And that still holds today. We have these amazing base models that you then take and tune with your LoRAs and your other systems.
Now the way that these used to be was they used to have a straight diffusion process where we had lots of little particles moving over. From Stable Diffusion 3 and more, we moved to a more flow matching process, which is what's the smallest pathway between A and B, between those smiles. And that led to a lot of the breakthroughs that you're seeing now. So models like the new GPT image 2. This is an example. I think it's out in the next few days. And now at incredible levels of detail. They can generate things like this. And the open-source versions are coming out even faster. I think BFL just released its latest model and I know of at least three more coming where you can almost control every pixel and it kind of understands what you're trying to say.
But that understanding isn't really enough because what you need to do is you need to be able to have that control. So a lot of you familiar with ComfyUI that came out of Stability. We've been creating a new autonomous system at II called IIAtory. And this all is generated on the fly. It builds any interface you need because this is how you bring it all together. You go from your control to your composition and then you can share this and you can bring stuff together because the base generators are getting good enough. But we need to go beyond the base generators to be able to create the experiences and the systems that we really want. And that's the real difficult bit because we didn't really have the ability to judge what's going on. It was always a bit probabilistic because it wasn't quite right. But now as the models are getting good enough that they're actually competent intelligence as I like to say, this is when the real breakthroughs are coming and this is where you can take what's in your mind and make it reality. In fact I think for a lot of the non-image generating people, media generated people, Claude Code was one of the first examples of this where you could build stuff almost on the fly. Like I said, that whole interface was built on the fly by our new model III agent or our new agent harness.
At the turn of the year, you had people like Andrej Karpathy, some of the top AI programmers in the world, going from 90% code being human written and 10% AI to now 90% of code being AI written because Claude 4.6 was the first actually competent intelligence. The new image models and video models and audio models from the open source like AEP to Zuno to the text to speech models are all now also competent. The outputs pretty much match what you want initially, but we still haven't quite built the systems to bring them all together to understand what we want and more.
I think that's because it's not been very long. So, this was the original Stable Video test of Will Smith eating spaghetti. That was the test after 'how many fingers am I holding up?' And you can really see the advances there. In fact, you know, as we move into 2026, you can now even have Will Smith having spaghetti across from someone with full audio, etc. And probably actually spaghetti eating Will Smith if you really want. This reminds me of when we kicked off and we had an astronaut riding a horse. And one of the real challenges that was just solved last year is a horse riding an astronaut. The models can pretty much pull anything together now. And that's pretty amazing as you imagine it because you can then take this and you can edit it dynamically.
But what are the models doing as they do this? When we had Stable Diffusion, we created Stable Video and Stable 3D by understanding that in that latent space representation of cats and Renoir and others was encoded a lot of physics, a lot of actual understanding. So you took the base image model and then we extended it with video, with 3D, with more. And now what you're seeing is increasing amounts of compute, better data, understanding what's actually inside these models more and more as they go forward.
And this becomes interesting because now you can move to almost real time. So this is from PA whereby you can take your AI clone and this isn't the cheapest thing and again there are open source alternatives coming soon and it can talk to you dynamically via Zoom and more because the models are getting fast enough, cheap enough and intelligent enough that they can have this encoded functionality.
And that opens up again a whole new world when you can combine the different media types to have real experiences because a holodeck, you know, all very good. I'm on top of the Alps, but it can be a bit lonely without the personalities. Of course, you hope they don't break out of the holodeck and then, you know, try to take over the world as happened in a few episodes. But then, you know, Star Trek isn't really reality. I mean, we can already see the androids that we have won't be like Data or Lore. I mean, they've really got pretty good personalities, even if underneath they are multi-tentacle shots that will probably kill us all, but that's a story for another day.
I think many of you now are also using these new tools to build these multi-pipeline systems. And again, it's interesting in that you've seen Comfy, you've seen other systems, Flora, the systems that Render are now rolling out take these multiple production pipelines to give you that level of control and understanding. But I think probably the next step is when it can customize and adapt to you. When it can understand your preferences to be able to create this type of stuff automatically like this was just all done with voice just me talking to the system and it hasn't integrated 3D consistency and others yet because I just haven't told it to. But where we're going again, you can have any type of interface for any type of system created almost dynamically and that suits everyone because everyone engages with media and their own creativity differently as well. And again, I think this is a big step up from what we've seen and the future is pretty much now.
And this comes at a time when again through extra training these models are learning physics. Like here's one of the latest video models. I believe this was VO3.1 actually, so not quite the latest. The ship sinks fast. The wood goes up. The sugar cubes actually dissolve. How did it understand that? Does someone give it a script saying, 'Oh, the sugar cubes dissolve, etc.'? No, it just knows the nature of sugar because it's been trained on so many things and it's built that latent space understanding as it approximates the physics of the world. And that's really what we have. The input classification destroying it down to its minimum semantic representation and rebuilding it allows us to approximate physics.
That's why from the generative diffusion type models to the JEPA type models Yann LeCun and others are talking about for semantic understanding we're really having big advances in things that can understand physics that can understand the real world. In fact, self-driving cars use diffusion models. You input what you see and Tesla made the bet that you didn't need to see lidar etc. You know, Waymos are pretty nice too and then it learns how to drive. How cool is that? That's a world model. That's a model that can create a holodeck and probably will have an Elon Musk one sooner rather than later. And again it's super powerful if you think about it because all you have is vision input coming in. How does it know what sugar is? It's somewhere in the training data set. It's somewhere in the language encoder. We found that when you took image and you took language models and you combined them, it would learn from the language models that were embedded in that. When you then extended it to video, the video would learn from the language models. In fact, when you look at models that are state-of-the-art or close to, such as from my old friends at Stability now at Black Forest Labs, Flux is a model that's built on a visual language model for Mistral. These models can learn from each other in different modalities and they start to approximate reality and the shortest route from A to B, from a boat made of sugar to how it sinks. That's pretty amazing if you think about it because that's what you need for a holodeck. You need to have a real representation of the physics of the world. One way you can do that is through an Unreal Engine type system or Unity. Those have encoded physics. The other is through a diffusion model or a JEPA model. But the reality is it's probably going to be a combination of these that come forward.
And the way they're learning physics again, I remember when you had ballerinas and it was a bit of a mess, you know, now you've got this and again I remember the Stable Diffusion 3.5 really. So we didn't quite have the arms and legs going properly or three and then we fixed that in 3.5 like this would have been unprecedented. This is the new Alibaba model. I mean, the dude's got moves, right? But again, this isn't surprising anymore. We get used to it just like driving a self-driving car dynamically.
But think about how this works. This entire model, and there's still some things like Dine, etc., is probably around about 20 GB, yet it can generate almost anything you can imagine. So, how far are you away? Well, this is the new Hunyuan world model, which is a very interesting one from the team at Tencent. So, you've seen a lot of these video models whereby you extended it and then it created worlds that you can now navigate. This one's interesting in that actually creates a multi-step process. And this is open source. It's actually about 5 GB. You can download it now. You generate the panorama, then you do the trajectory modeling, then the world expansion, and then finally you have the world composition that comes through. It's a multi-step process. That means that you can import and export the 3D meshes. And again, this is what we're going to need when we have holodecks again before hardlight holograms and things with the ability to describe and create. It's not just necessarily one diffusion model outputting and then you explore in. You want to have consistency of characters. You want things like the PA where you interact and talk. And you want that interface of control to be able to bring it together to be anything you can imagine. And I think again all these components are now coming into place for that. We just need a bit more scale. We need a bit more access and then we need to go forward from there as we generate the key frames, as we generate the interactions and more.
You also saw in this week actually another open-source model which is Lyra from Nvidia more restrictive license. You know the Nvidia are a bit like that but we love them still. This one you take an image and then you explore the Gaussian splat and then you can bring that into your Isaac simulator. You can open the doors. You can generate video feed forward as well to go forward and backwards because again if you tell one of these video models now explore this it will do that. Being able to bring the meshes and now you have high quality meshes backwards and forwards that's something again we're seeing for the first time that level of good enough, fast enough, I suppose cheap enough is really right there now and this fits with Jules's concept, you know, of neural rendering how can you then take that and map that to real world experiences. So this is the map of how it comes forward you kind of train it with a video diffusion transformer and again this is a transformer as a base mixed with a diffusion model, then you have the Gaussian splat for 3D upscaling and then you can integrate those splats backwards and forwards and a multi-component system. That's what enables this talent.
This then lets you create the worlds such as this, you know, like this is Doom guy in a world of screens playing Doom doing Doom. Again, this is the Genie model and I think Genie 4 and more will probably come out later this year. We might get generative GTA before we actually get GTA 6. I think it'll be a bit of a push, maybe one delay or two, but this is the type of technology and the type of models that we're looking at right now hitting the market. I think that means quite a lot because once you have creation then you need composition then you need the control and then finally you need to bring it all together to develop that iterative loop. I think that asset organization will be immense in this. Like once you have a character created with its own personality, you want to bring it backwards and forwards and we need real standards for that. And I think you know Render as a system has a real opportunity to build some great standards around assets with everything that's there.
We're going to need a lot more compute with the amount of pixels that we'll be shifting. A lot of models now are moving towards real time. You're seeing the consistency. You're seeing the one-step shots. And I think that you know that will be immense as you look at the system already straining like why didn't Anthropic release Mythos? Well partially it's because you know danger and things like that but you're already seeing the exploits being replicated by open source. I think a bigger deal here was the fact there just isn't enough compute out there and that's kind of scary when you think about it. You're running out and GPU prices are going up. DRAM prices unfortunately still aren't coming down. You know, I want an upgrade and we're not quite getting there.
I think that as these experiences go from being not competent enough to actually competent, from being not interactive to actually interactive, explore entire worlds together, generate and create them. Again, we'll need every single GPU we can get. And it might not be again the high quality stuff like you know this looks medium quality but now with technologies like FSR3 like DLSS and others you can upscale on the trajectories of these things very dynamically and that has a huge demand for the consumer graphics cards and more that are ideally suited for that. Okay, you might not quite like the hair in Resident Evil 3, Resident Evil Requiem and things like that, but I think we're seeing the advances again in these AI based upscalers in this asset generation and this asset control and creation. And we're just at the start of what's going to be the biggest boom in media ever. Not because necessarily we want to have the same things and the same different experiences individually, but we want to create brand new ones. And this creates a brand new opportunity for new economies. So last and last summer I released this book, The Last Economy. What happens when agents build AI? What happens when they can create assets and the cost of creation goes down to nothing? Well, stories still matter. You know, Taylor Swift is still going to sell out concerts. IP still matters. People like familiar characters. But the barriers to entry of creating amazing new things to creating holodeck type experiences, multiplayer ones are dropping faster than we've ever seen before and that's incredibly encouraging and again very democratizing.
I think that we continue to have the GPU shortage and that will be a real limiter just as we're seeing now with Frontier Intelligence being kept. But I'm incredibly optimistic with what I see on the open source and other pipelines of what's available to everyone. And I think that the systems here is do you give a damn about what you're creating? Because you'll be able to bring together again we're building a system for that others are too and we'll be open sourcing everything that we build so others can use it. You'll be able to bring it together dynamically in exactly the way you want and you'll be able to interact just like you saw with PA with your whole creative team and I don't think it's necessarily that you know the holodeck experience is individual it's more fun when you have an away team with you right I think the creative experience is also a lot more fun and the human experience when you create together this is why you know when I give talks and podcasts these days you know before we've got our new releases I've been encouraging people like, you know, it's very nice to have a meal with your family. It's better to create the meal together, you know, and I think generative media is similar. It's okay building by yourself, but it's a lot more fun, for example, creating an album together and a music video set with some friends in a really intentional way and pushing the boundaries of what's capable. Now, the intelligence is reaching that level of competence that it is accessible to everyone, but it's up to you what you do with it.
I think that the holodeck is near again minus the hardlight holograms etc. We'll be able to have that with our 3D glasses with the ocular occlusion and more. And again it's the most exciting time ever but we got to use it to create the stories that really matter positive stories. I know my friend Peter Diamandis is there with the Roddenberry XPRIZE and more like let's create those real amazing stories and let's create them together both on an individual basis and a communal basis because again that's what distribution is about. That's what decentralization is about. And I think again we're at this most exciting time ever and I'm very happy to be a part of the Render community. And I think there's a lot more to come. So you'll be able to see in the next few days some of that. We've had some big breakthrough advances in various areas and we'll be using ourselves every single GPU we can get. I think that this technology is too important to be just with a few people. That's why we kicked off Stable Diffusion. That's why we created Intelligent Internet to navigate the next page. And I think again the community here realizes that as well and it's up to us to use it the best we can. So with that, give you back a bit of time. Look forward to seeing you all in the community and all the amazing stuff you create. Thank you very much.
