Today we want to discuss a new class of AI models that are deeply rooted in space and time. A new era of AI, if you will, that's not only book smart but street smart. Think about it: we trained the world's most powerful models on written words. It was a decent starting point, readily available data. To be clear, language is an indispensable tool for humanity and society, but it's ultimately just a lossy abstraction of the real world and our being in the world. If I say the word 'slush,' what comes to mind? This venue, the talks, or maybe the friends you met in between? It's not often quite clear. Human intelligence also far exceeds the origin of language. For two-thirds of our existence, we didn't even have language. To really think about where human intelligence came from, it probably emerged over millennia of action and intent, acting out things in unbounded environments, collecting feedback, and seeing the consequences of our actions. I'd like you to really think about human and animal learning: toddlers learning to walk, monkeys learning to climb trees, trying yourself out, messing around, understanding your limits, often in a very playful way, doing more of what works and less of what doesn't. It's really not how we train machines today. We here on stage think that building superintelligence isn't just about scaling data and compute; it might ultimately require a fundamentally different approach to AI, teaching machines to truly play and explore. Truly intelligent machines probably must move, as this talk says, from words to worlds and acquire the capacity to perceive, anticipate, and improvise. I'll be honest, we could not have better panelists here with us today. John Hanke, founder and CEO of Niantic Spatial. He was previously VP at Google where he led Google Earth, Google Maps, and Street View. Early in his career, he also created one of the first massively multiplayer online video games, Meridian 59. His company, Niantic Spatial, is building a living model of the world that people and machines can talk to. Niantic, of course, has a rich history in spatial computing and AR, best known for Pokémon Go. That game's part of the business was sold for 3.8 billion earlier this year. I think that's worth a round of applause. Niantic Spatial still has 200 people, a very focused team, 270 million in funding to basically now build a new kind of map that AI can talk to. And then we have Pim Dit, co-founder and CEO of General Intuition, previously founded Metal, the largest platform for capturing and sharing gaming moments, about 10 million monthly active users, billions of gameplay clips every year. Fun fact: at age 14, he started the largest RuneScape server in the world. I don't know if we have any RuneScape players here, and he made millions before he even turned 18. His company, General Intuition, builds models for environments that require deep spatial and temporal reasoning. He recently raised over 130 million. It's actually Kla Ventur's largest seed bet since their check into OpenAI in 2018. That's probably worth a round of applause as well. With that, we'll dive right in. We'll start with you, John. I want to ground us in the real world. Take us back to the original idea for Niantic and maybe Pokémon Go as well.
We just wanted to have some fun and encourage people to go outside and explore a little bit, get some exercise, have fun with their kids or their friends. That was really the mission. That was the unifying thing. If you're all founders, it helps to have a mission like that that your team can get behind, that people feel good about what they're doing every day. So that's what did it for us.
What is it about being in the world? I know you like to take walks as well.
I think it's deeply part of our human DNA. We're evolved to know how to navigate really complex situations, like the layout of this conference center. Our brains just do that very naturally, but it's incredibly complicated. When you start trying to program robots to just navigate a simple path, much less understand all the content of the world and what other people and things might be doing, we're really terrible at it right now. That's why I'm so excited about the space because there's a ton of innovation in AI. If we can unleash that on robotics in the physical world, it could be transformative and significantly improve the quality of life for human beings.
On a high level, before we go into games specifically, but play as a higher-level concept, what is the role of play? How are physical worlds and play connected from your perspective? I think it's interesting that we both had gaming backgrounds. Dennis at Google had a gaming background as well. It's a great way to learn and explore new technologies. So I don't know, the connection here for me is just that being an on-ramp to develop really new tech and a place that's kind of forgiving so it can mature.
We developed technology for Pokémon Go in Niantic, the gaming company that we spun out into this new company. But it was the ability to work in that space of games for many years that let us mature it before it was really ready for any other kind of more serious application.
Is it fair to say that in some ways you refocused on your original mission? Even going back at Google, there was clearly this fascination with mapping the world. Then Pokémon Go was such a substantial success that maybe you got distracted a little bit, or was it part of the puzzle? Now you're fully focused back on that original idea.
Personally, I love games. I grew up, and that's what encouraged me to learn how to program because I didn't have money to buy games for my little computer. I typed in code from online and from magazines and wrote games. That was my entree in, but I've always loved maps too. I grew up with National Geographic magazine and the foldout maps. The ability to mix those things together has been a huge amount of fun and kind of the theme of the past 20 years of my life.
I know games were instrumental to your formative years as well, and ever since. So what does gaming mean to you?
I grew up with Tourette's. I struggled in school. I actually just told John, my first computer program was actually Google Earth. So maybe an applause to John for making everybody dream about the world instead of their own local cities. For me, it was really that I started my career reverse engineering RuneScape and eventually building the largest private server on RuneScape. That led me to eventually join Doctors Without Borders after a few years doing something completely different, mapping related, and then started a games company again. As part of that, we essentially help people create these clips which have interactions with the environment as well. Gamers love putting controller overlays and things on top of their videos. As a result, we have pretty much the largest dataset of interactivity in three-dimensional environments. For us right now, the bet is really that we can take this LLM-like bet for spatial temporal reasoning and move everything that people are currently doing in pre-training for robotics and for games into post-training.
There's clearly a very rich history of interlinks between AI and games. A lot of the foundational research labs started in gaming. You mentioned Demis briefly; he was a game developer before he became a research scientist. Nvidia, the world's highest-valued company today, actually started off in gaming with GPUs and 3D calculations. When did AI enter for you over the last couple of years? When you think about AI in gaming, what was the impetus for transitioning now to leveraging that game data to solve AI problems, and what was the inspiration for General Intuition?
It's good to start by explaining how text-based language models work. They generate a piece of text, then they take what they generated and run it through the model, and then they predict the next token or the next word. That doesn't allow for any observing of the environment to change. This is called autoregressive modeling; you're using the output of the model to model the next sequence. As a result, there is no sense of interaction. The environment has no way of adapting to its environment in real time. That's just not how the real world works. As humans, we invented text as a way of compressing the world around us. We needed to communicate what we're perceiving with each other in a common understanding, and as part of that, we compress essentially what we are perceiving down into words. Then we modeled that, and by the way, that's incredible. We modeled a compression method that we invented as humans, and we produced one of the most beautiful systems ever invented. Now I think what we need to do is take the lessons from how we modeled those systems and transfer them into modeling systems that actually represent interactions of how the world works. The world is fundamentally spatial, right? We perceive three-dimensionally, and text isn't. Games are just the physical world has a lack of ground truth data on three-dimensional interactions. If you try to start with a YouTube dataset or a video dataset, you have three problems. First, most YouTube videos are recorded with a mobile phone or a GoPro, so you have to solve for pose estimation. The problem is that it doesn't necessarily generalize super well to every environment, so that's information loss event number one. Information loss event number two is that you don't actually have the inputs and the actions, so you have to guess them or hire human labelers. The third is that your actual spatial temporal reasoning is largely dependent on where your eyes are actually looking, which those systems don't capture. It might perfectly capture a part of your vision, but you might be looking right when you make the decision. When you're playing video games, you're essentially simulating those optical dynamics with your hand because it's the only way to actually look around and do these things. Our bet is really that, the way that Anthropic focused on code, you can start by learning from this spatial temporal reasoning in video games and then transfer it out into other applications. That's why we're starting in video games. Because we're taking the LLM bet in terms of large-scale pre-training on this data, the bet is that we can just get all the data that labs are getting, but we just need a lot less of it because we already have this really good base.
I want to unpack with both of you these terms 'world model' and 'action model' respectively. To paraphrase what you just said, a big part of this ambition is really to have machines learn like humans most likely learn, through our actions in these environments, collecting the feedback from the environments, which is a dataset that simply doesn't exist in the real world. We're not all running around with brain-computer interfaces to measure our actions. We can measure the response in the environment, but video games are kind of the closest thing, or simulations are the closest thing we have to human action in worlds. It just happened to be virtual worlds. The theory is that if that can generalize, then the real world would just be N plus one, so to speak. Let's try because these are very vague terms, actually, and terms that are used in many different ways. When you think about world models or geospatial models, John, how would you describe those and why are they worth pursuing?
The thing that we're pursuing is different than what Pim is doing, which is incredibly cool. When we were preparing to sell our baby, our gaming business Pokémon Go, we looked at the technology that we built for augmented reality, which was built on about 30 billion posed images. These are images of the world taken by players, contributed to this common map with a lot of spatial metadata associated with them. Brian, my co-founder, and I asked ourselves, what if we constructed a map that was native to AI? We're all familiar with Google Maps and Google Earth. These are products designed for humans to look at and understand, so they're really written for our comprehension. AIs can kind of use those as resources, but not super effectively. If you want AIs to be able to reason about the world and accomplish things in the world, whether it's navigating a robot or solving a complicated problem having to do with real things, real cities, real actions that take place in the world, that needs to be grounded in real data about the world. We got super excited about this idea of building a new kind of map that was really made for AI, so that AIs can have that part of the brain that you alluded to earlier that we all have as humans, dedicated to understanding the world, building out maps about the world, and then reasoning about it. We think that's a huge opportunity, and we're building that model, but it's very much about the world as it exists. It has to be accurate or it's not useful for the kinds of problems we're trying to solve.
Often when people use the term 'world models,' if you Google that, a lot of what comes up is also these generative worlds where you're basically feeding actions into a system. We can now do generative video creation fast enough to basically render the next frame based on the history of the video, based on the actions that you feed into the system, and effectively create video games in the truest sense of the word, games that are just videos, playable videos at this point. Which is very cool but different from what we're doing.
Those are world models building worlds or random worlds. What you're doing is you want to build a model of the world of Earth, right? Tell us a little more about what would be possible. What are the first products we can imagine? Are you only going to be talking to robots, or us humans too?
I'm very long on robots. I'm long on humans too. I'm long on robots making life better for humans. I think there's a massive opportunity for robots to improve all of our quality of life, to make our cities more livable, our streets safer. There's a huge positive impact that can come from that, and we're seeing the beginnings of it. I got to visit with the founder of Starship earlier, who's doing their robots running here as well as in many other countries around the world. That first wave of deployed real-world robots is out there, a few thousand of them. But if you talk to companies like Tesla and others in the humanoid space, they're talking about millions of robots over the next few years. I really believe that's going to happen. Jensen Huang said it's the next, I think he said, 3 trillion dollar industry. He believes it's going to happen. I would encourage all of you to think about what kind of opportunities that's going to create for your businesses, what kind of impact it's going to have on your own personal life whenever this kind of labor force is really brought to the world. We need cleaner cities, we need cheaper housing. There are so many things that I think that kind of resource can help us achieve, but it needs to be spatially aware and geo-grounded, and we want to make our contribution to that.
A cousin, I think, of world models are action models or action policies. Pim, what is that?
