That's a great idea. I can write three to four custom opening lines tailored exactly to your slush talk that you can just pick from or tweak. I'm so sorry. Not sure how that got in here. Anyway, I've kind of had it with these large language models. So today we want to discuss a new class of AI models that are deeply rooted in space and time. A new era of AI, if you will. So that's not only book smart, but street smart. And think about it. We trained the world's most powerful models on written words and it was a decent starting point. It was readily available data. And to be clear, language is this indispensable tool for humanity and society. But it's ultimately also just a lossy abstraction of the real world and our being in the world. If I say the word slush, like what comes to mind for you? This venue, the talks, or maybe the friends you met in between? It's not often quite clear. Human intelligence also far exceeds the origin of language. In fact, for two-thirds of our existence, we didn't even have language. And to really think about where human intelligence came from, it probably emerged over millennia of action and intent, acting out these things in unbounded environments, collecting feedback and seeing the consequences of our actions. And so I'd like you to invite to really think about human and animal learning. You know, like toddlers learning to walk or monkeys learning to climb trees, like trying yourself out, messing around, understanding your limits, often in a very playful way. Doing more of what works, doing less of what doesn't. It's really not how we train machines today. And so we here on stage think that building super intelligence, it isn't just about scaling data and compute. It might ultimately require a fundamentally different approach to AI. Teaching machines to truly play and explore. And truly intelligent machines they probably must move as this talk says from words to worlds and acquire this capacity to perceive, to anticipate, to improvise. And I'll be honest, we could not have better panelists here with us today. John Hanke, founder and CEO of Niantic Spatial. He was previously VP at Google where he led Google Earth, Google Maps, and Street View. Early in his career, he also created one of the first massively multiplayer online video games, Meridian 59. And his company, Niantic Spatial, is building a living model of the world that people and machines can talk to. Niantic, of course, has a rich history in spatial computing and AR, best known for Pokémon Go. That games part of the business was sold for 3.8 billion earlier this year. I think that's worth a round of applause. And the Niantic Spatial still has 200 people, a very focused team, 270 million in funding to basically now build new kind of maps that AI can talk to. And then we have Pim Dit, co-founder and CEO of General Intuition. Previously founded Metal, the largest platform for capturing and sharing gaming moments. About 10 million monthly active users, billions of gameplay clips every year. Fun fact, at age 14, he started the largest Runescape server in the world. I don't know if we have any Runescape players here. And just to make millions before he even turned 18 years old. His company, General Intuition, builds models for environments that require deep spatial and temporal reasoning. He recently raised over $130 million. It's actually Kosla Ventures' largest seed bet since their check into OpenAI in 2018. That's probably worth a round of applause as well. And with that, we'll dive right in. We'll start with you, John. I want to ground us in the real world. Take us back to this original idea for Niantic and maybe Pokémon Go as well.
I mean, it was we just wanted to have some fun and encourage people to go outside and explore a little bit, get some exercise, have fun with their kids or their friends. That was really the mission. And that was kind of the unifying thing that, you know, if you all are founders, it helps to have a mission like that that your team can get behind, that people feel good about what they're doing every day. So that's what did it for us.
What is it about being in the world? I know you like to take walks as well.
I mean, I think it's deeply part of our human DNA in terms of we're evolved to know how to navigate really complex situations like the layout of this conference center. Our brains just do that very naturally, but it's incredibly complicated. And when you start trying to program robots to just navigate a simple path, much less understand all the content of the world and what other people and things might be doing in the world, we're really terrible at it right now, which is why I'm so excited about the space because there's a ton of innovation in AI. I think if we can unleash that on robotics and the physical world, it could be transformative and significantly improve the quality of life for human beings.
And just on a high level before we maybe go into games specifically, but play as a higher level concept, what is the role of play and how are physical worlds and play connected from your perspective?
You know, I don't know how is play connected to all this. I think it's interesting that we kind of both had gaming backgrounds. Dennis at Google had a gaming background as well. It's a great way to learn, I think, and explore new technologies. So I don't know, the connection here for me is just that being an on-ramp to develop really new tech and a place that's kind of forgiving so it can mature. We developed technology for Pokémon Go in Niantic, the gaming company that we spun out into this new company, but it was the ability to work in that space of games for many years that let us kind of mature it before it was really ready for any other kind of more serious application.
Yeah, I mean it's fair to say maybe that in some ways now you kind of refocus on your original mission because even going back at Google there was clearly this fascination with mapping the world and then Pokémon Go was such a substantial success that I don't know, maybe you got distracted a little bit or was part of the puzzle and now you're fully focused back on that original idea. Is that fair?
Yeah, I mean personally I love games. I grew up, that's what encouraged me to learn how to program because I didn't have money to buy games for my little computer. So I typed in code from online and from magazines and wrote games. So yeah, that was my entree in, but I've always loved maps too. So I grew up with National Geographic, the magazine, and the foldout maps. And the ability to mix those things together has been a huge amount of fun and kind of the theme of the past 20 years of my life.
I know games were instrumental to your formative years as well and then ever since. So what does gaming mean to you?
Yeah. So I grew up with Tourette's. And so for me, I struggled in school. I actually just told John my first computer program was actually Google Earth. Maybe an applause to John for making everybody dream about the world instead of their own local cities. And so for me it was really, I started my career reverse engineering Runescape and eventually building the largest private server on Runescape. And that led me to eventually joining Doctor Borders after a few years doing something completely different, mapping related, and then started a games company again. And as part of that, we essentially help people create these clips which have interactions with the environment as well. Right? Gamers love putting these controller overlays and things like that on top of their videos. And as a result, we have pretty much the largest data set of interactivity in three-dimensional environments. And so yeah, for us right now the bet is really that we can take this LLM-like bet for spatial temporal reasoning and move everything that people are currently doing in pre-training for robotics and for games into post-training.
There's also clearly a very rich history interlinks between AI and games. I mean just to spell it out, a lot of the foundation research labs started in gaming. You mentioned Demis briefly, he was a game developer before he became a research scientist. And Nvidia, the world's highest valued company today, actually started off in gaming with GPUs and 3D calculations. You know, I think AI entered for you like maybe over the last couple of years. When you think about AI and gaming, what was the impetus for transitioning now to leveraging that game data to solve AI problems and what was the inspiration for General Intuition?
Yes. So maybe it's good to start by really explaining how text-based language models work, right? So they generate a piece of text, then they take what they generated and they run it through the model and then they predict the next token or the next word. That doesn't allow for any observing of the environment to change, right? So this is called autoregressive modeling. You're using the output of the model to model the next sequence. And so as a result of that, there is no sense of interaction. The environment has no way of the generation has no way of actually adapting to its environment in real time. And that's just not how the real world works. So as humans, we invented text as this way of compressing the world around us, right? We needed to communicate what we're perceiving with each other in a common understanding. And as part of that, we compress essentially what we are perceiving down into words. And by the way, that's incredible. We modeled a compression method that we invented as humans, and we produced one of the most beautiful systems ever invented, right? And now I think what we need to do is take the lessons from how we modeled those systems and transfer them into modeling systems that actually represent interactions of how the world works. And the world is fundamentally space and time, right? We perceive three-dimensional, and text isn't. And games is just, you know, the physical world just has a lack of ground truth data on three-dimensional interactions, right? And so I'll maybe explain. If you try to start with like a YouTube data set or a video data set, you kind of have three problems. The first problem is that most YouTube videos for instance are recorded with a mobile phone or a GoPro, and so you have to solve for pose estimation. The problem with that is that it doesn't necessarily generalize super well to every environment. So that's information loss event number one. Information loss event number two is that you don't actually have the inputs and the actions. And so as a result, you have to guess them or you have to hire human labelers to do it. And the third is that your actual spatial temporal reasoning is largely dependent on where your eyes are actually looking, which those systems actually don't capture. They might perfectly capture a part of your vision, but you might be looking right when you make that decision. And so when you're playing video games, you're essentially simulating those optical dynamics with your hand because it's the only way to actually look around and do these things. And so our bet is really kind of like the way that Anthropic focused on code, that you can start by learning from this spatial and temporal reasoning in video games and then transfer it out into other applications. And that's why we're starting in video games. And then because we're sort of taking the LLM bet in a way in terms of just large scale pre-training on this data, the bet is that we can just get all the data that labs are getting, we just need a lot less of it because we already have this really good base.
I want to unpack with both of you these terms world model and action model respectively, so you can mentally pre-prepare for that. Just to paraphrase what you just said, I think a big part of this ambition is really to have machines learn like humans most likely learn, through our actions in these environments, collecting the feedback from the environments, which is a data set that simply doesn't exist in the real world. We're not all running around with brain computer interfaces to measure our actions. We can measure the response in the environment. But video games are kind of like the closest thing or simulations are the closest thing we have to human action in worlds. It just happened to be virtual worlds. But the theory is that if that can generalize, then the real world would just be N plus one, so to say. But let's try and, because these are very vague terms actually and terms that are used in many different ways, when you think about world models or geospatial models, John, how would you describe those and why are they worth pursuing?
Yeah, the thing that we're pursuing is different than what Pim is doing, which is incredibly cool. When we were preparing to sell our baby, our gaming business, Pokémon Go, we looked at the technology that we built for augmented reality, which was built on about 30 billion posed images. So these are images of the world taken by players, contributed to this common map with a lot of spatial metadata associated with them. And Brian, my co-founder, and I asked ourselves, well, what if we constructed a map that was native to AI? So we're all familiar with Google Maps and Google Earth. These are products that are designed for humans to look at and understand. So they're really written for our comprehension, and AIs can kind of use those as resources but not super effectively. So if you want AIs to be able to reason about the world and accomplish things in the world, it might be navigating a robot, it might be solving a complicated problem having to do with real things, real cities, real actions that take place in the world, that needs to be grounded in real data about the world. So we got super excited about this idea of building a new kind of map that was really made for AI so that AIs can have that part of the brain that you alluded to earlier that we all have as humans that's really just dedicated to understanding the world, building out maps about the world and then reasoning about it. So we think that's a huge opportunity and we're building that model, but it's very much about the world as it exists. It has to be accurate or it's not useful for the kinds of problems we're trying to solve.
I think often times when people use the term world model or if you would Google that, a lot of what comes up is also these generative worlds. It's like you're basically feeding actions into a system. And we can now do generative video creation fast enough to basically render the next frame based on the history of the video, based on the actions that you feed into the system and effectively create video games in the truest sense of the word. Games that are just videos, playable videos at this point.
Which is very cool, but different than what we're doing.
Exactly. And so I think those are world models building worlds or random worlds. What you're doing is really you want to build a model of the world of Earth, right? So tell us a little bit more what would be possible use cases, what are the first products we can measure? Are you only going to be talking to robots or us humans too?
