The government doesn't have to have equity ownership or be on a board to dictate who has access to what technology. And then that's now making other countries try to figure out how can I broker access to frontier intelligence if my country's companies don't make it. And I think the most logical way of doing this is by providing powered land and data centers, most likely for inference, and sort of translating into this high-level stat we have of there's about 138 billion of investment in sovereign AI. So if you count all the initiatives in the Middle East, in Europe, in Asia, and the US, that's how much money governments are spending to basically stand up data centers.
Hello everyone and welcome back to MTS. Today we are joined live by Nathan Benaich, who's the founder and general partner at Air Street Capital, and he is the author of the State of AI Report. And this report's been published every single year since 2018, I believe. Great. That's awesome. Yep. And now the ninth edition came out this morning, and it covers everything from the three-lab race at the frontier, AI increasingly helping build AI, and the enormous growth in inference, and the new challenge of controlling autonomous agents. So, Nathan, welcome to the show.
Pleasure to have you. There's lots to unpack. I know you guys just launched it this morning. Tell me, just before we dive into some of the details here, I want to know a little bit of how you actually put this report together. How do you identify what to cover? State of AI is tough. There's a whole slew of things that you can cover. And so, how do you think about it?
We've definitely been hill climbing the difficulty. Like, that's for sure. [laughter] But yeah, when we started this in 2018, you know, there was basically no canonical open-access document that wasn't a consulting report about progress in AI. And you know, as you guys know, it's really important to consider different dimensions. So in the report, we consider research, and we dive into that at the start to make it super clear that this is not a consulting report. And then we dive into industry trends, politics, safety, and then make predictions. And I think all those different angles are increasingly super important to the debate. As to how I do it, I mean, it's really an offshoot of my day job, which is investing in technology companies with a focus on AI that I've been doing for my whole career since like 2013 with Air Street. And so with that, I get a vignette into how labs are thinking of things, how real companies are kind of trying to make use of AI, how people in politics are debating the key issues of our time in different countries. And I spend time in Europe and North America, so I try to bring a bit of that global perspective to the forum. And then I'm able to track some emerging trends like safety. We started covering that independently maybe two, three years ago. You know, at the time we had a chart which was pretty controversial in the community where we counted the number of safety researchers who are exclusively focused on safety in big labs and estimated there was probably less than 200 people working on this. And to see how far that field has gone is, you know, pretty incredible. So I'd say just constant monitoring of the situation, you could say [snorts and laughter] that, and blood, sweat, and tears, really.
Oh my gosh. Okay. Well, you just gave a good overview of what other things we want to talk about today, from just the technical angle to also the politics, the geopolitics around it, the culture, everything around. Let's start a little bit with the technical angle. I think people are really curious on predictions around AGI. We kind of joke with monitoring the situation that we could rename it to monitoring the singularity. It's like originally we were like, 'Oh, we're going to put a clock timer on singularity is coming. It's near. It's nearer.' And when is that actually going to happen? You had some interesting reports here about AI is helping build AI. You say Anthropic internal index says Claude led 26% of measured model R&D in August. Tell me a little bit about what you learned and how did that inform potential AGI or RSI predictions.
Yeah, I mean, this is like the biggest topic I think in the last 12 months and certainly one right now. And the general idea has been, you know, for the longest time we've been figuring out how to get a computer to learn anything remotely useful from what humans do every day. And then we got that in different modalities like image, video, etc. They've all been falling. And then we're kind of moving into mushing all these different modalities together. And then now really smart people, or basically every smart person on the planet who's ambitious, is working on AI. So lots of different perspectives coming into producing better AI. So we have smarter people working on it. But then once you get smart enough models, can they start suggesting new paths for research, or maybe not even new ones, but just like automating the grunt work of doing AI research and implementation of these ideas? And I think it's like one of those grandiose ideas of like, can we create a machine that can create a better machine? And now the major labs like OpenAI and Anthropic have both been giving kind of vignettes into how they see this manifest on a day-to-day basis. And the stat we pulled out from Anthropic is that 26% of Claude's model R&D is getting guided by Claude. And it was sort of earlier this year there was this really nice post on building recursive superintelligence that Anthropic published, and you could see their developers accepting eight times more commits of code than they were just a couple months ago. And that's kind of correlating with model progress. And then they had this other experiment which I thought was neat where they had researchers kind of go down a research path, and at some point they forked to where humans continue to make decisions and then models made decisions, and then they had humans rate like, were the humans better? Were the models better? And they found with the more frontier systems that more and more of the research directions suggested by the AI were better than the human. Which for me I think is very exciting. I know there's some scary factor with this too, but when you kind of unwind where we came from with deep learning and DeepMind that showed that you could get a machine to, you know, play video games and then play Go and have these move 37. This whole idea that basically AI is like, can we discover the new global maximum, like the best solution ever to solve a problem that is way better than the local maximum that humans have designed and developed over their entire career. So I think that's like the idea with this ASI. It's like, can you get a machine to get us to a better place on all the problems that we really care about? And I think we're already seeing glimpses of it right now.
The other interesting stat that you guys pointed at too is not just how it's being used inside of development, but also the length of horizon tasks that it can actually cover and how that's even increased a lot just at the start of this year. So you had it from January to what it was over the summer. Tell me a little bit more about that.
Yeah. Yeah. If I recall the figures correctly, it's like...
Pull it up, but like the task length that's around three to six hours or something very short was succeeding like 18% of the time without any human intervention. And then now like tasks that take 36 hours or maybe even longer can be done at 18% accuracy again without human involvement. So the same kind of error rate but for much longer tasks. And I think you can see this anyway if you're like an avid user of Codex or Claude Code that you can just increasingly dump like whole projects or whole work, and you know, maybe give it some steering in between, and the thing just like goes.
Yeah. And it just goes. Yeah. The specific numbers you had, you said it used to be 18% was four to eight hours of human labor, and then now it's 32 to 64 hours of human labor that it can achieve. So we're seeing that too. I mean, it's interesting because people say that big concerns around AI safety and alignment often happen during these long horizon tasks, and the longer horizon tasks that you give an agent, the more you're at risk for hitting some of these considerations.
Yeah, I think there's another chart in there that I really liked, which is the Codex one about showing token usage amongst AI researchers at OpenAI, and it shows you that the vast majority of token usage is on implementing ideas like doing data pipelines, data cleaning, building infrastructure, but then a very, very small amount is actually figuring out what research to do. Yeah. And you know, I grew up in science. I did a PhD in computational biology and cancer research. And so I spent my life at the lab bench reading papers, trying to figure out like, well, what problem should I even solve? Like what's going to be useful in like three years, and then how do I get there, and like what's the most impactful experiment to run to answer a question? And so I think if we see token usage grow for that category, then that's another indication of RSI actually improving.
I want to shout out our sponsor, Lovable. You know the app you've been meaning to build, the internal tool, side project, or product you'd otherwise lose a weekend to. With Lovable, you get software you can ship today. [music] That includes an editable codebase you can inspect and change, plus two-way GitHub sync. Lovable also handles the backend and infrastructure, including managed Postgres, object storage, [music] hosting, payments, and more, so you don't have to wire it all up yourself. And through Lovable's MCP server, [music] you can create and deploy your project directly from the agents you already use. Turn ideas into software people love with Lovable, lovable.dev. [music] Now onto the episode. That comes down a little bit to taste and human judgment, like these earlier questions that we had around, oh, AI will never replace that. But if it can replace what we should be researching, what are the theses that we should be...
I mean, we'll find out in the 2028 State of AI Report.
Oh, 2027. Yeah. Yeah. Yeah. Way sooner.
Oh my gosh. Okay. One of the other areas I thought was really interesting is around robotics. We're saying robotics is having its GPT-2 moment. I really like that. I mean, of course, we're seeing tons of robotics here in the Bay Area, tons of companies who are working on that, but the average person doesn't really feel this. It's not really felt for regular consumers or even factory owners, whatever it might be. And so, tell me a little bit more about what you found there.
Yeah. Well, the general gist over the last few years has been like robotics is really hard, brittle. Like you see these DARPA videos of like the robot trying to open a door and like tripping over itself somehow. And then we went through a phase of labs starting to focus on it. You know, OpenAI had a team, they solved the Rubik's cube with a single hand. And then a few years later, famously disbanded that whole team. Robotics again was kind of in the dark ages. And then probably two, three years ago, the robotics community started to adopt language models and then basically bolt on a vision model. So basically a robot can perceive the environment and then have a language model read the pixels and describe what was going on, and then with that you could get better understanding of the environment, and you could basically tell the robot what to do in natural language, and it would try to communicate and kind of execute your wishes. And then now I think what's really cool is like these different modalities were starting to just merge together. Like we had separate image generators to make pretty pictures. We had video generators to make five-second clips. The video generators to become big had to understand the world because pixels need to be consistent, like the famous Michael Jackson spaghetti video. And then you have this next branch of world models and robotics models where you're trying to predict the next action that a robot should take to move something on a table. And now it seems like you're just mushing all that stuff together into robotics world models. And that's sort of what we're seeing in the report this year with this GPT-2 moment, where as you scale pre-training video across all sorts of different actions, you now get robots that can solve a task single or zero-shot with prompting.
And honestly, the thing I love the most about State of AI, apart from like after launch day I get to talk about it, is just going back in history, and so many times what is new is old. There was a company I was involved in many years ago, and they created a dataset called something something, and the idea was you would get human annotators to basically label actions of videos, like person is drinking water, person is doing this or whatever, and they tried to use convolutional neural networks back then to train visual common sense, and it just didn't work. But the idea was great, like you know, learn visual common sense from video because video represents reality, or at least real video represents reality. And so what needed to happen over the preceding five, six, seven years is just getting an architecture that can eat all this data and learn from it and not forget what it did before and kind of fuse all these different modalities together. So it's amazing that like, you know, whatever five, ten years later, these same ideas are working, but now we have the sort of infrastructure to...
What's old is new. No, it's pretty interesting around world models and how being able to get more accurate world models, being able to actually simulate exactly what's going to happen next, is a huge unlock for robotics. You don't need to just train on a lot of this human data. We've had quite a few people who are building world models, I think even from like general intuition, come on and share that and saying, 'Hey, there's a world where this gets so good that it kind of breaks what you'd think would be the scaling laws of what you might need for human-only data.'
Yeah. [clears throat] Yeah, for sure. And I think today one of our portfolio companies, Odyssey, released Odyssey 3, and there it's like a world model that is trained on a variety of different things, but they've put it on a car in India and it can drive off-road. They put it on a robot and it can take actions, and they put it in front of kids who can play video games through prompting, you know? So it's wild.
