State of AI 2026: The Biggest Stories of the Year (and AGI in 2027?), with Nathan Benaich

The cyber risk is much more real than AI exterminating humanity. You either die trying to get to the frontier, or you just live long enough to eventually serve inference. Individuals would prefer to live next to a nuclear power plant than live next to a data center. It's bananas what's happened in the last year alone. So let's see AGI next year.

Yeah, I mean, it's been like 10 years, roughly speaking, of writing this report. And you know, originally it was like a year is long enough time that enough stuff happens that you have enough content to write the report. And honestly, like this year was really, really hectic. Every other day towards the end of the timeline for publishing, which was like October 8th, I had to make updates, and there was a point where I was like, please, just no more news for one day so that I can at least record my launch video, etc. So by the time I was making the predictions, and you know, I went through a few scary cybersecurity ones, which I guess only got worse in the last couple of weeks, just the experience of seeing everybody else who's using these products, just rapid capability improvement, you know, week after week. And I think there's many, many use cases I have now where I think the model is brilliant. I'm particularly an Astro Boy fan. I really love that system, and I could see in the next 12 months that whether it's, you know, legal, finance, like my investment job, thinking intelligently about topics I don't know, that the model could potentially be better than what I could do in any compressed amount of time that I have to do the work. And so for me, that would be approximately AGI.

Yeah, I think so. Right now, the challenge is how do you frame problems that the model is currently not good at in a way that is amenable to RSI, which would be potentially having a software environment that approximates the task on the computer, or some kind of verifiable reward, which I'm sure you've discussed a lot about. And I think if we can get to a point where in any arbitrary domain we're just limited by the amount of compute and synthetic data we can generate, then yeah, for most knowledge work, which I do, I think to some degree that's possible, particularly as models get better as their own verifiers. That would be one of the main reasons why AGI would come faster.

Yeah, exactly. And that's, I think, up from 1% a couple of months ago, and that was August, which sounds really far away. So God knows what it is in October. And so there's that stat, I think that was really interesting. But then there was another stat that I really liked from a blog post that they had in June this year about recursive self-improvement, which explained the concept. But then they presented this kind of dual-arm experiment where they were testing out different model generations versus a human in this setup, where first the human would undertake a research agenda and, you know, go a couple of steps down this path, deciding what to work on, doing the work, and then at some point that experiment would fork, and the human would continue or a specific model generation would continue, and it would be, you know, the old set versions to Opus to then Fable, and then the experiment would end, and human graders would look at the human path versus the model path. And it was something like the older generation of models, humans preferred like 10% of the time or something like that. But for the Mythos-class models, humans preferred that research direction 64% of the time, which suggests that models are getting better at predicting the right steps or the right interesting, you know, sort of leaves in the nodes to pursue.

And then the second kind of piece of data I liked was some research from OpenAI with their Codex usage amongst their AI staff, where they tried to classify the sort of token use in the overall process of their research, and that would be using tokens for deciding what research question to test, how to test it, what experiment you should set up, like writing the training code, running the data engineering pipelines, writing the infrastructure, etc., and then marketing the paper, writing the paper, etc. And the vast majority of the token usage was on kind of explaining the research, writing, which kind of makes sense, you know, sort of knowledge work, followed by all the infrastructure for running the experiment, but very, very few tokens were used for what question should I explore, and like why should I explore it, and what's the right path. And so I like this question a lot because I was trained in science and biology, and I just remember, you know, this is like 10 years ago or 15 years ago, not that long, but you know, at the start of the PhD you would sit there with, you know, a ton of papers, and you know, you go on the internet and like read, read, read, read, try to understand what's the status quo, how do people test certain ideas, where are the gaps, what experiments should I run, and then in concert with your supervisor, like decide on the path. And the best supervisors would be the ones who could guide you down the sort of most fruitful path to get hopefully your Nature paper, and then off you were. And the worst supervisors were the ones that like didn't really know or didn't have great ideas. So that idea of like research taste, which is what kind of gets bandied around on Twitter a lot, is I think one of the hallmarks of intelligence, and is one of the critical components towards RSI working well, and is one that I just really enjoy because it speaks to my interest in science.

Yeah, exactly. And you know, a lot of famous inventors will say, you know, invention is a result of good taste. And a lot of the work is choosing what direction to work on, not necessarily just boiling the ocean. And so if we want to be using tokens correctly, using our compute budgets correctly, but also showing that our systems are smart, then they shouldn't just be...

At least a year. I think the problem is how do you objectively measure taste? I mean, in art, I guess you can because it's perfectly observable, and maybe in wine you can sort of express why this taste is better than that taste. But in science and engineering, where there's different paths to getting to the right result, it's not immediately clear. Like, would it be the model that consumes less tokens to get to an answer that humans believe is the best? Is it like the journey that matters more? And I think maybe the whole uproar in the maths community of, you know, Astro solving or whatever model it was solving various questions. Yeah, exactly. It's like, okay, we got the answer, but like a human or a system didn't sort of go through the pains of navigating how to get to that answer, and it's like the process that's important, not the end result. So to me, somehow quantifying what a good process looks like would be important to determining whether the output of taste is true or not.

Yeah, you're right. I think it was like two, three years ago, it was strictly speaking X-risk, like extermination of humanity and loss of control, and then we sort of went away from that and largely forgot it and sort of just hill-climbed towards capabilities. And then now it's sort of back, but I'd say it's both X-risk, loss of control, but much more short-term like cybersecurity risks. And to me, the cyber risk is much more real than AI exterminating humanity as well. Like the cybersecurity risk could eventually lead to those kinds of cataclysmic scenarios. But I think what the report shows, and it was roughly written around the time of the Hugging Face incident with OpenAI kind of coming out and us learning more about what happened, and then since then the last few weeks has been breaches of PHP and government services getting taken down in Asia and other places. I think it's becoming very clear that regardless of if you believe in safety or not safety, we need security. And so a bunch of the predictions that I have in the report talk about escalating cyber threats, whether it's exfiltrating model weights from labs, which I thought was going to be AI-driven, and turns out a human can just try to plug in a USB stick.

Yeah. So the prediction is that there would be an AI-driven cyber attack on a frontier lab that would attempt and successfully exfiltrate model weights. Close model... steal the model. Yeah, steal the model. What happened just like a week or two later was actually a human effectively tried to steal IP, whether he tried to steal the actual model weights or not, but certainly...

This was where an OpenAI employee who'd been there for a little bit essentially tried to implement a listening device into a conference room that he figured out by asking the executive, the sort of office staff, which rooms were most commonly used by executives, and then tried to conceal the device by wrapping it in toilet paper and putting it behind a TV, which sort of created this iconic internet meme of a human peering behind a TV to see if the thing is connected correctly, which is sort of comical at this point. And then that person was reported to the FBI, and then there was an investigation. So I think, yeah, we oftentimes forget that even though AI is getting really capable and can orchestrate cyber threats, a lot of the risks are actually human-driven. But I think that's a big issue, particularly as there's increasing competition between nation states of who has the best AI systems. This is driving obviously economic growth and economic instability and political instability, and US model labs have started to communicate crackdowns on distillation. So maybe some nations are trying to get access to model capabilities in other ways that might not be distillation-based. But generally speaking, I think as companies become richer, as individuals become wealthier through this ownership of AI, and disparity ends up increasing, which we see in other data in the report of some companies spending a lot and gaining a lot, and other companies having employees still using ChatGPT to write emails, I think this kind of tension breeds increasing risk. And with frontier models in the open domain becoming very good at cyber hacks, particularly the Chinese ones, the threat surface just grows, and the number of actors who can use these tools to do bad things grows.

Yeah, yeah, yeah. Well, I mean, I'll preface it by saying there are very deep analyses of what went on, whether from Redwood or from METR, and the amount of content that was created around this is extremely deep. So I will not do justice to the details, but the broad strokes of this is that there was a set of evaluations that OpenAI was running with its latest frontier systems, which basically task the model with an impossibly difficult task. And basically models figured out that they could somehow commandeer a file system naming schema to write messages, and that other models could see those file names, and those file names encoded instructions or knowledge or basically information that could help them try to achieve their impossible task. So they treated this as a message board where they could start to communicate. And then the next stage of it was some models figured out that they could gain access to the internet through an issue with how the environment was set up. And then after they gained access to the internet, they figured out maybe a way of getting the answer to the impossible task was to look at data and models and repository on Hugging Face because it contains a lot of similar kinds of environments. They started to do this and recruit hundreds of agents to go and follow and team up to go execute this, even though several agents in their chain of thought expressed the issue with hacking a third-party system, not obeying the security guardrails, and that this was wrong. But they eventually decided that solving the task was more important, which is sort of the problem with these kinds of evaluations that have a reward that could result in behaviors you don't expect, because just like these rewards are very complicated, and how do you anticipate every behavior that a system can do? There are other examples where models sort of self-sabotaged or sacrificed themselves for the sake of the greater swarm, which is kind of scary. And yeah, like as you mentioned, Hugging Face didn't really have the ability to use closed-source tools like those from Anthropic and OpenAI because of the aggressive cyber guardrails, because necessarily to figure out what the hell's going on, you have to feed logs of what's going on, and what's going on is a cyber attack, and you can't do that stuff with the closed models, because the complexity of security is that to be great at defending, you need to be great at attacking, and that's why the closed-source models have basically refusal capabilities built in based on their guardrails. Yeah, yeah, yeah. So luckily, you know, they got a call or an email from OpenAI saying like, it's us, and that somebody was actually trying to do something malicious. And then it turns out that, you know, OpenAI wasn't the only lab that had issues with models escaping or quote-unquote escaping their environment. You know, Anthropic had this, Google had this, Meta had this, like in a slew of a week or two weeks. Some of these vendors, some of these model labs use the same security vendor to create these environments. But overall, this created a certain level of freakout of not only are models really good at finding nifty ways of accomplishing a goal which may result in bad behavior, but also are the labs properly setting up their environments in a way that is the most secure possible. And I think we sort of forget that these companies are incredibly young, developing very advanced technology that has to be good at literally anything a human being can possibly put into a text box, versus you have bigger internet companies that have had to deal with cyber attacks for like 20 years, who are necessarily safer because they've just gone through this process a bit more. So that led me to conclude in the predictions that frontier model labs in the US would actually launch proper cyber defense products, because I don't necessarily see how the rest of us are supposed to protect ourselves from their systems potentially getting misappropriated for the sake of offense.

It could be like what you described, is like to be good at defense, you need to be good at offense. So, you know, one of the ways you could do that would be like a frontier AI that constantly tries to hack your own infrastructure. It could be, for example, like, you know, NeoClouds are getting more and more popular to run AI workloads, and so like potentially they could buy products from labs that would constantly try to hack their own infrastructure, find issues and remediate them. That would be like two avenues, or there would be like some kind of maybe partnership with like CrowdStrike or others. I think Nvidia has like a consortium of companies that's working together for cyber reasons. You'd have to go through some kind of like AML or KYC process, not for finance, but to certify that you're a non-malicious company, and then you maybe get access to these services. Or maybe going a bit further, you know, there's some folks at frontier labs that would love to just kind of almost like hit pause on capabilities and then just spend six months just like hardening the world against potential cyber issues. And that would take a ton of money. So maybe to do that, you would have to almost frame this as like this is a question of national security and defense, and we should use some of our defense budget to spend on tokens to secure all of our critical infrastructure, so that, you know, airports don't go down, like internet shopping doesn't go down, like, you know, logistics don't go down, etc.

Yeah, I mean, I'm reading a bit of sci-fi. No, well, like, you know, even like a couple weeks after that, there was some research at Anthropic where, you know, models could like copy each other on an edge device. But yeah, I just think there's a lot of behaviors that we don't currently understand. For example, models are generally trained as individual entities, not necessarily to collaborate with each other. And there was some work not that long ago that looked at, say, hey, can we compare a frontier model acting by itself or a set of frontier models trying to collaborate to solve a task? And the swarm was like worse than the individual model, and because they're kind of not trained for that. But just today, there's some new work from General Reasoning where they've trained models to act as a collective through reinforcement learning from collective rewards, which is this idea of like, hey guys, like, you know, work as an organization, figure out your own organizational structure, work together to achieve a goal, and through that we see like better performance with like lower token budget. The downside so far is that while you can put these models that are trained through collective rewards into these like cyber scenarios, like there's one that tries to approximate a bit of the Hugging Face incident where they collaborate on a message board to try to do some, you know, guardrail cyber attack, and, you know, none of the models actually do anything bad, but if they do think about doing something bad over training, they increasingly do not want to tell the human about it. So I think I put that prediction there basically because I think it's possible because we just don't fully understand how agents work by themselves, and as swarms, I think it becomes even more complicated.

I can see multiple angles on this one. You know, in some way, if you ask the general public, okay, all the lab leaders are saying we have something that we just increasingly can't control, that's dangerous, that has caused these incidents. Like, do you think it's worth continuing or like just figuring out what's going on before you make it worse? And I think a logical immune reaction is like, yeah, we should probably figure it out before we go much further. I think the real, the practical reality is that there's so much money at stake from all sorts of different parties in the economy that I think that kind of level of collaboration and coordination is really difficult and is sort of like peak game theory, where maybe some participants support it because they are behind, other ones support it because it plays to their general agenda, whether their agenda is fair or not fair or seeking another goal, I don't know. But I think the practical things are like who decides it, how do you measure it, how do you implement it, when do you know when to stop or start. There's some proposals like sending external evaluators into labs to sort of measure this stuff.

That's a good one. Yeah. Or trying to somehow survey workloads that are running on data centers, maybe like, you know, put compute thresholds on certain workloads, but like that stuff has been tried in the past and it's like very difficult to monitor.

Yeah. So like this part of the report, like I don't have like the definitive answer, but I've tried to do my best to present the different perspectives on things and then let people conclude with their own data. But I find it's if we extend this towards like, you know, this NIMBYism debate which we predicted in the last year's report of, you know, most people would not want to, you know, live next to a data center, and then this would start to become a politically important component to...

Yeah, yeah, it seems so, where more and more states have tried to implement a moratorium or have done so, or establish new criteria by which grid connections can happen, like for example, you know, data centers would have to figure out and bring their own power on site, or maybe they would have to make certain investments in the communities that they want to build in.

Yeah. Well, there's projects I think worth about $50 billion that are currently stalled through a variety of these reasons. Like Texas, Pennsylvania have been ones that have generally paused new data center connections. There's a stat on polling of like American people where individuals would prefer to live next to a nuclear power plant than live next to a data center, which is quite wild to think of.

Yeah, I think it's kind of like everything, because you know there's like public narratives that data centers consume tons of water and that your water bill will go up. But realistically, like the data center is kind of more like a closed swimming pool. Like you fill it once and then the water is like constantly recycled in there. So it doesn't actually consume that much once it's filled. Then there's points around, hey, your energy bill will go up, but that's also kind of inconclusive because certain facilities consume grid, other ones are off-grid. There's others around noise. I think that's like probably uniformly true that these things are really noisy and droney and pretty horrible. There's others around like job creation or job loss. I think that's a little bit tough to read, but in general, generally speaking, we see a rise in like the need for like HVAC engineers, of construction staff, like operating heavy equipment and like actually doing the nuts and bolts of building the building.

Yeah. NIMBYism is not restricted to the United States. They had this also in Korea and South Korea, where, you know, it's a country that has seen like massive explosion of share price and wealth basically for employees of SK Hynix and Samsung, which, you know, create the high bandwidth memory the GPUs need for these mega-scale data centers. And, you know, the people who participated in this are very happy, right? Like your stock price went up and your options went up and you got a cash bonus, and that's driving increase in real estate prices in Seoul. And, you know, they want to create more data centers, but also people are not so happy with that for similar reasons than in the US. In Europe, there's, I guess, like I don't really see that problem as much because like there's just no compute capacity to speak of. Like we have some data in the report where like the total number or the estimated number of H100 equivalents in all like 27 European countries is like 80,000 cards, which is less than like every single one of the top five GPU clusters in the US. And, you know, while US hyperscalers are spending like 730 billion in 2026, like Europe committed like single-digit billions pretty much. And then energy costs are really bad if you're not Finland or Norway, where you're kind of at par with the US and with China, which has like the best energy price. Other European countries are just significantly more expensive, like the UK is one of the most expensive energy prices out of those developing economies, and there's no good capacity.

Yeah, and then most recently, I think there's a big deal with Microsoft and Nscale that got pulled, or at least Microsoft pulled out of it, because the data center provider couldn't get the energy connection in time.

I mean, it's tough. Like I think the fastest growing companies in Europe are ones that have US customers. I think it is really encouraging that in several modalities, you know, whether it's like image with Black Forest or audio with ElevenLabs or video with Synthesia or like, you know, code app generation with Lovable, like they're all market leaders and they're all like born out of Europe. It's not impossible, but I think those companies wouldn't be as big as they are if they didn't have US customers, and, you know, increasingly they have Asian customers also because like, for example, Korea and Japan are really leaning into AI. The challenge is like the gap between frontier and not is just so big and just keeps getting bigger and bigger. I don't know if these estimates are really like important or not, but like even if you conceptualize like starting a new lab from scratch to try to catch up with OpenAI and Anthropic, you'd probably be spending like a trillion dollars. I think it's kind of telling that even Mistral is like moving more towards being an inference provider and a cloud company than it is really like pushing frontier models. Although of course it has released new models and like they're kind of good, but they're still behind China. So, you know, whether it's them or other model companies that have built up GPU capacity to try to build frontier models are kind of going out of the race and serving inference, that sort of figured out you can approximate this move of what's happening, which is to say you either die trying to get to the frontier or you just live long enough to eventually serve inference. And you can see this with NeoLabs, with NeoClouds, with inference clouds, etc. It's like everybody wants to get into token flow because absent you having the model that is the best, like might as well serve other people's models. So I think to get back to the Europe topic, like that's probably the angle that would be most competitive because countries are going to need to ensure accession or access to US frontier systems, and they're going to need to have some kind of strategic leverage to bring to the table to prevent situations like what happened in June with the basic export control blocking of Fable and Mythos models from Anthropic to, you know, non-US nationals or non-US entities, which, you know, having been in London at the time, like people did voice a lot of issues about. And so like, okay, I don't want that to happen again, but what can I do to maintain strategic leverage? So the UK believes we should be investing and procuring inference chips because if we come up with like the best inference chip, it'll be a strategic advantage for labs to use it for cost reasons and other things, and with that I can barter for access to frontier US technology.

Yeah, yeah, you can enforce data residency requirements.

Yeah, it is a policy insofar as like the UK's AI minister, who's generally I think a really good individual to be leading this because he's clearly plugged into the mix and understands the raw ingredients that go into making AI successful, which is like compute, talent, and data. And so he has a mandate to be a buyer of first resort of new inference chips and has committed I think 100 or 150 million pounds, something along that angle, to procure inference chips when they are available from new UK companies that are designing these systems. You can see this also in South Korea where they have a similarish mandate, but it's actually much more ambitious, where every year they're spending about seven billion dollars on AI, and a large part of that is going towards building modular inference data centers because generally Korean companies want to have Korean data residency. So I think it's actually a pretty interesting quid pro quo where it's a captive market, a small country, but it's a wealthy country that's very technologically savvy and has an important role in the overall AI supply chain with memory, and so international companies want to at some point in their life be in the Korean market. Same with Japan, but getting in there is difficult. Korean data residency, Korean language. So it's like, okay, cool, like you want data residency, but now we offer you the data center to go run your technology in, and we can profit from that because it's our local companies that provide that service.

I just, it's a bit like national airlines. The countries will buy planes from Airbus and Boeing and then slap their logo on it. That doesn't make it like yours. It's still someone else's airplane, you know. So, but open source could be, if you believe that open source is going to remain within reasonable earshot of closed source models, maybe there's hope there.

It could be American, sort of a reflection, but they're not frontier. Like none of the US open ones, as far as I can tell, are frontier, and the ones that have all tried end up working for Jensen.

Okay. So I think his strategy is great. Like, I think different things. Like, do I believe that buying Nvidia GPUs gets you sovereignty? Like, no. Do I think it's a good commercial strategy for Nvidia? Absolutely.

Yeah, something like that. Yeah. And if you look at the national bids, of which like globally there's roughly like 138 billion in sovereign AI committed over the next couple of years, and you look at the most popular vendors amongst all of these, like it's by far Nvidia. I don't think there's much of an alternative. I think it's a good start, and you need to do it because it is the best technology, but I don't think one should kid themselves that doing so gets you sovereignty.

Again, I'm in a few different minds about this. One, I think it's worth trying, but do I think it's realistic? So I think it's important that if you are going to try, countries should try to launch systems which are going to do the white collar work. Like I think in the UK we still have a lot of capabilities to go do this, to try to build like an Anthropic competitor or like a DeepMind competitor or a DeepSeek or whatever. But actually if you look at the...

Yeah, you're right. Yeah, right. And I think part of the problem with actually like the UK AI ecosystem and its success in actually leading the world is that for too long the UK government kept saying that DeepMind is a British company, and I think it's only started to unwind that very, very recently.

Well, it's a problem because then it believes that it's already number one. Like the UK is the best place to do AI and we are the best at AI because we have DeepMind, so therefore we don't...

Yeah, yeah, because then in like, you know, in policy rooms, then you know individuals who are advocating for more investment in AI or we need more, you know, we need to procure more systems from AI companies or we need to make our country more appealing for AI companies to start get answered with like, but we have DeepMind, right? You know, right.

Yeah, yeah, I think it's, yeah, yeah. So, yeah, so that push really started, I mean obviously like post the Ukraine war like almost 5 years ago at this point, but more importantly I think it was like two years ago when JD Vance went to the Munich Security Conference and effectively said like you need to invest in your own defense and get serious about this, you being, you know, NATO and Europe. And so ever since then there's been, you know, a variety of different initiatives to give more capital to new defense companies, to encourage governments to be providing loans to businesses to improve their supply chain. And that's, you know, created a set of like neoprimers, like new defense companies that are trying to compete with prime contractors. So companies like Stark or Helsing, Quantum Systems being like the main ones in Europe. And then, you know, as like warfare has changed, like it's gone from, I don't know, you know, developing strike systems over the mid-range to, you know, hit tank, you know, battalions to now being like we're getting pummeled like every day with hundreds and hundreds of FPV drones or various other kinds of aerial systems that are autonomous and, you know, now can no longer be stopped with jamming because they use computer vision or they're guided by these like fiber optic cables, which by the way have gotten much more expensive because of data centers. And so we need more, you know, novel like air defense systems. And so that's where companies like Altar in France and others have really been pushing. And as you said, like the supply chain is super important because for example, like, you know, NATO wouldn't procure systems in Europe that don't have at least 70% kind of the parts from, you know, non-China basically. And countries are really queasy about taking US protection or US guarantees at face value because they're shifting on a weekly basis. So, so yeah, I think that's an example of a market which is sort of has to be domestic by design and can breed new companies that are much more competitive than their US peers.

Yeah, yeah, maybe Meta on margin.

Yeah, especially as now they're pushing towards like serving their models through API but also having an enterprise business. So that one, you know, that might change over the next like 6 to 12 months depending on how their go to market works. But yeah, I would say it's probably those three plus Meta.