Today we are discussing why AI needs better benchmarks and what to think about the new ARC AGI 3. Welcome back to the AI Daily Brief. Today we are looking at the launch of ARC AGI 3. It's a new benchmark from ARC Prize that is specifically designed to test the interactive reasoning capability of AI agents. Now it is the latest in a sequence of benchmarks that are meant to deal with some of the problems of benchmarks. But to better understand what they are trying to respond to, it's worth going back and actually understanding what benchmarks' purpose is, what the problems with them are, and how people have tried to address those problems.
Benchmarks are effectively two things. They're a way to compare AI's performance in various areas, as well as a way to see how models are progressing over time. Historically, there have been two major categories of benchmarks that you see included with every new model release. The two categories are benchmarks around knowledge and benchmarks around function. Knowledge was the first big hill to climb with benchmarks like MMLU for general knowledge and GPQA which measures scientific knowledge. Over time, more difficult benchmarks were introduced like Humanity's Last Exam which features obscure knowledge not typically found in the training data.
As models developed, however, function became more important. SWE-bench is one of the best known of the functional benchmarks, testing the knowledge required to solve typical coding problems from GitHub. As agentic coding has risen in importance in the AI space, Terminal-Bench has arguably overtaken SWE-bench as the most important coding benchmark. Terminal-Bench tests not only coding reasoning, but also the model's ability to use a terminal. And interestingly, a lot of benchmarks have followed this pattern, starting off as a test of knowledge and then implicitly or explicitly also adding an element of testing functional capacity. Humanity's Last Exam, for example, began as a pure test of pre-trained knowledge, but now it's typically measured with web search tools enabled, making it a proxy for competency and tool use as well.
Now, very early on in the modern post-ChatGPT era of AI, benchmark saturation became a problem. All the way back in May of 2024 with the release of GPT-4o, all major models were already above 80% on MMLU with GPT-4o scoring 88.7%. Now, at the time, some other benchmarks were a little bit less saturated. 4o was a big breakout, for example, in GPQA, scoring 53.6%. But of course, with all of these benchmarks, it was only a matter of time. By last summer, the saturation problem had gotten much worse. At the time, o3 was OpenAI's daily driver. More difficult questions had been added to GPQA Diamond, and o3 still achieved 83.3% without using tools.
By that stage, most of the 2024 benchmarks had been abandoned or updated because of saturation. For example, the MATH benchmark was long gone, replaced by the AIME math test, which uses questions from a real-world math competition. o3 would score 88.9% on AIME math, foreshadowing that a specifically trained OpenAI model would achieve a gold medal performance at the International Math Olympiad a few months later. Fast forward to today, and once again, many of these benchmarks are getting saturated. GPT-5.4 is now up to 52.1% on Humanity's Last Exam with tools and 39.8% without, which is very close to Opus 4.6's 53% and 40% respectively.
SWE-bench was once again upgraded with GPT-5.4 scoring 57.7% on SWE-bench Pro. For Opus 4.6, Anthropic reported 81.4% on SWE-bench Verified but chose to highlight Terminal-Bench 2.0 more prominently where they scored a 65.4%. Now, it's difficult to keep track of all these numbers, but this chart shows the example of how performance on SWE-bench Verified progressed over the past year. Models from Anthropic, Google, OpenAI, and MiniMax, who produced the chart, are basically all up and to the right. They each began at different points in the middle of 2025, ranging from 55 to 70%, however, they've all arrived now at up near 80%.
Benchmark saturation then means that benchmarks no longer show particularly meaningful progress between each model generation. They also don't show meaningful differences between the models. And making this problem worse is the issue of benchmark maxing. Benchmark maxing refers to when a lab trains the model specifically to beat the benchmark, even if it has little relevance in the real world. This happens because the benchmarks are either completely known or semi-public, meaning model labs can train specifically for the test in order to have more impressive numbers when they come out.
One common perception and critique of Chinese labs is benchmark maxing in the extreme, which frequently leaves their models with a huge gap between their benchmark scores and real-world performance. In February, a variant coding benchmark called SWE-rebench was released containing a different set of problems and most of the Chinese models dived in the rankings, suggesting they were specifically trained against the narrow set of SWE-bench Verified problems. The Western models did drop as well, but not by nearly as much.
Another example was Meta with the release of Llama 4 Maverick last April. Meta was accused of testing multiple model variants on LMArena, which is a crowdsourced taste test platform for LLM performance. Platform users are presented with two samples and vote for the best one. Meta was accused of having tested models until they found the one that clicked most with users and launched as the second-ranked model on LMArena. You will recall that when people got their hands on Llama 4, they did not in almost any case think it was the second-best model available.
Between benchmark maxing and benchmark saturation, the net effect is the diminished significance of benchmarks as a tool for people to understand which models are good for and at what. Now, on top of all of that, there is just an inherent problem with traditional benchmarks. Most of these benchmarks tend to be narrowly focused on solving one particular type of task. Some are about recalling knowledge and some are about more complex skills, but they are focused on doing one thing within a very narrowly defined set.
We've talked in some episodes this week about the idea of task AGI, that at this point AI is really good at a huge array of knowledge work tasks, but where it struggles is in bringing tasks together. And in that light, it would be reasonable, I think, to say that while benchmarks might be good at demonstrating task AGI, they're not particularly useful in helping understand how AI does outside of that very narrow task. Math is a particularly good example of this with last year's models basically solving the very narrow field of competition mathematics. This was demonstrated in the IMO gold medal performances from OpenAI and Google. That is, of course, a completely different skill set than real-world mathematics.
To the extent the practical reality of deploying AI is understanding and dealing with its jagged frontiers, most traditional benchmarks just aren't all that helpful with that. Now, everything I'm discussing today are known long-standing problems, and there have been a ton of attempts to fix benchmarks over the years. One of the brute force methods is simply making the questions harder. We've seen this with SWE-bench and GPQA, which remained relevant deep into 2025 by simply changing the difficulty level. This gave the benchmarks at least a little more life and kept them relevant for hill-climbing performance, but it didn't really address the core underlying problem.
There are also benchmarks that were switched out for more practical tests. A key example here is the transition from SWE-bench to Terminal-Bench as a major coding benchmark. Terminal-Bench was intended to be a closer match to the way people actually use the models. It put models in a standard harness and tested their ability to use a terminal and other tools to solve coding problems. On some level, it was an improvement, but it still is dealing with saturation issues. And it also adds more complex variables. Particularly early on, for example, good coding models would fail tasks because they couldn't execute the tool calls properly.
Another approach has been trying to simulate real-world tasks. An early version of this idea was the SWE-Lancer benchmark developed by OpenAI last February. It tested coding ability against real-world tasks from Upwork that paid an aggregate of a million dollars. This allowed OpenAI to express their model's coding ability in dollar terms. The spiritual successor was GDPval released by OpenAI last September. It extended the real-world problem set beyond coding to encompass various types of white-collar work like making spreadsheets and slide decks.
One of the interesting quirks of GDPval was that it required the agent to build and deliver a polished work product. It quickly became clear that models were failing tasks not always because they couldn't do them but because the tool calls were failing. Now GDPval also has other challenges. For example, OpenAI went out and actually worked with experienced professionals to do a combination of AI and human review. Other evaluators like Artificial Analysis have gone and modified GDPval to be a strictly automated AI-only version and it remains one of the benchmarks that I think people are most interested in relative to all the others.
Now another major approach was looking at continuous agent performance with METR's long-task benchmark being the most well-known. This is that chart that we joked during a lot of last year as the bubble talk was increasing was effectively holding up the entire global market. The way that this test works involves giving models a set of coding problems that human coders could complete in a set interval of time ranging from a few minutes to several hours. The resulting chart has become one of the clearest demonstrations of model improvement. In the space of two years, we went from agents that could only complete tasks that take humans 5 minutes, in the case of GPT-4o, to agents that can complete tasks that take humans 10 hours in the case of Opus 4.6.
Now, the big problem with METR's test, and one that they've fully admitted, is that they're running out of tasks to test against. Their original task set included very few tasks that take more than a few hours. Now that agents can complete complex tasks that take 10 hours, METR is struggling to find a useful test set. Realistically, tasks that take human developers 10 hours aren't really tasks anymore. They're full-on software builds that introduce far more complexity into the test. In other words, METR can't really extend their benchmark without turning it into something fundamentally different. Meaning that this test even is effectively saturated.
Which brings us to ARC AGI. It began as the ARC Prize in the summer of 2024 based on former Google computer scientist François Chollet's approach to measuring machine intelligence. Introducing the prize, ARC wrote at the time, "AGI progress has stalled. New ideas are needed. Modern LLMs have shown to be great memorization engines. They are able to memorize high-dimensional patterns in their training data and apply those patterns into adjacent contexts. This is also how their apparent reasoning capability works. LLMs are not actually reasoning. Instead, they memorize reasoning patterns and apply those reasoning patterns into adjacent contexts. But they cannot generate new reasoning based on novel situations. More training data lets you buy performance on memorization-based benchmarks. But memorization alone is not general intelligence. General intelligence is the ability to efficiently acquire new skills."
ARC Prize's answer to this is a test that contains a series of abstract visual logic puzzles. The tasks are presented as a series of colored squares on a grid with squares added or removed according to a particular pattern. Two clues are given to teach the pattern. Then the task is to apply that pattern to a problem square. For example, the problem might require a yellow square to be placed next to a line of blue squares in various orientations. These are problems that are relatively easy for humans to solve, but proved to be difficult for LLMs. The tasks were also kept hidden, so the logic couldn't be trained into the models. Instead, the test was trying to measure an LLM's ability to learn new logic within context and apply it to a novel problem. Basically, it set out to be a pure test of reasoning ability rather than memorization of how to reason.
Early results were pretty compelling that this was a solid approach. At the time that ARC-AGI-1 was released, no models had come within 50% of human performance. Subsequent releases improved on this score, but the models seemed to be making genuine progress through reasoning. Then in December of 2024, OpenAI dropped a bombshell. A preview version of their o3 model had achieved a 76% score on low inference settings, exceeding the human score for the first time. On high settings, the score was 88%. The o3 model had been trained on the public dataset, but tested on a private dataset to achieve this score, so there was no risk the logic was trained into the model.
ARC wrote at the time, "This is a surprising and important step-function increase in AI capabilities," showing novel task adaptation ability never before seen in the GPT family models. At the same time, ARC announced that they would be updating their benchmark for 2025 with ARC-AGI-2. The new benchmark looked superficially similar to the first. It contained the same colored squares and was once again designed to be easy for humans and harder for LLMs. The key change was made to counteract the innovation that allowed the o-series models to outperform, which is test-time compute. It kind of seems quaint now, but at the time, the idea of making a model reason for longer was a paradigm-shifting innovation. With o3, OpenAI had extended test-time compute enough to maintain context between problems and learn iteratively throughout the test.
In order to pressure-test this approach, ARC added a new twist to the problems. Rather than simply adding a square according to the pattern, there were now three new styles of tests. Symbolic interpretation, where the LLM was tasked with interpreting more meaning within the symbols. For example, tasks where shapes needed to be colored differently according to how many holes they have. A second new set of tasks required applying multiple rules within the same problem set which they called compositional reasoning. And a final new set of tasks added context to the problem where logic was no longer universally applied but depended on context. For example, shapes with a red border need to be shifted to the right while shapes with a blue border need to be shifted to the left.
Again, all of these problems remain fairly simple for humans, but the additional complexities were designed to overload LLM context and test pure reasoning ability. The test held up well for most of 2025. Most model releases scored below 30%. At the very end of the year, and as this year got underway, things escalated dramatically. Gemini 3.1 Pro scored 77.1% at 96 cents per task in February. In March, Opus 4.6 achieved a 68.8% score. GPT-5.4 Pro achieved 83.3% and Gemini 3 Deep Think is the current leader at 84.6% and $13.62 per task. Basically, once again, as the benchmark got saturated, we needed something new.
Which gets us to ARC AGI 3. In an X post introducing the test on Wednesday, ARC writes, "Announcing ARC AGI 3, the only unsaturated agentic general intelligence benchmark in the world. Humans score 100%, AI less than 1%. This human-AI gap demonstrates we do not yet have AGI. Most benchmarks test what models already know. ARC AGI 3 tests how they learn." Now, the test is a complete rethink on the ARC AGI formula. The static grids of colored squares are gone. In their place, ARC has designed a series of 135 simple graphical games that require the LLM to manipulate the grid in real time. They have no instructions, so the model needs to explore the environment, figure out how it works, execute a plan, and adapt on the fly to what it sees.
In their early testing, ARC observed models failing by mistaking one game for another, carrying over theories between games, and failing to forecast cause and effect. ARC wrote, "ARC AGI 3 gives us a formal measure to compare human and AI skill acquisition efficiency. Humans don't brute force. They build mental models, test ideas, and refine quickly. How close AI is to that? Spoiler: not close. And unlike ARC-AGI-2, we are starting at ground zero. None of the frontier models can complete this test with any level of competency, each scoring less than 1%."
Google DeepMind's Xiaom shared one of Gemini's playbacks, which are all publicly available in the replay section of the ARC website. She wrote, "Poor Gemini straight thought it was playing Activision Tennis." Now, not everyone is a fan of how this is set up. LLM Scaling 01 writes the scoring of ARC-AGI-3 doesn't tell you how many levels the models completed but how efficiently they completed them compared to humans, actually using squared efficiency, meaning if a human took 10 steps to solve it and the model 100 steps, then the model gets a score of 1%. The implication, they write, is that this means scores are not comparable to the first two ARC tests.
On the other end of the spectrum, AI researcher Brandon Hancock commented on the elegance of the benchmark. He writes, "An alien species with zero knowledge of human language could ace ARC-AGI-3 on day one. I think that's beautiful. At a time when AI is dominated by language models, it's refreshing to have a frontier benchmark, the only one that I'm aware of, that requires zero language ability or cultural knowledge to solve. Intelligent does not mean speaks English or speaks Python. I'm reminded of classic first-encounter sci-fi story lines where intelligent species are able to communicate well before they hash out a common spoken or written language simply based on universal math, science, and reasoning concepts. AI has gotten complex enough that it behaves much more like an alien species than a next-token predictor at this point."
François Chollet, one of the creators of ARC-AGI, warned that this won't be the one benchmark to rule them all, commenting, "Keep in mind, ARC-AGI is not a final exam that you pass to claim AGI. The benchmarks target the residual gap between what's hard for AI and what's easy for humans. It's meant to be a tool to measure AGI progress and to drive researchers towards the most important open problems on the way to AGI." So, it's a moving target designed to track the frontier. As AI evolves, the benchmark evolves to spotlight the exact problems we haven't solved yet.
And I think maybe that's the big takeaway. The idea of trying to quote-unquote solve benchmark saturation is probably as simple as not assuming that benchmarks are going to last all that long. Just as we need innovation in the way that we build these models, we're going to need innovation in the way that we measure them. It'll be interesting to see how fast we have models that actually jump from one to some meaningful percent on ARC-AGI-3. But of course before long we'll need some other new thing to measure some other new capability. For now that is going to do it for today's AI Daily Brief. Appreciate you listening or watching as always and until next time, peace.
