François Chollet: Why Scaling Alone Isn’t Enough for AGI

Thanks so much for having me. I'm super excited to be here. Super exciting time to talk about AI.

Right. So NDEA is this new AGI research lab, and we are trying some very different ideas. And so our goal is basically to build this new branch of machine learning that will be much closer to optimal, unlike deep learning.

That's right. That's right. So what we're doing at NDEA is we're doing program synthesis research. And when I talk about program synthesis, often people ask me, oh, so are you doing like codegen? Are you building an alternative to coding agents? And that's actually not at all what we are doing. We are working at a much, much lower level than that. What we're actually doing is that we are trying to build a new branch of machine learning, an alternative to deep learning itself, rather than like coding agents. Coding agents are like this very, very high-level last-layer piece of the stack, and we're actually trying to rebuild the whole stack on top of different foundations. So we're building a new learning substrate that's very different from, you know, parametric learning, deep learning. So if you go back to the problem of machine learning, you have some input data, some target data, and you're trying to find a function that will map the inputs to the targets that will hopefully generalize to new inputs. And if you're doing deep learning, what you're doing is that you have this parametric curve that serves as your function, as your model, and you're trying to fit the parameters of the curve via gradient descent. And this is basically what we're doing, except we're replacing the parametric curve with a symbolic model that is meant to be as small as possible. It's like the simplest possible model to explain the data, to model what's going on. And of course, if you're doing that, you cannot apply gradient descent anymore. So we are building something that we call symbolic descent, which is like the symbolic space equivalent of gradient descent. The idea is to build this new machine learning engine that's giving you extremely concise symbolic models of the data you're feeding into it, and then we're going to make it scale. And so everything you're doing with machine learning today with parametric curves, we should be able to do it with symbolic models in the future in a way that will be much, much closer to optimality. Much closer to optimality in the sense that you're going to need much less data to obtain the models. The models are going to run much more efficiently at inference time because they're going to be so small. And because they're so small, they will also generalize much better and compose much better. You know, the minimum description length principle that the model of the data that is most likely to generalize is the shortest. And I think you cannot find a model like this if you're doing parametric learning. You need to try symbolic.

I mean, everybody is, you know, building on top of the LLM stack these days, which makes sense because, you know, the returns are there, like it's actually working. So it would seem very sensible for everybody to just be doing what seems to be the currently most productive path. But it's counterproductive to have everybody working on the same thing. Like I personally don't think that machine learning or AI in 50 years is still going to be built on this stack. I think this is a stack that is very promising. Maybe it even gets us to AGI, but it's not as efficient as it should be. I think it's inevitable that the world of AI will trend over time towards optimality, and so I'm trying to sort of like leapfrog directly to optimality, like to build the foundations of optimal AI today. But in general, you know, our vision is very ambitious, and I'm not saying that we're going to be successful. Like we have maybe a 10 or 15% chance of success, but that is enough that it's worth trying, right? And I think in general, like among listeners, if you have a big idea and it has very low chance of success, but if it works, it's going to be big, and no one else is going to be working on it, right? It's not something popular. It's not something if you don't do it, no one else will do it. And this is basically our situation. If you're in this situation, then you should try a chance, you know, should go and work on it.

Yeah.

The reason it's important is that again, if we don't do it, no one else will do it, right? So it's worth trying. Even if we don't succeed, it's worth trying.

Yeah, absolutely. I think it has surprised many people. It definitely did surprise me. If you look at why everything is starting to work so well with coding agents, it's really because code provides you with a verifiable reward signal. And I think right now we're in this situation where any problem where the solutions you propose can be formally verified and you can actually trust the reward signal, it's not just some guess made by a model. Any domain like this can be fully automated with current technology, with the LLM-based stack. And code is sort of like the first domain to fall, but there will be many others in the future. I think mathematics is also primed to see a revolution in the next few years for the same reasons, again, because the domain just gives you verifiable rewards.

Yeah. Yeah. Absolutely. I mean, writing essays is, you know, the typical example of a domain that's not verifiable. And so what you're going to see is that progress of reasoning models and baselines on this type of domain is, you know, is going to be very slow because the stack we're using, like the LLM stack, is very, very reliant on its training data. It's basically just operationalizing the training data. And for writing essays, the training data is coming from human experts like annotating answers, and that's costly. So you're going to see this very, very slow progress. Maybe it's even going to stall. But for any verifiable domain, like take code for instance, what was the big unlock is when people started creating this code-based training environment for post-training, where the reward signal, the verification signal is provided by things like unit tests and so on. And so that means that the model was not just working from human annotations. It was actually trying its own things, verifying the answer, and generating a lot more training data in the process. So a much denser coverage of the problem space, and not just coverage in terms of like is the answer right or wrong, but also starting to build models of the execution traces, right? So that the models could start incorporating an execution model, very much the way that human programmers, you know, when they look at code, they're sort of like executing the code in their minds, they keep track of variables and so on. Is also what the models are trying to do now, and this is why it's working so well. And it's possible because you're working with this very formal, fully verifiable environment. You cannot do that with essays. You cannot do that with, you know, law or many other problems.

Yeah. So my definition of general intelligence, you know, many people around the industry these days, they say AGI is going to be a system that can automate most economically valuable tasks. And to me, that definition is about automation. It's not about intelligence, not about general intelligence. So my definition is AGI is basically going to be a system that can approach any new problem, any new task, any new domain, and make sense of it, like model it, become competent at it, with the same degree of efficiency as a human could. So meaning it's going to need basically the same amount of training data and training compute as a human would, which is very little. Like humans are really, really data efficient. So general intelligence is human-level skill acquisition efficiency on the same scope of tasks that humans could potentially learn to do.

Absolutely. I think that's a trajectory that we're on right now. And I think it's already true that in principle, current technology can fully automate at human level or beyond any domain where you have verifiable rewards, right? And code being the first one. And I think figuring out AGI, figuring out like human-level learning efficiency over arbitrary tasks, that's probably going to take a different sort of technology, a different mindset, a different approach.

With enough compute, everything starts looking like everything else. Every computer is a great equalizer. Every approach starts looking the same. And I think it's possible in principle to build something that looks a lot like AGI on top of the LLM stack, but it's not going to be LLMs per se. It's going to be this new layer, perhaps, you know, it's going to be even a few layers above, not just one layer above, but a few layers above. But you can build it on top of LLM because LLMs are kind of computers, right? I see. I do believe, however, this would be the wrong thing to do because it would be very inefficient. I think AI research will have to trend towards not just efficiency, but in fact optimality over time. And for this reason, future AI in a few decades, it's not going to be this harness on top of a reasoning model on top of a baseline. It's going to be much, much lower than that.

I mean, I, you know, I've been doing deep learning for a very, very long time, and initially my mindset was that deep learning was going to be able to do everything.

That's right. That's right. I was trying to deploy models for natural language processing, in fact, in 2014. And from that work, you know, I actually started developing this open source library, which I released, in fact, exactly 11 years ago, March 2015. So it was Keras, and then it got popular, and then I'd end up sort of like doing less of the research that I started Keras for and more of working on the framework itself, just because it has really, really good product-market fit. And so my take, you know, around that time, around like 2015, 2016, was that deep learning was extremely general, that you could do everything with deep learning, that you didn't need anything else. It was Turing complete. So my take was basically that deep learning was differentiable programming. So anything you would do with software, you could in principle train a deep learning model on the right inputs and outputs to do the same thing. And in 2016, I was doing research at Google Brain on trying to train deep learning models to help with reasoning problems, and in particular, first-order logic problems, theorem proving, and so on. And I started finding that you could not really get gradient descent to encode sort of like reasoning-style algorithms. It was not because the models could not represent these algorithms. It was because gradient descent could not find them. Right? So the problem was that it wasn't about deep learning not being trainable or anything like that. Like that was not the problem. The problem was gradient descent. Gradient descent would not find generalizable programs. It would instead end up doing overfit pattern matching, right, over sequences of input tokens.

With a lot more data, and also I think models today, they're a lot more compressive of the data, which is why they generalize better.

That's right. That's where the idea came from. And I was like, you know, at the time, back in 2016, 2017, I was like, okay, we're going to need a benchmark to capture the ideas. We're going to need a program synthesis benchmark. And my mental model for that was ImageNet. I was like, oh, I'm going to make the ImageNet of reasoning. So I started brainstorming a few ideas around like 2017. I explored many different things. I tried working with in part cellular automata, like a setup where you show a model cellular automata outputs and it must recreate the program that generated them, like that sort of thing. And eventually I settled on the ARC-AGI format around like early 2018. You know, I was doing this on the side. It was a side project. Like my main project was developing Keras at Google. I wasn't moving very fast on that. So summer 2018, I wrote the ARC task editor, and then I started just making lots of tasks by hand. And about one year later, I had made 10,000 tasks. And so I wrote up the paper that was explaining what this was about, what the big idea was, like intelligence as skill acquisition efficiency. And I published all of that in 2019.

Yeah, people who worked on program synthesis knew about it. But a lot of people who worked on deep learning, on scaling up LLMs, they didn't really care for it. And part of the reason why is because LLMs did not work well or at all on the benchmark. But a benchmark to capture the attention of the research community, it needs to start working a little, right? If it's too hard, people are just going to dismiss it.

Yes.

Yeah. Absolutely. If you look at prior performance on ARC V1 first and then V2, so baselines were scoring extremely low on V1, like sub-10% basically. And I mean, it was true of the original like GPT-3 scoring zero, but that's even true of the latest baseline LLMs today, you know, as of March.

Without reasoning, yeah. So the base models, so performance of baseline LLMs on V1 stayed very, very low, even though in the meantime, you know, we had scaled up these models by 50,000x, right? So it was really telling you that, you know, more scaling up pre-training alone was not going to crack the benchmark. This was not enough to demonstrate that the model had fluid intelligence. And then the moment models started performing well on ARC-1 was with the first reasoning models, in part the OpenAI o1 and then o3 models, which by the way, they were demonstrated by OpenAI on ARC because it was the one unsaturated reasoning benchmark that was really showing that this model was different, that new capabilities that we had not seen before. And so with reasoning models, you start seeing this sudden like step function change on ARC-1. And so ARC-1 was really the benchmark that signaled that at this moment in time something was happening and something big. Yeah, something big. Like new capabilities were emerging. Like reasoning was new and different. And it was actually not obvious at the time. Like, you know, I don't know if you remember when o3 preview was announced by OpenAI.

Yeah, December 2024. And like, sure, it was like huge, like step function progress on ARC, but it was very expensive. We did not really have product-market fit effectively. But if you looked at ARC results, you knew that this was big and important. And then we released ARC-2, which was the same format but more difficult, like with more composition at the level of the reasoning chains. And what happened is that so the earliest reasoning models started very, very low on ARC-2, and then around the same time as coding agents started working, you saw this... Yeah, so very, very recent, just a few months ago, you saw this very, very fast saturation of ARC-2. And so again, like ARC-2 signaled that yes, there was this new set of capabilities emerging. I think the benchmark did a really good job at capturing the advent of reasoning models and then the advance of agentic coding, like this new paradigm where if you have verifiable rewards, then you can basically fully automate the domain, which by the way is true of ARC. Like ARC does provide a verifiable reward.

That's right. So not necessarily codegen per se, but the frontier labs have been targeting ARC-2, and the progress you saw on ARC-2 is actually a result of this very, very large-scale targeting. So what you can do to solve ARC-2 is you ask your reasoning model to make more tasks like those in the benchmark, and then you try to solve them using, let's say, program induction, for instance, still using your reasoning model. Then you verify the solution. Again, it's verifiable, so you can trust the answer. And then you fine-tune the model on the successful reasoning chains. And then you keep repeating. Like you generate new tasks, you solve them, you verify the solution, you fine-tune the model on the reasoning chains. And you can keep doing this millions of times, right? Like you just need to spend more money.

And the new paradigm in AI is basically that any domain where this is true, where you have the ability to join these true verification signals, you can run this kind of loop, right? If you can run this kind of loop, you can brute force mine effectively the entire space and get extremely high performance. This is basically the process through which ARC-2 was saturated. So what it tells you is that it's not so much that the models have higher fluid intelligence than they did with the first models. It's just that you have this new paradigm of post-training. And this is exactly what led to agentic coding. So it does matter. It is valuable. It is useful.

Yeah, absolutely. When it comes to competency, there's always a trade-off between intelligence and knowledge. If you have more knowledge, if you have better training, you need less intelligence to be competent. And that's exactly what happened with the rise of coding agents, right? The models don't have higher fluid intelligence per se. They don't have like a higher IQ, so to speak. It's just that they're way better trained, and they're way better trained in two ways. So they're not just trying to autocomplete code anymore. They're actually trained via trial and error in these post-training environments with, you know, true reward signals. And also they're trained to embed this model of code execution, right, where they learn to keep track of the value of variables over an execution cycle. And that's what's leading to this extremely strong product-market fit of agentic coding today. And it's really, it's completely changing software engineering.

Yeah. Yeah. There's a lot of progress right now. It's driven by custom harnesses around the task. And the harness is basically a way for the human programmer to input into the model like higher-level solution strategies, basically. I mean, to me, the fact that you need humans to engineer these harnesses is also a sign that we're short of AGI today. Because if we had AGI, you know, AI would just make its own harness. It would not need to be told how to solve a problem. It will just figure it out. But it is very effective. Like harnesses, I don't think they get us closer to AGI in any sense, but it's a very valuable area of research because that can lead to task automation at scale.

Yeah, absolutely. So if you look at V1, V2, it was really focusing on your ability to produce like causal models of a pattern that was just given to you, like the data was given to you. So it was static, it was passive, and really focused on modeling. And V3 is completely different. We are trying to measure agentic intelligence. So it's interactive, it's active, like the data is not provided to you. You must go get it. The idea is that your agent is dropped into a new environment, which is kind of like a mini video game. And it's not provided any instructions. It's not told what to do. It's not told what the goal even is or what the controls even are. And it must figure out everything on its own via trial and error. So we are not just measuring, you know, the AI's ability to model its environment. We're also looking at its exploration efficiency, its ability to acquire goals on its own, like goal setting, and of course its ability to plan through the model of the environment that's created and to execute the plan. And so together, you know, all of these abilities, we call that agentic intelligence. And we are looking for AI systems that could learn to play these games and, you know, crack them with the same degree of action efficiency as a human. If you look at the human, they are dropped into this new environment. They try a few things. They start understanding how things work. They can solve the environment, you know, in a few hundreds to thousands of actions. We're trying to look for AI systems that could match this efficiency. And by the way, we know that all of these test environments in ARC-3 are solvable by humans with no prior training because we actually tested them on regular people. Yeah. At first, you just see this screen and, you know, you have these keys available, but you don't know what they do, and you must figure out everything from scratch. And humans are really good at that, by the way. They're really good at exploring efficiently, at making sense of something new, and eventually cracking the game. And frontier models today, they're not very good at it.

Yeah, I mean, I'm pretty curious to see how frontier labs are going to react to V3 and how they're going to start to target it. It is designed to be more resistant to the same kind of harness strategy as what we saw for V2.

In particular, like of course you can try to just make more ARC-AGI-like games and then train your agents in them. But the thing is, we've deliberately tried to create a private set of environments that is significantly different from the public set. Like you can look at the public set, it's actually not giving you that much information about what's in the private set. In the private set you will have very different games with very different concepts, and also the public set is meant to be substantially easier. Performance on the public set is not actually representative of how well the system would perform. So for this reason it's going to be harder to target, and that makes it a better test of fluid intelligence as opposed to a test of how much effort you put into cracking it.

Yeah, we set up an entire video game studio, right, to create them. So we got over 250 games, and you know, they're pretty quick to play. Like each game takes you maybe 10 minutes or a bit less to play from scratch, like upon first contact. And we have like 250 plus. And we set up this very productive game studio where any given week we had multiple games in progress. We had this pipeline including, you know, design, implementation, review, human testing, and many, many iteration cycles to make sure that the game comes out right.

Right. We hired a team of game developers and we built our own game engine.