He Co-Invented the Transformer. Now: Continuous Thought Machines [Llion Jones / Luke Darlow]

This podcast is supported by Cyber Fund. Hey folks, I'm Omar, product and design lead at Google DeepMind. We just launched a revamped vibe coding experience in AI Studio that lets you mix and match AI capabilities to turn your ideas into reality faster than ever. Just describe your app and Gemini will automatically wire up the right models and APIs for you. And if you need a spark, hit I'm feeling lucky and we'll help you get started. Head to a.studio/build to create your first app. Two for AI Labs is a research lab based in Zurich. They've got a team of amazing ML engineers and research scientists. They're doing some really cool stuff. If you look at their website, for example, you can see what their approach was for winning the ARC AGI 3 pub competition which closed out a few months ago. They are hiring amazing ML engineers and research scientists. They also care deeply about AI safety. So if any of that is a fit for you, please go to Two for AI Labs and give it a go.

The audience will know I'm a huge fan of Kenneth Stanley's ideas. His book, Why Greatness Cannot Be Planned, changed my life. It was absolutely insane. What he was speaking to is that we need to allow people to follow their own gradient of interest unfettered by objectives and committees. Because that is how we do epistemic foraging. When you have too many agendas involved in the mix, you kind of end up with a gray goo and you don't discover interesting novelty and diversity. I suppose that's basically the thesis of your company, Sakana, is to lean into those ideas.

And what are those processes that curtail freedom as a company matures? How would you describe that?

There's a phenomenon called audience capture, and I think there might be a phenomenon called technology capture. In the early days of Google, it was quite open-ended. Transformers is now the ubiquitous backbone of all AI technology, and it's a huge achievement that you're involved in. But there's a similar story with OpenAI. They're now starting to see all of these commercialization opportunities. They're going to become LinkedIn, an application platform, a search platform, a social network. I guess this could happen to you guys. There's a very strong chance, especially with your new paper, the Continuous Thought Machines, it could be a revolutionary technology, but then it will become obvious how it could be commercialized, and that's how those pressures come in.

Yeah. So we are a victim of our own success and this basin of attraction. There are so many basins of attraction. Sarah Hooker spoke about the hardware lottery, and this is a kind of architecture lottery. It actually made me think of the agricultural revolution, which is that this kind of phase change happened and all of the folks that had these skills that were so necessary, these diverse skills for living and surviving, they died out. That's actually quite paradoxical because we need those skills to take the next step. We're now in this regime. We've got the term foundation model, and the implication is that you can do anything with a foundation model. In the corporate world, we used to have data scientists, ML engineers doing these architectural tweaks even in midsize enterprise, and now we just have AI engineers who are just doing prompt engineering. So you're saying that the fundamental skills that we need to be diverse to think of new solutions and new architectures, they're dying out.

There's also this interesting thing that even in spite of great new research, I was speaking to Seb Hoger and he's got all of these new architectural ideas and OpenAI aren't implementing them. Google are doing this diffusion language model which is quite cool. I'd like to know your opinion on why that is. There's a few philosophies floating around like this concept of a universal representation, that there are universal patterns and the Transformer representations resemble those in the brain. It's rather led to this idea of, well, we don't need to use different architectures because if we just have more scale and more compute, then all roads lead to Rome. So why would we bother doing it any differently?

May I also submit that there could be an additional reason. I love that Fractured Entangled Representations paper. There's this shortcut learning problem, and I think that there's a little bit of a mirage going on here. There might be problems with these language models that we're not fully aware of. There's also this thing that we're seeing that we are starting to bastardize the architecture. We know we need to have adaptive computation for reasoning. We know we want things like uncertainty quantification. What we're doing is we're bolting these things on top rather than having an architecture which intrinsically does all of these things that we know we need.

You're touching on something fascinating there because we were talking about the need for adaptivity and adaptive computation. I'm really inspired by Randall Balestriero's spline theory of neural networks. We've had him on many times. You can look on the TensorFlow playground and see what happens when you have a ReLU network on this spiral manifold. You'd be forgiven for thinking that these things are basically a locality-sensitive hashing table because they partition the space and they can predict the spiral manifold. But we want to do something a little bit more different than that. It also comes into this impostor thing because just tracing the spiral manifold but not continuing the pattern, there's a big difference between that. From an impostor perspective, just tracing the pattern is not learning it abstractly or constructively. If we learned it constructively, you speak about this in your paper, this complexification, the abstract building blocks, and you can do adaptive computation. You understand the spiral. That means that with adaptive computation, you can continue the spiral and then you can update the model's weights so it has adaptivity because that's so important for intelligence. We know that we need models that can do these things. But for some reason, they're so sophisticated, they're almost better than an adaptive intelligent system because they tell us exactly what we want to hear. They seem so intelligent, but we know that they're missing these fundamental properties. I'm still fairly skeptical when I see video generation models. We went through a phase where you could detect them because of the number of fingers on somebody's hand. With more data, with more compute, with better training tricks, they submit. Now they usually do have five fingers. But did we fix the problem or did we just use more brute force to force the neural network to know it's five fingers, where something that actually had a much better kind of representation space? It's almost mad that it's controversial to say that we should represent a spiral like a spiral. Something that could do that generally, if it represented a human hand the way that maybe I represent a human hand, then maybe it would be much easier to count how many fingers are on a hand. It's unfortunate that they work so well. It's unfortunate that scaling works so well because it's too easy for people to just sweep these problems under the carpet. You guys have possibly created what I think might be the best paper of the year. This could actually be the innovation which takes us to the next step. You got the spotlight at ICLR as well this year. Congratulations on that. I think that's testament to how amazing this paper is. The CTM, the Continuous Thought Machine. It's actually not that far outside of the local minimum that we're stuck in. It's not as if we went and found this completely new technology. We took quite a simple biologically inspired idea of the fact that neurons synchronize, and not even necessarily in a biologically plausible way. Brains don't literally have all their neurons wired together in a way that they work out their synchronization. But it's the sort of research that I want to encourage people to do. The way to sell it is quite easy. I think at no point did we have to worry about being scooped. That stress was taken away from us completely. There was no pressure to rush out with this idea because we thought, well, there's probably somebody else working on exactly this. I think the reason that we were able to get a spotlight is because we were able to create such a polished paper. We took the time to do the science properly, to get the baselines that we wanted and do all the tasks that we wanted to try. Encouraging researchers to take a little bit more of a risk, to try these slightly more speculative long-term ideas, I think the sad thing is I don't think it's necessarily a very difficult thing to sell. I want to have the CTM as a poster child of it works. It was a bit of a risk. We didn't know if we were going to find something interesting, but it was our first shot and we did find something interesting and it became a successful paper.

If we do find a system which can acquire knowledge, design new architectures, do the open-ended type of science that you're speaking to, can you see a future where at some point the locus of progress will be mostly driven by the models themselves?

Could you introspect on that? Is it because you feel we need supervision because the models don't yet understand? There's this path dependence idea. We need to do supervision because we have the path dependence so we can guide the generation of the language models. Maybe in the future the language models will just understand better themselves. But there's also the output dimension which is that we want to produce artifacts that extend the phylogeny of human interest. We want it to be human relevant.

Oh, interesting. I wasn't aware of that. Yeah. So what to do when that day comes for AI scientists is a broader discussion. I think now is a good segue to talk about this paper in a little bit more detail. This Continuous Thought Machines, you were just pointing to it before. Luke, first of all, introduce yourself and set this thing up for us.

My name is Luke. I am a research scientist at Sakana AI. My primary sector of research is this Continuous Thought Machines. It took us somewhere in the region of about eight months working on this project with the whole team. I did a lot of the work, but we also had a lot of people in different areas and doing different parts of it. An eight-month life cycle for a paper seems a bit long for AI research at the moment. But to the actual technical points of the paper, we call it Continuous Thought Machines. It originally had a different name. We called it Asynchronous Thought Machines before, but every single time people asked us what's the asynchronous part, it became a bit confusing. So Continuous Thought Machines basically depends on three novelties. The first one is having what we call an internal thought dimension. This is not necessarily something new. It's related conceptually to the ideas of latent reasoning. It's essentially applying compute in a sequential dimension. When you start thinking about ideas and problems in this domain and in this framework, you start understanding that many problems that look intelligent or solutions to problems that look intelligent are often solutions that have a sequential nature. For instance, one of the primary tasks that we tested in the Continuous Thought Machines was this maze solving task. Solving mazes for deep learning is quite trivial. It's really easy to do if you make the task easy for machines. One of the ways to do this is you give an image of a maze to a neural network like a convolutional neural network and it outputs an image same size of the maze and it's zeros where there isn't a path and ones where there is a path. There's some really brilliant work showing how you can train these in a careful way and scale them up essentially indefinitely. This is fascinating and a really interesting idea of how to solve this. However, when you take that approach out of the picture and you ask what is a more human way to solve this problem, it becomes a sequential problem. You have to say go up, go right, go up, go left, whatever the case may be to trace a route from start to finish. When you constrain that simple problem space and you ask a machine learning system to solve it like that, it turns out to actually get much more challenging. So this became our hello world problem for the CTM. Applying an internal sequential thought dimension to this is how we went about solving this. Two other novelties that we can touch on and talk about. We sort of rethought the idea of what neurons should be. There is a lot of excellent research in cognitive neuroscience particularly exploring how neurons work in biological systems. Then we get on the other side of the scale how deep learning neurons work, which the quintessential example is a ReLU. It's off or on in a sense. This very high level abstraction of neurons in the brains feels a little bit myopic. So we approached this problem and said, let's on a neuron by neuron basis, let this neuron be a little model itself. This ended up doing a lot of interesting work on how to build dynamics in the system. The third novelty here is, as I said before, we have this internal dimension over which thinking happens. We ask the question, what is the representation for a biological system when it's thinking? Is it just the state of the neurons at any given time? Does that capture a thought, if I can be controversial and use the term thinking and thought? My philosophy with this is no, it doesn't. The concept of a thought is something that exists over time. So how do we capture that in engineering speak? We instead of measuring the states of the model that is recurrent, we measure how it synchronizes, how neurons synchronize in pairs along with other neurons. This opens up the door to a huge array of things that we can do with this type of representation.

You were talking about this sequential nature of reasoning. Devil's advocate, there was that Anthropic biology paper and they were talking about planning and thinking. They were saying that this thing is planning ahead. I think your system actually we can say does planning. It's actually different computationally. Can you explain that?

Yes, I think the boundary in terms of computation from a Turing machine perspective is really interesting. The notion of being able to write your tape, read from a tape, and then write again to be in a Turing complete system is obviously an incredible idea that has completely changed the world. I think the primary difference with Transformers versus what we're trying to do with the CTM is that the process that the CTM thinks in, we can apply that internal process to breaking down a problem. The problem itself can have a single solution and you could do that in one shot. As I explained with the maze, you could just process that in one shot. But there are certain phrasings of problems that are real problems that doing so becomes exponentially more challenging. In the maze task, a really good example is that if you try to predict 100, 200 steps down the path in one shot, no models that we could train, not even our model could do that. We needed to actually build an auto-curriculum system where the model first predicted the first step and then when it could predict the first step, then we started training it on the second and third and fourth step. The sort of resultant behavior of this is where it gets interesting. One of the ways that I like to do research and that I encourage people who work with me to do research is understand the behavior of a model. We're getting to a point now where the models that we build are demonstrably intelligent in ways that keep surprising us. Breaking that down into a single set of metrics or even a finite single metric about performance seems maybe not to be the right way to do it for me. Understanding the behavior and the actions that those models take when you put them in a system and train them in a certain way seems to reveal more about what's actually going on under the hood.

Very cool. I didn't pick up on this. So you're doing a fixed number of steps. You have like a context window. Did you say that you've set that around 100 steps?

For the maze task, the model always observes the full image at every step. The CTM will observe the full image. For argument's sake, those images could be tokens from a language, the output of a language model, those inputs could be numbers that the model has to sort. Whatever the case may be, it should be agnostic to data. That's how we've tried to build it. In the maze task, the model can continuously observe the data no matter where. It can look at the whole image simultaneously, but it uses attention to retrieve information from the data. It has, let's call it, 100 steps that it can think through. What we do is we pick up at some point the model solves three steps through the maze. So it says I'm going to go up, up, and right. Then it's correct. But then it makes the wrong turn. At that point, we stop supervision. We only train it to solve the fourth step, one more than what it could. In practice, we do it five, but the principle holds. When you do that, it's a self-bootstrapping mechanism. I think the intuitive listener will understand how that extends to other domains, other sequential domains like language prediction, many tokens ahead, that sort of thing.

I'm really interested in this idea of adaptive computation. The first question is how sensitive was the performance to the number of steps? The next question would be could you have an arbitrary number of steps, which means that perhaps based on uncertainty or some kind of criterion, you could do fewer steps? The final question is could you have potentially an arbitrary or unbounded number of steps?

Yeah, really super question. I'll answer the uncertainty question first about the sensitivity to steps. A very good example of this is we just trained the model on ImageNet classification. Our loss function is quite simple. What we do is we run it for, for example, 50 steps and we pick up two distinct points. The first one is where is it performing the best, i.e., where is the loss the lowest, and the second one is where is it most sure or where is it most certain. Those give us two indices between 0 and 49 inclusive. We apply cross entropy at both of those points. We just make the loss the average of the cross entropy at those points. What this does is it induces a behavior where easy examples are solved almost immediately in one or two steps, whereas more challenging examples will naturally take more thinking. It enables the model to use the full breadth of time that it has available to it in a natural fashion without having to force it to happen.

So you've decided to model every neuron as an MLP, which is really fascinating. Talk about that, but also there's this notion of synchronization. I think you use the inner product to determine the extent to which the parameters are synchronized, and this unfolds over time as the driving force. Can you explain that in a bit more detail?

Absolutely. I think it's a good point to explain the neuron level models as we call them in the paper, or NLM, first because it ties into this. You can imagine a recurrent system is a state vector that is being updated from step to step. We track that state vector and that state vector unfolds. For each individual neuron, each i-th neuron in the system, we have an unfolding time series. It's a continuous time series, well, it's discrete, but it's a continuous value. Those time series define what we call the activations over time. Synchronization is quite simply just measuring the dot product between two of these time series. So you have a system of d neurons and essentially you have d over 2 squared different synchronization pairs. Neuron one can be related to neuron two by how they synchronize, and neuron one can also be related to neuron three, etc. The neuron level models function by taking in a finite history like a buffer of neuron activations coming in. Instead of being just a ReLU activation, they use that history as information to process a single activation out, and that is what moves from what we call pre-activations to post-activations. The principle here is that this might seem rather arbitrary. Does it help for performance? Turns out it does, but that's not really the catch-all solution here. That's not what we're after. What we're after here is trying to do something biologically plausible. Find the line somewhere between biology, which is how the brain implements things in the biological substrate that we have, versus deep learning, which is highly parallelizable, super fast to learn, backpropable, all of the nice properties that have got us this far. Find a line somewhere where we can take some sprinkling of biological inspiration but still train it with deep learning. It turns out that neuron level models is a nice interim that we can do this with. The concept of synchronization is applied on top of the outputs of those neuron level models.

On the scaling, I think the time complexity is quadratic in respect of the dimension of the synchronization matrix. In your paper, you were talking about subsampling to improve the performance. How did that affect the stability? Were there any things that cost you doing that?

Yeah, it's a neat question. In terms of stability, what we found was kind of fun, and this was a sentiment that we had throughout the experiments that we ran with this paper. It tended, no matter what we tried it on, it just kind of worked with all spreads of hyperparameters. The problems that you have with backprop through time typically with recurrence models like RNNs and LSTMs, it's a challenge. You run for many internal ticks with the RNNs or the LSTMs and the learning seems to break down. But the fact that we use synchronization in some sense touches all of the neurons through all of the time, so it really helps with gradient propagation. A nice interesting point that's maybe a bit oblique to what you asked about synchronization is we have a system of d neurons and, like I said earlier, there are d over 2 squared possible combinations. This essentially means that our underlying state or underlying representation to the system is quite a lot larger than what you would get with just taking those d neurons. As to what that means in terms of downstream computation and performance and the things that we can do with this is what we're actively exploring right now.

You guys used an exponential decay rate. You have the system that unfolds over time. It would be maybe a little bit too constrained if the synchronization between any two neurons depended on the same time scale. For instance, there are neurons in your brain that are firing over very long time scales and very short time scales. The way that they fire together impacts other neurons and causes those neurons to fire. But everything in biological brains happens at diverse time scales. It's why we have different brain waves for different thinking states. Besides that point, what we do with the exponential decay in the Continuous Thought Machines is it allows us for a very sharp decay to say that these two neurons that are pairing together, what only really matters is how they fire together right now. But if we had a very long and slow decay, essentially that's capturing a global sense of how those neurons are firing over an extremely long period of time. This was essentially a way of us capturing this idea of how different neurons could maybe fire together very quickly and other neurons can fire together very slowly or not at all. This lets that representation space that I spoke about, that d over 2 squared representation space, lets it again become more rich, and we can enrich that space with more subtle tweaks to how we compute those representations.

We were speaking about this yesterday, Luke, that when folks apply Transformers to things like the ARC challenge or things that need reasoning, we need to do lots of domain specific hacks. The architects who were the winners of last year's challenge did depth-first search sampling. Some folks have been experimenting with using language representations or DSLs. Part of this is to do with the reachability of language. Language is quite dense, which means you can kind of monotonically increase. But if I understand correctly, your system might have some interesting properties for reasoning and for discrete and sparse domains, and also for sample efficiency because we want to build a system that can actually do well on things like the ARC challenge. Can you explain in simple terms why you think this architecture could be significantly better than Transformers for doing those things?

I think a lot of the really fascinating work in the last few years that I found fascinating in the literature of language models has been related to what one can actually call a new scaling dimension. I in some sense see chain of thought reasoning as a way of adding more compute to a system. That's obviously just one small part of what that really is and what that really means. But I think it's quite a profound breakthrough in some sense. Now what we're trying to do is have that reasoning component be entirely internal yet still running in some sort of sequential manner. I think that's rather important. You spoke earlier about Gemini's diffusion language modeling, and I think that there are a lot of different directions that are exploring this right now. I do think that the Continuous Thought Machine with the ideas of synchronization and multi-hierarchical temporal representations gives a certain flexibility on that space that other people are not yet exploring. The richness of that space, being able to project the next step to solve the ARC challenge and the next 100, the next 200 steps, to be able to break that down into a process that a model can then very quickly search that process in its high-dimensional latent space, becomes something that feels like a good approach to take.

Do you see any relationship between this architecture and Alex Graves' Neural Turing Machine?

Yes, that's really interesting. I do. I think that one of the most challenging parts about working with a Neural Turing Machine is the concept of writing to memory and reading to memory because it is a discrete action. That has its own challenges associated with it. I wouldn't go so far as to say that the Continuous Thought Machine is definitively Turing complete, but the notion of doing reasoning in a space that is latent and letting that space unfold in a way that is rich towards a different set of tasks. This actually brings me to a point that I find quite interesting that I'd like to share with you. Consider again the ImageNet task or any sort of classification task. It's a nice test bed. There are many images that are really easy and there are many images that are really difficult. When we train, for instance, a ViT or a CNN to do this task, it has to nest all of that reasoning in the same space. It has to put all of its decision-making process for a very simple obvious cat versus some complex weird underrepresented class in that dataset, and it has to nest it all in parallel in a way that we get to the last layer and then we classify. I think breaking that down where you have different points in time where you can say now I'm done, I can stop, versus now I'm done, I can stop, lets you take a dataset or take a task and actually naturally segment it into its easy to difficult components. I think we know that curriculum learning and learning in this continuous sense again seems to be a good idea. It's how humans learn. If we can get at that architecturally and just have that fall out in a model, again, this seems like something worth exploring.

I'm not sure if you know much about model calibration and how neural networks tend to be poorly calibrated. Oh, go for it, Tommy.

Yes. On that point, I think maybe the most exciting thing about your paper is, we were talking about path dependence and having this understanding which is built step by step, this process of complexification. Maybe this is apropos in the theme of world models in general and also active inference, and I say active inference in big quotes because it's not Karl Friston's active inference, maybe adaptive inference or something like that. We want to build agents that can continue to learn, that can update their parameters, and most importantly can construct path-dependent understanding. Because that's completely different to just understanding what the thing is. How you got there is very important. This architecture potentially allows these agents using this algorithm to explore trajectories in spaces, find the best trajectories, and actually construct an understanding which carves the world up by the joints.

Yeah, that's a really neat perspective. I haven't actually thought about it like that, but yes, I think that particular stance becomes really interesting when you think about ambiguous problems because carving the world up in one way is as performant as carving it up in another way. Perhaps the hallucination in language models is carving the world up in some fine way, but it's just not performant in our measure. This is hallucination, and actually that's not true, but in some other trace down the path of wanting to carve the world up through an autoregressive generation of tokens, you end up in a different carve up of that world. Being able to train a model that can be implicitly aware of the fact that it is actually carving up the world in a different way and explore those manners, those descents down the carve up, is something that we're after. I think it's quite an exciting approach to be trying to take a stance of let's break up this problem into small solvable parts and learn to do it like that, and how can we do this in a natural way without too many hacks.

Yeah, it's something I've been thinking about. As much as I love Chollet's measure of intelligence, for him adapting to novelty is getting the right answer, and the reason why you gave that answer is very, very important. In machine learning, we have this problem that we come up with this kind of cost function that rather leads to this shortcut problem. We could just build a symbolic system, we could be GOFAI, and we could say we need to do this principled kind of construction of knowledge maintaining semantics. Well, we're not doing that. We're doing a hybrid system. But there must be some natural way of doing reasoning where, in spite of the end objective being this cost function, because of the way that we traversed these open-ended spaces, we can actually have more confidence mechanistically that we're doing reasoning which is aligned to the world.

I think that's a great way of seeing this particular avenue of research. Obviously we're not the only people thinking like this and we're not the only ones trying to do this. What we have is an architecture that's amenable to it, and surprisingly so. It wasn't again the goal. It's not the goal to do this type of research. It's not the goal to be able to break the world down into these small chunks that we can actually reason over in a way that seems natural. Instead, what we did was pay respect to the brain, pay respect to nature, and say, well, if we build these inspired things, what actually happens? What different ways of approaching a problem emerge? Then when those different ways of approaching a problem emerge, what big philosophical and intelligence-based questions can we then start to ask? That's where we're at right now. It might feel at times, especially for me, too many questions and too few hands to answer those questions. But the fun and exciting thing and the encouraging thing that I can try to encourage other younger researchers out there is that you do what you're passionate about and figure out how to build the things that you care about and then see what that does. See what doors that opens up and see how to explore deeper into those domains.

We were talking about this yesterday, weren't we? That you can think of language as being a kind of maze. What is to stop us from taking this architecture and building the next generation language model with it?

That's honestly, as you know, something that I am actively trying to explore right now. I think the maze task gets really interesting when you add ambiguity to it, when there are many ways to solve the maze. Honestly, this isn't something I've tried yet, and maybe it's something I should try next week. But essentially, you can imagine an agent or the CTM in this case observing the maze and taking a trajectory. Surprisingly, we saw this. We have a section in our recently updated paper on arXiv, the final camera-ready version of this paper, where we added an extra supplementary section that is not in the main technical report. That supplementary section is basically, hey, we saw this cool stuff happen, and we list I think 14 different interesting things that happened while we were doing the research that obviously didn't make it into the paper, but we wanted people to know about these strange things that happened. This is one of the strange things. We watched during training what was happening. At some time during training, maybe halfway through the training run, we could see what the model would do. It would start going one path in the maze and then suddenly it would realize, oh no, I'm wrong, and would backtrack and then take another path. Eventually it gets really good and it does some sort of distributed learning in this because it's got an attention mechanism with multiple heads. So it can actually just figure out how to do this pretty well and refine its solution. But sometime early on in the learning, it descends multiple paths and comes back and backtracks. We have a really fascinating set of experiments that also showed, and this we actually have some supplementary material online showing this, and I don't really know what this says. It's kind of a deep philosophical thing. If you're trying to solve a maze but you don't have enough time, turns out that there's a faster algorithm to do it. This blew my mind when I saw it. If we constrain the amount of thinking time that the model has but still get it to try solve a long maze, instead of tracing out that maze, what it does is it quickly jumps ahead to approximately where it needs to be and it traces backwards and it fills in that path backwards, and then it jumps forward again, leapfrogs over the top and traces that section backwards, and then leapfrogs. It does this fascinating leapfrogging behavior that is based on the constraint of the system. Again, this is just an observation we made. What that means in a deep sense and how it's related to giving a model time to think versus not, and is it enough time to think, what happens, what different algorithms does the model learn when you constrain it in this way, I find that quite fascinating and an interesting thing to explore. Does it tell us something about how humans think? Does it tell us something about how we think under constrained settings versus open-ended settings? There are a number of cool questions you can ask on this front.

You guys are both huge fans of population methods and collective intelligence. Because we can scale this thing up and we can scale it out. What would it mean to scale this thing out not only in a kind of trivial parallelization but in terms of having some kind of weight sharing between parallel models? What would that give you potentially?

This is a fun area of research. One of the active things that we're trying to explore in our team is concepts of memory, long-term memory, and what this means for a system like this. An experiment that one can construct for instance is to put some agents in a maze and let them try solve this maze, not how we did it in the paper, but in a very constrained setting where an agent can only see maybe a 5x5 region around it. We give that agent some mechanism for saving and retrieving memories. The task, if you wish, is to solve that maze, find your way to the end. The model needs to learn how to construct memory such that it can get back to a point where it's seen before and know I did the wrong thing last time and go a different route. You can then see this with parallel agents in the same maze with a shared memory structure and see what actually happens when you can all access that memory structure and have a shared global, almost like a cultural memory that we can access and solve this global task by having many agents trying to use this memory system. I do think that memory is going to be a very key element to what we need to do in the future for AI in general.

The subject of reasoning came up just a second ago. I think there's a perception that recently we made a lot of progress in reasoning. It's actually one of the main things that I think people are working on. We released a dataset recently called Sudoku Bench. I was actually quite happy to see it come up organically on your podcast a few weeks ago with Chris Moore. I wanted to tell you a little bit about this benchmark because I think I've been having a little bit of issue promoting it because it doesn't on the surface sound particularly interesting. Sudoku has a sort of feeling that it's already been solved. How interesting can a collection of Sudokus be for reasoning? We're not talking about normal Sudokus. We're talking about variant Sudokus. What variant Sudokus are are usually normal Sudokus, put the numbers one to nine in the row, the column, and the box, but then literally any additional rules on top of that. They're all handcrafted. They all have extremely different constraints. Constraints that actually require very strong natural language understanding. For example, there's one puzzle in the dataset where it tells you the constraints of the puzzle in natural language and then says, oh, by the way, one of the numbers in that description is wrong. You have to be able to meta reason about the rules themselves even before you start solving the puzzle. There are other puzzles where you have a maze overlaid on the Sudoku and the rat has to work out a way through the maze by following a path to the cheese. But then there are constraints on the path that it takes of what numbers and what they can be add up to. It's difficult to really describe how varied these variant Sudokus are. I think they're so varied that if anyone was actually able to beat our benchmark, they would necessarily have to have created an extremely powerful reasoning system. Right now, the best models get around 15%, but only on the very simplest and the very smallest Sudoku puzzles in the set. We're going to be putting out a blog post about GPT-5's performance, and it is a jump, but it's still completely unable to solve puzzles which humans can solve. What I really like about this dataset, and actually was the catalyst for me creating it in the first place, was that there was a quote by Andrej Karpathy saying, okay, so we have all this data from the internet, but what you really want if you wanted AGI, you wouldn't want all of the text that humans have ever created. You would actually want the thought traces in their head as they were creating the text. If you could actually learn from that, then you would get something really powerful. I thought to myself, well, that data must exist somewhere. My first thought was maybe philosophy, like there's a type of philosophy where you just write down your thoughts without thinking, like stream of consciousness. I thought maybe that could work. But then when I wasn't thinking about it and I was in my leisure time, I was watching a YouTube channel called Cracking the Cryptic, where these two British gentlemen will solve these extremely difficult Sudoku puzzles for you. Sometimes their videos are four hours long and they're professionals, this is their job. What was perfect, I realized, is they tell you in agonizing detail exactly what reasoning they used to solve those particular puzzles. We, with their permission, took all of their videos, which represents thousands of hours of very high quality human reasoning, like thought traces, and scraped them and made that available for imitation learning. We did try to do this internally. Turns out that I did a little bit too much of a good job of really creating a very difficult benchmark. We're still trying to get that stuff working and we'll publish it if we have some success. I want to really sell the fact that this reasoning benchmark really is different. Not only do you get something that's super grounded, you know exactly if it's right or wrong, so you can do RL to your heart's content, but you can't generalize very easily. Each puzzle is deliberately designed by hand to have a new and unique twist on the rules called a break-in that you have to understand. Right now, despite all the progress we've made, the current AI models can't take that leap. They can't find these break-ins. They'll fall back to, okay, I'll try five, I'll try six, I'll try seven. The reasoning becomes really boring and nothing like what you see in the transcripts that we've open-sourced from this YouTube channel. I just want to put the challenge out there that this is a really difficult benchmark and I think progress on this benchmark will really mean progress in AI generally.

Could you reflect a bit? After watching this Cracking the Cryptic YouTube channel, how diverse were the patterns? Chris was saying to me, oh, these guys go on Discord servers, they get these creative crazy ideas, and I'm obsessed. Maybe I'm just being idealistic, but I love this idea of there being a deductive closure of knowledge. That there's this big tree of reasoning and we're all in possession of different parts of the tree to different depths. The smarter and the more knowledgeable you are, the deeper down the tree you go. In this idealized form, there is one tree and all knowledge kind of originates or emanates from these abstract principles. We could in principle build reasoning engines that could just reason from first principles, and it might be computationally irreducible, so you have to perform all of the steps. It feels like because we're not in possession of the full tree, what we need to do is kind of fish around. We fish around to find Lego blocks. Oh, that's a good Lego block. I can apply that to this problem. Maybe that's just what we need to do in AI for the time being, is we need to just acquire as much of the tree as possible. But could we just do it all the way down?

Guys, it's been an absolute honor having you on the show. Just before we go, are you hiring? Because we've got a great audience of ML engineers and scientists, and I think working for Sakana would be the dream job.

It might be the opportunity of a lifetime, folks. So get in touch, guys. Seriously, thank you so much. It's been an honor having you both on the show.

Thank you so much. It's been great.