Today, I'm chatting with three of my AI researcher friends from whom I learn a lot every time we talk. They also happen to be at somewhat open-ish labs and companies, so you guys can actually say things on the record. I'm joined by Beren Millidge, who is the CTO of Zyphra, which is developing open source models. John Schulman is the chief scientist at Thinking Machines, previously a co-founder of OpenAI, and led the RLHF work that led to ChatGPT. And Charlie O'Neill is head of model training at Baseten. The first question I have: If we're in 2036 and we don't have billions of crazy superintelligences running around that have radically transformed the world, what is the most likely reason that doesn't end up being the case? Other than exogenous political shocks, or there's a war, or they ban AI or something. What is the most likely technical reason that 2036 isn't a crazy alien superintelligence world?
There's been a classic thing, almost like Moravec's paradox, where we think of the AI as, "If it can do this, it's going to be amazing." If it can solve these hard maths problems, if it can win at chess, blah, blah, blah… Then it solves these things, and it's not that impactful. Obviously, it's somewhat impactful, but not everything. If somehow that continues, and there's never the true spark of generalization that occurs, I think that could lead to the AI just being extremely good at everything that people put into a benchmark or put into an environment. But there's still some persistent sim-to-real gap which is somehow blocking everything. I think this is unlikely. We do actually see this kind of generalization even from RL in practice already. But if it is just ridiculously hard to generalize meta-learning, plus we don't solve continual learning and it's just super hard and impossible… This would be my default scenario in that case.
I agree with that. Humans have a lot of advantages over models now. Each time a new model comes out, it'll catch up in some of these areas. But you end up getting bottlenecked by the places where the model is weaker and where it has worse judgment, or the models can't check themselves well enough. There's this cycle that keeps repeating where a new model comes out and people are blown away and they're like, "This is it. This is AGI." But then they use it a bit, and it starts to feel dumb after a month or so. That cycle just might keep going. It's hard to predict how many times it's going to repeat. Right now, you don't get explosive growth in capabilities because you still get bottlenecked enough when you're trying to do research and engineering. Even if the model can write way more code than a person, it doesn't make you 100X more productive. So maybe there are just more of these cycles than we would expect.
For me, it's a question of how far off the global optimum of "a learner you could have on a chip" is from the transformer + RL, basically the current recipe. People imagine that once you have an agent which is better than all humans at AI research, even if it's 0.1% better than all humans, then the fact that you can run hundreds of thousands, if not millions, of these in parallel — and you can run them much faster as chips speed up — is going to outweigh every other bottleneck. You're eventually going to hit this very fast takeoff with regards to self-improvement. I could imagine that if we continue along the trajectory that we're currently on with that paradigm, where it's basically self-attention, RL, scaling up RL environments… Think about what happened with Moore's law. We had this very nice straight line and that held for a really, really long time. But there were so many discrete discontinuities and innovations that had to happen to keep that scaling law going. The same thing has happened with LLMs. We had this pre-training scaling law, and then that was hitting diminishing returns. Then we came up with RL and solved that, and then we got this new diminishing returns curve to hit that made it keep looking like a straight line going up. So if it requires another one of those discontinuities to solve, I'm not sure that the current method of training LLMs with these RL environments, even RSI-targeted RL environments, would be able to discover that discontinuity. If not, we're probably going to hit this asymptotic curve.
But do you think the discontinuity will be harder than anything that's come since 2012?
If we had the answer to that, we'd kind of have the ability to implement it. But maybe we should distinguish between a discontinuity which adds to the current paradigm, which is cumulative — there's something beyond the RL that we have to discover, and maybe they're capable of connecting the dots in that straight line — or, again, how far off the global optimum are we? Do we have to go back and throw out gradient descent and neural nets in general? I don't think, if you continue to scale up the current paradigm, an LLM, no matter how many LLMs you're running, is necessarily capable of discovering that if it's too far away. The only hope really is if deep learning just can't get us to an AI which can at least dominate human research and human development, including the human ability to come up with new paradigms and so forth. Or, I don't know, maybe humans would also never have discovered the next learning architecture. But to the extent humans could have discovered it eventually… But it just seems like… If you just look at the progress that's happened since 2012 till now, and you just continue that on — I know it's just been powered by huge amounts of compute scaling and so forth — it would be weird if it just didn't get to the point where it could dominate humans, at least in R&D, especially over the next few years.
Ryan Greenblatt was on the podcast recently. He made this point that I'd be curious to get your thoughts on. You could imagine, as AIs get more and more capable, that they're capable of making progress on simulations which incentivize getting better at not only AI R&D, but at science generally. This is a thing that all the labs are targeting and many startups are targeting. Another intuition pump is if you look at the Elo score of chess bots since the '80s. There's a very linear increase in Elo over time. But there's this huge discontinuity as they cross the human range, from human experts always winning against AIs to human experts never winning against AIs, as this linear increase in Elo happens.
I agree with your point that so far, AI capabilities have not been that big of a deal in terms of their end economic impact in the world. But that is because they're slowly rising in Elo relative to humans. I agree it would be very surprising. The only way for this to not happen is if, as you said, it somehow asymptotes just before. Because we're already pretty close, in my opinion, to where we'll start crossing the human Elo score. So we'll need to asymptote before that. That's the only way — in this scenario you pose where somehow we're sitting here in 2035 and everything is normal — for this to happen, I think. The only other way is there's some dramatic regulation on AI. This is what I see as the most likely way for this scenario to happen, actually, rather than a technical thing.
I think there's different kinds of research. There's research in the autoresearch style where the objective is already specified very cleanly and you're optimizing that objective. I think everyone is picturing that if we continue along this path of making pre-training loss go down and making our environments have the reward on them go up, that's going to lead to improvement. But maybe what Ryan is talking about is this much more open-ended type of science which is required for paradigm shifts, where we can't specify the objective, and the AIs are definitely not able to specify that objective either. We have to be really, really careful about how we specify objectives for any of these things.
Maybe your point is that the nature of the breakthroughs that have happened since 2012 is that we have found… In 2012, people weren't saying… I'm assuming, I don't know, you guys were there. Or at least John, you were there. But I was not. I was in primary school. Actually, John, I'm curious for your wisdom of the ages, or wisdom of being in the trenches way back when. Presumably, a big breakthrough was realizing that next token prediction is the… You wouldn't have thought that the nanoGPT speed run is the thing to be optimizing for in 2014. But now that we have come to this new paradigm, you would think to do a speed run on that and have AIs get really good at that. But maybe there's a next inner loop to optimize that the AIs wouldn't anticipate. There's an outer loop of revenue or something that eventually should be strong, but it's a very slow outer loop.
In fact, I remember in the early OpenAI days having the intuition that just minimizing log loss wasn't going to get you to intelligence. Because the important bits are accounting for such a small fraction of the loss that it was going to be overwhelmed by noise. So just training a language model on next token prediction wasn't going to learn the interesting things you want it to learn. We needed to craft better objectives that would put more emphasis on the important things. You can make all sorts of arguments for this. You could say, "Oh, humans probably don't learn how to model everything in our environment. Most people can't create a photorealistic reproduction of some kind of scene they've looked at. So we must need a better objective." But then it turned out that it just worked anyway. As you were pointing out, the inner loop, even in current AI research, of post-training benchmarks or whatever, doesn't necessarily translate into what users like.
Oh, yeah. The whole field relies a lot on generalization and it's very hard to predict when you're going to get generalization, or when you're going to get some kind of out-of-distribution generalization. We know that if you train on the task you care about, you're going to do better. But the most important advances are often types of generalization that we have no right to expect. For example, from just pre-training on this very naive next-token-prediction objective to various tasks of interest that require understanding of the input in some deep way, or learning some skill from pre-training that's very rare and not heavily represented. Then also generalization from these verifiable tasks to less verifiable ones, this is also a type of generalization that there's no reason a priori to expect.
This is an interesting question, because one intuition pump you could have for why you would see some sort of singularity very rapidly — without even scaling up the inputs to AI progress that are not just AI labor — is that before every single 7-figure experiment you run, you spend an equivalent amount of compute on AI labor. So you just have automated versions of you guys spending a century thinking about what is the optimal experiment to run, doing small-scale ablations, developing literally a century's worth of theory, going back even before deep learning. Before you decide what experiment to run, you're doing extremely optimal setting up of the experiment. Then you do a century of thinking after the experiment is over, where you're analyzing what happened and what the next experiment to run is.
If you think hard enough, you probably could have expected some of these things beforehand. There is probably some very clever way to do a small-scale experiment that'll let you build the theory that then will generalize to the large-scale experiment. So I would expect that we're nowhere near the ceiling of how well you can do research. I would imagine a future where AI is doing a lot of analysis and theory building, spending a comparable amount of compute to the amount that you're spending on the experiments themselves, doing various kinds of analysis and building a theory around what we've seen so far.
I think there are really concrete examples of this when the objective is well specified. All thinking can do is update your posterior based on the bits that you've gotten since you formed your prior. You can't gain any new bits from just thinking. But when the objective is well specified and there is this data sitting around, I imagine there will be this big speed-up in the current paradigm we're in. A good example of this is if you got an AI to think about the Kaplan scaling laws. An AI at this point would have noticed, "Oh, they've just taken these intermediate checkpoints and didn't account for the annealing, and so this is wrong." That would have been caught years earlier. We would have cut off a year or two of progress just from that observation from an AI. Again, once the objective is well specified, which is lower pre-training loss or whatever, there are many, many good examples where if you just thought about it a bit more, you would have been able to cut down significantly on things that you've done. So muP, and how learning rate scales with model size, and realizing that model width is important in that as well. I feel like you can really back out a lot of these things and cut off a lot of low-hanging fruit. I would imagine a 10x speed-up if our thing is just, "Maximize the objective we're currently on." But I don't see how that generalizes at all to coming up with the right objective in the first place. Just thinking doesn't necessarily buy you the right objective in the first place.
I think this is really the key question for any kind of very rapid RSI from current AIs. How well can AIs generalize to learning their own objectives? To have any kind of self-propelling automated loop, we need the AI to propose objectives, optimize them, figure that out, propose a new objective, and have this not go off the rails at any point for a long, long time. To come back to Moravec's paradox, there might be a case of Moravec's paradox where we think this kind of autonomy and being self-encapsulated — so we can think of what we should do ourselves and then go do it and have this loop — is super easy because we always do this. Obviously, evolution needs to create creatures that can survive by themselves for long periods of time. And this just might be something that for some reason is really hard for the AI, in the same way that locomotion stuff is really hard but math is super easy despite being super hard for us.
But doesn't the time horizon increasing suggest that that's—
Yeah, exactly. This is another possibility, but I agree, there's no obvious evidence for this. In fact, the fact that our agents are now super persistent and it's quite easy to do this is kind of evidence against this. But this would potentially be one of the reasons why we just don't get this immediate takeoff, if this is hard.
If you look back from 2012 till now — or maybe from when you started doing your research till now — what part of all the innovations that have happened since that time, including purely engineering ones, including purely conceptual ones, seems like the thing that would be the last thing humans would have to do before AI totally automates AI R&D?
Probably just iteratively asking the right questions. If you can get the AI to do any experiment, you still need to decide what experiments to do. Right now I think AIs are not very good at this compared to coding the experiment. Whenever we talk about research, they propose a bunch of miscellaneous things which are very, very tiny steps. Or even going from DeepMind's approach of, "We're going to solve intelligence by learning to play games at a superhuman level," to one random researcher like Radford being, "I'm going to try and just predict the next token of a very wide swath of data"… Even once Radford had discovered that, it took a while before people decided to scale it up, because we had to come up with the idea of scaling laws and the fact that you could very reliably predict these things.
I would say that the last job for humans, or the role for humans that'll last the longest, is defining the objective and deciding what we actually want. In that vein, something like deciding how the AI assistants should behave, or what it means to be helpful, or what the objective is when we're doing RL from human feedback, is one such thing. Then later, defining constitutions and model specs is another one. Even if the AIs can do all the technical work, we'll still have to do a lot of that and decide what we actually want. Alignment is the final job.
Alignment is sort of the answer. But alignment itself can be decomposed into specification of the objective, or figuring out what the right objective should be, and then actually achieving or optimizing the objective you've defined. I think the first one is not going to go away anytime soon. If I think about a post-training team and why you need a lot of people on the team, it's just because there are a lot of different areas where you have to figure out how the model should behave. It would be very hard to automate the whole thing, just because someone has to think about how the model should behave in this area.
Jane Street started using Antithesis to test its software in early 2025, and the team was so impressed by the product that it decided to invest in the company. I recently caught up with Ron Minsky, who co-leads Jane Street's tech group, to ask how Antithesis actually plugs in. The thing that I think is most impressive about Antithesis is that we started using it on a team that was building high-assurance software and being really careful. Nonetheless, it was able to shake out bugs that were otherwise going to be really hard to find. That's important both because it helps make those systems more reliable and because it helps the teams that build them move faster. This matters more and more as code production is increasingly automated. I think, in general, as we've been using agents more and more, the key problem you run into is the verification bottleneck: just the time it takes for people to look at code and figure out, is that actually something you want to accept into your production software? Tools that make testing better are just incredibly helpful there. They ease the verification bottleneck and make it possible for you to get more stuff done and move faster, because you can have more confidence that the code generated by the agent isn't introducing new problems. To see how Antithesis fits into your development process, go to antithesis.com/dwarkesh.
What is the story for why there isn't huge consolidation in model providers? There are just so many things that point to centralization here. If you step back over the course of years, is there something that is going to prevent that?
I think distillation is the main thing that fights against the centralizing force. Basically anything that can be learned through RL can be distilled very easily, because it's a small number of bits. It's something that you can learn from a small amount of data. If you can get trajectories from the model that show a behavior, you can easily distill it. I think distillation is one of the things that fights centralization. There is also a possibility that there'll be company-specific models, that it'll be possible to learn from deployment and have a company continually improving its own model. Such a system could be provided by the current oligopoly of model providers or some other currently smaller company. But I think that'll change the game a bit. I also want to point out that continual learning, honestly, doesn't stop distillation. Even if your model is improving every day, people could be distilling it every day. The loops could just operate at the same pace.
That makes sense. So copying model behavior… I guess you need to know yourself what the right distribution to prompt is in order to get the relevant model behavior?
Oh, yeah. For just distilling with supervised learning, the prompt distribution is extremely important. It's very non-trivial to distill a model, even if you have full access to it and have the chain of thought and everything. It's non-trivial to distill all of the useful capabilities from it, because you need to prompt the model with something. You need to prompt it with realistic prompts. You need to have a really wide distribution of realistic prompts. One thing that's been coming out recently is that some of the Chinese companies are probably using these router services which are designed to allow people in China to use the US frontier models, which would otherwise be blocked in China. There are all these router or proxy services that allow people in China to use these models, mostly for coding. And these router services are collecting and selling some of the data. This is a very useful data set for distillation because it gives you the perfect prompt distribution.
I think this is one of those things where AIs help a lot. If you actually look at the frontier pipelines, or the Chinese models that they've actually put in their papers, they get seed prompts from somewhere, which is some combination of humans and this kind of data. Then they synthesize a vast coverage from those seed prompts using their existing models or the other frontier models. You can automate an awful lot of this prompt distribution gathering and environment creation. Humans need to provide increasingly fewer bits as the models get better.
But it still seems you're bottlenecked by having a service which has users going through it.
Not necessarily. That's obviously very helpful, but theoretically, you can just think about what users want. But the whole point is that the user says, "Make me an application like this. Oh, that didn't work. I actually want you to make this new feature. But actually, let's step back and do this other thing." Capturing that whole trace is the thing. Or to the extent you could have done that anyway, then you just have RSI. Ultimately, if you have this fully automated loop, that is basically RSI. The AI is deciding the data, it's deciding the training. That is the loop. But it depends how much human information you need. At some point, if you're just like, "I want traces that look like this," you prompt that to the model. The model will be able to come up with a pretty good approximation. But what if you want to do, "Make me a really good politician," and then it has to anticipate de novo how a discussion in the Senate halls would go or something? I just feel like there are going to be a lot of things which are— Ironically, this is actually easier for the distillers than the frontier labs. The distiller's just like, "I want a good politician." They go to the frontier model. The frontier model already knows how to be a good politician, so it just generates those traces. Whereas if you actually want to build the first model that does this, you have to actually somehow get data on what politicians do every day and build that. It's actually much easier to say, "I want something like this," and then get the AI to produce a billion variations, than to actually create the thing like this to begin with.
I think you can actually make a really concrete prediction based off this observation that the Chinese labs have this router data. The thing that started this originally was I was saying, "Isn't it weird how Sonnet 5 and Opus 5 are almost objectively worse models than GLM-5.3 and Kimi K3, even though they've had access to not only distillation but logit distillation from Mythos?" The counter was that the prompt distribution really, really matters. You need to see what users are doing so that you can distill these behaviors and things in. I think the prediction from this is that the frontier labs don't necessarily have much of an advantage, if at all, in RL environments now. Yes, user distribution matters for general behavior and so on, but the best measure of a capability is the very, very hard RL environments you've made at the frontier. If you have access to those RL environments as Anthropic, and you have access to logit distillation, and you've still made a worse model, then maybe—
Then real-world deployment matters more than the environment. That's really interesting. But they had to incentivize those capabilities in the first place in Fable, or the frontier model. So it's weird that they can't incentivize them again with a smaller model or something. Maybe we're just in this weird uncanny valley where trying to copy that frontier model too much, the student-teacher gap, whatever it is, is just too large. People have made this point with Opus. The difference between Opus 4.6 and Opus 5 is that Opus 5 really feels like it's got this AI-as-a-judge checking every possible thing it's done. That's why it uses so many tokens. It tries to think about all these things, but it doesn't necessarily have the big model smell of Fable to know when to stop doing that, or when's a good path to go down. The reach exceeds the grasp.
I would offer a slightly different hypothesis. I would say there are a couple of different axes for the environments you can create. One of them is difficulty and the other is realism. It's comparatively easy to create a lot of difficult environments that involve doing a much more complicated task or doing something that requires a lot more cleverness. You could say this is the benchmaxxing distribution, because a lot of the most prominent benchmarks just involve doing some very hard puzzle-like task that's easy to verify. Then there's the realism axis, where you want the model to be good in the realistic coding agent setting where there's multiple back-and-forths with the human and there's multiple objectives. The labs who are crafting the model behavior for the first time need to push in both directions. To get good model behavior, you need to really push on the realism axis and have rubrics or some kind of human feedback that's informing the reward function you use there. But if you try to do distillation naively, you end up just matching the teacher on the benchmaxxing distribution. If you don't have enough of the environments that really exercise the capabilities in these trickier realistic settings, then you're not going to get those into your student model. I think maybe one thing that's happening is the big models generalize better from the tricky narrow tasks to these more realistic tasks. If you have a really good realistic prompt distribution for distillation, you can match the big model really well. But if you only have this distribution of easily verifiable tasks, then you can match the big model on all the benchmarks, but you do worse on this broader distribution. That might even explain something about the smaller Anthropic models, like Sonnet 5, though it's hard to predict exactly what they're doing to post-train those models. It could also be that they're always changing their post-training stack, and they just got a few things wrong in some of these models. I don't know… they turned something up too high and created some quirks that people really don't like. It's really easy to screw up post-training in some way that doesn't show up in benchmarks.
Just one other very basic point is that the frontier AI labs buy all their data from big data companies. The Chinese can also just buy the same data from data companies. And they are, right? And they are. Exactly. There's a lot of people being annoyed about this, but if they have exactly the same data and they can buy that, they can also distill. It means it's quite easy to keep up, really.
The other question I had is how the first models that are capable of automating AI R&D will actually be trained. There's a toy version, which is this thing that Ryan was talking about. You just have GPT-8 try to build GPT-3 size models that are really good at inner loop type challenges: beating video games that require continual learning, or just getting to a certain loss with the least amount of compute, et cetera. But John, I think you had an interesting point that maybe that's not the way it actually will happen in practice. So I'm curious, by the point at which you have AIs that are actually capable of automating AI R&D, how are they probably trained?
We'll probably do some combination of learning from human feedback to absorb the researchers' taste, and just creating a lot of practice environments which involve doing multi-step research projects. People will in practice do some combination of those two things and, each iteration, patch whatever seems to be most broken in the last iteration. Researchers will be using the AIs a lot and will notice that they have some consistent weaknesses. Those things will either be patched by collecting human feedback or creating environments.
Maybe a useful way to think about this is how much of the lineage we roll back and then let self-play from there. In the limit, you're picturing just giving them a GPU and maybe neural nets or something and saying, "Okay, figure out how to train a model to do these particular tasks." The way it currently works is we go up to the very edge of the lineage and say, "Okay, here are the bugs Anthropic has found in their training stack in the last few months. We'll turn those into environments." You need to train and get better on the frontier. So you obviously lock in all the previous history of the lineage. But you could imagine a world in which you roll back to before GRPO or something. Then you have environments which try to get it to discover the best form to RL models on, and then maybe you roll further and further back… But I think we will still be so compute bottlenecked that people will just keep staying at the frontier and essentially diffing the bugs and whatever improvements they found since the last model version, turning those into training environments. Which is also really good for having non-stale, new data between model generations. Again, this is basically continual learning within the AI lab, of distilling the last three months of AI research progress through environments and RLHF-type stuff back into the model itself. And it is distilling. That's maybe why some of us feel like it's asymptotic. You're always just trying to get the last three months of progress. That progress is being contributed to by AIs, of course, but it also still has humans in the loop. It feels like you're just constantly inching closer and closer to what the human researchers are finding and capable of doing. The one thing I will say, though, is obviously if you're just distilling on trajectories, you can never go above it. But environments can go quite a far way above what a human can do. It's very easy to design an environment that no human can solve, but the AI can obviously still try and solve it. That would be the path to go ahead of just what the human AI research is.
Do you have an example in terms of RSI, of what kind of training set?
Nanochat speedrun, but doing it even faster than a human speedrunner. I feel like in AI research especially, it's very easy to define goals. You could say the loss needs to be 1.3 or something, and no human can get that now. But that's an extremely measurable, verifiable task. If the AI gets that, then great. Or I don't know, building a 100 million parameter model that beats Minecraft. That's maybe too easy, but beats a much more complicated game or something.
Isn't it crazy that 100 million parameter models beat Minecraft? We're calling that too easy? Imagine if you said that five years ago.
I would say a lot of research is not exactly like that, though, where it's hill climbing on a well-defined goal. It's more like, here's an intuition we have about some way models should be better. We also have some idea for an algorithm that seems to go a little bit in this direction. So let's come up with a task that is designed to show signs of life on this approach, and see if we get those signs of life. If we do, we can make successively more realistic versions of the task. It's a lot more guided by intuition.
The inner loop is to test for that intuition rather than the test itself leading to the insight.
Right. You're not directly optimizing for the eventual objective you care about or the practical production objective. You're relaxing your objective a little bit. You're saying, "Let's relax on the realism axis a little bit and find some methods that actually work, and then try to get back to realism later after the method matures a little bit." There's also research that's more oriented towards explaining things and developing a theory. Often we don't have mathematical theories in machine learning that are that predictive. But we have a lot of more informal theories for what's going on. Presumably the models will be trained on some combination of all of these tasks. Some will be very easily verifiable, some will be LLM-as-a-judge or just ask the human, "Does this look reasonable?" The hope would be that these would all generalize to these much harder, more vague, fuzzy kinds of tasks. It probably will to some extent. Whether it generalizes enough that the loop can become self-sealing without humans being in the loop at all is unclear.
Maybe taking a step back. Here's what it seems to me the plan for AI research going forward is. You tell me if you think it's going to work or if you agree with this characterization. The bet is that we will scale up RLVR training across millions of diverse environments, across hundreds of different kinds of domains. What will emerge at the other end is an agent which has learned these basic skills — or less than basic skills — around being persistent, being able to triage information and context, eventually having end-to-end optimization of working with other agents and things like that. Such an agent will be very sample efficient within the context — you've done research on how you scale up in-context learning to make it arbitrarily long, but you keep scaling it up. And what comes out the other end will be something that basically functions like a drop-in remote worker over the course of a week or a month. First of all, do you agree that that is the bet the labs are making? And second, is that enough? Basically learning how to learn within these simulacra within a data center, and then getting deployed into the real world, but not actually learning from real-world deployment… only learning these meta skills from the simulated environments in the data center.
I think it's now hard to separate out how much of the labs' effort is going towards direct RSI versus making generally intelligent models that they can continue to deploy to collect revenue to fund the next big training run. For the latter, yes, that's probably just the bet they're making. It's very clear, the pattern of where these environments are going over the last few years. Anthropic's lineage of environments is a very clear example of this. First, we just focus on coding and we're going to get really, really good at that. Then, from the task horizon that we've got from coding — which is probably the lowest-hanging fruit in terms of data available on the internet to create environments, and their own internal stuff that they can turn into environments — then we're going to generalize. We're going to go up to finance next, and literally just so much Excel data and all that sort of stuff in the RL training. Then it's PowerPoints. It's this long tail of the working economy. That seemed to work really well. A lot of the other labs, even the open source labs, have now realized that that was the correct bet to make.
But what is the implication from that? When I had Dario on the podcast, the thing I asked him was, if you truly expect models which will be human-like in their ability to learn on the job, why would you try to bake in all these skills of working with PowerPoint or something? Wouldn't you just expect the model to be able to pick that up while it's deployed?
There's multiple different explanations. One is just that we expect models to get there soon, but they're not there yet, so why not amortize these skills into the model training? Another is that we're not concentrated on making it really good at widely deployed work. We just want it really good at RSI. This is just a way for us to get revenue so that we can pour it back into a model that is actually really good at doing RSI development. Then once the singularity happens, the thing that comes out the other end will be really good at all the things which seem like bottlenecks to the current generation of models.
John, I don't know if you have takes on how one should construe why there is so much task-specific knowledge in these models if the path is this kind of generalization.
If the models were good enough at learning in context, then in theory, you wouldn't need to train them on finance. They would just be able to read all the books on the fly and figure out how to do everything in the appropriate jurisdiction. You could argue that you need to do a lot of this domain-specific training just to make them more efficient. Even if they were smart enough to figure this out on the fly, you still might want to do a bunch of RL and bake all these intuitions into the weights, so the model would be more efficient at runtime. In practice, it does seem like model providers are going domain by domain and trying to strengthen the models in the highest value domains. I'd say that that's one of the answers to why the models have gotten so much better. It's just because the model providers have covered a lot of the high-value domains and the most common types of skills. Another thing is just that it's not that expensive to do both at the same time. The models are massive. They can easily afford, in terms of their parameters, to learn everything. There is likely some transfer. Even if finance is not specific, the information is important for RSI. Just the general meta-learning of how to figure out what's important, how to have taste, how to do long-horizon work is potentially generalizable. There's not that much RSI data in the world as well. It's hard to generate and requires a lot of effort. So if you can amortize in this other data, you get some transfer from it. You already have masses of compute and masses of parameter space, so why not do that as well as, obviously, the direct commercial intent of selling a model?
I'll add that there's one question about whether this current paradigm of doing sim-to-real will be the dominant one forever. You look at what the real-world tasks are like. Then you try to create a bunch of environments that can be simulated in the data center, and you can do RL on them. Obviously, this has been very successful. But it has a lot of weaknesses, because a lot of things are just hard to simulate, especially if they involve interacting with a bunch of humans in real time. So there's some question about whether sim-to-real will be the dominant framework forever.
I think sim-to-real has to be the dominant framework while sample efficiency is low, because right now you need thousands and thousands of interactions with the humans. No human is going to sit there and be in the loop of RL training. So we have to simulate that now to get the samples you need. But obviously, if sample efficiency improves a lot, you'd expect learning from deployment to become a much bigger part of it. Though there are also other things you could do. You can learn off-policy, so you can take all the traces, and even without resimulating everything, you can potentially learn something from them.
Jane Street just launched a new competition, and it's its most ambitious one yet: design a protocol-emulator ASIC. Basically, if you have a chip you want to test, you can connect it to this ASIC, and the ASIC will simulate realistic traffic. That way, you can see how the chip responds without having to plug it into a live system. Jane Street is looking for flexible, general-purpose designs, not single-protocol emulators. When I was chatting with them, they suggested that I start by trying to implement what are apparently three very common protocols: UART, SPI, and I²C. Jane Street also mentioned that they hope more ambitious designs will tackle low-speed USB, Ethernet, and any other protocols that flex your chip's specific architecture. Importantly, your design should be reprogrammable rather than smashing a bunch of specific protocols onto a chip. If a new protocol comes out after your ASIC is taped out, your chip still needs to be able to handle it. How exactly it does that is up to you. But there is one hard constraint: your design must target an open-source 130-nanometer process node. That's because Jane Street will pay to tape out the most novel submissions and send physical copies to the winners. The competition is open until January 18, 2027, and working in teams is highly encouraged. Go to janestreet.com/dwarkesh to download the template code and get started.
I want to ask more about this, because it's weird that you have 50% of compute that's spent on inference that is not directly helping the model become better. One of the key advantages you'd expect digital minds to eventually have is that, unlike a human who gets to have 50 years of real-world experience, a model will get to experience, through all its instances, millions of years of deployment across all kinds of economically relevant work in the economy. Right now, that data is just not, in a meaningful sense, helping the model get better. It seems so obvious that eventually models should be able to learn from this data. Once they do, you would have something that almost feels like a widely deployed intelligence explosion, because the model is assimilating so much information across all these deployed instances. When do you expect this kind of hive-mind, crazy shit to start happening?
I think broadly, at a very basic level, this is already happening… just in the next generation of models. Right now, you can obviously take your deployment data and put this in the pre-train or the mid-train of future models, especially if you do some kind of filtering or some kind of judgment or annotation or synthesization of that.
How much do you think that explains the generation-over-generation improvement?
I think it explains quite a bit. I don't know whether the labs do this, because theoretically, they claim not to train on people's data. But the Chinese 100% do. They definitely get this advantage. This is basically what distillation is. They take the models, they get some fraction of their deployment data by pinging the model, and then they train their next generation of models on it. They can certainly do it on their own models as well. There's no reason not to whatsoever.
I completely agree with this. If you zoom out far enough, this is definitely happening. What we're all picturing, the holy grail of continual learning, is this very organic, live loop of an individual model getting an experience and live-updating on the spot and learning from that. A lot of things break when you zoom into that level of granularity. But the big labs are doing this. The closed models are doing this. There are also early signs of life of people using open-source models doing this at a much faster cadence. A good example is probably Composer. Harvey's doing the same thing with legal agents. You have some sort of model, and you are getting very specific environments from the data that you have for that particular task, and things that users are complaining about, and all the feedback that you're somehow extracting from your specific deployments. A lot of these companies have the advantage over the big labs in that they can use this data really, really well. Then they will create environments. They will do a big post-train of Kimi K3. They will go deploy it. They might do some online learning as well, like Composer did online… basically REINFORCE for a long time. There's still a human in the loop. There's still a human saying, "Okay, these are the signals we care about. Here's how we're going to create environments from the data that we have." It's still a longer cadence than maybe the one that you're thinking of, but it really is happening. Eventually that loop will become faster and faster.
The Composer thing is interesting because this is where, in Cursor, people press Tab or they don't press Tab on the next completion that the model suggests. Based on that, every single day, Composer gets better at predicting the next—
That was the old Tab model. They actually did the same thing not just for the Tab model, but for the actual generative model.
It's hard because when you do online reinforcement learning, you don't have groups. You just have one user saying one thing, and then you get one rollout. So you have a big variance-reduction problem. Cursor's fuzzy answer to this was, "We have very good heuristics which are able to estimate how much better than average this response was, or how much worse than average this response was." Then they would do this big REINFORCE update. Their solution to whether it got worse or not was that if it improved on CursorBench, they would deploy the new model every five hours. If it didn't, they would throw that version out.
I think your biggest problem is actually just not knowing what the reward function should be for natural data. If you use some kind of superficial signal, like did they accept the edit, that might get reward-hacked in some way.
But isn't this a bigger issue with the sim-to-real thing, where the longer-horizon tasks get, the harder they are to simulate within a data center? It seems to me that even in coding, we're already getting to the point where there's not some year-long coding task that doesn't eventually require you to talk to a client or interact with the company or interact with users. If you think about the gamut of things we would want AI to be capable of, eventually superintelligence should be able to run a business, or start a new business and make it profitable, or have a profitable day trading in the markets, or win a court case. These are all things which are very hard to simulate in a data center. An inherent part of the learning there is interacting with the real world. Maybe they learn how to get better at these things from the transfer between sim-to-real. But alternatively, maybe you do need weight updates from these kinds of interactions in order to get better at them. If that is the case — if transfer isn't strong enough and you do need weight updates — then the fact that the models are quite sample-inefficient is maybe a deeper problem. The reason I'm curious about this is that by default, I don't see how you don't get some kind of crazy recursive self-improvement within the next 10 years. But the one reason why that might not happen is that in terms of the sample efficiency of weight updates, models just seem way far behind humans. They're plausibly a millionfold behind humans in terms of how much data a human sees from birth to adulthood versus how much a model sees from cold start to finishing training. This is all to say, first of all, is there going to be good transfer between simulations and the extremely long-horizon, really complicated real shit that we want the AI to do in the real world? And if not, does that really mean that the lack of sample efficiency in these models comes to bite us?
Maybe the way I'd break down the two types of tasks — the ones in which models get good and the ones where models will still continue to struggle — is whether the task is cumulative, or whether you have this non-stationary distribution where you have to keep learning and relitigating a bunch of stuff. An example of a cumulative task might be RSI. It's theoretically possible to have a less-than-a-million-token Python file which from scratch trains a model that is capable of recursive self-improvement. Every discovery that you make is a line in the sand that you hold. If it's true that for RSI we don't need to discover a new attention variant or whatever, then once you've discovered attention, and once you've discovered mixture of experts, and once you discover GRPO, you just add that to the training stack and that's there. A good example of this is 5.6 Sol training, 5.6 Terra, or whichever one OpenAI told us it trained. It didn't have to go back and discover attention. It basically would have called a bunch of scripts, like pre-training.sh and post-training.sh, and just done that. That's an example of a cumulative task. I think the real world — and the reason people are thinking so much about continual learning — is not really a cumulative task. Imagine in a law firm, you have an agent acting as a legal associate. That's a very non-stationary distribution. You have to be able to fit in your context all the relationships between all the important people at that company, which are also changing all the time. You have all these implicit ways about how things are done, where to find information, et cetera. That's not as clean an example of a cumulative task as RSI is. I think there will be this breakdown between tasks. But if the labs realize that — and they do believe that RSI is cumulative in the sense that we don't need to go back and discover some brand-new architecture or whatever — then maybe more and more effort and compute gets focused on that versus the other tasks.
It's so unfortunate that RSI happened to be easier than being a paralegal.
I would say today's models are weaker than humans in a lot of different ways. Some of them might have to do with sample efficiency in a certain regime. In some regimes, models are very sample-efficient, like learning in context. But then there might be some medium-length regime where they're less sample-efficient, because humans can do some kind of weight update more efficiently than models. I think being less sample-efficient in certain regimes might be one of the sources of weakness. But I think there are other sources of weakness that are completely different from that.
For example, having lower diversity of thought than humans, or being bad at certain kinds of long-horizon judgments. I think a lot of what people call taste is something about behavior that works in the long run, and that people have realized works in the long run. Not everything, but some aspect of taste. Especially for something like software engineering, I think a lot of taste is 'What are the systems that are going to be maintainable and work well in the long run of this project?' There are a variety of weaknesses of models which limit RSI along with other things. Some of them are related to sample efficiency, and some of them aren't.
Maybe an interesting thought experiment is this. Let's say you were able to give a model a context window of a trillion tokens, or whatever you would have needed to fit in your experience prior to, let's say, RLHF. It's got all that experience in the context window, and it has the same sample efficiency and in-context learning ability as it does at a million tokens. Do you think taste is then solved? Would it be able to make the same judgments that you did? Or is there something fundamentally missing, apart from just a longer context window with the same sample efficiency?
It would have to be trained to learn from that context. Either it would have to be trained to learn the right update to make from that context, or it would have to generalize.
So you don't think you can just dump it all in, your whole life, your research experience?
You still need the data to train it on long context. Even if you could theoretically get a trillion context, you would need a trillion lengths of data to train it. Right now you have 10k context, you can't just dump in a million.
Yeah, I'm just asking if you had that.
In theory, I think, yes. This really just comes down to the question of how meta-learnable taste is from shorter-horizon episodes. I feel like there's no obvious reason it's super long, because humans somehow developed taste without having many long episodes. We don't live to be 10,000. We develop pretty quickly. If you think about even a PhD, the difference between a first-year PhD student and a final-year student or postdoc, that's five years maybe. They've only done maybe 10-30 research projects in total. But somehow they develop taste quite quickly from a relatively short succession of small things. Theoretically, it's possible to develop it like that. The AI obviously will have vastly more experience in which to develop taste, to meta-learn it. Then the question is how well that generalizes to really long-horizon things, which I think is really unsolved at this point. We don't know.
Going back to this question, eventually there should be a regime where AIs are learning a ton from each individual instance of deployment that they have. Currently you could say there's a fuzzy meta process by which models do improve from deployment. But I feel like it's a very weak feedback loop. Do you see this on the horizon, where there's this hive mind kind of learning that's very rapid, and if so, how exactly does it happen?
I would say that whether we get a hive mind that learns from all of its deployment experience is in a big part about incentives, rather than being a technical question. Companies aren't going to want to have the model provider learn from all of their deployment, because that might just reduce the advantage of their business. I think the economics of this will pressure, not necessarily weight updates to one big common shared model, but modules that get subbed in. A very obvious example of this is a LoRA, but it might be something else. There's been a lot of work to try and fit an arbitrary context length into a fixed size. This is all the linear attention stuff. And cartridges, which are essentially KV caches trained to be very, very compressed to fit in a lot of information. That's another example of something that companies may be willing to sign up for, if that gets subbed into the model and it's not actually changing the base underlying model itself. There are many different versions of learning from your data in real time. The latter ones are not really helping the big labs, because they are just these modules. But I think the economic pressure will force the labs to go down that path first before they can embark on this...
Which economic pressure, though? I feel like even if you have a bunch of cartridges or LoRAs or whatnot, you can still just take all these traces and dump them into the pre-training of your next generation of models.
Yes. It may be a more indirect form of learning that the big labs are getting. That's obviously still really valuable to them. But I can't imagine a world in which we start off with, 'We're going to just directly train this one big model on all the exact data that we're getting.' No, I think it will definitely go through stages, because this is assuming there's one discontinuous event where suddenly we fix weight updates continuously. In practice, I think it's much more likely to be that the cartridges and stuff allow you to specialize in deployments. Then you generate traces, you put that in your model, and three months later you come out with a model which is better at this stuff. You specialize it again, you consolidate it again, and then eventually we'll just make this leap faster and faster. Instead of releasing a model every three months, now it's every week, and then every day, and then every hour, at which point we've basically solved it.
I think this is a good point as well, because you asked how far off the current paradigm we are from being able to do this. We've done a bit of research into this, and people have done a lot of research. At a really large scale, when you wash out enough noise and you have large enough batches, this outer-loop process of putting data into mid-training and creating our own environments does work in some sort of continual learning regime. But the problem is, when you zoom in close enough at a micro level — I've got one model and I'm trying to update it again for a law firm or something, and I'm trying to do that very continuously with a relatively small amount of data — all the methods kind of break down a bit. If I SFT the model on just successful traces, off-policy or on-policy, eventually in the very iterative regime, when you're doing hundreds of these micro-updates, you see catastrophic forgetting. You see forgetting of previous information learned on top of the base model that was much earlier on, and you see degradation of general capabilities. On-policy distillation seems to push this horizon out a little bit, but it still eventually succumbs to the same thing. RL is good at getting capabilities in, but it's not as good at getting knowledge in, this very explicit knowledge of, 'Ah, okay, this person does this at this law firm, and this is a very specific process we find.' You have to pour in a lot of compute to create the right environments to get the knowledge in with RL.
