The following is a conversation all about the state-of-the-art in artificial intelligence, including some of the exciting technical breakthroughs and developments in AI that happened over the past year, and some of the interesting things we think might happen this upcoming year. At times, it does get super technical, but we do try to make sure that it remains accessible to folks outside the field without ever dumbing it down. It is a great honor and pleasure to be able to do this kind of episode with two of my favorite people in the AI community, Sebastian Raschka and Nathan Lambert. They are both widely respected machine learning researchers and engineers who also happen to be great communicators, educators, writers, and X posters.
Sebastian is the author of two books I highly recommend for beginners and experts alike. First is Build a Large Language Model from Scratch and Build a Reasoning Model from Scratch. I truly believe in the machine learning/computer science world, the best way to learn and understand something is to build it yourself from scratch. Nathan is the post-training lead at the Allen Institute for AI, and author of the definitive book on Reinforcement Learning from Human Feedback. Both of them have great X accounts, great Substacks. Sebastian has courses on YouTube, Nathan has a podcast. And everyone should absolutely follow all of those. This is the Lex Fridman podcast. To support it, please check out our sponsors in the description, where you can also find links to contact me, ask questions, get feedback, and so on. And now, dear friends, here's Sebastian Raschka and Nathan Lambert.
So I think one useful lens to look at all this through is the DeepSeek, so-called DeepSeek moment. This happened early in 2025, when the open-weight Chinese company DeepSeek released DeepSeek-R1 that I think it's fair to say surprised everyone with near state-of-the-art performance, with allegedly much less compute for much cheaper. And from then to today, the AI competition has gotten insane, both on the research level and the product level. It's just been accelerating. Let's discuss all of this today, and maybe let's start with some spicy questions if we can. Who's winning at the international level? Would you say it's the set of companies in China or the set of companies in the United States? And Sebastian, Nathan, it's good to see you guys. So Sebastian, who do you think is winning?
Winning is a very broad term. I would say you mentioned the DeepSeek moment, and I think DeepSeek is definitely winning the hearts of the people who work on open-weight models because they share these as open models. Winning, I think, has multiple timescales to it. We have today, we have next year, we have in 10 years. One thing I know for sure is that I don't think nowadays, in 2026, that there will be any company having access to a technology that no other company has access to. And that is mainly because researchers are frequently changing jobs, changing labs. They rotate. So I don't think there will be a clear winner in terms of technology access. However, I do think there will be, the differentiating factor will be budget and hardware constraints. I don't think the ideas will be proprietary, but rather the resources that are needed to implement them. And so I don't see currently a winner-takes-all scenario. I can't see that at the moment.
Nathan, what do you think?
You see the labs put different energy into what they're trying to do, and I think to demarcate the point in time when we're recording this, the hype over Anthropic's Claude 3.5 Opus model has been absolutely insane. I mean, I've used it and built stuff in the last few weeks, and it's... it's almost gotten to the point where it feels like a bit of a meme in terms of the hype. And it's kind of funny because this is very organic, and then if we go back a few months ago, we can see the release date and the notes when Gemini 1.5 from Google got released, and it seemed like the marketing and just, like, wow factor of that release was super high. But then at the end of October, Claude 3.5 Opus was released and the hype has been growing, but Gemini was before this. And it kind of feels like people don't really talk about it as much, even though when it came out, everybody was like, this is Gemini's moment to retake Google's structural advantages in AI. And Gemini is a fantastic model, and I still use it. It's just kind of differentiation is lower. And I agree with Sebastian; what you're saying with all these, the idea space is very fluid, but culturally Anthropic is known for betting very hard on code—which is the Claude Code thing—and it's working out for them right now. So I think that even if the ideas flow pretty freely, so much of this is bottlenecked by human effort and the culture of organizations, where Anthropic seems to at least be presenting as the least chaotic. It's a bit of an advantage, if they can keep doing that for a while. But on the other side of things, there's a lot of technology from China where there's way more labs than just DeepSeek. So DeepSeek kicked off a movement within China, I'd say kind of similar to how ChatGPT kicked off a movement in the US where everything had a chatbot. There's now tons of tech companies in China that are releasing very strong frontier open-weight models, to the point where I would say that DeepSeek is kind of losing its crown as the preeminent open model maker in China, and the likes of Zhipu AI with their GLM models, MiniMax's models—Kimi Moonshot, especially in the last few months, has shown more brightly. The new DeepSeek models are still very strong, but that's kind of a... it could look back as a big narrative point where in 2025 DeepSeek came and provided this platform for way more Chinese companies that are releasing these fantastic models to have this new type of operation. So these models from these Chinese companies are open weights, and depending on this trajectory, the business models that these American companies are doing could be at risk. But currently, a lot of people are paying for AI software in the US, and historically in China and other parts of the world, people don't pay a lot for software.
So some of these models like DeepSeek have the love of the people because they are open weight. How long do you think the Chinese companies keep releasing open weight models?
I would say for a few years. I think that, like in the US, there's not a clear business model for it. I have been writing about open models for a while, and these Chinese companies have realized it. So I get inbound from some of them. And they're smart and realize the same constraints, which is that a lot of top US tech companies and other IT companies won't pay for an API subscription to Chinese companies for security concerns. This has been a long-standing habit in tech, and the people at these companies then see open-weight models as an ability to influence and take part in a huge growing AI expenditure market in the US. And they're very realistic about this, and it's working for them. And I think that the government will see that that is building a lot of influence internationally in terms of uptake of the technology, so there's going to be a lot of incentives to keep it going. But building these models and doing the research is very expensive, so at some point, I expect consolidation, but I don't expect that to be a story of 2026, where there will be more open model builders throughout 2026 than there were in 2025. And a lot of the notable ones will be in China.
You were gonna say something?
Yes. You mentioned DeepSeek losing its crown. I do think to some extent, yes, but we also have to consider though, they are still, I would say, slightly ahead. And it's not that DeepSeek got worse, it's just that the other ones are using the ideas from DeepSeek. For example, you mentioned Kimi: same architecture, they're training it. And then again, we have this leapfrogging where they might be at some point a bit better because they have the more recent model. And I think this comes back to the fact that there won't be a clear winner. It will just be like that. One person releases something, the other one comes in, and the most recent model is probably always the best model.
Yeah. We'll also see that Chinese companies have different incentives. For example, DeepSeek is very secretive, whereas some of these startups are like MiniMax and Moonshot AI. Those two literally have filed IPO paperwork, and they're trying to get Western mindshare and do a lot of outreach there. I don't know if these incentives will change the model development, because DeepSeek famously is built by a hedge fund, Highflyer Capital, and we don't know exactly what they use the models for or if they care about this.
They're secretive in terms of communication, they're not secretive in terms of the technical reports that describe how their models work. They're still open on that front. And we should also say on the Claude 3.5 Opus hype, there's the layer of something being the darling of the X echo chamber, the Twitter echo chamber, and the actual amount of people that are using the model. I think it's probably fair to say that ChatGPT and Gemini are focused on the broad user base that just want to solve problems in their daily lives, and that user base is gigantic. So the hype about the coding may not be representative of the actual use.
I would say also a lot of the usage patterns are, like you said, name recognition, brand and stuff, but also muscle memory almost, where you know, like, ChatGPT has been around for a long time. People just got used to using it, and it's almost like a flywheel: they recommend it to other users and so on. One interesting point is also the customization of LLMs. For example, ChatGPT has a memory feature, right? And so you may have a subscription and you use it for personal stuff, but I don't know if you want to use that same thing at work. Because it's a boundary between private and work. If you're working at a company, they might not allow that or you may not want that. And I think that's also an interesting point where you might have multiple subscriptions. One is just clean code; it has nothing of your personal images or hobby projects in there. It's just like the work thing. And then the other one is your personal thing. So I think that's also something where there are two different use cases, and it doesn't mean you only have to have one. I think the future is multiple ones.
What model do you think won 2024, and what model do you think is going to win '25?
I think in the context of consumer chatbots, it's a question of: are you willing to bet on Gemini over ChatGPT? Which I would say, in my gut, feels like a bit of a risky bet because OpenAI has been the incumbent and there's so many benefits to that in tech. I think the momentum, if you look at 2024, was on Gemini's side, but they were starting from such a low point. And RIP Bard and these earlier attempts at getting started. I think huge credit to them for powering through the organizational chaos to make that happen. But also it's hard to bet against OpenAI because they always come off as, as so chaotic, but they're very good at landing things. And I think, personally, I have very mixed reviews of o1, but it had to have saved them so much money with the high-line feature being a router where most users are no longer charging their GPU costs as much. So I think it's very hard to dissociate the things that I like out of models versus the things that are gonna actually be a general public differentiator.
What do you think about 2025? Who's going to win?
I'll say something, even though it's risky. I think Gemini will continue to make progress on ChatGPT. I think Google's scale, when both of these are operating at such extreme scales, is a factor. Google has the ability to separate research and product better, where you hear so much about OpenAI being chaotic operationally and chasing the high-impact thing, which is a very startup culture. And then on the software and enterprise side, I think Anthropic will have continued success as they've again and again been set up for that. Obviously Google Cloud has a lot of offerings, but I think this Gemini name brand is important for them to build. Google Cloud will continue to do well, but that's a more complex thing to explain in the ecosystem because that's competing with Azure and AWS rather than on the model provider side.
So in infrastructure, you think TPU gives them an advantage?
Largely because the margin on NVIDIA chips is insane, and Google can develop everything from top to bottom to fit their stack and not have to pay this margin, and they've had a head start in building data centers. So all of these things that have both high lead times and very hard margins on high costs. Google has a kind of historical advantage there, and if there's going to be a new paradigm, it's most likely to come from OpenAI, where their research division again and again has shown this ability to land a new research idea or a product. Like Deep Research, Sora, o1 thinking models—all these definitional things have come from OpenAI, and that's got to be one of their top traits as an organization. So it's kind of hard to bet against that, but I think a lot of this year will be about scale and optimizing what could be described as low-hanging fruit in models.
And clearly there's a trade-off between intelligence and speed. This is what GPT-5 was trying to solve behind the scenes. It's like, do people actually want intelligence, the broad public, or do they want speed?
I think it's a nice variety, actually, or the option to have a toggle there. For my personal usage, most of the time when I look something up, I use ChatGPT to ask a quick question, get the information I want fast. For most daily tasks, I use the quick model. Nowadays, I think the auto mode is pretty good where you don't have to specifically say thinking or non-thinking. Then again, I also sometimes want the Pro mode. Very often what I do is, when I have something written, I put it into ChatGPT and say, "Hey, do a very thorough check. Are all my references correct? Are all my thoughts correct? Did I make any formatting mistakes? Are the figure numbers wrong?" or something like that. And I don't need that right away. It's okay—I finish my stuff, maybe have dinner, let it run, come back and go through this. I think this is where it's important to have this option. I would go crazy if for each query I had to wait 30 minutes or even 10 minutes.
That's me. I'm sitting over here losing my mind that you use the router and the non-thinking model. I'm like, "How do you live with that?" That's my reaction. I've been heavily on ChatGPT for a while. I never touch non-thinking. I find its tone and then its propensity for errors—it just has a higher likelihood of errors. Some of this is from back when OpenAI released o1-preview, which was the first model to do this deep search and find many sources and integrate them for you. So I became habituated with that. So I will only use GPT-4o thinking or Pro when I'm performing any sort of information query for work, whether that's a paper or some code reference that I need. And I will regularly have five Pro queries going simultaneously, each looking for one specific paper or feedback on an equation or something.
I have a fun example where I needed the answer as fast as possible for this podcast before I was going on the trip. I have a local GPU running at home, and I wanted to run a long RL experiment. Usually, I unplug things because you never know when you're not at home. I accidentally unplugged the GPU. My wife was already in the car, and it was like, "Oh dang." Basically, I wanted as fast as possible a Bash script that runs my different experiments and the evaluation. And it's something I know. I learned how to use the Bash interface or Bash terminal, but in that moment I just needed like 10 seconds, give me the command.
This is a hilarious situation. But yeah, so what did you use?
So I did the non-thinking fastest model. It gave me the Bash command to chain different scripts to each other and then the thing is, you have the tee thing where you want to route this to a log file. Top of my head I was just in a hurry, I could have thought about it myself.
By the way, I don't know if there's a representative case: wife waiting in the car... you have to run, unplug the GPU, you have to generate a Bash script. This sounds like a movie, like Mission Impossible.
I use Gemini for that. So I use thinking for all the information stuff and then Gemini for fast things or stuff that I could sometimes Google. It's good at explaining things and I trust that it has this kind of background of knowledge and it's simple. And the Gemini app has gotten a lot better. It's good for those sorts of things. And then for code and any sort of philosophical discussion, I use Claude Opus 3.5. Also always with extended thinking. Extended thinking and inference-time scaling is just a way to make the models marginally smarter. And I will always hedge on that side when the progress is very high because you don't know when that'll unlock a new use case. And then sometimes I use Grok for real-time information or finding something on AI Twitter that I knew I saw and I need to dig up and I just fixated on. Although when Grok-3 came out, the Grok-3 what is Super Heavy, which was like their pro variant, was actually very good and I was pretty impressed with it, and then it just kind of muscle memory lost track of it with having the ChatGPT app open. So I use many different things.
Yeah. I actually do use Grok-3 Heavy for debugging. For like hardcore debugging that the other ones can't solve, I find that it's the best at. It's interesting because you say ChatGPT is the best interface. For me, for that same reason, but this could be just momentum—Gemini is the better interface for me. I think because I fell in love with their best needle in the haystack. If I ever put something that has a lot of context but I'm looking for very specific kinds of information to make sure it tracks all of it, I find at least that Gemini for me has been the best. So it's funny with some of these models, if they win your heart over—for one particular feature at one... on a one particular day, for that particular query, that prompt, you're like, "This model's better." And so you'll just stick with it for a bit until it does something really dumb. There's like a threshold effect. Some smart thing and then you fall in love with it and then it does some dumb thing and you're like, "You know what? I'm gonna switch and try Claude or ChatGPT." And all that kind of stuff.
This is exactly it. You use it until it breaks, until you have a problem, and then you change the LLM. And I think it's the same how we use anything, like our favorite text editor, operating systems, or the browser. I mean, there are so many browser options: Safari, Firefox, Chrome. They're relatively similar, but then there are edge cases, maybe extensions you wanna use, and then you switch. But I don't think there is anyone who types the same thing, like the website, into different browsers and compares them. You only do that when the website doesn't render, or something breaks. So that's a good point. You use it until it breaks, and then you explore other options.
On the long context thing, I was also a Gemini user for this, but the GPT-4o release blog had crazy long context scores, where a lot of people were like, "Did they just figure out some algorithmic change?" It went from, like, 30% to 70% or something in this minor model update. So it's also very hard to keep track of all of these things, but now I look more favorably at GPT-4o's long context. So it's just kind of like, "How do I actually get to testing this?" It's a never-ending battle.
Well, it's interesting that none of us talked about the Chinese models from a usage perspective. What does that say? Does that mean the Chinese models are not as good, or does that mean we're just very biased and US-focused?
I do think that's currently the discrepancy between the model and the platform. I think the open models are more known for their open weights, not their platform yet.
There are also a lot of companies willing to sell you open-model inference at a very low cost. With things like OpenRouter, it's easy to look at multi-model things. You can run DeepSeek on Perplexity. I think all of us sitting here are like, "We use OpenAI GPT-4 Pro consistently." We're all willing to pay for the marginal intelligence gain. And these models from the US are better in terms of the outputs. I think the question is: will they stay better for this year and for years to come? As long as they're better, I'm gonna pay to use them. There's also analysis that shows that the way Chinese models are served—you could argue this is due to export controls or not—is that they use fewer GPUs per replica, which makes them slower and gives them different errors. It's speed and intelligence. If these things are in your favor as a user, a lot of users in the US will go for this, and I think that is one thing that will spur these Chinese companies to compete in other ways, whether it's free or substantially lower costs, or it'll breed creativity in terms of offerings, which is good for the ecosystem. But the simple thing is the US models are currently better, and we use them. I've tried these other open models, and I'm like, "Fun, but I don't go back to them."
We didn't really mention programming. That's another use case that a lot of people care about. I use basically half-and-half Cursor and Claude, because they're... I find them to be a fundamentally different experience and both useful. What do you guys... You program quite a bit, so what do you use? What's the current vibe?
So, I use the Codeium plugin for VS Code. You know, it's very convenient. It's just like a plugin, and then it's a chat interface that has access to your repository. I know that Claude is, I think, a bit different. It is a bit more agentic. It touches more things. It does the whole project for you. I'm not quite there yet where I'm comfortable with that, because maybe I'm a control freak, but I still would like to see what's going on. And Codeium is kind of the sweet spot right now, where it is helping me, but it is not taking completely over.
I should mention, one of the reasons I use Claude is to build the skill of programming with English. I mean, the experience is fundamentally different. You're... As opposed to micromanaging the details of the process of the generation of the code, and looking at the diff, which you can in Cursor if that's the IDE you use, and changing, altering. Looking and reading the code and understanding the code deeply as you progress, versus just thinking in this design space and guiding it at this macro level, which I think is another way of thinking about the programming process. Also, we should say that Claude just seems to be somehow a better utilization of Claude Opus 3.5.
It's a good side by side. You can have Claude open, you can have Cursor open, you can have VS Code open, and select the same models on all of them—and ask questions. It's very interesting. Claude is way better in that domain. It's remarkable.
All right, we should say that both of you are legit on multiple fronts, researchers, programmers, educators, Twitterers. And on the book front, too. So Nathan, at some point soon, hopefully has an RLHF book coming out.
It's available for preorder, and there's a full digital pre-print. I'm making it pretty and better organized for the physical thing, which is why I do it, because it's fun to create things that you think are excellent in the physical form, when so much of our life is digital.
I should say, according to Perplexity here, Sebastian Raschka is a machine learning researcher and author known for several influential books. A couple of them that I wanted to mention, which is a book I highly recommend, Build a Large Language Model from Scratch, and the new one, Build a Reasoning Model from Scratch. I'm really excited about that. Building stuff from scratch is one of the most powerful ways of learning.
Honestly, building an LLM from scratch is a lot of fun. It's also a lot to learn. And like you said, it's probably the best way to learn how something really works, 'cause you can look at figures, but figures can have mistakes. You can look at concepts and explanations, but you might misunderstand them. But if there is code, and the code works, you know it's correct. There's no misunderstanding. It's precise. Otherwise, it wouldn't work. And that's the beauty behind coding. It doesn't lie. It's math, basically. Even though with math, I think you can have mistakes in a book you would never notice. Because you are not running the math when you are reading the book, you can't verify this. And with code, what's nice is you can verify it.
Yeah, I agree with you about the "Build a Large Language Model (From Scratch)" book. It's nice to tune out everything else, the internet and so on, and just focus on the book. But, you know, I read several history books. It's just less lonely somehow. It's really more fun. For example, on the programming front, I think it's genuinely more fun to program with an LLM. And I think it's genuinely more fun to read with an LLM. But you're right. That distraction should be minimized. So you use the LLM to enrich the experience, maybe add more context. The rate of "aha moments" for me on a small scale is really high with LLMs.
100%. I also want to correct myself: I'm not suggesting not to use LLMs. I suggest doing it in multiple passes. Like, one pass just offline, in focus mode, and then after that... I mean, I also take notes, but I try to resist the urge to immediately look things up. I do a second pass. It's just more structured this way. Sometimes things are answered later in the chapter, and sometimes it just helps to let it sink in and think about it. I would highly recommend using LLMs when reading books. For me, it's just not the first thing to do; it's the second pass.
By way of recommendation, I'll say I do the opposite. I like to use the LLM at the beginning to lay out the full context of what is this world that I'm now stepping into? But I try to avoid clicking out of the LLM into the world of Twitter and blogs. Because then you're down this rabbit hole. You're reading somebody's opinion, there's a flame war about a particular topic, and all of a sudden you're in the realm of the internet and Reddit. But if you're purely letting the LLM give you the context of why this matters and what the big picture ideas are... Books themselves are good at that, but not always.
This is why I like the ChatGPT app: it gives the AI a home on your computer where you can focus on it, rather than just being another tab in my mess of internet options. And I think Claude Code does a good job of making that a joy. It seems very engaging as a product designed to be an interface that your AI will then go out into the world. It's something intangible compared to OpenAI's models, it just feels warm and engaging, whereas they can often be as good but just feel a little bit rough around the edges. Whereas Claude Code makes it fun to build things, particularly from scratch where you don't have to worry, because you trust it'll make something. This is good for websites and refreshing tooling and stuff like this, which I use it for, or data analysis. For my blog, we scrape Hugging Face to keep download numbers for every dataset and model over time now. And Claude was just like, "Yeah, I've made use of that data, no problem." And I was like, "That would've taken me days." And then I have enough situational awareness to be like, "Okay, these trends obviously make sense." But that's just a wonderful interface where you can have an intermediary and not have to do the awful low-level work that you would have to do to maintain different web projects.
All right. So we just talked about a bunch of the closed-weight models. Let's talk about the open ones. So tell me about the landscape of open LLM models. Which are interesting ones? Which stand out to you and why? We already mentioned DeepSeek.
Do you wanna see how many we can name off the top of our head?
Yeah, yeah. Without looking at notes.
DeepSeek, Kimi, MiniMax, Z.ai, Moonshot. We're just going Chinese.
Let's throw in Mistral AI, Gemma, OLMo, the open-source model by AI2. Actually, NVIDIA had a really cool one, Nemotron 340B. There, there's a lot of stuff especially at the end of the year. Qwen might be the one—
Qwen was the obvious name I was going to say. I was trying to get through the—You can get at least 10 Chinese and at least 10 Western. I think that... OpenAI released their first open model since GPT-2. When I was writing about OpenAI's open model release, they were like, "Don't forget about GPT-2," which I thought was really funny 'cause it's just such a different time. But GLM-4 is actually a very strong model and does some things that the other models don't do very well. I think that selfishly I'll promote a bunch of Western companies. Both in the US and Europe have these fully open models. I work at the Allen Institute for AI, where we've been building OLMo, which releases data and code. And now we have actual competition from people that are trying to release everything so that others can train these models. There's the Institute for Foundation Models and LM360, which had their K2 models of various types. Apertus is a Swiss research consortium. Hugging Face has SmolLM, which is very popular. And NVIDIA's Nemotron has started releasing data as well. And then Stanford's Meerkat project, which is making it so there's a pipeline for people to open a GitHub issue and implement a new idea and then have it run in a stable language modeling stack. So this space—that list was way smaller in 2024—I think it was just AI2. So it's a great thing for more people to get involved and to understand language models, which doesn't really have a Chinese company that has an analog. While I'm talking, I'll say that the Chinese open language models tend to be much bigger, and that gives them higher peak performance as MoEs, where a lot of these things that we like a lot, whether it was Gemma and Nemotron, have tended to be smaller models from the US, which is starting to change. In the US and Europe, Mistral Large 3 came out, which was a giant MoE model, very similar to DeepSeek architecture, in December. And then startups like RCAI and both Nemotron and NVIDIA have teased MoE models way bigger than 100 billion parameters—in the 400 billion parameter range, coming in this Q1 2026 timeline. So I think this kind of balance is set to change this year in terms of what people are using Chinese versus US open models for, which I'm personally going to be very excited to watch.
First of all, huge props for being able to name so many of these. Did you actually name LLaMA?
All right. Can you mention some interesting models that stand out? You mentioned Qwen 2.5 is obviously a standout.
I would say the year is almost bookended by DeepSeek V3 and R1. And then on the other hand, in December, DeepSeek V3. Because what I like about those is they always have an interesting architecture tweak—architecture tweak that others don't have. Otherwise, if you want to go with familiar but really good performance, Qwen 2.5 and, like Nathan said, also Jamba. And I think Jamba, what's interesting about it is it's the first public or open weight model that was really trained with tool use in mind, which I do think is a bit of a paradigm shift where the ecosystem was not quite ready for it. By tool use, I mean that the LLM is able to do a web search, to call a Python interpreter. And I do think it's a standout because it's a huge unlock. Because one of the most common complaints about LLMs are, for example, hallucinations, right? In my opinion, one of the best ways to solve hallucinations is to not try to always remember information or make things up. For math, why not use a calculator app or Python? If I ask the LLM, "Who won the soccer World Cup in 1998?" instead of just trying to memorize, it could do a search. I think mostly it's still a Google search. So ChatGPT and GPT-4o, they would do a tool call to Google, maybe find the FIFA website, and find that it was France. It would get you that information reliably instead of just trying to memorize it. So I think it's a huge unlock, which I think right now is not fully utilized yet by the open source, open weight ecosystem. A lot of people don't use tool call modes because it's a trust thing. You don't want to run this on your computer where it has access to tools, could wipe your drive or whatever. So you want to containerize that. But I do think that is a really important step for the upcoming years to have this ability.
So a few quick things. First of all, thank you for defining what you mean by tool use. I think that's a great thing to do in general for the concepts we're talking about. Even things as sort of well-established as MOEs. You have to say that means Mixture of Experts, and you kind of have to build up an intuition for people what that means, how it's utilized, what are the different flavors. So what does it mean that there's just such an explosion of open models? What's your intuition?
If you're releasing an open model, you want people to use it; that is the first and foremost thing. And then after that comes things like transparency and trust. I think when you look at China, the biggest reason is that they want people around the world to use these models, and I think a lot of people will not. If you look outside of the US, a lot of people will not pay for software, but they might have computing resources to run it. I think there can also be data that you don't want to send to the cloud. So the number one thing is getting people to use models, use AI, or use your AI that might not be able to do it without having access to the model.
I guess we should state explicitly, so we've been talking about these Chinese models and open weight models. Oftentimes, the way they're run is locally. So it's not like you're sending your data to China or to whoever developed it in Silicon Valley, whoever developed the model.
A lot of American startups make money by hosting—these models from China and selling them. It's called selling tokens, which means somebody will call the model to do some piece of work. I think the other reason is for US companies, like OpenAI is so GPU deprived. They're at the limits of the GPUs. Whenever they make a release, they're always talking about like, "Our GPUs are hurting." And I think in one of these GPT-4o release sessions, Sam Altman said, "We're releasing this because we can use your GPUs. We don't have to use our GPUs, and OpenAI can still get distribution out of this," which is another very real thing, because it doesn't cost them anything.
And for the user, I mean, there are users who just use the model locally like they would use ChatGPT. But for companies, I think it's a huge unlock to have these models because you can customize them, you can train them, you can add more data post-training. Like, specialize them into, let's say, law, medical models, whatever you have. And you mentioned Llama; the appeal of the open-weight models from China is that the open-weight models also have licenses are even friendlier. I think they are just unrestricted open source licenses where if we use something like Llama or Gemma, there are some strings attached. I think there's an upper limit in terms of how many users you have. And then if you exceed, I don't know, so and so many million users, you have to report your financial situation to, let's say, Meta or something like that. And I think while it is a free model, there are strings attached, and people do like things where strings are not attached. So I think that's also one of the reasons, besides performance, why the open-weight models from China are so popular, because you can just use them. There's no catch in that sense.
The ecosystem has gotten better on that front, but mostly downstream of these new providers providing such open licenses. That was funny when you pulled up Perplexity and said, "Kimi-k2-thinking hosted in the US." Which is just like an exact... I've never seen this, but it's an exact example of what we're talking about where people are sensitive to this. But Kimi-k2-thinking and Kimi-k2 is a model that is very popular. People say that has very good creative writing and also in doing some software things. So it's just these little quirks that people pick up on with different models that they like.
What are some interesting ideas that some of these models have explored that you can speak to, like that are particularly interesting to you?
Maybe we can go chronologically. I mean, there was, of course, DeepSeek. DeepSeek R1 that came out in January of 2025, if we just focus on this year. However, this was based on DeepSeek-V3, which came out the year before in December 2024. There are multiple things on the architecture side. What is fascinating is you can still... I mean, that's what I do with my from-scratch coding projects. You can still start with GPT-2, and you can add things to that model to make it into this other model. So it's all still kind of like the same lineage... It is a very close relationship between those. But top of my head, DeepSeek, what was unique there is the Mixture of Experts. I mean, they were not inventing Mixture of Experts. We can maybe talk a bit more what Mixture of Experts means. But just to list these things first before we dive into detail. Mixture of Experts, but then they also had Multi-head Latent Attention, which is a tweak to the attention mechanism, where this was, I would say, in 2025, the main distinguishing factor kind of between these open-weight models. Different tweaks to make inference or KV cache size. We can also define KV cache in a few moments. But to kind of make it more economical to have long context, to shrink the KV cache size. So what are tweaks that we can do? And most of them focused on the attention mechanism. There is Multi-head Latent Attention in DeepSeek. There is Grouped-query Attention, which is still very popular. It's not invented by any of those models. It goes back a few years. But that would be the other option. Sliding Window Attention, I think OLMo 2 uses it if I remember correctly. So there are these different tweaks that make the models different. Otherwise I put them all together in an article once, where I just compared them. They are very surprisingly similar. It's just different numbers in terms of how many repetitions of the Transformer block you have in the center. And, like, just little knobs that people tune. But what's so nice about it is it works no matter what. You can tweak things. You can move the normalization layers around to get some performance gains. And OLMo is always very good in ablation studies, showing what it does to the model if you move something around. Does it make it better or worse? But there are so many, let's say, ways you can implement a transformer and make it still work. The big ideas that are still prevalent is Mixture of Experts, multi-head latent attention, sliding window attention, group query attention. And then at the end of the year, we saw a focus on making the attention mechanism scale linearly with inference token prediction. So there was Qwen2.5, for example, which added a gated delta net. It's kind of inspired by State space models, where you have a fixed state that you keep updating. But it makes essentially this attention cheaper, or it replaces attention with a cheaper operation.
And it may be useful to step back and talk about transformer architecture in general.
Yeah, so maybe we should start with GPT-2 architecture. The transformer that was derived from the "Attention Is All You Need" paper. The "Attention Is All You Need" paper had a transformer architecture that had two parts, an encoder and a decoder. And GPT went just focusing in on the decoder part. It is essentially still a neural network and it has this attention mechanism inside. And you predict one token at a time. You pass it through an embedding layer. There's the transformer block. The transformer block has attention modules and a fully connected layer. And there are some normalization layers in between. But it's essentially neural network layers with this attention mechanism. So coming from GPT-2 when we move on to GPT-3, there is, for example, the Mixture of Experts layer. It wasn't invented by GPT-3; it's a few years old. But it is essentially a tweak to make the model larger without consuming more compute in each forward pass. So there is this fully connected layer, and if listeners are familiar with multi-layer perceptrons, you can think of a mini multi-layer perceptron, a fully connected neural network layer inside the transformer. And it's very expensive because it's fully connected. If you have 1,000 inputs and 1,000 outputs, that's one million connections. And it's a very expensive part in this transformer. And the idea is to kind of expand that into multiple feedforward networks. So instead of having one, let's say you have 256, but it would make it way more expensive, because now you have 256, but you don't use all of them at the same time. So you now have a router that says, "Okay, based on this input token, it would be useful to use this fully connected network." And in that context, it's called an expert. So a Mixture of Experts means you have multiple experts. And depending on what your input is—let's say it's more math-heavy, it would use different experts, compared to, let's say, translating input text from English to Spanish. It would maybe consult different experts. It's not as clear-cut to say, "Okay, this is only an expert for math and for Spanish." It's a bit more fuzzy. But the idea is essentially that you pack more knowledge into the network, but not all the knowledge is used all the time. That would be very wasteful. So during the token generation, you are more selective. There's a router that selects which tokens should go to which expert. It adds more complexity. It's harder to train. There's a lot that can go wrong, like collapse and everything. So I think that's why OLMo still uses dense... I mean, you have OLMo models with Mixture of Experts, but dense models, where dense means... So, also, it's jargon. There's a distinction between dense and sparse. So Mixture of Experts is considered sparse, because we have a lot of experts, but only a few of them are active. So that's called sparse. And then dense would be the opposite, where you only have, like, one fully connected module, and it's always utilized.
So maybe this is a good place to also talk about KV cache. But actually, before that, even zooming out, like, fundamentally, how many new ideas have been implemented from GPT-2 to today. Like, how different really are these architectures?
Like the Mixture of Experts. The attention mechanism in Llama 3, that would be the Group Query Attention mechanism. So it's a slight tweak from multi-head attention to Group Query Attention, so that we have that. I think they replaced LayerNorm by RMSNorm, but it's just like a different normalization there and not a big change. It's just like a tweak. The nonlinear activation function, people familiar with deep neural networks, I mean, it's the same as changing sigmoid with ReLU. It's not changing the network fundamentally. It's just like a little tweak. And that's about it, I would say. It's not really fundamentally that different. It's still the same, same architecture. So you can convert one from one... You can go from one into the other by just adding these changes, basically.
It fundamentally is still the same architecture.
Yep. So for example, you mentioned my book earlier. That's a GPT-2 model in the book because it's simple and it's very small, so 124—124 million parameters approximately. But in the bonus materials, I do have OLMo from scratch, Llama 3 from scratch, and other types of from-scratch models. And I always start it with my GPT-2 model and just tweak the—well, add different components and you get from one to the other. It's kind of like a lineage, in a sense.
Can you build up an intuition for people? Because when you zoom out, you look at it, there's so much rapid advancement in the AI world. And at the same time, fundamentally the architectures have not changed. So where is all the turbulence, the turmoil of the advancement happening? Where are the gains to be had?
So there are the different stages where you develop the network or train the network. You have the pre-training. Now back in the day, it was just pre-training with GPT-2. Now you have pre-training, mid-training and post-training. So, I think right now we are in the post-training focus stage. I mean, pre-training still gives you advantages if you scale it up to better, higher quality data. But then we have capability unlocks that were not there with GPT-2, for example. ChatGPT is basically a GPT-3 model. And GPT-3 is the same as GPT-2 in terms of architecture. What was new was adding the supervised fine-tuning and the reinforcement learning with human feedback. So it's more on the algorithmic side rather than the architecture.
I would say that the systems also change a lot. I think if you listen to Nvidia's announcements, they talk about these things like, "You now do FP8, you can now do FP4." And what is happening is these labs are figuring out how to utilize more compute to put it into one model, which lets them train faster and that lets them put more data in. And then you can find better configurations faster by doing this. So you can look at, essentially, the tokens per second per GPU is a metric that you look at when you're doing large-scale training. And you could get... You can go from, like, 10K to 13K by turning on FP8 training, which means you're using less memory per parameter in the model. And by saving less information, you do less communication and you can train faster. So all of these system things underpin way faster experimentation on data and algorithms that is kind of like... It's this kind of loop that keeps going where it's kinda hard to describe when you look at the architecture and they're exactly the same. But the codebase used to train these models is gonna be vastly different—and you could probably... the GPUs are different, but you probably train GPT-NeoX-20B way faster in wall-clock time than GPT-2 was trained at the time.
Yeah. Like you said, they had, for example, in the mixture of experts, this FP4 optimization, for example, where you get more throughput. But I do think this is... for the speed, this is true, but it doesn't give the model new capabilities. It's just: how much can we make the computation coarser without suffering in terms of model performance degradation? But I do think... I mean, there are alternatives popping up to the transformer. There are text diffusion models—a completely different paradigm. And there is also... I mean, although text diffusion models might use transformer architectures, it's not an autoregressive transformer. And also Mamba models. It's a state-space model. But they do have trade-offs, and currently, there's nothing that has replaced the autoregressive transformer as the state-of-the-art model. For state-of-the-art, you would still go with that thing, but there are now alternatives for the cheaper end, alternatives that are kind of making compromises, but it's not just one architecture anymore. There are little ones coming up. But if we talk about the state-of-the-art, it's pretty much still the transformer architecture, autoregressive, derived from GPT-2 essentially.
I guess the big question here is, we talked quite a bit about the architecture behind the pre-training. Are the scaling laws holding strong across pre-training, post-training, inference, context size, data, synthetic data?
I'd like to start with the technical definition of a scaling law—which kind of informs all of this. The scaling law is the power law relationship between... You can think of the x-axis, kind of what you are scaling, as a combination of compute and data, which are kind of similar, and then the y-axis is like the held-out prediction accuracy over next tokens. We talked about models being autoregressive. It's like if you keep a set of text that the model has not seen, how accurate will it get when you train? And the idea of scaling laws came when people figured out that that was a very predictable relationship. And I think that that technical term is continuing, and then the question is, what do users get out of it? And then there are more types of scaling where, OpenAI's o1 was famous for introducing inference time scaling. And I think less famously for also showing that you can scale reinforcement learning training and get kind of this log x-axis and then a linear increase in performance on the y-axis. So there's kind of these three axes now where traditional scaling laws are talked about for pre-training, which is how big your model is and how big your dataset is, and then scaling reinforcement learning, which is like how long can you do this trial and error learning that we'll talk about. We'll define more of this, and then this inference time compute, which is letting the model generate more tokens on a specific problem. So I'm kind of bullish, but they're all really still working, but the low-hanging fruit has mostly been taken, especially in the last year on reinforcement learning with verifiable rewards, which is this RLVR, and then inference time scaling, which is just why these models feel so different to use, where previously you would get that first token immediately. And now they'll go off for seconds, minutes or even hours, generating these hidden thoughts before giving you the first word of your answer. And that's all about this inference time scaling which is such a wonderful kind of step function in terms of how the models change abilities. They kind of enabled this tool use and enabled this much better software engineering that we were talking about. And this is, when we say enabled, almost entirely downstream of the fact that this reinforced learning with verifiable rewards training just kind of let the models pick up these skills very easily. So let the models learn. So if you look at the reasoning process when the models are generating a lot of tokens, what it'll be often doing is it tries a tool, it looks at what it gets back. It tries another API, it sees what it gets back and if it solves the problem. So the models, when you're training them, very quickly learn to do this. And then at the end of the day, that gives this kind of general foundation where the model can use CLI commands very nicely in your repo and handle Git for you and move things around and organize things or search to find more information, which if we're sitting in these chairs a year ago is something that we didn't really think of the models being doing. So this is just kind of something that has happened this year and has totally transformed how we think of using AI, which I think is very magical. It's such an interesting evolution and just unlocks so much value. But it's like, it's not clear what the next avenue will be in terms of unlocking stuff like this. I think that there's—we'll get to continual learning later, but there's a lot of buzz around certain areas of AI, but no one knows when the next step function will really come.
So you've actually said quite a lot of things there, and said profound things quickly. It would be nice to unpack them a little bit. You say you're bullish basically on every version of scaling. So can we just even start at the beginning? Pre-training, are we kind of implying that the low-hanging fruit on pre-training scaling has been picked? Has pre-training hit a plateau or is even pre-training still something you're bullish on?
Pre-training has gotten extremely expensive. I think to scale up pre-training, it also implies that you're going to serve a very large model to the users. So I think that it's been...
Loosely established, the likes of GPT-4 and similar models were around one trillion parameters at the biggest size. There's a lot of rumors that they've actually gotten smaller as training has gotten more efficient. You want to make the model smaller because then your costs of serving go down proportionately. These models, the cost of training them is really low relative to the cost of serving them to hundreds of millions of users. DeepSeek had this famous number of about five million dollars for pre-training at cloud market rates. In the OLMo-1 section 2.4 paper, we just detailed how long we had the GPU clusters sitting around for training, which includes engineering issues, multiple seeds, and it was about two million dollars to rent the cluster to deal with the headaches of training a model. So these models are pretty accessible; a lot of people could get one to ten million dollars to train a model, but the recurring costs of serving millions of users is really billions of dollars of compute. I think you can look at a thousand GPU rental you can pay 100 grand a day for. And these companies could have millions of GPUs. You can look at how much these things cost to sit around. So that's kind of a big thing, and it's like, if scaling is actually giving you a better model, like is it going to be financially worth it? We will slowly push it out as AI solves more compelling tasks, like the likes of Claude 3 Opus, GPT-4.5, making Claude Code just work for things. I launched this project called the ATOM project, which is American Truly Open Models, in July, and that was like a true vibe-coded website, and I have a job to make plots and stuff. And then I came back to refresh it in the last few weeks, and it's like Claude 3 Opus versus whatever model at the time just crushed all the issues that it had from building in June and July. It might be a bigger model. A lot of things go into this, but there's still progress coming.
So what you're speaking to is the nuance of the Y-axis of the scaling laws, that the way it's experienced versus on a benchmark, the actual intelligence might be different. But still, your intuition about pre-training, if you scale the size of compute, will the models get better? Not whether it's financially viable, but just from the law aspect of it, do you think the models will get smarter?
Yeah. And I think that... And this sometimes comes off as almost like disillusioned from leadership at AI companies saying this, but they're like, 'It's held for 13 orders of magnitude of compute, so why would it ever end?' So I think fundamentally it is pretty unlikely to stop. Eventually we're not even going to be able to test the bigger scales because of all the problems that come with more compute. I think that there's a lot of talk on how 2026 is a year when very large Blackwell compute clusters, gigawatt-scale facilities for hyperscalers, are coming online. And these were all contracts for power and data centers that were signed and sought out in like 2022 and 2023, before or right after ChatGPT. So it took this two-to-three-year lead time to build these bigger clusters to train the models. While there's obviously immense interest in building even more data centers than that. So that is kind of the crux that people are saying: these new clusters are coming. The labs are gonna have more compute for training. They're going to utilize this, but it's not a given. I've seen so much progress that I expect it, and I expect a little bit bigger models. I would say it's more like we'll see a $2,000 subscription this year. We've seen $200 subscriptions. That could 10X again, and these are the kind of things that could come, and they're all downstream of this bigger model that offers just a little bit more cutting edge.
So, you know, it's reported that xAI is gonna hit that one gigawatt scale early '26, and full two gigawatts by year end. How do you think they'll utilize that in the context of scaling laws? Is a lot of that inference? Is a lot of that training?
It ends up being all of the above. So I think that all of your decisions when you're training a model come back to pre-training. If you're going to scale RL on a model, you still need to decide on your architecture that enables this. We were talking about other architectures and using different types of attention. We're also talking about a mixture of experts models. The sparse nature of MoE models makes it much more efficient to do generation, which becomes a big part of post-training. You need to have your architecture ready so that you can actually scale up this compute. I still think most of the compute is going in at pre-training. Because you can still make a model better, you still want to revisit this. You still want the best base model that you can. And in a few years that'll saturate and the RL compute will just go longer.
Are there people who disagree with you and say pre-training is dead? It's all about scaling inference, scaling post-training, scaling context, continual learning, synthetic data?
People vibe that way and describe it in that way, but I think it's not the practice that is happening.
It's just the general vibe of people saying this thing is dead—
The excitement is elsewhere. The low-hanging fruit—
—in RL is elsewhere. For example, we released our model in November... Every company has deadlines. Our deadline was November 20th, and our... For that, our run was five days, which compared to 2024 is a very long time to just be doing post-training at a model of 30 billion parameters. It's not a big model. And then in December, we had another release, which was just we let the RL run for another three and a half weeks, and the model got notably better, so we released it. That's a big amount of time to just allocate to something that is going to be your peak—...for the year. So it's like—
There's these types of decisions that happen when training a model where they just can't... They can't leave it forever. You have to keep pulling in the improvements you have from your researchers. So you redo pre-training, you'll do this post-training for a month, but then you need to give it to your users. You need to do safety testing. So it's just like... I think there's a lot in place that reinforces this cycle of updating the models. There's things to improve. You get a new compute cluster that lets you do something more stably or faster. It's like you hear a lot about Blackwell having rollout issues, where at AI2 most of the models we're pre-training are on 1,000 to 2,000 GPUs. But when you're pre-training on 10,000 or 100,000 GPUs, you hit very different failures. GPUs are known to break in weird ways, and on a 100,000 GPU run, you're pretty much guaranteed to always have at least one GPU that is down. And you need your training code to handle that redundancy, which is a very different problem. Whereas what we're doing like, 'Oh, I'm playing with post-training on DGX H100s,' or you have your book, or people learning ML, what they're battling to train these biggest models is just—...massive distributed scale, and it's very different. But that's somewhat different than whether these... Like, that's a systems problem—...in order to enable the scaling laws, especially at pre-training. You need all of these GPUs at once. When we shift to reinforcement learning, it actually lends itself to heterogeneous compute because you have many copies of the model. And to do a primer for language model reinforcement learning, what you're doing is you have two sets of GPUs. One you can call the actor and one you call the learner. The learner is where your actual reinforcement learning updates are done. These are traditionally policy gradient algorithms. Proximal Policy Optimization (PPO) and Group Relative Policy Optimization (GRPO) are the two popular classes. And on the other side you're going to have actors which are generating completions, and these completions are the things you're going to grade. Reinforcement learning is all about optimizing reward. In practice, what you can do is have a lot of different actors in different parts of the world doing different types of problems, and then you send it back to this highly networked compute cluster to do this actual learning where you take the gradients and you need to have a tightly meshed network where you can do different types of parallelism and spread out your model for efficient training. There's just a lot of... Every different type of training and serving has these considerations when you scale. We talked about pre-training, we talked about RL, and then inference time scaling is how do you serve a model that's thinking for an hour to 100 million users? I'm like, 'I don't really know about that, but I know that's a hard problem, and in order to give people this intelligence, there's all these systems problems, and we need more compute and more stable compute to do it.'
But you're bullish on all of these kinds of scaling is what I'm hearing. On the inference, on the reasoning, even on the pre-training?
Yeah, so that's a big can of worms, but there are basically two... The knobs are the training and the inference scaling where you can get gains, and so in a world where we had, let's say, infinite compute resources, you want to do all of them. So you have training, you have inference scaling, and training is like a hierarchy. It's pre-training, mid-training, post-training. Changing the model size, more training data, training a bigger model gives you more knowledge in the model. Then the model, let's say, has a better... It's like a better base model. Still, we call it a foundation model, and it unlocks... but you don't, let's say, have the model be able to solve your most complex tasks during pre-training or after pre-training. You still have these other unlock phases where you have mid-training or, for example, post-training with RL that unlocks capabilities that the model has in terms of just knowledge in the pre-training. And I think, sure, if you do more pre-training, you get a better base model that you can unlock later. But like Nathan said, it just becomes too expensive. We don't have infinite compute, so you have to decide: do I want to spend that compute more on making the model larger? It's like a trade-off. In an ideal world, you want to do all of them. And I think in that sense, scaling is still pretty much alive. You would still get a better model, but like we saw with GPT-4o, it's just not worth it. I mean, it's like... because you can unlock more performance with other techniques at this moment, especially if you look at inference scaling. That's one of the biggest gains this year with o1, where it took a smaller model further than pre-training a larger model like GPT-4o. So I wouldn't say pre-training scaling is dead, it's just that there are other more attractive ways to scale right now. But at some point, you will still want to make some progress on the pre-training. The thing that's also to consider is where you want to spend your money. If you spend it more on the pre-training, it's like a fixed cost. You train the model, and then it has this capability forever. You can always use it. With inference scaling, you don't spend money during training, you spend money later per query, and then it's also about the math: how long is my model going to be on the market if I replace it in half a year? Maybe it's not worth spending 5 million, 10 million, or 100 million dollars on training it longer. Maybe I will just do more inference scaling and get the performance from there. It maybe costs me two million in terms of user queries. It becomes a question of how many users you have and then doing the math, and I think that's also where it's interesting, where ChatGPT is in a position. I think they have a lot of users where they need to go a bit cheaper, where they have that GPT-4o model that is a bit smaller. Other companies that have... let's say if your customers have other trade-offs. For example, there was also the Math Olympiad or some of these math problems where ChatGPT or they had a proprietary model, and I'm pretty sure it's just like a model that has been maybe fine-tuned a little bit more, but most of it was during inference scaling to achieve this peak performance in certain tasks. You don't need that all the time. But yeah, long story short, I do think all of these pre-training, mid-training, post-training, inference scaling, they are all still things you want to do. It's just finding... At the moment, in this year, it's finding the right ratio that gives you the best bang for the buck, basically.
I think this might be a good place to define pre-training, mid-training, and post-training.
So, pre-training is the classic training: one next token prediction at a time. You have a big corpus of data. And Nathan probably also has very interesting insights there because of OLMo. A big portion of the paper focuses on the right data mix. So, pre-training is essentially just training a cross-entropy loss, training on next token prediction on a vast corpus of internet data, books, papers and so forth. It has changed a little bit over the years in the sense people used to throw in everything they can. Now, it's not just raw data. It's also synthetic data where people rephrase certain things. So synthetic data doesn't necessarily mean purely AI-made data. It's also taking something from an article, a Wikipedia article, and then rephrasing it as a Q&A question or summarizing it, rewording it, and making better data that way. Because I think of it also like with humans. If someone, let's say, reads a book compared to a messy—no offense, but, like, Reddit post or something like that, I do think you learn—
There's going to be a post about this, Raschka.
Some Reddit data is very coveted and excellent for training. You just have to filter it.
And I think that's the idea. It's like if someone took that and rephrased it in a, let's say, more concise and structured way—I think it's higher quality data that gets the LLM maybe the same... You get the same LLM out of it at the end, but it gets there faster. It trains faster, because if the grammar and the punctuation is correct, it already learns the correct way versus getting information from a messy way and then learning later how to correct that. So, I think that is how pre-training evolved and why scaling still works is that it's not about just the amount of data, it's also the tricks to make that data better for you, in a sense. And then mid-training is... I mean, it used to be called pre-training. I think it's called mid-training because it was awkward to have pre-training and post-training, middle, right? It sounds a bit weird. You have pre-training and post-training, but what's the actual training? So, the mid-training is usually similar to pre-training, but it's a bit more specialized than pre-training. It's the same algorithm, but what you do is you focus, for example, on long context documents. One reason you don't do that during just pre-training is because you don't have that many long context documents. We have a specific phase. And one problem of LLMs is also still... It's a neural network. It has the problem of catastrophic forgetting. So, you teach it something, it forgets other things. And you wanna... I mean, it's not 100% forgetting, but it's like no free lunch, you can't... It's also the same with humans. If you ask me some math I learned 10 years ago, I wouldn't know, I would have to look at it again.
Nathan was actually saying that he's consuming so much content that there's a catastrophic forgetting issue.
Yeah, I'm trying to learn so much about AI, and it's like I was learning about pre-training parallelism. I'm like, 'I lost something and I don't know what it was.'
I don't want to anthropomorphize LLMs, but I think it's the same kind of sense how humans learn. Quantity is not always better because you have to be selective. And mid-training is being selective in terms of quality content at the end. So the last thing the LLM sees is the quality stuff. And then post-training is all the fine-tuning, supervised fine-tuning, DPO, reinforcement learning with verifiable rewards, human feedback, and so forth—the refinement stages. And it's also interesting, the cost thing, right? For pre-training, you spend a lot of money on that right now. RL a bit less. RL, you don't really, I would say, teach it knowledge. It's more like unlocking the knowledge. It's more like a skill learning, like how to solve problems with the knowledge that it has from pre-training. There are actually three papers this year, or last year, 2024, on RL for pre-training. But I don't think anyone does that in production.
Toy, toy examples for now.
Toy examples, right. But to generalize, RL post-training is more like the skill unlock, where pre-training is like soaking up the knowledge essentially.
A few things that could be helpful for people. A lot of people think of synthetic data as being bad for training models. You mentioned the DeepSeek-V3 almost-OCR, which is Optical Character Recognition paper. A lot of labs did. AI2 had one, Nougat had multiple. And the reason each of these labs has these is because there are vast amounts of PDFs and other digital documents on the web that are in formats that aren't encoded with text easily. So you use this Almost CR, or DeepSeek OCR, that we called our Almost CR, to extract what can be trillions of tokens of candidate data for pre-training. And pre-training dataset size is on the order of trillions—it is measured in trillions of tokens. Smaller models from researchers can be something like 5 to 10 trillion. Qwen is documented going up to like 50 trillion, and there are rumors that these closed labs can go to like 100 trillion tokens. And just getting this potential data to put in, they have a very big funnel, and then the data you actually train the model on is a small percentage of this. This character recognition data would be described as synthetic data for pre-training in a lab. And then there's also the fact that ChatGPT now gives wonderful answers, and you can train on those best answers, and that's synthetic data. It's very different than, early ChatGPT, with lots of hallucination data, when people became grounded in synthetic data.
One interesting question is, if I recall correctly, OLMo-3 was trained with less data than specifically some other open-weight models, maybe even OLMo-2. But you still got better performance, and that might be one of the examples how the data helped.
It's mostly down to data quality. I think if we had more compute, we would train for longer. I think we'd see that as just something we would want to do. And especially with big models, you need more compute, because we talked about having more parameters and we talked about knowledge. Essentially, there's a ratio where big models can absorb more from data, and then you get more benefit out of this. It's like one of these. Any logarithmic graph in your mind is like a small model will level off sooner if you're measuring tons of tokens, and bigger models need more. But mostly, we aren't training that big of models right now at AI2, and getting the highest quality data we can is the natural starting point.
Is there something to be said about the topic of data quality? Is there some low-hanging fruit there still where the quality could be improved?
It's like turning the crank. Historically, in the open, there's been a canonical best pre-training dataset that has moved around between whoever has the most recent or best effort. Like AI2's Dolma was very early with the first OLMo, and Hugging Face had FineWeb. And there's a DCLM project, which has been like a—it stands for DataComp Language Model. There's been DataComp for other machine learning projects, and they had a very strong dataset. And a lot of it is the internet is becoming fairly closed off, so we have Common Crawl, which is hundreds of trillions of tokens, and you filter it. It looks like scientific work where you're training classifiers and making decisions based on, how do you prune down this dataset into the highest quality stuff and the stuff that suits your tasks? Previously, language models were tested a lot more on knowledge and conversational things, but now they're expected to do math and code. To train a reasoning model, you need to remix your whole dataset. And there are actually wonderful scientific methods here where you can take your gigantic dataset and sample a lot of really tiny things from different sources. So you say you have GitHub, Stack Exchange, Reddit, Wikipedia. You can sample small things from them, and train small models on each mix and measure their performance on your evaluations. You can just do basic linear regression, and it's like, 'Here's your optimal dataset.' But if evaluations change, your dataset changes a lot. So a lot of OLMo-3 was new sources for reasoning to be better at math and code, and then you do this mixing procedure and it gives you the answer. I think a lot of that's happened at labs this year, as there are new hot things, whether it's coding environments or web navigation, and you just need to bring in new data. You need to change your pre-training so that your post-training can work better. And that's like the constant evolution and the redetermining of what they care about for their models.
Are there fun anecdotes of what sources of data are particularly high quality that we wouldn't expect? You mentioned Reddit sometimes can be a source.
Reddit was very useful. I think that PDFs are definitely one.
Yeah, AI2 has run Semantic Scholar for a long time, which is what you can say is a competitor to Google Scholar with a lot more features. And to do this, AI2 has found and scraped a lot of PDFs for openly accessible papers that might not be behind the closed walled garden of a certain publisher. So, like, truly open scientific PDFs. And if you sit on all of these and you process it, you can get value out of it. And I think that a lot of that style of work has been done by the frontier labs much earlier. You need to have a pretty skilled researcher that understands how things change models, and they bring it in and clean it. It's a lot of labor. I think at frontier labs, when they scale researchers, a lot more goes into data. If you join a frontier lab and you want to have impact, the best way to do it is just find new data that's better. Rather than the fancy, glamorous algorithmic things like figuring out how to make o1. That's the sexiest thought of a scientist. 'Oh, I figured out how to scale RL.' There's a group that did that, but most of the contributions come from—
—saying, 'I'm gonna make the data better,' or, 'I'm gonna make the infrastructure better so that everyone on my team can run experiments 5% faster.'
At the same time, I think it's also one of the closest guarded secrets, what your training data is, for legal reasons. And so there's also, I think, a lot of work that goes into hiding what your training data was essentially—tuning the model to not give away the sources because you have legal reasons.
The other thing, to be complete, is that some people are trying to train on only licensed data, where Common Crawl is a scrape of, like, the whole internet. So if I host multiple websites, I'm happy to have them train language models, but I'm not explicitly licensing what governs it. Therefore, Common Crawl is largely unlicensed, which means that your consent really hasn't been provided for how to use the data. There's another idea where you can train language models only on data that has been licensed explicitly, so that the kind of governing contract is provided. I'm not sure if Appertus is the copyright thing or the license thing. I know that the reason that they did it was for an EU compliance thing, where they wanted to make sure that their model fit one of those checks.
On that note, there's also the distinction between the licensing. So some people, like you said, they just purchase the license. Let's say they buy a book online, like an Amazon Kindle book, or a Manning book, and then use that in training data. That is a gray zone, 'cause you paid for the content and you might want to train on it. But then there are also restrictions where even that shouldn't be allowed. That is where it gets a bit fuzzy. And I think that is right now still a hot topic, and also big companies like OpenAI, they approached private companies for their proprietary data and private companies are becoming more protective of their data because they know, 'Okay, this is going to be my moat in a few years.' And I do think that's the interesting question, where if LLMs become more commoditized, and a lot of people learn about LLMs, there will be a lot more people able to train LLMs. Of course, there are infrastructure challenges. But if you think of big industries like pharmaceutical industries, law, and finance industries, I do think they, at some point, will hire people from other frontier labs to build their in-house models on their proprietary data, which will be another unlock with pre-training that is currently not there. Because even if you wanted to, you can't get that data. You can't get access to clinical trials most of the time and these types of things. So, I do think scaling, in that sense, might be still pretty much alive if you also look in domain-specific applications, because we are still just looking at general purpose LLMs: ChatGPT, Anthropic, and so forth. They are just general purpose; they're not even scratching the surface of what an LLM can do if it is really specifically trained and designed for a specific task.
I think on the data thing, this is one of the things that happened in 2025, and we totally forget it, is Anthropic lost in court and owed $1.5 billion to authors. Anthropic, I think, bought thousands of books and scanned them and was cleared legally for that because they bought the books, and that is kind of going through the system. And then the other side, they also torrented some books, and I think this torrenting was the path where the court said that they were then culpable to pay these billions of dollars to authors, which is just such a mind-boggling lawsuit that kind of just came and went. That is so much money—so much money from the VC ecosystem.
These are court cases that will define the future of human civilization, because it's clear that data drives a lot of this, and there's this very complicated human tension of... I mean, you can empathize. You're both authors. And there's some degree to which, I mean, you put your heart and soul and your sweat and tears into the writing that you do. It feels a little bit like theft for somebody to train your data without giving you credit.
And there are, like Nathan said, also two layers to it. Someone might buy the book and then train on it, which could be argued fair or not, but then there are the straight-up companies who use pirated books where it's not even compensating the author. That is where people got a bit angry about it specifically, I would say.
Yeah, but there has to be some kind of compensation scheme. This is moving towards something like Spotify streaming did originally for music. You know, what does that compensation look like? You have to define those models and think through all of that. One other thing I think people are generally curious about, I'd love to get your thoughts. As LLMs are used more and more, if you look at even arXiv, but GitHub, more and more of the data is generated by LLMs. What do you do in that kind of world? How big of a problem is that?
Largest problem is the infrastructure and systems, but from an AI point of view, it's kind of inevitable.
So it's basically LLM-generated data that's curated by humans essentially, right?
Yes, and I think that a lot of open source contributors are legitimately burning out. If you have a popular open source repo, somebody's like, 'Oh, I want to do open source AI. It's good for my career,' and they just vibe code something and they throw it in. You might get more of this—
Yeah, so I actually have a case study here. I have a repository called mlxtend that I developed as a student, around 10 years ago, and it is a reasonably popular library still for certain algorithms, especially for frequent data mining stuff. And there were recently two or three people who submitted a lot of PRs in a very short amount of time. I do think LLMs have been involved in submitting these PRs. Me, as the maintainer, there are two things. First, I'm a bit overwhelmed. I don't have time to read through it because, especially as an older library, that is not a priority for me. At the same time, I also appreciate it because I think something people forget is it's not just using the LLM. There's still a human layer that verifies something, and that is in a sense also how data is labeled, right? One of the most expensive things is getting labeled data for reinforcement learning from human feedback phases. And this is kind of like that where it goes through phases, and then you get higher quality data out of it. And so I don't mind it in a sense. It can feel overwhelming, but I do think there is value in it.
It feels like there's a fundamental difference between raw LLM-generated data and LLM-generated data with a human in the loop that does some kind of verification, even if that verification is a small percent of the lines of code.
I think this goes with anything, where people think also sometimes, 'Oh, yeah. I can just use an LLM to learn about XYZ,' which is true. You can, but there might be a person who is an expert who might have used an LLM to write specific code. There is this human work that went into it to make it nice, throwing out the not so nice parts to kind of pre-digest it for you, and that saves you time. I think that's the value-add where you have someone filtering things or even using the LLMs correctly. This is still labor that you get for free. For example, reading an article—let's say a Substack article. I could maybe ask an LLM to give me opinions on that, but I wouldn't even know what to ask. I think there is still value in reading that article compared to me going to the LLM because you are the expert. You select what knowledge is actually spot on, what should be included, and you give me this very executive summary. This is a huge value-add because now I don't have to waste three or five hours to go through this myself, maybe get some incorrect information and so on. So I think that's also where the future still is for writers, even though there are LLMs that can save you time.
It's fascinating to actually watch—and I'm sure you guys do this—but for me to look at the difference between a summary and the original content. Even if it's a page-long summary of a page-long content, it's interesting to see how the LLM-based summary takes the edge off. Like, is the signal it removes from the thing?
The voice is what I talk about a lot.
Voice? I would love to hear what you mean by voice, but sometimes there are literally insights. By removing an insight, you're fundamentally changing the meaning of the thing. I'm continuously disappointed how bad LLMs are at really getting to the core insights, which is what a great summary does. Yet even if I have these extensive extremely elaborate prompts where I'm like really trying to dig for the insights, it's still not quite there. I mean, that's a whole deep philosophical question about what is human knowledge and wisdom and what does it mean to be insightful and so on. But when you talk about the voice, what do you mean?
So when I write, I think a lot of what I'm trying to do is take what you think as a researcher, which is very raw. A researcher is trying to encapsulate an idea at the frontier of their understanding and put what is a feeling into words. I try to do this with my writing, which makes it come across as raw but also high-information in a way that some people will get and some won't. That's the nature of research. I think this is something language models don't do well. They're all trained with RLHF, which is designed to take feedback from a lot of people and average how the model behaves. And I think that it's going to be hard for a model to be very incisive when there's that sort of filter in it. This is a wonderful, fundamental problem for researchers in RLHF: provides so much utility in making the models better, but also the problem formulation has this knot in it that you can't get past. These language models don't have this prior in their deep expression that they're trying to get at. I don't think it's impossible. There are stories of models that really shock people. I would have loved to have tried Bing Sydney. Did that have more voice? Because it would often go off the rails on people in what is historically obviously, a scary way—like telling a reporter to leave his wife—is crazy to potentially put in general adoption. But that's kind of like a trade-off. Is this RLHF process in some ways adding limitations?
That's a terrifying place to be as one of these frontier labs and companies, because millions of people are using them.
There was a lot of backlash last year with GPT-4 getting removed. I've never used the model, but I've talked to people at OpenAI where they get emails from users that might be detecting subtle differences in the deployments, even in the middle of the night. They're like, 'My friend is different.' They find employees' emails because they are so attached to this set of model weights and a configuration that is deployed. We see this with TikTok. I don't use TikTok, but supposedly in five minutes the algorithm gets you. It's like it's locked in. Those are language models doing recommendations. I think there are ways that you can do this. Within five minutes of chatting, the model just gets you. And that is something that people aren't really ready for. I think—don't give that to kids... at least until we know what's happening.
There's also going to be this mechanism. What's going to happen with these LLMs, as they're used more and more... Unfortunately, the nature of the human condition is such that people commit suicide. Journalists will report extensively on the people who commit suicide, and they would very likely link it to the LLMs because they have that data about the conversations. If you're really struggling, if you're depressed, if you're thinking about suicide, you're probably going to talk to LLMs about it. And so journalists will say, 'Well, the suicide was committed because of the LLM.' And that's going to lead to the companies, because of legal issues, taking the edge off the LLM. So it's going to be as generic as possible. It's so difficult to operate in this space because you don't want an LLM to cause harm to humans at that level, but this is also the nature of the human experience: to have a rich conversation, a fulfilling conversation, one that challenges you and from which you grow. You need that edge. And that is something extremely difficult for AI researchers on the RLHF front to solve, because you're actually dealing with the human condition.
A lot of researchers at these companies are so well-motivated, Anthropic and OpenAI culturally want to do good for the world. And it's such a... I'm like, 'I don't want to work on this,' because a lot of people see AI as a health ally, as someone they can talk to about their health confidentially, but then it bleeds into talking about mental health, where it's heartbreaking that this will be the thing where somebody goes over the edge, but other people might be saved. And I'm like, 'I don't...' There's things that, as a researcher training models, I don't want to train image generation models and release them openly because I don't want to enable somebody to have a tool on their laptop that can harm other people. I don't have the infrastructure in my company to do that safely. But... There's a lot of areas like this where it just needs people that will approach it with the complexity and just kind of conviction because it's just such a hard problem.
But also, we as a society, as users of these technologies, need to ensure that we're having the complicated conversation about it versus just fearmongering. 'Big tech is causing harm to humans,' or 'stealing your data,' all that kind of stuff. It's more complicated than that. And you're right. There's a very large number of people inside these companies, many of whom you know, many of whom I know, that deeply care about helping people. They are considering the full human experience of people from across the world, not just Silicon Valley. People across the United States, people across the world—what that means, what their needs are. It's really difficult to design this one system that is able to help all these different kinds of people across different age groups, cultures, mental states, and conditions.
I wish that the timing of AI was different regarding the relationship of Big Tech to the average person. Big Tech's reputation was so low, and with how AI is so expensive, it's inevitably going to be a Big Tech thing. It takes so many resources, and people say the US is, quote-unquote, 'betting the economy on AI' with this build-out. To have these be intertwined at the same time makes for such a hard communication environment. It would be good for me to go talk to more people in the world that hate Big Tech and see AI as a continuation of this.
And one of the things you recommend... One of the antidotes that you talk about is to find agency in this whole system. As opposed to sitting back in a powerless way and consuming the AI slop as it quickly, rapidly takes over the internet. Find agency by using AI to build stuff, build apps... So, one, that actually helps you build intuition, but, two, it's empowering because you can understand how it works and what the weaknesses are. It gives your voice the power to say, 'This is bad use of the technology, and this is good use of technology.' And you're more plugged into the system, so you can understand it better and you can steer it better as a consumer.
I think that's a good point about agency. Instead of ignoring it and saying, 'Okay, I'm not going to use it,' I think it's probably long-term healthier to say, 'Okay, it's out there. I can't put it back.' Like the internet and computers back when they came out. How do I make best use of it, and how does it help me to up-level myself? The one thing I worry about, though, is if you just fully use it for something you love to do, the thing you love to do is no longer there. And that could potentially lead to burnout. For example, if I use an LLM to do all my coding, now there's no coding. I'm just managing something that is coding for me. Two years later, if I just do that eight hours a day, have something code for me, do I still feel fulfilled? I mean, is this hurting me in terms of being excited about my job and what I'm doing? Am I still proud to build something?
So on that topic of enjoyment, it's quite interesting. We should just throw this in there, that there's this recent survey of about 791 professional developers, professional meaning 10 plus years of experience.
That's a long time. As a junior developer?
Yeah, in this day and age. So there are also aspects on many fronts that are surprising. So they break it down by junior and senior developers. But it just shows that both junior and senior developers use AI-generated code in code they ship. So this is not just for fun or intermediate learning things. This is code they ship. And so 25%—like, most of them use around 50% or more. And what's interesting is, for the category of over 50% of your code that you ship is AI generated, senior developers are much more likely to do so. But you don't want AI to take away the thing you love. I think this speaks to my experience—these particular results I'm about to say. Together, about 80% of people find it either somewhat more enjoyable or significantly more enjoyable to use AI as part of their work.
I think it depends on the task. From my personal usage, for example, I have a website where I sometimes tweak things on the website. I personally don't enjoy this. So in that sense, if the AI can help me to implement something on my website, I'm all for it. It's great. But then, at the same time, when I solve a complex problem, if there's a bug and I hunt this bug, and I find it, it's the best feeling in the world. You get so much joy. You feel great. But now if you don't even think about the bug and you just go directly to the LLM, well, you never have this kind of feeling, right? But then there could be the middle ground where you try yourself, you can't find it, you use the LLM, and then you don't get frustrated because it helps you move on to something that you enjoy. And so looking at these statistics, I think also the difference is what is not factored in; it's averaging over all the different scenarios. We don't know if it's for the core task or if it's for something mundane that people would not have enjoyed otherwise. So, in a sense, AI is really great for doing mundane things that take a lot of work. So for example, my wife the other day, she has a podcast for book discussions, a book club, and she was transferring the show notes from Spotify to YouTube, and then the links somehow broke. And she had in some episodes, because there are so many books, like 100 links, and it would have been really painful to go in there and fix each link manually. So I suggested, 'Hey, let's try ChatGPT.' We copied the text into ChatGPT, and it fixed them. And instead of two hours going from link to link, it made that type of work much more seamless. I think everyone has the use case where AI is useful for something that would be really boring, really mundane.
For me personally, since we're talking about coding, and you mentioned debugging... A lot of the source of the enjoyment for me, more on the Cursor side than the Cloud side, is that I have a friend, I have a—what's that called? A pair programmer. It's less lonely. You made debugging sound like this great joy. I would say debugging is like a drink of water after you've been going through a desert for days. So you skip the whole desert part where you're suffering. Sometimes it's nice to have a friend who can't really find the bug, but can give you some intuition about the code, and you're together with that friend going through the desert, and then together find that drink of water. At least for me, maybe it speaks to the loneliness of the programming experience. That is a source of joy.
It's maybe also related to delayed gratification. Even as a kid, I liked the idea of Christmas presents—having them, looking forward to them—better than actually getting them. I would look forward to the day, but then it's over and I'm disappointed. It's similar to food, I think. Food tastes better when you're really hungry. And you're right with debugging, it is not always great. It's often frustrating, but then if you can solve it, then it's great. There's a sweet Goldilocks zone; if it's too hard, then you're wasting your time. But I think that is another challenge: How will people learn? In the chart we looked at, we saw that more senior developers are shipping more AI-generated code than the junior ones. It's very interesting, because intuitively you would think it's the junior developers, because they don't know how to do the thing yet. It could mean the AI is not good enough yet, or it could also mean experts are more effective at using it. They know how to use it better, they review the code, and they trust it more. One issue in the future will be: How do you become an expert if you never try to do the thing yourself? The way I learned is by trying things myself. Like math textbooks: if you just look at the solutions, you learn something, but I think you learn better if you try first. You appreciate the solution differently because you can put it into your mental framework. If LLMs are here all the time, would you actually go through the length of struggling? Struggle is not nice, right? But if you use the LLM to do everything, at some point you will never really take the next step and then you will maybe not get that unlock that you would get as an expert using an LLM. So, I think there's a Goldilocks sweet spot where maybe the trick here is you make dedicated offline time where you study two hours a day, and the rest of the day use LLMs. But I think it's important also for people to still invest in themselves, in my opinion, to not just LLM everything.
Yeah, there is—we as a civilization—that we each individually have to find that Goldilocks zone. And in the programming context as developers. Now, we've had this fascinating conversation that started with pre-training and mid-training. Let's get to post-training. A lot of fun stuff in post-training. So, what are some of the interesting ideas in post-training?
The biggest one from 2025 is learning this reinforcement learning with verifiable rewards. You can scale up the training there, which means doing a lot of this kind of iterative generate-grade loop, and that lets the models learn both interesting behaviors on the tool use and software side. This could be searching, running commands on their own and seeing the outputs, and then also that training enables this inference time scaling very nicely. And it just turned out that this paradigm was very nicely linked, where this kind of RL training enables inference time scaling. But inference time scaling could have been found in different ways. So, it was a perfect storm where models change a lot, and the way that they're trained is a major factor in doing so. And this has changed how people approach post-training... dramatically.
Can you describe RLVR, popularized by DeepSeek R1? Can you describe how it works?
Yeah. Fun fact, I was on the team that came up with the term RLVR, which is from our Tulu 3 work before DeepSeek. We don't take credit for being the people to popularize the scaling RL, but as academics get, as an aside, is the ability to name and influence the discourse because the closed labs can only say so much. One of the things you can do as an academic is you might not have the compute to train the model, but you can frame things in a way that ends up being, I describe it as: a community can come together around this RLVR term, which is very fun. And then DeepSeek are the people that did the training breakthrough, which is they scaled the reinforcement learning where you have the model generate answers and then grade the completion if it was right, and then that accuracy is your reward for reinforcement learning. So reinforcement learning is classically an agent that acts in an environment, and the environment gives it a state and a reward back, and you try to maximize this reward. In the case of language models, the reward is normally accuracy on a set of verifiable tasks, whether it's math problems, coding tasks. And it starts to get blurry with things like factual domains. Like that is also, in some ways, verifiable or constraints on your instruction, like respond only with words that start with A. All of these things are verifiable in some way, and the core idea of this is you find a lot more of these problems that are verifiable and you let the model try it many times while taking these RL steps, these RL gradient updates. The infrastructure evolved from Reinforcement Learning from Human Feedback, where the score they were trying to optimize was a learned reward model of aggregate human preferences. So you change the problem domains and that let the optimization go on to much bigger scales, which kind of kickstarted a major change in what the models can do and how people use them.
What kind of domains is RLVR amenable to?
Math and code are the famous ones, and then there's a lot of work on what is called the rubrics, which is related to a word people might have heard is LLM-as-a-judge, which is like for each problem, I'll have a set of problems in my training dataset. I will then have another language model and ask it, 'What would a good answer to this problem look like?' And then you could try the problem over and over again and assign a score based on this rubric. So that's not necessarily verifiable like a math and code domain, but this rubrics idea and other scientific problems that might be a little bit more vague is where a lot of the attention is, where they're trying to push this set of methods into these kind of more open-ended domains so the models can learn a lot more.
I think that's called Reinforcement Learning with AI Feedback, right?
That's the older term from it that was coined in Anthropic's Constitutional AI paper. So it's like a lot of these things come in cycles.
Also, just one step back for the RLVR. I think the interesting thing here is that you ask the LLM, let's say, a math question, and then you know the correct answer, and you let the LLM, like you said, figure it out, but how it does it... I mean, you don't really constrain it much. There are some constraints you can add like 'use the same language, don't switch between Spanish and English.' But let's say you're pretty much hands-off. You only give the question and the answer, and then the LLM has to—you know, just the task to arrive at the right answer—but the beautiful thing here is what happens in practice is that the LLM will do a step-by-step description, like a student or a mathematician, how you would derive the solution. It will give you, or it will use those steps, and that actually helps the model to improve its own accuracy. And then like you said, the inference scaling. So inference scaling loosely means basically spending more compute during using the LLM during inference, and here the inference scaling is that the model would use more tokens. In the R1 paper, they showed the longer they train the model, the longer the responses are. They grow over time. They use more tokens, so it becomes more expensive for simple tasks, but these explanations, they help the model with the accuracy. There are also interesting papers showing what the model explains does not necessarily have to be correct or maybe it's even unrelated to the answer, but for some reason, it still helps the model. It's the fact that it is explaining. And I think it's also, again, I don't want to anthropomorphize these LLMs, but it's kind of like how we humans operate, right? If there's a complex math problem, let's say in a math class, you usually have a note paper and you do it step by step. You cross out things. And the model also self-corrects, and that was, I think, the aha moment in the R1 paper. They called it the aha moment because the model itself recognized it made a mistake and then said, 'Ah, I did something wrong, let me try again.' And I think that's just so cool that this falls out of just giving it the correct answer and having it figure out how to do it, that it kind of does in a sense what a human would do. Although LLMs don't think like humans, it's kind of like an interesting coincidence and it... And the other nice side effect is it's great for us humans often to see these steps. It builds trust, but also we learn we can double check things.
There's a lot in here. I think—There's been a lot of debate this year on if the language models like these aha moments... I think the aha moments are kind of fake because in pre-training you essentially have seen the whole internet. So you have definitely seen people explaining their work, even verbally, like a transcript of a math lecture. 'You try this, oh, I messed this up.' And what reinforcement learning, this RLVR is very good at doing, is amplifying these behaviors because they're very useful in enabling the model to think longer and to check its work. And I agree that it is very beautiful that this training kind of... The model learns to amplify this in a way that is just so useful at the final answers being better.
I can give you also a hands-on example. I was training the Qwen 2.5 base model with RLVR on MATH 500. The base model had an accuracy of about 15%. Just 50 steps, like in a few minutes with RLVR, the model went from 15% to 50% accuracy. And the model... You can't tell me it's learning anything about, fundamentally about math in—
The Qwen example is weird because there've been two papers this year, one of which I was on, about data contamination in Qwen—specifically that they train on a lot of this special mid-training phase that we—
—that we spent a minute on because it's weird—
And so, because they train on problems that are almost identical to MATH.
Exactly. And so you can see that basically the RL, it's not teaching the model any new knowledge about math. You can't do that in 50 steps. So the knowledge is already there, in the pre-training, you're just unlocking it.
I still disagree with the premise because there's a lot of weird complexities that you can't prove because... One of the things that points to weirdness is that if you take the Qwen 2.5 so-called base model and you... You could Google like 'math dataset, Hugging Face,' and you could take a problem and what you do if you put it into Qwen 2.5 base, all these math problems have words, so it'd be like 'Alice has five apples and takes one and gives three to whoever,' and there are these word problems. Why people are suspicious of these Qwen-based models is if you change the numbers but keep the words, Qwen will produce, without tools, a very high accuracy decimal representation of the answer, which means at some point it was shown problems that were almost identical to the test set, and it was using tools to get a very high precision answer, but a language model without tools will never actually have this. So it's kind of been this big debate in the research community. How much of these reinforcement learning papers that are training on Qwen and measuring specifically on this math benchmark where there's been multiple papers talking about contamination, how much can you believe them? I think this is what caused the reputation of RLVR being about formatting, because you can get these gains so quickly and therefore it must already be in the model. But there's a lot of complexity here; it's not really like controlled experimentation, so we don't really know.
But if it weren't true, I would say distillation wouldn't work, right? I mean, distillation can work to some extent, but the thing is, the biggest problem is research contamination, because we don't know what's in the data. Unless you have a new dataset, it is really impossible. And the same, you mentioned the MATH dataset, where you have a question, an answer, and an explanation given, but then also even something simpler like MMLU, which is a multiple-choice benchmark. If you just change the format slightly, like, I don't know, you use a dot instead of a parenthesis or something like that, the model accuracy will vastly differ.
I think that could be a model issue rather than a general issue.
It's not even malicious by the developers of the LLM, like, 'Hey, we want to cheat at that benchmark.' It has just seen something at some point. I think the only fair way to evaluate an LLM is to have a new benchmark that is after the cutoff date when the LLM was deployed.
Can we lay out what would be the recipe of all the things that go into post-training? And you mentioned RLVR was a really exciting, effective thing. Maybe we should elaborate. RLHF still has a really important component to play. What kind of other ideas are there on post-training?
I think you can take this in order. You could view it as what made o1, which is this first reasoning model, possible or what will the latest model be. They actually have similar interventions at these stages, where you start with mid-training, and the thing that is rumored to enable o1 and similar models is really careful data curation, where you're providing a broad set of what are called reasoning traces, which is just the model generating words in a forward process that is reflecting, like breaking down a problem into intermediate steps and trying to solve them. At mid-training, you need to have data similar to this so that when you move into post-training, primarily with these verifiable rewards, it can learn. And then what is happening today is you're figuring out which problems to give the model and how long you can train it for and how much inference you can enable the model to use when solving these verifiable problems. So as models get better, certain problems are no longer... The model will solve them 100% of the time, and therefore there's very little signal in this. If we look at the GRPO equation, this one is famous for this because essentially the reward given to the agent is based on how good a given action, a completion, is relative to the other answers to that same problem. So if all the problems get the same answer, there's no signal. So they're finding harder problems, which is why you hear about things like scientific domains, which are so hard. Getting anything right there, if you have a lab or something, it just generates so many tokens, or much harder software problems. So the frontier models are all pushing into these harder domains where they can train on more problems and learn more skills at once. The RLHF link to this is that RLHF has been and still is like the finishing touch on the models, where it makes them more useful by improving the organization or style or tone. Different things resonate with different audiences; some people like a quirky model, and RLHF could be good at enabling that personality, and some people hate this markdown bulleted list thing that the models do, but it's actually really good for quickly parsing information. In RLHF, this human feedback stage is really great for putting this into the model at the end of the day. It's what made ChatGPT so magical for people. And that use has actually remained fairly stable. This formatting can also help the models get better at math problems, for example. So it's like the border between style and formatting, and the method that you use to answer a problem, is actually, they're all very closely linked in terms of when you're training these models, which is why RLHF can still make a model better at math, but these verifiable domains are a much more direct process for doing this because this makes more sense with the problem formulation, which is why it all ends up forming together. But to summarize: Mid-training is giving the model the skills it needs to then learn. RL with verifiable rewards is letting the model try many times, putting a lot of compute into trial-and-error learning across hard problems. And then RLHF would be: finish the model, make it easy to use, and kind of just round the model out.
Can you comment on the amount of compute required for RLVR?
It's only gone up and up. I think Greg Brockman was famous for saying they use a similar amount of compute for pre-training and post-training. Back to the scaling discussion, they involve very different hardware for scaling. Pre-training is very compute-bound, which is like this FLOPs discussion, which is just how many matrix multiplications can you get through in one time. And because RL, you're generating these answers, you're trying the model in the real-world environments, it ends up being much more memory-bound because you're generating long sequences and the attention mechanisms have this behavior where you get a quadratic increase in memory as you're getting to longer sequences. So the compute becomes very different. When in pre-training we talk about a model, I think if we go back to the Biden administration executive order, it's like 10 to the 25th FLOPs to train a model. If you're using FLOPs in post-training, it's a lot weirder because the reality is just like how many hours are you allocating, how many GPUs for? And I think in terms of time, the RL compute is getting much closer because you just can't put it all into one system. Pre-training is so computationally dense where all the GPUs are talking to each other and it's extremely efficient, where RL has all these moving parts and it can just take a long time to generate a sequence of a hundred thousand tokens. If you think about GPT-4o taking an hour, it's like, what if your training run has to sample for an hour and you have to make sure that's handled efficiently? So I think in GPU hours or just wall clock hours, the RL runs are probably approaching the number of days as pre-training, but they probably aren't using as many GPUs at the same time. There are rules of thumb where in labs you don't want your pre-training runs to last more than a month because they fail catastrophically. And if you are planning a huge cluster to be held for two months and then it fails on day 50, the opportunity costs are just so big. So people don't want to put all their eggs in one basket, which is like... GPT-4 was the ultimate YOLO run and nobody ever wanted to do it before where it took three months to train and everybody was shocked that it worked. I think people are a little bit more cautious and incremental now.
So RLVR is more, let's say, unlimited in how much you can train and get benefit, where RLHF, because it's preference tuning, you reach a certain point where it doesn't really make sense to spend more RL budget on that. So just taking a step back with preference tuning: there are multiple people that can give multiple explanations for the same thing and they can both be correct, but at some point you learn a certain style and it doesn't make sense to iterate on it. My favorite example is if relatives ask me what laptop they should buy, I give them an explanation or ask them, 'What is your use case?' Like they, for example, might prioritize battery life and storage. Other people like us, for example, we would prioritize RAM and compute. And so... but both answers are correct, but different people require different answers. And with preference tuning, you're trying to average somehow, like you are asking the data labelers to give you the preferred answer and then you train on that. But at some point, you learn that average preferred answer. And there's no reason to keep training longer on it because it's just a style, whereas with RLVR you literally let the model solve more and more complex, difficult problems. And so I think it makes more sense to allocate more budget long-term to RLVR. Right now we are in RLVR 1.0 land where it's still like that simple thing where we have a question and answer, but we don't do anything with the stuff in between. So there were multiple research papers, also by Google for example, on process reward models that also give scores for the explanation, how correct is the explanation. And I think that will be the next thing, let's say RLVR 2.0 for this year, focusing in between question and answer, like how to leverage that information to improve the explanation and help it to get better accuracy. But then... So that's one angle. And there was a DeepSeek Math-V2 paper where they also had interesting inference scaling there where, first, they had developed models that grade themselves, a separate model. And I think that will be one aspect. And the other, like Nathan mentioned, it will be for RLVR branching into other domains.
The place where people are excited are value functions, which is pretty similar. So process reward models are kind of like... Process reward models assign how good something is to each intermediate step in a reasoning process, where value functions apply value to every token the language model generates. Both of these have been largely unproven in the language modeling and this reasoning model era. People are more optimistic about value functions for whatever reason now. I think process reward models were tried a lot more in this pre-o1, pre-reasoning model era, and a lot of people had a lot of headaches with them. So I think a lot of it is the human nature of like... Value models have a very deep history in reinforcement learning. They're one of the first things that were core to deep reinforcement learning existing, is training value models. So right now in the literature, people are excited about trying value models, but there's very little proof in it. And there are negative examples in trying to scale up process reward models. These things don't always hold in the future. We came to this discussion by talking about scaling. And the simple way to summarize what you're saying is you don't want to do too much RLHF, which is essentially that the signal scales. People have worked on RLHF for language models for years, especially an intense interest after ChatGPT. And the first release of a reasoning model trained with RLVR, OpenAI's o1, had a scaling plot where if you increase the training compute logarithmically, you get a linear increase in evaluations, and this has been reproduced multiple times. I think DeepSeek had a plot like this. But there's no scaling law for RLHF where if you log-increase the compute, you get some performance. In fact, the seminal scaling paper for RLHF is Scaling Laws for Reward Model Overoptimization. So it's like that's a big line to draw with RLVR and the methods we have now and in the future; they will follow this scaling paradigm, which is like the best runs you can let run for an extra 10x and you get a few x performance, but you can't do this with RLHF. And that is just going to be field-defining in how people approach them. While I'm a shill for people academically to do RLHF, a good way to describe it is like: to do the best RLHF you might not need the extra 10 or 100x of compute, but to do the best RLVR you do. So I think there's a seminal paper from a Meta internship. It's called something like The Art of Scaling Reinforcement Learning with Language Models. What they describe as a framework is ScaleRL. And their incremental experiment was like 10,000 V100 hours, which is like thousands or tens of thousands of dollars per experiment, and they do a lot of them. This cost is just not accessible to the average academic, which is a hard equilibrium where we're trying to figure out how to learn from each community.
I was wondering if we could take a bit of a tangent and talk about education and learning. If you're someone listening to this who's a smart person interested in programming and AI, I presume building something from scratch is a good beginning. Can you take me through what you would recommend people do?
I would personally start by implementing a simple model from scratch that you can run on your computer. The goal is not to have something you use every day for your personal projects. It's not going to be a personal assistant replacing an existing open-weight model or ChatGPT. It's to see exactly what goes into the LLM, what comes out, and how the pre-training works, preferably on your own computer. Then you learn about pre-training, supervised fine-tuning, and the attention mechanism. You get a solid understanding of how things work, but at some point you'll reach a limit because home models can only do so much. The problem with learning about LLMs at scale is that it's exponentially more complex to make a larger model because the model doesn't just become larger. You have to think about sharding your parameters across multiple GPUs. Even for the KV-cache, there are multiple ways to implement it. One way to understand how it works is just to grow the cache step-by-step by, let's say, concatenating lists, but then that wouldn't be optimal on GPUs. You would pre-allocate a tensor and then fill it in. But that adds another 20 or 30 lines of code. For each thing you add code, and the trick with the book is to understand how the LLM works. It's not going to be a production-level LLM, but once you have that, you can understand the production-level LLM.
So you're trying to always build an LLM that's going to fit on one GPU?
Yes. Most of them do. I have some bonus materials on some MoE models. One or two of them may require multiple GPUs, but the goal is to have it on one GPU. And the beautiful thing is you can self-verify. It's almost like RLVR. When you code these from scratch, you can take an existing model from the Hugging Face Transformers library. That library is great, but if you want to learn about LLMs, I don't think that's the best place to start because the code is so complex; it has to fit so many use cases. Since people use it in production, it's really sophisticated and intertwined. It's not linear to read.
It started as a fine-tuning library and then grew to be the standard representation of every model architecture. Hugging Face is the default place to get a model, and Transformers is the software that enables it so people can easily load a model and do something basic with it.
And all frontier labs that have open-weight models have a Transformers version of it, like from DeepSeek to GPT-2. That's the canonical weight that you can load there. But even Transformers, the library, is not used in production. People use SGLang or vLLM, and it adds another layer of complexity.
We should say that the Transformers library has like 400 models.
So it's the one library that tries to implement a lot of LLMs and so you have a huge codebase, basically. It's like huge. It's like, I don't know, maybe millions... hundreds of thousands of lines of code. Understanding the part you want to understand is finding the needle in the haystack. But what's beautiful is you have a working implementation and so you can work backwards from it. What I would recommend doing, or what I also do, is if I want to understand, for example, how Llama 3 is implemented, I would look at the weights in the model hub, the config file, and then you can see, 'Oh, they used so many layers.' They use, let's say, Group Query Attention or Multi-Head Attention in that case. Then you see all the components in like a human-readable 100 lines of config file. And then you start, let's say, with your GPT-2 model and add these things. The cool thing here is you can then load the pre-trained weights and see if they work in your model. And you want to match the same output that you get with a Transformer model, and then you can use that as a basically as a verifiable reward to make your architecture correct. And then it's kind of... sometimes it takes me a day. With Llama 3, the challenge was RoPE for the position embeddings. They had a YaRN extension, and there was some custom scaling there and I couldn't quite match these things, and in this struggle you kind of understand things. But at the end, you know you have it correct because you can unit test it. You can check against the reference implementation, and I think that's one of the best ways to learn really. To basically reverse-engineer something.
I think that is something that everybody interested in getting into AI today should do. I think that's why I liked your book. I came to language models from this RL and robotics field. I had never taken the time to just learn all the fundamentals, and this Transformer architecture I described as being so fundamental, like deep learning was a thing that I had to learn in the past, and people need to do this. I think where a lot of people kind of get overwhelmed is, 'How do I apply this to have impact or find a career path?' Because AI language models make this fundamental stuff so accessible, and people with motivation will learn it, and then it's like, 'How do I get the cycles on goal to contribute to research?' I'm actually fairly optimistic because the field moves so fast that a lot of times the best people don't fully solve a problem because there's a bigger problem to solve that's very low-hanging fruit, so they move on. And I think that a lot of what I was trying to do in this RLHF book is take post-training techniques and describe how people think about them influencing the model, and what people are doing. It's remarkable how many things I think people just stop studying. So I think people trying to get narrow after doing the fundamentals is good, and then reading the relevant papers and being engaged in the ecosystem, it's like you... The proximity that random people have online from the leading researchers... Like, no one knows who all the... Like, the anonymous account on X in ML is very popular for whatever reason, and no one knows who all these people are. It could just be random people that study the stuff deeply, especially with the AI tools to just be like, 'I don't understand this, keep digging into it,' I think is a very useful thing. But there's a lot of research areas that just like are maybe three papers that you need to read and then one of the authors will probably email you back. But you have to put in a lot of effort into these emails to understand the field. Like I think it would be for a newcomer easily weeks of work to feel like they can truly grasp what is a very narrow area, but I think going narrow after you have the fundamentals will be very useful to people because it's like I became very interested in character training, which is like how you make the model funny or sarcastic or serious and like what do you do to the data to do this? And it's like a student at Oxford reached out to me and was like, 'Hey, I'm interested in this,' and I advised him. And that paper now exists, and it's like, I don't know, there's like two or three people in the world that were very interested in this. He's a PhD student, which gives you an advantage, but like, for me, that was a topic I was waiting for someone to be like, 'Hey, I have time to spend cycles on this,' and I'm sure there's a lot more very narrow things where you're just like, 'Oh, it doesn't make sense that there was no answer to this,' and I think that it's just like there's so much information coming that people are like, 'I can't grab onto any of these,' but if you just actually stick in an area, I think there's a lot of interesting things to learn.
Yeah, I think you can't try to do it all because it would be very overwhelming and you would burn out if you tried to keep up with everything. For me, I haven't kept up with computer vision in a long time, just focused on LLMs. But coming back to your book, I think this is also a really great book and a really good bang for the buck because you want to learn about RLHF. I wouldn't go out there and read RLHF papers because you would be spending two years...
Some of them contradict. I've just edited the book, and I was like, there's no chapter where I had to be like, 'X papers say one thing and X papers say another thing, and we'll see what comes out to be true.'
Just to go through some of the table of contents, some of the ideas we might have missed in the bigger picture of post-training. So first of all, you did the problem setup, training overview, what are preferences, preferences data and the optimization tools, reward modeling, regularization, instruction tuning, rejection sampling, reinforcement learning. Then constitutional AI and AI feedback, reasoning and inference time scaling to use in function calling, synthetic data and distillation, evaluation and then open questions section, over-optimization, style and information, and then product UX, character and post-training. So what are some ideas worth mentioning that connect both the educational component and the research component? You mentioned the character training, which is pretty interesting.
Character training is interesting because there's so little out there, but we talked about how people engage with these models and how we feel good using them because they're positive. But that can go too far; it can be too positive. And it's essentially: how do you change your data and decision-making to make it exactly what you want? Like, OpenAI has this thing called a Model Spec, which is essentially their internal guideline for what they want the model to do, and they publish this to developers. So essentially, you can know what is a failure of OpenAI's training, where they have intentions they haven't met yet, versus what is something they actually wanted to do and that you don't like. And that transparency is very nice, but all the methods for curating these documents and how easy it is to follow them is not very well known. I think the way the book is designed is that the RL chapter is obviously what people want because everybody hears about it with RLVR, and it's the same algorithms and the same math, but it's just that you can use it in very different documents. So I think the core of RLHF is how messy preferences are. It's essentially a rehash of a paper I wrote years ago, but this is the chapter that'll tell you why RLHF is never ever fully solvable, because the way that even RL is set up assumes that preferences can be quantified and that multiple preferences can be reduced to single values. And I think it relates in economics literature to the Von Neumann-Morgenstern utility theorem. That is the chapter where all of that philosophical, economic, and like psychological context tells you what gets compressed into doing RLHF. So you have all of this, and later in the book, it's like: you use this RL math to make the number go up. I think that's why it'll be very rewarding for people to do research on, because quantifying preferences is something that humans have designed a problem for in order to make preferences studyable. But there's fundamental debates like, for example, in a language model response, you have different things you care about, whether it's accuracy or style. And when collecting data, they all get compressed into, 'I like this more than another.' That is happening, and there's a lot of research in other areas of the world regarding how you should actually do this. I think social choice theory is the subfield of economics around how you should aggregate preferences. And I went to a workshop that published a white paper on, 'How can you think about using social choice theory for RLHF?' So I want people that get excited about the math to come and stumble into and learn this kind of broader context. I think there's a fun thing; I keep a list of all the tech reports that I like for reasoning models. In Chapter 14, where there's a short summary of RLVR, there's just a gigantic table where I list every single reasoning model that I like. So there's just... I think in education a lot of it needs to be like, at this point, what I like, because the language models are so good at the math. For example, the famous paper Direct Preference Optimization, which is a much simpler way of solving the problem than RL. The derivations in the appendix skip steps of math. For this book, I redid the derivations and I'm like, 'What the heck is this log trick that they use to change the math?' But doing it with language models, they're like, 'This is the log trick.' I'm like, 'I don't know if I like this, that the math is so commoditized.' I think some of the struggle in reading this appendix and following the math, I think is good for learning. And I...
Yeah, so we're actually returning to this often, just on the topic of education. You both have brought up the word 'struggle' quite a bit. So there is value. If you're not struggling as part of this process, you're not fully following the proper process for learning, I suppose.
Some of the providers are starting to work on models for education, which are designed to not give... actually, I haven't used them, but I would guess they're designed to not give all the information at once and make people work to do this. I think you could train models to do this, and it would be a wonderful contribution. Where like all of the stuff in the book, you had to reevaluate every decision for it, which is such a great example. I think there's a chance we'll work on it at AI2, which I thought would be so fun.
It makes sense. I did something like that the other day for video games, for example. I sometimes play video games for my pastime, video games with puzzles. You know, like Zelda and Metroid. And there's this new game where I got stuck, and I really got stuck and was okay. I don't want to struggle for two days, so I used an LLM. But then you say, 'Hey, please don't add any spoilers. Just, you know, I'm here. What do I have to do next?' And the same thing you can do for math, where you say, 'Okay, I'm at this point. I'm getting stuck. Don't give me the full solution, but what is something I could try?' You know, where you kind of carefully probe it. But the problem here is I think it requires discipline. And a lot of people who enjoy math, but there are also a lot of people who need to do it for their homework and then it's like this shortcut. And yeah, we could develop an educational LLM, but the other LLM is still there, and there's still a temptation to use the other LLMs.
I think a lot of people, especially in college, they understand the stuff they're passionate about, they're self-aware about it, and they understand it shouldn't be easy. Like, I think we just have to develop a good taste, talk about research taste, like school taste about stuff that you should be struggling on and stuff you shouldn't be struggling on. Which is tricky to know, because sometimes you don't have good long-term vision about what would be actually useful to you in your career. But you have to develop that taste, yeah.
I was talking to my fiancee or friends about this, and it's like there's this brief 10-year window where all of the homework and all the exams could be digital. But before that, everybody had to do all the exams in blue books because there was no other way. And now after AI, everybody's going to need blue books and oral exams because everybody could cheat so easily. It's like this brief generation that had a different education system that, like, everything could be digital, but you still couldn't cheat. And now it's just gonna go back. It's just very funny.
You mention character training. Just zooming out on a more general topic, for that topic how much compute was required? And in general, to contribute as a researcher, are there places where not too much compute is required where you can actually contribute as an individual researcher?
For the character training thing, I think this research is built on fine-tuning about seven billion parameter models with LoRA, which is essentially only fine-tuning a small subset of the weights of the model. I don't know exactly how many GPU hours that would take.
Not doable for every academic. The situation for some academics is so dire that the only work you can do is doing inference where you have closed models or open models and you get completions from them and you can look at them and understand the models. And that's very well-suited to evaluation, where you want to be the best at creating representative problems that the models fail on or show certain abilities, which I think that you can break through with this. I think that the top-end goal for a researcher working on evaluation, if you want to have career momentum, is that Frontier Labs pick up your evaluation. You don't need to have every project do this. But if you go from a small university with no compute and you figure out something that Claude struggles with and then the next Claude model has it in the blog post, there's your career rocket ship. I think that's hard, but if you want to scope the maximum possible impact with minimum compute, it's something like that, which is just get very narrow and it takes learning of where the models are going. So you need to build a tool that tests where Claude 4.5 will fail. If I'm going to start a research project, I need to think where the models in eight months are going to be struggling.
But what about developing totally novel ideas?
This is a trade-off. I think that if you're doing a PhD, you could also be like, 'It's too risky to work in language models. I'm going way longer term,' which is like, what is the thing that's going to define language model development in 10 years? I think that I end up being a person that's pretty practical. I mean, I went to my PhD where it was like, 'Eh, I got into Berkeley. Worst case, I get a master's, and then I go work in tech.' And so I'm very practical about it, so I'm like, the life afforded to people to work at these AI companies, the amount of... Like, OpenAI's average compensation is over a million dollars in stock a year per employee. Any normal person in the US, to get into this AI lab is transformative for your life. So I'm pretty practical about it. There's still a lot of upward mobility working in language models if you're focused. And the outcomes, it's like, look at these jobs. But from a research perspective, the transformative impact in these academic awards, it's like being the next Yann LeCun is from not working on, not caring about language model development very much.
It's a big financial sacrifice in that case.
So I get to work with some awesome students, and they're like, 'Should I go work at an AI lab?' And I'm like, 'You're getting a PhD at a top school. Are you gonna leave to go to a lab?' I'm like, 'I don't know.' If you go work at a top lab, I don't blame you. Don't go work at some random startup that might go to zero. But if you're going to OpenAI, I'm like, 'It could be worth leaving a PhD for.'
Let's more rigorously think through this. So where would you give a recommendation for people to do a research contribution? The options are academia: get a PhD. Spend five years publishing. Compute resources are constrained. There's research labs that are more focused on open-weight models, and so working there. Or closed frontier labs, research labs. So OpenAI, Anthropic, xAI, and so on.
The two gradients are: the more closed, the more money you tend to get, but also the less credit you get. So in terms of building a portfolio of things that you've done, it's very clear what you have done as an academic. Versus if you are gonna go trade this fairly reasonable progression for being a cog in the machine, which could also be very fun. So I think it's very different career paths. But the opportunity cost for being a researcher is very high because PhD students are paid essentially nothing. So I think it ends up rewarding people that have a fairly stable safety net, and they realize that they can operate in the long term, which is wanting to do very interesting work and get a very interesting job. So it is a fairly privileged position to be like, 'I'm gonna see out my PhD and figure it out after because I want to do this.' And at the same time, the academic ecosystem is getting bombarded by funding getting cut and stuff. So there's just so many different trade-offs where I understand plenty of people that are like, 'I don't enjoy it. I can't deal with this funding search. My grant got cut for no reason by the government,' or, 'I don't know what's gonna happen.' So I think there's a lot of uncertainty and trade-offs that, in my opinion, favor just taking the well-paying job with meaningful impact. It's not like you're getting paid to sit around at OpenAI. You're building the cutting edge of things that are changing millions of people's relationship to tech.
But publication-wise, they're being more secretive, increasingly so. So you're publishing less and less. And so you are having a positive impact at scale, but you're a cog in the machine.
I think, honestly, it hasn't changed that much. So I have been in academia; I'm not in academia anymore. At the same time, I wouldn't want to miss my time in academia. But what I wanted to say before I get to that part, I think it hasn't changed that much. I was working in, like, I was using AI or machine learning methods for applications in computational biology with collaborators, and a lot of people went from academia directly to Google. And I think it's the same thing. Back then, professors were, you know, sad that their students went into industry because they couldn't carry on their legacy in that sense. And I think it's the same thing. It hasn't changed, I think, that much. The only thing that has changed is the scale. But, you know, cool stuff was always developed in industry that was closed. You couldn't talk about it. And I think the difference now is, well, your preference. Do you like to talk about your work and publish, or you are more in a closed lab? That's one difference, the compensation, of course. But it's always been like that, I think. So it really depends on where you feel comfortable. And also, nothing is forever. The only thing right now is there's a third option, which is starting a startup. That's a lot of people doing startups. Very risky move. But it is a high-risk, high-reward type of situation, whereas joining an industry lab, I think, is pretty safe. Also upward mobility. Honestly, I think once you have been at an industry lab, it will be easier to find future jobs. But then again, you know, it's like, yeah, how much do you enjoy the team and working on proprietary things versus how do you like the publishing work? I mean, publishing is stressful. It is. You know, the acceptance rate at conferences can be arbitrary, can be very frustrating, but also high reward if you have a paper published, you feel good because your name is on there. You have a high accomplishment, and you know.
I feel like my friends who are professors seem on average happier than my friends who work at a frontier lab. To be totally honest. Because that's just grounding and the frontier labs definitely do this 9/9/6, which essentially is shorthand for work all the time.
Can you describe 9/9/6 as culture that, I believe you could say was invented in China and adopted in Silicon Valley? What's 9/9/6? It's 9:00 AM to 9:00 PM.
Six days a week. What is that, 72 hours? Okay. So, is this basically the standard in AI companies in Silicon Valley? More and more this kinda grind mindset.
Yeah, I mean, maybe not exactly like that, but I think there is a trend towards it. And it's interesting. I think it almost flipped. Because when I was in academia, I felt like that because, as a professor, you had to write grants. You had to teach and you had to do your research. It's like three jobs in one, and it is more than a full-time job if you wanna be successful. And I feel like now, like Nathan just said, the professors in comparison to a lab, I think they have less pressure or workload than at a frontier lab because...
I think they work a lot. They're just so fulfilled. By working with students and having a constant runway of mentorship and a mission that is very people-oriented. In an era when things are moving very fast and are very chaotic, it's very rewarding to people.
Yeah, and I think at a startup, it's this pressure. It's like you have to make it. It is really important that people put in the time, but it is really hard because you have to deliver constantly, and I've been at a startup. I had a good time, but I don't know if I could do it forever. It's an interesting pace, and it's exactly like we talked about in the beginning. These models are leapfrogging each other, and they are constantly trying to take the next step compared to their competitors. It's just ruthless, I think, right now.
I think this leapfrogging nature and having multiple players is actually an underrated driver of language modeling progress, where competition is so deeply ingrained in people, and these companies have intentionally created very strong culture. Like, Anthropic is known to be so culturally, deeply committed and organized. I mean, we hear so little from them, and everybody at Anthropic seems very aligned. And it's like being in a culture that is super tight and having this competitive dynamic is a thing that's gonna make you work hard and create things that are better. So that comes at the cost of human capital, which is like you can only do this for so long, and people are definitely burning out. I think I wrote a post on burnout as I've tread in and out of this myself, especially trying to be a manager, full-mode training. It's a crazy job. The book After Steve by Patrick McGee talks about how hard the Apple engineers worked to set up supply chains in China, and he said they had 'saving marriage' programs, and he said in a podcast, 'People died from this level of working hard.' So I think it's just like it's a perfect environment for creating progress based on human expense, and there's a lot of... the human expense is the 996 that we started this with, where people do really grind.
I also read this book. I think they had a code word for if someone had to go home to spend time with their family to save the marriage. It's crazy. Then the colleagues said, 'Okay, this is red alert for this situation. We have to let that person go home this weekend.' But at the same time, I don't think they were forced to work. They were so passionate about the product that you get into that mindset. And I had that sometimes as an academic, but also as an independent person, I have that sometimes. I overwork, and it's unhealthy. I had back issues, I had neck issues, because I did not take the breaks that I maybe should have taken. But no one forced me to; it's because I wanted to work, because it's exciting stuff.
That's what OpenAI and Anthropic are like. They want to do this work.
Yeah, but there's also a feeling of fervor that's building, especially in Silicon Valley, aligned with the scaling laws idea, where there's this hype where the world will be transformed in a scale of weeks and you want to be at the center of it. And then, you know, I have this great fortune of having conversations with a wide variety of human beings, and from there I get to see all these bubbles and echo chambers across the world. And it's fascinating to see how we humans form them. And I think it's fair to say that Silicon Valley is a kind of echo chamber, a kind of silo and bubble. I think bubbles are actually really useful and effective. It's not necessarily a negative thing because you could be ultra-productive. It could be the Steve Jobs reality distortion field, because you just convince each other that breakthroughs are imminent, and by convincing each other of that, you make the breakthroughs imminent.
Burn Hobart wrote a book classifying bubbles, but essentially one of them is financial bubbles, which is like speculation, which is bad, and the other one is like, I don't know the term, but effectively for build-outs, because it pushes people to build these things. And I do think AI is in this, but I worry about it transitioning to a financial bubble, which is...
Yeah, but also in the space of ideas, that bubble, you are doing a reality distortion field, and that means you are deviating from reality, and if you go too far from reality while also working, you know, 996, you might miss some fundamental aspects of the human experience, including outside Silicon Valley. This is a common problem in Silicon Valley: it's a very specific geographic area. You might not understand the Midwest perspective, the full experience of all the other humans in the United States and across the world, and you speak a certain way to each other, you convince each other of a certain thing, and that can get you into real trouble. Whether AI is a big success and becomes a powerful technology or it's not, in either trajectory you can get yourself into trouble. So you have to consider all of that. Here you are, a young person trying to decide what you want to do with your life.
The thing that is... I don't even really understand this, but the SF AI memes have gotten to the point where 'permanent underclass' was one of them, which was the idea that the last six months of 2025 was the only time to build durable value in an AI startup or model. Otherwise, all the value will be captured by existing companies and you will therefore be poor, which... like, that's an example of the SF thing that goes so far. I still think for young people going to be able to tap into it, if you're really passionate about wanting to have an impact in AI, being physically in SF is the most likely place where you're going to do this. But it has trade-offs.
I think SF is an incredible place, but there is a bit of a bubble. And if you go into that bubble, which is extremely valuable, just get out also. Read history books, read literature, visit other places in the world. Twitter is not, and Substack is not the entire world.
One of the people I worked with is moving to SF, and I need to get him a copy of Season of the Witch, which is a history of SF from 1960 to 1985, which goes through the hippie revolution, all the gay community taking over the city and that culture emerging, and then the HIV/AIDS crisis and other things. And it's just so recent, and there's so much turmoil and hurt, but also love in SF. No one knows about this. It's a great book, Season of the Witch. I recommend it. A bunch of my SF friends who get out recommended it to me. And I think that's just living there... I lived there and I didn't appreciate this context, and it's just, like, so recent.
Yeah. Okay, let's... We talked a lot about a lot of things. Certainly about the things that were exciting last year. But this year, one of the things you guys mentioned that's exciting is the scaling of text diffusion models, just a different exploration of text diffusion. Can you talk about what that is and the possibility it holds? So, different kinds of approaches than the current LLMs?
Yeah, so we talked a lot about the transformer architecture and the autoregressive transformer architecture, specifically, like GPT. It doesn't mean no one else is working on anything else. People are always on the lookout for the next big thing. Because I think it would be almost stupid not to. Because sure, right now, the transformer architecture is the thing, and it works best, and there's nothing else out there. But it's always a good idea to not put all your eggs into one basket. So, people are developing other alternatives to the autoregressive transformer. One of them would be, for example, text diffusion models. And listeners may know diffusion models from image generation, like Stable Diffusion popularized it. There was a paper on generating images. Back then, people used GANs (Generative Adversarial Networks). And then there was this diffusion process where you iteratively denoise an image, and that resulted in really good quality images over time. Stable Diffusion was a company; other companies built their own diffusion models. And then people are now like, 'Okay, can we try this also for text?' It doesn't make intuitive sense yet, because it feels like it's not something continuous like a pixel that we can differentiate. It's discrete text, so how do we implement that denoising process? It's kind of similar to the BERT models by Google. Like when you go back to the original transformer, there were the encoder and the decoder. The decoder is what we are using right now in GPT and so forth. The encoder is more like a parallel technique where you have multiple tokens that you fill in in parallel. GPT models do autoregressive generation, completing the sentence one token at a time. And in BERT models, you have a sentence that has gaps. You mask them out, and then one iteration is filling in these gaps. Text diffusion is like that, where you are starting with some random text, and then you are filling in the missing parts or refining them iteratively over multiple iterations. The cool thing here is that this can do multiple tokens at the same time. So, it has the promise of being more efficient. Now, the trade-off is, of course, how good is the quality? It might be faster, but now you have this dimension of the denoising process. The more steps you do, the better the text becomes. You can scale in different ways. Researchers are trying to see if that is maybe a valid alternative to the autoregressive model in terms of giving you the same quality for less compute. Right now, papers suggest that if you want to get the same quality, you have to crank up the denoising steps, and then you end up spending the same compute you would on an autoregressive model. The other downside is that while parallel sounds appealing, some tasks are not parallel, like reasoning tasks or tool use, where you might have to ask a code interpreter to give you an intermediate result. That is tricky with diffusion models. So there are some hybrids. But the main idea is: how can we parallelize it? It is an interesting avenue. Right now, there are mostly research models out there, like LaMDA and some other ones. I saw some by startups, but there is no big diffusion model at Gemini or ChatGPT scale yet. But there was an announcement by Google, where they said they are launching Gemini Diffusion, and they put it into the context of their Nano 2 model. They said basically for the same quality on most benchmarks, we can generate things much faster. You mentioned what's next. I don't think text diffusion will replace autoregressive LLMs, but it will be something for quick, cheap, at-scale tasks. Maybe the free tier in the future will be something like that.
