David Luan: DeepSeek’s Significance, What’s Next for Agents & Lessons from OpenAI

This is such a good question. Are we going to get exceptionally good at coding and math but still not be able to tell a joke or something? This is a great topic to debate. I have a very strong view.

Yeah, thanks for having me. This is going to be a lot of fun because we've known each other for what, like more than 10 years now? You know, I remember when you originally joined OpenAI, and I was like, 'That seems interesting, but I wonder, you know, a cool career move.' And then obviously you were, as always, prescient long before everyone else.

Yeah, so I still remember the morning that everybody started waking up to DeepSeek news. I woke up, I looked at my phone, I had like five missed calls. I was like, 'What is going on?' And the last time something like that happened was when SVB collapsed, because all my investors were calling me to get our funds out from SVB and First Republic and all that stuff. So I was like, 'Something really bad must be happening.' And I checked the news, and it's like stocks are crashing because DeepSeek R1 is out and all that stuff. I instantly was like, 'Wow, people have really missed the memo on what actually happened here.' DeepSeek, I think, was incredibly good work. I have so many thoughts on team culture and composition and all that other stuff we can get to later, but it was just really incredible work. But it's part of this broader story arc of: we figure out first how to make new ML systems smarter, and then we figure out how to make them more efficient. So this was like the talk of the tick of o1 really. And what everybody got wrong was just because you could make more intelligence happen at a lesser price doesn't mean that you stop consuming more intelligence. If anything, you consume even more of it. So once the market woke up to that, now we're back to sanity.

I think what's going to happen is that people want to build the smartest models possible, but that sometimes it's not always inference efficient. So I think what we're going to start seeing more and more of is, whether people talk explicitly about this or not, people are going to train these humongous teacher models on as much compute as they can get their hands on, and then they're going to try to figure out internally in their own labs how to render it down to something that runs really fast and is efficient for customers. The biggest thing I'm seeing right now is that every AI use case is in concentric circles of complexity. At the very inner circle of complexity will be something like just having a good old chat conversation with a base LLM, which we were able to do back in GPT-2 pretty competently. Every incremental circle of intelligence, maybe it's being able to do mental math or coding, or later on agents, or later on drug discovery, requires smarter and smarter models. But every previous ring of intelligence becomes almost so cheap as to be commoditized.

There's actually an interesting paper trail and podcast trail of me talking about the recipe for how to build AGI that's years old at this point. I'll contribute to that trail. So now we get to this point in time. But back in even 2020, we always said, at the time we were starting to see GPT-2 had come out, GPT-3 I think was cooking or maybe done by then, we were starting to think about four. We were living in this world where people were not sure whether or not all you needed was next token prediction to solve all of AGI. My view, and a couple views of people around me, was actually that the answer would be no. The reason why the answer was no is because an LLM trained to do next token prediction, by definition, is penalized for discovering new knowledge because new knowledge was not part of the training set. As a result, what we needed to do was go look at what other ML paradigms we know can actually discover new knowledge. We know RL can do that, and search can do that. Even AlphaGo was maybe the first time it went to public consciousness that we could discover new knowledge using RL. The question always was when we were going to combine LLMs with RL to get systems that had all of the knowledge of what humanity already knew and the ability to go build upon it. The reason why the initial DeepMind path of just doing RL alone didn't work was because it was initialized randomly. Those models that were playing Atari, the amount of time it would take something that knew nothing about the world to rediscover human language and rediscover how to coordinate and learn details about how to file your taxes would take forever if done purely in an RL setting. So now I think that philosophy has really been borne out by seeing how successful these models that combine both these paradigms are.

This is a great topic of debate. I have a very strong view. The answer is that these models are better at generalizing than you think. Everybody's like, 'Ah, I played with o1 and it seems like it's a little better for math, but wait for it to think and it kind of maybe it's a little bit worse than chat or whatever.' I think those are just blips on the way to glory for how these things are built. Today, we already have signs that getting better through problems where you have an explicit ability to test whether or not the model correctly solved it, which is what we've seen from DeepSeek, does lead to transfer on some slightly fuzzier problems that seem in a similar domain. The field is working hard, my team and others, to figure out how to learn human preferences around these much more complicated tasks and then just do RL to go satisfy those. You have to be able to build a model to essentially verify that output is good law or that output is a good healthcare diagnosis. Obviously, that's a much harder problem than verifying a math proof or did that code run. The fundamental thing that I view we are arbitraging is this gap between how good these models are at determining whether they did a good job compared to generating the right answer. We've always seen that these models are better at determining whether they've done a good job than they are at generating the answer. To some extent, what we're doing with all this RL stuff is exploiting that to force it to try over and over again to satisfy its own sense of whether or not it did a good job.

There are just so many. Where do we even start? In no particular order, and I'm only probably going to itemize like three of the problems that we've had to solve. I think one of the first ones is just figuring out how you can even build an organization and a process to be able to reliably turn out models. One of the things that I always say to my team and to people that I collaborate with is that today, if you run a modern AI lab, your job isn't to build models. Your job is to build a factory that reliably turns out models. If you think about how you think about that, it totally changes what you invest in. Until there was repeatability here, there was not a lot of forward momentum. We've just gone through this arc over the last couple of years of going from alchemy to industrialization in terms of how these things have been built. Without that, there was no substrate for these things to work. I think the next part, this is a space where you have to go slow to go fast. I continually believe that the thing that people always gravitate to because they think it's cool and sexy are the algorithms, but if we look at what actually drove a lot of this stuff, it's solving the engineering. How do you do giant massive clusters that you can reliably keep up for long enough, and if a node goes down you don't end up wasting a bunch of time? To be able to push the frontier at scale, now with this whole RL thing, we're going to be quickly moving to a world where there will be lots and lots of data centers, each of which are going to be doing a lot of inference on the base model and maybe testing it on new environments, maybe that a customer has brought to bear to learn how to improve the model, and sending those new knowledge back to a centralized place where the model can learn how to be smarter. There are a lot of really tough engineering problems that have to be solved.

I think we just have counterexamples. AlphaGo was original thinking. We watch, if you go back even to the old OpenAI work where we were using RL to play Flash games, if you're of a certain age you probably remember Miniclip and stuff like that. Great sinks of time in middle school. But it's so funny to watch that become the substrate of AI. We were working on just using our algorithms to try to solve many of those games at once, and you just learn that they learn really quickly how to discover, often times, speedrun techniques by glitching through walls and stuff to solve platformer levels. That's stuff that humans have never done before.

I think you use the model. I love it.

Okay, so I'm super excited about agents. I remain super excited about agents. I still go back to 2020, 2021 when the first wave of truly powerful LLMs like GPT-4 were coming out. You'd go play with them and you're like, 'Wow, there's so much promise. It's made me a great rap song. It does great roasts sometimes. It does three-digit addition acceptably.' And you're like, 'Please order me a pizza,' and it just cosplays being a Domino's Pizza rep. It just can't do it. It's obviously a major gap in the utility of these systems. So even since then, I was pretty sure that we had to go solve the agent problem. We started working when I was at Google on problems that actually still are called tool use, how do you expose affordances to the LLM to decide when it should go do something. Back then, the RL literature had always called it agents, but the general public didn't yet have a word for it, so we tried to come up with a new term called a large action model instead of a large language model. That kind of had a little bit of traction, and then the world decided that it was going to be called an agent, and now everything's an agent and doesn't mean anything anymore, which is very sad. But it was really cool to be the first modern agent company. When we started Adept, the best open source LLMs were not good, so we had to train our own model because there was also no multimodal LLMs like image input LLMs. GPT-4V came so much later. So we had to do everything end to end from scratch. It's kind of like starting an internet company in 2000 and having to go call TSMC to build your own chip. It's just insane. Along the way, what we learned was that really early on, LLMs out of the box without any of the new RL stuff we're doing today, they're behavioral cloners. They kind of do what they've seen in the training data, and that means that they're really liable to go off the rails because the moment they end up in a situation they've never seen before, the generalization tends to be bad and it does something unpredictable. So at Adept, we were always focused on useful intelligence. What did utility mean? It wasn't ship a cool demo that went viral on Twitter. It was put this in the hands of someone so they don't have to do the shuffle things around in your computer grunt work that most knowledge workers have to do. Those knowledge workers care about reliability. One of our early use cases was invoice processing for people. Everyone loves invoice processing for these agentic models. It feels like a natural place to start. It's a great hello world. At the time, nobody had really done these things before, so let's choose an obvious hello world. We chose that, we did Excel, some other ones. But if this thing ends up deleting a third of your QuickBooks entries one in seven times, you'll never use it again. Reliability remains an issue. Even today, Operator is super impressive and seems to be a cut above cloud computer use so far, but if you look at both of those things that are already out, the biggest challenge is that they've both focused on end-to-end task performance. Go in there and you type, 'Hey, I'd like for you to find me five vacation sites that I could go to this weekend,' and it'll go do an approximation of that, but the end-to-end reliability of that is super low, requires lots of interventions. We're still not at a point where businesses really trust it in a fire-and-forget way. That's what we've got to solve.

I can talk at a high level about it. Basically, there are two things you have to do. One is an engineering problem. The engineering problem is how do you expose, in a model-legible way, what it can do. Here are the APIs you can call, here are the UI elements you can call. Let's go teach you a little bit about how Expedia.com works or how SAP works. That's a little bit of research and engineering. That's kind of step one, giving it a sense of what it can and can't do and basic abilities to go do stuff. The interesting part happens in the second component, which is how do you teach it to plan and reason and replan and follow user instructions, and later on actually even be able to infer what the user actually meant and go do that for them. That's a huge research problem, and it differs a lot from regular old LLM work because regular LLM work is like, 'Let's go generate a piece of text.' Even the reasoning work we're seeing today with math problems, there's an answer at the end, so it's like a single step even if it's thinking for a bajillion chains of thought. It's really taking one step for you, which is 'I've given you the answer.' With this, it's this whole multi-step decision-making process that involves backtracking and involves trying to predict the consequences of an action you take for the future and realizing, 'Hey, the delete button is probably dangerous.' You have to do all the work to teach the model that in a basic setting, and then you set it in sandboxes to learn on its own terms. The best analogy for this, I'll jam in, I think it was Andrej Karpathy who wrote this on the internet, is that modern AI training is kind of like how a textbook is organized. First, you have all the exposition of some physical process, and then you've got some sample problems. The first part is pre-training, the sample problems are supervised fine-tuning, and the RL step happens when you have the open-ended problems at the back that maybe have an answer in the back of the textbook. We're just following that process.

If I were to ding the field right now on one thing, it is that there has been a massive lack of creativity on how people interface with increasingly smart LLMs and agents. We are in the, do you remember when the iPhone came out, the App Store came out, people started making all of these apps like hit this button to make the burp noise and here's a beer that you can pour into your mouth by tilting the phone. Our interfaces today feel like that, and that is so sad because chat is a super limiting, low-bandwidth way to get things done. In some ways it's easy, but in many other ways, I don't want to have a seven-turn conversation to decide what toppings I'm going to have on my pizza. The lack of creativity there has been really bugging me. I think part of the reason why is that the amazing product designers that could be helping us figure this stuff out, a lot of them don't yet deeply understand the limitations of the models. This is changing quickly, but conversely, so far the people who have been able to advance the technology have always just thought about it as 'I'm here to go deliver a black box' and not 'I'm here to go deliver an end-to-end experience.' When that changes, I'm excited to see things like systems where when you interact with the agent, it is itself synthesizing this multimodal user interface for you to best elicit what it needs from you, and to have shared context between the human and the AI. Instead of the current paradigm where you're chatting to each other, it's like you are doing something together on your computer and looking at the screen, more like parallel rather than perpendicular.

I think Operator is super impressive, by the way. It's just that right now the whole field is missing that last chunk. It's like self-driving. I forgot how many, it must have been over a decade, maybe even 15 years ago, we had amazing demo videos. I think having done the self-driving podcast yesterday, I think it was 95 where they did the ride across the country like 99% off. So are we going to have to wait 30 years? No, I don't think so, because I think we actually have the right tools in our toolbox now. I think this recipe for how to build a level agent will work pretty well.

The main milestone I'm looking for in the agent space is, during training time, I work at one of these main labs. The milestone I'm looking for is I have a recipe where I can hand this agent in training any task and come back end days later and it's 100% at it. Not just like humans have given us another 5% points more reliability, but the agent has figured out how to solve the thing.

I actually have a huge amount of uncertainty on this one, but my current view is that I personally think AGI is really not super far away. When you say AGI, how do I define it? The model that can do anything useful that a human does on a computer, that's one part of the definition. The other part that I also like is that it's a model that can learn how to do that thing as fast as a human can, like a generalist human can. I think either of those are really not that far away. But I also don't think that it's going to be deeply transformational and diffuse through society really quickly because, as we know through Amdahl's law, once you really speed up one particular thing, something else becomes a bottleneck and your overall amount of speed up is less than you think. So I think what's going to happen is that we'll have the tech, but there will be a massive capability overhang where society's ability to actually use these things productively will lag quite a while.

I think it's people. It's people, processes, it's figuring out how to co-design the interface with the decisions that startups are making on how to use the models. It's going to be social acceptance. Imagine you have this model that pops out tomorrow and says, 'I have invented a brand new way of doing X, everybody should go use this thing.' Humans have to go make their peace with it and decide if this is actually a better solution. That's not going to be as fast as we think.

I'm actually pretty sure that's what's going to happen. I'm obviously biased, I want that to happen. I think that's a good bet because at the end of the day, I still really believe that even in a world with AGI, human relationships really matter. Knowing and owning your customer and being more in tune with them about what they need is going to be more important than simply controlling this artifact that many other labs will have.

I think that we'll just get new tools in the toolbox for how we interface with computers. Today, people still use the command line. That's a really important part of people's productivity. People still use the GUI. In the future, people will still use voice interfaces, but also AI will use more ambient computing, and they'll have the standard UI thing we were talking about earlier. But I think the metric we should be looking at is just what is the amount of leverage per unit energy a human spends with computing. I think that is going to continue to go up and to the right with these systems.

You probably want the hypothetical legal specialist to know some basic facts about the world, make people go do a general college degree before law school. Exactly. So I think there will be specialized models, but I don't want to bury the lead by just saying there will be specialized models. I think they'll be specialized models not for technical reasons, but for policy reasons. That's juicy. What does that mean? It's just like you have a couple companies who just really don't ever want their data comingled with each other. Or you've got a big bank with a sales and trading division and an investment banking division. The AI employees or LLMs that power the stuff, just like how those employees today can't share information, should not be able to share information even remotely through its weights.

I actually don't believe that we take what we have today exactly and then just pull ahead the cluster from two years from now and everything will magically work. I do think scale is going to be a major factor, but my confidence actually comes from looking at what are the remaining open problems and trying to have an estimate for how hard they are. I think that if a super hard thing like we need to go replace gradient descent, or we can only do AGI with a quantum computer or something like that, I don't think that's in the cards.

There are two things that I do. One of them is that what I've learned is that, and this is what's so cool about this field, sometimes you just look at a result and you look at, especially if there is a methodology that gets published with it which is rare now, you just look at how they did it and you're like, 'Wow, this is actually simpler than how we used to do this and the results are better.' When that happens, it almost always becomes part of the deep learning canon. You just have this moment where you're like, 'This is actually really beautiful.' I think that's the main one. Then the other one is benchmarks. Part of the hype of the field has been that a lot of benchmarks that are good but not that aligned with what people need from these models have just become so important in people's development processes that they're all kind of gamed. I actually think that evaluations are so hard, measurement is so hard. Way more prestige and attention should go to that than actually many other things that we're doing right now.

Amazon is a super interesting place. I feel like I've learned so much in a short amount of time there. Amazon is super serious about building generally intelligent systems, especially generally intelligent agents. What I think is really cool about it is that I think everybody at Amazon understands that computing itself is changing from the primitives that we all know and love to a call to a large model or a large agent being probably the most important compute primitive in the future. People really care, which is awesome. I think what's interesting about Amazon now is I cover agents, and what's been really cool is you get to see just the breadth of everything that agents touches in a company as big as Amazon. What's also awesome is that Peter and I have started this new San Francisco-based research lab for Amazon. A lot of that was because folks in the highest levels of Amazon really believe that we have to make new research breakthroughs to solve those remaining problems we were talking about earlier on the path to AGI.

I always pay attention to things that look like they might help us better map model learning to compute, can we use more compute more efficiently. That just gives us a huge multiplier over what we can do. But I honestly actually spend more of my time looking at data centers and chips because I just find it so fascinating. It's so cool, and there's some interesting plays being made there now.

I think there are two different jobs that have to be solved from data labeling. There may be more, but the two that come to mind for me first is teaching the model the very basics of how to do a task by cloning human behavior. If you have super high quality data, then you can use that to better elicit something you already see during pre-training. Then I think the second job is to teach the model what good and bad looks like for tasks that are fuzzy. I think both of those will remain really important, but I think this middle chunk of just spamming human data labels to marginally improve models that can already kind of do a thing, that's going to be the job of RL.

The one that I keep coming back to is the importance of building the team culture in the right way. I think we've always kind of known it, but I've become even more convinced that hiring really smart, energetic, intrinsically motivated people earlier on in their careers is actually one of the best engines for productivity. In this field, every couple of years the optimal playbook changes. If people are too overfit to the previous optimal playbook, they actually slow you down. So I think it's a lot better to bet on new folks coming in than I had previously thought. The other one that I've changed my mind on is I used to think that building AI would actually have real long-term technical differentiation that you can compound on. I used to think, 'If you get really good at text modeling, it'll help you, so you should obviously just become the winner in multimodal. If you're good at multimodal, you should obviously become the winner in reasoning and in agents.' These things should compound. In practice, I've seen so little compounding. I think people are all trying relatively similar ideas.

It's definitely correlated, but it's not deterministically true that you will then obviously win the next change.

Similarly to my belief about digital agents, I think that we have a lot of the raw materials. Interestingly enough, digital agents gives us an opportunity to de-risk some of the hard problems in physical agents before you have to do all the costly stuff of real world items.