I definitely agree that continual learning is really the thing. It's really the thing that we're building. But I don't really think this is like a problem that's ignored and off the path of what we're doing currently. I think it is what we're working toward.
What are like the other research areas within alignment that you're paying attention to or that you think are promising?
A lot of the like longer-term challenge with alignment is about generalization. What are the values that the model falls back on?
What are the things that you need to figure out to be able to really make models work well in some of these other spaces?
Jakub is the chief scientist of OpenAI. I think literally one of the most important people on the planet. And today on Unsupervised Learning, I got to ask him literally everything that I've been thinking about and I know a bunch of people in the ecosystem have too. We talked a lot about model progress, what's required to make long-running agents work, as well as the really interesting work OpenAI has done in the AI for science world and the progress he sees in that over the next years. We talked a lot about how companies should be thinking about model building in this moment, when they should be doing reinforcement learning, how they should be thinking about the evolution of harnesses and the impact that will have. We hit on a lot of his really interesting research, including the work he's done around alignment, the work that OpenAI broadly has done around math competitions. And we also talked about this focusing moment in OpenAI and what it means for the research organization and how he runs his team. Literally just such an awesome opportunity to talk to someone who is driving so much of the change that has revolutionized this space in the world. I hope folks enjoy this wide-ranging conversation as much as I did.
I feel like you are the perfect person to talk to about all the questions everyone has in the ecosystem. What's happening with model progress? A lot of companies are thinking about how they should be building things based on what's happening with the models. A lot of people at a societal level are thinking about the impact AI is going to have on science and broader society. And you've been at the forefront of the space for pretty much every generation of improvement these past years and so really excited to have you on the podcast.
I think I'll start with one of the juiciest things you said which is, you know, four months ago I think you and the open team talked about aiming for a system with research level intern capabilities by September of this year. So coming up, I think that's what, six months from now. And then a more fully automated AI researcher by March 2028. And so I guess, you know, checking in four months later, how are you feeling about those timelines?
Yeah, I think over the last months, the change that really happened is we've seen this explosive growth of coding tools.
It's an understatement. Yeah, we've definitely like really kind of gone to a place in OpenAI where we use Codex for the majority of actual coding. And so I think for most people, the act of programming has changed quite a bit. So I definitely see this as a signal that something here is on track. The other kind of like very interesting update over the last few months to me has been the progress on the math research capabilities. Also the results we've kind of seen in physics and other fields. I think this kind of level of capability, this level of ability to provide insight when combined with ability to access infrastructure, ability to use maybe more compute at test time that's something that Codex is using currently, and very strong improvement in general intelligence which I also expect over the next couple of months. Yeah, it's something we're still very much planning for and very focused on.
And how do you know when you've gotten there? Like what's a workflow you might look to to say, hey, okay, I think we've got these research intern level capabilities?
The way I would distinguish a research intern from a full automated researcher is the span of time that we would have it work mostly autonomously or the specificity of the task that has to be given. So I don't expect we'll have systems where you kind of just tell them, oh, like, you know, go improve your model capability, go solve alignment, and they will do it. Not this year. I think we might get there at some point, but I think for more specific technical ideas, like I have this particular idea how to improve the models, how to run this evaluation differently, I think we have the pieces that we mostly just need to put together.
Karpathy released, you know, a pretty viral version of using some of these models to improve some of his, you know, obviously way less complex models than what you guys are building here, but did that feel like generally in the spirit of some of what these tools might look like?
Yeah, I think it's in the spirit. Yeah, I mean I expect it to look like a pretty continual evolution from kind of where Codex is now. I think towards a bit more autonomy, running for a longer time. But yeah, I think we'll see a lot of this sort of application. I think in general we'll see more autonomous and higher compute use of these models for different things.
You mentioned kind of like the math and physics side and obviously you've had these really impressive breakthroughs in math on some interesting different kinds of competition problems. Maybe, you know, I think for our listeners it intuitively makes sense how progress in coding directly translates to something like helping with AI research. How does math and physics progress also tie into this?
The biggest role that focusing on these math benchmarks has played for us is as a general benchmark and a north star for how to improve this technology. Like math is very measurable, right? It's much easier to tell whether you've actually solved the math problem than whether you've even produced a good piece of software. And also it can get very hard, right? So you can have things where it's very definite whether you've solved them, but it can be arbitrarily pretty much hard to actually solve them. You know, I would say up until not too long ago, my perspective has been like, well, okay, our models are not maybe able to solve simple math problems. Okay, our models are able to solve simple enough problems but are not able to solve IMO level problems. So clearly there is just a gap in the intelligence of these models that is very measurable, very easy to run at. It's very clear what we need to do and this has been kind of our north star for reasoning models and so forth. Now of course that is changing quite a bit, right? And we have kind of reached these milestones that we've been working towards of IMO gold level, solving IMO problem six, and making progress into research level mathematics. And from this point, I think there still is utility in continuing to measure progress on this. I think there's also definitely transfer that you can get from getting better at mathematical reasoning to getting better at AI research. You know, a lot of our best researchers are mathematicians we're training or from other theoretical fields. But definitely we are very much changing how we think about these north stars and we are very focused on how the models, the next models that we're producing are actually useful in the real world, useful especially for research but also for other economically valuable activities and for other fields of science and especially maybe more applied sciences. And the reason for this shift is because we believe the models are now capable enough, not as smart as people always, but capable enough to actually materially change the economy, change how things are done. And so, yeah, we feel a lot of urgency about that.
In the early days, picking a domain like math that is so hard to solve, but then easy to verify whether you did it, like it's kind of the perfect place to get started. And I think code obviously shares a lot of attributes to that. You know, possible to check and verify and great for reinforcement learning. I think one question that a lot of people are thinking about is, okay, we've seen reinforcement learning work incredibly well in these domains where you can verify it rather easily. A lot of valuable tasks in the world, medicine, law, finance, you know, there's some level of the ability to do that, but it's certainly not to the same extent that math and code are. And so I think a lot of people are trying to figure out, you know, are we going to see similar improvements? You know, obviously code and math, the rates of improvement have been so astronomical and shocking.
Yeah, I definitely expect so. I think an interesting duality that we think about a lot is for these more general tasks, for these tasks that are kind of harder to evaluate, they share a lot of commonalities with just longer horizon tasks, right? Because if you think about even a very well specified math or coding problem, again, if it's something that you need to work on for like a year, then even if it's very clear what the criteria of success are in the long term, like what to do on your first day of working on it is a pretty open-ended problem. And so I kind of believe these difficulties coincide and they're very clearly the next frontier for how these systems develop. And I think we've definitely seen very encouraging signs both on just our ability to scale RL on these more general domains. And I think also we can scale efforts that have a lot of promise.
In these other domains, it feels like one of the hardest things to know is just what was success in a task, right? And you can imagine there's going to be whatever the problems you are that are facing code or math that are short-term tasks and then longer-term tasks, feels it will be amplified in the space that is outside of those, right? Where a short-term legal task or medical task may be harder to run thousands of iterations on, right? And figure out was that done correctly and then those longer-term tasks like even harder. I'm curious like how you even conceptualize that research challenge, like what are the things that you need to figure out to be able to really make models work well in some of these other spaces.
Yeah, I think I come back to this reality of like how do we make the models work for a very long time and how do we teach them to evaluate kind of partial progress.
Yeah. I mean I think if you look at even outside of RL, like where that sort of progress on longer horizons is coming from, right? Like I mean as the models kind of become more consistent from just pure supervision in pre-training, they gain some idea of like, you know, what does a good partial artifact here look like. And so I think even if we weren't scaling RL very meaningfully we would see an elongation of these horizons over time.
Yeah, it's definitely a research challenge to figure out how to leverage these new ideas from RL and so forth to apply this to general domains. But I'm quite optimistic about that.
Yeah. And it's interesting. It sounds like part of your mental model is like the models themselves being able to check progress with some sort of cadence that is, you know, reliable enough from the outside at least. It's not totally clear if we've seen like generalization in RL yet. It feels like we clearly you seem to have some techniques that really optimize models around whatever we choose to focus on but it's like almost feels like an older school version of ML of like one thing at a time. Is that like, you know, I guess would you agree with that characterization and like how do you kind of see this current climate?
Well, we are buying a lot of compute, right? Because we still believe a bit less and we believe more than ever to some degree. Yeah, we've seen new techniques and I think new ways to scale, but that is kind of the lens through which we've been viewing things. Yeah, I think there is a certain amount of complexity that we need to grapple with and kind of everyone needs to grapple with because, you know, we're no longer really purely building a brain in the sky that's completely isolated from the real world, right? Like if you actually, if you want this model to do medical research, if you want it to cure cancer at some point it needs to learn about the real world in a meaningful way, you know, maybe conduct some experiment and learn from its results. And for that you need to figure out how to actually connect it, right? And that is going to involve something that goes in the direction you described but I don't think that goes counter to actually finding and scaling the simple algorithms that we've been developing.
I feel like I talk to a lot of companies and one of the main questions everyone seems to be asking these days is like should we be doing our own reinforcement learning? Like take an open source model and we have some data on a task that people do. We have evals because we know our domain pretty well. Is this something that makes sense for us to do or should we just wait for the models to continue to get better at some of these things? You know, what advice would you give for the many builders that listen to the podcast as they think through the extent to which they invest on the reinforcement learning side?
Reinforcement learning definitely can be a very data efficient way to really improve the model at some sort of task, right? There is a much more data efficient way of learning that we know, which is like learning in context, right? And this is maybe the most fundamental way that people teach these models, you just prompt them with examples, with instructions for what you want. I expect that learning is going to get much better over time. And so I think it definitely really matters that the models can adapt to your context. They can adapt to the kind of tasks you care about. So I think that will be very important. I'm not sure if replicating the current pipeline is going to be the right way to go about it. But yeah, it's definitely a problem that we're thinking about.
Yeah. So it's almost like, yeah, you still have to do the work, like you still should figure out what the evals are that matter, gather the data, the examples, but it may just turn out in the future you're far better off just feeding that into this context than trying to do anything on your own model.
Yeah, I think that's quite plausible.
And I think that, you know, obviously people have seen the success of tools like Codex which I know you've obviously been a key part of and wondered like, hey, do we need to build our own harnesses or our own ways of using these things for our own domains whether it's legal or finance or healthcare or do we kind of just take the harnesses that the large models do and use them within the context that we have. Any thoughts around that?
Like the implementation of the harness shouldn't really be a limitation for a very long time. I think we'll be able to get much more general harnesses that people can use for all sorts of other domains. I mean I think Codex is pretty good actually if you try using it for things beyond coding.
That's so interesting. Like a much more general harness being something that's almost adaptive to or just works across whatever the specific set of tools you have in your domain or specific set of things you want to expose to the model.
Yeah. I mean I think and it's also worth thinking about like what is the ultimate interface that we want to interact with the model with. So the models give some UI hard for humans, right? They can build their own UIs. They can do things that people would find very time-consuming. But I definitely think there is also just a lot of space to enable the models to access the current interfaces that we use for people, right. So I think like we want to have AIs on Slack for example that are plugged into our context and are able to learn from it and able to realize these existing things, right. So definitely there is some meet in the middle here but definitely I believe long-term, by default the AI should meet you where you are and if not that would be because it has new abilities, not because it has limitations.
Yeah, it's an interesting point that basically today it feels like these harnesses are so bespoke to certain environments, but over time as you add more and more skills and tools and models can navigate across those effectively, it's like there'd just be a general way, like the way humans have, that makes a tremendous amount of sense. I guess I'm curious, you know, obviously I'm sure like every day you see kind of crazy stuff on the research side at this point, like what are the milestones that are still meaningful to you as you think about like it would be pretty crazy if I did a run one day and saw like X or Y, like what are the things you're paying most attention to?
Yeah. I mean at this point it really is about research, right? Like is it about, can the model discover new things, can it execute on a longer horizon research problem.
It's almost like looking for some sort of insight that you're like, oh, someone on my team had come up with that, that would have been pretty intrigued by.
Yeah, we've actually had some minor but I think quite impactful ideas come from even like GPT 5.2 Pro that we're using entirely. But you know, I think it's still very very small compared to where I expect it to be.
Yeah, I mean it seems like almost inevitably these models are going to get better. They will be used in research. They'll be used in science more generally. You're like one of the first people interacting directly with these models as like research partners almost at this stage. Anything you've learned around the right way to do that or do you think about like what a research organization, as these models continue to get better, might look like?
Yeah, I think we're definitely kind of at a transition point where the short-term immediate quality of the model is about to be a quite determining factor for the pace of our research progress because the models are going to drive a lot of that. And so that definitely requires rewiring some intuitions about how to run a research organization. You know, normally you kind of try to not be too focused on immediate quality. You try to be much more focused on the longer term. I think we have a lot of very exciting stuff queued up that we are kind of working towards but I feel a lot of urgency to kind of yes, to actually execute on it and to actually use these advances in model intelligence to accelerate research on the AI and especially AI alignment.
Yeah, it's such a fascinating point because I've heard you talk before about running a research organization and I feel like in the past it was like giving people the space to pursue a lot of things that weren't directly, you know, hey, this is for a month or two months of progress, but it's like what are the ideas that are really going to drive things forward. But it makes total sense that we're in a time now where you're like, look, everything we do will be so much better if we just focus on this in the short term and make it better. It must be fascinating to navigate that and these maybe further off research ideas at the same time and running an organization.
Yeah. Yeah. It's definitely something we spend a lot of time on with Mark nowadays. Yeah.
Right now you have a ton of compute as a company, but you obviously you have great scaling laws on the pre-training side, you have great scaling on the RL side, you have probably lots of experiments going on that have nothing to do with either of those vectors, but are like interesting new ways. How do you even think about allocating compute across all of this stuff?
Yeah, it can get very complicated, right? Because there's so many things that we need to do. One thing we've been, one kind of discipline we've started keeping is we try to make sure we just explicitly budget a large chunk of our compute to the most scalable methods, to the things that we believe are the most responsible for driving general model intelligence. And you know, even if it's not the most efficient allocation of compute at all times because, you know, if you're allocating so much compute to one experiment or one set of experiments, there's so many things you can accelerate a little bit of that compute elsewhere. But I think it's easy to kind of with all the interesting and important things that we're doing, I think it'll be very easy to kind of paralyze all of it and not really end up doing the things that we believe are most important. You definitely want to understand the empirical evidence. You definitely want to make sure your evaluations are in order and the experimental rigor is there. And then you also want to apply some regularization based on like, okay, do we understand this method? Do we actually expect it will scale? Do we expect this is something you can actually build on in the future? Is this kind of a one-off? Right. And based on that determine the priority.
Yeah, it's so interesting. Probably find all the ways that you know you could improve things but they feel maybe like off a little bit to the side of where you think the overall arc of progress is and so you end up leaving some of these like low-hanging fruits to some extent because really the most important thing is finding the future direction and then the scaling within that and devoting compute toward that. Obviously the place where we talked about Codex a lot and the success of coding and it feels like last year was like the year of just incredible hill climbing on coding. I'm curious, you know, obviously Codex has been a super successful product in many ways, like Anthropic was kind of first to this market, you know, Claude Code was a dominant product there. What do you kind of, reflecting on that, I guess what do you make of the success Anthropic's had in this space?
Yeah, I think it's a matter of really focusing your product direction on where you believe the next application of the technology is, right? And if you look at the prioritization we've had on our product, I mean we have been working on cutting products but they have kind of been a secondary thing compared to our main priorities. And the interesting thing is that is not very reflective of the priorities of the research organization within OpenAI. I think given that we've kind of had this explosive success of ChatGPT, you know, I think ChatGPT is going to evolve quite a bit but as it was in 23, right, is this particular product that's maybe not, you know, I think it's definitely quite aligned with our vision of where AI is going, but it's not really representative of everything that it enables. And so the majority of our work in research has been focused on that future thing. And I think increasingly it has decoupled from our short-term product strategies, right? Yeah. I'm very confident about the things we've been building and the things we are building on the research on the model intelligence side. You know, a lot of our reprioritization and increased focus on the product side is about actually getting to deploy them and the belief that actually they are the thing that really matters now.
Yeah. And now it feels like clearly the whole company priority is so locked in and focused on this and you've seen just incredible improvement in Codex in recent months. For all the developers that listen to the podcast, like if again it's almost like hard to comprehend what the world looks like as these models keep hill climbing on longer and longer tasks, like what do you think will look different in their lives or how will they be using Codex in three, six months? I realize three months and six months are very different timelines in this world, but take whichever, whatever in between point you'd like.
I would expect just a gradual increase in just the level of autonomy you feel comfortable for the model, just the vagueness of description that can work with, the level of supervision it needs. I think we're not very far from models that can work autonomously for a couple days. Maybe use quite a bit more compute than they're using now and produce much higher quality artifacts on their own.
Do you have a gut instinct on like, you know, there's always been this question of like will the world, you know, do you need that software engineering skill set to supervise these models running for a few days or like, hey, does it turn out at some point of being able to run for a while, anybody can use coding agents and supervise them to some sort of output?
I mean I think definitely for a lot of outputs you already don't need much experience, right? I think still the distinction I would draw between an intern here and really an autonomous researcher or software engineer would be that if you want to build something bigger, you probably still want to apply supervision, you still kind of want to have an overarching thing, you want to recognize what building blocks fit in and which don't. But yeah, I definitely expect that desired skill set to shift quite a bit over time towards this more general vision setting.
You know, I guess on the research side, I feel like there's been, maybe like a month ago, I feel like all anyone could talk about was continual learning and there's just, you know, it was in the Zeitgeist. There's all these new labs starting to go focus on continual learning. Some folks left OpenAI to go focus on that. I'm curious, I think part behind that is a belief that RL alone either won't get us there or will get us to some level of very inefficient scaling and it's kind of different than the way humans learn. I think even I've heard you say before that RL is still very different today than the way that humans learn. What's your take on that whole movement?
Yeah, I am a little bit confused by it because in my mind the whole excitement that we've had, I mean even if you look at the titles of the GPT-3 paper, right, like it is that this class of models is actually capable of continual learning, right? It's capable of learning to learn in context, right? That has been really the driving force behind the excitement to scale these GPT models further. That has been the premise for why we really need to teach them with RL to learn in context more efficiently. And so I definitely agree that continual learning is really the thing, right? Like it's really the thing that we're building, but I don't really think this is a problem that's like, oh, you know, it's kind of ignored and off the path of what we're doing currently. I think it is what we're working towards.
Yeah. Like in your mind, this is like the single best path to get there is to continue to kind of scale the pre-training and RL.
I think that is kind of how we've made the most progress on this problem so far and you know, I think there are, I think that there definitely are more ideas, more steps. I think also a lot of improvements that will just come from scale.
Yeah, and I guess like, you know, we have a lot of folks listening that maybe have been able to do a lot of simpler things with these models and then they try to do some of these more complex, you know, I don't know, call it 100-step or longer-term tasks and they're like, oh, the models don't work for this yet. And I think it's harder, you on the inside constantly feel this improvement but for them it feels like, hey, this is like night and day away from being able to do this much longer thing. How do you kind of articulate to them the set of things that need to be true for these much longer steps to happen? Is it around kind of checking in more often as you were talking about before or I feel like there's just this belief among the research community of like, oh, all of these tasks will be solved in the next year or two and then in the wild a lot of people maybe not totally grokking that improvement line that we've been seeing.
Yeah. I mean I think a lot of that prediction comes from just looking at historical improvement lines, right? And but I think increasingly we can roughly see the shape here. I do think a lot of this is about just the models becoming intelligent enough to recognize whether they're making progress. I think some of this is like this very pragmatic work of like are the models actually, can they actually access all the context, all the files, all the infrastructure they need to do the work you want them to do. Which yeah, I remember like in the past when we were discussing the road map that we're taking with RL, you know, I definitely view like, okay, we just need to teach the model to kind of reason with its own tokens as kind of the priority and then of course we'll need it to use tools like the environment, you know, at some point we definitely need to teach it to see, right? At some point, we need to teach it to use a physical body, right? Like, but yeah, I mean, I think we're definitely well into the stage where it really needs to interact with the environment and it really needs to see and you know, someday soon we'll really care about robots, but yeah.
Yeah. I mean, it does feel like a lot of the times when I hear people complain about, oh, a model can't do X or Y, it's like literally just because you haven't fed, or connected it to systems or fed enough context into it. Actually, I do wonder if context was universally applicable and able to flow into these things. Like I feel like a lot of these problems would actually just be solved with today's models. You know, I want to talk about some of the AI for science stuff that you guys have been working on. And one thing in particular, you know, I feel like the coding stuff is something that everyone feels very viscerally, in every company they're using these tools and getting tons of productivity. You know, on the math side, not all of us competed in IMO competitions and necessarily have as much of an intuitive feel for some of these breakthroughs. And so one of them I know that was really interesting that you guys did is you used some compelling work around like first proof, right? And I think these are like very different problems than kind of traditional competition math. I wonder if you could just speak a little bit to that because I think it's just a space that our listeners might be less familiar with and kind of less familiar with understanding the implications of models being able to do pretty cool work here.
