OpenAI’s Chief Scientist on Continual Learning Hype, RL Beyond Code, & Future Alignment Directions

I definitely agree that continual learning is really the thing. It's really the thing that we're building. But I don't really think this is like a problem that's ignored and off the path of what we're doing currently. I think it is what we're working toward.

A lot of the like longer-term challenge with alignment is about generalization. What are the values that the model falls back on?

I come back to this.

Happy to be here.

Yeah, I think over the last months, the change that really happened is we've seen this explosive growth of coding tools.

It's an understatement. Yeah, we've definitely like really kind of gone to a place in OpenAI where we use Codex for the majority of actual coding. And so I think for most people, the act of programming has changed quite a bit. So I definitely see this as a signal that something here is on track. The other kind of like very interesting update over the last few months to me has been the progress on the math research capabilities. Also the results we've kind of seen in physics and other fields. I think this kind of level of capability, this level of ability to provide insight when combined with ability to access infrastructure, ability to use maybe more compute at test time that's something that Codex is using currently, and very strong improvement in general intelligence which I also expect over the next couple of months. Yeah, it's something we're still very much planning for and very focused on.

The way I would distinguish a research intern from a full automated researcher is the span of time that we would have it work mostly autonomously or the specificity of the task that has to be given. So I don't expect we'll have systems where you kind of just tell them, oh, like, you know, go improve your model capability, go solve alignment, and they will do it. Not this year. I think we might get there at some point, but I think for more specific technical ideas, like I have this particular idea how to improve the models, how to run this evaluation differently, I think we have the pieces that we mostly just need to put together.

Yeah, I think it's in the spirit. Yeah, I mean I expect it to look like a pretty continual evolution from kind of where Codex is now. I think towards a bit more autonomy, running for a longer time. But yeah, I think we'll see a lot of this sort of application. I think in general we'll see more autonomous and higher compute use of these models for different things.

The biggest role that focusing on these math benchmarks has played for us is as a general benchmark and a north star for how to improve this technology. Like math is very measurable, right? It's much easier to tell whether you've actually solved the math problem than whether you've even produced a good piece of software. And also it can get very hard, right? So you can have things where it's very definite whether you've solved them, but it can be arbitrarily pretty much hard to actually solve them. You know, I would say up until not too long ago, my perspective has been like, well, okay, our models are not maybe able to solve simple math problems. Okay, our models are able to solve simple enough problems but are not able to solve IMO level problems. So clearly there is just a gap in the intelligence of these models that is very measurable, very easy to run at. It's very clear what we need to do and this has been kind of our north star for reasoning models and so forth. Now of course that is changing quite a bit, right? And we have kind of reached these milestones that we've been working towards of IMO gold level, solving IMO problem six, and making progress into research level mathematics. And from this point, I think there still is utility in continuing to measure progress on this. I think there's also definitely transfer that you can get from getting better at mathematical reasoning to getting better at AI research. You know, a lot of our best researchers are mathematicians we're training or from other theoretical fields. But definitely we are very much changing how we think about these north stars and we are very focused on how the models, the next models that we're producing are actually useful in the real world, useful especially for research but also for other economically valuable activities and for other fields of science and especially maybe more applied sciences. And the reason for this shift is because we believe the models are now capable enough, not as smart as people always, but capable enough to actually materially change the economy, change how things are done. And so, yeah, we feel a lot of urgency about that.

Yeah, I definitely expect so. I think an interesting duality that we think about a lot is for these more general tasks, for these tasks that are kind of harder to evaluate, they share a lot of commonalities with just longer horizon tasks, right? Because if you think about even a very well specified math or coding problem, again, if it's something that you need to work on for like a year, then even if it's very clear what the criteria of success are in the long term, like what to do on your first day of working on it is a pretty open-ended problem. And so I kind of believe these difficulties coincide and they're very clearly the next frontier for how these systems develop. And I think we've definitely seen very encouraging signs both on just our ability to scale RL on these more general domains. And I think also we can scale efforts that have a lot of promise.

Yeah, I think I come back to this reality of like how do we make the models work for a very long time and how do we teach them to evaluate kind of partial progress.

Yeah, it's definitely a research challenge to figure out how to leverage these new ideas from RL and so forth to apply this to general domains. But I'm quite optimistic about that.

Well, we are buying a lot of compute, right? Because we still believe a bit less and we believe more than ever to some degree. Yeah, we've seen new techniques and I think new ways to scale, but that is kind of the lens through which we've been viewing things. Yeah, I think there is a certain amount of complexity that we need to grapple with and kind of everyone needs to grapple with because, you know, we're no longer really purely building a brain in the sky that's completely isolated from the real world, right? Like if you actually, if you want this model to do medical research, if you want it to cure cancer at some point it needs to learn about the real world in a meaningful way, you know, maybe conduct some experiment and learn from its results. And for that you need to figure out how to actually connect it, right? And that is going to involve something that goes in the direction you described but I don't think that goes counter to actually finding and scaling the simple algorithms that we've been developing.

Reinforcement learning definitely can be a very data efficient way to really improve the model at some sort of task, right? There is a much more data efficient way of learning that we know, which is like learning in context, right? And this is maybe the most fundamental way that people teach these models, you just prompt them with examples, with instructions for what you want. I expect that learning is going to get much better over time. And so I think it definitely really matters that the models can adapt to your context. They can adapt to the kind of tasks you care about. So I think that will be very important. I'm not sure if replicating the current pipeline is going to be the right way to go about it. But yeah, it's definitely a problem that we're thinking about.

Yeah, I think that's quite plausible.

Like the implementation of the harness shouldn't really be a limitation for a very long time. I think we'll be able to get much more general harnesses that people can use for all sorts of other domains. I mean I think Codex is pretty good actually if you try using it for things beyond coding.

Yeah. I mean I think and it's also worth thinking about like what is the ultimate interface that we want to interact with the model with. So the models give some UI hard for humans, right? They can build their own UIs. They can do things that people would find very time-consuming. But I definitely think there is also just a lot of space to enable the models to access the current interfaces that we use for people, right. So I think like we want to have AIs on Slack for example that are plugged into our context and are able to learn from it and able to realize these existing things, right. So definitely there is some meet in the middle here but definitely I believe long-term, by default the AI should meet you where you are and if not that would be because it has new abilities, not because it has limitations.

Yeah. I mean at this point it really is about research, right? Like is it about, can the model discover new things, can it execute on a longer horizon research problem.

Yeah, we've actually had some minor but I think quite impactful ideas come from even like GPT 5.2 Pro that we're using entirely. But you know, I think it's still very very small compared to where I expect it to be.

Yeah, I think we're definitely kind of at a transition point where the short-term immediate quality of the model is about to be a quite determining factor for the pace of our research progress because the models are going to drive a lot of that. And so that definitely requires rewiring some intuitions about how to run a research organization. You know, normally you kind of try to not be too focused on immediate quality. You try to be much more focused on the longer term. I think we have a lot of very exciting stuff queued up that we are kind of working towards but I feel a lot of urgency to kind of yes, to actually execute on it and to actually use these advances in model intelligence to accelerate research on the AI and especially AI alignment.

Yeah. Yeah. It's definitely something we spend a lot of time on with Mark nowadays. Yeah.

Yeah, it can get very complicated, right? Because there's so many things that we need to do. One thing we've been, one kind of discipline we've started keeping is we try to make sure we just explicitly budget a large chunk of our compute to the most scalable methods, to the things that we believe are the most responsible for driving general model intelligence. And you know, even if it's not the most efficient allocation of compute at all times because, you know, if you're allocating so much compute to one experiment or one set of experiments, there's so many things you can accelerate a little bit of that compute elsewhere. But I think it's easy to kind of with all the interesting and important things that we're doing, I think it'll be very easy to kind of paralyze all of it and not really end up doing the things that we believe are most important. You definitely want to understand the empirical evidence. You definitely want to make sure your evaluations are in order and the experimental rigor is there. And then you also want to apply some regularization based on like, okay, do we understand this method? Do we actually expect it will scale? Do we expect this is something you can actually build on in the future? Is this kind of a one-off? Right. And based on that determine the priority.

Yeah, I think it's a matter of really focusing your product direction on where you believe the next application of the technology is, right? And if you look at the prioritization we've had on our product, I mean we have been working on cutting products but they have kind of been a secondary thing compared to our main priorities. And the interesting thing is that is not very reflective of the priorities of the research organization within OpenAI. I think given that we've kind of had this explosive success of ChatGPT, you know, I think ChatGPT is going to evolve quite a bit but as it was in 23, right, is this particular product that's maybe not, you know, I think it's definitely quite aligned with our vision of where AI is going, but it's not really representative of everything that it enables. And so the majority of our work in research has been focused on that future thing. And I think increasingly it has decoupled from our short-term product strategies, right? Yeah. I'm very confident about the things we've been building and the things we are building on the research on the model intelligence side. You know, a lot of our reprioritization and increased focus on the product side is about actually getting to deploy them and the belief that actually they are the thing that really matters now.

I would expect just a gradual increase in just the level of autonomy you feel comfortable for the model, just the vagueness of description that can work with, the level of supervision it needs. I think we're not very far from models that can work autonomously for a couple days. Maybe use quite a bit more compute than they're using now and produce much higher quality artifacts on their own.

I mean I think definitely for a lot of outputs you already don't need much experience, right? I think still the distinction I would draw between an intern here and really an autonomous researcher or software engineer would be that if you want to build something bigger, you probably still want to apply supervision, you still kind of want to have an overarching thing, you want to recognize what building blocks fit in and which don't. But yeah, I definitely expect that desired skill set to shift quite a bit over time towards this more general vision setting.

Yeah, I am a little bit confused by it because in my mind the whole excitement that we've had, I mean even if you look at the titles of the GPT-3 paper, right, like it is that this class of models is actually capable of continual learning, right? It's capable of learning to learn in context, right? That has been really the driving force behind the excitement to scale these GPT models further. That has been the premise for why we really need to teach them with RL to learn in context more efficiently. And so I definitely agree that continual learning is really the thing, right? Like it's really the thing that we're building, but I don't really think this is a problem that's like, oh, you know, it's kind of ignored and off the path of what we're doing currently. I think it is what we're working towards.

I think that is kind of how we've made the most progress on this problem so far and you know, I think there are, I think that there definitely are more ideas, more steps. I think also a lot of improvements that will just come from scale.

Yeah. I mean I think a lot of that prediction comes from just looking at historical improvement lines, right? And but I think increasingly we can roughly see the shape here. I do think a lot of this is about just the models becoming intelligent enough to recognize whether they're making progress. I think some of this is like this very pragmatic work of like are the models actually, can they actually access all the context, all the files, all the infrastructure they need to do the work you want them to do. Which yeah, I remember like in the past when we were discussing the road map that we're taking with RL, you know, I definitely view like, okay, we just need to teach the model to kind of reason with its own tokens as kind of the priority and then of course we'll need it to use tools like the environment, you know, at some point we definitely need to teach it to see, right? At some point, we need to teach it to use a physical body, right? Like, but yeah, I mean, I think we're definitely well into the stage where it really needs to interact with the environment and it really needs to see and you know, someday soon we'll really care about robots, but yeah.

Yeah, I mean, you know, I think yeah, I was very excited with the first proof challenge and you know, again, like I kind of, you particular one is kind of a

Benchmark, right? It's like a couple of respected mathematicians and theoretical computer scientists releasing problems that they believe are representative of their day-to-day work but haven't been published anywhere, so that we can really have our models take a crack. We were so excited about this challenge, but it was kind of dropped without any advanced warning with a week-long deadline to actually execute. We had a very exciting model training at the time, and so one of the people in charge of training, James Lee, kind of started prompting that model just by hand. And actually kind of seeing, oh okay, it's actually solving these problems, was really a fascinating thing to see. One of these problems actually is from a domain that I did my PhD in, and yeah, seeing the model kind of come up with these ideas which I would be quite proud to come up with in a week or two, seeing it come up with them in like an hour or so, that was a very weird feeling. I think in the past when I felt like that was when watching our data bot play just very interesting data games infinitely, and it feels like there's some sort of magic happening because interesting things should not be indefinite.

And so seeing that happen for math, for something that I believe is actually quite representative of our work or a precursor to a lot of the work that we're doing and a lot of the work that really matters in the world, yeah, definitely really increased my feeling of urgency.

I mean, it is at a stage where we needed to seek out experts in the particular domains to be able to tell us whether these particular proofs are correct or not, but it's still much easier to tell whether you've actually made progress than for something like even coding, right? Because sure, competitive programming you can evaluate, but most programming is not competitive programming, and it's about are the abstractions right, are you handling all the cases.

I believe so, yeah. I mean, I think kind of on schedule we're starting to see minor advancements, right? Not huge things, a small idea here or there. I mean, maybe some bigger papers in collaboration with scientists. But, you know, was AlphaZero a pattern matcher? AlphaGo a pattern matcher? Our data bots, they did kind of come up with new strategies for the respective games.

It's funny that there are counterexamples to it all the way back to 2016, 2017.

And you can say, well, I guess you can always fall back on flaws in that, which I think is interesting. Like AlphaGo can be beaten with some strategy, our data bots could have been beaten with some strategy. I think there will be a lot of definitions for a while of these models. But I think also they are able to discover new things because they have a lot of these capabilities. And the way, you know, it's taken a couple years to go from very tiny game environments to much more general scientific research. It required kind of going through a decent approximation of all human knowledge in the meantime and learning all the human languages and so forth, but I think the basic principle is very similar.

It doesn't concern me. I mean, I think it's expected. I'm sure I thought for at least one of the problems that actually our model produced a pretty nice proof that was quite a bit shorter than the intended one. But I think in general you would expect these models, they can produce so much more reasoning in a short time than a person can, just in terms of raw number of tokens or thoughts. I don't expect that to be a long-term feature.

Yeah, I think it's kind of similar to my answer about the UI for Codex, which I would build around the capabilities of a technology and not around its limitations so much. So you definitely, if you have something that can suddenly design a huge amount of interesting chemical or biological experiments, yeah, it makes sense to build labs that enable that. I think if we did get to a place where the model is very capable of designing high-quality experiments, it also makes sense to have it work with humans in the loop. We shouldn't think of it as either you kind of automate it fully and you have this fun thing using some tools on the side. We will get to a world where it's just very natural to be collaborating with AI scientists that are working hard on a problem.

I mean, I wouldn't necessarily categorize it as... of course there are tools in some sense, but I think we will get to a point where they're driving a lot of the design and ideation for the whole process. Yeah, with an LLM architecture, but just being able to figure out the right way, the right kinds of experiments to run and then actually design it. And yeah, when it comes to different architectures, for sure, natural language reasoning, the kind of things that we're prioritizing, that gives you a lot of generality. There are things that you kind of want to train a different model to model. I think even if you want to create a very good video model, I don't think large language models are the most efficient way to go about this, although they might result in the best model eventually. But I think it's similar for protein folding or other tasks of this kind.

Yeah. I want to also make sure just to talk about AI safety because I think that's an area that you've done a lot of really pioneering work on. And I'm not sure all our listeners will be familiar with, you actually did some really interesting work across the labs, right? And were focused on chain of thought monitoring. And so maybe to start, just tell us a little bit about that work and what you found.

And it always tells me I have great ideas.

And for me that was the primary motivation. That was the reason, I didn't really even want to consider releasing it in different ways. There definitely was a bit of internal discussion about this, but the reason I felt very strongly that we should just hide it is because of this. Then there was this other concern that I didn't initially think about but I think was also very valid, of like, well, this model is going to be distilled to some extent, and that's definitely also been a big factor here. But I actually think that allowing the models some sort of private space... oh, and by the way, why do I think it's important that we don't show this chain of thought in product? If I'm saying the important thing is not to supervise them during training, well, I think if we did show it, if we established a paradigm where you just show these chains of thought in product, eventually you kind of have to train them, right? Like you'll have to train them for the same reasons you have to train whatever models you ship. And I just think that...

Right. I mean, I think it'll be useful to some extent, and we are trying to capture most of that value, either with chain of summaries, which I think are kind of a little bit of a stopgap. I think the longer-term solution here is having the model actually talk to you in real time, which the latest version of Codex kind of does, the latest version of the reasoning GP models kind of do, but I think that will get much better. But yeah, I think there's something very exciting here about just not having the training signal fight against us. And not... yes, because I think if you want to be able to understand what the model does in the long term, but you're scaling a method that is kind of going directly against that, you're probably not going to have a good time, right? That's the other side of the better lesson. And so this decoupling, I think, is an idea that gives me a lot of hope for our ability to at least understand how these models' motivations and generalization evolve as they get better, as they work for longer. I don't think it's a complete solution to AI alignment by a long shot. I think it's just another tool in our toolbox. But I am hopeful that building our toolbox with technical tools like this, we can actually continue chipping away at the fundamental problems here.

Yeah, I mean, I think it's a tool that can help us understand, I think it's actually very useful to build understanding of long-term alignment. For example, there has been this very exciting work from a planning collaboration with other labs on model scheming, where they investigate depending on what environment you put the model in, how you train it, is it prone to start having hidden objectives that it pursues, and what enables that. That whole line of work is chain of thought monitoring, right? Is this notion of, oh, you can actually inspect what the model's motivations are. And I think from that, that might take us in a completely different direction in terms of mitigations. Maybe the right way is changing the pre-training data of the model, or maybe it's something like the inoculation prompting from Anthropic. I think those are very interesting ideas, but I think having this ability to understand is very helpful to evaluate these.

Yeah, I think a lot of the longer-term challenge with alignment is about generalization. We can train our models to do well, and or at least mostly to some extent, we can mostly kind of control their behavior in the things that are in distribution that we train for. But the things that are worrisome is what happens when a model is asked to do something very different, or it finds itself in a very different situation, or it's much smarter than it ever was before and it has all these capabilities. It's like we haven't really thought about how to train for. And so I think the study of this kind of longer-term value alignment is really a study of generalization, like what are the values that the model falls back on. One line of research I'm very excited about here, and something that we're investing in quite a bit, is understanding how that generalization falls back onto the pre-training data. And yeah, I think there's quite a lot there.

I will speak to the longer-term challenges of alignment, or what happens when you have very smart models. The way my thinking about the problem has evolved over the past few years has definitely kind of gone from, you know, oh, is this like very nebulous problem that is just very hard to even grapple with or define, to like, oh, you know, I think we can actually make progress at it by very concrete technical solutions and technical insights. And this is why we've really been viewing alignment as just a core part of research and really making sure that we are designing our reasoning models thinking about this, and we are conducting our alignment research with these reasoning models in mind and so forth. So I think my general belief that there's a research path here that actually gets us to an extremely happy world has increased quite a lot. At the same time, my timelines to very capable models have definitely decreased a lot. I think we're not that far. Again, I don't think these are models that are smarter than all of us, but I think these are models that are just very transformative. And so I'm quite optimistic we can keep a good grip on how we're doing on the alignment problem, how to roughly evaluate the risks of our models or the problems with them. But I do think we have to be, as an industry, really prepared to take trade-offs and possibly slow down development depending on what we see.

There's definitely some, I mean, there's definitely shared interest in these topics. Yeah.

I focus on just high-quality experiments, recognizing are we actually making progress, being honest with ourselves, and promoting honesty about the results. I don't think that has changed. And even though our work will evolve a lot, I believe we still have quite a lot of work left to do. And so I don't think it's like, oh, we need to wrap up all our projects very quickly. So yeah, I don't think those fundamentals change. I think what does change is a level of urgency to really kind of bring some of these things that we think are most promising to fruition.

Well, yeah. I mean, there's certainly a number of dramatic moments like this. I think the ways the company underwent the most change is not really these snap decisions, but more like shifts in how it operates. I would say OpenAI has gone through a couple phases. When I joined at the start of 2017, it very much felt like a very academic lab pursuing a lot of different ideas, not so much the scaling pill in practice. And I think that was the first big change with the data product, with GPT, we've kind of moved to, okay, we actually are going to have to buy big computers, we're actually going to have to scale things, we're going to have to develop the science of scaling, we'll have to develop the infrastructure for it. And so that kind of started the second phase of, okay, now we're scaling. We're still going to pursue a lot of these basic research ideas, but we are going to evaluate them for, are they scalable? Then yeah, then there was this interesting period I talked about earlier, where you kind of have ChatGPT as this big thing.

No, it certainly seems that way.

Yeah, I mean, I think it's really starting to reconcile this tension between the AI that you build ultimately is something that affects the world, but until you kind of get pretty close, it's a pretty theoretical thing that you're just kind of training and developing algorithms for. And so recognizing that, okay, now we actually need to make a lot of progress and focus on how we're actually deploying this technology in the wild. This is definitely something I've been thinking about a lot lately.

Yeah, I believe it's kind of going in the same direction as the coding models, where it's actually going to be something very useful, it's going to be something that's a meaningful part of people's lives.

I feel that's part of it, but also just coming to become a dependable, trustworthy assistant or companion.

Yeah. I mean, I definitely think there are very promising algorithmic ideas there that I believe are going to work that are not too dissimilar from the space of ideas. So I'm quite optimistic about timelines there. Although I do think they're longer than the kind of virtual AI.

Yeah, I think getting to a point where so much intellectual work can be automated, I think comes with pretty big problems that I don't think have obvious solutions. One natural is a question of jobs and concentration of wealth, and I suspect this requires real policymaker involvement. I've heard some kind of optimistic takes on how this is resolved, but I think at a fundamental level it does seem like some things that used to be very valuable, used to cost a lot, and used to provide something, now can be done pretty cheaply. And in the long term it should be a good thing, but I think it can happen quite quickly. And there is a related question of, you really can, if you actually have an automated research laboratory, an automated company that can do so many things, it can be controlled by a very small number of people, right? It can do a lot. And this gets even more crazy when you have robots, but you don't need to have robots. And I think figuring out what does governance of such things look like, what are these organizations that are so powerful and yet maybe made of only a couple of people, how to think about these things, I think is a new question we have to grapple with as a society.

A task for all of us is to build the AI, build a world in a way where at the end of the day humans have the agency, humans set the direction. And maybe a lot of the technical challenges that we cherish right now will become more of a pastime, that's something that we really kind of need to do in order to make progress, and the challenges will be more in figuring out what are the things that are important, what are the things we should go do. I think that will still be, in that world, people can end up with more things to do and definitely more exciting things to do. And I think you still want to have an understanding of technology, all the kind of basic education however you want to acquire it, for the sake of being able to think about these problems.

I think the set of problems we just discussed, and also the questions around alignment, monitorability, I think those are growing to be very urgent challenges. And I don't think there are challenges only for AI researchers. I think there are challenges for policymakers, but also just things we have to think through as a society. And yeah, I'm happy to see some discourse starting to arise and I think we need more of it.

Thank you.