The big thing that we are targeting is producing an automated researcher. So automating the discovery of new ideas. The next set of evals and milestone that we're looking at will involve actual movement on things that are economically relevant. And I was talking to some high schoolers and they're saying, 'Oh, you know, actually the default way to code is vibe coding.' I do think you know the future hopefully will be vibe researching.
Thanks for coming Jacob and Mark. Jacob, you're the chief scientist at OpenAI. Mark, you are the chief research officer at OpenAI and you guys have both the privilege and the stress of running probably one of the most high-profile research teams in AI. And so we're just really stoked to talk with you about a whole bunch of things we've been curious about, including GPT5, which was, you know, one of the most exciting updates to come out of OpenAI in recent times. And then stepping back, how you build a research team that can do not just GPT5 but Codex and ChatGPT and an API business and can weave all of the many different bets you guys have across modalities, across product form factors into one coherent research culture and story.
And so to kick things off, why don't we start with GPT5? Just tell us a little bit about the GPT5 launch from your perspective. How did it go?
So I think GPT5 was really our attempt to bring reasoning into the mainstream. Prior to GPT5, we had two different series of models: you had the GPT kind of 2, 3, 4 series which were kind of these instant response models, and then we had an O series which essentially thought for a very long time and then gave you the best answer that it could give. So tactically, we don't want our users to be puzzled by, you know, which mode should I use, and it involves a lot of research in kind of identifying what the right amount of thinking for any particular prompt looks like and taking that pain away from the user. So we think the future is about reasoning more and more, about agents, and we think GPT5 is this step towards delivering reasoning and more agentic behavior by default. There is also a number of improvements across the board in this model relative to O3 and our previous models, but our primary focus for this launch was indeed bringing the reasoning mode to more people.
Can you say more about how you guys think about evals? I noticed even in that launch video there were a number of evals where you were inching up from, you know, 98 to 99% and that's kind of how you know you saturated the eval. What approach do you guys take to measuring progress and how do you think about it?
One thing is that indeed for like these evals that we've been using for the last few years, they're indeed pretty close to saturated and so yeah, for a lot of them, inching from like 96 to 98% is not necessarily the most important thing in the world. I think another thing that's maybe even more important but a little bit subtler: when we were in this like GPT2, GPT3, GPT4 era, there was kind of one recipe: you just pre-train a model on a lot of data and you kind of use these evals as just kind of a yardstick of how this generalizes to different tasks. Now we have different ways of training, in particular reinforcement learning on like serious reasoning where we can pick a domain and we can really train a model to become an expert in this domain, to reason very hard about it, which lets us target particular kinds of tasks, which will mean that we can get extremely good performance on some evals but it doesn't indicate as great generalization to other things. So we think about it in this world, we definitely think we are in a little bit of a deficit of great evaluations. And I think the big things that we look at are actual marks of the model being able to discover new things. I think for me the most exciting thread and actual sign of progress this year has been our model's performance in math and programming competitions, although I think they are also becoming saturated in a sense. And the next set of evals and milestone that we're looking at will involve actual discovery and actual movement on things that are economically relevant.
Totally. You guys already got number two in the AtCoder competition. So there's really only number one left.
Yeah. I mean I think it is important to note that these evals like AtCoder, IMO are actually real world markers for success in future research. I think a lot of the best researchers in the world have gone through these competitions, have gotten very good results, and yeah, I think we are kind of preparing for this frontier where we're trying to get our models to discover new things.
Yeah, very exciting. Which capability from GPT5 before the release surprised you the most when you were working through the eval bench or using it internally? Were there any moments where you felt like this was starting to get good enough to release because it was useful in your daily usage?
I think one big thing for me was just how much it moved the frontier in very hard sciences. We would try the models with some of our friends who are professional physicists or professional mathematicians and you already saw kind of some instances of this on Twitter where you can take a problem and have it discover maybe not very complicated new mathematics, but some non-trivial new mathematics. And we see physicists, mathematicians kind of repeating this experience over and over where they're trying problems and saying, 'Wow, this is something that the previous version of the models couldn't do.' And it is a little bit of a light bulb moment for them. It's able to automate maybe what could take one of their students months of time.
Well, GPT5 is a definite improvement on O3. For me, O3 was definitely like that moment where the reasoning models became actually very useful on a daily basis. I think especially for working through a math formula or a derivation, it actually got to a level where it is fairly trustworthy and I can actually use it as a tool for my work. And yeah, I think it is very exciting to get to that moment. But I expect that, well now as we're seeing these models actually able to automate, well yes, like we're saying, solving contest problems over longer time horizons, I expect that was quite small compared to what's coming over the next year.
What is coming in the next one to five years? It would be just at whatever level you're comfortable sharing, what does the research road map look like?
So the big thing that we are targeting with our research is producing an automated researcher. So automating the discovery of new ideas. And of course, a particular thing we think about a lot is automating our own work, automating ML research. But that can get a little bit self-referential. So we're also thinking about automating progress in other sciences. And I think one good way to measure progress there is looking at what is the time horizon on which these models actually can reason and make progress. And so now as we kind of get to a level of near mastery of these high school competitions, let's say, we get to maybe on the order of one to five hours of reasoning. And so we are focused on extending that horizon both in terms of the models' capability to plan over very long horizons and actually able to retain memory.
And back to evals, that's why I think eval of the form of how long does this model autonomously operate for are of particular interest to us.
And actually maybe on that topic, there's been this huge move toward agency and model development. But I think at least the state that it's in currently, users have sort of observed this trade-off between too many tools or planning hops can result in quality regressions versus something that maybe has a little bit less agency, the quality is at least observed today to be a bit higher. How do you guys think about the trade-off between stability and depth? The more steps that the model is undertaking maybe the less likely the tenth step is to be accurate versus you ask it to do one thing it can do it very well and to have it keep doing that one thing better and better but more complex things there's sort of that trade-off. But of course to get to full autonomy you are taking multiple steps, you're using multiple tools.
I think actually the ability to maintain depth is a lot of it is being consistent over long horizons. So I think they are very related problems. And in fact I think with the reasoning models we have seen the models greatly extend the length over which they are able to reason and work reliably without going off track. Yeah, I think this is going to remain a big area of focus for us.
Yeah. And I think reasoning is core to this ability to operate over a long horizon because you know, you imagine yourself solving a math problem, right? You try an approach, it doesn't work, and you have to think about what's the next approach I'm going to take, what are the mistakes in the first approach, and then you try another thing. And the world gives you some hard feedback, right? And then you keep trying different approaches and the ability to do that over a long period of time is reasoning and gives agents that robustness.
We talked a lot about math and science. I'm curious to get your take on: do you think some of the progress that we've made can actually extend similarly to domains that are less verifiable? They're sort of less of an explicit right or wrong.
Oh yeah, this is a question I really like. I think if you actually truly want to extend to research and discovering ideas that meaningfully advance technology on the scale of months and years, I think these questions stop being so different. It is one thing to solve a very well-posed constraint problem on the scale of an hour, and there's a finite amount of ideas you need to look through and that might feel extremely different from solving something very open-ended. But even if you want to solve a very well-defined problem that is on a much longer scale, like prove this Millennium Prize problem, well that suddenly requires you to think about what are the fields of mathematics or other science that might possibly be relevant, are there inspiration from physics that I must take, what is the entire program that I want to develop around this, and now these become very open-ended questions. And it's actually hard to, for our own research, if all we cared about is reduce the modeling loss on a given data set, measuring the progress on that, are we kind of actually asking the right questions in research, actually becomes a fairly open-ended affair.
Yeah. And I think it also makes sense to think about what the limits of open-ended means. I think a while back Sam tweeted about some of the improvements that we were making in having our models write more creatively. And we do consider the extremes here as well.
Right. Let's talk about RL because it seems like since O1 came out, RL has been the gift that keeps giving. Every couple months OpenAI puts out a release and everyone goes, 'Oh, that's great, but this RL thing is going to plateau. We're going to saturate the evals. The models won't generalize or there's going to be mode collapse because of too much synthetic data for whatever.' Everybody's got a laundry list of reasons to believe that the gains and performance from RL are going to tap out and somehow they just don't. You guys just keep coming out and putting out continuous improvements. Why is RL working so well and what if anything has surprised you about how well it works?
RL is a very versatile method, right? And there are a lot of ideas you can explore once you have an RL system working. A long time at OpenAI, we started from this before language models, right? We were thinking about RL is this extremely powerful thing of course on top of deep learning which is this incredible general learning method. But the thing that we struggled with for a very long time is what is the environment, how do we actually anchor these models to the real world or should we simulate some island where they all learn to collaborate and compete. And then of course came the language modeling breakthrough and we saw that if we scale deep learning on modeling natural language we can create models with this incredibly nuanced understanding of human language. And so since then we've been seeking how to combine these paradigms and how to get RL to work on natural language. Once you do, then you kind of have the ability to actually execute on these different ideas and objectives in this extremely robust rich environment given by pre-training. And so yeah, I think it's been a real, yeah, I think it's been perhaps the most exciting period in our research over the last few years where we've really found so many new directions and promising ideas that all seem to be working out and we're trying to understand how to compare.
One of the hardest things about RL for folks who are not practitioners of RL is the idea of crafting the right reward model. And so, especially if you're a business or an enterprise who wants to harness all this amazing progress you guys are putting out, but doesn't even know where to start. What do the next few years look like for a company like that? What is the right mindset for somebody who's trying to make sense of RL to craft the right reward model? Is there anything you've learned about the best practices or an approach of thinking of using this latest family of reasoning techniques? What is the right way I should think about even approaching reward modeling as a biologist or a physicist?
I expect this will evolve quite rapidly. I expect it will become simpler. Maybe two years ago we would have been talking about what is the right way to craft my fine-tuning data set and I don't think we are at the end of that evolution yet and I think we will be inching towards more and more human-like learning which RL is still not quite. So I think maybe the most important part of the mindset is to not assume that what is now will be forever.
So I want to bring the conversation back to coding. We would be remiss not to say congrats on GPT5 Codex which just dropped today. Can you guys say a little bit more about what's different about it, how it's trained differently, maybe why you're excited about it?
Yeah, so I think one of the big focuses of the Codex team is to just take the raw intelligence that we have from our reasoning models and make it very useful for real world coding. So a lot of the work they've done is kind of consistent with this. They are working on having the model be able to handle more difficult environments. We know that real world coding is very messy. So they're trying to handle all the intricacies here. There's a lot of coding that has to do with style, with just softer things like how proactive the model is, how lazy it is and just being able to define in some sense a spec for how a coding model should behave. They do a lot of very strong work there and as you see, they're also working on a lot better presets. Coders, they have some kind of notion of this is how long I'm willing to wait for a particular solution. I think we've done a lot of work to dial in on for easy problems being a lot lower latency, for harder problems actually the right thing is to be even higher latency, get you the really best solution and just being able to find that preset sweet spot for easier problems versus harder.
What we've found is the previous generation of the Codex models, they were spending too little time solving the hardest problems and too much time solving the easy problems. And I think that is actually just probably out of the box what you might get out of O3.
Maybe just on the topic of coding since you guys are both competitive coders in prior lives. I know you've been at OpenAI for almost a decade now, but I was struck by the story of Lee Sedol, the Go player who kind of famously quit Go after he lost to AlphaGo multiple times. And I think in a recent interview you guys were both saying that now the coding models are better than your capabilities. And that gets you excited. Say more about that. And how much would you say you code now? Well, if you're hands-on keyboard, you can talk about OpenAI generally, but how much code is written by AI now?
In terms of coding models being better? I mean, I think yeah, I think it is extremely exciting to see this progress. I think the programming competitions have a nice kind of encapsulated test of ability to come up with some new ideas in this boxed environment and time frame. I do think if you look at things like the IMO problem six or maybe some very hardest programming competition problems, I think there's still a little bit of headway to go for the models but I wouldn't expect that to last very long. Historically I've been extremely reluctant to use any sort of tools. I just used Vim pretty much.
Yeah. Eventually I think especially with this latest coding tools, like GPT5, I've really kind of felt like okay, this is no longer the way. You can do a 30-file refactor pretty much perfectly in 15 minutes, you kind of have to use it. And so I've been learning this new way of coding which definitely feels a little bit different. I think it is a little bit of an uncanny valley still right now where you kind of have to use it because it is just exciting so many things but it's still a little bit not quite as good as a coworker. So I think our priority is getting out of that uncanny valley.
Yeah. But yeah, it's definitely an interesting time.
Yeah, definitely. To kind of speak to the Lee Sedol moment, I think AlphaGo for both of us was a very formative milestone in AI development. And at least for me, it was the reason I started working on this in the first place. And maybe partly because of our backgrounds in competitive programming, I had this affinity to building these models which could do very well in these forms of contests. And going from solving eighth-grade math problems to a year later hitting our level of performance in these coding contests, it's crazy to see that progression. And you kind of imagine or like to think that you feel a set of the feelings that Lee Sedol felt too, right? It's like wow this is really crazy, and what are the possibilities, and this is something that I took decades to do and it took a lot of hard work to get to the forefront of. So you really do feel an implication of that is these models, what can't they do? And I do feel like already it's kind of transformed the default for coding. This past weekend I was talking to some high schoolers and they were saying, 'Oh, you know, actually the default way to code is vibe coding.' I think they would consider, oh, maybe sometimes for completeness you would go and actually do all of the mechanics of coding it from scratch yourself, but that's just a strange concept to them. Like why would you do that? Just vibe code by default. And so yeah, I do think the future hopefully will be vibe researching.
Yeah. I have a question about that which is what makes a great researcher? When you say vibe researching, there's a big part of vibe coding is just having good taste in wanting to build something useful and interesting for the world. And I think what's so awesome about tools like Codex is if you've got a good intuition for what people want, it helps you articulate that and then basically actualize a prototype very fast. With research, what's the analog? What makes a great researcher?
Persistence is a very key trait. What is different about research when you're actually trying to, I think a special thing about research is you're trying to create something or learn something that is just not known, right? It's not known to work, you don't know whether it will work. And so always trying something that will most likely fail. And I think getting to a place where you are in a mindset of being ready to fail and being ready to learn from these failures. And of course with that comes creating clear hypotheses and being extremely honest with yourself about how you're doing on them. I think a trap many people fall into is going out of the way to prove that it works, which is quite different from believing in your idea and its significance is extremely important, and you want to persist with that, but you have to be honest with yourself about when it's working and when it's not so that you can learn and adjust.
Yeah, I think there are just very few shortcuts for experience. Through experience you kind of learn what's the right horizon to be thinking of a problem. You can't pick something that's too hard or it's not satisfying to do something that's too easy. And I think a lot of research is managing your own emotions over a long period of time, too. There's just going to be a lot of things you try and they're not going to work. And sometimes you need to know when to persevere through that or sometimes when to switch to a different problem. And I think interestingness is something you try to fit through reading good papers, talking to your colleagues, and you kind of maybe distill their experience into your own process.
When I was in grad school, there's a big part, I'm a failed machine learning researcher. I was in grad school for bioinformatics. But a big part of my research advisor's thrust was about picking the right problems to work on such that you could then sustain and persist through the hard times. And you said something interesting which was there's a difference between having conviction in an idea and then being maximally truth-seeking about when it's not working. And though both those things are sometimes in tension because you kind of go native on a topic or a problem sometimes that you have deep conviction in. Have you found, is there any sort of heuristics you found are useful at the taste step, at the problem-picking step that help you arrive at the right set of problems where that conviction and truth-seeking is not as much in zero-sum tension as other kinds of problems?
Yeah, to be clear I don't think conviction and truth-seeking are really in a zero-sum tension. I think you can have a lot of belief in an idea and you can be very persistent in it while it's not working. I think it's just important that you're kind of honest with yourself how much progress you're making and you're in a mindset where you're able to learn from the failures along the way. I think it's important to look for problems that you really care about and you really believe are important. And so I think one thing I've observed in many researchers that inspired me has been really going after the hard problems, looking at the questions that are widely known but not really considered tractable and just asking why are they not tractable or what about this approach, why does this approach fail. You're always thinking about what is really the barrier for the next step. If you're going after problems that you really truly believe are important, then that makes it so much easier to find the motivation to persist with them over years.
And in the development of, like during the retraining phase of GPT5 for example, were there any moments where there was a hard problem, the original initial attempts that were being made to crack that problem weren't working and yet you found somebody...
Persisted through that. And what was it about those stories that comes to mind that worked well that you wish other people and other researchers did more of?
I think on the path there, along the sequence of models, both the pre-trained models and the reasoning models, I think one very common theme is bugs. Both just silly bugs in software that can stay in your software for months and invalidate all your experiments in a way that you don't know. Identifying them can be a very meaningful breakthrough for your research program. But also bugs in the sense of having a particular way of thinking about something that is a little bit skewed, which causes you to make the wrong assumptions. Identifying those wrong assumptions and rethinking frames from scratch. I think both for getting the first reasoning models working or getting larger pre-trained models working, we've had multiple issues like that that we've had to work through.
As leaders of the research org, how do you think about what it takes to keep the best talent on your team? And on the flip side, creating a very resilient org that doesn't crumble if a key person leaves?
