There's a lot of excitement leading into GPT-5. I'm wondering what the energy is like internally at OpenAI leading up to such a big launch.
We have a research roadmap. In the last several years, this roadmap hasn't changed much at all.
Interesting. It's able to cut the thinking time that it needs to extract the information by several factors.
You are head of research. So, how do you personally balance that tension between the product side of the organization and the research side?
One thing that we're really excited about is to kind of raise the bar in terms of what we consider acceptable for open source.
Let's talk about some of the lessons that you learned from GPT-4.
And GPT-5 is one of the first models we have that marries both the pre-training paradigm and the reasoning paradigm together. We do believe that this technology is going to raise the quality of life for most people.
Mark, thanks for joining me today. Good to see you again.
So I want to start with a broad question which is there's a lot of excitement leading into GPT-5. I'm wondering what the energy is like internally at OpenAI in the months and then more recently weeks leading up to such a big launch, such an important model. What is that energy like? What is it like inside of the company?
Every single launch, you know, it comes with high emotion, right? And you have that feeling when you're starting out to do a project where people are excited and then you have a period in the middle where there's always this kind of internal uncertainty, right? Is this model going to be good? Is it going to kind of hit the expectations? And then kind of really seeing kind of near the home stretch everything come together. The energy picks back up and I would say right now the feelings are very strong. People are excited to get this model out and yeah, we're excited to show it to the world.
Yeah, I've tested it and it is absolutely incredible. And I'm really just starting to figure out where it really excels, where its personality and the tone and everything are. So, but I visited you guys a few weeks ago and Greg Brockman had mentioned that OpenAI still sees itself as a research lab and you are head of research. So how do you personally balance that kind of tension between the product side of the organization and the research side of the organization?
Yeah. I mean I think it's important to take stock of, you know, we're here to do research and the research really is the product, right? Every time we make a big breakthrough, that is something that will lead to a lot of value and utility for people and of course the product itself enables us to do more research, right? So I think these two things go hand in hand. It's a delicate balance, you really don't have one without the other and that's kind of the view that I hold in my head. We want our research to have contact with the world. We want people to be able to experience kind of all this intelligence that we're building and we're very lucky that it's resulted in a very successful product too.
Let's talk about some of the lessons that you learned from GPT-4 and how you applied it to GPT-5. So from an outsider's perspective, there doesn't seem to be a large amount of new publicly available data that can be applied to a brand new foundation model. Is that a correct assumption? And if so, how did you solve that data scarcity problem?
Right. So I would say it's somewhat accurate but not fully accurate. You can always improve and expand the envelope of the data that you put into the model. So we're always looking for more sources of publicly available data, more sources where we can license data from. But I do think there's a couple axes by which GPT-4 and GPT-5 differ. So when you look at GPT-4, it's really this culmination of scaling the pre-training paradigm. And GPT-5 is one of the first models we have that marries both the pre-training paradigm and the reasoning paradigm together. So it really heavily leans on our O series of models as well. And one of our philosophies here is we want people to get reasoning when they need it and get a very fast response when that's the more appropriate mode to be in. And they don't have to kind of pick, hey, do I need reasoning for this? Do I not need reasoning for this? And back to the question of data. One thing that we've also been exploring in GPT-5 is the use of synthetic data. And this is data that isn't written by humans, but is generated by models. And we've had a very healthy synthetic data program led by one of our researchers, Sebastian Bubeck. And, you know, it's bearing fruits and it helps us improve coverage in areas that we really want to shine.
Yeah. Okay. So, I was going to ask you all about synthetic data. So, let's skip to that. One of my questions was, are you using synthetic data? Obviously, the answer is yes. There are some folks in the industry who say synthetic data and models trained on synthetic data from prior generations of models can only really be marginally better than their predecessor. What do you think about that? What's your take there? So how do you think about the synthetic data in future generations?
Right. I mean, we really do believe in the potential for synthetic data to be, you know, higher quality and to improve the model in meaningful ways beyond just kind of really broadening or kind of deepening surface level knowledge in a particular category. Of course, this is a research program we're still pursuing. I think it still has a lot of room to go. But I think we've seen enough signs of life that we've decided to use some of that to power GPT-5.
And are you able to speak to the mix of synthetic versus human data for GPT-5 and maybe how that compares to the mix for GPT-4?
I do think the exact mix is something that, you know, we want to keep to ourselves but over time it's becoming more and more.
Are there any categories of knowledge where synthetic data really excels? Are there any categories of knowledge like math, science, coding? Those seem like maybe the obvious ones, but you tell me like where does synthetic data really work and maybe where does it fall short, right?
Right. And I think fundamentally I believe in the promise of synthetic data in a very broad-based setting. And it's really kind of up to us to choose which domains that we really want to kind of unleash our set of tools on. You know, we care a lot about code in the GPT-5 release and certainly that's one of the areas that we've emphasized but it's by no means the only way that we view synthetic data.
Okay. And just to continue on that, are there areas that synthetic data have fallen short for you or that you maybe wouldn't apply synthetic data to?
I don't think so. And I mean, there's certainly areas that are more amenable, but I don't think that we, you know, consider the techniques in synthetic data to be not general in a deep sense.
So as you know, let's like rewind months, maybe even years when you start thinking about the architecture for GPT-5. What are some of the early bets that you made that maybe at the time you were a little bit nervous that hadn't been proven yet and ended up really working out with a big model like this?
Right. It's going to be the coming together of a lot of advances in architecture, in optimization, in reasoning and even just kind of in the core infrastructure that we're building. So, you know, we have teams that are exploratory teams in all of these domains, right? Exploratory architecture teams, exploratory optimization teams and in the early phases of a project like this, you know, they're coming up with their scattershot set of ideas and over time we refine those to the bets that are really working. And it's nice to kind of see that kind of winnowing of the ideas and the refinement of some of these ideas and you integrate them all together and it creates this thing that's really kind of a combination of a lot of these innovations on all of these axes.
Were there any early bets that really surprised you though? Like you didn't know it was going to work or you had some doubts and then you just saw incredible results from it?
Yeah. I mean, one thing that I want to stress is kind of the coming together of reasoning and pre-training here, and that might sound pretty obvious on the surface, right? Like, why can't you just kind of get the best of both worlds very easily? There's actually a lot of work done by our post-training team led by Max Schwarzer. And, you know, it took a lot of work to make these reasoning models a lot faster, a lot more robust, a lot more reliable so that we could be able to combine both of these two paradigms that we've been working on somewhat independently into one surface that, you know, people can access the best of both worlds with.
When you're thinking about where to invest your compute in pre-training versus RL, how do you think about that mixture as a decision of really compute investment and really monetary investment because of compute?
Right. So I mean today we invest heavily in both, right? And it's really a function of on the RL side it's a new paradigm, there's a lot of very promising work, a lot of kind of like GPT-2 era style work of just exploring all of the different things that influence RL and it feels very early, right? And so there's a lot of things to explore but at the same time on the pre-training side there's also a lot of energy right now, right? There's the work in synthetic data like what I just talked about and I think still a lot of healthy work in optimization and architecture.
As you're training the model, as you're doing post-training on the model, how do you decide when a snapshot is like, hey, this is the one we're going with? Because obviously you're going to continue to improve it. That's what you guys do. But like how do you decide this is the one we're putting out? It's ready. It's baked. And we'll wait for subsequent iterations for the next versions.
Yeah. Well, I mean, I think it's a little bit of an art, right? And you want to strike this balance of, you know, you want to pursue perfection. You want to pursue something that, you know, it doesn't have kind of any kind of nitpicks or any, you know, you play around with the models a lot. It has to pass the vibe check. And it shouldn't have any kind of small pathologies or, you know, like any behavior that you're worried about. And you have to toe the line of kind of, you know, waiting too long and kind of training what you consider the perfect model on all axes and something that just like feels good and ready to deploy. So I think we've struck a good balance here and the post-training team does a great job of kind of vibe checking all the models.
All right. All right. You mentioned vibe check. What is Mark Chen's vibe check? How do you make sure that the model is good for you, for your personal life, for your work life? What does your vibe test look like?
Yeah, I mean, I check it on a couple different axes. There are a couple of math problems that are my go-to math problems.
Oh, sure. Sure. So there's a fun one that, I think I shared on Twitter a while back. But, it has this very simple description. Basically, you want to create a random number generator, uniform random number generator modulo 42 and you have access to random number generators modulo every single prime less than 42. And this is actually still a relatively unknown problem. A lot of models I think still can't get this consistently right. But you can see a progression of solutions until you get to the most optimal solution here. And you can kind of see improvement in models in terms of how creative they are in being able to generate steps towards the best solution here.
Yeah. Yes. Like I understand those cutting edge math problems, the frontier math problems, the ones that are really hard. Do you have anything that is more like day-to-day tests?
So I think I have two others, you know, one are just kind of generating user interfaces, right? A lot of kind of physical simulations that are very visual, right? You can quickly get a test for like does the physics look right, right? I think you see classic examples of this of, you know, balls bouncing around hexagons, but you can also do kind of like fluid simulations or other kind of user interfaces. And that gives you a sense for, you know, how powerful is the model? How good is the underlying physics? And how good is it at kind of generating robust code and aesthetic code. Another thing that I like to test it with is creative writing. You know, I write a lot day-to-day as well. And yeah, sometimes, probably one of the biggest use cases I use personally in my day-to-day is having it comment on, revise, you know, as a thought partner in documents that I write. And so we really just kind of like does it have an intuitive grasp for what good style is, you know, is it persuasive? Is it compelling? Those are kind of the things that I kind of test for.
Did you see a noticeable improvement from GPT-4 to GPT-5 for creative writing?
Yeah, I do and I think other users will notice that, too.
Do you ever test humor, comedy?
I always find the models struggle with that. So, that's like kind of the golden benchmark for me.
Yeah. I mean, I think there was a time in GPT-2, GPT-3 era where the models weren't generating that realistic human text, and it would be funny in this kind of like, you know, it's just a little bit off. But, I think now that, you know, we're in much more realistic text land. It might take a little bit of time before the models become funny again.
Yeah. A lot of dad jokes I've noticed. Yeah. I think it's getting better. The reasoning models are surprisingly good at humor. I do think that ultimately could be a good test of reasoning, right? Can you kind of like figure out deeply what makes something funny?
One thing that I use models for frequently is getting life advice and I don't know if you do this as well, but I, you know, some complex situation in my life where there's different kind of angles I need to think about it on and it's kind of a difficult decision. It just helps me understand all of those angles. Helps me understand maybe something I hadn't thought of, a new way to approach the problem. Do you ever use it for that?
Yeah. No, I love it as a brainstorming partner. And yeah, it's, you know, when you need advice on a situation, when you have some kind of complex problem where, you know, there's pros and cons of every single kind of decision you could take. Sometimes I use it as kind of something you can confide in, right? And it often times provides some interesting perspectives.
You know, over the last few months, even back to the beginning of the year, China has open sourced a number of incredible models. They've made a lot of innovations on efficiency. Did you take any lessons or techniques from those open source models, open source papers as they were coming out and apply them to GPT-5 or like how has it adjusted your thinking and your approach to your own internal models?
Right. So I want to, one thing that I'm very proud of at OpenAI is that we have a research roadmap. It's something that we put a lot of effort into and it scopes out kind of what path that we want to take towards AGI and it prescribes a couple of things you want to do in the short and medium term but also in the long term. And in the last several years this roadmap hasn't changed much at all and it still hasn't changed. Yeah. Yeah. Even in light of model releases from, you know, from DeepSeek and others. And I think that's really one of the strengths at OpenAI, right? We have conviction in the path that we want to take, we know what kind of ideas it implies and we just pursue those ideas and we're not a very reactionary company in our research roadmap. And at the same time, you know, Chinese labs have been doing really great work too, right? Like I think DeepSeek in particular, they do phenomenal architecture research. They write very, very efficient kernels and I think there's kind of lessons you can learn from that. But by and large the research roadmap hasn't changed and we are still executing on kind of the plan that we had 6 months ago, a year ago.
Yeah, that's actually kind of surprising to hear. So I assume the research roadmap is more broad categories of achievements you're looking to make like a certain amount of efficiency or are they specific techniques that you're looking to test and implement?
