Hello, I'm Andrew Mayne, and this is the OpenAI Podcast. In this episode, we're going to speak with OpenAI co-founder and president, Greg Brockman, and Codex engineering lead, Thibault Sottiaux. And we're going to talk about Agenda coding, GPT-5 codex, and where things might be heading in 2030.
Just bet that the greater intelligence will pan out in the long run.
It's just really optimized for what people are using GPT-5 within codex for.
How do you make sure that AI is producing things that are actually correct?
We're here to talk about codex, which first, I've been using it since actually since I worked here with the first version of this. And then now you guys have the new version of this. I was playing with it all weekend long. And I've been very, very impressed by this. And it's amazing how far this technology has come in a few years. I would love to find out the early story. Like, where did the idea of even using a language model to code come from?
Well, I mean, I remember back in the GPT-3 days seeing the very first signs of life of take a doc string, a Python definition of a function name, and then watching the model complete the code. And as soon as you saw that, you knew this is going to work. This is going to be big. And I remember at some point we were talking about these aspirational goals of imagine if you could have a language model that would write a thousand lines of coherent code, right? That was like a big goal for us. And the thing that's kind of wild is that that goal has come and passed. And I think that we don't think twice about it, right? I think that while you're developing this technology, you really just see the holes, the flaws, the things that don't work. But every so often, it's good to like step back and realize that like actually things have just like come so far.
It's incredible how used we get to things improving all the time and how to just become like a daily driver. You just use it every day and then you reflect back to like a month ago. This wasn't even possible. And this just continues to happen. I think that's quite fascinating, like how quickly humans adapt to new things.
Now, one of the struggles that we've always had is the question of whether to go deep in a domain. Right. Because we're really here for the G, right, for AGI, general intelligence. And so to first order, our instinct is just push on making all the capabilities better at once. Coding has always been the exception to that, right? We really have a very different program that we use to focus on coding data, on code metrics, on trying to really understand how do our models perform on code. And, you know, we've started to do that in other domains, too. But for programming and coding, that that's been like a very exceptional focus for us. And, you know, for GPT-4, we really produced a single model that was just a leap on all fronts. But we actually had trained, you know, the codex model. And I remember doing like a Python sort of focused model. Like we were really, really trying to push the level of coding capability back in 2021 or so. And I remember when we did the Codex demo, that was maybe the first demonstration of what we'd call Vibe coding today. I remember building this interface and having this realization that for just standard language model stuff, the interface, the harness is so simple. You're just completing a thing and maybe there's a follow-up turn or something like that. But that's it. for coding that you actually, this text comes to life, right? You need to execute it, that it needs to be hooked up to tools, all these things. And so you realize that the harness is almost like equally part of how you make this model usable as the intelligence. And so that is something that I think we kind of knew from that moment. And it's been interesting to see as we got to more capable models this year and really started to focus on not just making the raw capability, like how do you win at programming competitions, but how do you make it useful, right? Training in a diversity of environments, really connecting to how people are going to use it, and then really building the harness, which is something that Thibaut and his team have really pushed hard.
Could you unpack a harness, what that means in simple terms?
Yes, it's quite simple. You have the model, and the model is just capable of input-output. And what we call the harness is how do we integrate that with the rest of the infrastructure so that the model can actually act on its environment? So it's the set of tools, it's the way that it's looping. So the agent loop, as we refer to it as the agent loop. And it's in essence, it's fairly simple. But when you start to integrate these pieces together and really train it end to end, you start to see like pretty magical behavior and an ability of the model to really act and create things on your behalf and be a true collaborator. So think about it a little bit as, you know, the harness being your body and the model being your brain.
Okay. It is, It's interesting to see how far it came, like GPT-3 days where you literally had to write commented code and say, this function does this with its Python, put your hashtag in front of that, whatever. And it's just interesting to see how the models have now become just naturally, intuitively good at coding. And you mentioned that trying to determine between a general purpose model or saying how important code is. Was it just outside demand, people telling me one of these models is better at code? Or was this coming internally because you guys wanted to use this more?
Both, yeah absolutely both. And I remember, you know, in I think 2022 is when we worked with GitHub to produce GitHub Copilot. And the thing that was very interesting there was that for the first time you really felt what is it like to have an AI in the middle of your coding workflow and how can it accelerate you. And I remember that there were a lot of questions around the exact right interface. Do you want ghost text so it just does a completion? Do you want a little drop down with a bunch of different possibilities? But one thing that was very clear was latency was a product feature. And the constraint for something like an autocomplete is that 1500 milliseconds, right? That's like the time that you have to produce a completion. Anything is slower than that. It could be incredibly brilliant. No one wants to sit around waiting for it. And so the mandate that we had, the clear signal we had from users and from the product managers and all the people thinking about the product side of it is get the smartest model you can subject to the latency constraint. And then you have something like GPT-4, which much, much smarter, but it's not going to hit your latency budget. What do you do? Is it a useless model? Like, absolutely not. The thing you have to do is you change the harness, you change the interface. And I think that that's like a really important theme is you need to kind of co-evolve the interfaces and the way that you use the model around its affordances. And so super fast, smart models is going to be great, but the incredibly smart, but slower models, it's also worth it. And I think that we've always had a thesis that the, you know, that the returns on that intelligence are worth it. And it's never obvious in the moment because you're just like, well, it's just going to be too slow. Why would anyone want to use it? But I think that our approach has very much been to say just that the greater intelligence will pan out in the long run.
It was hard for me to wrap my head around where that was all headed back when working on the GitHub copilot, because at that point, were used to, like you said, the completion, ask to do a thing, it completes a thing. And I think I didn't really understand how much more value you would get out of building a harness, adding all these capabilities there. And it just seemed like all you just need is the model. But now you realize the tooling, everything else matters can make such a big difference. And you brought up the idea of modalities. And now we have CLI, Codex CLI. So I can go in the command line, I can do this. There's a plugin for VS Code, so I can go use this there. And then also I can deploy stuff to the web and do that. And I don't think I fully kind of comprehend the value of that. And so like, how is this something you're using? How are you kind of deploying these things yourself? Like, where are you finding the most, you know, utility out of it?
I think just to go back a little bit, like the first signs that we saw is like, we had a lot of developers at the company outside of the company, like our users use ChatGPT to help them debug like very complex problems. And one thing that we clearly saw is like, people are trying to get more and more context into ChatGPT. And you're trying to get bits of your code and stack traces and things. And then you paste that and you present that to a very smart model to get some help. And interactions were starting to get more and more complex up to some point where we realized like, hey, maybe instead of the user driving this thing, maybe let the model actually drive the interaction and find its own context and then find its way and be able to debug this hard problem by itself so that you can just sit back and watch the model do the work. So it's like sort of like reversing that interaction that led to this, I think, thinking a lot more about the harness and giving the model the ability to act.
And we iterated on form factors. I mean, I remember at the beginning of the year, we had a couple of different approaches. We had sort of the async agentic harness, but we also had the local experience and a couple of different implementations of it.
We actually started to play a little bit with this idea of running it in the terminal. And then we felt that was not AGI-pilled enough. We needed the ability to run this at scale and remotely and just close the laptop and have the agent just continue to do its work. And then you can maybe follow it on your phone and interact with it there. That seemed very cool. So we pushed on that. But we actually had a prototype of it fully working in a terminal. And people were using that productively at OpenAI. We decided to not launch this as a product. It didn't feel like polished enough. It was called 10X because we felt like it was giving us this 10X productivity boost. But then we decided to just experiment with different form factors and really go all in with the async form factor initially. And now we've kind of gone back a little bit of that and re-evolved and said, hey, actually, this agent, we can bring it back to your terminal. We can bring it in your IDE. But the thing that we're really trying to get right is like this entity, this collaborator that's working with you and then bringing that to you in the tools that you're already using as a developer.
And there are other shots on goal as well. So we had a version where there was a remote daemon that would connect to a local agent. And so you kind of could get both at once. And I think that part of the evolution has been that there's almost this matrix of different ways you could try to deploy a tool. There's this async. It has its own computer off in the cloud. There's the local that it's running synchronously there. You can blend between these. There's been a question for us of how much do we focus on trying to build something that is externalizable, that is useful in the diversity of environments that people have out there versus really focus on our own environment and try to make it so that things work really well for our internal engineers. And one of the challenges has been we want to kind of do all of this. We ultimately want tools that are useful to everyone. But if you can't even make it useful for yourself, how are you going to make it extremely useful for everyone else? And so part of the challenge for us has been really figuring out where do we focus and how do we achieve the sort of biggest bang for the buck in terms of our engineering efforts? And, you know, for me, one of the things that's been an overarching focus has been we know that coding and building very capable agents is one of the most important things that we can do this year. At the beginning of the year, we set a company goal of an agentic software engineer by the end of the year. And figuring out exactly what that means and how to substantiate that and how to bring together all the opportunity and all the kind of compute that we have to bear on this problem. Like that has been a great undertaking for many, many people at OpenAI.
So you mentioned that you had the tool 10X and that was an internal tool. And that seemed to be something at some point you said, oh, this is really useful to other people. It's got to be hard to decide when to do that and when not to. and how much to sort of prioritize that. We've seen cloud code has become extremely powerful, which I imagine is probably a similar story with something that was used internally and then became something deployed. When you start to think about next steps of, where do you decide to take it next? Where do you decide to put the emphasis? You mentioned before, I can now run things in the cloud, run these web, do these kind of agentic-like tasks where I walk away. And my problem is just, it's such a new modality. It's really, really hard for me to think about. But sometimes these things have to sit around for a while and people sort of discover them independently. And have you found that internally that somebody says, oh, now I get it?
I'd say absolutely. Right. And I think that, you know, my perspective is that we kind of know the shape of the future, right, of the long term. It is very clear that you're going to want an AI that has its own computer that is able to run, you know, delegate to a fleet of agents and be able to solve multiple tasks in parallel. You should wake up in the morning, you're sipping your coffee, you know, answering questions for your agent, like providing some review, being like, oh, no, this wasn't quite what I meant. This workflow clearly needs to happen. But the models aren't quite smart enough for this to be the way that you interact with them. And so having an agent that is really there in your terminal, in your editor to help you with the way that you do your work that looks very similar to the way you would have done it a year ago. That's also the present. And so I think that the way that we've seen it is almost we're blurring together. Here's what the future looks like, but we also can't abandon the present and thinking about how do you bring AI into code review and how do you make it so that it appears proactively and does work for you that's useful. And then you have a whole new challenge as well of if you have a lot more PRs, like how do you actually sort through those to the ones that are the ones you actually want to merge? And so I think we've kind of seen all of this opportunity space and we've seen people start to change how they develop within OpenAI, how they even structure their code bases.
Yeah, I think there are two things to that effect that really combine and mean, you know, this is where we're at today is one infrastructure is hard. And we would love for, you know, all of everyone's code and like tasks and packages to be like perfectly containerizable. And so we can run them at scale. That's not the case. Like people have very thorough and complex setups that probably only runs on their laptop. And we want to be able to leverage that and meet, you know, people where they are so that, you know, they don't have to configure things specifically for Codex. That gives you this very easy entry point into experiencing, you know, what a very powerful coding agent can do for you. And it's also at the same time, lets us experiment with, you know, what the right interface is. Six months ago, we weren't playing with these kinds of tools. And this is all very new and evolving fast. And we have to continue to iterate here and innovate on like what the right interface and what the right way to collaborate with these agents are. And we don't feel like we have really nailed that yet. that's going to continue to evolve, but bringing it to like a zero setup, extremely easy to use out of the box, you know, allows a lot more people to benefit from it and like play with it. And for us to get the feedback so that we can continue to innovate, that's very important.
I remember at the beginning of the year talking to one of our engineers, who I think is really fantastic. And he was saying that ChatGPT, we had this integration where it could automatically see the context in this terminal. And he's like, it's transformative because he doesn't have to like copy paste errors. He just like can instantly be like, hey, like, you know, what's what's the bug? And it would just tell him and it was great. Right. And you realize that it was an integration that we built that was so transformative. It wasn't about a smarter model. And I think that one thing that's very easy to get confused by is to really focus on only one of these dimensions and be like, which one matters? Because the answer is they kind of both matter. And the way I've always thought about this, I remember when we were originally releasing the API back in 2020, is there's two dimensions to what makes an AI desirable. There's intelligence, which you can think of as one axis. And then there's convenience, which you can think of as latency. You could think of as cost. You could think of as the integrations available to it. And there's some acceptance region, right? Where it's like, if the model's incredibly smart, but it takes you like a month to run it or something, like you still might, right? If what's going to output is such a valuable piece of code or cure for a certain disease or something like that, okay, fine, like it's worthwhile. If the model is incredibly not that intelligent, not that capable, then all you want to do is autocomplete. So it has to be incredibly convenient, zero cognitive tax for you to think about what it's suggesting, that kind of thing. And where we are is, of course, somewhere on the spectrum now. We now have smarter models that are reasonably less convenient than autocomplete, but still more convenient than you have to sit around and wait for a month for the answer to appear. And so I think that a lot of our challenge is figuring out when do you invest in pulling that convenience to the left? When do you invest in pushing the intelligence up? And it's a massive design space. It's what makes it fun.
Yeah. I don't know if you remember, but I made an app that was featured on the launch back in 2020, AI channels.
And that was, yeah, the challenge was, GPT-3 app was very capable, but I had to write these like 600 word prompts to get it to do stuff. And because it's six cents per thousand tokens in the latency, I'm like, I don't think this is the world for this right now. Yes. And then GPT-3.5 and GPT-4, then all of a sudden you see all that capabilities. And it was hard for me to say why, but then you see that all of a sudden the things that come together. And you mentioned, you know, the idea of just having, you know, the model be able to see the context inside of the, you know, where you're working. And I remember when I was copy pasting using ChatGPT into my workspace and it reminded me going into a grocery store and refusing to get a cart and just carrying everything to the checkout. I'm like, this is terribly inefficient. Once you put things on wheels, it works really well. And I think we're seeing all kinds of those unlocks now. Now the problem I deal with is when I sit down to work on something is, do I go into CLI? Do I go use the VS Code plugin? Do I go into Cursor? Do I use some other tool? And how do you guys figure this out?
Right now, we're still at the experimentation phase where we're trying different ways for you to interact with the agent and bring it where you're already productive. So, for example, Codex is now in GitHub. You can mention Codex and it will do work for you. If you do add Codex, fix this bug or move the tests over here, it will go and run off and do it with its own little laptop on our data centers. And you don't have to think about it. but if you're working with files in a folder, you know, then you have that decision that are you going to do it in your IDE? Are you going to do it in terminal? What we're seeing is users are developing, like power users are developing very complex workflows with the terminal more. And then when you're actually working on a file or a project, you prefer to do it in the IDE. It's a bit more of a polished interface. You can undo things. You can see the edits. You know, it's not like just scrolling by you. And then the terminal is just an amazing, also vibe coding tool where, you know, if you don't really care that much about the code that's being produced, you know, you can just generate a little app. It's much more about that interaction. It elevates the interaction more instead of focusing on the code. So it's more focused on the outcome. And it's just sort of like depends on what you want to do, but it's still very much an experimentation phase right now. And we're trying different things out. And, you know, it's going to continue like that, I think.
Yeah, I really agree with that. And I also think that a lot of our direction will be more integration across these things. I guess people are capable of using multiple tools, right? You already have your terminal, your browser, your GitHub web interface, your repo on your local machine. Each of these is something people have kind of learned when it's appropriate to reach for what tool. And I think that because we're in this experimentation phase, that these things can feel very disparate and very different. And like, you have to kind of learn a new set of skills and the affordances of the relevant tool. And I think that a lot of as we're iterating what's on us is to really think about how do these fit together. And so you can start to see it right with the codex IDE extension being able to run remote codex tasks. And I think that ultimately our vision is that there should be a AI that has access to its own computer, its own clusters, but is also able to look over your shoulder. Right. They can also come and help you locally. And these shouldn't be distinct things.
Right. And it's like this one coding entity that is there to help you and collaborate with you. Like when I collaborate with Greg, you know, I don't complain that sometimes you're on Slack. Sometimes I talk to you in person.
Sometimes we interact like through a GitHub review. Like this seems like very natural when you interact with other humans and collaborators. And this is also where, you know, how we're thinking about Codex as an agentic like entity that is really meant to just supercharge you when you're trying to achieve things.
So let's talk about some of the ways of using it, like agents.md. Do you want to explain that?
Yeah, agents.md is a set of instructions that you can give to Codex that lives alongside your code. So that Codex has a little bit more context about how to best navigate the code and accomplish the tasks. There are two main things that are useful to put in agents.md that we find is like helping with it's like a compression thing where it is a little bit more efficient for the agent to just read codex.md. instead of like exploring the entire code base. And then preferences that are actually not clear in the code base itself, where you would be like, you know, actually tests should be over here or, you know, I like things to be done in this particular fashion. And those two things, you know, preferences and then explaining to the agent how to navigate the code base effectively are very useful things to have in agents.mg. Yep.
And I think that there's something deeply fundamental here of how do you communicate to an agent that has no context what you want, what your preferences are, and to try to save it a little bit of the kind of spin up a human would require, right? We do this for humans, right? We write readme.mds. And this is just a convention for a name of a file that an agent should go look at. But there's also something that's a little point in time, right? That the agents right now don't have great memory, right? It's like if you're running your agent for the 10th time, Has it really benefited from the nine times that it went and solved a hard problem for you? And so I think that we have real research to do to think about how do you have memory? How do you have an agent that really just goes and explores your code base and really deeply understands it and then is able to leverage that knowledge? And so this is one of the examples, and there are many, where we see great fruit on the horizon for further research progress.
It's a very competitive landscape now. There was a point where, you know, OpenAI kind of came out of nowhere for a lot of people. And all of a sudden there was GPT-3, then there was GPT-4. And then I think Anthropics building great models. And Gemini, you know, from Google has gotten really good. How do you guys see the landscape? How do you see your placement there?
I mean, I think that there's a lot of progress to be had. I focus a little less on the competition and a little more on the potential, right? Because we started OpenAI 2015 thinking that AGI is going to be possible, maybe sooner than people think. And we just want to be a positive force in how it plays out, right? And really thinking about what does that mean, trying to connect that to practical execution has been a lot of the task. And so as we started to figure out how to build capable models that are actually useful, right, that can actually help people, actually bringing that to people is this really critical thing. And you can look at choices that we've made along the way, for example, releasing ChatGPT and making ChatGPT free tier available widely. That's something that we do because of our mission, because we really think about we want AI to be available and accessible and benefit everyone. And so in my view, the most important thing is to continue on that exponential progress and really think about how to bring it to people in a positive and useful way. So I really look at where we're at right now is that these models, like there's the GPT-4 class of pre-trained models. There's reinforcement learning on top of it to make it just much more reliable and smart. It's like you think about if you've just sort of read the internet, you've just observed a bunch of sort of human thought. And you're trying to write some code for the first time, you're probably going to have a bad time of it. But if you've had the ability to actually try to solve some hard code problems, you have a Python interpreter, you have access to the kinds of tools that humans do, then you're going to be able to become much more robust and refined. So we now have these pieces working together in concert, but we got to keep pushing them to the next level. It's very clear that things like being able to refactor massive code bases, like no one's cracked that just yet. There's no fundamental reason we can't. And I think the moment you get that, I think refactoring code is one of the killer use cases for enterprise, right? It's, you know, if you could bring down the cost of code migrations by, you know, 2x, I think you'll end up with 10x more of them happening. Think about the number of systems that are stuck in COBOL. And there's no COBOL programmers being trained, right? It's just like it's strictly like, you know, building liability for the world to have this dependency. Like the only way through is by building systems that can actually tackle that. So I just think it's a massive open space. The exponential continues and we need to stay on that.
My favorite thing today that happened was there was a tweet from OpenAI, which was showing people how to use the CLI to switch from the completions API to the responses API.
That's a great use. I expect to see more of that. You know, where you have special instructions given to Codex in order to go do like refactorings reliably. And then you just set it off and it does it for you. That's like a wonderful thing. migrations are some of the worst things. Nobody wants to do migrations. Nobody wants to change from one library to the other and then make sure that everything still works. If we can automate most of that, that's going to be a very beautiful contribution.
I think there's a lot of other ground as well. I think that security patching is a good example of something that I think will become very important soon, and that's something we're being very thoughtful about. I think that being able to actually have AIs that produce new tools, right? You think about how important the Unix set of standard tools has been and AIs, they're actually able to build their own tools that are useful for you, are useful for themselves. You can actually build up a ladder of complexity there or utility there to be able to just like continue to improve this flywheel of efficiency. AIs that are actually really not just writing code, but able to execute their own, be able to administer services or be able to do SRE work and things like that. I think all of that is on the horizon. It's like starting to happen, but it's not really happening yet in the way that we would like to see.
One big one that we cracked internally at OpenAI and then we decided to release it as a code review, where we started to notice that the big bottleneck for us was with increased amounts of code needing to be reviewed is like the amount of review simply that people had to do on the teams. And so we decided to really focus on a very high signal codex mode where it's able to review a PR and really think deeply about the contract and the intention that you were meaning to implement and then look at the code and validate whether that intention is matched and found in that code. And it's able to go layers deep, look at all the dependencies, think about the contract and really raise things that some of our best employees, some of our best reviewers wouldn't have been able to find unless they were spending hours really deeply thinking about that PR. And we released this internally first at OpenAI. It was quite successful and people were upset actually when it broke because they felt like they were losing that safety net. And it accelerated teams and including the Codex team tremendously. The night before we released the IDE extension, one of the top engineers on my team was like cranking out 25 PRs. And we were finding quite a few bugs automatically. Codex was finding quite a few bugs. And, you know, we were able to put out an IDE extension that was almost bug free the next day. So the velocity there is incredible.
And it's very interesting that for the code review tool in particular, people were very nervous about having this enabled because I think our previous experience with every auto code review experiment that we've tried is that it's just noise. Right. You just get an email from some bot and you're like, oh, another one of those things. You ignore it. And I think we've had kind of the opposite finding from where we are now. And it really shows you when the capability is below threshold, it just feels like this thing is like totally net negative. I don't want to hear about it. I don't want to see it. Once you kind of crack above some threshold of utility, suddenly people want it, right, and get very upset if it gets taken away. And I think also our observation is if something kind of works in AI right now, one year from now, it'll be incredibly reliable, incredibly mission critical. And I think that that's where we're going with code review.
Part of the interesting things there with CodeReview as well is like bringing humans along and really have this be a collaborator, including and review. And one thing we taught a lot about is like, how can we raise those findings so that you are actually excited to read this finding and you might even learn something, including when it's wrong. Like, you know, you can actually understand its reasoning. Most of the time, like actually more than 90% of the time it's right. And you often learn something as the person who authored the code or someone who is helping review the code.
Yeah, just circling back to what we were saying earlier about the rate of progress and sometimes stepping back and thinking about how things were earlier. I remember for GPT-3 and for GPT-4 really focusing on the doubling down problem. Do you remember if the AI would say something wrong and you'd point out the mistake?
Oh, it would argue with you.
Oh, yeah. It would try to convince you that it was right. We're so far past that being the core problem. I'm sure it happens in some obscure edge cases, just like it does for humans. But it's really amazing to see that we're at a level where even when it's not quite zeroed in on the right thing, it's highlighting stuff that matters. It has pretty reasonable thoughts. And I always walk away from these code reviews thinking like, huh, OK, yeah, that's a good point. I should be thinking about that.
We're now just getting to launch a GPT-5. And as the recording of this podcast, we now have GPT-5 Codex.
