OpenAI’s CPO on how AI changes must-have skills, moats, coding, startup playbooks, more | Kevin Weil

The AI models that you're using today is the worst AI model you will ever use for the rest of your life. And when you actually get that in your head, it's kind of wild. Everywhere I've ever worked before this, you kind of know what technology you're building on. But that's not true at all with AI. Every two months, computers can do something they've never been able to do before, and you need to completely think differently about what you're doing.

Our general mindset is in two months there's going to be a better model and it's going to blow away whatever the current set of limitations are. And we say this to developers too. If you're building and the product that you're building is kind of right on the edge of the capabilities of the models, keep going because you're doing something right. Give it another couple months and the models are going to be great and suddenly the product that you have that just barely worked is really going to sing.

Libra is probably the biggest disappointment of my career. It fundamentally disappoints me that this doesn't exist in the world today because the world would be a better place if we'd been able to ship that product. We tried to launch a new blockchain. It was a basket of currencies originally. It was integration into WhatsApp and Messenger. I would be able to send you 50 cents in WhatsApp for free. It should exist. To be honest, the current administration is super friendly to crypto. Facebook's reputation is in a very different place. Maybe they should go build it now.

Thank you so much for having me. We've been talking about doing this forever and we made it happen.

Yep. Mine, too. My wife Elizabeth sent me one of hers. So, I'm right there with you.

You know, there have been a handful of times in my career when you're working on a project or a product internally and the internal usage just explodes. This was true, by the way, when we were building Stories at Instagram. More than anything else in my career, we could feel it was going to work because we were all using it internally and we'd go away for a weekend, you know, before it launched. We were all using it and we'd come back after a weekend and we would know what was going on and be like, 'Oh, hey, I saw you were at that camping trip, how was that?' You were like, 'Man, this thing really works.' Imagen was definitely one of those. So we've been playing with it for, I don't know, a couple months. And when it first went live internally to the company, there was kind of a little gallery where you could generate your own. You could also see what everyone else was generating, and it was just like non-stop buzz. So, yeah, we had a sense that this was going to be a lot of fun for people to play with.

Yeah, especially social things because you have a very tight network as a company socially. So you know each other and you're experts in your product hopefully and so there's some sense in which if you're doing something social and it's not taking off internally you might question what you're doing.

I think it's just the style people love and the model is really capable at emulating style or understanding what, you know, it's very good at instruction following. That's actually something that I think people are starting to discover with it. But you can do very complex things. You can give it two images. You know, one is your living room and the other is a whole bunch of photos or memorabilia or things you want and you say like tell me how you would arrange these things. Or you can say, 'I'd like you to show me what this will look like if you put this over here and this thing to the right of that and this one to the left of this, but under that one,' and the model actually will understand all of that and do it. It's incredibly powerful. So, I'm just excited about all the different things people are going to figure out.

It's getting there. It's getting there. There's this quote I love, which is AI is whatever hasn't been done yet because once it's been done, when it kind of works, then you call it machine learning. And once it's kind of ubiquitous and it's everywhere, then it's just an algorithm. So I've always loved that we call things AI when they still don't quite work and then you know by the time it's like an AI algorithm that's recommending you follow, you know, oh that's just an algorithm but this new thing like self-driving cars, that's I think to some degree we're always going to be there and the next thing is always going to be AI and the current thing that we use every day and is just a part of our lives, that's an algorithm.

It's, I mean, there's something like that with everything. If I showed you when GPT-3 launched, right? I wasn't at OpenAI then. I was just a user, but it was mind-blowing. And if I gave you GPT-3 now, I just plugged that into ChatGPT for you and you started using it, you'd be like, 'What is this thing?' I like this like mess. Slop, slop.

The craziest thing about what you just shared actually is like I don't know, ChatGPT 3.5 was like a couple years ago and imagine what life will be like in a couple years from now. We're going to get to that where things are going, what you think is going to be the next big leap, but I want to start with the beginning of your journey at OpenAI.

If I'm remembering the timeline right, we communicated at Planet. I was leaving and I was planning to just go take some time, you know, like I wasn't going to stop working, but I was also happy to take the summer. This is like maybe April or something. I was like, 'Cool, I'm going to have the summer with my kids. We're going to, you know, go up to Tahoe or something and I'll actually get to hang out rather than what I usually do going up and down and all that.' And then, you know, Sam and I had known each other lightly for a bunch of years. And he's always involved in so many interesting things, you know, like companies building fusion and all these things. So, he'd always been somebody that I would like call occasionally if I was starting to think about my next thing. Because I like working on big like tech forward sort of, you know, next wave kind of things. And so I called him and I think Vinod also helped put us in touch again and this time it wasn't like oh you should go talk to like these guys working on fusion, he said actually you know we're thinking about something you should come talk to us. I was like, 'Okay, that sounds amazing. Let's do it.' And it goes really fast, really, really fast. Like, I met most of the management team in a brief period of time, a few days, and they were telling me, 'Look, we're going to basically move as fast as we want to move.' And it kind of if you talk to everyone, everyone likes you, we're ready to go. Sam came over for dinner and we had a great evening together just like talking about OpenAI and the future and getting to know each other better. And at the end I was like I was going to go in the next day for like a bigger round of interviews. And Sam was saying, you know, hey, it's going really well. We're really excited. And I said, cool. So how do I think about tomorrow? And he said, oh, you'll be fine. Don't worry about it. And if it goes well, like we're basically there. And so I go in the next day, meet a bunch of people, have a great time, like I really enjoyed everybody I met with. In any interview, you can always second guess yourself, you know, like, oh, I shouldn't have said that thing or that thing I gave a bad answer on. I wish I could redo. But I came away feeling like I think that went pretty well. And I was expecting to hear like that weekend basically because they'd sort of set expectations soon as you know if this goes well, we're ready to go. And I didn't hear anything. And then it was like Monday, Tuesday, Wednesday, I still didn't hear anything. And I reached out to folks on the OpenAI side a couple of times, still nothing. And I was like, 'Oh my god, I screwed it up.' Like, I don't know where I screwed it up, but I totally screwed it up. I can't believe it. And I was going back to Elizabeth, my wife, and being like, 'What did I do? Where do you think I...' you know, getting all crazy about it. And then it's still nothing and finally it was like nine days later they finally got back to me and it turned out you know there was like a bunch of stuff happening internally and this that and the other thing and you know there's just a million things happening and they finally were like oh yeah that went well let's do this and I was like oh okay cool let's do it. But it was like nine days of agony and they were just super busy on some internal stuff and there I was like fretting every single day and re-going over every line of our interview process.

Yeah, totally. They might just be busy. I give them a hard time about it still.

Yeah. Have a little bit of chill.

I think it's probably the pace. Maybe it's two things. One is it's the pace. The second is, you know, everywhere I've ever worked before this, you kind of know what technology you're building on. So, you spend your time thinking about what problems are you solving? Who are you building for? You know, how are you going to make their lives better? How are you going to, is this a big enough problem that you're going to be able to change habits? You know, do people care about this problem being solved? All those like good product things. But the stuff that you're building on is like kind of fixed. You know, you're talking about databases and things, and I bet the database you use this year is probably 5% better than the database you used two years ago, but that's not true at all with AI. It's like every two months computers can do something they've never been able to do before, and you need to completely think differently about what you're doing. There's like something fundamentally interesting about that makes life fun here. There's also something, and you know we'll maybe talk about evals later, but that it also really in this world of everything we're used to with computers is about giving a computer very defined inputs. You know if you look at Instagram for example there are buttons that do specific things and you know what they do and then when you give a computer defined inputs you get very defined outputs. You're confident that if you do the same thing three times, you're going to get the same output three times. LLMs are completely different than that, right? They're good at fuzzy, subtle inputs, all the nuances of human language and communication. They're pretty good at also, they don't really give you the same answer. You probably get spiritually the same answer for the same question, but it's certainly not the same set of words every time. And so, you're much more, it's fuzzier inputs and fuzzier outputs. And when you're building products, it really matters whether you know there's some use case that you're trying to build around. If the model gets it right 60% of the time, you build a very different product than if the model gets it right 95% of the time versus if the model gets it right 99.5% of the time. And so there's also something you have to get really into the weeds on your use case and the evals and things like that in order to understand the right kind of product to build. So that is just fundamentally different. You know if your database works once it works every time and that's not true in this world.

Yeah, sure. I think the easiest way to think about it is almost like a quiz for a model, a test to gauge how well it knows a certain set of subject material or how good it is at responding to a certain set of questions. So in the same way you take a calculus class and then you have calculus tests that see if you've learned what you're supposed to learn, you have evals that test how good is the model at creative writing, how good is the model at graduate level science, how good is the model at competitive coding. And so you have these set of evals that basically perform as benchmarks for how smart or capable the model is.

Yeah, unit tests in general for models. Totally.

Well, it gets back to what I was saying. You need to know whether your model is going to, there are certain things that models will get right 99.95% of the time and you can just be confident. There are things that they're going to be 95% right on and things that are going to be 60% right on. If the model's 60% right on something, you're going to need to build your product totally differently. And by the way, these things aren't static either. So, a big part of eval is if you know you're building for some use case. So, let's take our deep research product, which is one of my favorite things that we've released maybe ever. The idea is with deep research for people who haven't used it, you can give ChatGPT now an arbitrarily complex query like it's not about returning you an answer from a search query which we can also do. It's here's a thing that if you were going to answer it yourself you'd go off and do two hours of reading on the web and then you might need to read some papers and then you would come back and start writing up your thoughts and realize you had some gaps in your thinking. So you go out and do more research. It might take you a week to write some like 20-page answer to this question. You can let ChatGPT just like chug for you for 25-30 minutes. You know, it's not the immediate answers you're used to, but it might go work for 25-30 minutes and do work that would have taken you a week. So, as we were building that product, we were designing evals sort of at the same time as we were thinking about how this product was going to work. And we were trying to go through like hero use cases. You know, here's a question you want to be able to ask. Here's an amazing answer for that question and then turning those into evals and then hill climbing on those evals. So, it's not just that the model is static and we hope it does okay on a certain set of things. You can teach the model. You can make this a continuous learning process. And so as we were fine-tuning our model for deep research to be able to answer these things, we were able to test is it getting better on these evals that we said were important measures of how the product was working. And it's when you start seeing that and you start seeing performance on evals...

These models are intelligences and intelligence is so fundamentally multi-dimensional. So, you can talk about a model being amazing at competitive coding, which may not be the same as that model being great at front-end coding or back-end coding or taking a whole bunch of code that's written in COBOL and turning it into Python, you know, like, and that's just within the software engineering world. And so I think there's a sense in which you can think of these models as incredibly smart, very factually aware intelligences, but still most of the world's data, knowledge, process is not public. It's behind the walls of companies or governments or other things. And same way if you were going to join a company you would spend your first two weeks onboarding. You'd be learning the company specific processes. You get access to company specific data. The models are smart enough you can teach them anything but they need to have the raw data to learn from. And so there's a sense in which yeah, I think the future is really going to be incredibly smart broad base models that are fine-tuned and tailored with company specific or use case specific data so that they perform really well on company specific or use case specific things. And you're going to measure that with custom evals. And so, you know, what I was referring to is just like these models are really smart. You need to still teach them things if the data is not in their training set. And there's a huge amount of use cases that are not going to be in their training set because they're relevant to one industry or one company.

Well, one of my, so this is something Ev Williams used to say back at Twitter that's always stuck with me, which is no matter how big your company gets, no matter how incredible the people are, there are way more smart people outside your walls than there are inside your walls. And it's why we are so focused on building a great API. We have 3 million developers using our API. No matter how ambitious we are, how big we grow. By the way, we don't want to grow super big, there are going to be, there are so many use cases, places in the world where AI can fundamentally make our lives better. We're not going to have the people. We're not going to have the know-how to build most of these things. And I think like I was saying, the data is industry specific, use case specific, you know, behind certain company walls, things like that. And there are immense opportunities in every industry and every vertical in the world to go build AI based products that improve upon the state-of-the-art. And there's just no way we could ever do that ourselves. We don't want to. We couldn't if we did want to. And we're really excited to power that for 3 million plus developers and way more in the future.

Yeah, I mean, we try and have a sense of where we're trying to go, you know, point ourselves in a direction so that we have some rough sense of alignment. Thematically, I don't for a second, and we do quarterly road mapping, you know, we laid out sort of a year-long strategy. I don't for a second believe that what we write down in these documents is what we're going to actually ship, you know, three months from now, let alone six or nine. But that's okay. There's, I think it's like an Eisenhower quote. Plans are useless. Planning is helpful. Which I totally subscribe to, especially in this world. It's really valuable if you think about quarterly road mapping, for example. It's really valuable to have a moment where you stop and go, okay, what did we do? What worked? What went well? What didn't go well? What did we learn? And now what do we think we're going to do next? And by the way, everybody has some dependencies. You need the infrastructure team to do the following things. Partnership with research here. And so you want to have a second to kind of check your dependencies, make sure you're good to go, and then start executing. We try and keep that really lightweight because it's not going to be right. You know, we're going to throw it out halfway because we will have learned new things. So the moment of planning is helpful even if you're only going to, you know, it's only partially right. So that's I think just expecting that you're going to be super agile and that there's no sense writing a three-month road map, let alone a year-long road map because the technology is changing underneath you so quickly. We really do try and go very strongly bottoms up kind of subject to our overall directional alignment. We have great people. We have engineers and PMs and designers and researchers who are passionate about the products they're building and have strong opinions about them and are also the ones building them and so they have a real sense of what the capabilities are too which is super important and so I think you want to be more bottoms up in this way and so we operate that way. We are happy making mistakes, we make mistakes all the time. It's one of the things I really appreciate about Sam, he pushes us really hard to move fast. But he also understands that with moving fast comes we didn't quite get this right or you know we launched this thing it didn't work we'll roll it back. You know look at our naming, our naming is horrible.

It's absolutely atrocious and we know it and we'll get around to fixing it at some point but it's not the most important thing and so we don't spend a lot of time on it but it also shows you how it doesn't matter. Again, ChatGPT, the most popular fastest growing product in history. The models are, it's the number one AI API and model. So clearly it doesn't matter that much. And we name things like o3-mini-high.

So, we do...

Product reviews and things like that, like you would expect. There isn't a ritual because there isn't—we would never want us to be blocked on launching something, you know, waiting for a review with me or Sam. If we can't get there, if I'm traveling or Sam's busy or whatever, that's a bad reason for us not to ship. So obviously for the biggest, most high-priority stuff we have a pretty close beat on it, but we really try not to, frankly. We want to empower teams to move quickly and I think it's more important to ship and iterate. So we have this philosophy that we call iterative deployment, and the idea is like we're all learning about these models together. So there's a real sense in which it's way better to ship something even when you don't know the full set of capabilities and iterate together in public. We kind of co-evolve together with the rest of society as we learn about these things and where they're different and where they're good and bad and weird. I really like that philosophy. There's also a bit of—I think the other thing that ends up being a part of our product philosophy is this sense of model maximalism. The models are not perfect. They're going to make mistakes. You could spend a lot of time building all kinds of different scaffolding around them. And by the way, sometimes we do because sometimes there are things, kinds of errors that you just don't want to make. But we don't spend that much time building scaffolding around the parts that don't match that because our general mindset is in two months there's going to be a better model and it's going to blow away whatever the current set of limitations are. And so if you're building—and we say this to developers too—if you're building and the product that you're building is kind of right on the edge of the capabilities of the models, keep going because you're doing something right because you give it another couple months and the models are going to be great and suddenly the product that you have that just barely worked is really going to sing. And that's kind of how you make sure that you're really pushing the envelope and building new things.

Yeah, absolutely.

Yeah, I mean, kudos to Anthropic. They've built very good coding models. No doubt we think that we can do the same. Maybe by the time this podcast is shipped we'll have more to say, but either way, all credit to them. I think this intelligence is really multi-dimensional. And so I think there's the model providers—it used to be that OpenAI had this massive model lead, you know, 12 months or something ahead of everybody else. That's not true anymore. I like to think we still have a lead. I'd argue that we do, but it's certainly not a massive one. And that means that there are going to be different places where the Google models are really good or where Anthropic's models are really good or where we're really good and our competitors are like, 'Ah, we got to get better at that.' And it actually is easier to get better at a certain thing once someone's proved it possible than it is to forge a path through the jungle and do something brand new. So, I just think, yeah, as an example, it was like nobody could break four minutes in the mile and then finally somebody did and the next year 12 more people did it. I think there's that all over the place and it just means that competition is really intense and consumers are going to win and developers are going to win and businesses are going to win in a big way from that. It's part of why the industry moves so fast. But all respect to the other big model providers, models are getting really good. We're going to move as fast as we can and I think we've got some good stuff coming.

I think being first helps, which is one of the reasons why we're so focused on moving quickly. We like being the first to launch new capabilities, things like deep research. Our models are very—they can do a lot of things, right? So they can take real-time video input, you can have speech to speech, you can do speech to text and text to speech. They can do deep research, they can operate on a canvas, they can write code. And so ChatGPT can kind of be this one-stop shop where all the things that you want to do are possible. And as we go forward and we have more agentic tools like Operator where it's browsing for you and doing things for you on the web, more and more you're going to be able to come to this one place to ChatGPT, give it instructions and have it accomplish real things for you in the world. There's something fundamentally valuable in that and so we think a lot about that. We try to move really fast so that we are always the most useful place for people to come do things.

I don't know, maybe I should have expected this, but one of the things that's been funny for me is the extent to which you can kind of reason when you're trying to figure out how some product should work with AI, you can often—or even why some AI thing happens to be true—you can often reason about it the way you would reason about another human and it kind of works. So, maybe a couple examples. When we were first launching our reasoning model, we were the first to build a model that could reason, that could instead of giving you just a quick, system one answer right away to every question you asked—it was the third emperor of the Holy Roman Empire, like, here's an answer—you could ask it hard questions and it would reason the same way that if I asked you to do a crossword puzzle, you couldn't just snap fill in everything. You would be, 'Well, okay, on this one across I think it could be one of these two but that means there's an A here so that one has to be this—oh wait, you know, backtrack kind of step by step build up from where you are.' Same way you answer any difficult logistical problem, any scientific problem. So this reasoning breakthrough was big but it was also the first time that a model needed to sit and think and that's a weird paradigm for a consumer product. You don't normally have something where you might need to hang out for 25 seconds after you ask a question. And so we were trying to figure out, you know, what's the UI for this because it's also not like with deep research where the model is going to go and think for 25 minutes sometimes. It's actually not that hard because you're not going to sit and watch it for 25 minutes. You're going to go do something else. You're going to go to another tab or go get lunch or whatever. And then you'll come back and it's done. When it's like 20, 25 seconds or 10 seconds, it's a long experience. It's a long time to wait, but it's not long enough to go do something else. And so, you actually need—and you know, so you can think like if you asked me something that I needed to think for 20 seconds to answer, what would I do? I wouldn't just go mute and not say anything and kind of shut down for 20 seconds and then come back. So, we shouldn't do that. We shouldn't just have a slider sitting there. That's annoying. But I also wouldn't just start babbling every single thought that I had. So we probably shouldn't just expose the whole chain of thought as the model's thinking. But you know, I might go like, 'Huh, that's a good question.' I might approach it like that and then think and you're sort of like maybe giving little updates. And that's actually what we ended up shipping. You have similar things where you can find situations where you get better thinking sometimes out of a group of models that all try and attack the same problem and then you have a model that's looking at all their outputs and integrating it and then giving you a single answer at the end. I mean, sounds a little bit like brainstorming, right? Like I certainly have better ideas when I get in a room and brainstorm with other people because they think differently than me. So anyways, there's just all these situations where you can actually kind of reason about it like a group of humans or an individual human and it sort of works which I don't know, maybe I shouldn't have been surprised but I was.

Yeah, we learned from that actually. Because when we first launched it, we kind of gave you the subheadings of what the model was thinking about but not much more. And then DeepSeek launched and it was a lot and we kind of went, you know, I don't know if everyone wants that. There's some novelty effect to seeing what the model's really thinking about. We felt that too when we were looking at it internally. It's interesting to see the model's chain of thought, but it's not—I think at the scale of like 400 million people, you don't want to see the model kind of babble a bunch of things. And so what we ended up doing was summarizing it in interesting ways. So instead of just getting the subheadings, you're kind of getting like one or two sentences about how it's thinking about it. And you can learn from that. So, we kind of tried to find a middle ground that we thought was an experience that would be meaningful for most people, but showing everybody like three paragraphs is probably not the right answer.

Yeah, I don't know if maybe I'm—maybe this is one of those things I believe that most people don't believe, but I actually think chat is an amazing interface because it's so versatile. People tend to go, 'Oh, chat. Yeah, well, that's just like, you know, we'll figure out something better.' And I kind of think this is—it's incredibly universal because it is the way we talk. Like I can talk to you verbally like we're talking now. We can see each other and interact. We can talk on WhatsApp and be texting each other but all of these things is this sort of unstructured method of communication and that's how we operate. If I had some more rigid interface that I was allowed to use when we spoke I would be able to speak to you about far fewer things and it would actually get in the way of us having maximum communication bandwidth. So there's something magical and by the way in the past it never worked because there wasn't a model that was good at understanding all of the complexity and nuances of human speech and that's the magic of LLMs. So to me it's like an interface that's exactly fit to the power of these things. And that doesn't mean that it always has to be just like—I don't necessarily always want to type but you do want that very open-ended flexible communication medium. It may be that we're speaking and the model's speaking back to me, but you still want that very sort of lowest common denominator, no restrictions way of interacting.

By the way, I think there are—it's not that it's only chat either. Like there are if you have high volume use cases where they're more prescribed and you don't actually need the full generality. There are many use cases where it's better to have something that's less flexible, more prescribed, faster at a specific task and those are great too and you can build all sorts of those. But you still want chat as this baseline for anything that falls out of whatever vertical you happen to be building for. It's like a catch-all for every possible thing you'd ever want to express to a model.

It's an area where we're evolving a lot. I'm really excited about it. Frankly, I think if you go back a couple years when ChatGPT was just getting started—obviously I wasn't at OpenAI—but we were more of a pure research company at the time. ChatGPT, if you remember, was a low-key research preview for many years. It wasn't a thing that the team launched thinking it was going to be this massive product. It was just a way that we were going to let people play with and iterate on the models. And so we were primarily a research company, a world-class research company. And as ChatGPT has grown and as we've built our B2B products and our APIs and other things, now we're more of a product company than we were. I still think OpenAI should never be a pure product company. We need to be both a world-class research company and a world-class product company. And the two need to really work together. And that's the thing that I think we've been getting much better at over the last like six months. If you treat those things separately and the researchers go do amazing things and build models and then they get to some state and then the product and engineering teams go take them and do something with them, we're effectively just an API consumer of our own models. The best products though are going to be—as I was talking about with deep research—it's a lot of iterative feedback. It's understanding the problems you're trying to solve, building evals for them, using those evals to go gather data and fine-tune models to get them to be better at these use cases that you're looking to solve. It's a huge amount of back and forth to do it well. And I think the best products are going to be product design and research working together as a single team to build novel things. So that's actually how we're trying to operate with basically anything that we build. It's a new muscle for us because we're kind of new as a product company, but it's one that people are really excited about because we've seen every time we do it, we build something awesome and so now every product starts like that.