We're in this transition from ChatGPT being a thing that answers questions to a product that actually goes and does tasks for you in the real world. You should be in control of any actions that it takes. The best way to do that is to kind of co-evolve together. Sam has sort of talked a little bit about generational differences. So what is it like if you're in your early 20s, you're using ChatGPT? If you're younger, it's just a core part of the way you operate your life. There's sort of an always-on nature. You realize that you have this super assistant in your pocket that can not just answer any question, but it can teach you anything that you want to learn. How far out are model capabilities being delivered? Are there things that are being worked on today that might not make their way into models until mid-2026? It's unpredictable. Sometimes things take longer than expected. Other times you see these capabilities that you didn't expect at all that are kind of emergent and all of a sudden something just works. It's a completely different way of building products. Computers can like do things that they couldn't do two months ago and we're constantly in that state.
Today I'm so excited to welcome Kevin Weil. He is the Chief Product Officer of OpenAI, coming off the back of a storied career in companies like Twitter and Facebook. So much to talk about. So first of all Kevin, thank you for making the time. You've been busy today already. It's morning in San Francisco and you've already launched some things.
Yes, we have. And thank you so much for having me. It's my first live Substack, so I'm excited.
Well, hopefully many more to come. You are really keeping us on your toes. And so, just an hour before we went live, there was another spate of product launches from OpenAI. What do they tell us about your vision for ChatGPT and where the product is going?
The biggest thing that we launched this morning—we launched like six different things this morning—but I think the most important one kind of for the long-term future of AI is we launched a series of connectors that can connect to your either personal data or if you're an enterprise, your enterprise data. So this is connectors into Google Docs, to Gmail and Calendar, to SharePoint, to OneDrive, Dropbox, Box, Linear, all of these different tools that you use every day to get things done. With the rise of our reasoning models, connecting them into the services and the data that you use helps the models be way more useful. So it's not just that you can now ask, you know, at work for example, if you connect it into your Google Docs or SharePoint, you connect it into the docs that you use every day, suddenly ChatGPT can get all of this context on the enterprise and on the state of projects and on the latest about any particular thing that's going on. You wouldn't ever have an employee at your company that you didn't give access to docs. It's like where the conversation and the strategy and other things are happening. So now you have ChatGPT that has the ability to do it. And today they're read-only. So they're able to access the information but not create on the other end. But you can imagine going forward, another big part of this is that ChatGPT should be able to take actions, should be able to help you write a document or create a presentation or write tasks to your task management system, and ultimately combine all of these together to begin really working like an employee. Right? So that's the hint, right, that direction of moving ChatGPT from something we interact with with a challenge and response into something that is more evolved and feels like it's really doing work for us as an employee.
But we're going to spend some time digging into that. It's such a big vision. Let me ask about you. You're the Chief Product Officer of what I think is the most important company in the world today. So, how does that feel like to you day-to-day and week-to-week?
I mean, it's a privilege. It's the most exciting place I've ever worked. And I've been very fortunate in my career to work at a bunch of awesome places with really great co-workers. But I think, you know, the opportunity in front of us, the rate at which AI is changing all of our lives means we have an ability to make a really big impact and we take that super seriously. So, I get to work with amazing colleagues. I get to kind of, you know, have a front row seat to the way that our models are evolving and the way this is all impacting our lives and hopefully we get to build some products that make a difference in your life and in the lives of everybody listening here.
Well, nearly a billion people have made at least voted with their mouse finger to use ChatGPT just since November 2022. Technology always changes us. The television did. We got TV dinners and we got water cooler discussions. The internet obviously changed us in different ways. Cars as well. We got big box retail. We got the suburbs. We got the picket fences. Desperate Housewives. How might AI products reshape our everyday life?
Well, I think one of the other interesting things you get when you have these waves of technology is you start by doing the thing that you were doing with the previous wave but sort of with the new medium. You know, the first TV advertisements were people standing on a stage reading their radio advertisements and then people slowly figure out that you can actually, you know, do what we have as commercials today which are much more interactive and sort of dynamic. And so, you know, we're still probably in that mode of people when they look at the impact that AI can have in their lives or in their work, they're kind of like, okay, I have these processes. How do I, you know, sprinkle AI on top of this to make it better, faster, etc. And like that's fine. That's all good. As in previous technology transitions, the power comes from completely reimagining the work that you're doing from first principles using the new technology. You know, so mobile wasn't just about, you know, a computer in your pocket. It was, you know, you have access to GPS and you have these totally new ways to interact with technology. I think over the next, you know, year, we're all going to be in the process of reinventing the way that we do things with AI. And the fun part is the technology is moving so fast. It's not—I mean faster than any technology I've ever worked with before, I've ever seen in my career. So even as we're reinventing, the technology is gaining new capabilities and it's just an exciting time to be alive.
I mean it is, that rate of change is quite something. I did a—I was looking for something today which I know would not have been possible four months ago. So, I was using o3 to help me find a little portable mobile 5G router for when I travel and I just need to get good quality signal. And of course, o3 went into all the technical specifications of the radio frequency chipset and said, well, you know, this earlier router has got a slightly better Qualcomm chipset they no longer use and you'll get one extra signal bar in rural environments in the US but not in Europe. And I'm sitting there going, this is kind of insane. But I'm really curious about those types of behaviors. What are the surprising behaviors that you've seen emerge around ChatGPT from your users? Sam has sort of talked a little bit about generational differences, but give us a flavor of things that you've learned about behavior that are just really hard for product teams to research or figure out a priority.
Well, I do think—I mean Sam has talked about the way that, like I said, the people today, I think a lot of people are sort of sprinkling AI onto their existing workflows and it's young people that are, you know, this is native to them in a way that it's not native to, you know, those of us who grew up without AI. And I mean my kids just—they're like, of course you can talk to a super powerful AI that, you know, can customize itself and answer any question you have. And kids that are graduating college these days as engineers are like, they don't know any other way to write code than using, you know, AI editors like Cursor and Windsurf and it's just completely natural to them and so it gives them superpowers in a way. And it's one of the reasons that we look a lot at how our youngest users or young users, you know, late teens, college users, things like that are using the product because it teaches us a bunch.
Can you characterize those differences though? So, what is it like if you're in your early 20s, you're using ChatGPT? How does it feel different to somebody who might be in their mid-30s or early 40s?
There's sort of an always-on nature. You just—you realize that you have this super assistant in your pocket that can not just answer any question, but it can teach you anything that you want to learn. And so when you go about life that way, the rest of us are like trying to remember, you know, trying to think about the processes that we go through and how we can reimagine them. If you're younger, you might not have had those processes. And so you built them from the ground up with AI. And so it's just a core part of the way you operate your life. And in some ways, they're ahead and the rest of us are catching up.
A lot of people are scared of this technology. A lot of people are, you know, nervous. I've just come back from Brussels and spoken to people in a different sector and you do get that sense of fear. What could change in the product to address that?
One thing that I think is really important is people ask, you know, what should I do about AI? How should I think about it? And my answer is always just use it. And you know, of course, I think everybody should try ChatGPT, but whether it's us or any of the others, just start using it because it's the number one way to realize it's not this like super scary thing that you read about. It just helps you get more done and you're like, 'Oh, this is great. I now have this incredible new thing that can be part of my life and help me get things done and help me automate a bunch of the boring work that I have.' So, the number one thing is just start using it. Also given the rate that the technology is improving, if you don't start using it now, it's going to be even harder to sort of catch on to it. And if you believe that AI is going to be a big part of our lives, that's not a train you want to miss. So, you know, that's the number one thing. Just start using it. But then, you know, we think a lot, especially as you get towards agents and other things, we think a lot about making sure that the user is in control. So you don't want the, you know, as you're getting to use ChatGPT or any other AI, you don't want it to go off and do a bunch of things for you without you feeling in control. And so, you know, it's one thing if it's like answering a question for you, reading some docs and summarizing them, things like that, but as we're in this transition from ChatGPT being a thing that answers questions to a product that actually goes and does tasks for you in the real world, you should be in control of any actions that it takes. And over time as the models get better and you begin to trust them more then sure you can give it, you know, sort of more leash and trust it to take more actions autonomously but you should control that every step of the way and I think that's one of the most important ways that we're looking to build trust.
I mean there's a lot that will change as you move to more and more of these agent-based workflows and the models be able to use tools and I guess we have to co-evolve and definitely we should look into those questions. I'm just curious about your own experience. A lot of us have these 'oh' moments when we use LLMs, right? There's something sublime that happens or there's two years of your work that gets returned in 5 seconds. What was your most recent 'oh' moment using your own products?
Well, so I'll give you—this one's a little bit more, I don't know, pragmatic, but it was a really meaningful thing for me. We have—we were talking about kids as people were sort of coming on and one of our sons had a minor surgery and it was one of those things where all odds were that it wasn't going to be a big deal, but there's a small chance that it was a really bad thing, you know? And so they do the surgery and like they take the thing and they go biopsy it and you're waiting to hear back. And as a parent, you're nervous, even though you know logically that the odds that it's anything bad are really small. And at some point, we got a letter in the mail with a bunch of doctor-speak on it that looked pretty intimidating, frankly. You know, there were a bunch of words I didn't understand characterizing what this thing was. And it didn't say like you should worry about this or you should not worry about this. It just characterized it and then finished. And I couldn't get a hold of the doctor. She was in surgery or something, you know? And so there I was like, 'Oh my god, what does this mean?' And so I took a picture of it. I put it in ChatGPT and I said like, 'Should I be worried?' And I said, 'Can you explain this to me like I'm five?' And ChatGPT did it and it was like, 'No, this is totally fine. Everything—nothing to worry about.' And I actually ended up not being able to get a hold of the doctor for 72 hours because she was just super busy. And that 72 hours would have been a terrible 72 hours for me as a parent if I was just sitting there brooding. And ChatGPT was able to answer. And that's us with access to great healthcare. You think about the impact of this all over the world where people don't have the same access. It's really powerful and I think that's sort of an underappreciated part of ChatGPT.
I mean that is a wonderful story and I'm glad that your son is healthy and comes out of that well but it also speaks to the power of this particular product, right? It's a really complex product and there's no way I can talk to you without asking you about how that complex product finds its way in that funny dropdown in the top left of ChatGPT. There must be something there. You guys are all so brilliant, there must be some internal joke about, well, how should we order it? What should we call the next one? What's really going on there? And you know, shouldn't we all just be using o3 and o4 if we're in a hurry?
It's a totally fair question. And you can also—you can and you should make fun of us for our naming as well. It kind of comes back. So, we have this philosophy of iterative deployment, which is these models are—I think AI is going to change all of us. It's going to change the world. It's going to change society. And we believe that the best way to do that is to kind of co-evolve together, is to get these models out there and put them in people's hands, help them understand and also, you know, they help us discover the capabilities of the models and the weaknesses and other things. And so we sort of learn together and can iterate really quickly and improve. So that's part of it. The other part of it is we're building a lot of new capabilities as we go. And if we took the time to only, you know, we just had like one model and we just had to build everything into one model, we'd end up moving a lot more slowly. And, you know, things would be simpler for sure, but we'd end up moving a lot more slowly because sometimes it's easier to build a new model with a certain set of capabilities that's great at certain things, not as good at other things. And so together you have sort of a collection of models that can do lots of things. Each individual model has its strengths and weaknesses. And so basically we've just optimized for going faster and getting more capabilities in people's hands at the expense of a bit of confusion. And then over time as we sort of gain more control over some of the new functionality, we understand it better. We build it back into the core model. So you have models like GPT-4 that can do a lot of things well. And you know, this is what we're trying to do with our forthcoming GPT-5 is take a lot of the things that we've learned and build more of the capabilities into a single model so that it's easier for people to reason about. It's like, what model do I use? Just use GPT-5. And you know, in a perfect world, it knows how hard the question you ask it is. And so it knows whether it should give you an answer like this or whether it should think for a while. And you know, that's what we're shooting for.
So and you know, in a way that's lifting the cognitive load that is currently on the user because I will sit there and I'll think, do I have time? Is it a complicated question? Does it need to go to the reasoning model o3? Does it need a longish prompt for o3 because it might get the wrong end of the stick if I give it a shortish prompt. And in a way what you're talking about is crunching that all and building it into the model itself.
Yeah. And look, what I don't want to do is overpromise and say, 'Oh, in the future, like we'll only ever have one model and it's going to all be simple' because we're also, you know, say we launch GPT-5, we're then going to have a bunch of new capabilities beyond that that we're trying to build and experiment with and we're going to want to get those out to people and deploy iteratively and so on. So I expect you always have this sort of phenomenon of there are new models and then you've got sort of your workhorse models and then you've got some of the new ones that have certain frontier capabilities that we're experimenting with and learning with together and then over time as those mature they all kind of get merged back into single models.
So it seems that a lot of the velocity is actually about getting through the learning loop, right? And you just run all these horses at the same time so you can gather enough data about what models work against what capabilities in a way that helps you develop and deliver on GPT-5. And I hope you'll share the date with us down to the minute.
So we can time ourselves. But does that also mean that, you know, how far out are model capabilities being delivered? Are there things that are being worked on today that might not make their way into models until mid-2026?
Yeah, it's a good question. It's one of the most interesting things about working here is you kind of have a sense of what's coming and you know I'm not in the research team so I'm getting this, you know, as I collaborate and work with the researchers. I would say like, you know, on the product side we have a decent sense of what's coming in the next say three months, maybe a hazy sense over the next six months and beyond that it's harder to say. You know, you have a certain set of capabilities, you know, where like you know a bit, but you see things coming through the haze a little bit and sometimes capabilities are, you know, it's research, right? So it's not like you just have the formula and you just turn the crank. We're uncovering new things and it's unpredictable. So sometimes things take longer than expected. Other times you see these capabilities that you didn't expect at all that are kind of emergent and all of a sudden something just works.
Can you give an example of one that just worked that you weren't expecting?
Well, I mean, Deep Research is an interesting example where for a while there were a handful of researchers thinking about the—it was like, okay, we could probably make the model able to do this iterative kind of research where, you know, with Deep Research, you give the model an arbitrarily complex query to go research something that would probably take you a week and it will go off and it'll do like a hundred searches, but not all at once. It'll do three or four or five and then it'll reason about the results that it gets back. Try and understand how they pertain to what you asked and what gaps are still there and then it'll go off and do some more searches and maybe think again. Maybe it'll write some code for a little while as it thinks and then go do some more, you know. And so it's this iterative—I mean it's what you would do if someone made you write a very complex research report. You wouldn't—you go do some research.
Sorry Kevin, I don't do that anymore. I just go to Deep Research and I actually can't remember how to do it on my own. But yes, I understand, right? You sort of inch your way across and you figure out exploration strategies and go down dead ends and come back.
Yeah. And so to your question, that was something that some folks were like, okay, this is coming together, but it's not clear exactly when it will come together. And so there was a small team of researchers that just believed in this and were working to make this real. And you know for a while it wasn't good enough. It wasn't good enough. And then there were some advances and all of a sudden you're like, okay, this is getting good enough. And somewhere in that time frame we put also a product and engineering team working on it with them. And then you have the thing that I think really is the magical part of OpenAI when you get a research team and a product and engineering team just in the same room all bringing their unique skills to bear and you understand the problem you're trying to solve and so you're bringing back use cases, creating evals and benchmarks for how you measure whether you're successful against those use cases. The research teams are taking that and using that to improve the model itself and you get this tight loop of the model improving towards a particular product. And it's, you know, I think our best products are the ones that we build that way and Deep Research is a good example.
That's quite—I mean that's a novel way of thinking about product development. I mean if I think about the history of product development quite often it was, you know, before the '90s, before the consumer internet, it was run by engineers and they'd say we've got a new chip that can do this and you then try to figure out some software that can do useful things on that chip. I think the big breakthrough of the consumer internet was to put product managers at the heart of product development and you did, you know, we talked about lean and iteration and being very data-driven and user-centric. And now we're getting to this new model which I would characterize as not at all a return to the pre-internet product engineering but something that is quite novel because the researchers discover something that's a new capability and then you have to have a very rapid discussion about how can that capability be productized and then this word you used which I guess it means how can it be measured to see whether it's actually doing its trick. So is it really a new discipline that is evolving at this point?
I think it's a completely different way of building products. It's certainly different than anything I've ever done in my career. And you know within research there's kind of a spectrum, right? There are parts of our research team that are just like Deep Research. It's almost academic in nature because they're trying to just—they're looking for new breakthroughs. They're trying to find things that nobody has ever figured out before. And those kinds of things, you don't want to be product-driven at all because you just want to give a lot of room for exploration and fundamental breakthroughs. And then there's kind of the other end of research that's more on the post-training side where you really are trying to teach the models to do specific things very well and those teams tend to be much more partnered with product and engineering teams with a common goal and then it's kind of a spectrum in between. I think the right way for us to be is—we certainly don't want to be entirely product-led. That's not the magic of this place. It's not maybe entirely research-led either because it's good to know, you know, feedback about what problems you can solve for people and how we can make the biggest impact in the world. It's really sort of a combination of both with research really at the core though. And I've loved it. It's the most fun in the world and it moves super fast. Computers can like do things that they couldn't do two months ago and we're constantly in that state. But when you're working with research, then it's more than just looking at a scaling law and saying, 'Oh, we've just signed off another 100,000 GPUs, therefore it will be able to do this in six months when the training run is done.' There's more to it than that.
But one of the things I find fascinating is how do you map those capabilities against products? And you talked about evals, which I guess is evaluations. What is the structure of an eval? And is it what replaces what was in an old product requirements document that we might have had 15 years ago?
Yeah, sort of. I think in some ways for understanding where the model is good and where it's not, if you think of the model as sort of an intelligence of some sort. Intelligence is so multifaceted. People are smart in a million different ways and one smart person is better than another in certain areas and worse than another. So one way to think of evals is as a way to measure capabilities and intelligence of models on different dimensions. So you can have evals around, you know, how good it is at solving USAMO math olympiad style problems and another around how good it is at chemistry and another one about how good it is at creative writing. But are you using the public benchmarks for that, the sort of ARC-AGI and MMLU and GPQA as your way of measuring some? And then also for when we're building specific products, I think one of the most effective ways to build products is to take the skill that you want the model to have in order to meet the product need, turn that into an eval so you can actually understand how good you are at it and also how you're getting better over time. But one of the fascinating things is like the eval that we all used, you know, a year ago to measure models, they're all very kind of cut and dried. Like you're testing against math and with math there's a right answer. You can talk about creative writing evals though. And with creative writing there's no answer. So how do you grade that? Right? That's one problem. The other is like as you start to take on more complex tasks, you're not just answering questions. You're actually trying to automate some multi-step workflow. There may be ambiguity in the right way to do that. If I'm an AI booking a...