Hello. All right, welcome back. As you saw from earlier today, a lot of exciting announcements around Box AI agents that can transform how you work with your most important data. I'm incredibly excited to have an amazing guest, Kevin Weil, the CPO of OpenAI, leads all the product functions at OpenAI and wanted to have a little bit of a conversation on the future of AI. Obviously, what you're doing at OpenAI and ChatGPT and then where agents are going in the workplace. Kevin, welcome. Thanks for coming.
Thanks so much for having me.
Yeah, appreciate it. So, first of all, what is it like to be Chief Product Officer of OpenAI and how did you get into this role? How did we get here? It's been almost a year actually.
I used to... Sam and I have known each other for a bunch of years and he is involved in so many interesting things. He's investing in fusion companies and things like that. And I'm a science nerd by background. So, I used to just always, whenever I was thinking about doing something new, I would give Sam a call and just say like, 'What is interesting right now?' Because he's always 5 years into the future.
Okay. So, he's like your nuclear phone-a-friend.
Yeah. And so this one time I gave Sam a call and he said, 'Actually, why don't you come in and chat?' And then as soon as I came in and talked to people, I was just like, you know, you haven't given me an offer yet. But this is the most interesting thing in the world. And if you give me an offer, I will sign whatever you give me. Just let me come work here. And it's been a blast.
That's... I mean not always the best negotiation strategy, but it really wasn't.
I acknowledged that up front and I was just like I just want you to know where my position is here. It'll all somehow work out.
So you've had an amazing career, largely across consumer technology and then increasingly over time a lot on the B2B side. Now you're in a world that combines those two. And we definitely want to hear any breaking news that you want to share about what we can expect from ChatGPT. But you said this is sort of the year of AI agents and a number of context that's emerged. I think that's basically true for everybody that's tuning in. If I go back two years ago, the big ChatGPT moment, this moment when we all saw the potential for AI and a chat interface. Now I think we're seeing the next rev of that which is the potential for AI agents and how do we interact with them? So give us a little bit of a landscape. What are AI agents going to look like? In particular how are they going to show up in ChatGPT? Walk us through that.
This is the year that ChatGPT goes from answering questions for you to doing things for you in the real world. And it's agents that are the vehicle for that. So we said this was going to be the year of agents. We kicked it off in January by launching an agent called Operator that can just use the web the way you use the web. It can browse. It understands what it's seeing. It understands how to search and what different elements on the web are and what's likely to happen if you click on something. And it can just accomplish things for you on the web. You can order DoorDash or go look at Box documents. Actually, you had a great tweet when we launched it about having Operator use Box to assemble a data room.
Yep. As if you were doing an acquisition. It's actually... actually what was a quick sidebar. So Operator's fantastic. So anybody listening should definitely check it out. It's very surreal because you're watching the AI use your software and so that gives you this little window into it. And there were actually a couple moments in there I was a little bit mortified of like gosh we do need to make that more usable. Like it didn't know to click that one button for this particular function. And so it was actually like weirdly this incredible usability study using AI because you're just like it wasn't trained specifically on Box and has to figure out how to use it, right? And so you just see these incredible kind of moments in the product experience. So browser kind of AI agents was the kickoff to the year. What would happen next?
That was the beginning. And then I think the one that really opened people's eyes, certainly opened my eyes to the power of agents was Deep Research. Where the idea is you go from ChatGPT which normally you ask a question you get an answer to, and some of the reasoning models which we can talk about as well, you ask a question and it'll think like the way you and I would think if we got asked a hard question for 30, 45 seconds. Deep Research, the idea was well what if we let these models think for 20 minutes and they could go browse the web. If you ask a hard question they could go browse the web. Say you're looking to understand the impact of tariffs across industries by state and you want to, maybe you're a small business and you're veteran-owned and so you also want to look at arbitrarily complex kinds of things that if you were going to go do that work might take you like four or five days. It's going to go do a bunch of searches, understand what it learns based on that, reason about what it's learned, figure out, no, there are still gaps here. Okay, so I'm going to go do more work over here. And you end up with this, often like 20-page document. And to me the most interesting thing about it is rather than a lot of what ChatGPT does which is like saving you five minutes here, a little bit more efficiency over there, summarize this document here, all of that's super valuable but Deep Research was a bunch of like when I asked Deep Research queries I just wouldn't have been able to do that research because I don't have five hours or five days or whatever. Often I don't have the expertise to do even what I'm asking and yet I can have this agent go off and do all this work and come back with a meaningful well-sourced document for me.
Yeah. So that's been a big game changer. One of the things we're doing now by the way is especially around the partnership that we have together, we're starting to let Deep Research connect not just to the web but to internal sources of knowledge. So think your Box documents, Google documents, SharePoint, Jira, things like that. And I think that's going to be the next really big kind of magic moment because we all have so many documents. And everything changes all the time.
It's great. There are lots of them. More documents. Imagine if we can together make them way more useful and allow you to introspect, dive into any number of documents even across services to get answers. I think there's going to be another magic moment there.
We totally agree and anything that relates to documents and AI we definitely believe lots of magic moments to come. I want to circle back for one second on the Deep Research because we're going to bring you back to the interop point and you guys have been I think great stewards of interoperability in AI. But on the Deep Research side we basically said wow like this is searching the web in this completely different way and it's an amazing breakthrough. But it was really like you compressed five or 10 minutes of work into three seconds. Deep Research to your point was all of a sudden now you can start to do work that you probably never would have even thought before because even the barrier to imagine like I'm going to go ask somebody this question and then I'm going to go make them spend a day of work to get back to me. It just basically reduces all of the use cases you have. So Deep Research is this incredible breakthrough. How much of the work is time being spent going across the internet to find lots of sources versus how much is the model kind of just in its own head rethinking about the problem? How much is the research part versus the thinking part and what is the breakthrough on reasoning models that led to this being possible?
Yeah, one of the really cool things so the models now can use a variety of different tools and if you think about from the perspective of the model being able to search the web is just a tool. It can also write Python code. So if it wants to do a bunch of data analysis over some of the data that's come back, it'll actually write a little Python program. Same thing if it wants to visualize the results in a graph or if it gets an image back and it wants to better understand the image. I've seen it write Python code to do image analysis as it thinks. It's absolutely wild. But it can employ all of these tools as it reasons. And that's really been the big breakthrough. And then enabling the model to get good at sort of interspersing these things so it can use a tool get some information back reason about that then go okay well the thing I need to know next is this and so maybe I'll write some Python code to do that and then I'll search and then I'll reason a little bit more and then I'll search again and write some Python and then use this other tool and then use memory it gets to know you over time through your usage and it can bring all of these things together into one final answer that's like kind of been the big unlock.
Yeah. When let's say generating Python code for some data analysis kind of sort of task. Is it always starting from scratch or does it have a library to kind of pull from of okay this is the common thing I should build or use for this type of use case? Is everything ephemeral?
Everything is ephemeral. Everything starts from scratch but you know it's a... the model is very good at writing Python and so it very quickly can... the things that are in libraries that you might import, I mean it can import them too but it can also just write it.
Was there some moment in the past 12 months where then the reasoning model dynamic kind of was this discovered? Did... was it an emerging property of AI that you guys kind of figured out? How did we get here?
You can think of it kind of like if I ask you a question, who was the third Roman emperor or something like that, you either know it or you don't. The model knows it. So the model just says here's the answer. If I ask you a harder question, I ask you to solve a crossword puzzle or a sudoku. Or even think about booking a flight. If you have to book a flight through multiple places you're like okay well flights to this place get in this time, the overlay. Okay. No, that's not going to be a long enough time. I might miss my next. You're forming hypotheses. Using what you know to validate those hypotheses. You either say okay yes that can work now let me go to the next phase or no it can't let me go back to the drawing board and think again. That's what these models are doing. And it was a really big breakthrough and it's in the same way that if you had to give a snap answer to every single thing in your life you probably wouldn't do a lot of good thinking. The ability to reason has unlocked the model's capabilities and we just see huge jumps in the model's intelligence, ability to solve hard problems, things like that.
It becomes very convenient to kind of anthropomorphize what you just described in the way that we do this. But you have this time compression capability of the power of compute so do you imagine that these thinking tokens that are kind of streaming out of the model while it's doing reasoning will that eventually with let's say GPU performance improvement still be more or less instantaneous and it's compressing all of that or how far are we off from that being possible?
Well, I think that's actually one of the... you see the difference between you let the model think for one second for simple questions. Actually o3 has this new search mode where it'll think for like a minute of search and that's actually pretty cool. You get kind of a mini Deep Research with a minute of thinking. You've got full Deep Research where it might take 20 minutes. But what if we instead could let the model think for like 3 days or 3 months or what happens in that case? When you think about fundamental science for example, you don't make breakthroughs in fundamental science by thinking about something for 20 minutes. You devote weeks, months, years of your life to it. What if you let these models do that? I think that's actually going to be a big part of our future.
Very core to OpenAI's mission to think about what if we could speed up the pace of fundamental science research and innovation so you imagine a world where we will basically deploy a problem in the form of some kind of task or whatnot at some cluster of computers and all AI agents and just check in on it and see did it come back with something that was relevant for our work and you might have these very long-running agents kind of completing work for us.
Yeah, absolutely. And you're going to have different tasks that take different amounts of time. Some are still going to be very quick. Summarizing a document is going to get faster and faster and it'll be more or less instantaneous. But I think as we go forward, you're going to be, today you can only give models can solve incredible problems relative to two years ago. Relative to a year ago, relative to 6 months ago. It's kind of mind-blowing and we get used to it very quickly. But they still can't solve everything. And the way that they're going to be able to do that over time, I think, is you're going to give them more room to think to solve hard problems, which is kind of intuitive.
What is the pace that you're seeing from the rate of model improvements happening? So to your exact point if I look back at ChatGPT kind of 3.5 versus some of your latest models GPT-4.1, o3 or o4 mini I mean it's incredible because you kind of look back and you're like I can't believe that was the breakthrough model. Because our expectations now have been heightened so much. But what do you think about the pace and especially on the axis of cost of the tokens have dropped dramatically, in probably some cases like 100x plus, the accuracy rates have improved dramatically in some cases kind of 10x plus depending on the data type. Are we... is this trajectory going to continue? There's this thing called scaling laws if you want to maybe comment on what that looks like. Where are we in that journey of kind of saturating or plateauing versus no the acceleration is just still happening.
Everything that we see says that we are not plateauing and there's a lot of room to continue growing. So the crazy thing about where we are right now, every I don't know three months computers can do something that computers have never been able to do in the history of the world before. And that's just... that didn't used to be our world and now that's just the world we live in. But you've got two exponentials that are happening at the same time. Model capabilities are increasing and you mentioned we have two ways that we scale the intelligence of the models. One is the traditional way where you just do bigger and bigger pre-training runs. And that still works. It's an expensive way to scale but it still does work. And then you have this inference time compute, the reasoning where you can just essentially allow the models to think for longer. And both of those make the models more intelligent. So, as we improve on either axis, models get better and we don't see any stop to that anytime soon. And then on the cost side, you mentioned if you compare GPT-4 when it originally came out a couple years ago to some of our models now, there's a 500x cost difference. So, you know, Moore's law was twice the number of transistors on a chip every 18 months. This is if you go let's even say it's 100 over two years drop every 12 months. It's a faster exponential than Moore's law. It's just a wild place to be in this current moment.
Are you willing to commit to this as being the trend? Can this be the sustained trajectory that we're on?
That's what we hope. I mean, look, you've seen us, most companies think about putting things behind a paywall. We take things from behind the paywall and give them to people for free. We drop the cost of our API repeatedly whenever we can basically because what we see, our mission is about bringing AI and AGI to the world and the smarter the model we can offer for the lower price means that more use cases become ROI positive means more adoption means the world progresses faster and so that's... this is what our mission is, we want to... we're going to keep doing this to every extent that we can. It's been great to see.
And there was this debate at the very beginning of the year with this breakthrough open source model from DeepSeek and the big debate was this idea of okay if we bring down the cost of the AI models does that then reduce compute usage and to your exact point every time we've lowered or seen any lowering of costs on our kind of AI expenses actually usage goes up more than commensurate with the cost drop because all of a sudden you can start to throw AI at more problems that weren't affordable before or they weren't ROI positive or alternatively when you get that efficiency to come in then you get breakthroughs like reasoning which all of a sudden then reintroduce a 50 or 100x increase in the amount of AI usage for a particular task which are both interesting kind of examples of the compute kind of growing again. Are you more constrained by do you think improvements on the hardware architecture or algorithmic improvements in the form of the training runs and kind of what results you're getting out of that or model whatever you're doing from a post-training model efficiency standpoint where would you like to see the bigger breakthroughs happen at this point?
The good news is we see a bunch. I mean the hardware is getting better every single year. We are making continued algorithmic improvements, we and the whole community. I think that's one of the reasons you're seeing cost drop to the degree that it is because it's actually just getting cheaper to inference these models. And you're seeing intelligence scale. So you've got like and you said this earlier, but you've got intelligence going up, you've got accuracy going up, you've got cost coming down dramatically. The models also get safer as they get smarter. So when you think about the various aspects of AI safety, the more recent models we've launched are also the safest models that we've ever launched. They're more controllable. They, as they reason, they're harder to jailbreak, things like that. And so you've got basically this set of four things that are all going in the direction you would want them to go. And it's creating an opportunity for incredible progress.
