All right, we just wrapped up a conversation with Kevin from OpenAI. I, as the resident dodo, got most of it, but I'm pretty sure I missed some. I'm pretty sure Varun grasped a lot more. Varun, what do you think of it and what stood out for you?
So Kevin is the CPO of OpenAI. He's the one that led Deep Research and basically all their products. Dude, the thing that stood out for me is somebody from Anthropic had told me that code is going to be totally automated by 2027. Kevin's like, hold up, it's probably going to be done this year or the next. So very interesting insights, how they think about product, where's the commoditization. He actually argues against commoditization. He says sure, they're all commoditized, but there is gaps in capabilities and those gaps keep changing over time. I mean, there's so much interesting to take away if you're a user. There's so much to take away about timelines and predictions on what will happen over the next year, the year after that, and so on and so forth. There are a bunch of business ideas, right? TI mined out some business ideas that Kevin thinks will work in this world of AI. I mean, this is a very fascinating conversation. I don't want to spoil it for you, so please watch it and yeah, subscribe. It's because you guys watch and subscribe that we get to have these fun conversations. And yeah, roll the intro.
Today's episode, Varun, is going to be a special one because people get to see somebody outside of the same two mugs that they see every week on the show. We have Kevin Weil from OpenAI joining us. Kevin, for folks who might not know, could you tell us a little bit about what you do at OpenAI and why are you so special?
Why am I so special? Well, I have the privilege to be the Chief Product Officer at OpenAI, which is probably the most fun I've ever had in a job. You know, I've been fortunate to work a bunch of places in my career. I was an engineer in the early days of Twitter when it was about 40 people, up through when it was 4,000 or so. Spent most of my time there as head of product. Was head of product at Instagram. Was co-creator of Libra, the crypto project out of Meta. Spent a bunch of years building satellites at Planet, which is also an incredible experience. But you know, every other place that I've been outside of OpenAI, you kind of have a basic sense of what computers can do. You know, and you're building products, you're thinking about who are you building for, what problems you're trying to solve, but technology is technology. You know, databases improved but not that fast. Like your database this year is, you know, 5% faster than your database last year, but it kind of does the same things. The crazy thing about OpenAI is that like every two months we make computers do something computers have never been able to do before in the history of the world. And so like your conception of what technology can do changes and it means that you have to almost like rethink your product every two or three months, which is, I mean, it's intense, but it's super fun.
All right, so guys, the reason why Kevin is special is because his resume is bigger than my Wikipedia page. That's the main reason. Kevin, thank you so much for joining us. Varun, I know you have a ton of questions, so why don't you start?
Awesome. Kevin, thanks for joining us. Can you tell us, you know, just how it's been at OpenAI? How has it been sort of, you know, working on product? Because OpenAI is two companies, right? You're a research firm and at the same time you're a product company. And a lot of people have spoken about, you know, the models, they're going to keep getting better and research, you know, there's great research happening here, there's great research happening in other companies. But OpenAI has also captured the world's imagination as a product. ChatGPT, just ChatGPT, it is now a verb. I use it as a verb. So what's it like, sort of, you know, what does your day-to-day look like?
Yeah, it's been awesome. I mean, when you go back, OpenAI started what, like 10 years ago now, and was originally a research company. It was started with, you know, a handful of at the time crazy people who said, you know what, we can build AGI, we can build artificial general intelligence. And at the time nobody really believed them. They were kind of off in a corner, but they were right. They figured something out. They started to build some amazing demos, you know, robot hands that could solve Rubik's Cubes, things like that. And then they happened on these scaling laws that showed that the more compute, the more data you apply, these models started getting really good. And as you said, OpenAI was very quickly a world-class research company. But now, with ChatGPT, with our Enterprise products, our API products, other things that we're thinking about, we can't just be a world-class research company. We have to be a world-class research company and a world-class product company. And actually, increasingly, I don't think we can be sort of these two things separately. We have to work really well together because, like I said before, every two months your conception of what computers can do changes. And if you're operating as two separate companies, then the best thing you can do is sort of throw, you know, some improvements over the wall at each other. And that's not going to be how we build the best products. The best products, and we're seeing this with things like Deep Research, where it was really an integrated product, research, and design team. That's when you can bring together what people want and, you know, a sense of the problems you're trying to solve from the product and engineering side with the incredible breakthroughs that are happening with the models and their capabilities. And when you bring those two together early on, then you get magic. If you're throwing things over the wall at each other, you know, you could build cool stuff, but I don't think you're going to build the best stuff. And the market is so competitive. We have, you know, great companies building great models right, left, and center. The only way that we're going to win is if we really operate as one unit. And that's increasingly how we're setting ourselves up.
So your day-to-day is mostly thinking about where the puck is going with research and then saying, well, here's where product needs to accelerate to meet it, and here's what the entire whole thing looks like in six months from now or a year from now?
Well, it's both, you know, because it's one of the interesting things about working with models is the models these days are so incredible, they're not really intelligence-limited. They're in some sense like teach-limited. Like you can teach them to do anything you want them to do, but you need to have the data, the RL, the environments that teach it to do something great. And so there's actually an increasing amount of product work that goes into what are your evals and how are you, what are you going to teach the model. And there's nuance to it. So, you know, if you teach the model on, again, you train it against a bunch of competitive coding contests, then it's going to get really good at competitive coding. But competitive coding is not exactly the same thing as like real-life, you know, coding a frontend app. And so you really need to get into what problems are you solving very specifically and make sure that you're teaching the model the right thing. So that's a part where like actually the product goes backwards into research. And then like you said, there's all sorts of use cases where the research goes forward into product. There's some new capability and we're like, okay, how can we build something awesome with this? And it's why you need both groups working together. You get the best of both worlds and you build something amazing.
I see they spend so much time between research and product, there's what ends up happening is you have very little time to end up naming the products.
So Kevin, just for a little context, Varun's an engineer by training, and I'm also a comedian on the side. I just want to give context that I'm not just a random person showing up to make jokes. All right, go on.
No, it's, we deserve all the crap we get from people about our naming. It's absolutely horrendous. But we know it and we lean into it. You know, we do Reddit AMAs every once in a while and I was doing one with Sam and someone's question was, can you guys improve your naming? And we were like, oh man, how do we respond to this? You know, we're sitting in a room going back and forth and in the end I just responded with two letters: no. Like we're trying to make AGI happen and the biggest consumer feedback is the names are kind of confusing. You should just name everything AGI 1, AGI 2, AI 3. It's just easier.
Yeah, I'm excited though. You know, Sam talked about this a little bit. As we bring the models back together, we've had this bifurcation where historically we had all the GPT models, right? You have GPT-3, 3.5, 4, now 4.5. And these models are fed by bigger and bigger pre-training runs. And then we had this breakthrough last year with reasoning. And because the models were incredible, o1 preview, o1, you know, o3 that's coming soon, can do unbelievable things. I mean, the benchmarks, they're off the charts. They can do things that, you know, by all the scaling laws, you would have had to pre-train a model that was 100,000 times the size to get there. So they're incredible in what they can do. But at the time, especially, they didn't have all the bells and whistles that our GPT models did. They couldn't search the web, they couldn't, you know, handle file uploads, etc. They're just really good at reasoning. And because of that, we gave them a new name and we sort of branched the series because we couldn't really combine them. They didn't have the same set of capabilities. But there were times when you really wanted to use one and times when you really wanted to use the other. And you know, I think that was the right decision even though it was confusing because it helped us get these things out into the world. And now people are making advances with them. They're doing amazing things with the models, whether it's like coding or frontier science or other things. And so, you know, in most things we optimize for velocity, we optimize for putting these new tools in the hands of people so they can do more cool things faster. But we bifurcated. And now as we've sort of gotten familiarity with these reasoning models, we're teaching them to use all the tools that the GPT series can do. So they can handle file uploads and they can search the web and they can use Python as they reason, which is kind of crazy, by the way. Like as they're reasoning to solve a problem, they can write code to help them understand it better. Super cool when it happens. But so now the O series and the GPT series are starting to have the same sort of capabilities and it gives us an opportunity to bring them back together. And that's what we're planning to do with GPT-5.
So we are working on simplifying, even though we are ethically terrible at naming. Do you have a, this is a hard question, but do you have a timeline for GPT-5? I just, I had to ask.
It's, I mean, soon enough that we're talking about it, right? So there's a little bit of uncertainty because we do want to actually simplify as we do this, which means all of the tools and everything that people are used to need to work really well. So we are having to teach the reasoning models a bunch of new skills. And it's research, so, you know, some things have error bars on them. So I won't give you a time, but it's soon enough. We're like, we're out talking about it. We're very serious about it. People are working on it as I speak.
Kevin, you know, I'm not an engineer, right? I'm a user of the products and I love them, by the way. I think you guys are doing a great job. You know, as someone who's making the products, right, like as time has gone by, you know, we've seen competitors rising, right? And people almost say that, okay, the models could get commoditized, so the actual product and the use cases become that much more important. And now you guys also must have noticed that, okay, everyone's going to sort of, like, if OpenAI leads the way, then the others are sort of going to come out with similar products. So as someone who's making the products, are you guys now conscious of, you know, what the competition is doing while you're making the product? And how do you guys think about, you know, continuing to maintain this sort of first-mover advantage and a lead? As someone who works on the products, like, how do you guys think about that?
It's a great question. And you said, I'll sort of disagree slightly with the framing of the question a little bit, which is I actually don't think that models are becoming commoditized. I think they're moving really quickly and they're getting better fast. I mean, honestly, the rate at which models are gaining intelligence is unbelievable. Sam was talking a couple weeks ago and said that if you take constant intelligence, take a certain level of intelligence as your benchmark, the cost of that intelligence is coming down by a factor of 10 every year. You think about Moore's Law, which is, you know, basically defined our age for the last 60 years. It's doubling of the number of transistors on a chip every 18 months. And so that's twice every 18 months. This is a factor of 10 every 12 months. So I mean, massively steeper curve, right? And of course, intelligence isn't staying constant. It's dramatically increasing at the same time as costs are dropping. I mean, just like a series of trends that are unbelievable. But I don't think that these models are commoditizing. They're just all, like, because we're on such a steep trajectory, people catch the lead fast. But that doesn't mean that the lead isn't valuable. So I think even a three to six-month lead is super important because it means that you're the first to new capabilities, which means you can be the first to launch new products that meet those capabilities. And since these capabilities are new to the world, these products have never existed before, things like Deep Research, right? And you're the first to do those, you are sort of leading the way and everyone else is following. And I think that's a valuable place to be. So, you know, we're going to do our absolute best to keep the lead that we have. Other people do great research and there are aspects where other labs are in the lead and like more power to them. But we are hyper-competitive and we definitely watch what other folks do. But we want to lead the way. Our mission is to ensure that AGI benefits all of humanity. We do that by trying to put AI in everybody's hands, you know, both in the first-party products that we build with ChatGPT and others, and by making an API that, you know, three million developers are using on a regular basis to embed AI in like every tool and company and product out there. So like, we're going to do everything we possibly can to bring AI to as many people as we possibly can.
Yeah, I have a question here, right, which is actually, it's more like a personal observation, which is, look, I was about to turn, I'm just being frank, I was about to turn off my ChatGPT subscription and move to Claude for a while.
Yeah, this is like a short while ago. But then Deep Research came out. And I was on the regular plan, the Plus plan. And then Deep Research came out and I heard it was a Pro plan. And I was like, you know what, it's probably not going to be that good. But then I saw a lot of tweets about it. So I saw one tweet, people saying this is phenomenal. Then I saw another one. I was starting to get four more, right? And I saw a third one and I saw a fifth one. I said, you know what, screw it, let me go get o1 Pro, right? Let me get the Pro plan. And to be honest, it's been a long time since I had the kind of holy moment that I had with Deep Research. Like, Deep Research is great. It's like, you know, Perplexity has a Deep Research, you know, Grok has a Deep Research. Nothing comes close to the Deep Research you have. How much of that is, you know, the underlying models? How much of that is your work on the product? Because Deep Research is phenomenal. I use it every day. You know, you were resource-constrained at some point, right? You can only do a certain number of searches. I made another account and upgraded so I can do more. I use it for the simplest of queries because, you know, it's going through Reddit and like a bunch of blog posts that nobody's ever seen in the world, collecting all that. It hallucinates very, very little. It's a phenomenal, in my opinion, that's it feels like AGI. Because AI always felt like I was getting content out, but it was generic in nature. But with Deep Research, it's now filled with insights, nuggets of insights. Like if you ask it, how many people on average does it take to make a AAA game? Now, the earlier versions of AI would be very flaky about it, but now I get specific examples saying this one did this many, this one did this many, and it's just so useful. How much of that was you in product and how much of that is the research underneath?
It's the longest compliment Kevin's received, by the way.
No, no, you can keep going though. You can just keep talking about Deep Research. You're doing a great job.
Yeah, I'll be honest, actually, I think Deep Research is the best product that we've launched since ChatGPT itself. It's just magical. And it's one of the first, internally we talk about these like 'feel the AGI' moments when you just get goosebumps and you start to, you know, intellectually you see the trends in AI and models increasing in intelligence and all these things. And it's sometimes kind of hard to actually reason out like, you know, what is this going to feel like when we really have AGI around us at all times. But every once in a while you get a hint of it. And Deep Research was one for me because you just, I mean, it's so many of the things that I use ChatGPT for, it's saving me like 5, 10 minutes at a time. And that's awesome, right? I'll take all the time I can get. I've got like a job, three kids, you know, family, all these things going on. But in a lot of them, there are things that I could have done without ChatGPT. It's just ChatGPT makes them better, faster, etc. Deep Research was one of the first places where it's just, I could not have done these things. You know, I was like, I'm a physics nerd, so I was like trying to learn about muon colliders. And I had it go do this big thing for me and in 20 minutes it came back with like a 15-page report on muon colliders. There's just, I just wouldn't have done that, you know. Ditto, like doing some medical research for some stuff with my son. I don't have the capability to do that kind of research. And it went off and did it for me and provided me a bunch of peace of mind. And so there's just, it wasn't just saving a little bit of time, it was fundamentally doing something for me that I could not have done myself. And that's, it's eye-opening, right? And back to your question, I think, I mean, I think there's credit starts with the research team. They did an unbelievable job. But it is one of the examples I was telling you where we have a research team and a product and engineering and design team all working together from the beginning, making sure that the problems we're trying to solve for people match up with the ways that we're like, you know, training our model. And as we're training to get the model to like get better at certain skills, those are directly tied to the product we're trying to create. And it really is the magic of bringing these teams together. So I think it's an example of something we couldn't have created if we were operating, you know, as a research team over here and a product team over here.
Interesting. And one follow-up question on Deep Research, and I guess not on Deep Research, but you know, for example, I've tried 4.5 recently. And 4.5 has this very distinct big model feel. Like, I don't know how to explain it, but it's almost like when I use a smaller model, like let's say o3-mini, I can tell it's taking me too literally. I don't know how to explain it, but it's just one of those vibe things I get with a model. But like, 4.5 writes really well. It's able to pick up nuance in what I've said. It's able to pick up edge cases in what I've said. And from my understanding, 4.5 doesn't have any reasoning. This is just a lot larger of a pre-training set. Are you seeing, are you now going to see with this level of new pre-training, with this level new, combined with reasoning, we're going to have something new that the world's never seen? Is that, do you expect a huge capabilities jump? Or does reasoning not matter if the model is already that large?
No, no, I think reasoning definitely matters. And you can see it in the benchmarks, right? Like if you're looking at some of the more academic, sciency benchmarks, whether it's like GPQA or FrontierMath or ARC-AGI, these kinds of things, 4.5 doesn't hold a candle to like o1, let alone o3, our forthcoming frontier reasoning model. But there's also, like you said, there are these sort of softer evals, these things that are squishier and more human and like softer that 4.5 is just unbelievable at in ways that are very hard to quantify. But you can certainly see it in like human feedback. You can see it in, you know, A/B tests where you compare 4.5 next to other models and ask people which one they prefer, and you see a massive preference for 4.5. So I think it goes to the fact that there really are, there are two ways that we know of today to scale the intelligence of models. You can do a bigger and bigger pre-train and you can do more and more RL on top of it to teach it to reason. And the answer in the end isn't one or the other, it's to continue to do both. So these bigger and bigger pre-trained, they get the soft skills, they have better world knowledge, they're more interesting to talk to, they feel somehow more, you know, human or alive isn't exactly the word I'm looking for, but they feel more real to you.
Do you have a theory why?
I think you can encode a lot more subtlety at the end of the day in a bigger model. The human world is full of dimension and subtlety and nuance. And when you take a, like, if you think about going the opposite direction, if you take like a 4o model and you distill it down into say 4o-mini, you get a model that depending on how you distill it can keep most of the like, the skills on a particular dimension that you care about. So if you want to build a really good small coding model, you can totally do it. You can take 4o and distill it, or take o1 and distill it, and produce a really good coding model. And it will, you know, maybe not exactly be at the same benchmarks on coding, but like not very far. And it will be a much smaller model. But if you then try and talk to it, it's not a super fun model to talk to. It sort of lost its personality in some ways. Yeah, and like if it's a coding model, who cares, that's not why you're using it. So I mean, small models are awesome in a lot of ways, but there's some sense in like which as dimensionality collapses, you're losing subtlety and nuance. And there's something about that that makes things very like pleasant to talk to. And I think the same happens when you're going the opposite way and expanding it. There's just a lot more room, if you will, in the added dimensionality to like encode more nuance. You know, that's a little bit viby, but I think there's some science behind it as well. But it seems to be true in everything that we see. And 4.5 is just, I mean, if you're asking a model to represent a particular voice, if you're asking a model for relationship advice, if you're using it for like more of the soft human things, 4.5 is the best thing out there by far. And you won't know it until you try it. And then you try it and you're like, oh, I totally get it.
