You're watching TVPN. Your background looks way different because you have a whiteboard behind you because we're breaking down the X's and O's of the GPT5 launch today. GPT5 launched from OpenAI. Uh really quickly, there is some other news. Firefly Aerospace stock opened at $70 in NASDAQ debut. This is the company that landed on the moon. Very cool.
Very cool. Um there there are a few other stories going on, but we're going to skip most of them because we're going to be focusing on chat GPT today on GPT5. We have a bunch of uh a bunch of guests coming on. We have a stacked lineup. We'll pull that up, but we'll break down the x's and o's of the matchup. So, of course, Open AI, here's our here's our lineup. We have something like 15 guests today. uh a ton of folks from OpenAI, a ton of people that build on top of OpenAI and uh can comment on what's going on with CH GPT. Um but of course this battle is between OpenAI and the timeline. It's the it it's they got to get the vibes right.
It's war. It's it's the timeline's in turmoil over whether or not this is a good model, what it means for the industry, what it means for AGI timelines. Everyone's got their take. Everyone's posting memes. There's been a ton of funny ones already. We'll take you through them, of course. But let's break down the offense today. We have Sam Alman, the founder, CEO. He briefly got cut from the team in November of 2023, but he's back leading the team for the 2024 2025 seasons. He seems healthy. He's doing great today. Uh he went on at 10:00 a.m. to break down the launch of GPT5. Uh he has a couple of key plays in his playbook, in his arsenal. Uh he's got a solid ground game, lots of quick posts, hitting the timeline, probably in lowercase. Then he might air it out with a couple thousand-word essay. We've seen him do this before. It's a bit of a hail mary. Maybe a thousand a couple thousand days away. Maybe we're in the soft singularity, but he's very strong there with the long post when he needs to be. It's up his sleeve if he needs it. Um then he can also pull out the vague posting. He was doing this last night. Posted a picture of the Death Star. No one knows what it means. Maybe it was taking a shot at the doomers who are on the defense today. So he's also known for driving supercars. that lets him get to the office faster. He's saving time and money. You can save time and money by going to ramp.com. Easy to use, corporate cards, bill pay, and accounting and a whole lot more all in one place. And so he is uh he also gave apparently, this is a rumor, he gave every Open AAI employee who's been with the company for more than two years $1.5 million. A lot of people say 1.5 million, that's not enough for a big house in San Francisco, but it is enough for a supercar. So that's probably why he picked that number and that's why that's what the OpenAI team will be doing with that money. They'll be buying Aston Martin Valkyries, Pagani Huayas, McLaren Sabers for Ferrari Daytona SB3s. Uh they can get a Koig set game. They could get a Singer DLS or Bugatti Veyron. It would have to be used. They could also get the Bentley Bakalar. There's only 12 of those ever made. Uh it's an open top two-seater roadster. It's coach built. So that's going to run you 1.5 million, but that's perfect. You just got the 1.5 million bonus. So put it to work. Spend it all in one place on a car. This is financial advice.
Yes, exactly. Then you got Greg Greg Brockman. He's joining at noon. He's he's extremely well-rested. He's actually coming off a sabbatical right now. That's very exciting. Uh he should be injury-free for the rest of the season. Uh he cut his teeth at MIT and uh then he got drafted by Stripe in 2010. Uh Microsoft tried to do a trade deal during the 2023 chaotic trade deal trade window that opened up post Sam Alman ouster. Uh but he stuck with the open AI team and now he's president of the company. Then you got Mark Chen. He's coming on at 11:30 today. Uh he's the chief research officer. The rumor is that he turned out a maxed out contract to head the Metal Lamas, but he's sticking with the OpenAI team. He was an MIT undergrad. Also worked at Jane Street before joining OpenAI in 2018. Then we got Sarah Frier coming on the show at 12:30. She's the CFO of OpenAI. It's her job to find bank accounts big enough to find to fill all the cash they're raising. It's it's a tough job. You got to find Okay. This bank account, will it hold 10 figures? Will it hold 11 figures? Will it hold 12 figures? Like a lot of cash in this one. Exactly. Exactly. She's also going to be defining the non-GAAP metrics that will be catnip for Ben Thompson in just a few years. We're excited to talk to her about how she's measuring the success and the health of their business. Obviously, it's not just revenue. It's not just topline, bottom line. We're going to want to know about queries. We're going to be want to know about DAUs, all those non-GAAP metrics. That's where people are going to be tracking when IPO uh when IPO day comes hopefully soon. And then we also have Brad Lightcap. He's joining at 2:35. He entered the league as an investment banker. Let's give it up for the investment bankers. They don't get enough credit around here, but we love the investment bankers. Then he got drafted by Y Combinator before joining OpenAI as CFO in 2018. Now he's the chief operating officer. And then we have uh Max Schwarzer. Uh he's in charge of post training, fine-tuning these models, getting them into the fight bit fighting performance to put on a display of authority on GPT5 launch day. Now let's flip it over to the defense. They're going up against the timeline. They're going up against the vibe checks. We got the Doomers. The doomers. They're led by Ellie Yudkowsky. Admittedly, everyone knows this. No one debates this. The doomers have had a terrible season, but you'd expect to see at least a few hail marys about GPT5 creating bioweapons thrown up on the timeline today. Probably won't be bangers, probably won't get a thousand likes, but you'll be seeing them here and there, mostly in the replies. We've also seen some doomers talking about uh GPT5 being available to every government employee, and Eleazar had some harsh words about that. don't give the keys to Sam Alman. Don't give the keys to the government to open AAI. Uh he was upset about that. But in general, the doomers not putting much of a fight up today. Then you got Claude. Uh interesting. Claude was caught playing for the wrong team earlier this week. Anthropic, they're on defense today. Uh but we saw them take out OpenAI's key pinch hitter, Claude. Uh the Claude Code API was playing for the OpenAI team, but they shut that down and Claude is no longer pinch hitting for OpenAI. Uh then you got the Elon stands. Uh the ground game's going to be there. It's going to be track uh it's going to be strong. The Elon stands are going to be tracking the benchmarks relentlessly. We know XAI loves to benchmark and all the Elon stands are going to be calling out GPT5 for any any misaligned benchmarks. If they fail humanity's last exam, it's over. It's over. Uh they'll also toss up the occasional unhinged conspiracy theory. Uh moving on, Gemini. Uh the betting lines have shifted big time. People thought Gemini was out of the game. They're so back. Poly Market has Gemini at what 75% chance of being the best model towards the end of the month. This is of course based on the LM Arena more vibes-based benchmark. But uh Gemini will probably be quiet today. They usually don't try and front run press releases. They usually try and sit back, let the model speak for themselves, let the API credits work their way through the latest YC demo day batch and get the product into the hands of people. And so, expect to see a big uh glossy conference in a couple weeks demoing uh Gemini 3 should be a good rebuttal from the Geminis. Uh then you got the Metal Lamas. Zuck's been on a poaching spree. He's rebuilding the team during the off season. Uh now he has a stacked roster and he's ready to go duke it out. But no one knows exactly what's going to be in the playbook. Is he going to go consumer? Is he going to go API? Is he going to turn into a hyperscaler? We don't know. But we know they got a stack team. They got Alex Wang. They got Nat Freedman. They got Daniel Gross. They got tons and tons of other researchers. They've been raiding every other team. Completely reset the salary cap for the league. And it's been uh it's been an absolute clinic in terms of recruiting over there at Llama. Then you got the final benchmark, Arc AGI. This benchmark stands. GPT5 couldn't get past this defense. And uh RK AGI, you know, sitting there right in the end zone just swatting him down. Swatting him down all day. You think you think you you think with super intelligence around the corner? RKGI denied. Denied. Uh Tyler, give us the update on ArcGI. Where does everything stand? How GPT5 do? Does it matter? Should we care about ArcGI? We love the team behind them, but is it an important benchmark? Should we be tracking it today?
Um, yeah. Okay. So, so there's RKGI V1 and V2, right? And V3. I actually don't know if No one's been No one's even tested V3. No one's even really close there. But how we doing on V1? GPT5 is at 65.7. Unfortunately, that's going to be 1% just short of Grok 4 at 66.7. Okay, Arc AGI 2, 9.9%. Grok 4 16%. So, absolute kind of brutal, you know, ARC AGI mogging.
Rough showing. Rough showing. people have have accused Gro 4 of being slightly benched maxed. You know, this is, you know, they might have a team, but um what's the what are the pros and cons? We know the cons of benchmarking uh of benchmaxing. You're overfitting on something that might not actually drive consumer value. It might not actually solve real world problems. It might not increase DAUs or revenue or ARR or anything that really matters. It might not even get us closer to super intelligence. Give me the counterargument. Why is benchmaxing good?
The bull case for benchmarking. A bench maxing. Break it down for me. Yeah. So I I think the idea is basically um this is almost like a non-AGI build kind of take, right? So if you don't have a a super general intelligence, um your ability to benchmax basically proves your ability to um solve some like kind of specific task. So, so there's this um thing about the the gas station. Yeah. It's called getting spiky. Getting spiky. Getting adding more spikes to the spiky intelligence. Yeah. I think it was Rune who had this this tweet about the gas station benchmark. Right. I I don't care if he said something like uh I I don't care about um AI solving gas stations if it has the gas station benchmark. Something like that. Um, yeah, but the idea is like if if you if the if making the gas station benchmark run said, 'My bar for AGI is an AI that can learn to run a gas station for a year without a team of scientists collecting the gas station data set in in capital letters.' Yeah. And then my take is basically I don't care how they got to the like I don't care how they made it run the gas station. I care how fast that it runs it. If it if we can run the gas station with AI, if you have a team who's you know your benchmaxing team, that just proves that like if you have some task that's like really important that you want to get done, they can just figure it out. So it's like RL for business. This is like the same thing RL for law. All these like specific verticals if you doing the thinking machines, right? RL for businesses come into your organization, understand the most the most valuable business processes out there that could potentially be RL against that could be turned into a benchmark and then and then, you know, bench hacked because I don't care if you're hacking, you know, if I have translate this type of document to this type of document for my business. If you can do it with 100% accuracy, I don't care that you benched it. Yeah. Exactly. Like like benchmarks right now are not like economically valuable. Like if you if you're really that much better at MMLU, it's like is are you producing that much value? Probably not. But if you have if you make some new benchmark that's, you know, your tax benchmark, I think Anthropic just released that fairly recently. That's like I don't care if you benchmax on that. It does way better because then it's going to it's going to do the task. Yeah. Yeah. Yeah. Yeah. That makes sense.
Um what about um the what does it say that it feels like open AI seems capable of bench hacking? It seems like they've opted not to. Is that because bench hacking has a risk of giving you negative aura? Because if you're accused and found guilty of bench hacking, you could it it it often reveals that you're not building this one beautiful, you know, super intelligence to rule them all.
Yeah. I think it's also like maybe we're just looking at the wrong benchmarks. Like maybe they're um there's a bunch of like interesting benchmarks about like there's this one I really like. It's the Minecraft benchmark where you have to like build you like give it some castle and how how good it looks or there's the one you always see about um the unicorn. Yeah, that's and it's um so you use this like math package that does like grass and stuff but you ask it to to draw a unicorn. I've seen that. Yeah, those are really good because it kind of shows the creativity stuff like that.
Uh walk us through TBPN bench and what we will be benchmarking the uh the AIS against going forward. Have you heard about this reps of 225? That would be close but it's difficult because uh the humanoids kind of change that and you can just use normal actuator. This is this is truly for a large language model. You feed in our data set. We have a public data set a private data set presumably at some point but walk us through TBPN bench.
Yeah. So so I'm yet to try this on GP5. I don't think it's out yet like for public use at least. I don't have it. Um but I can I can tell some of the questions. Right. So so the first one um I have this picture of a horse. You have to guess the breed. Yep. So, um, let me see. I think why I don't want to say it in case D5 is listening, but it is may or may not be a Caspian horse. Okay. Um, and it's failing right now. 03 is failing. 03 is failing. Oro is failing. I haven't tried every Yeah, we got to try Grock and Gemini. Bring it all out. Horse identification. This seems extremely hackable, but at the very least, if we get one scientist to be to go off and collect the horse data set and then and then uh and then bench hack it, I think we will have done our job. Yeah. So that's the first question. The second one is a it's I have two pictures of before and after of uh this guy and it's which peptide didn't take to achieve this body transformation. Yep. Yep. Yep. Um so it fails there. It fails there. So you have a data set of of what peptide does what to the human body. Where'd you find that? Well, you know, Wikipedia has a lot of this stuff. Okay. Okay. You'd think they'd be able to it'd be able to cheat this around with 03. Just reason who is this person? go look up what they've said they've taken and then boom, you have Well, at first with 03 when I was prompting it, I would like save the the photo, but then it would have the metadata or the the file name would be like Caspian horse or something. Yeah. Yeah. Okay. And then and then the third one, the third one, um, I pass in an audio file of a car revving. Has to pick which one. It has to pick. It has to identify the car. The car. Yeah. From the engine note. From the engine. And it's not doing it currently. It's No, it wrong. This is This is a good benchmark. We humanity's real last exam. Yes, exactly. So, I think those are pretty solid. I have some more. Obviously, I don't want to make them public in case anyone's going to try to, you know, benchmark this. Of course. Of course. We'll see. Hopefully.
It's funny because um I was I was mentioning the other day this this app that my dad had of like tracking the like you just set your phone up and it just automatically detects which birds are in your backyard. Yeah. So, yeah. I mean, this has to be extremely solvable. It's just something that it it reveals the lack of like general general intelligence when when you have to go and and collect the horse data set which should just be out there or the engine note data set which should just be out there. Um but but but clearly we are in the age of go on the on the individual problem and we are looking at like the power law of capabilities. knowledge retrieval is clearly a you know 12 billion dollar a year market that consumers will pay for that will probably grow significantly. Um and and then health and therapy and shopping and all the other features that PGCMO laid out uh in her post. This is kind of like you know what will be rldled against because those are key pockets of value in the in the consumer economy and the same thing will happen in the business economy but in the B2B context you'll probably see an individual startup building on top of an API but even then most of the most of the model platforms offer kind of RL as a service fine-tunes as a service something where if you're starting to spend tens of millions of dollars they will do some customization on top of the model so that could be the regime for the next few years as we go into this like you know uh instead of like this centralizing AI force there's only one company there's actually like a camberan explosion of a ton of companies doing a bunch of different things so anyway let's go to signals post signals not happy with the launch he says okay I've seen enough this launch felt like attending a funeral hosted by minimalists uh they're unveiling tech that should feel magical real breakthroughs but the whole vibe was grayscale grief the set design looked like if mood disorder got bow got a bow house grant Um, I don't know what a bow house grant is exactly. Uh, even the storytelling arc, chart styles, the eulogy, tributes, then closing on someone's health battles. What exactly are we uh are we as the audience mourning? It feels like they're trying to get you to pre-install a therapist. Uh, potentially great products, sure, but the emotional tone was so damn DOA. Uh, incredibly strange all around. I I think a like like it's weird because we're in this world and this is a question that I want to noodle on all day is is will this be the last launch of a GP of a number GPT model because like you don't hear about new new versions of Google going out. You just it just got better and better and better. Same thing with Amazon when they were optimizing for hey it's faster. We have more on our cat. Take away was is that the product matters more than the model. Yes. now and probably will for potentially a very long time. And when we were watching the stream, I was cheering because they gave the feature of you can now talk to the model and get it to trigger a deep reasoning workflow or get it to give you a quick answer in natural language. And so it's it's abstracting even fur even more of the UI into the actual text interface. And so I think in terms of like surprise and delight and and I don't know, it's like you you everyone kind of rips the Apple thing, but Apple does a great job of being euphoric and and and happy with somewhat minor product changes and like maybe that's more of where they'll go is just, hey, there's these new features and here's how these things and Apple will spend 10 minutes on stage talking about like shifting an icon around and stuff and it's like I thought it was interesting they just sort casually mentioned that they're they're deprecating the old models. I think it's great, which makes sense. I think it's great. I don't want the model picker anymore, but are you upset? A lot of people are going to be upset about that. I think they're getting rid of 4.5. Oh. Oh, really? And you're and you're and you're a 4.5 fan. I love 4.5. But I I would imagine that the future is if I ask it to think really hard about the pros and the writing style, it would then do a pass on with 4.5 and but it would only trigger that when it needs to. It's not going to give you that because if I'm just asking for, hey, regurgit regurgitate a bunch of facts or write some code or or put together a table of data, like it's not going to need to pull 4.5 off the shelf. Just like it's not always going to pull Python off the shelf. It's not always going to pull web browsing off the shelf. And so I I'm I'm not I'm not sure that I necessarily want 4.5 there as a selection criteria. I would like all this to be tucked behind a UI and have something that's actually cleaner and less frustrating to use. I think it'll lead to higher retention. Yeah. For the average like normie that no one knows what 4.5 is like. That's true. That's true. Uh anyway, Chris Pikes is open and anthropic are duking it out. Meanwhile, consumer surplus is growing. Um, we also have very good news. Um, we also have uh uh the details from Mike Nuke over at a RKGI. Full GPT5 is along the V1 paro frontier. Uh, that's cost versus performance. OpenAI said they focused on other goals like UX and reliability. Our testing supports this. Uh, Mini GPT5 is super impressive accuracy for cost. In fact, based on cost efficiency, Mini could have entered ARC Prize 2024 and likely won first place. We are still verifying GPTO OSS or as Rune says GP GP toss um results soon. Nano GPT appears overfit. Performance is commodity. Uh and France Chalet is also chiming in with the top line. Yes, production team needs the deck. Oh, we don't have a deck today. We're just going through Yeah, we're just we're we're just riffing through the timeline uh the timeline uh tab and just pulling up some random posts. So So you're free to pull those up, but also we can just read through them. Um Ashley Vance is saying, 'But but model switching was my job. Model switching is out and we are into the future. Just talk to the model. Just talk to the model and ask it what you needed to do and it will switch for you. It will pull the right tool for the job.' Anyway, um the other question that I have for the OpenAI folks today is on the nature of secrets. So, in 0ero to1, TL has this concept that that discovering a secret is key to building a startup and it's a key insight and I I was joking with you know the super intelligence or or GPT5. Could my first prompt be teach me exactly how to build GPT5? And then I go to Meta and I say I know how to do it. I have the prompt. I have the I have the result. And of course the answer is no. Of course, OpenAI would never leak the most frontier capabilities into the model. But can you build a super intelligence? Can you call it super intelligence if it doesn't if it can't tell you how to build super intelligence? One read on what the secret might be is that the app was the most important thing all along. And if you if you create this narrative that that super intelligence is, you know, weeks or months away and you get a bunch of people that go and try to compete on raw intelligence, meanwhile, you build a consumer app business with billions of users. Yeah, it's like a seems like a pretty good strategy. I guess one quick thought is is how do you how do you rate uh Sam's vague posting from yesterday with the Death Star in the context of this new uh in the context of of of the release today?
It's a great question. There's a bunch of reads on it. One is just that like the Death Star is to some degree like Stargate and you have to Oh, wait. He if this is the apocalypse, I figured I'd at least tune in live. Um, you have to like the the impact of this of GPT5 is not one crazy super intelligent model that does everything. It's just a more user-friendly higher retention lower churn consumer model that weaves its way into all aspects of daily life and improves performance and efficiency all over the place. And so you have to build this massive cluster to serve all of that. I don't know. What's your read on it?
I don't know. I think it just was I think it was dramatic. It was provocative. People didn't like it. It is provocative because there are many other like super mega structures that are in in sci-fi history that are positive or positive. Yeah. Yeah. And this is this is like but it gets the people going. Yeah. I don't know. Is it is is it a metaphor for someone else that's going to attack? Is he I mean I mean the image is from the viewpoint of someone looking at the Death Star. Is he saying he is seeing a Death Star being built on the horizon? Is that something else? Is that another company, another organization? Is that the government? Is that the is that is that legal? Here's a read from uh Bubble Boy says, 'I am an expert on bubbles, so it brings me no joy to say that the AI bubble is popping this time next year.' Mhm. Uh he is updating his timelines. When you promise infinite scaling and don't produce it, the calculus changes. I don't think it will be bad for most companies, but those who built their entire business model model around making the best LLMs are unfortunately going to struggle as models become more of a commodity. Again, OpenAI is a my read on it is a consumer app business, right? They still have a big enterprise business, but but by you know their recent valuations are are predicated on their this incredible consumer business that they built. Yep. Uh Bubble Boy says the end user doesn't care much if Claude is 5% better than GPT5. They care about cost, speed, and utility, especially at scale. Things will be going. The obvious play now is shorting Nvidia and dumping. Uh okay, start getting into financial advice territory here, bubble boy. Uh but um interesting uh again kind of goes back to what I was saying earlier uh in that um if you were raising billions to make a lab and I think the potentially anthrop you know we'll see what happens in the coding market but there's some clear winners emerging and then on the consumer side um you know expecting a power law outcome and and it's hard to see anyone unseeding uh chatbt completely agree. Uh I want to dig in more, but we have our first guest. Let's welcome him to the stream. What a day. Mark, how you doing?
Hey, pretty good. Nice to see you guys again.
You uh congratulations on the launch. Uh take us through it. Uh are you were you actually live or are you wearing the same thing and you recorded it yesterday?
I'm actually live. I don't know why, but we do.
Yeah, it's gone. I mean, we're big fan. We're big fans of live. I mean, it just allows you to be mo the most reactive to the most new information. Um, give us g give us the core thesis that you are trying to get across. I think that there are a few narratives out there. Um, we we've been enjoying the one that's, you know, this is a dominant consumer product. They just made it a better consumer product and people are going to use the product and get more value out. I saw a bunch of things in the presentation where I was like that's going to make my daily usage of of chatt better. At the same time, we're in this we're in this world of oh the models the numbers matter and the scale matters and this and that and this and that and uh and it's a it's a fine line and it's a dance and and we're in a transition phase away from benchmarks and away from talking about the size of the bubbles. But what was your core thesis like what did you want to get across to the listener?
Yeah, I mean fundamentally I think from a research perspective, we've been working on reasoning models for several years now. And I think until now you've had this really clunky interface. You have to pick, you know, GBD4o or you have to pick o3. Um, and for the longest time, we've known that o3 gives you better answers across the board. It's just too slow, right? I mean, you often don't want to just sit there and wait for the model reason it out. So we've done a lot of work to push the speed, the performance of our reasoning models such that these can come together and work in a very seamless way. And so I think you know above everything we're trying to move the world into this agentic reasoning world. We believe that's the future. And on top of that you know you pointed something out which uh I really resonate with. Post training is a huge part of this release. We really wanted to highlight uh Max Schwarzer and his team who did a phenomenal job and they've made the model just really that much more useful for consumers, for businesses. It's a monster at coding. So yeah.
Um on the on the speed of reasoning, you're obviously the chief research officer. um is are are you more optimistic about getting speedups there from I don't know algorithmic design software optimizations or new hardware just let Moore's law carry on or find new AS6 or we saw Sarah Brris posting yesterday about um the incredible speed that they're getting 3,000 tokens a second on GPTO OSS um and I'm wondering what levers obviously we pull all of them but But but what what what path of the tech tree are we should we be like most focused around most tracking and uh and most excited about?
Yeah, I mean as a person who represents research, I control the things that I can't control and I think a lot of that focuses on algorithms, right? Simple algorithms that are scalable that we can pump a lot of compute into. Um we also do care about the hardware improvements that are stacking up. Um with the open source release, you see thousands of people, right? um really kind of serving these models creating really great inference stacks and those are really great lessons for us to pull from you know how what's the ceiling of the speed in which we can serve these models.
um what uh what can you tell us about the actual um like like user experience of speed um I was I I've just like last week I finally got to a place where like for a lot of tasks I'm I'm firing off a 4o query and an o3 Pro query. I just have I have two tabs. Yeah. o3 tab. Exactly. And and and I'm wondering um uh what user experience patterns you think can uh help people balance between those? Is this like just something that we're like different patterns that we're going to learn over time or different uh or or or are there going to be certain problems of user experience that are purely solved just by better product design, better speed and we don't even need to learn these because I remember like you you know when you when you prompted a uh an image generator you used to have to say like don't no six fingers five fingers please or like don't make mistakes and now you know the models kind of have that baked in. Um but but but how how are you thinking about the user experience of getting the user um the results in the right amount of time?
Yeah. I mean this is one facet of why we believe so much in reasoning. It's just because all the scaffolding you used to have to give the model. All these small hints they go away, right? Like the model can examine its own outputs. It can review them. It can be like hey look like I'm just counting the fingers here. Why are there seven? And um and it can kind of fix that, right? It does a lot of iterative generation. and does a lot of fixing things on the fly. And so we think one of the benefits of bringing reasoning to the world is really to kind of remove the need for scaffolding. And with GBD5, right, we know how clunky that that experience is with uh switching between 4o and o3. Actually, I mean, there's so many stories. Um I was just talking to someone yesterday, right? They're like, 'Hey, well, you know, I've used 4o my whole life, right? It's the Frontier model.' And I'm like, 'Hey, well, have you tried o3?' And they're like, 'Why would I try o3?' you know, three is less than four and so you need to get out of that world. Um, you know, GPD5, I think it's a one-stop shop, reasoning and non-reasoning. Um, and we've really tried to make it kind of just pareto optimal.
Yeah. Yeah. It's absolutely crazy to just take a bunch of letters and smash them together and expect people to pick up on that as a name or a brand. Chat GBT, TBPN, we're both kind of in the same insane gambit. But fortunately, it's worked out and I think people have have gotten over the hump. But it rolls off the tongue TV. Yeah, sort of. Except our friend David Sandra keeps flipping the letters. A lot of people do that. But at a certain point, yeah, you do break through and Chachi PT has, but uh but keeping the model numbers simpler uh m makes a ton of sense. Um uh talk to me about the pace of play for research to actual product like a lot of and and on that note the line between your personal philosophy on the line between research orgs and engineering product orgs.
Yeah. I mean so our research operates on a variety of different time scales, right? we have teams that they scope out a bunch of ideas and then they start to kind of narrow in on um the promising ideas as they get closer to a run. Um and then you kind of see a winnowing of ideas as you get closer to launching a flagship model, right? And um there's always this kind of like uh um explore more exploratory to more kind of concrete and execution focused pipeline. Um and we're pulling on ideas across the board here, right? There's a lot of work in architecture optimization. Seb was on stream. He pointed out improvements in synthetic data. So there's really a lot of work that goes into creating one of these models. And you know it's hard to say like oh this model was about this breakthrough just because right now we have this machine that's producing breakthroughs on all these axes and um even across several paradigms. Right. So it's all that coming together that produces the experience that you guys feel.
Yeah. Can you talk to me about um the legacy or future of 4.5? Um I remember I was talking to you and I was like I haven't been using it a lot and you looked at me like I was crazy. You were like ah it's so good and I was talking to Tyler and he was like uh our our intern here and he was saying like yeah the people who really like understand how good it is use it. Um but but I was I was wondering is there a world where that is a tool in the tool chest for GPT5 in the same way that Python is or web browser is? And if if it detects that I want something with more more emotional pros or more thoughtful writing, it can do a whole bunch of research, collect a bunch of raw raw text, and then kind of do a 4.5 pass that I believe is more expensive maybe, and maybe doesn't make sense for every single query, but um could be a a feature in the loop or a tool that is pulled into the overall product experience.
Yeah, absolutely. Um, speaking of 4.5, it's also a very smart model, right? Um, and one of our bars in creating GPD5 was to make sure that on a lot of the axes we cared about that it was able to outshine 4.5. And I I think even in some of the soft ones like creative writing, I I think that um was the case and and that's what makes us so confident with with the name. Um, I think we're able to really rely on all of the architecture advancements, all the kind of post-training advancements, all the synthetic data advancements to create a model that's better than 4.5, but much faster and much cheaper.
Yeah, it feels kind of like we're I remember, wasn't the second iPhone called the iPhone 3G, and the number literally corresponded to a specific technology? And now when you get the iPhone 14, it doesn't mean it's 14 megahertz or a gigahertz or inches big like it doesn't like the number is abstract and it speaks to a bucket of features and it feels like there's I mean this was the first day of kind of re-educating folks on what the nomenclature means going forward. Um, have you talked about an annual release schedule or like or or because there's the iPhone cadence and then there's the Google cadence which was like Google search just got better every year for two decades. Um, I it it it feels like at a certain point you want to just be shipping as fast as possible. How do you think about the culture of shipping updates that you know you find something that feels like hey that could make the customer more delighted or the user more delighted and we don't need to do a big training run for it so let's get that out today um and let's tell people about it like how are you thinking about fast iteration versus splashy announcements?
Right so on the product research side I think it makes a lot of sense to think about you know what's the cadence of release and you know uh what are the feature sets that we want build and I actually think there's enough great research happening there that we don't have to worry about oh you know is there going to be a drought or a long stretch without enough features to launch but one thing that's important for us is to be able to provide the people doing the exploratory work some buffer from that right it's hard to do really great exploratory research in an environment where you feel pressured to do release after release after release and so we let that be a little bit of a lazier pipeline not meaning that the work itself is lazy but we give it space really to mature uh and to flourish and you know once it's ready uh we can ship things across across that fence. So um that's kind of philosophically how we organize. We have a product research org uh still very much entrenched in the research and they care about the release cadence um and they're able to draw from all of the research that's happening um you know algorithmically and in scaling and in RL.
Yeah. Uh talk to me about tool use and how that's growing. I was I was kind of noodling on this idea that you know the I was I was thinking about the IMO and how uh it at least from the reporting it sounded like OpenAI's model didn't use tools for that and that's an incredible achievement but it's kind of like artificial like I don't I don't care if the model doesn't use tools I I use everything possible and uh even if even if an LLM can can memorize every fact I'm fine. fine with an LLM looking stuff up in a traditional database, spinning up a spreadsheet, like use whatever tool you want. Just give me the correct answer. Um, but do we have is it important to give to surface to the user the variety of tools that are in the GPT5 tool chest? I noticed something magical happened when I was using GPT uh I was using o3 Pro. I sent an image in and I asked to estimate the height of a desk and it wrote like a thousand lines of of Python image interpreter and was like, you know, interpreting pixels and I was like, I didn't even think to trigger Python. It did.
Yeah. Yeah. Yeah. No, he was right. It was crazy. But the the really funny thing was that it was just a standardized desk. It was just like it could have just Googled like how how tall is an average desk or something. Uh or just memorized it. It probably was just already in the weights that it knows that a desk is like 36 inches tall. But it it did a ton of work and it still got it right. It fact checked it a bunch of different ways. But but but but I've noticed that now I can I I I can pull different things. Make a table. Don't make a table. Write some Python for this. Don't write some Python. And it kind of gives me the feel of like a super user to some extent. Um but I'm wondering how you're thinking about what is further down you like you've given Chat GPT a computer as Ben Thompson said. you've you've given kind of the core tools the Python ripple the uh the the web browser um what how are you thinking about kind of the long tale of tools that you want to bring to bear and how does that interface I know that there's API integrations and all sorts of different surface area there but give me some context on that.
Yeah I mean our reasoning models are pretty cute right I mean I think they um you know when you look at their behavior right they they know the height of the desk but they'll still go verify it five different ways you know it's all consistent, give you that median answer. And um I think that's really what makes these models so powerful. And when you think about tool use generically, right, like we want the models to use that reasoning ability to just be able to like zero shot um a new tool, right? It you should be able to kind of minimally get instructions about how the tool works and just be able to know how to use it, right? And humans do this all the time. You get a new tool, you start experimenting with it, and then you don't need too much scaffolding and you just go and go and use it and understand it. So we want our reasoning models to use their reasoning to be able to use a broad selection of tools. And of course there are a couple that you really do care about. You know in in coding it's very important for you to be able to execute code. Um it's really important in personalization for you to be able to get context from your calendars and uh
From basically the digital world. So I think there's a range of tools we want familiarity with, but beyond that, we want the model to be smart enough to just generalize and use tools zero-shot.
Yeah. Talk to me more about personalization. I feel like there's a world where I'm maybe underutilizing ChatGPT as an app because I don't have it wired up to a non-relational database where it can just stuff data from. You know, it already has memory and it's doing kind of rollups and there's some sort of saving of context. But when we were talking to Kevin Wheel, I was kind of like, well, I don't really have a GitHub repo that's active that I want to dump code in regularly for my one-off tasks. But for that image generation, like understanding the height of the desk, it's like, well, if I'm doing that a lot, maybe I want to have a tool built that lives in the world that my chat interface can interact with on an ongoing basis and contribute to and modify and kind of wind up instantiating a piece of software that's even more long-lived and then every successive query is even faster. So yeah, how do you think about different ways to increase personalization?
Yeah, I mean I think memory is huge. So we have teams surrounding memory and also personality. And when you look at memory, I think it's just we have so much context built up about ourselves that the model doesn't have. And our memory team's been really hard at work. You know, there's a surface level of just gathering facts about you, but there's also stuff about just kind of thinking very deeply about who you are, what your motivations are. And even you could think about, you know, you're trying to do some code-based tasks, right? You're a developer. Shouldn't the model just be trying code out, you know, and just kind of leveraging all that memory, kind of its thoughts about what you want to do to just help you kind of be doing work all the time? So yeah, we do think memory is a huge part of making the model more personalized to you and it should just make use of all that passive signal about you that it observes or all of that interaction and just help you accomplish your goals.
Got it. What do you think it'll take for AI to start making novel discoveries? That's been a critique over the last year is everybody's so excited, everybody's using these products every day and in their work and life and yet it still feels like we're missing that. Dwarkesh has talked about potentially that being around continual learning but I'm curious what you think.
So one thing to underscore is I think the models are already phenomenally creative in certain ways. So when I've looked at our performance on contests, you know, I've done these contests before, sometimes you have this mental classification of these problems require more creativity or these ones require less and one of the big surprises for me was that the model can get some of the ones which I intuitively think require more creativity and you know it often does come up with these solutions that I consider quite ad hoc and really don't pattern match to anything I've seen before. When you look at advancing science or mathematics or a field like this, one thing that construct in which humans work sometimes is there are kind of theory builders in mathematics for instance, there are mathematicians whose role are to kind of build out this theory and almost to kind of create Olympiad-style sub-problems which often other mathematicians who are very good at that kind of style of work can do and I do think kind of the model will increasingly contribute on that side first, right? If there's some mechanical like, hey, I really don't know how to simplify this expression, I really don't know how to get this result, it can really do that quickly for you. We're trying to increase the envelope such that the model's getting towards that theory building side and you know being able to create creative hypotheses and all these components are very useful for what I consider the ultimate goal, which is being able to automate some of our own work and our own research.
How are you thinking about the layers of mixture of mixing? Like I remember GPT-4, I don't know if this was ever confirmed, but mixture of experts model. This is kind of widely understood in the industry. Now are we in the era of a mixture of models that have mixture of experts? Like how many mixtures are going on? How does GPT-5 actually work? Is there a taxonomy or architecture diagram that you can kind of walk through to explain what GPT-5 is because it feels so much different than GPT-3.
Yeah. I mean one of our probably the pinnacle of our research roadmap and our path to AGI, when you look at the levels of AGI, the top level is what we describe as organizational AI and what this means is collections of agents working together often like we might in a company towards a shared goal, right? And you would imagine that these agents probably subspecialize in ways maybe similar to what humans do maybe in their own more efficient ways. And I think you know effectively work together to accomplish some goal. So we very much care about exploring this vision seeing if that's much more effective than one single big brain working on a problem and I think there are reasons to think why it could be so and yeah I think that is one of the things that we're after.
Yeah, on that note of specialization, how are businesses working with GPT-5 or how do you expect them to work with GPT-5 in terms of coming to OpenAI and asking for special capabilities or fine-tuning or you know any sort of RL on this particular problem in my world. I have this specific data set. It's not public, but I want you to benchmark on it. I want you to get a 100% on the gas station bench or whatever. You know if I have a certain business and I'm willing to invest in some overfit RL because it will create immense economic value for my business or it'll solve some fundamental problem. How are businesses going to be using GPT-5 over the next few years?
No, that's a great question. So I think that this is a chance to kind of highlight one of the results that we've accomplished over the last couple weeks which is our ATCoder results. So this is a relatively unknown programming contest but it involves really the pinnacle of the best coding contest contestants in the world. And what they do is they're put in a room and they have to solve an optimization problem. This is something that's actually very real-world relevant. So you can imagine an optimization problem as something like what Uber might have. You have let's say riders and you have drivers and you want to kind of create a system where you match them as quickly as possible with the least amount of cost for instance. And so we've really created a system that can solve optimization problems at the level of the best in the world, right? And these truly are the kind of the best heuristic solvers in the world. And so we have an organization led by Alexander Madry, it's called Strategic Deployment and what they do is for a select handful of customers who really have that beefy problem that they need to solve to just go and provide that value, right? And I think there's a lot we can do there. I think there's a lot of very valuable optimization problems in the real world and we're really excited to partner with people because I think this creates a template for directly having AI provide economic value and really catapulting certain industries forward.
On the research side, what unique advantages do you think you and your team have given your position in the market with the incredible user adoption and the incredible usage from those users? It's not just DAUs but it's actually the number of queries, semi-analysis estimated at like 71% of all queries going through ChatGPT. What advantages does that confer from a research perspective?
Yeah, I mean a lot, right? And I think it allows us to kind of deeply understand use cases. It allows us to understand the frontier of where humans are kind of finding value, where they're not finding value, which areas that we need to improve the models on. It gives us a lot of signal into how users are deriving value, when they derive value.
And what is that signal? Like I see the thumbs up, thumbs down button. I'm sorry, I don't push it very often. I'm not doing my job apparently. But I know that you can figure out whether or not I'm satisfied. Just stop booing me, Jordy.
Okay, Mark. I promise you for the next 100 ChatGPT responses, I will be honest with my thumbs up, thumbs down just to help you do it.
We have tons of people luckily who do.
Oh, that's great. Okay, so you do get a lot of thumbs up, thumbs down. And I'm sure I have done it occasionally. But I also imagine that there's a ton of other signal in there. You know with the TikTok algorithm or any social algorithm it's very easy, time on site, but with ChatGPT obviously it's exciting when we hear okay 30 minutes a day or some rumored number of minutes, it feels correlated with usage, it feels correlated with value that's being delivered. You can obviously look at churn metrics and all that stuff but what other pockets of signal are you finding? Are you finding people just... I mean I remember the story about Google where they were trying to figure out how to handle misspellings and create the definitive database. Do you know this story where they were trying to develop the definitive database of how to spell things and they were taking a bunch of shots at it and they figured out that the best most rich source of data was just if you type in financial into Google and you misspell it oftentimes then you will just correct it yourself and the second query you send will be spelled correctly. So they can just look at two similar queries. What's the second one? That's the correct spelling. So yeah, what other pockets of signal are you finding that are translating into the research environment? What are you excited to go deeper on?
Yeah, so I'd love to first talk about the DAU signal because I think that's something that a lot of companies track, but we find actually a lot of danger in tracking it too closely. And one of the recent blog posts we pushed out was one on sycophancy, right? If you just, hey, we're going to boost responses where users say thumbs up, you know, it creates a condition for a model.
I just want to say, Mark, I love everything you're doing on this front. This entire interview has just been fantastic. You just... We'd love to have you back on the show tomorrow.
Yeah. Yeah. Clear problems, right? The model just starts kind of sucking up to you totally and saying like, hey, you're right. And even in complicated situations where I think objectively, collectively we'd be like, hey, this person's in the wrong. The model starts saying, hey, you're right. The other person's gaslighting you. This other person's kind of...
And people deal with this in the real world. They'll go to a friend, they'll tell them about a situation, and the friend will give them advice, but maybe it's not the fullness of the situation, right? Maybe they left out some key facts and the friend is like, oh yeah, that other person definitely is in the wrong. And they skipped over some important details.
Yeah. No, exactly. Exactly. And we don't want our models to fall into this trap where it's just trying to get you to like what it says. And so you know, we rolled back a lot of changes that produce that kind of behavior. And really the way I think about daily active users today is we need to be opinionated about the features that we build into the future. I think we have a lot of ideas here but we have to let that drive, you know, build for the future. Build for the things that you think they'll want and maybe don't necessarily know they want necessarily today. And then use DAU as kind of this byproduct, right? A way to track that you're on the right track here. So yeah, I mean we want to be careful here. We don't want to fall into these traps of like 3, 4 years from now that this turns into kind of engagement bait or something.
Yeah. How much time has the research team been focused on efficiency specifically? It felt like summer was a good window before kids come back to school and start maxing out queries, a good time to increase efficiency. And I know the cost of GPT-5... every time there's a new model I'm like this is the best it could ever be, it's good enough, bake it on an ASIC, I just want it for free and I want it like in milliseconds. But that's just me being grumpy I guess.
We've done a lot of work, we've been building out our teams. We focused a lot on scaling. I think Greg's going to come on a little bit later and he's been spearheading a lot of that work. So yeah, no, honestly, it's become a bigger and bigger focus for us, especially in the last couple of months.
On the... I mean this is somewhat related to the sycophancy thing, but I'm interested to know like what do you think is driving the GPT tone? You know how like the em dash is a thing and then it's not a newspaper, it's a way of life and there's these little flourishes like that that come through and in kind of a tell that it was written. And in a lot of ways I love it because when I get a deep research report I like that it's using the same Wikipedia style tone. Like I want consistency there. I don't want it to be like oh today it looks like it's a Vice News article and today it looks like it's written by someone at BuzzFeed. I like that it's consistent in many ways. But why is that happening? Do you think that bigger models like 4.5 kind of were able to solve that or do those kind of local minima like wells happen even in bigger models? Is there anything from a research perspective that can stop GPT having its own voice or is it fine that it has its own voice?
Yeah. That's a really great question and I think you know as you scale up models, as the models become more intelligent they kind of have a just deeper innate understanding of tone, right? And so you expect that to improve just naturally as you make the models more powerful, bigger, better reasoners. But one thing that I think gets lost a lot is each individual company has a lot of impact in terms of how they shape the default tone. And you know we publish a document called the spec, it kind of lays out how we expect the model to sound in certain cases. Lays out a lot of examples for that. And I think we use the spec in many ways, right? We have people come in and see, hey, was this thing generated in accordance with what we would hope to generate from our spec? And this is a living document, right? It evolves over time. And so I think, you know, each company kind of has a very opinionated take on what they think the model should sound like. And it's not an accident that the models sound a certain way. I don't think just naturally every company is going to train the same kind of voice into their model.
Totally. Well, thank you so much for hopping on. Congratulations on the big launch. We'd love to have you back soon to talk more. We could go in a million different directions, but we'll let you get back to it. We know it's a big day. So, have a great rest of your day.
It was a great conversation.
Talk to you soon. And we will tell you about Restream, one live stream, 30 plus destinations, multistream and reach your audience wherever they are. This stream is made possible by Restream. OpenAI just did a live stream. If you're trying with Restream, if you're trying to do a stream, you got to get on Restream, so it's everywhere. And we will bring in our next guest, Greg Brockman, the president of OpenAI. And we'll bring him in. Greg, how you doing?
Congratulations. How are you feeling? How's the company feeling? It's been such a wild journey. Just take me through a little bit of the vibes in the company and how you got here to today.
Well, I'm excited. The whole company's excited and honestly, I'm just so proud of the team. Like, it's just been amazing to watch people come together, not just for this launch. And you know, the funny thing is behind the scenes that people are always putting on the last minute adjustments and polish and scaling up the capacity and there's always something that goes wrong before launch day. There's a lot of people who worked late into the night or really crunched to bring this release to the world. And you know it's a little bit like the duck that's, you know, under the water. But that also describes the whole OpenAI history right? Is that I think that we have put in many years worth of investment to the techniques used to produce this model. And really it's across just every function within OpenAI that has come together to make this a reality.
Yeah, I mean, you've been there for every GPT release. How do you think about summing up each iteration in kind of like one line? Because GPT-1, GPT-2, GPT-3, these feel like similar architectures, at least history's kind of compressed them into similar architectures. But how do you think about the progression of just the big numbered releases?
Yeah, it's interesting because in some ways it's a punctuated equilibrium, but on the inside it looks very smooth, right? Even before the GPT series formally began, the first result that really sort of set this path to be something that we were heading down and that was clear that we were going to pursue it was the unsupervised sentiment neuron which was an LSTM in like 2017. So a different architecture from today's transformers and it was the first time that you could train a model to predict the next element. So, we predicted the next character on Amazon reviews and we were able to get semantics out, right? Because you expect, okay, yeah, it's going to learn where the commas go, what maybe what nouns and verbs are, but the idea that it would learn a state-of-the-art sentiment analysis classifier, that was mind-blowing. And so, I remember seeing that result in 2017 is like, we have to scale this up. We have to see where it goes. And so, GPT-1 was like, I think a good sign of life of you train on all the public data you can get and you use transformer and that you were able to get state-of-the-art on various downstream benchmarks, right? So you have a model, it clearly learned some representation, something useful about the data that it was shown and it's applicable. You can use it for various tasks, but we didn't really think very hard about the generation side. GPT-2 was the first time that we were like, all right, let's actually like the samples we're getting from it, the things it actually generates, they're kind of cool. And I remember reading in the GPT-2 blog post we have this unicorn story where it generates some fictional story about a herd of unicorns and it was just so cool. It was like wow it wrote a story that's actually kind of interesting. It doesn't totally make sense but there's something here. There's some real spark of intelligence within this model. GPT-3 was the first time that we had a model that was actually something people would... it was just barely above threshold for something people would want to use. And I remember working on the GPT-3 API. This was our first real product. And it was actually the hardest product, the hardest project in total I've ever worked on because it just felt like maybe no one wants to use this model. We don't really know what it's useful for. And it certainly was the case that GPT-3 was a great demo machine. You can make really awesome just like tweets and cool little apps and it would give you quick answers, but it didn't feel very reliable. And then GPT-4 was something that actually felt like it had true real-world utility. It was above some threshold. It was something that was helpful for health. It was something that was helpful for starting to be good at coding. And GPT-5 I think just sets a whole new standard for the reliability, for the utility. Things like coding I think are just clearly we're already on this trajectory of transforming software engineering this year. I think are really on trajectory now to be revolutionized. So just really exciting to see that whole arc.
Yeah. When did the API opportunity like really click for you? Because I do remember companies in that era that quickly unlocked the power of the API and grew tremendously. When did that opportunity click? Because you said initially that you kind of had some concerns, kind of doubts, how useful was it going to be and then when did the consumer opportunity click?
Well, we in 2019, end of 2019 had GPT-3. We knew we needed to build a product to be able to actually continue the mission to be able to raise capital. But what did we want to build? Right? We're really here because we believe in AGI that's going to have this powerful positive transformative effect on society. We want to be part of it. And so we thought, well, maybe we could build something in health. And then you realize, okay, well, we're going to sell to hospitals and we're going to maybe hire... let other people do that. Exactly. Right. It's just like you have to go into one domain and that means giving up on the general, right? It's like it feels like you're going to become one particular thing but we kind of want to be supporting all industries at once and so the idea was let's build an API and let people figure it out but this is totally not the way you're supposed to build a startup, right? You're supposed to have a problem, no one cares about the technology behind it, add value to that problem, focus on just that one thing. And so that's why that project was so hard and in January of 2020, February of 2020, I with the team were going around trying to just find anyone that would be willing to try this API and we were driving to different offices in San Francisco being like hey we have this cool model and it was hard enough to get people to take the meeting much less to sign up their company for it. It was actually very fortunate we found a couple of good partners. And it was fortunate that that happened then because March 2020 suddenly that was COVID, we weren't driving around to people's offices to try to beg them to use this budding new technology. So it was really six months worth of grind of really trying to turn... like when we started with GPT-3 I remember it was, you know, the inference code was not very well optimized. It was like I don't know 150 or maybe 250 milliseconds per token or something. And we just optimized, optimized, got it down to like 50 milliseconds per token which by the way today's models run much faster than that which is kind of amazing for me just like seeing how much faster we're able to run them with much greater intelligence. And I remember setting two goals for the team. One was actually find one customer who's willing to pay. So literally get a dollar in for this thing. And the second is get a use case that we use at OpenAI every day. That first one happened within the first couple months. So actually that moment I was like all right this thing is probably going to work. But in order to get there we had to do a bunch of just scaling the API and really doing the product work. But that second one took much longer, right? And that wasn't really until ChatGPT. And so if you fast forward a couple years because this was mid-2020 when we first got the API into the world, ChatGPT we didn't release until November of 2022. So you're talking like a decent period of two years there, a little bit longer. And I remember we were building, you know, people have talked about we were going to call it maybe Chat with GPT-3.5. We had a sort of precursor product called WebGPT that was built on 3.5 that we were literally paying contractors to use. Right? So this was all throughout 2022. We basically had the ChatGPT precursor that we had to pay people. They would not pay us. We had to pay them to use this thing. And that's wild. The moment for me that really clicked was actually when we finished training GPT-4. So that was August 8th of 2022 which actually is like three years ago now. It's actually pretty wild to realize that almost to the day. And we did the initial post-train of GPT-4 and honestly I had a bunch of bugs in there. It was like broken for a bunch of different reasons but the model was like extremely creative. It was actually really interesting. It took us like about a year and a half to get to the point that the creative writing of our models matched that initial one that was buggy for various reasons. And I remember, you know, we had an instruction following data set that was post-trained on. So it's really we had collected examples of here's a human asking for a thing, here's what the model should do. So it's really not trained to do multi-turn. So I asked it a question, it gave a response, but then I was like, well, what if we just ask another question? And it actually was able to leverage that full context. It actually was able to have a coherent chat. And the moment that we saw that we were like, okay, this thing is capable not just of being post-trained to do this very specific thing, but it can generalize, right? It can kind of do the intelligent thing even though it wasn't directly trained for it. It was just so clear this was going to be the killer application. And so then we were planning on launching GPT-4 in early 2023. And we had this chat infrastructure we've been working on and it's so clear okay we're going to have to release the infrastructure and the model and it's going to be this amazing killer product. And so just almost as infrastructure ahead of getting the real thing out, I was excited for us to do ChatGPT and that's why we did it then and see that come to life in November. So I think that for me I was really focused on GPT-4 as the model. This is going to be the chat moment that's really going to work and kind of missed the fact because every time you see these new models you just sort of see only flaws in the previous ones and so missed the fact that GPT-3.5 was something that no one had really tried before in the broad sense of society and that it was something that was already useful and that people would respond to.
Was GPT-3 kind of like the main pivot point for shifting the company towards LLMs? Because in the prehistory of OpenAI there were a lot of maybe expensive training runs. I don't know how much financial risk was taken with like the OpenAI Five project or the robotics projects, but it feels like at a certain point the chat became like the main financial risk vector. So I guess the question is like when it feels like GPT-3 was the moment when you shifted. I'm also interested in hearing about Ben Thompson called OpenAI the accidental consumer company and I'm wondering when that narrative set in for you, like when did it become clear that this was going to be a really really powerful consumer application?
Yeah. Going from paying people to use your product to people saying hey we want to give you money for this. Yeah. A very important transition it turns out. Yeah. So it's a great question. I would say that if you rewind to the beginning of OpenAI, you know there's many people who thought that in retrospect say that we set out to prove that scale is how you make progress in this field but it's almost the other way around. Scale was the thing that worked, right? That we tried a bunch of things that didn't pan out. And it really the first time we saw this concretely was in our Dota project. I remember my collaborators Jakub and Szymon trained the very first little agent on like 16 cores or something and left it running on their desktop over the weekend and we came back and it was this very constrained mini environment but the model was doing something smart. Was actually able to solve this kiting environment and that was pretty cool. And then they and the team just kept scaling up, right? That we had all these free cores that were just sitting idle on AWS at the time and they just kept throwing more compute at it and every time they would do that, the model would just get better. And so when you look at something like that, you're like, well, you just have to see where this goes. You have to push it until it hits the wall, right? And our goal with Dota was actually to develop new reinforcement learning algorithms because the common wisdom at the time was well the existing reinforcement learning algorithms, it doesn't scale. Everyone knows that. But the question from Jakub and Szymon was well why do we believe that? Has anyone actually tested it? And no one had really tested it. And so I think that that ethos of saying you have to push the existing techniques to the wall until they break. And then once they break you actually have a baseline to overcome and you win either way, right? Either it just exceeds all the humans in terms of the specific capability that you're trying to exercise which was the case for Dota or it hits a wall and now you have a real problem to solve. And so I think that ethos really got embedded in our DNA and you know at the same time I think that we were really thinking about how do we get to AGI, right? And really I spent a lot of time thinking about that question of where's this company going and how do we actually achieve it and you start to do some math in terms of the kind of compute that it would take to get to AGI and you just start to realize you're going to have to build really big computers and those are extremely expensive and so I think that from these early foundational results and thinking we kind of realize the path that we're going to have to walk.
So, it seems like there's been a few walls that we've scaled up through and then maybe hit them. There's been talk of like a pre-training wall. Now, we're putting tons of resources and compute towards reinforcement learning. Is there a third scaling curve that we're going to be talking about in the next few years? Are we continuing to scale up those two primary vectors? Is that too high level of an abstraction in terms of how we should be thinking about just progress along the vector of scale? Like give me the up-to-date thinking on just the fruits of scale.
Yeah, I'd say fundamentally deep learning. I think that you know people talk about the bitter lesson. It's almost this exploration into how do you convert compute into intelligence, right? Through some particular techniques to do that that we're kind of constantly fleshing out. And the thing that's really amazing is if you rewind to I don't know even the 1940s for the McCulloch-Pitts neuron which is kind of the precursor to neural nets. If you look at that paper, they have all these diagrams that actually look very similar to the kinds of diagrams we draw now of multi-layer neural nets and things like that. Like the basic idea of what we're trying to do has not really changed in almost 80 plus years, which is just a wild fact, right? It means there's something deeply fundamental about the thing that we are pursuing. And that idea itself, I think, kind of came from trying to model the information processing of the brain. And it's imperfect and not an exact analogy to biology and all these reasons that it should fail or that people have said this thing is doomed. But the results are undeniable at this point. I mean some people try but it's really hard to kind of close your eyes and sleep on this in my mind. And it's very interesting if you look at, you can find quotes from the mid-1960s of people trying to poo-poo the whole direction saying that these neural net people have no new ideas. They just want to build bigger computers. And you could basically say something very similar today. What we are trying to do... one moment.
You know, we're all human.
A proof of humanity right there.
Exactly. So what we're all trying to do is find novel ways of taking compute and really harnessing it. And sometimes you hit a wall. But these walls tend to be ones that you can drill through, right? What we found is every time you scale up everything, all of your engineering, all of your sort of scale and variance, all these things, they get stressed to the next level. It's almost that the tolerances become tighter and tighter. It's like launching a 10x bigger rocket means you need to be like 100x more precise on everything, but it doesn't mean that the fundamentals of the science are different. So pre-training, there's definitely been a lot of discussion of data wall. Doesn't mean it's fundamental, right? It just means that we need to be better and more precise at what we're doing. There's RL, which has been something that has kind of come from spending a small amount of compute to much larger amounts of compute now. And then there is a third way that we're really harnessing compute, which is compute at test time. And we published some scaling laws around this. And all three of these things multiply. Like that's the amazing thing. And of course the compute and the harnessing of it is the fundamental goal, but that you get these multiplicative effects out of all of it through the quality of your engineering implementation, right? Through the quality of the data sets, through a bunch of the refining work that you do and there's lots of different techniques and ideas and that's what makes this field so rich and why progress is just going to continue apace.
What about on the infrastructure side? You guys have been busy scaling up. What can you share on that front?
Well so I run a team called Scaling at OpenAI and we really focus on building the infrastructure for scaling and this is in partnership with really everyone across the company. It's almost a misnomer that our team is called Scaling because fundamentally this whole team and effort is about scale. But what we really try to do is to both on the physical infrastructure side deliver as much compute as humanly possible and that is in partnership with companies like Oracle, SoftBank and others that we've been able to deliver just increasing amounts of compute to OpenAI. But we're constantly thinking about how do we just deliver more flops and do it more efficiently, earlier, cheaper, more power efficient, all of those kinds of questions. There's the software infrastructure side as well and really thinking about how do you coordinate massive numbers of GPUs in order to work across one synchronous training run. How do you coordinate that for reinforcement learning? How do you deploy that into production and bring these models to life at massive scale? And I think that every single layer of the stack there is innovation required. And that's something that's very easy to miss. Like one way I think about research is that there is, and this is kind of the view from Jakub who's now our chief scientist, that there's a research stack and you can kind of think of the top of it is people running experiments and coming up with new ideas for how to sort of utilize data or something like that. There's a middle of the research stack of people thinking about how do you sort of take these different ways people are running experiments and be able to train in novel ways and kind of put together the pieces differently. And then there's a bottom of the research stack which is like writing CUDA kernels to get the absolute max out of the GPUs. And at every single layer here you get a multiplicative factor through innovation. So it all comes together as one big whole.
On scaling, I'm interested to hear about just if we think about the impact of AGI or the impact of AI just being some sort of maybe quantitative GDP metric or qualitative just impact and good, is there an important factor of scale with just not even the flops that are going into the models, into the pre-training, into the RL, into the test time inference, but actually just the flops that are going into the usage of AI within humanity broadly. And I feel like that might maybe be the next scaling curve that we're seeing as more people use models, they see improvements all over the place. Is that something that we should be tracking to see kind of instead of these S-curves, we want to see the continual exponential.
I think that's a great perspective, right? Because at the end of the day, I mean, if you look at kind of the shift from something like Dota, which we pursued in order to, you know, we wanted to do new algorithmic development, but really it almost validated how we scale up existing algorithms. But there was no illusion of delivering direct economic benefit from it, right? To the current models where we are still, we're starting to end the era of pushing on these academic benchmarks, right? You look at things like the IMO at this point. Models are able to get gold medal on it, like these the hardest academic benchmarks that are available are sort of no longer the guiding light of progress for these models. To where we actually want to be is for AI to be helping everyone, right? To be something that uplifts humanity. And that's the final metric, right? Is how much does it actually benefit everyone? How much value does it bring to the world?
Yeah. Not just health bench. It's actually how many people did you solve their healthcare problem.
Exactly. Yes. Yes. Yes. And that's the actual goal. And that's what's exciting, right? Is it's like we're moving from the lab to reality. And I remember in the early days as we were thinking about how do we measure our progress towards AGI. We always sort of dreamed that one day we would be able to measure it this way. And you can think of revenue maybe as a proxy metric for value delivered to the world. It's not perfect, but it's at least something, right? You can think of the distribution of how much compute goes into it, how many people are using it. But fundamentally what we're after is how much do we really uplift humanity through this technology.
Yeah. I mean, I might be misreading it, but I'm pretty sure that was the Kurzweil philosophy was that like total number of flops getting immense, not necessarily all in one data center for one model. It was that compute broadly would be so wide.
Yes. Yes. And I remember like on that chart you can see total compute of all human brains. Yeah. Which really suggests a particular vision of how this technology will be rolled out.
Yeah. Distributed, the phones count as an impact. The Wi-Fi router counts for the impact of the internet just like the phone does. Not just the big pipe that's going, the backbone of the internet that actually matters. Deep research hit product, almost everybody I know at least in the US says he's reading 30 pages of deep research a day basically, he loves it. He's making books with it. But why have agents broadly come around a little bit slower than people may have expected? Is it just that using computers is actually a much harder, computer use is just a really hard challenge or you know I think going into this year everybody said this was the year of agents, booking, you talking about flight booking, but people were saying 2025 is the year of agents and I would say that it's the year of deep research and not a lot of these other broader use cases.
Sure. Well, 2025 isn't quite over yet. So, that'd be my response. And I'm very much on the... I think that progress in this field, the way that it tends to work is that if something kind of works with the current generation of models, it will be extremely reliable with the next generation of models. And I think that where we've been is that deep research is the... if you rewound a year, that was the like we kind of had something working. And then this year, it's been just incredible. And I think that agents, specifically computer use agents are something we've kind of had working and again, the year is not over. I think there's a lot of rapid progress to be made. But I think that maybe part of it too is that the agents that we're about to see, I think are a little different from maybe what we would have pictured five years ago. Like I remember having a debate with some friends on do you want an agent that does the flight booking because the problem is it's actually a very high bar to beat the flight booking UI because there's so many preferences that are entailed in that, right? And you really have to know kind of what mood you're in, like are you okay with taking the extra layover and all these kinds of questions. And that actually there's so much other stuff that happens in your life that is toil or drudgery or that's something that you're not an expert in, you're supposed to be... think about health, right? That like every patient really is the doctor if you're coordinating across multiple specialists. There's no doctor that helps you with that, right? That's really on you and that there you actually can have AIs that are just text only that actually able to add massive value and then frees up your time if you want to go book the flights yourself. And so I think that really finding the right problems that have high leverage, right? That really add value to people and also thinking about the other side of how to make sure these agents are responsible with the trust that you put in them, right? That the more that you give an agent access to your email, the more you really have to trust that it's going to do right with whatever your task is and send the right email to the right people and be able to segment your information, all these kinds of questions. And so I think that there's both a practical how do you get to adoption but also just like where are the most important leverage points in a person's life.
You also missed coding agents because it's been the year of deep research but I feel like it's also been the year of coding agents. How is that developing at OpenAI? I've noticed that I'll hit o3 Pro and it'll wind up writing a bunch of code for me and I didn't even ask it to. Then you have specific products for coding. How do you see the evolution of software development evolve? How are you seeing OpenAI customers use coding tools and how good is ChatGPT or GPT-5 on coding?
Well, software engineering is definitely being revolutionized in front of our eyes. It's been happening and GPT-5 is the best coding model in the world right now. It's the default now in Cursor, which I think is a really huge statement of the quality of the model and that it's just so good across every function of writing code, understanding codebase, being able to use tons of tools, being able to do agentic work. That yeah, it's like I'm not a front-end developer at all but actually now I am, right? And I think that you are too, right? If you just talk to the model you can produce incredible things and so I think that there's this...
Real empowerment, if you think about what computers were supposed to be, right? Computers are supposed to be a more productive thing. But then somehow when we started out with computers, you have to contort the human to the machine, right? Writing assembly language and like all these very abnormal things for a human to do. And as we've moved to tools, ultimately, you know, in the current generation now, GPT-5, suddenly the computer comes closer to you, right? That you just express your intent and you don't think about exactly which language and what version of different libraries. The model is something you can delegate to. And so we are very committed to programming and to making our models continue to be the best they possibly can be.
Must a super intelligence be able to explain how to build super intelligence?
So it's a great question. I think that where we're going is a world, and we're already seeing it, where these models help us produce the next generation of models, right? They also help us really supervise tasks that are too hard for humans to supervise on our own, right? If the model writes a 10,000-line program for you, reviewing that is probably going to be quite burdensome. But if you can have a model that you trust that maybe isn't as capable as the one that wrote all that code, or maybe there's a team of agents that work together to write all that code, but you have a team of reviewer agents, this is the kind of thing that you can actually bootstrap trust. And I think that this is one of the most important things. And also interestingly, 2017 is when we had the first language results. We also had some results or some vision on how you can actually bootstrap supervision beyond the scale of tasks that humans are able to supervise directly. And so I think that we're heading to a world where we now have these chain of thought models. We've been advocating very strongly to preserve the integrity of the chain of thought. Right? So that means don't directly optimize it to look good, even though there will be lots of temptation to do it for various reasons. Really make sure that there's no pressure on the model to obfuscate its thoughts within that chain of thought, because then you can really see what it's up to. And I think there's further techniques to even make it more faithful and more rigid to what the internal monologue of the agent is. And so I think that there's actually a lot of promise in terms of interpretability, in terms of supervision, in terms of being able to scale to just much more sophisticated tasks.
Yeah. I guess my question is like there, how much information in the world can be derived from first principles reasoning versus true secrets that need to be discovered by interacting with the world directly? Because it feels like it would be very difficult to... I'm just wondering about like how intellectual property interfaces with super intelligence, or how if you play this out a lot, how there's all these hard-won... Doresh has talked a little bit about this with continual learning. There's all these little subtleties that maybe they're not secrets, maybe they're not true trade secrets. You don't think to lock them down, but they're just things that haven't been codified online or anywhere. They haven't been given to anything that is surfaceable by the model. And I'm wondering, is it just we need to build up new knowledge in every fact from first principles and kind of go through the history of humanity's pursuits of knowledge, or do we just need to onboard more and more information? Or maybe it's both. I don't know. It's just something I've been noodling on.
Yes, it's a great question. I would say all of the above. Select all star. So I think that the answer is very similar to what it is for humans, right? How does a human generate new knowledge? How do we accomplish new things? First, you want to be grounded in the wisdom of the past, right? You really want to understand what have people tried, what worked, what didn't work. You want to go and read the biographies of various people and understand those. But you also want to try things out, right? You want to make some mistakes in a contained environment in a way that you actually can see the effect of your hypothesis. And then you want to be able to learn from those. And I think that being able to really start to scale up these systems and be able to integrate them with the world is a very big process and milestone that we're currently embarking on, right? To move from a world of totally hermetically sealed reinforcement learning environments to thinking about how do you actually put real world interaction in there. And you think about things like robotics, like you're going to need to have that at some point, right? You're going to need to have some sort of interaction with the real world and to have models that are able to produce new materials, right? To be able to actually solve various diseases. For them to be able to really help people, right? That, you know, we already have models that are great at use cases like therapy. But to really get to the next level of something that can just really help every person accomplish more and accomplish whatever their goal is, it would be very helpful for that model to actually have some real world experience with doing that very thing. And so I think that figuring out how to bring all this together is ultimately what our mission is about. And we do this not in isolation, but really as part of a much broader community.
It seems like it's advantageous to have the most dominant consumer app in that environment. So, congratulations. Jordy, do you have a last question?
Last question. What do you hope to see out of Washington DC in the next year, year or two, not thinking super long term in terms of, you know, basically promoting innovation within the United States. Obviously, the admin cares a lot about AI and has been making moves, but what else would you like to see or where would you like them to double down?
Yeah, I've been very, very impressed with how much the administration has engaged with the technology and really tried to figure out how can we help and ensure that American AI continues to lead and really sets the standard for the world. And I think that that is the lens that I would really encourage thinking through, right? Is like this technology is changing very fast and that fast plus government is not usually an ideal combination, but this is the reality that we have. It's the opportunity we have and I think that the question in my mind is less about any specific regulation or strategy, but it's really being calibrated. It's really having a very tight feedback loop, right? Being able to react to, okay, we have a new model. These are the capabilities we see on the horizon. How do we make sure that we get the most uplift and benefit from it and thinking strategically about not just how do we do this for Americans, right, but how do we actually do this for the world and promote democratic values? And so to me, the most important thing is that motivation, right, is the question that is asked and the ultimate sort of motivation behind what gets implemented.
Yeah, that makes a ton of sense. Thank you so much for joining us. Jordy, are you gonna hit the gong for GPT-5 and the whole... congratulations on the massive historic day and thank you so much for stopping by. We'll talk to you soon.
Bye. Cheers.
Really quickly, let me tell you about figma.com. Think bigger, build faster. Figma helps design and development teams build great products together. And we are joined by Sarah Friar, the CFO of OpenAI next. And we are going to bring her in in just a minute.
Still swinging. The gong's still swinging. And I'm going to tell you about Vanta.com. Automate compliance, manage risk, improve trust continuously. Vanta's trust management platform takes the manual work out of your security and compliance process and replaces it with continuous automation. Whether you're pursuing your first framework or managing a complex program, we need one more second. Tyler, any other questions that we should be asking for the OpenAI folks? Anything top of mind?
What's on the timeline? Is the timeline still in turmoil or has it settled?
So, I think the general vibe is like this model was not benchmaxed, but if you actually get to use it, it's pretty solid.
One thing, it failed QPN bench.
It did not get the horse breed. Correct.
Get the horse breed. Wait, so you have it? You have access to...
Yes, I have access. But I've seen other things on the timeline. We can talk about it later, but it seems like a really good model.
That's amazing. Great to hear. Well, welcome to the stream, Sarah. Good to meet you. How are you doing? Congratulations. A historic day. Thanks so much for taking the time to talk to us. How you doing?
I'm doing great. I mean, how could you not be doing great on the day when GPT-5 launches? It's been a long time in the making and we're so happy it's out.
Yeah, fantastic. Walk me through your role and what GPT-5, what this launch means specifically for you. And yeah, well, let's just start there.
Finance has to be, you guys have to be the unsung heroes at OpenAI. There's just a lot of big numbers, bills coming in for crazy training runs and you have to underwrite these against future revenues and I'm sure you've developed many models to figure that out. But yeah, walk me through what your role at OpenAI and what today means for you.
Yeah, absolutely. So I'm OpenAI CFO, but the finance can be the unsung heroes, but they are an amazing team so I'm going to shout out to them.
They're heroes to us. They're heroes to us. It's a complex world that we're all living in and there are a lot of B's on the end of a lot of the numbers that we look at. Look, what is our role?
Number one is just making sure we have a healthy, high growth business. It's been incredible watching just first of all the number of weekly actives. 700 million people using ChatGPT every week and I'm assuming after today we should see a very nice little bump in that number.
This is going to be a gong-heavy segment, Jordy. I think we're going to have a lot of soundboard for the big number. So congratulations.
I love it. And I've never met a number I didn't like. I think the other part of the business that you know, and then we have to do... is clapping. We have to do this balance of the consumer business, the enterprise business and then API business which I think of as somewhat enterprise. And balancing that out. So enterprise adoption has also been exploding. I probably do, I mean interestingly as a CFO I probably meet four to five customers a week. It's a part of my job I actually love. We have about 5 million paying business users right now from banks to biotech. I was talking to the CFO.
And so that number is individual companies?
That is individual seats at companies.
Seats at companies. Got it.
So what I would say about that number is it's crazy to have done that in just two and a half years because enterprises, right, you got to put your big boy, big girl pants on to go sell to an enterprise, right? They want to make sure that you have the table stakes of security, SSO for signing on, you have HIPAA compliance if you're selling to healthcare and so on. They want to know that other people have done it. So they're often looking for that case study. But they also want to be, you know, the innovator right at the front. And so that to grow that scale of business in just two and a half years blows my mind. And it's not just big, big businesses which I could talk at length on, but it's also small mom and pop, you know, literally the people who really keep the lights on in most countries are also gravitating to ChatGPT which is wonderful. And then on the developer side, four million developers have built in our platform and the question there is like that could be a developer inside a big company like Grab, it also could be the next startup founder that's Y Combinator getting going with the next multi-billion dollar unicorn business. And so we see the whole gamut there and that's important to us as well because very mission aligned, right? How are we going to get AGI to all of humanity if we don't do it through this ecosystem? So a big part of my, you asked my role, big part of my role is just keeping that business really healthy, making sure we always have the headlights on so people know the decisions they're making from a business standpoint. Huge part of what the team does.
The other big part of my role, compute. If I didn't talk about that in my first breath, you all should correct me.
I mean, it's making sure we think compute is a massive competitive differentiator.
I give so much kudos to Sam and the team, but particularly Sam because no matter how big a number we look at, Sam always wants to go bigger and he's been right. It is...
He's never met a number he doesn't want to add a zero to.
That too, maybe more, maybe logarithmic, maybe two zeros. And he's, but he has been very right and if you know, you've just had a long conversation with Greg Brockman, I think he does such a good job of kind of really explaining what a completely different world an AGI-ed world is or an AI world is. And so I think when people get all caught around the axle of like, you know, what is a gigawatt of compute and oh my god you guys want to have 10 gigawatts and that's more than the compute of like Ireland since I grew up there, and now you kind of look back on that and you're like those numbers already look small for a world where everyone will have access to intelligence and we're really starting to see what that can mean when you look at the demos today around things like healthcare and education and so on.
Can you talk to me about non-GAAP metrics and what you think is going to be useful to track? We were talking to Mark Chen about this and he was saying, you know, DAUs are great, time on site is great, but that's not as impactful of a metric for OpenAI as it is necessarily for a social network or an entertainment app. And there can actually be some problems that come up with that. So, it feels like there might be some tension in the organization eventually or just publicly about what metrics are worth optimizing for. And then there's also the financial community that wants non-GAAP metrics to track the health and progress of the business. And then of course over decades we see companies eventually roll back some of those non-GAAP metrics as the business gets more complex. So how do you think about the development and sharing of non-GAAP metrics and what do you think is actually interesting and provides signal to the business and the investor community?
I'm kind of smiling to myself because when anyone normally says talk to me about non-GAAP metrics, I can see like most of people's eyes roll back in their head. I live for non-GAAP metrics. I would love to do that. Look, I think in a CFO seat, first of all, it's really important to think about input metrics and output metrics and things like revenue, which is a GAAP metric as well as a non-GAAP metric, they're very laggy. Like if you are spending your whole time focusing on the revenue number in an operator seat, you are completely missing what's going on with the business. So I push my team a lot to get out of kind of ultimately what the P&L looks like and I'll come back to it though and go way upstream and say what are the true input metrics that tell us about the health of our business. And so I think it does start with that funnel of monthly actives to weekly actives to daily actives because we do, I mean our mission is literally AGI for the benefit of humanity. So we know how many billions of people live on the planet. The fact that we're starting to be able to talk in billions and percentage of the world's population, it blows my mind. Right? Today 85% of our users are outside the United States. And I love that stat. And in fact, if you go look at where are the big populations of users, it just tracks global population, right? It's countries like India, Indonesia, Brazil, Vietnam, like the Philippines, like go to anywhere that has big population. The US too of course, but that will be your tracker. So that's kind of number one when I think of an input metric. From there on the consumer side, you're right. Things like time in app, I've actually always had somewhat of a love-hate affair with, but I think in this case because we're giving people intelligence, teaching them how to use that, I actually think is where time in app does become important and one of the things we've really seen with ChatGPT are people are spending more time with it now. You know, we balance that with things like mental health and so on, making sure that we're not creating bad things like we might have seen in prior eras of computing, but I think we're just getting started on that front. Beyond that, like when we go into areas like the API, I don't look only at usage, right? I can look at tokens per minute as a usage metric, but I look at things like latency. I actually try to look at the elasticity of demand, right? We know that developers want performance, they want intelligence, but they also want to make sure the API is always up, and they want price. And they're often willing to trade across those three things, right? It's a kind of a linear program depending on what your use case is. And so I think it's important that we are offering things to developers that allow them to optimize across those three metrics. For example, so that's kind of your input metrics. And again, I could wax lyrical, but I won't. But then you go to what you really ask. So investors on the other side, right? They want to see a P&L. They're like, I want to be able to compare you to other companies. I want to be able to create a maybe a DCF. Like I want to think about fundamental valuation for a company if I'm going to invest in it. And so, you know, today what I really try to push investors on is we are not a company that should be optimizing for free cash flow today because there's just too much opportunity. Like that point about compute. We have to make a decision on compute today with an eye to what we're going to need in two to three years because data centers don't just spring up overnight. Like they're not mushrooms. They literally take time and effort. The thing we have at, frankly I would say is three years ago we didn't have enough foresight to say how big could ChatGPT because it didn't exist. It's just a shame on us if we keep doing that over and over. So there can be a bit of a mismatch between our belief on revenue because we don't yet know the product versus the input which is the cost today on compute. And so getting investors comfortable with the fact that there's probably losses for a period of time. I say probably because ChatGPT just generally the revenue models continue to surprise to the upside but at least for now we should be in big investment mode. And then you kind of said it well like as companies mature you move to more GAAP metrics, right? If you look at the large, the Mag 7, in many cases they're looking at like real GAAP net income. So the whole way down to the bottom of the P&L, we're just not there yet and we should take advantage of that advantage because we can invest as a private company.
How do you think about timing fundraisers? From my understanding or rumors, the last, you know, the most recent financing was very oversubscribed and at the same time you're still committing to capex in the future that is a multiple of current, you know, the current run rate. And so you in the CFO seat, I'm sure you're trying to find this balance of like what does the business need today while not diluting the company too much, knowing the growth rate of the business.
I mean that's exactly right, that's the art, not the science of it, is that, you know, we did just come off the back of closing out the sleeve of investment that we could take down in this current round led by SoftBank and it was massively oversubscribed, which comes back to I think the market really waking up to the fact that AI is a generational opportunity and the scale that it requires is like something people have not even seen before, right? It's, you know, people talk about the internet or like the railways. They're good analogies or transistors. I think Sam always goes back to they're good analogies, but I do think this is bigger than everything that's come before. So there's a, you know, taking down $40 billion, which we just did in this round, that certainly felt like that gave me a lot of confidence. Appreciate that. A lot of confidence to then go out and do large compute deals, right? We announced the large deal with Oracle for example and to be able to keep working with all of our supply chain Microsoft, CoreWeave, Oracle, Nvidia and so on. But at the same time, you know, in a world where our valuation has gone up, you know, at pace with our revenue, you do get an opportunity to keep coming back to market and not take that same dilution because you're getting that higher valuation for the work and the output that you've created. So it is a bit more of an art than a true science. I think for now we will continue to need to fundraise in order to fund that compute but I think we want to start getting more sophisticated like just pure equity fundraising for everything is an expensive way to fundraise and I think we're probably getting to the stage at a company where we can be a little bit more kind of broad in how we think about funding overall and even just working frankly with our supply chain because you know our success with bringing this era of AI into being is their success too. And I think these companies are realizing that.
What about partner? Last question. Partner selection on the compute front. There's not a lot of companies in the world or firms that can really be a...
You should update your LinkedIn title. We saw someone yesterday works for Discord is in charge of their cloud buying and his LinkedIn title was I have full responsibility over buying our entire cloud budget. And it was clearly like a huge flag, but I'm sure you're in direct message with every single person that's relevant in the industry. But yeah, but I'm curious around like, you know, a lot of people have been excited about developing data centers over the last couple years in hopes to win...
comp, you know, the business of companies like OpenAI, but I think when you guys are evaluating partners, I imagine that scale is such a massive factor. And so a single small data center is not really going to move the needle. You guys need to be thinking in terms of mega projects.
Yeah, I mean I think that's exactly right. I mean it started with our partnership with Microsoft and it's kind of, it makes me smile now to go back and look at that original kind of large fabric for pre-training because I think it was only in the maybe 20 megawatt sort of size. And you know now we're talking gigawatts even just this year. And you're right that when we think about like what is perfect compute for us or strategically the right compute for us we are definitely thinking about large scale, we're thinking about flexibility, right? We're learning a lot about pre-training, post-training, test compute, even like where the different kind of scaling is happening. We're kind of recognizing there's more of a blurred line often between what people think of as inference, investors always are like your inference compute and your training compute, it's like you know literally it's vanilla ice cream and chocolate ice cream when in reality there's like a bit in the middle that is something of both. We also need to think about things like where, you know, latency, where do we want to put our footprints around the world, that very global weekly active user base, right? As they use ChatGPT, you don't want to slow the model down, right? The beauty of the intelligence is like the real-time nature of it. And then when we get into big compute like where there's lots of tokens being used like deep research, image gen, video as that comes online, like all the work you saw today actually just even on voice, like that really quickly means that you got to make sure your compute is near your users. And so it is a big plan that's coming together but you're right, like small is just not that useful to us. But...
What about pushing partners to take risks? From my understanding, you guys are pre-committing to certain, you know, basically spend levels, but at the same time, I imagine you want people to say, 'Here's what we know we're going to need, but we want you to build, you know, this much capacity so that we have the sort of incremental capacity built in.'
Yeah, we want, I mean being extensible is really important. And we do want to see partners like I think Oracle OCI has done a really nice job of that of kind of starting, we started with like one large, it felt really large at the time, data center footprint in Abilene in Texas and now that has really multiplied up into multiple sites that can all be connected and that's a good example of a partner who has the capability to start in one way but to be able to show you a path to maybe 5x-ing just in that single footprint. That said, we are finding that as we go around the world, there is an ability to go work with governments. For example, we just made an announcement in Norway, made an announcement in the UK. This is the first time in my professional career I've seen countries come to the table and want to do commercial deals like wall-to-wall ChatGPT. I think the government of Estonia put ChatGPT into all of their high schools or I can't remember, was it university level. But that's kind of wowing. And hand in hand with that they are viewing AI infrastructure as incredibly strategic for their population. And you know it's a whole other level of selling versus, you know, I've seen enterprise, large enterprises before but never anything at this scale.
Last question. Whose idea was it to give every federal agency ChatGPT for a dollar a year?
I imagine that had to... you could have gotten more than a dollar. The CFO must be really upset here. $10, that's 10 times as much money. Now...
This is one where I think it's really important. OpenAI is, you know, in some ways a US asset, a national asset, and we want to make sure we're accelerating our government, like all of the resources as we think about, you know, western democracy and so on that we are absolutely putting our technology into those hands.
It's that guy Kevin Wheel, he's been moonlighting for the US government. It's like, which team are you playing for Kevin? Are you on OpenAI or are you on the US government?
Kevin just did his basic training. I don't know if I'm allowed to tell you that, but I was hearing all about it yesterday.
I saw some photos. They look great.
Yeah, it's a good thing. It's even better for Kevin. Yeah, it's great.
Last question for me. I, you know, the open source model launched two days ago and there's this world where like you have this dominant, the accidental consumer company, you have this dominant consumer app that's generating so much revenue. Then you have B2B and enterprise and API and that looks more like a cloud provider. But then is there a world where the Red Hat Linux of open-source LLMs is an OpenAI division and that there's actually serious revenue and profit that comes from helping companies implement an open-source large language model? Like Red Hat built a pretty fantastic business for a long time on top of open source Linux implementations. So like...
Yeah, I mean I think it's the right question to be asking. I mean I think step one was getting, yeah, you got to get it out, two days ago our second open source model out, and getting seeing what that traction is and then seeing what the community needs. I think it's important to leave space for a community to develop, right? That is the beauty of open source is that ecosystem that develops and that was true with Linux. It's true in areas like crypto too. But I do think you'll find over time that as enterprises want to deploy it, like I now date myself, but when I was a research analyst at Goldman Sachs back in the day, I covered software and I covered Red Hat actually. Oh, that growth. I wrote a research report called 'Fear the Penguin' at one point because seeing Linux being deployed, but then you started to understand that for an enterprise you couldn't depend on like patching and upgrading to happen via community model like you needed some of the rigor that goes with an enterprise business where you kind of know when you need maintenance, if you need a bug patch and so on. And so that did allow Red Hat to grow an incredible business. So I don't know if it's us or we'd be supportive of others, but I think we are so excited to see open source out there and getting incredible feedback and I think we want to do that ahead of GPT-5 to keep coming back to like we're here to grow this ecosystem.
Well, we'll give you market cap credit for it anyway even if it's early stage. Well, thank you so much for coming on. This is fantastic. We'll talk to you soon.
Thank you, sir. Great to see you both. Take care. Have a good one. Cheers. Bye.
Up next, we have DD Credo from Kudo. I believe I'm pronouncing that correctly. Let me tell you about Graphite code review for the Age of AI. Graphite helps teams on GitHub ship higher quality software faster. You can get started for free at graphite.dev. And let's bring in our next guest. How are you doing? Welcome to the stream.
Welcome.
Ooh, very clean background. I know it's probably virtual, but whatever you got going on looks fantastic. You look great. How are you doing? Are you excited about GPT-5?
Oh, I'm so excited. It's awesome.
It's actually like everybody's talking about the coding capabilities, please. But no one is really talking about the code review capabilities and I'm going to talk about that today.
Yeah. Yeah. Break it down. How are you using it right now?
Yeah. So we just enabled it in our platform. It's the default model for both our IDE plugin, our CLI, our Git plugin. And yeah, we're using it to generate very high quality code reviews, catch bugs before they hit production, help enterprises verify that their code is aligned with their best practices.
Mhm. See, it's super exciting. I can share my screen and show a few things if that makes sense.
You can, everything you share will be live. It'll be a little bit please. But I want to know also while you're getting that set up, I want to know about what changes materially do you think happened in GPT-5 specifically for code and code review? Do you think there's more data going into the model, more data going into the pre-training, post-training? Anything else? Anything that you're noticing that you're like, 'Oh, there's a specific upgrade here. They must have done something to get there.'
Yeah. Yeah. I think it's a great point. So, I think it's all of the above. So it's scaling of both the pre-training but probably a lot of the reinforcement learning and basically using that at scale to verify that code gets generated in high quality and then also basically catching bugs. And when you do it with reinforcement learning you have the actual ground truth. So once you scale that you can get the model to be a lot better at that.
How steep is the power law right now in just programming languages? Is it basically all Python, JavaScript and then a really hard fall-off or is it actually important for coding models if they want to be adopted widely to be like truly multi-language and get all the way down into the long tail of like the Rust and the C and all the different languages that are out there.
Yeah. Yeah, for sure. It's important to, I mean the majority of the market is in the JavaScript, TypeScript, Python, like the majority of the early adopters I would say. But then when you get to enterprise use cases you get a lot of Java and the models are getting pretty good at those languages as well. For sure.
Are you excited about, I mean how do you think about the difference between like the improvements to GPT-5 from the consumer's perspective versus at the API level? I always found it a little confusing that ChatGPT was available as an API and you could interface with the chat, I believe you could interface with the ChatGPT model via the API. And there's a little bit of like a line blurring there, but are there features that you think are cruft and you want to kind of rip out for an API use case or do you just say, 'Hey, give us the kitchen sink and we'll work from there and it's actually helpful to have, you know, a coding model that can still have a web browser.'
Yeah. Yeah. I think basically it's a lot about we consume the model through the API and it's really the same model that drives the consumer product. But for us since our use cases are a lot about agentic use cases, the more the model gets better at using tools and gets better at kind of listening to very, very specific instructions, following instructions is critical for the enterprise use cases. Because for us unlike the broader market we believe that for enterprises you need to have very specific agents that are defined with specific set of instructions, prompts and tools and permissions. And the more the models get trained with that type of environment, the better they end up serving the enterprise market, which is really where we're focused on.
My question is, I wonder like you said like very specific instructions are important. When are we going to get an agent that I can just turn loose in a codebase and say like just go improve it? Like just go hunt around, do like rewrite that, like when you get a good open-source contributor on a team that just becomes nerd sniped by the project that you're building on. They will just go around and find little ways to improve this, documentation needs to be a little better. Let's rewrite this test case over here. Let's add a little bit more functionality to this class or function. How far are we from that?
Yeah, I think the models are getting better and better at that part of basically kind of running loose in a codebase. Yeah. But they do need the guardrails in place and this is kind of where we're focused on. Like a lot of the talk in the market is around the code generation side. You know, let the agent loose and give it a task and it will just going to go around and run for hours and do it. What we're seeing is that the real challenge is now shifting towards how do I verify that the code is aligned with the best practices? How do I make sure that it's well tested, well reviewed, doesn't break anything. So that's I think the next frontier and really developers going forward are not going to write a lot of the code by hand. They're going to spend most of their time reviewing code and that's the next frontier and that's what we're really here to tackle.
Very cool. Anything else Jordy?
Well, thank you so much for joining, giving us some extra context on the GPT-5 launch. We will talk to you soon. Have a great rest of your day and thank you for joining.
Thanks. Cheers.
Talk to you soon. And let me tell you about Profound. Get your brand mentioned on ChatGPT. That seems more relevant than ever. Reach millions of consumers who are using AI to discover new products and brands. I forgot to ask about this. We'll have to come back to this but I want to know if...
the Founders, Powers, MongoDB, Indeed, Mercury, DocuSign, Zapier, Ramp, Ro, Gong, Workable, Majury, Asleep, US Bank, Chime, Clay...
Okay, okay, okay, we get it. They got some logos. There is this question of like, okay, even if you're like, okay GPT-5 is more incremental, more of an evolution than a revolution, it's like, okay well then let's talk about how it affects every other business and every other aspect of the economy, what should you be focusing on? And is like, do any of the updates from GPT-4 to GPT-5 change how you're positioning your brand for AI search? That's certainly an interesting question to dig into. Anyway, we have Zach Lloyd from Warp coming into the studio. Welcome to the stream for the second time. Welcome back.
Uh, you know, yeah. So, I mean, a lot of what stuck out to me, I'm mostly a consumer of consumer AI apps. I'm very excited about not needing to mess around with a model picker anymore. But take us through the biggest improvements from the software development side.
Yeah, I mean, it's a major step up from the prior OpenAI models. It's, I mean, it's doing agentic workflows in Warp for a much longer period. It's just a smarter general model. Like we eval it against all of our benchmarks and it's up there at state-of-the-art which is, you know, from our perspective it's awesome to have multiple competitive models that our users can benefit from. So definitely a huge improvement from GPT-4.1.
Yeah. So it seems like not the, you know, Claude Code killer, but certainly in the same conversation, in the same football stadium if we're using a sports metaphor. How much, you know, one thing that stood out is the cost reduction. How about do you think that developers will care about that versus just, you know, what it can do from an output standpoint?
I think developers do care about value. So sort of like quality to cost ratio. I think it's the more you get into like the individual developer and the small team, the more that that matters. Whereas if you're at the enterprise level, I feel like it's a little bit less price sensitive. So yeah, I mean you can see it as different apps change their pricing what the reaction of the developers is. You've probably seen this with Cursor and seen this with Claude Code and so developers really, really are looking for something that's cost effective. So the fact that the cost is a little bit lower is actually a big deal.
Do you think we're in the Lyft-Uber 2015 arc where the prices are subsidized and the prices will go up? Do you think that there's a price war on the horizon now that the frontier models seem to be similar capabilities? Do you think that someone will try and raise a bunch of money, cut prices a bunch, and steal a bunch of users? Like, how do you think that plays out?
It's an awesome question. I mean, my hope is that we get to a world where there is price competition at the model layer. So, Warp is very much at the app layer, right? And so, our value prop is like we can give our users who are mostly developers the best model access. And so to the extent that it's not one sort of model provider running away with that and having pricing power, it's better for us just candidly. And so, you know, my hope would be something like the model world ends up a little bit like G-Cloud, AWS, Azure. That's our best end state where all of these models are, you know, sort of similarly powerful and a little bit more commoditized. I don't think it's been like that, but it's getting a little bit more like that. And so, the more that there's more than one show in town, I think that's generally good for Warp and actually is good for developers because it will put competition, the competition will put pressure to bring the prices down. But I don't know, like I also think that people will definitely pay for quality. And so if there is a meaningful delta in quality on the frontier models then I think that like whoever has the quality delta will have a lead temporarily but I'm not sure that that lead will be sustainable. We'll see.
How do you think the developer community should plan around model deprecation over the next, you know, one to two years? Like how much, you know, from, I don't know that I've gotten a reaction yet from, I don't know if there's general frustration yet from people. You know, we've heard on the consumer side Tyler on our team here loves 4.5 and so he was a little bit disappointed to hear that. But what are you seeing on the developer side?
Yeah, I think it's a little bit different for people who are like building apps on LLMs versus people who are using LLMs as like an accelerator to doing coding. And like, you know, at Warp actually we do both like we're an application level stack and like it's actually very easy for us to go to the latest model and so it doesn't really bother me. I don't know. I don't know what type of app you would be building where it's like it's really important that it's like GPT-3.5 or GPT-4 or something like that. I think like generally we want the most intelligent tokens at the best cost. So I don't see that being like too big of an issue honestly.
What about open source? Does that feel like something that will be in the playbook? Is the markup on closed source models high enough that there will be a significant price delta and or is the frontier kind of indifferent to closed source, open source?
So if there was a comparable open-source option that would be awesome. I think that the economics of it again, it doesn't seem like a perfect analogy to me between like open source software and open source models. So open source software it's like you have a big community of people who, you know, for the love of coding are building a really awesome product. For open source models it's like you just need a crazy amount of capital to train something that's on the frontier and so I don't know how that happens. And so what we've seen is like the open-source models are competitive at the quality level that they're at but the quality level that they're at is not the same as the frontier models and I don't really see why that would change. And so I don't know, in Warp it's like we were serving some open source models, but they're just not as good. And so there's I think a more limited use case for them right now. And I don't really see economically why that would change. In fact, I would be surprised if anyone was spending billions of dollars to train a model and just kind of put out the weights. Like I don't get the business strategy there but maybe that will happen, that would be awesome.
Is there a world where you're like this idea of like smarter models either orchestrating dumber, cheaper models or like using or distilling models into more narrow formulations that can be run more efficiently. We've talked to a few companies that do this for businesses. Like you just want a model that just filters for profanity and you can run it on, you know, a gaming graphics card. And so it's basically super, super cheap or super fast. I'm wondering about like in the coding world, coding agent world, any of that, like where are the opportunities to kind of fan out and use an ensemble of models instead of just hit everything with the smartest, best? It feels like because of the funding environment, everyone can kind of justify like a high cloud bill, but and most people don't admit that it's hurting the bottom line, but it feels like at some point it kind of has to eventually.
I mean, I think that's a very real thing. Like even in Warp we don't use the biggest, most powerful model for every task. And so there's certain things like...
For Warp, deciding whether or not we should summarize a conversation is a good example. You hit the context window and you're like, okay, is this a good spot to summarize? Is this a good spot to encourage a user to start a new conversation? We use a much more inexpensive and also low latency model. The other trend is that these very powerful models tend to have much higher latency, and so we do a mixture of models, and that's totally a real thing. But I think the predominant use case as a developer is going to be: I want to tell an agent to do something. I want it to be harder and harder. I want it to run for longer and longer. And to do that, you kind of want in general the most intelligent model. And so, until the models have a sort of S-curve type shape, I think it's going to be more of a quality game than a cost game for most of these things.
Doesn't it feel like they have an S-curve shape right now?
Certainly does from a consumer perspective.
That's interesting. From a coding perspective, I feel like we're still accelerating. The difference again between the last version of GPT and this version of GPT is probably bigger than the difference between 4.1 and 4, and 4 and 3.5. It's a big deal, and same thing with the Anthropic models, and I'm sure that we'll see something from Google where it's an acceleration. And I think that there is maybe an underappreciation of how much left there is to solve here. Because even when you're doing a real coding task as a pro, despite all the demos you see on Twitter where someone asks an agent to build an app, that's a lower level of difficulty than doing what a pro developer does with one of these models. And the models still don't produce great code a lot of the time. There's a lot of kind of handholding that has to go into it. And I think that we are still seeing an acceleration in terms of the models actually becoming not just okay competent engineers but really, really good engineers.
Yeah. Do you care about benchmarks?
We care a ton about benchmarks.
But your own internal benchmarks or...?
We do both. So, plug for Warp, we're number one on TerminalBench, which is the public terminal benchmark, and we're top five on SWE-bench, which is the coding benchmark. And then the only way in my opinion that an app at our layer in the stack can really improve is by measuring the progress. And so we have our own internal set of evals that we run across all these models as well, which are coming from real use cases. And that again is an advantage of being a product that's in the wild that has a lot of users, is that we can sort of see where the models are failing, where they're working, and so we're very big on that actually.
Awesome. Well, thank you so much for stopping by. We will talk to you soon.
Sure. You'll have a busy afternoon.
Shout out by the way to the OpenAI team, very, very helpful in working with us to get GPT-5 to be awesome in Warp. And one more shameless plug: we have a discount code for people who want to try GPT-5 in Warp. It's $5 GPT-5.
Okay. Thank you for having me, guys.
Yeah. We'll talk to you soon. Thanks. Cheers.
Tyler, any updates from the timeline while you're thinking about what the latest vibe check is in the war between OpenAI's...
Choice for OpenAI. You have something?
From Reggie James, front of the show: 'Half of my timeline says this is the closest we've been to AGI. The other half of my timeline says we officially just hit AI stagnation. I love tech.'
Well, we will be going deeper deciding whether or not this is stagnation or hyper intelligence takeoff. And we will be joined by our next guest, Riley from Charlie Labs.
Hey guys, thanks for having me.
Good to see you, Riley. How you doing?
What's happening? I'm doing fantastic. We've been heads down with GPT-5.
How long have you had it? How long did you get the preview? I feel like it gets rolled out to early adopters a little bit earlier, but has it been weeks, months? How long have you had it?
We're a couple weeks, like two or three.
What was the first thing you... Charlie liking it?
Charlie loves it. And also I love what Charlie does with it.
Yeah. What does Charlie do with it? What was the first thing you did with GPT-5?
Oh yeah. How'd they come back?
Just really good. Much better than o3, which was much better than any other model we've run before that.
Interesting. And yeah, so let's zoom out. What do you do? What do these evals measure? Walk me through it.
So Charlie is a TypeScript-focused coding agent that operates much more like a human does. So less like IDE application terminal and more joins your GitHub and Slack and Linear workspaces. And it interacts with the team the same way other humans do. And then our evals are a mix of code review, because part of Charlie's job is to review PRs from humans as well as his own, and then code authoring, so opening PRs and pushing commits.
So when you develop your own evals, I imagine you try and keep those out of any training data. You want those to be held private. Is that correct?
Yes. And it's getting even harder with web access now because they're too good at finding things.
They're finding everything. It's funny. And then talk to me about the shape of the actual problems in the eval. Are there some easy questions, some hard questions, some extremely hard questions? Like how are you formulating those? What's the shape of an individual task? Is it scored out of like a hundred? How do you think about developing a good eval?
A mix of hard to very hard. The easy ones are just a waste of money and time at this point, especially with 5. There's a bunch that it's just not going to get wrong. And then we're mostly doing the PR ones, which look kind of like SWE-bench in the sense that we're taking an issue to start with. But instead of giving the issue like in a Docker container already, we trigger a comment on the issue that says, 'Hey, Charlie, go make a PR for this.' And then Charlie does his thing and then the PR comes up. And then we score that PR against a whole bunch of things like correctness to a known solution that's correct as well as code quality, testability, and some softer things like descriptions.
Who are the biggest customers or users for like a TypeScript-focused coding agent?
It's a wide range of mostly modern apps. Pretty much any web app these days is going to be like a Next.js type app. And then all the way into like back end, Charlie himself is written in TypeScript. And there's very little front end.
Anything else? What else you got?
I just want to say I love the name Charlie. It's one of my favorite agent names that we've had on the show.
Yes. It's right up there with Pig and what was the other one? Well, I don't think that was an agent, but...
That was an agent, but...
But yeah, it's a good one.
Yeah, congrats on locking it down.
Yeah. What about cost and that side of the business? Is there any movement there or anything where you require movement or you need movement to really unlock new capabilities in the business or new markets?
Not really for us because we're operating kind of at a human level. We do value-based pricing. So we charge per PR or per commit. And because that's comparing to such expensive actions that humans are doing, the challenge for us is more actually living up to the promise than doing it cheap.
Yeah. Are you having... but then doesn't the cost reduction announced today, isn't that great for business?
Yeah. I mean, it's good overall, but like that's our problem is not that the models are expensive. It's that they're... I mean, they're getting really smart, but I'll always take more.
Like, for instance, since the beginning of August, we've been testing 98% of the code that got merged into our codebase was written by Charlie.
Not 30, not 50, 98%. And that's coming through PRs. That's not like autocomplete in an IDE type thing.
That's crazy.
Yeah. What does that mean for the future of like who are you hiring? I imagine that you're still an engineering-heavy organization that's just puppeteering and orchestrating agents. But where do you see the future of software development as a career path going?
Yeah. Are new CS grads cooked?
I think if they get really good at using the AI, no. If they try and take an approach of getting really good at writing code by hand, for sure. What we're mostly looking for hiring is people who are able to see things at a much higher level and plan further out because with tools like Charlie, you can write so much more code so quickly that it's more important to see where you're going and take the right path than it is to be able to write it quickly.
Very cool. Well, thank you so much for stopping by. Good luck with the rest of your day and congrats on an upgrade to everything that you do.
Tell Charlie to have fun out there.
Have some fun. Thanks a lot, guys.
We'll talk to you soon.
All right. Let me tell you about numeralhq.com. Sales tax on autopilot. Spend less than five minutes per month on sales tax compliance.
Salestaxsuperintelligence.com.
A number of the fellas in the chat got access to 5. Break it down for us.
Reg says it's pretty good. The writing ability feels a little nerfed. Says, 'The way it writes feels a little programmatic rather than sounding human, reverts to using points even for things like blog posts and also uses overly complicated language for simple stuff.' Techno Chief says, 'It's crazy fast.' Dan Ratliff says, 'Yeah, I was just going to say that, very, very, very fast.' Z Jean Ahmed says, 'Junior devs are barbecued.'
Tyler, anything from your side before we talk to Germa from Vercel?
I think maybe a good way to vibe check on at least on the timeline is that it's almost like a 4.5 kind of thing where it comes out, people are like, 'This model totally sucks. Look at the benchmarks. It's not some massive improvement. It's like a bar, not a step change at all.' But then you start playing with it and it's actually like, okay, this is actually a good model. Like a lot of the stuff I'm seeing people post, like, oh, that's actually really interesting output stuff like that. But seems good.
Can we do the green text eval, green text bench?
Yeah, we got the TVPN intern.
Yes. Yes. Yes. Yes. We'll let you cook on that and then we will move on to our next guest, GMO Ralph from Vercel, coming in to TVPN for the second time. Great to see you, GMO. How you doing? I like the action hall. Thank you. Welcome to the stream.
Do you think GPT-5 could beat me, you, a couple of the boys here on Dust 2 in Counter-Strike?
Yeah. It depends on the frame rate, right? Like on a long enough timeline, we're cooked.
We're cooked, but we might frag it short term and we might be faster.
Amazing. Yeah. We got to... I mean, I'm sure we'll get to GPT-5, but what's your reaction to the world model stuff from Google? Do you have an idea of where that's going as a product?
It feels like a GPT-2 level technology, very much a research-focused technology. I'm sure OpenAI is working on something too and a lot of the labs will work on it, but what's your theory behind the generative video game world model stuff that's going on?
I mean, number one, super fascinating, right? I think when we think about the future, I always think about Jensen's: the future of applications will be that pixels are generated, not rendered. So as much as we're really excited today that GPT-5 and v0 are really good at writing code that then renders interfaces, I think it's also cool to dream of a world where we're just going directly from GPU to pixel grid, right? And but if you remember like a couple years ago and maybe a decade ago, there was a lot of excitement of video games that were going to be live streamed from the cloud.
Where your input, your keyboard, you could have a very thin client. Your input, your keyboard, your mouse movement was going to be dispatched to the cloud.
We're going to have GPUs near you.
Google Stadia was big there. And then xCloud was Microsoft's game and is still Microsoft's, actually still pushing it very heavily. Awesome tech, but not mass adoption.
But if you look at, a lot of these technologies are being really successful in letting people get more creative and test things out. A lot of the use cases that we see for v0 and vibe coding are almost like a communication tool. I want to prototype something. I want to see what's possible. I want to explore the latent space. And I think those world models are going to be incredible just to inspire what the future of games could look like, right? Just getting ideas for actually then shipping them in a real 3D engine model. I think short term, I think long-term all bets are off. Someone was saying in the chat, you know, junior devs are roasted or barbecued. I think that's not quite true.
Same for like 3D engine developers.
Give us the bull case for junior devs staying off the barbecue.
So the bull case for I think people in general is that you move from... I mean the progression in the industry has been assistant to agent to team of agents, agent orchestrator. It's still really useful to have a human be the one that's sort of like managing the team. So you're moving from like junior dev to junior manager. Especially as these tools become more agentic, in the new version of v0 that's coming up really soon, you're starting to notice that v0 sort of splits the task between a little team. You have the designer of the team. You have the PM of the team that's sort of working on the spec. You have the architect. You have the engineer. I don't know if you saw Claude Code announced. I think it's like a slash security review.
You think of that as having a security team or team of agents or security researcher at your disposal. So junior dev as like a vertical skill might be a little barbecued, but junior bench manager... so I think it's just going to be the junior dev is so much more powered in this world if you allow yourself to be and you keep up with what these tools can do and I think you stay at the cutting edge.
Yeah. I mean the obvious bull case is if someone's a college student today, they can learn to code truly AI natively. They don't have to say, 'Oh, we're an AI native organization now. We have to upskill and kind of retrain people how to think.' They can just naturally start to think with these capabilities. Sam Altman posted about how we'll look back on, you know, 93% of humanity was subsistence farming. And if you ask those people what they think about our email jobs, they'd be like, 'You guys are crazy.' And it's almost like in the near future, midterm future, maybe even long-term future, it's like the number of individual contributors will be extremely low and almost everyone will be a manager and you'll become a manager much faster. You'll just be managing agents and then you'll be managing people who manage agents. But the job of almost everyone will become managerial. Maybe that's what happens. I don't know. I'm not 100%. But that's what that made me think.
Someone asked me yesterday, you know, what do you think the future of the market of monitors looks like? Like does it stay flat? Do people get more monitors because they're going to like Dogecoin trader analyst when like...
In the future everyone has the hedge fund six monitor setup or...
In the future everybody's just going to be at work on their phone. I mean I've noticed that when I was an individual contributor I had three monitors. I was programming on all the screens and now I use my laptop during the show and then most of my work is done on my phone, phone calls and then firing off tech messages. Yeah, maybe we actually shift away from monitors and go further into voice interfaces. You call the lead of my agents and then that agent relays it to some...
I'm very optimistic on voice by the way because I've now seen it. We're cooking on a better mobile experience for v0.
And I was going back and forth with my head of mobile and he was talking to v0 and I was writing down and I'm a pretty fast typer, but he beat me with voice using the local model on the phone. So there's still the question of like edge latency versus cloud latency kind of like what we talked about with 3D. But I do think voice is going to play an increasingly exciting role in programming which is kind of wild. I would have never imagined. I've always been about like typing benchmarks in WPMs. Voice is coming.
Yeah. Yeah. Yeah. How do you think about competition broadly in developer tooling, code gen? I mean it right now it seems like there's just so much demand.
It feels like massive TAM expansion moment. Every company's ripping... TAM expansion moment, but at the same time winners will emerge. Obviously you're playing to win. And yeah, I'm curious, you know...
Yeah, on some level we're playing both sides of the bet. What we announced today that's really exciting is v0 with GPT-5 support.
So you can go to v0.dev/gpt5 and we'll use GPT-5 in combination with our model pipeline that makes it really good at vibe coding especially for non-technical folks. But we also on the Vercel AI Cloud side of things, we open sourced basically you can create your own vibe coding platform powered by any model.
I was joking about this with Tyler. Vibe code me a vibe coding platform please.
Yeah. Vibe code me a billion dollar company. Yeah. No mistakes. But basically we are giving people that as a starter kit.
Sure. And by the way, the fundamental question that a CEO asked me the other day was, is vibe coding a product or a feature or is it both? You know, it's TBD. The case for feature is okay, so there's going to be lots of systems of record. Think Salesforce, Snowflake, Databricks, and increasingly they're going to incorporate codegen capabilities into their platforms. They can use a lot of these capabilities that we just open sourced and you'll go to their existing place where you have the data, kind of like what we've talked about for decades of like are you bringing compute to the data, are you bringing vibes to the data, right? Are you bringing codegen to your own platform? So...
You used to bring like a dashboard builder and it would have a couple widgets and now I could just potentially if I'm plugged into some sort of data source, some system of record, I could say vibe code this app on top of it. There's some tool, Retool's played in this space, Zapier a little bit, but yeah, I mean this feels like we're getting, we're not fully in the just the pixels are generated, but we're generative UI, generative application on top and that being bespoke and ad hoc.
I also think it's important to understand the line between consumer vibe coding and just generating ephemeral software and websites and things like that versus enterprises which will have a lot of different use cases. When I look at the vibe coding market and I see businesses that are almost entirely consumers just creating things for fun, I think that has to be a tough business because it's a hyper-competitive market and consumers are flaky. They'll create something for fun but they'll churn in month two because they're not running a real business. Whereas a business knows, hey, we'll pay for this on a long-term basis because we have a use for it all the time from this product manager to an engineer over here to somebody in marketing, etc.
Yeah. The other side of the equation is how do you make these vibe coding tools work really well for enterprises? Frankly, the most surprising emergent thing that I've learned is just how much demand there is in enterprises for vibe coding. And this is because a lot of the traditional thing has been the people that understand the business are sitting over here.
The people that understand the code are sitting over here and their communication is fraught with peril. Like they don't speak the same language. They kind of like resent one another. I love to tell this story. I was meeting with a CEO of a very successful company. He was telling me that asking a feature to his own engineers felt like petitioning the government.
Yeah. Even though he's the CEO, it's like he's struggling to make the case and please get me in your next sprint, get me this feature.
So vibe coding actually solves that problem. All of the PMs, designers, marketers, business users that previously only had access to what, Jira and to-do lists and project management tools and writing PRDs and so those kinds of things. They weren't able to ship PRs. They weren't able to ship software and now they can. And so the opportunity is how do you actually make this secure? How do you make it high quality? How do you create guardrails? And those are tricky problems. And I'm really happy that some of them are easy to overcome and at least for us, and some of them are active areas of research, but I think the enterprises really have a strong case for this.
Yeah. Can you walk me through like tool use? I mean, we were talking to the OpenAI folks about GPT-5 being like really a summation of standing on the shoulders of giants. You get a Python REPL, you get a web browser, you get the ability to kind of run cron jobs. Now there's voice and all sorts of different tools kind of wrapped up into one, multiple models. You can trigger reasoning chains if it wants. It can do all these different stuff. And that's actually the benefit of like this isn't just a bigger model. It's a next version of a thing. It's more like switching from the iPhone 12 to 13 than going from the iPhone to the iPhone 3G. It's not just a new technology that's in there. But in the world of vibe coding, what are the tools that you want to think about adding? I know that basically every vibe coding platform recommends a database. But we were talking to Harley at Shopify yesterday and there's a world where if I go to a vibe coding platform and I say I'm building an e-commerce website, it should probably just be like, hey, I'm going to do Shopify under the hood and I'll vibe code the landing page on top. But how are you thinking about the landscape of like tools that you could pull in? Because there's open source repos that are like full projects that you could pull in and then just start customizing on top of. It's kind of this big continuum.
Yeah, there's a couple layers. On the foundation model layer, what you want is a model that is exceptional at tool calling.
Whether it has built-in tools or whether you register them yourself, this is a sort of silent war that has been going on. Like if you talk to devs, what are you optimizing for? Tool calling quality. Why? Because to demystify the word agent, what an agent is, it's a loop of tool calling that builds up context over time.
So to give you an example concretely of v0. v0 is becoming more and more agentic over time. One of the things that it can do is it can take a screenshot of the thing that's building and reflect on it. So today I live vibe coded to an audience of web3 and crypto engineers and I told v0, hey make this dark mode and initially v0 does me dirty. He changes some things with dark mode and then it kind of astonished me because I was like oh I have to now explain to this audience. It then takes a screenshot, looks at it, and keeps fixing it. And I was like, this is literally a developer that's alive on autopilot. And the reason it's on autopilot is because he has access to these tools like looking at the web browser. Another one is research. I vibe coded an example of build me a Substack clone for cryptocurrency news. And the agent didn't know what the cryptocurrency news were. So I started doing research on the internet of okay, Ethereum passed certain price and whatever. So and then you're talking about the tools over the internet. So to demystify another topic, MCP is really exciting because it's a new protocol for registering tools that your agent doesn't locally have. So those tools that I just talked about, we gave them to v0. Here's a deep research tool. Here's the screenshotting tool. And those will likely become the new services. When you think about like AWS of today, if AWS was an AI cloud, which is kind of what we're trying to build at Vercel, you think a lot of those tools are going to become as a service, like bring me the research as a service, bring me browsing and screenshotting as a service and so on. But then you have MCP, which allows you to okay, I need to sell something online. All right, so now there's an MCP for Shopify. Now there's an MCP for Stripe.
So it's really exciting like now it's like the ultimate choice for a builder and you don't have to go and learn all these things. You don't have to... this is almost like a discontinuity of the Valley trend of like if we build amazing documentation they will come. This is more so if the agent picks you, they will come, right? And so there's a lot of figuring out right now like how do I make my infrastructure, how do I make my product to be loved by these agents? And MCP promises to be one of these first things that you are in control of.
That makes a ton of sense. Last question. Someone on your team named Josh is in the chat. He wants to know what does he need to do to get a Twitter badge?
Oh well. Yeah. 100k downloads of the AI CLI. I think we've been talking.
Okay. Good work. The gauntlet's been thrown down.
Thank you. It's on your work cut out for you.
It's burned into the immutable record of this live stream and the future training runs.
Best of luck, accountable now.
We're going to hold you accountable to that. GMO. Great seeing you. We'll talk to you soon. Congratulations.
Let me tell you about Finn.ai. The number one AI agent for customer service. Number one in performance benchmarks. Number one in competitive bakeoffs. Number one in IR in G2. Number one in having an Irish founder.
That's right. And we will invite our next guest to the stream from factory.ai. Welcome to the stream. How are you doing? Good to see you.
Hey, how's it going? Glad to be here.
Great. Thanks so much. Kick us off with an introduction on you and the company.
Yeah, my name is Eno, co-founder, CTO at Factory. We are building a platform for enterprise software developers to perform what we call agent-driven software development. So basically more than just code, bringing agents into every stage of the software development life cycle. So think coding, code review, maintenance, incident response, documentation. We think agents should be a part of all of this and we think that they should be driving a lot of that menial component while you think at the high level about how to plan and structure the work.
There's so many different... like enterprise is a narrow category, it's you know, not consumer I guess, but it's such a wide category. Is there a beach head? Is there a certain type of project within different industries or specific industry that's getting especially a large amount of value out of Factory these days?
Yeah, totally. I think that one thing that we see a lot and typically when we say enterprise we're thinking greater than 1,000 engineers, right? Like 2,000, 3,000. And one reason why we focus on that larger scale, you tend to have these large organizations where the bottleneck is not code, right? The bottleneck is how do we plan a migration of 185 code bases to this new framework and there are 3,000 developers that are going to touch this over the next 6 months and an SI just told us the quote is $80 million to do it. And we have to figure out how to...
