Benedict Evans: OpenAI’s Moat Problem & the Future of Software

If you're Sam Altman, you've got a commodity technology. You're competing with people who have giant legacy cash flows. You don't have your own infrastructure, don't really have any differentiation, but you've got massive mind share. So, what do you do? You try and swap that for hard assets and you try and talk your way into a self-fulfilling prophecy. And then you've got this middle case where you could almost kind of call it like improvised software. And now AI coding means coding is way cheaper and easier, and also means that there's a whole bunch of stuff that you couldn't do in software that now you can. And so there will be way more software.

Thank you.

I suppose there's two ways we could talk about this. One of them is to talk about all the stuff that OpenAI has been trying to do. I think the other is to talk about the problem. And the problem is that, as far as we can see, there is no winner-takes-all effect or network effect in a foundation model. So there's nothing that you can do that means other people can't make a model as good as yours, which is what we're used to from the software industry. That's how Windows and Google and Facebook and Instagram and TikTok and iOS... What we're kind of used to in tech is that software by definition has no capital or very little capital, but it has network effects, and so it tends to produce monopolies or near monopolies, and that produces high margins. And so that's what happened with Windows and Mac OS and Google and so on. Well, the question with LLMs is, like, they're very expensive and very hard, and so you can kind of fail to get onto the ladder or fall off the ladder, for like Microsoft on the one hand and Meta for the meantime on the other hand. But there's nothing that... there's no lever that you can decide you're going to pull whereby you'll just pull ahead of everybody else and they won't be able to catch up, which is like Google versus Bing. Doesn't matter how much money and how hard Microsoft works, Bing will never catch up with Google. So that means if we've got like, pick a number between three and six or maybe more organizations that can make a frontier model and they keep leapfrogging each other every couple of weeks or every couple of months. So that's one problem.

First of all, I push back slightly on that comparison and then answer the question. And the first is if you actually look at the market shares, Google Cloud, Azure and AWS are actually in quite different businesses. AWS is mostly infrastructure. Microsoft is mostly services. Google is mostly scrambling to catch up in a very distant third place despite having basically invented cloud. So they're kind of not all the same, I think. But the challenge, I suppose, is where do we get to equilibrium on the foundation model itself? Is it that these things will be able to create ecosystems above themselves? So it will be like iOS and Android, and so they will create value, they will create value capture, they'll build tools and all sorts of stuff on top of the raw foundation model, and so you'll have to choose, right? All our company is going to standardize on ChatGPT, which isn't what happened in cloud. You as a business, if you go and buy a SaaS app, you don't like think, well, we're going to log into it with our AWS account. Certainly as a consumer, like I can't remember which cloud Snap uses and I don't care. And if I install Uber, like which cloud does Uber use? Who cares? So it may be that the models are able to kind of build something that looks more kind of like Windows, which is what Sam Altman talked about a lot at the end of last year, or it may be that they're basically commodity infrastructure and they're sold at marginal cost, and maybe you barely make a return on the investment, or maybe you don't make a return on the investment itself, and what you're doing is making a return on everything you built that you do with this on top of it. And that's where these companies do get different, because if you are Meta or Google, you've got this whole other highly profitable business which now needs to have LLMs inside it powering all sorts of capabilities and features. And you probably want them to be your LLMs rather than somebody else's. But you may not necessarily need to make money from the LLM by itself anymore than Meta. I mean, obviously Meta has a cloud, but Meta doesn't even try and doesn't sell the cloud. They just use it to power their own stuff. So those are sort of the two possible outcomes. It may be that, you know, this stuff ends up as commodity infrastructure that's sold at zero margin. It may be that no, there's two of these or three of these and they capture a lot of value. There's a sort of price equilibrium question of do you get pricing discipline? Do you have a small number of companies and they hold the prices high or not? Like as I wrote about this last month, like ask your favorite economist, like there are words, there are terms for these kinds of questions, like named after economists who first asked the question generically. But I think the kind of going back to Fidji and OpenAI, the problem at the moment is that the raw chatbot by itself isn't a great product, and most people struggle to work out what to do with it. You've got a small number of people, mostly in tech and also outside tech, who are very self-optimizing and have certain kinds of jobs that map very well to the kinds of stuff that LLMs are very good at doing. If you look at the usage data, something like 10% of the population is using these things every day, but another 50% are using it every week or every month. So most people who have a ChatGPT account can't think of anything to do with it today. And so you can ask, well, is that because the models have to get better? And we had this conversation before. Is this because the models have to get better? Or is it because habits have to change? Or at the extreme, could you say, well, people only use Google once a month, once a week too, which doesn't seem like a good answer. Or do you have to build a whole bunch of stuff on top, which is back to this commodity infrastructure point? And so it seems like what a lot of what the question now is like, oh, are we all just going to use ChatGPT or Anthropic or Claude as ChatGPT, or is that going to be like an invisible API call buried underneath that's used by some other thing, and it's the other thing that makes all the money. The really kind of interesting comparison here would be TSMC, because what happened in chips was that there's no network effect, but with each generation it got more difficult and more expensive, and like the numbers dropped with each generation. And now there's basically one company at the frontier and two or three that are a generation behind, and then five or 10 that are a generation behind that. And TSMC makes lots of money, but like they don't capture all the value of the tech industry. Like, you know, as a consumer, you don't know that TSMC makes a chip in your iPhone, and they don't get like a cut of every App Store sale. So that's sort of the puzzle. This great strategic problem for OpenAI is you basically got commodity technology. You don't have a network effect or winner-takes-all effect. You've got 900 million weekly active users, but most of them are not using it every day and can't think of anything to do with it. And only 5% of them are paying for it. So the usage is like a mile wide but an inch deep. And so you've kind of got to swap the mind share and the momentum that you have for something more durable. So there's a story yesterday that they're trying to do deals with private equity firms to get into the private equity firms' businesses. They did this video sharing app at the end of last year that didn't work. And there's an app store. Oh no, there's another app store. No, are we on the second or the third app store now? I've kind of lost count. And there's an e-commerce integration, but like guess what? E-commerce integrations are really, really hard and complicated because there's millions of merchants and billions of SKUs, and Google and Meta have failed at that twice. So it turns out that OpenAI went, eh, no, we probably can't build that ourselves right now. So there's this sort of challenge of how do you get from having the mind share and one of the good models to having some kind of a durable platform business where you've got developers, users, corporate accounts, something that's locked in that doesn't just depend on you having the best model next week and the week after that and the week after that.

And the net is that a better model doesn't solve the problem. This is something I wrote like a year ago, I think, where I said, what do you mean when you... it was something like, what is a better model? And which sounds like a crazy thing to say, but I always remember like years ago when iPhone was a hot new thing, one of the New York late night TV shows, they would go onto the street and they'd show people last year's phone and say, it's a new phone, what do you think? And they'd go, wow, it's amazing, because like you couldn't really tell actually. And you know, if you're doing like very specific, very hardcore stuff and you're pushing these models to the limit, then you'll be able to, oh, it didn't do that and now it can do that. With most stuff, the models are jagged in lots of different ways. You don't really know whether it will be able to do that or not, or how well it will be able to do that. And most of the stuff that you try, it can sort of do to varying degrees. And now it can sort of do it slightly better or maybe slightly less sort of. But if you've got a bunch of use cases where you need the right answer as opposed to sort of the right answer, then saying that the model is better doesn't mean anything. I mean, literally is literally meaningless because what you're telling me is I asked the model to compile 50 things and last month it would get 10 of them wrong. So I'd have to check all 50, and now it will get eight of them wrong or 12, but probably eight. So again, I'll have to check all 50. So actually nothing's changed. And the point that things change is not when it goes from 90% right to 91% right to 95% right. The point it's either right or not right. And if you don't have that, then you've got to kind of sit and think much harder about what you do with this thing and how you use it and how you build tooling and software around it. Part of the challenge here, I think, is which actually we could kind of segue to talk about, you know, the sort of the SaaS apocalypse, a lot of the challenge is working out how to ask for what you want. And so for the sake of argument, even if the model had general AI and was like actually really superhuman, not superhuman the way that branding people talk about it, was actually really AGI in all senses, it's still quite hard for you to describe what you want.

Well, so two or three things. Firstly, I think very obviously it only works if you're using it a lot already. And so if you haven't worked out a bunch of stuff to do with this, and most people have not, then you don't see the memory. OpenAI did this data at the end marketing at the end of last year where they gave everyone a cute little graphic of how many messages they'd done and told everyone what decile they were in. And so I went on Reddit and grabbed like 300 screenshots of people posting these. And it turns out that if you did a thousand posts, if you did a thousand prompts last year, you're in the top 20%. So basically 80% of people hit return less than a thousand times last year. So average of less than three times a year. Obviously charts growing up in the year. So you can't really average it across the year. But like if you typed return a thousand times in the whole year, you're not really using this every day. And so there's no memory there. Then the sort of a subsidiary point would be, okay, is that a network effect or is that stickiness? It's probably more stickiness. And how sticky and what happens if you just ask it to tell you everything it knows about you and then paste that into Claude or vice versa. So it was kind of unclear whether that was a thing. I mean, the whole thing does feel very sort of 1997 in a lot of ways in that it's clear this is a huge deal and it's already starting to work, but that doesn't mean you know how anything's going to turn out. And one thing that occurred to me looking at ChatGPT as a product is it's kind of like trying to differentiate the web. The chatbot itself is kind of like trying to differentiate a web browser in that you've got an input box and an output box. And how can you make them different if the whole point is that you can type in anything and get anything out? You can make the underlying rendering engine better, like Chrome has a better rendering engine, fine, and you can make the underlying LLM better, but the actual product that's presented to the user, it's an input box and an output box and a couple of buttons around the edges. I mean, it's the same with logos. Like, you've seen the joke that all the chatbot logos look like buttholes, you know, all these little spots, you know, but again, like, what are you supposed to show? Because like, how could you, what would, is it even possible in principle to make the UI of a chatbot different? Because isn't the whole point that it's completely universal and there's no UI? So you've got this kind of, you know, back to Fidji Simo, OpenAI sort of has this problem that first of all the model itself is the same as everybody else's more or less. And if you're again, if you're the kind of person that watches this podcast, you're probably thinking, no, no, no, it's really different. But like for somebody who's only using this once a week, they really don't see the differences. And then it's not clear what you're supposed to build on top of it. And is that going to be you? And even if you worked it out, why wouldn't everybody else build it too?

Oh yeah. Well, this is so, so Kevin Weil and Mike Krieger both made this point on stage like 18 months ago or something. And then Fidji Simo made it in like the second hour of a podcast I was listening to at the gym. You know the joke that like anything, the most secure encryption on earth is anything you say in the second hour of your podcast because no one will ever hear it. Well, I actually listened all the way through. And she said, which is also what Mike and Kevin had said, is, you know, the way it works is you turn on your computer and you look at your phone in the morning, you've got an email from the research group that says, hey, guess what? We've got this cool thing. And then your job is to go and do something with it. So it's like, you know, you get the email and then we've got a new voice model. It's like, okay, well, I guess we're adding a microphone button today. Like, that's not you... you start from the technology. You don't control the product strategy, which is of course how science works, but you don't know what's going to happen. And you don't know what's going to get built. You know, obviously you've got like Sam and Dario and so on are like setting that fundamental research strategy, but you know, the thing comes back in six months and it may work, it may not. You don't know what's going to work. You don't know if that will happen. So, and you know, fairly or unfairly, I opened the last essay I wrote taking that quote from Simo and then comparing it with Steve Jobs famously saying you can't start with the technology and work forward to the product. You've got to start with the user experience and work backwards to the technology. And you know, this is sort of, it's not like anybody doesn't know this, but that's sort of inherent in where we are is that this technology is continually shifting and changing and evolving and you don't know what it'll be able to do next week or next month. So you kind of don't know what it is that you're trying to build. You're a product... you're a strategy taker, not a strategy setter.

Well, so you eventually get to a point that people have actually built a different product as opposed to an input box and an output box to a model. Not at all clear to me that an agent is a consumer-facing... building your own agents is a consumer-facing thing. It seems to me either you shouldn't have to know what it's doing or that should be submerged inside some product. Now I think Stripe did, like they copied autonomous cars and did like level zero to level five for agents. And so, you know, level five is that like the agent knows that you're out of dog food and buys you more dog food and you don't even know and it just kind of appears like fully autonomous ordering and fully autonomous purchasing, fully autonomously solving the problem. And where a sort of level zero is like, you know, here's a picture of a code, work out what it is, and the agent goes and your model goes and calls a bunch of agents to solve that question for you. You don't have to know that that's what it's doing. And so there's certainly sort of a spectrum of what do you mean by agent? Do you just mean that the LLM can call different tools but you don't know about it? Or do you mean you can actually get the LLM to go and do a thing for you with or without multiple tools? It's a very kind of... it's a slightly kind of fuzzy term. It's a little bit like saying metaverse, you know, you don't really know what somebody meant when they... they might have meant VR. VR is real. They might have meant games. Games are real. But when they said metaverse, you didn't really know what it was they were talking about. It's kind of the same with agents. It kind of gets again to this sort of... it almost kind of deepens the problem in that the more the capabilities expand, the more jagged the thing is. And so the harder it is to know whether it will be able to do X or Y, the harder it is for you to kind of mentally map, is this the kind of thing that it would or would not be able to do. So for example, these models struggle massively to read PDFs. But that's not a thing that you would like know from looking at it that it will be completely unable to read a PDF, like because why? There's not some intuitive reason why you could kind of deduce, well, of course it won't be able to do that. And some of this is just familiarity. You know, it took a while to work out, you know, you can't... you could ask Google that it's not going to be able to answer this. But it is this sort of puzzle of how do you build product around representing that. At what point does your model of what it does, the frontier gets smooth enough or your mental model maps closely enough to the jaggedness of the frontier that you can kind of understand what it could or couldn't do, and at what point do you have to know you actually have to sit and map that to a particular use case. But there's another way I think about this, which is to say like with every new thing, you start by getting it to do the stuff you're already used to. And it kind of takes time to build new things that are native to the new technology. And you can probably split that apart further just because obviously every slide needs to have three bullet points and say like there's sort of a third step which is at what point can you kind of pull the whole thing inside out and do something that just doesn't bear any kind of connection to the way that you might have thought about the old thing. You know, the sort of obvious example would be, you know, the progression from, okay, Flickr having a mobile app to Instagram to Snap or TikTok because Instagram is taking the desktop experience. Indeed, arguably still Instagram is still kind of taking the desktop experience except that it adds filters which no one uses anymore. But Snap says, but this has a camera, so why aren't we starting with a camera? And then TikTok says, and it's a social network, so why aren't we starting with that? And so you're sort of pulling the whole thing out. And TikTok isn't like a mobile version of Flickr or indeed a mobile version of YouTube anymore. It's something else. And we're still at the stage of, you know, taking a PDF of your catalog and putting it on your company website as we try and work out what we should do with AI. And of course, that was a really big deal and it's still a really big deal that lots of companies where having a PDF of your catalog on the website is really good like 30 years later.

The problem is that's self-selected behavior. That's people... what can people think of? Whereas, you know, it's again a Steve Jobs quote. You know, what you want is to say, aha, I've realized a thing that you could do with this that hasn't occurred to anybody, because that's where you create billion-dollar companies, trillion-dollar companies. You change the question and you realize that maybe I'm not going to do it like that. I've realized I can solve that. Here's this problem. You didn't really realize that problem existed and I'm going to go and work out a way of solving it in some way that doesn't even look like the problem either. And then that's how you really change things. Yes. It's like step one, OpenAI can look at all the data paths and they can pay for the data paths. The problem there, of course, is that they also have to map that against trying to make money. So I think when you look at the usage data they released last autumn, very little of it was actually e-commerce. There's loads more people using it for porn, but like you can't make money from that. You can make money from e-commerce. So they try to do e-commerce and advertising. Also they hire a bunch of e-commerce and advertising people from Meta. So that's what they know how to do. I mean, this comes back to my point, is this commodity infrastructure, which is... and it's also this is a kind of a quote from Craig Federighi at Apple. So people are having a go at Apple that you haven't got any LLMs, and Craig says, well, we don't have Uber or YouTube either. He didn't actually call it that, obviously, but we don't have a video sharing site, we don't have a taxi service. We provide the platform for other people to do that. And you can't, as a platform, you can't invent all of those things. Google couldn't invent... Apple and Google could not invent everything that was done on the iPhone and Android, and Microsoft couldn't invent everything that was done on the PC. The trap in that, of course, is that you can also say that about Microsoft on the web. Like, Microsoft didn't invent any of the stuff that we did on the web. I mean, that's not quite true. They invented Expedia and like they tried to do all sorts of stuff. But in the end, all this interesting stuff on the web wasn't just... you still did it on a Windows PC, but it was done with other people's tools, other people's software, other people's capabilities, and the PC in effect was commodity infrastructure that you used to access the web. It became a Chromebook, which is actually what a Mac is now mostly for most people. So, you've got this on the one hand, on the other hand, like on the one hand, can you possibly create all of the billions of use cases that people are going to come up with, the millions of use cases that people are going to come up with this stuff? No. But you want to be the place that you want them to build it for your thing and not be commodity infrastructure for everyone else. You want to be kind of somewhere in between those two. The puzzle is that, you know, this was my point, is, you know, you don't sign into your SaaS app with your corporate AWS account. You sign into it with Okta or something, but, you know, no consumer has an AWS account. No company has an AWS... you know, that's not the right level of abstraction or aggregation. It's just infrastructure, and it does feel at the moment like the LLM is infrastructure. And, you know, another historical analogy is like, I mean, I'm just old enough to remember like the early '80s with PCs, people said that what you should do is you should buy a PC and then you should buy a database program or you should buy like a software development program and you should make your own software. So you're a retailer and you want some inventory software, you should buy a PC and you should buy C++ or whatever it would have been, Turbo Pascal, I don't know, wasn't... I don't know, whatever the programming stuff would have been, and you should program your own inventory management software, or you should buy like a database program and you should make your own inventory management software in the database program. And neither of those were the right way of thinking about this. In fact, I found this wonderful quote from The New York Times from 1980 talking about VisiCalc spreadsheets, and this guy says, of course it was a guy who says, well, I used to code DCF models in machine code and it would take me 20 hours, and now with VisiCalc I can do it in 15 minutes. So tell that next time you hear a software developer saying like AI is a completely different thing and nobody has ever abstracted software like this before. Like, yeah, we've been doing this for 30 years, 50 years. So this is this sort of question is like where do you create the applications and the use cases, who does it, how close does that go to the raw chatbot, how far up the stack does a chatbot do it. So now, you know, it was ChatGPT apps which failed like two or three times, and of course it's OpenClaw skills as well or whatever OpenClaw calls them.

Not really. I mean, this week they've got all the fire. They've got all the juice. Whatever the word is, I don't know. This week, this week, next week, next week it'll be something else. OpenClaw is kind of interesting to me because it looks a lot kind of like desktop Linux in that like suddenly you've been having to use this monolithic thing from these big companies where you couldn't really get your hands on the metal. You were just like, you had to get your API key and you type stuff in and they tell you what you could do. And now you can like build it yourself with your own hands. It's like the Homebrew Computer Club in the '70s and you can make it yourself. And then the other side of this, of course, is, and it just needs to be a little bit more polished and then it'll be ready for everybody, which of course Linux people have been saying for 30 years and don't understand, like the whole culture of it is just wrong to do that. And you know, you ask it to... what the other interesting thing about open core is it shows you why Google and Apple haven't shipped this. Like, tidy up my inbox. Okay. I deleted all your messages. Great. You're welcome.

Yeah. And to the point that you're just making, and also Anthropic presumably, you're not in the camp of believing that people are going to be building their own software with Claude Code. It's funny, this is like, it's almost like a straw man except there are like presumably real rational sentient people who actually say this stuff. I don't think it's particularly interesting to explain why people aren't going to code their own Stripe. I think what's more interesting is to kind of think about a broader taxonomy of software, which is that you have like the big enterprise ERPs and what they call systems of record. So like you need to store the same data in the same way for 50,000 people and a billion transactions a year. No, you can't have David Brent from The Office like making his own. Then you have tasks and workflows where that single system, like call it SAP, is too inflexible, and so you break it out, unbundle it into something dedicated and vertical, you get vertical tools. And so this is why the typical big US company today has, depending on your numbers, like 400 to 500 vertical SaaS apps. I think you guys invested in Frame.io, I think.

Yeah, and Frame.io is unbundling Google Sheets and Dropbox. Like theoretically, that's what people were doing. They were managing the video workflows in Google Sheets and Dropbox and email, and it says, no, we're going to turn this into a dedicated tool. And I don't know, theoretically you could have done that in Oracle, but like, no. And so you've got the vertical tool that's specialized and you've got the horizontal general purpose tool, and then you've got this middle case of using Google Sheets and email or using Excel or exporting a CSV or something where you could almost kind of call it like improvised software, where it's not a process that's being done over and over again in the same way, the same time, and has to be recorded and tracked and has to have compliance and everything else, and it's not kind of enough of it or big enough or no one's realized it's big enough to make a dedicated tool, and so you're doing it in Excel or Tableau or Google Sheets or tweets or email or something. And now on the one side, AI coding means coding is way cheaper and easier, and also means that there's a whole bunch of stuff that you couldn't do in software that now you can. And so there will be way more software, and that will pick up many more of those use cases that weren't automated before, either because they were too small or because you actually couldn't automate that thing with software before, and now with AI you can.

Well, this was going to be the other half of it, is now you've got, you expand this category of like improvised software, and so where something it might have been that you do it as get the CSV and do it in Excel, and or maybe a Perl script if you're that kind of person, now maybe you'll ask to do it for you, analyze this for me, do this for me. And often it's like stuff that you couldn't have done in Excel before, you know, it's like look at all those PDFs, look at all those PowerPoints, and then look at this PowerPoint, is there something in all of those PowerPoints that we haven't said in this one, which is the kind of thing that's got the sort of fuzzy answer that it's kind of right that works very well with an LLM, except it wouldn't work if there were PDFs because you can't read PDFs, so you have to tell people it can't read PDFs. So that sense of like this sort of middle ground, and maybe my taxonomy is wrong, but I think that's more interesting as a way of looking at it than just saying, oh, everyone will just vibe code their own thing. No one will vibe code their own ERP or their own Frame.io, but they may ask Anthropic or Gemini or ChatGPT, can you do this thing for me in a way that's sort of analogous to they might have done it in Excel.

Way more software.