I'm here with Benedict Evans. We worked together at Andreessen more than 10 years ago. Benedict is a well-known newsletter author, probably needs no introduction for people watching this. We're here in Singapore. We just came here for an AI conference. This is, you do about a quarter newsletter, three-quarters conference nowadays, we're speaking. That's what brought you out here, right?
And the newsletter is now like 175,000 or something like that, you said.
Yeah, something like that. It wobbles a bit from day to day.
And it started out, you started as a mobile analyst and you became like a broader tech analyst. Is that the evolution?
Yeah, that's one way to put it. I mean, I think...
You were at Orange, is that right?
A long time ago. Yes. Long time ago. Yes. Just when it was all becoming horribly French. There's like... we were chatting before this and I said like the thing in tech is that the point that you understand something is often the point that you should be moving on to pay attention to something else. So I started my career in the dot-com bubble as an equity analyst and I was covering mobile stocks and at that time mobile was kind of dynamic and exciting and sexy and disruptive and...
They turned into water companies. They were going into water companies. Utilities.
Oh, utilities. Yeah. Okay. They were going to connect everybody in the world and then they did and like now what? They were like Marc Andreessen's phrase, they were like the dog that caught the truck, right? And I went and worked in strategy and a bunch of media and telecom things and yeah, I was analyzing, looking at smartphones because that had suddenly become the center of the industry and no one understood it. Now like it happened, time to look for different questions.
Well, it's funny because I think when we were overlapping it was right in the middle of the smartphone dividend, the smartphone explosion. And just to, you know, we actually, there's a few things. One is the smartphone dividend. That's a useful concept, right? Like that the rise of a billion smartphones meant that everything that went into them became cheaper and that enabled VR headsets, that enabled drones, right? All this stuff.
Yeah. All the components that came out of it. Yeah. So smartphone sales are now from memory like one and a quarter, one and a half billion units a year. And all the supply chain from that, all of those components is then available off the shelf if you want to buy 5,000 of them or 10,000 of them. All the Wi-Fi chips and the batteries and the cameras and all the other bits. And before, if you wanted to put a computer into something, you basically needed to use PC components. So ATMs and so on are all basically PCs, like elevators are basically PCs. And that has size and power and cost constraints. And then smartphones become the thing and then all those components are available and so that's what gets you drones and connected light bulbs and all the other bits and pieces around the edge of that.
One of the things I think people don't appreciate is they think, for example, like the consumer, they think like the military has like special gear and it's got its own kind of supply chain. And often the military supply chain is often just a subset of the consumer supply chain because you sell a billion units of this and maybe you have 100,000 or a million units of a military thing.
It's actually, it's almost kind of the reverse now in that it used to be so that the way I think about this is like in the past, like before we were born, the intelligence agencies would get the cool new stuff first and then the military would get it and then big corporations would get it and eventually consumers would get it like 30 years afterwards. So this is like the version is like microwaves were invented for NASA, right? And eventually consumers get them.
Yeah. Yeah. Or like GPS was invented to guide missiles.
Exactly. And now it's used for tagging cat photos. And the shift is like a combination of the stuff getting cheap enough that it can be for consumers instead of you needing a billion dollars to have one, and then the scale of consumers once it gets cheap enough. And so now the way it works is the consumers get the new stuff and the military gets it 10 years later, because that's how long it takes the bureaucracy to assimilate that, a, the bureaucracy, b, to harden it and productize it and turn it into what you need if it's going to get shot at or it's going to be cold or hot or warm or whatever it is.
Yeah, that's, it's funny. Does it really improve through that process? I know people think it does, but I'm not sure it does relative to the cost of not using the pretty good product versus whatever improvements come from the delay to harden it. I'm not sure if it actually, I don't know.
I think, but there's clearly, there's a sort of a process of you have to put it into a fighter jet. You don't replace the avionics in a fighter jet every six months. Right. Well, but you know, but yeah, you know, that's the kind of the core of it is the cutting edge of the innovation is for consumers and then that flows back through to everything else.
That's right. Well, you know, I was going to say maybe in China you do. Maybe in China, like I think what happened with the consumer drones, they got good at quadcopters and that's led them to their new form. Have you seen EHang? It's like the Chinese flying cars.
You know, I played this clip like a year, year and a half ago. And people said, um, it kind of, you know, the teal one, like we wanted flying cars, we got 140 characters. And I was like, uh, a lot of people done a riff on that, but I was like, we wanted to fly in cars, we got them with Chinese characters. Okay. Yeah. And the thing is when I put that up there, people are like, that's not a car. It's a, you know, it's a copter, right? That's not a car. It doesn't have wheels. But it solved the problem differently, right? And actually I think was one of your lines is like, uh, unfair comparisons are often the best kind of comparisons, right?
Yeah. Um, I remember seeing a bunch of flying cars when we were at Andreessen Horowitz. I think Marc Andreessen, he said it was like, they're all like houseboats and a houseboat is a crap house and a crap boat.
Yes, that's right. And, um, you know, thinking of it as a flying car is like the wrong term. It's better to think of it as like a much, a small, much better, much cheaper helicopter.
Maybe, maybe. But the point is that they now, the other thing is it's meant for short hops and like city to city where you fly over the traffic. And they've got this, Uber was going to do this, by the way, before they decapitated Uber. Like the low-altitude economy was something they were thinking about. And a lot of things get like cut off in the West and then they appear fully formed in China. Like consumer drones, for example, you know, Chris Anderson, um, he was very early on drones and that got blocked by the FAA. And so consumer drones were hobbled in the US and that's why DJI arose in China. So lots of things get blocked in the West and they arise in China because of that.
Um, anyway, coming back up. So, uh, so smartphones, I mean, I think you and, uh, Horace Dediu of Asymco, I was on his pod a while ago. I think you're two of the best. He's also like European or something like that.
French. Yeah, he was, he was at Nokia.
I mean, there's an interesting kind of like information. Do you know him?
Yeah, I know. Yeah, he's great. He's a great guy. Um, part of it was, it was like, and there was a moment in time when there weren't many people who really understood this and were industry analysts and were able to talk in public.
So there were people inside Nokia or Goldman's or Bain or wherever who had all the data, but they couldn't publish the data and they weren't allowed to say stuff in public. Or if they were writing analysis, it was analysis for public markets investors or something, right? And so there were very few people who were like, knew that you could go and take Apple's reports and make a chart of unit sales and make a chart of ASP and knew what ASP was or knew what ARPU was. Now there's like an explosion of this. So there's huge, huge numbers. And particularly if you look at AI now, there's like 10 people who do a really, really good 200-page deck of every possible AI chart.
Oh, is that right? Interesting.
Yeah. Um, and so that whole thing shifted. But at the time, yes, it was me and Horace and like Ben Bajarin, um, were like the only strate-, like, you know, a handful of people who understood this and could do the charts and are allowed to do the charts. And so that was sort of, you know, being at the right place, right time got me a lot of attention.
Yeah, it's interesting. I think like you and, um, uh, Ben Thompson of Stratechery, and I'm not sure if Horace had a newsletter, but you guys were newsletters before Substack productized it. Sort of like Rogan was a podcast before that became productized as a category. Um, and are you on Substack or were you on Ghost?
No. Yeah. No, I'm still on my old cobbled-together stack of Mailchimp plus Memberful plus Squarespace.
Why, why don't you, you don't want to move to something?
It's just a pain to move. It's a heavy lift to move platform and you sit and do the analysis and you're like, is this a good use of like a week of my time, right?
Maybe, maybe it might be at this point. Substack's pretty good, but I mean, it's your, you know, your separate Substack thing, which is, do you want it to be on your newsletter or your Substack?
Yes, that's true. Yeah, because it's a platform and you get the advantage of, I mean, this is something we can talk about. You know, it's Chris Dixon's line of come for the tool, stay for the network. You go on Substack, they will get you new subscribers. Ghost won't get you subscribers. Yes. On the other hand, they now, they control who your readers are and you don't, which is always a thing of a network.
Well, I mean, you still mail out to everybody.
You do, but then they're trying to get you to use their website and their algorithm to decide who reads what.
True. But there's always these kind of questions like, do you want to go with the people who will give you an audience and in exchange for that, they're deciding that they'll give you the audience. There's always a trade-off with the distribution.
Yeah. Um, I think Ghost is another option and I think, uh, Ghost and Beehiiv are the two others that people use. Ghost is like, you know, I saw Ghost when it was very early and, uh, I just thought it was so good for what it was like. It was, um, I mean, not even for, it's just a very polished, thought-through, for an open-source product, it's unusually polished. John O'Nolan's very, very good.
Um, it's funny, you know, like, um, on that, uh, well, but there's a bunch of things we can talk about, but the whole newsletter thing, it's sometimes there's things that are like newsletters or podcasts that are what I consider lowercase in technology before they become uppercase. Like for example, Odeo, you know, like what Twitter was, Twitter was a podcasting company before it became Twitter. And the time constant, they just got the time constant wrong where you needed, which is hard to predict, that microblogging would take off first and then it required like AirPods and everybody being online for a long time and maybe, you know, COVID before podcast really exploded and the term was around in lowercase. You could even argue it needed like 5G or 4G. If you're listening to it in the car, then you need a fast enough network.
Yes. Bandwidth is a constraint.
Yes. Um, and then the time works. So, what do you think is lowercase today that's going to become uppercase? Like what exists in tech that people are like, oh, yeah, that exists, but it's going to go big? I have some ideas. I want to hear yours.
Interesting question. Um, I think there's probably the answer, if I was a consultant and trying to whiteboard this, is I would be looking around AI because that's a new platform. Um, and, you know, there's a lot, all the old white space got filled in and now you've got a whole bunch of new white space. So deterministically there should be a bunch of those things here. AI, which I'm sure we'll talk about, does feel very sort of mid-90s in that you're like mid-90s internet in that like, well, is this a browser? How do you use it? What's it for? How would you get to it? How does this work? Where's the value, where's the value capture going to be? Um, I'm not sure that there's like, maybe one answer is like, I'm too old and I'm not like spending too much time looking for like weird, weird stuff around the edges. The last one of these that I spotted personally was Shein, which is it Shein or Shine? I'm told it's Shen for speaking to people there. Like I haven't worked out how to pronounce it. Um, that was an interesting one that it was, maybe you could also say it was the last of the ones that you could spot because suddenly, wait, what is this thing that's at the top of the iPhone app store charts all the time.
Um, suddenly that thing exploded and that's like probably the largest pure-play apparel retailer on earth.
Yeah. And like Shein and Temu.
Yeah, that's right. They're now getting hit with the tariff stuff and, you know, we'll see.
Yeah, tariffs plus, um, plus a de minimis rule, um, in the US.
But that's only the US market and that's not, you know, I don't know what fraction of their sales that is, like a third of their sales or half, quarter of their sales or something. So that was like, that was a thing that was interesting. I'm not sure there's not like a new thing that I'm watching that I've noticed recently. Um, I'm sure there will be, you know, I keep looking.
So, I have a few. We were talking about the glasses, right? Like I think smart glasses are sort of like the most predictable thing after the iPhone and...
Oh, yeah. That's, I put in a different category. I was sort of thinking like what's stuff that's being used now that people haven't quite noticed is being used yet.
I see. Well, I guess that glasses, glasses definitely your next thing. Sure. So, well, I guess I would sort of bundle VR headsets, AR headsets, uh, you know, like that with glasses and say that that's just, glasses are sort of the next version for goggles. But okay, so that's one that we'd agree on. Except the question is, as you said, is it going to be watches or phones? How big does that get?
Right. Yeah. Um, I think, um, I think, uh, you know, just like podcasts grew to mean like video podcasts and so on, the robot dogs are interesting. Um, they are fun to play with. Uh, and they're getting way cheaper now, right? They went from the Boston Dynamics kind of things. So, the home robot as a toy, I think, is probably going to become more and more popular, like a Christmas present kind of thing at first, right? Um, because I see kids playing with them and they just love them just as a toy. And the, you know, it's kind of like, um, the robot dog, the drone as like a starting to become a like a Christmas present kind of thing. I think that that becomes a thing. Um, and eventually, like we were talking about this at the Museum of the Future in the UAE, they clad these so that it doesn't, it's not just like a skeleton of a robot dog, but it actually looks like an animal and that completely changes your perception of it, right? Um, so I think that that'll be a thing.
Um, and with respect to AI, and so let's do, I mean, there's AI, there's Bitcoin, there's China, there's drones, there's biotech, there's actually several different areas that I'm tracking. I'm tracking a bunch of these various singularities, whatever. Um, they're not all really actually singularities in the technical sense of going to infinity, but ramps, curves.
Curves. That's right. Yeah.
Um, with AI, there's, uh, you know, one way of thinking about it is, uh, like now we're two and a half years in, let's say, let's call the ChatGPT moment, right? And, um, it's interesting because it, uh, I think what people really overestimated was how much it's agentic intelligence versus amplified intelligence, like that to say you still have to prompt it. So prompting is like higher-level programming. Number one, you still have to verify the output and that means you kind of need to know what it is you're looking for. For example, if it spits out a bunch of mathematical symbols in an area of math that you don't know, then you have to be Terence Tao to verify it. It might be gibberish. It might be real. Who knows, right? And, uh, so the prompting and verifying are actually the bottlenecks in many areas. Now, Karpathy and I, you know, the Andrej Karpathy, were just having a discussion on this like a week or so ago. And the thing about verifying is if you're using the GPUs that we have built in and you're looking at images or video or front-end code, right, like the user interface, your eye can just instantly pick out and you can verify pretty quickly. So for that side of things, AI is quite good. Anything that's images, video, your ear can also pick out audio, right? Um, and front-end. But when it's backend stuff, right, when it's like database code, when it's like crypto, when it's mathematical equations that you don't have like GPUs, you can't just like hit it with your eyes and quickly detect it, right, whether it's correct or not. You have to deep read it carefully.
Right. So it can generate reams of text, but then you have to verify it.
Exactly. That's right. Maybe you have some thoughts on that.
Well, so it's funny. I was talking to John Borthwick the other day and he said, Benedict, you think in slides. Um, so that we do too. We both think in slides. So I have a slide. Yes. Um, and maybe there's sort of, I'll talk about the slide and there's sort of an observation around it. I think a lot of discussion of LLMs is sort of hunting for like, what's the right, it's like with machine learning, the right way to conceptualize it was this is pattern recognition. And we're still sort of hunting for the right way to conceptualize LLMs. Um, the slide is that traditional software is deterministic and does things that are easy to explain to machines. In fact, automation, machine tools, sewing machines, typewriters, adding machines, right? Things that are easy to explain to a computer. There may be things that are very hard for people to do, but they're easy to explain. So it's hard for you to drill a hole 100 times or to calculate a mortgage in your head, but it's easy for you to write down the logical steps to explain how you do this. So that's traditional software, right? Databases, data processing, the whole 60s, 70s mainframe thing. Um, machine learning is stuff that's hard to explain to a computer. So it's hard to explain why that credit card transaction is weird. It's hard or how to move your hands or something.
Yeah, it's hard to explain why that's a picture of a dog and not a cat.
And you think it's easy until you try and do it and then it's like you try to make a mechanical horse. It always falls over until robotics comes along. Um, so that was machine learning. I also think that, as a kind of quiz for you, do you think machine learning is still AI or is that now just software?
Well, so the way that, uh, I think there's a process that once it's been around for a while, it's not AI anymore. It's funny. So I think like within the field technically the division would be machine learning would be, uh, you know, everything up to linear logistic regression, um, and, you know, SVMs, all that kind of stuff. And then right at the point you start doing deep learning and you have large neural networks, now you start getting into what people would call modern AI. So, um, ML is almost like the boundary of understandability, you might say, right, where you can write clean equations and like really understand what's going on. And to me, the most surprising and confusing, I still don't feel I know what the phenomenon is, but I still find it magical, is something called the double descent problem. You know what that is? Basically, normally when you're fitting to data, um, you want to have the fewest possible parameters because you can overfit, right? And so your error goes down and then your error starts going up on the holdout set because you train your model in machine learning and you want the minimum number of parameters to be able to explain the training data and predict the test data. And if you overfit, then you're no longer predicting out-of-sample stuff. But double descent is when you do AI, you get actually a second wind when you start going to a very highly parameterized model and the error actually drops again, right? And, um, which is just a really weird phenomenon that there's papers on this and so on. And, um, it's one of the most counterintuitive things about the whole thing that just having these gigantically parameterized models would generalize.
Well, right, because it violates that.
That's the biggest difference.
Go ahead. There's other things people might say is the biggest difference. I mean, I think there's one of the ways I sort of think about the term AI is that people kind of use it like technology. The word technology.
Yeah, that's right. That anything new is technology. Anything your parents had is technology. I'm a stickler for precision. There's different ways that you can say what do we mean by the word AI. I feel like AI has almost become like the word metaverse where like you don't know what somebody means when they say it.
But to continue my slide. So there's the first point is there's deterministic software which is stuff that's easy to explain. There's machine learning which is stuff that was hard to explain which basically machine learning solved. And now an LLM is maybe stuff that's easy to explain to an intern. It's something where if you had to go away and have a kickoff meeting and spend half an hour working out how we're going to do this project, then an LLM probably can't do that. But if it's something that you could explain in 10 seconds or 20 seconds, then an LLM is going to be able to do that. And part of the problem is, are you able to explain it even to yourself? Could you explain it to another human being? Can you actually kind of shut your eyes and conceptualize how is it that I'm going to explain what it is that I want this thing to do?
It's so, what you're saying is very important because, um, you know, there's like several different angles I want to take off of that. Um, you know, in one sense, I had this tweet, we're living in the age of the phrase, right? So the prompt for the AI, um, or the 140-character tweet, or actually in crypto, like 14 words, 13 words, 12 words can be your crypto reset phrase, right? These like are phrases of power, right? In AI, in social, in crypto, right? Like these strings of characters that do a lot, you know, and they're spells, they're spells, right? And the thing about it is, um, the crisper you are as a manager, like, you know, if you're a really good engineering manager, you're great at prompting AI because crucially you don't just say, hey, code this. You say, hey, you know, try and use, you know, React for this. You can use React Native for, you know, the iOS and Android interfaces. Use Tailwind. Use the more, in a sense, vocabulary terms you have, the better you can prompt something with. And you have to use those vocabulary terms correctly. And what that meant is, for example, I realized with DALL-E, you know, when that was first, you know, before the chat moment, I was like, wow, art history is now an applied subject, knowing like Cezanne and Picasso and what, you know, these various kinds of obscure styles. Suddenly you can be like, boom, style it like this, style it like this, and it'll do that, right? You could say the same thing for music.
Like what exactly is it that's being done there? There is a word for that. So, and you have to know the word.
That's right. Exactly. So, you can upload a track and you can say, what style is this? How do you caption this? Right? You ever seen, you know, like the restaurants with the fancy menus and they don't say, um, they don't say tomatoes, they say like...
There's a word, there's things that are theoretically, um, subjective. Yeah. But there are within the professions, there is a particular term for doing that particular thing.
Exactly. It's like difference between like red versus burgundy and, you know, crimson and what have you. They've got precise words which mean something and then you can summon greater precision with those precise words. And so what a way I was thinking about, you know, what you're saying is that, and I've written about this, AI is like undocumented APIs, right? So normal API, every function is like written out and it's like you can do this and you can do that and it's like got 20 functions and here's everything is there, right? With AI, it can do lots of things that even the people who wrote it up, so there's, it's much more mysterious as to what it can do. You just have to try things, right? So the way I was thinking about this from a different angle was to think about GUIs. But what a GUI is doing, there's several things that a GUI is doing. One of them is it's telling you all the features that the developers have created. And part of the reason that was a revolution is, A, you knew what they were and you didn't need to memorize keyboard commands. Yeah. But B, you can actually have more stuff because you're not constrained by the number of keyboard commands that you can write down. So you can have hundreds of functions instead of like, you know, you can just put them all, you can just add more to the menus. Yes. Um, but the other part of it is that the GUI is telling the user a whole bunch of accumulated decision and institutional knowledge about what the right things to do at this point would be, right? And so if you're in a workflow as opposed to just a blank screen, you know, it's one thing if you're in like Photoshop or Excel. Yeah, it can prompt you on the prompt, but if you're in a workflow in Salesforce, then there's a decision taken that says, I'm going to offer the user these five options here and not 750 options.
Yeah. This is, you know, Karpathy has talked about this also, but I do think there's room for AIOS, right? Like in a sense, uh, and we can talk about crypto in a second, but I think AI and crypto are both actually operating system level innovations. And for example, it may be someone who just does as an app or like a downloadable thing and just does as a layer on top of the Mac. But if you have the full context of all the actions that are happening on your Mac, you can suggest which apps to use, suggest which apps to download, suggest, hey, you probably want to change these keyboard settings. And so like there's, uh, you know, it's funny to put it this way, but Clippy is finally vindicated. Clippy, but for everything, right? And, um, because Clippy can now be really, really, really, really smart, right? Like, you know, it was Andreessen's line is like everything in tech works. It's just when, right? And, uh, and even the thing that's interesting about the Clippy thing is, uh, somebody also made a point which is that, um, you actually want to put faces on your AI, avatars on your AI agents. So you could pick from Clippy or 10 other kinds of things. And the reason you want to do that, uh, this is counterintuitive, but people like you and I can use ChatGPT and, you know, Claude and what have you because we're familiar with interfaces, but the reason they're actually intuitive to 100 million people is they're used to chatting with another human on the other side. So they're already modeling the chat box as being a human-like response because they've been using WhatsApp or Facebook Messenger or Instagram chat or something like that for a long time, right? Um, but when it's outside of that chat box environment and it's like suggesting on the screen, you kind of want a face to pop up so they can associate, okay, this person is suggesting this because that's who they are and they kind of map that personality onto the AI agent. And so you could choose from different kinds of Clippies that would give you prompts on what to do or it just does it for you. That's another possibility.
But I don't think people like it when it does it. They want to be able to approve it before they do.
I think there's a sort of sense in here of how people conceptualize what this thing is and how it works. I remember, um, John Borthwick at Google showing me a chart, a Google Trends chart of best versus cheap.
Best versus cheap. Uh-huh. Yeah.
So the best does this and cheap does that.
Crossing over time. So Google Trends. So what's the frequency of the word best? So it starts with like cheap phones and it goes to best phones.
Yes. And so the thesis was that this was shifting from the internet as price comparison where you already knew what you wanted and it's at the top of the, and that's so the bottom of the funnel to the internet as recommendation, curation, suggestion where you're looking for suggestions.
That's so interesting. So let me see if I can understand the psychology. So when, so it starts from 20, it starts from 2004. Okay. So in 2004, you go on the internet and you already know what you want and you look for the cheap, what is the cheap X and then you put in a SKU or you put in a product or something. Whereas over time that goes down and best goes up and best goes up and crosses it. It's a perfect X on the chart, unfortunately. And but the thesis is you're going further up the funnel. You're looking more and more for, I want someone on the internet to tell me the best X or Y where previously you'd have got that from the magazine or newspaper or something.
There's two or three, there's two things about that. The first is, you know, Andy Grove's thing about the paired metrics. So Andy Grove, um, whenever anybody's optimizing like sales, for example, they will usually, or recruiting, they'll start by optimizing quantity, but there's, you know, you can sometimes optimize quantity and then quality drops off, right? So quantity is easy to measure. It's like just the number of people we hired or whatever, but quality is how good were they, right? And so that's the, the second paired metric is usually a quality metric. And so quantity is cheap, right? And people start with cheap and then quality is best and they go to best. So that's another lens on this. A third lens, what I thought, my explanation, maybe it's different than what actually happened was when people are just trying out a space, they just want the cheap version to try it out. And once they've committed to a space and like, for example, they cheap digital camera, cheap, cheap drone or something like that. I want to try it out, right? And they want to try it out at low cost.
And then once they're committed to a space, then they're like, I want the best drone out there now because I want to, the analysis then would be cheap drone versus best drone, right? Um, but I think the, there, that's what I thought you were saying. So, but you're saying cheapest to best overall.
Overall. But I'd love to see it category by category. I wouldn't be surprised to see that happen category by category, but maybe not.
Well, there's a different point there, which is sort of what I was talking about in our panel this morning, which is this infinite product. So how do you know what to buy? Yeah. And it used to be that you'd start with a magazine and then you go to the internet to find the cheap place to buy it already knowing what you wanted. And now you go to the internet to find like, where's, what's the, what's the right place to do this. So the internet has become much more kind of a default. But actually the thing, the thought that prompted me to that was you can also go and play with Google Trends. And I did a chart, played with like how, why, where, what, like more kind of basic questions. And you really need to be inside Google to do that analysis properly.
Exactly. Yeah. But it's that sense of how much are people doing conversational queries into Google as opposed to typing keywords into Google and things that are not really a Google query, like what is a, is not probably doesn't help Google, but that's still how people use it. People were trained for years to not, to, to remove all prepositions, to remove all that stuff and just do keyword-ese. And now we're trained the opposite to write full and complete English sentences.
Like prompting is the new searching, but it's a completely different, you know, behavior, right? Um, go ahead.
Well, I was going to say this is one of the, um, there's a sort of tangential point of that. Like one of the, like the early easy obvious things that people have deployed with AI, with LLMs on the internet is different kinds of, is sort of natural language queries or different, not so much natural language, but like different kinds of query. So the canonical one people talk about is Walmart saying now you can search for what should I buy to take on a picnic, which isn't a database query. And for Google, for Walmart or for Amazon five years ago, that search just wouldn't work because unless there's a product that's like tagged with picnic, it's not going to come up. Whereas now there's an LLM with a world that has some sense of what, how you might answer that question.
Yes. Is it a world model?
It is, it's at least a web model. It's a different kind of query anyway. It's not, you're not doing a SQL query. You're doing something else.
That's right. And I think, you know, one of the things that's interesting is, um, computers are, we knew they were very good at that first kind of deterministic computation, the SQL query, the calculations. That's what they're built for doing math, right? Yeah. And now they've gotten good at probabilistic kinds of things, right? So this would be like system one and system two thinking, right? Probabilistic is like the quick impression and then, you know, this is like the logical calculation. So it's actually good at the hard, the thing that's harder for humans is the, you know, like long involved mathematical calculation can do that errorlessly. And now it can also do the other kind of thing. And so it does suggest that there would be some synthesis of that eventually where an AI can, I mean, this is like AI tool use or what have you, like it detects that it needs to go to system one and it starts invoking Python for that. And this is getting better, but, um, it's surprisingly not amazing two and a half years in, right? Uh, when it needs to go deterministic.
Well, so I, um, I wrote, the last long thing I wrote about this was about looking at Deep Research which OpenAI launched. And one of the kind of traps in looking at the new thing is to test it based on what was important to the old thing. So, you know, to look at the Apple II and say, does this match the uptime of a mainframe? No. So it's useless. Well, no, but that's not the right question. Yes. Um, can you build an Excel model on an iPhone? No, but that's not the point. It can still replace PCs. And the reason I mention this is, so Deep Research, OpenAI launched this thing and it's whatever it was, $100 a month or whatever. So, but then you look at the marketing page and the marketing page shows it doing a research project about mobile, which as we said, I know a lot about, and it got the answers wrong. And it, that's verifying.
See, you knew, you could tell that it was wrong, but it looked polished.
Exactly. So this is the thing. And it got stuff wrong in several levels. Um, people can remember now what I wrote like two months ago. And so there was a specific, it was make a table which shows mobile smartphone adoption in a bunch of countries and then the operating system market share. And then this is like an intern teaching moment because first of all, what does adoption mean? Does that mean unit sales share, installed base, app store sales? Like what metric specific are you asking me for? Yeah. Then it had given a source for the number it had come up with, which was, um, Statista. And Statista is an aggregator that steals other people's data and repolishes it. Yeah. And when you jump through a bunch of registration hoops, you discover that the actual source was, um, I think Kantar. It's an ad agency. It's part of GroupM. I think it was part of one of their, it's consumer survey data. So it's a proper, proper, proper company. Yeah. Yeah. So it was actual proper consumer survey data. Um, but the two things. So then when you go to the Kantar chart page, you discover that Deep Research had got the numbers the opposite. So it had flipped the percentages.
And then it had also said right back, it removed, it had copied them from the website wrong.
And then the other source it gave was StatCounter. And StatCounter was just using the same wrong data, which is a traffic measure. Right? So that's not going to tell you what adoption is because high-end phones get used more and iPhones get used more. So, and there's a bunch of things in here where you'd like, this is what I'd expect from an intern, right? I would go back and say, no, this is what I mean by adoption and this is a good data source and that isn't, right? And it was like a great first version. Um, the problem is, A, it had to copy the number out wrong, which is not what I would expect for an intern or at least not a good intern. But secondly, I'd have to be a mobile analyst to know any of these things.
Exactly. And that's a verifying thing that I was getting at.
This is the kind of the core of it is all these people were looking at Deep Research and saying this is fantastic for researching things you don't know anything about. I was like, no, no, it's not. It's fantastic if you need a bunch of material about something you know a lot about.
Exactly. So that's the thing is that's why I think AI in its current incarnation is better thought of as amplified intelligence because the better, the more you know about a field, the better you are at prompting because you got better vocabulary and the better you are at verifying because you know more facts about it and you have more cross-cutting checks. And that is less true for the visual area, but just identifying that is a very important limitation where you have a completely different system you can use for the visual stuff, which is just your eyes, right? Um, you don't have to use the, you know, we have different hardware for quickly seeing, you know, this way, the hands or something like that, whereas if that was...
It's a monkey brain, exactly, right? So that's now an interesting question. This is, you know, Karpathy and I are discussing this, is there some way to turn some or a subset of the non-visual things into visual cues where you could see what's wrong immediately? So give me, I'll give you a small simple example. Let's say you generate an audio file, right? You know, like a spectrogram of an audio file, right? You could maybe immediately see if there's some artifact there.
Right? That's a trivial example. So I think this, it's a fascinating concept. I would wonder whether that's the right split.
Okay. It's at least one split I found useful for now. What are you thinking?
Well, so the split I was thinking was that the natural language generation to make text is perfect. So the text is always grammatically correct.
But the model underneath, like the facts presented in the text might be wrong.
Yes. And that's sort of deceptive to us because we see the text is correct and it looks confident.
Yeah, that's right. Whereas in an image, um, like you ask it for a picture of a person and everything's perfect except the person's got six hands.
Mm-hmm. I'm not sure conceptually what is it that that's flattened. Is that you're seeing two things in one layer or is it that, do you see what I mean?
I see what you mean. I think, or is it that it's a different level of, well, maybe then maybe there's a different point here, which is, um, you ask for an image of a car. Yeah. And the car, you like, I actually did this. Ask for a fantasy 1960s French sports car. Right. It will look French. It will look like a sports car. It will have four wheels. It might have two steering wheels.
Yes, that's right. The two steering wheels is the equivalent of a grammatical mistake or a spelling mistake in the text generator.
Yes. Because however, it may also be that the balance of the car is all wrong and it would flip over if it tried to go around a corner. But you'd have to be an automotive expert to know that. So I'm saying that like there's different levels of error.
That's right. What you're saying is the two steering wheels is like a spelling error, but spelling errors are very rare for AI, whereas the two steering wheels is a common error.
Right. And I think that has to do with, um, just the way diffusion models work versus how transformers work. That'd be like one high-level answer I'd give where it's doing like kind of, it's more local with the diffusion model and, um, you can be locally correct with the steering wheel but globally incorrect, whereas locally correct with spelling is usually correct. That's like maybe one that's useful.
Um, the second is that, um, with there's only a small space, I think like for example, we are optimized to recognize faces so we can detect very subtle differences in faces. But if I gave you like five different sheets of like static noise, right, even if there are very clear patterns, like mathematically, like these are all like Fourier transforms of the same object and this is the one I add, they're just like total noise to you. A computer would be like, these 12 are the same and this one is odd one out, right? So, um, in a sense, our eyes are optimized for a very low-dimensional set of things, which are the things that occur in the real world, like those are the things we can pick out.
Right. Which is also that our eyes are like, dogs are better at motion than us.
Yes. Even eyes are different depending on the species.
That's right. So because of that, we actually have a, uh, like because they can't detect patterns in static. That's like too high-dimensional a space. I think text is kind of like that because it can describe one of the most, I mean, surprising things to me about how AI has evolved. We were asking, we were talking about this question before, is I was surprised you could get so much mileage out of pure text.
The reason is so much, what's so much mileage out of pure text, right? Um, and the reason I was surprised by that is, you know, you'd think...
You mean like reasoning and or stuff that looks like reasoning?
Reason and also spatial manipulation, like picking, like having cameras, having eyes, seeing the world, reasoning about it like a baby and so on and so forth. It is amazing how much of that world humans have assigned machine-readable labels to with text. And the way that, you know, it's just very surprising how well that worked. Like language, what I'm trying to say is in a few, in like 40 words, you can describe, it's like code, you can describe many, many, many different kinds of things in like 40 words, right? And it's just more general. Um, it's one of those things where if you're, sometimes when you're really close to a space, you're actually more surprised by a breakthrough than if you're farther away. And I should say like, you know, even looking, seeing all the style transfer stuff in the mid-2010s and seeing ImageNet and seeing the benchmarks and so on and so forth, I was surprised that it got, you know, a markup chain is it well, um, if you saw the stuff before GPT-3, right, uh, it was like semi-coherent, but it didn't look like it was converging on something, you know, it just looked like, you know, it repeat itself many times and what have you. And the fact that it broke through to what it did just based on language was so counterintuitive. And it's, I think it's because it's such a high-dimensional thing. It captures so many different aspects of the world, like anything you could perceive in the world, there's a word for it. There's many words for it. Um, and then we also have billions of people who've been typing those words for two decades, right? So in a sense, like the entire internet, the video games and social media are like this bootstrapper for AI. Anyway, so on the other hand, AI is very bad at spatial stuff. You know this thing called ARC? Francois Chollet has this, uh, benchmark.
Yeah. And so he has this benchmark that actually got beaten by the recent, um, you know, ChatGPT release. And he's got like a new one. Uh, and it's almost like a Tetris-y kind of thing that's got some degree of logic and spatial type stuff that AI finds it hard, but humans still find it easy. It's kind of like maybe the next generation CAPTCHA. Um, and it's visual more than it is verbal, right? So for whatever reason...
Is it something that would be hard to explain in words?
Yes, I think kind of it's about like this is here and it's almost like Minesweeper, you know, Minesweeper where you click and it expands and so on and so forth. Um, I think AI, because it started with words, it doesn't do well with the spatial side of things. Now, on the other hand, what the Chinese are working on in particular is physical robotics. Obviously Elon's working on it and so on and so forth, but China's way ahead on the physical supply chain. So like physical AI is robots and those definitely have cameras and XYZ and spatial and rotation and so forth. So there's some eventual fusion, you know, like the self-driving cars have gathered hundreds of millions, billions of miles at this point. So there's some fusion of the web, which is words, and the world, which is, you know, spatial, that will get you like a completely, you know, maybe a fusion set where it can reason about the world as it is. Knows how tall Everest is. Someone, some robot has hiked it. You know, like Google Street View. You might eventually imagine a bunch of humanoids walking the world just like that.
You know, I wrote a thing years ago about Street View and Yahoo and the sort of thing I was kind of poking away at is that basically every big internet system is a mechanical Turk. Ah. And the question is where do you put the people? Where are the humans? Yes. Yeah. And with Google search, the people are, um, everybody, A, everybody making a link on a web page and B, everybody using Google.
Whereas with Yahoo, they tried to like have a bunch of people in an office. Yeah. Doing it in the middle office, making a hierarchical list of all the websites on the internet, which became impossible. Yeah. And with Street View, you just pay a bunch of people to drive down every street in the world, which is actually not impossible. It's just expensive.
It's just expensive. It's really, it's an interesting computation. It's not obvious that it would be feasible to... It's funny, you know, the Yahoo thing. Yahoo, uh, you know, I think got started in like the early mid-90s, right? I think 94-ish, 93, something like that.
And the thing about it is, it like Yahoo had to kind of get to its limit before it was obvious that you needed something like Google because like, um, web pages had to be suffused with, um, at the time they put on-page spam and so on and so forth. Uh, you had to kind of top out, you had to get enough web pages that the hierarchical model broke down. Um, you had to get enough economic value that people were really incentivized to game the system and so on and so forth before, you know, maybe Yahoo could have self-disrupted, but before something like Google was there, Yahoo almost built out enough of the web economy to make something like Google necessary, you know. Um, anyway, so one thing I wanted to talk about, uh, I want to go through various other areas, but what is AI disrupted? What is AI going to disrupt? Right. So what is it already disrupted? So search, it's taken points off of Google share. Um, you know, like Stack Overflow, you know, their queries are down. Um, image search, because now image search is image generation. Obviously video, obviously, um, many different kinds of specialty apps, uh, will, um, you know, things that are, uh, for example, like various sales tools that make templated emails and things like that, those all, you know, change. I'm not sure Salesforce, I mean, Salesforce is, you know, certainly they're using AI, but the entire Salesforce model, like spamming people with email, I'm not sure that's going to last in the age of AI because you can spam so many of them now, right? Um, so those are some of the, you know, obviously robotics, obviously protein folding and whatnot. Um, what is it going to disrupt that people haven't thought about yet? And I can give some ideas as well.
Well, one answer is we don't know. It's like trying to ask that question about the internet in 1994.
Sure. And the joke is always that newspapers thought the internet would be great because they'd save on printing and they at first probably thought it was good for them.
I did a slide in my, the last presentation I did because it struck me that people would always say, well, you know, Uber didn't sell software to taxi companies and Airbnb didn't sell software to hotels. They redefined what those things were. So I went and did a chart of, well, what happened to taxis versus what happened to hotels? Actually rather unsurprisingly what happened, um, taxi medallions, Uber demolishes taxis mostly. Um, Airbnb is mostly additive to hotels. Why is that? I think because Airbnb is a different kind of experience in a hotel. It's not a substitutional experience in the same way. It's complementary. Half of business travel, half of hotels are business. There's another whole bunch that's conferences. There's a bunch that's about like, I mean, just, you know, okay, so two examples. Like my fiance works for, goes to fly to Milwaukee. She arrives in town at 10:00 at night. She needs a gym. She's got a client meeting the next morning. And then she's got a flying back to New York. She doesn't want to go and stay in some random stranger's hotel, which she's got no idea what it's going to be like. She wants, you know, a very specific brand promise from a random stranger Airbnb. She wants to stay in a hotel. She will stay in a hotel. She's not going to stay in an Airbnb. Um, the other side of this is I think there's a more general point and same thing. I arrived in Singapore at 2:00 this morning. I'm not going to go and work out whether this Airbnb is any good. I can stay in a hotel. I think there's a more general point which is that like everything is probably disruptive to someone at some point in the value chain, but it kind of depends on the industry quite how much and in what sense. So like the iPhone demolished the existing cellular industry, didn't really have any effect on telcos. Telcos kind of hoped that they were going to do all these services, but that was never going to happen. But mobile operators today are basically the same companies that they were 20 years ago with more basically the same share price. Um, because their business was not in anything that the iPhone changed except that they're providing massively more data than they were in the past. The business is basically owning sites and owning spectrum and connecting them up and selling that to consumers. The same thing with like online travel booking completely demolished the travel agent industry, didn't really change the airline business. And airlines had to do a bunch of stuff around loyalty and pricing and maybe pricing became much more transparent and so on. But at the end of the day, their business is owning or leasing airplanes and buying fuel and owning landing slots, right, and maintaining the aircraft. Um, and so there now, of course, the counterargument would be you could have looked at taxis and say, well, clearly that's not going to get changed by the internet except maybe you'll be able to book a taxi more efficiently until it comes along and changes it. But the point is you can't, there's this sort of very naive view that says, oh, well, the software will just destroy everything, right? Right. And the answer is, well, it kind of depends. It's path dependent.
That's true. And like one of the ways that I sort of think about this is that like the tech industry kind of comes and changes everything in industry and resets how it works and then leaves and goes off and works. And, you know, the joke about how consultants are seagulls.
Yeah. They come, they fly, crap everywhere, make lots of noise and fly out, right?
And so if you think about what happened to books or music, no one in the tech industry cares about music anymore, right? Like...
Well, yeah, Spotify does, but it's not the main event.
Yeah. Recorded music is like 20 billion dollars a year. It's like a rounding error in the scale of the tech industry. It has no, streaming means it has no strategic leverage for Apple or Google. Suno is interesting though. So the AI created, but for the last 20 years, you know, 20 years ago, the internet completely screwed the music industry and since then it left and doesn't care. Same thing in books. Like all the conversations around books right now, some of which were about Amazon, are book industry conversations. I think there's something similar happening now with video generation and Hollywood. Like everybody in Hollywood like got over the panic and now everyone is sitting and looking at this and thinking, okay, well, this saves a bunch of second unit stuff. So one way, but they're all like, all the questions for what does this mean are questions for people in LA. So one way of thinking about it is conversation is proportional to derivative rather than absolute value. So let's say you have a sigmoid that's going like this and then it flattens out, right? So when it's, um, like a nullity or ubiquity, you know, when it doesn't exist or when it's everywhere, when 0% or 100%, it's just not notable. It's not worth talking about, right? People use Uber or Dropbox a lot more today than when they were talking about Dropbox and Uber a lot, right? So the conversation is maximum at the time of maximum growth and then it's just much less because now it's like not notable. It's just a feature of the environment, right? So you can do, um, Google Ngrams.
Yeah. That show exactly this. I think that'd be a great, that'd be a great graphic. So you can do them for like steel. Mm-hmm. Or, and some of these railroads, steel, because it starts in 1800. Um, and of course, some of them you look at it and you go, oh, I'm actually seeing a chart of World War II in some of these where you see steel suddenly does that or shipping suddenly does that in World War, and that's not obvious, right? Because conversations or like attention is focused on change rather than absolute value.
Well, I always used to do a, um, I always used to be fascinated by elevators. I get these kind of autism spectrum fascinations about things. Um, and I, there's a chart I did of the number of people employed in the US as elevator attendants, which is a perfect bell curve.
It's all curves up and down. And this is because first half of the 20th century, you didn't have any elevators. You employ a lot of elevator attendants, right? Second half of the 20th century, they become automatic. You have a button and you can go and find all this advertising. Why were they at the beginning? Was it just like switchboard operators? Was it technical enough?
