FULL INTERVIEW: Dylan Patel Says We’re Still Underestimating AI

When are we going? Me, you, the International Space Station. Let's break it down. You know, the space tourism industry is quite a fun one, right? Would you go? Would you do the Blue Origin thing where they blast you out past the Karman line? Good enough for Katy Perry? It's not good enough for you. What's going on?

Yeah. Yeah. It's not enough just to go back. I think I would do it. It's even... It's like 90 seconds, right? Yeah. But it's better than being hanging out at... all the cool stuff that astronauts do, right? Like, you know, put water and then like they're bubbling and then you like try and drink the water or like... So they'll be unplugging the GPU, plugging it back. Oh, yeah. Yeah. Yeah. Yeah. Yeah. That's how you pay for your space tourism. You got to go unseen ship 90 seconds of service.

So, I think everyone freaks out. Oh my god, putting stuff in space is expensive. But if you look at like Starship launch cost and they keep falling, you're fine, right? I think by the end of the decade the cost of space launch will be fine. The heat dissipation, I mean it's a challenge but you just put a massive, massive effectively radiator and it's fine, right? By the end of the decade you'll be good. I think the big challenge is that chips are just really unreliable, right? And so how do you deal with a couple things, right? Satellites can only be so large before they need start needing a lot of support and structure before they tear themselves apart. So when you look at the launches, these things are shooting out tiny satellites and many of them. Okay, so you can't have a big fully connected cluster of chips. And then on top of that, how do you deal with any random error? On Earth you have techs running around the data center unplugging stuff, putting in spares, things like that. What do you do in space? You RMA it to the factory where they might unsolder it and resolder it and then test it and it works and go back out. Sometimes it is just trashed. But that's the challenge to me. I feel like maybe the pattern we should be looking at is like how often do the Tesla self-driving chips need to get serviced because that's the team that would probably be building or bridging the gap there. Like the Starlink satellites, sure, they go down, but the service works, you're just relying on some sort of 90% uptime stuff's coming down, but most people that are in a Waymo, the chip keeps working, right? Most people that are in a Tesla self-driving, you don't hear about Tesla owners being like, 'I love FSD, but I'm constantly in the shop getting my custom silicon chip unseated and reseated,' right?

There's this whole like, if you look at Nvidia GPUs, when you first turn on the cluster about 10 to 15% of them fail RMA in the first two weeks. Wow. And that's fine. Like you have to reseat them, whatever. And the industry knows how to deal with this, right? And over time Hopper is now at 5% but Blackwell's still 10 to 15%. Right. Actually started out higher than that. And when a new generation comes out, it's going to be higher than 10-15%. It'll have its curve gradually decline down. But who's going to... are you going to test it and burn it in on the ground, or are you going to say 5% of my chips or 10-15% of my chips are trashed because someone can't go up there and do these things or am I saying, 'Oh, I need robots who can do all this stuff in space and now that's an additional engineering problem.' When sacks of meat are actually very cheap.

I think it's a big vibe shift from Nvidia, right? Before they were like, 'All right, I got this big GPU. Everyone's going to use this GPU. Software ecosystem of the GPU is so good. It's one size fits all. Everyone try, you know, like everyone's trying to make all these specific point solutions, but we've got the thing that's good at everything.' Okay. And then they had a vibe shift, right? They launched this thing called CPX, okay, which is a chip made for prefill. Okay. Prompt processing, creating a KV cache, and also good at video generation and image generation. And that's coming out later this year. Press release. They were really talking about video generation as well. So, you've got CPX, you've got the standard GPU, now you've got the Grok chips, and they all fill a different niche, but really it screams, 'Oh crap, we don't really know exactly where AI is going,' which I don't think anyone does, right? I mean, it's moving so fast, the software, the model architectures, etc. So, we're just going to engineer solutions that are along multiple points of the Pareto optimal curve and then, you know, one of them will win, right? And I think it's a big vibe shift from Nvidia. Also, they just knew OpenAI was going to do this serious deal, so they freaked out, but...

So, you have people thinking like, oh, latency matters in terms of where our data center is. It doesn't matter at all. What matters is, as we've moved from chat applications which were like search response immediately, chat applications let's say response takes 10, 20, 30 seconds. You've got agents, you know, I don't know, my Claude code is working in the background for a long time, right? Doesn't matter where the data center is, but what does matter is that these streams of inference take 30 minutes versus 10 minutes versus 5 minutes. And for a lot of people, I'm fine to spend 10x the price on something that completes 10x faster. Yeah. And so Cerebras sort of just makes a ton of sense there. So, OpenAI, they've got these long horizon... there's Codeex 5.2 extra high thinking or whatever. It's terrible. Can you guys teach them how to market?

So, I mean Cerebras can speed up multiple users for sure. The question is sort of like where you use it and that's where they have to figure out where within Codeex, right? Because there are times where Codeex is running for like 10 hours and sometimes you don't mind, right? Like screw it, I've put out this nice prompt, go and work on it, refactor my code, do this thing, do this task. Other times I want this iteration feedback loop. So how do you expose it to the user without saying, 'Hey, actually there's another toggle.' So your permutation is 18 times. Well, hopefully like a really robust model router, but it feels like that's been a process. Yeah. So, the OpenAI deal is like for 750 megawatts. It's not that much capacity on the order of what OpenAI talked about. You know, by the end of 2018, they'll be at like 16 gigawatts of that. So, it's like the absolute cutting edge, the most price insensitive customers in that specific use case of this is the type of prompt that you need to return fast, then you'll get the speed up potentially. Right. Right. And they've got to figure out how to do it from a product exposing to the user, etc. But it's clearly something where there is demand, right? Like I don't know, Andrej Karpathy doesn't care if he's spending a thousand bucks per agent per second or whatever, right? Like whoever it is, these super cracked engineers don't care at all and then obviously there's a long tail of like actually cost does matter for most people and so all along that curve they've got to have solutions, right?

I mean this has been rumored for a long time, right? Like people are saying Tesla, Tesla, Tesla for the longest. And it's harder with a public company. Yeah. Yeah. And then a bit ago people were like, 'Oh, SpaceX.' I'm like, 'Wait, this makes no sense.' But there was a very coordinated narrative pump. Oh, yeah. Like data center at the end of last year and it was like almost perfectly telegraphed.

Yeah. So the V2 Starlink satellites I think are the only ones they've launched. Maybe they've launched a few V3s but the V3s are coming soon and those are like 100x more bandwidth each. Right. And more power. And just more power. And so when I'm just thinking of like can you scale this thing up at all? It's like are they two orders of magnitude off? Are they three orders of magnitude off being something that looks like an H100? I think the metric is like 50. It's either 50 kilowatts a ton or something like this per satellite for V3. If they let's say from V3 to whatever the compute thing is, they double it again, get to 100. I think the V2s are like 25. So if you get to 100 kilowatts per ton for launch, it's only 150 or so Starship launches. I think that's so reasonable. Maybe not '28, maybe it takes '29, but like you know it's so reasonable. The question is cost and reliability and what happens when the chip fails? How do you service it? That kind of stuff. How do you deal with having clusters be much smaller instead of these big clusters, even for inference big clusters are useful.

Yeah. So for the longest time Google's had one main line of TPUs, right? All made by Broadcom and then sort of next year they've diverged it, right? Where Broadcom makes a TPU and MediaTek makes a TPU. These two TPUs are focused at different things and they're fabbed at... they're both fabbed at TSMC. Everything at the end of the day goes to ASML, right? I want to go there next, but... goes to ASML. So fabbed by TSMC regardless, but both of these TPUs are focused on different things and they've actually got a third project for another kind of TPU. They also see this need to proliferate along the curve of like, hey, do I care a lot about super high amounts of flops, not that much memory? Do I care a lot about super fast on-chip memory only? Do I care about 3D stacking memory? Do I care about this sort of general purpose middle ground AI chip, which is what an H100, a Blackwell, a TPU looks like today? There's sort of like, oh, we need to hit the entire Pareto optimal curve and it's like, okay, within this there's training versus inference differences and like what numeric you want and all these other things. There's so much complexity there. Everyone sort of is diverging their roadmaps once they're at a sufficient scale, I think.

Yeah. So, it's a couple of things, right? In 2023, everyone thought that scaling was pre-training. Yeah. Right. More parameters, more data. And that's very difficult to split across data centers. And has Google been able to do that? And Google's been able to do that to an extent, right? So what they've done is they've got, they don't have the largest individual data center campus, but what they do is they do these regions where it's like, hey, each data center is roughly 40 miles apart from each other. So in Nebraska and Iowa and then in Ohio, they've got these complexes. And now they're building one in Oklahoma, Texas. You know, these complexes where there's all these data centers pretty close to each other. So it's not really cross-data center across the world, right? It's just across like region. Yeah. And then that makes a lot of the difficulties a lot easier. Flip side is we've also moved to RL, right? And majority of the time of the chips is spent generating data, right? Only doing forward passes through the model. And then you only send the final tokens that you verified sort of back to train on to the training, right? So then you end up with like oh instead of in pre-training scaling you need to synchronize all the weights every 10, 20, whatever seconds. When you're doing these rollouts and especially as things get more and more agentic in training, you might not only need to send not the entire weights but just the tokens that are relevant. So way smaller amount of data and way less frequently, right? Minutes at a time instead of seconds at a time. Yeah. And so you've got this like now it's become like reasonable where oh actually multi-data center training is completely reasonable and people do this, people do multi-data center, multi-chip training. Sure. Right. You know you do your inference on one set of chips and you do your training on another set of chips. So like Anthropic does this. I don't know if Google does this, but Google's kind of already got the cards. Yeah.

Yeah. Yeah. So, sidebar before I answer the question because I think it's fun. In the US it's insane to say turn off your AC for AI, right? And the general public hates AI already. But in Taiwan they've had droughts before and they've turned off water to entire cities. They're like, oh, you get water three days of the week and then the fab still gets supplied water. It's like this is, you know, you've got to understand the mindset. We are not ready as weak Americans to do this. No, but at the end of the day, water and power are certainly less big of constraints now. Now you've got to imagine like the semiconductor industry is used to doubling the amount of transistors made every year or two. Part of that is more slow, part of that is more capacity. Whereas the energy industry in America wasn't. And so initially people were not creative, they're like let's do these kinds of gas plants. It's like well no, now we've realized, yes there's three main manufacturers of turbines and then you've for a dual combine cycle. Then you've got IGTS, but you've also got medium-speed reciprocating engines, right? Like turns out Cummins can make like a million diesel engines a year and like those can make electricity. Like if I don't give a and I put it in West Texas, easy. So now it's more of like a regulation thing, a supply chain thing. Power is not a constraint in so far like that much, right? I think it certainly is a constraint still today. It was the biggest constraint in '24-'25 data center capacity power because the industry was not ready. People have woken up. They've been shocked to the system. Now you've got tens of gigawatts being deployed. Next year 30 gigawatts are being added and we think the power is there for it. What was it this year? It's... or this year is like I think it's like 18ish, 10ish. 15 to 18ish, sorry. So, so almost a doubling. Yeah. Almost a doubling. Yeah. Wow. And when you look at TSMC and the crew, there is not really oh this random, you know, there's 12 people making medium-speed reciprocating engines that you can now convert to make power at some random data center. No, no, no. There's ASML, right? There is one set of spice, like, you know, that's it, right? And so, and then the flip side is like, okay, when you have 12 vendors, everyone's got a little bit of slack capacity, you know, there's more likelihood, you know, people are like, oh, turbines, you can't get... you can call a broker and you can get a turbine. You might be paying 50% more, 2x more, but you can get a turbine. Yeah. Right. Like it's... you can't get a 3-nanometer fab. You cannot get a 3-nanometer fab. Exactly. And so when you talk about what's the... the baton got passed from semiconductor shortages in '23 to power and data centers in '24-'25. '26 we're still swinging the pendulum but it will fully beat semiconductors again in '27, right? And so we see this across the entire space of the ecosystem. It's not just TSMC, it's also memory, both... because both of them have built at a certain pace. Now TSMC has been expanding at some rate. Mhm. The memory makers in fact have just not expanded capacity basically. They've not built new fab since 2022 because their cycle is so undulating. Yeah. And so when you look at it, it's like, oh, even if they wanted to double capacity, they need to build the fabs. Yeah. Right. And building the fabs, it is the most complex building humans make, right? It's the entire air of a clean room circulates itself every 1.5 seconds. What? And you don't even feel it when you're inside. Really? It's like that. And it's like parts per billion of particles, right? Like it's actually insane how you could get coughed in the face by someone who has COVID and not get COVID because it gets circulated so fast it doesn't even hit you. It's like that meme of like the spraying when someone's talking and then it just gets circulated. So another sidebar is everyone knows COVID really popped off in Wuhan, right? Wuhan also is home to China's largest memory company YMTC. And so when they were welding people into their homes, the people who worked in the fab still went to work. Wow. It was because it's, you know, one, it's national importance, but two, like these people aren't getting sick. This fab is way too clean. Crazy. Sorry, Jordy.

It's terrible comms. Yeah. Like I told my Oracle context like who the hell is in charge of the Twitter? Like what are you doing? Nvidia did something similar last year when the whole TPU mania was going on. It was... yeah. It was like we're thrilled with Google's progress with the TPU. That said, Nvidia chips are the only... it's like no one asked you to comment. I mean, like I'm sure a handful of people in your DMs and random, but that doesn't mean... doesn't project confidence. It's sort of the lion shouldn't concern themselves with the sheep and like okay, Nvidia is a lion. Maybe Oracle is a little bit more bumpy, but I think Oracle is fine. People are just freaking out because OpenAI is peak, you know, people are peak negative on OpenAI right now because of how good Anthropic's been killing it. Yeah, I think it's just kind of silly like they need to...

They need to hire someone to do comms, like a Lulu or something, right? Both Nvidia and Oracle, because what are you doing?

It's, you know, Jensen's not been as famous as other CEOs for as long, and yet he's so important now. And if you know Jensen, how he is in meetings, I feel like there's two Jensens, right? There is like PR, good at PR, just good at talking, good at making people hyped up and believe what he's doing. He's great at standing on stage holding up the chip, delivering like a sermon.

And then there's the real Jensen, which is like a business killer and actually just knows about every aspect of the supply chain, right? All the way from niche semiconductor design and manufacturing stuff all the way to energy, power, data center, and then doing the business deals too, right? And so you've got this whole range of things that he's good at and he's a killer in. And clearly he was in a meeting where he was being a killer and negotiating supply contracts or something.

And that's why he was like, you know, still a killer. Like, no, we never said we committed to a hundred billion, you know. And it's like, I don't know, where did you even get the hundred billion dollar number from?

And it's like, well, you did go on CNBC and make a big deal out of it. I think people would assume that it was, but they did say in the press release, and remember these are early talks, but they just kind of jumped the gun. This was the height of the press release economy.

Good energy back at home with the roommates.

I wanted to... yeah, I got one more thing. So yes, over the weekend it was sort of drowned out by all the Justice Department stuff, but...

No.

I didn't realize that was related. Yeah, that makes a lot of sense.

Yeah. What does the fab look like if you have no humans inside? Like that's probably his long-term thing is like, yeah, there will be an Optimus. No one. Like the number of people working in a fab is like irrelevant.

Well, it's like a nesting of cleanliness, right? For example, you've got this wafer you've put down, let's say you put down copper and now you're moving it from one area to another. Well, it needs to be stored in a vacuum, but easiest way to store a vacuum or an inert gas. And that's like the thing that's being transported in, but then around that you want it to be super clean as well.

If you don't, then the copper starts getting oxidized, it affects your yields, all this sort of stuff happens. And so, you kind of want it to be a nested layer of like, well, this thing inside the FOUP tool is super clean and then the thing feeding it is super clean and then the thing it sits in is super clean because that's how you get to zero particles. Well, because in the transportation devices it's like parts per trillion, and maybe FOUP, it's called FOUP, Front Opening Unified Pod, but it's called a FOUP, it's like the thing that moves and it carries the wafers. And then the fab is like parts per billion, and you've got this nesting relationship so everything is super clean.

I'm bullish on robots, like super bullish on robots, but only for like, not for fabs that have like TSMC's Arizona fabs or, okay, let's say TSMC Tainan, which I think produces like, you know, indirectly hundreds of billions of dollars of global GDP. Even directly, it's like still tens of billions of dollars, has like 5, 10,000 people in it. Like it's like irrelevant in terms of the number of people who work there in terms of the overall economic value that's created, right? It's like how many people fold laundry or how many people wash dishes or how many people do construction work. Like these are way bigger markets for robotics.

Yeah, I think it's like a curve of what they will accept. It's, you know, one, you push someone into the corner, they're going to start swinging, right? And I'm very concerned that China does this, right? Do you push them too far into the corner? Do they say screw this, we're going to start being a lot more aggressive? We're going to do more military actions or even just invest twice as much in global supply chain take over Africa more than they already have, Latin America, etc. Or just take over Taiwan, right? Because if I can't have the chips, what value is there in Taiwan existing in its current state, right? So there's this game theory aspect.

At the same time, you don't want China to be able to, if you believe AI is going to do what I think many at least in San Francisco think it's going to do, which is completely revolutionize humanity and cause GDP growth to accelerate, do you want to have China also own that technology and their ability to integrate that into their military and all these other things much faster? So there are these competing interests where is the right line? And some people think it's like, hey, yeah, sell them AI model... well, I think Dario would say don't even sell them AI model access.

Yeah, I think so. I think Anthropic does not sell AI access to China. They loop it through, and you can see this in the traffic data, they go through Korea and Japan, other places, but like, and so they get it.

And then the other side is sort of like I think the Ben Thompson view, which is, and I think I'm more sympathetic to that although I think I'm not exactly aligned with that, which is like, and we've been saying like don't sell them equipment, don't sell them equipment, don't sell them equipment. And my view, my argument is more economic in the sense of like, if you sell them tens of billions of dollars of equipment they can make hundreds of billions of dollars of AI value or chips with that equipment, whereas if you sell them AI model access and it costs them this much to get the economic, you know, they're not able to...

Yeah. And so that's sort of the question that is at foot here, right? Do we want them to capture all this value of the supply chain and equipment or by buying the chips or using the models and services? And we've seen across many stacks, China refuses to accept using American ecosystem and they'll wait many years before they develop their own. Whether it was like, hey, they didn't use Windows, they figured out a bootlegging economy, or they didn't use Visa and eventually they came out with Alipay and WeChat Pay or whatever it's called, and these things are way better than Visa in fact, right? Lower transaction cost and higher volume.

Really? Yeah.

I'm a fan of TempleOS, you know.

Okay. Yes. Yes. So, I think everyone's like, cloud code is for coders, and it's like, no. No. Cloud code is for people who don't code now. Yes. Right. And that's the big realization this year.

We've got a couple folks now in the firm who have psychosis, but Douglas Olaflin, who is like, you know, SemiAnalysis number two. He's president, you know, he's my boy. He's the one, in fact, he's the one who encouraged me to make a Substack a long time ago.

I had a WordPress blog and I was consulting on the side, but I was like, okay, let me do a Substack now. Because I saw him making money off it. I was like, why are you getting paid for this? Like there were multiple times where he wrote something and I was like, I could do way better.

And like obviously it was good because we both taught each other a lot of things and we've been great friends and eventually he joined SemiAnalysis. But his background is he was a hedge fund analyst and then he decided to do a Substack, walk, hike the Continental Divide Trail for like six months walking from Mexico to, you know, and then came back to doing Substack and tried to do a fund.