CEOInterviews.AI
Start App
Dylan Patel
Dylan Patel
Founder, CEO, and Chief Analyst, SemiAnalysis
Watch

Dylan Patel on Why AI's Biggest Infrastructure Gains Come From Co-Design

📅 Jul 06, 2026 The AGI Post 10 MIN 128 VIEWS 21 SEGMENTS · 3 SPEAKERS
read the full feature: https://theagipost.com/posts/hardware... original video:    • Why Hardware-Software Co-Design Is AI's Re...   Sequoia Capital talks with Dylan Patel of SemiAnalysis about AI infrastructure, inference economics, chip architecture, CUDA, neoclouds, and why the biggest gains come when model architecture, kernels, interconnects, and hardware are optimized together. 0:00 - AGI Post feature recap 0:03 - Sequoia source and Dylan Patel setup 0:44 - Founder story: truck tent, cheap motels, semiconductor conferences 0:59 - Recap: the chip stack and co-design thesis 1:59 - Idea...

What Dylan Patel said

Written from the verified transcript and checked against it. Every figure links to the moment it was said.

Dylan Patel argued that AI progress comes from co-designing model architecture, serving software, interconnects, and hardware as one system, not from any single layer. He disagreed with the host's premise that recent gains were mostly hardware-driven, citing a roughly 30x improvement on DeepSeek from Hopper to Blackwell, but noting model-layer gains like smaller Qwen models (27B parameters, 2B active) were even larger. He described a throughput-interactivity curve that shapes all infrastructure decisions, from low-latency speculative decoding to batch processing. Patel projected 20 gigawatts of data center capacity added this year and over 30 next year, yet argued demand could still outpace supply as models like Mythos 5 and Fable 5 expand the TAM far beyond Opus. He contrasted Nvidia's NVLink switches (72 GPUs) with Google's switchless ICI (8,000 chips), explaining how these choices favor different model architectures. He predicted many custom chip efforts will hit local minima, leaving a durable market for general-purpose AI compute.

Key takeaways

  1. Hopper to Blackwell delivered roughly 30x improvement on DeepSeek's most optimized deployment.
  2. This year adds about 20 gigawatts of data center capacity; next year over 30 gigawatts.
  3. Nvidia's NVLink connects 72 GPUs via switches; Google's ICI connects 8,000 chips without switches.

Numbers and commitments

FigureWhat it refers toTypeAt
30x improvement on DeepSeek from Hopper to Blackwell metric 2:03
27B parameters total parameters in a smaller Qwen model metric 2:03
2 billion active active parameters in that Qwen model metric 2:03
20 gigawatts data center capacity added this year guidance 4:39
30 gigawatts data center capacity added next year guidance 4:39
72 GPUs GPUs connected by Nvidia's NVLink metric 6:43
8,000 chips chips connected by Google's ICI metric 6:43

Chapters

  1. 0:00Co-design as the real driver
  2. 1:59Hardware vs model layer gains
  3. 3:16Throughput-interactivity curve
  4. 4:39Compute supply and demand crunch
  5. 6:08Nvidia vs Google network topology
  6. 7:59Custom chips and local minima

Questions asked in this interview

4
  1. 0:44You might be asking, what does this have to do with hardware-software co-design?
  2. 3:17Is that the most important one?
  3. 6:22Why is that? What's the fundamental difference there?
  4. 6:39So you think OpenAI and Anthropic are converging to very different model architectures?
Narrator 0:03 ↗
Today, we're covering Sequoia Capital's YouTube interview, why hardware-software co-design is AI's real 100X, with Dylan Patel of SemiAnalysis. He unpacks the AI supply chain, chips, memory, networking, power, data centers, neo clouds, and who actually gets capacity. But the part we found most interesting is his co-design argument. The biggest gains do not come from a better model, a better kernel, or a better chip in isolation. They come when model architecture, serving software, interconnects, and hardware are optimized as one system.
In the conversation, Dylan told the story of living in a tent on the back of his truck and in cheap motels while blogging, researching, and listening to semiconductor textbooks. Later, he would travel to more than 40 conferences a year. You might be asking, what does this have to do with hardware-software co-design? This is what gave Dylan the deep appreciation for the gnarly interconnected nature of the chip stack. Tools, chemicals, fabs, software, and ultimately is what led to the success of SemiAnalysis.
He formed one of the earliest and most robust thesis on AI co-design. The idea that you cannot cleanly split progress into model, software, and hardware buckets. The big jumps come when labs co-optimize all three. First, Shawn asks whether the recent gains mostly came from hardware. Dylan disagrees. He says Hopper to Blackwell helped a lot. He also says smaller models got much better, and DeepSeek got faster because the model shapes, kernels, and chips fit together. Here he is explaining that answer.
Dylan Patel 1:59 ↗
Yeah, Shawn, I completely disagree with you, by the way.
Shawn 2:01 ↗
That's great. Great, that's why I'm asking you.
Dylan Patel 2:03 ↗
Okay, so I think one way is to look at it as these three different layers. In that sense, from Hopper to Blackwell, which is all we've had over the last 3 years, roughly 30x improvement on DeepSeek. On the most optimized deployment, which you can see on Inference X, there's about a 30x improvement. But over the last 3 years, we've had way more improvement in intelligence per watt, a lot of that coming from the model layer. If you look back 3 years, it's GPT-4. Now it's like, maybe like Qwen, one of the smaller Qwen models that's 27B parameters total and like 2 billion active is way better. So you've got this huge improvement on model layer, you've got this pretty sizable improvement on hardware, but it's that co-design layer, and I think that's what's important. If you look at the architecture of any of these models, but DeepSeek is the most famous one that's public.
Narrator 2:53 ↗
Second, he says, serving AI is a choice between speed and packing more work onto the same compute. If one user needs a fast response, you use a small batch and spend more. If you are processing documents overnight, you pack many users together and save cost. Here he is explaining that curve.
Dylan Patel 3:16 ↗
Peak.
Shawn 3:17 ↗
Is that curve like the most important curve in your opinion? The throughput and your interactivity curve? Is that the most important one?
Dylan Patel 3:23 ↗
Yeah, I think most things in hardware infrastructure, model application layer, everything is downstream of that curve. Is it something that needs to be super fast, super low latency, and I don't really care about the cost, so I make batch size very low, and I use techniques like speculative decoding or multi-token prediction heavily, and there are so many possible techniques there. Or is it something where, actually, I'm batch processing a ton of documents, and I don't really care about all these things. I don't use these techniques that actually are worse on cost efficiency to help you with speed for an individual user because I just want to pack a bunch of users. I don't care if the document takes all night to process. And right now the way we treat AI infrastructure is it's like one size fits all, but over time we're going to get to the point where there's stuff where you have batch workloads or you need instant response and there's the whole curve that's going to matter for users. And so we see this with Anthropic, right?
Narrator 4:15 ↗
Third, Dylan says more compute is coming online, but demand can still stay ahead. He says this year may add about 20 gigawatts of data centers, and next year may add more than 30. But when a model gets much more useful, people find many more tasks to run on it. Here he is explaining why that can keep the crunch going.

10 more exchanges in this transcript

Sign in free to read the rest of this interview. No card required.

Sign in to read the full transcript

Cite this transcript

APA, MLA, BibTeX
APA

Patel, D. (2026, July 6). Dylan Patel on Why AI's Biggest Infrastructure Gains Come From Co-Design [Interview transcript]. The AGI Post. CEOInterviews.AI. https://ceointerviews.ai/interview/1085937/

MLA

Dylan Patel. "Dylan Patel on Why AI's Biggest Infrastructure Gains Come From Co-Design." The AGI Post, 6 Jul. 2026. Transcript, CEOInterviews.AI, https://ceointerviews.ai/interview/1085937/.

BibTeX 
@misc{patel2026_1085937,
  author       = {Dylan Patel},
  title        = {Dylan Patel on Why AI's Biggest Infrastructure Gains Come From Co-Design},
  howpublished = {Interview transcript, The AGI Post. CEOInterviews.AI},
  year         = {2026},
  month        = {jul},
  url          = {https://ceointerviews.ai/interview/1085937/},
  note         = {Speaker-attributed transcript with timestamps}
}