I think it's really fun inside of SemiAnalysis because we have 90 people, and a big chunk of them are technologists and engineers across the whole supply chain. And then a big chunk are people who are formerly at hedge funds. You see these arguments organically: people say, 'Oh, that doesn't matter,' and then someone says, 'But cost,' and the engineers say, 'No, no, no, but this technology is the coolest.' It fights itself out. We're pretty informal. Given that I was a moderator, you can imagine. You don't wrestle with a pig because a pig enjoys it, right? Exactly.
We're here in the SemiAnalysis office with Dylan Patel. I'm Sean from Sequoia, and my partner Sonia Huang. It's pretty insane what you've done. Semis five years ago were not very sexy in the West; they were sexy in the East, but people here had kind of forgotten about them. You did not forget. You went very long and created probably the premier research company in the space, educating the world on the state of the art from technical details to supply chain to the bigger picture. There are rumors that SemiAnalysis recently passed $100 million in revenue. I don't know how accurate those are, but whatever the numbers, you guys are crushing.
It's as accurate as the information is.
Yeah, cool. There are also rumors that you might start a venture fund. I hear all the time in the ecosystem people wanting affiliation with SemiAnalysis. You've built this trusted brand, and whatever you do, it's working. This is clearly just the beginning of the journey for you. Congratulations on all of that. But how did this happen? What's your background? How did you get to where you are now?
Well, when I was a young boy coming out of the womb... No. So I grew up in a small business. My parents had a motel; we lived in the motel and ran a gas station. I joke that the first neural network I trained was racially and visually profiling people based on when they entered the gas station to pick which cigarette. The cigarettes were extruded across the top, and I was too short to reach them. Technically it wasn't legal to sell cigarettes at that age, but I had to move the step stool over to the right area.
I started working my first job before it was legal too, so it's good experience.
Well, I didn't get paid. It's a family business.
We had our motel and across the street was our gas station. Someone would walk in, and if an old white lady with curly hair walked in, I'd move the step stool over to where the Camels were. Different age, demographic, profession, race—I'd move the step stool. I joke that's the first neural network I trained because if I waited for them to tell me, I'd have to move it over and step up versus just being ready. Menthols versus 100 slims, all these things. I grew up in family businesses, lived in a motel. It all goes back to my 8th birthday in May. In April, the Xbox 360 was announced. For my birthday, I didn't ask for the Xbox; I asked for it for Christmas. I thought there was no way they'd give it to me for Christmas, so I asked for my birthday. Christmas came, I got it. Fast forward a couple months, my cousin from Alabama, who also lived in a motel, was coming for spring break. I bragged many times on the phone, 'Yeah, I got an Xbox.' Then the Xbox broke—the red ring of death. I had to open it up and short the temperature sensor, which fixed it. That's how I got into hardware. I opened Pandora's box. By 12, I was on forums reading and posting a lot. This was around when Reddit ate all other forms, so I became a moderator of Android, Apple, Google, and hardware forums, watching Intel, Nvidia, AMD. I built a PC. I was watching, reading, posting a lot, and moderating some. I watched smartphones develop from very simple to technologically more advanced than PCs in many ways architecturally, and same with GPUs. I always had an economic tinge because I grew up in a small business. There was a time when neckbeards on the internet loved AMD GPUs, and I personally bought an AMD GPU for price performance, but when it came to what's technically better, I'd say Nvidia is better because they use a smaller chip for better performance and power efficiency, and their margins are better. I'd talk about how Nvidia's margins were better than AMD's in the GPU landscape. It was very fun.
And you were 12 at the time. I started moderating when I was 12, but this is all through my teenage and high school years. Do you have any other weird hobbies, or was it just semis?
I played a ton of Starcraft. At one point, I was grandmaster on the North American ladder in Starcraft 2.
Very serious. So you've gotten obsessively good at multiple things.
They were decent. I had mostly A's, but classes I found boring, like Spanish, I got not the greatest grades. I speak fluent Spanish by the way, so it's really dumb. I didn't learn Spanish till later. My grades were fine enough for Asian parents. I was better than most of the school, but I wasn't try-hard maxing for all A's.
Okay, so you're very much a student of the internet. That's how you developed this expertise. At what point did you decide to start SemiAnalysis, and what's been the biggest surprise since starting the company?
I went to school and got a few degrees in stuff not related to semiconductors. I was a quant for two years at a small quant risk firm. Then a culmination of events happened. I got screwed out of a bonus; I made my company many millions of risk-free revenue by exploiting a market thing, well over $10 million, and someone else took credit. I lost the social contract with the company. Also, my grandparents lived with us in the motel. My grandmother got dementia, forgot who I was, fell down stairs, and passed away. That happened in early 2020. There were also some girl things. I was very sad. Then COVID happened, and my brother said, 'Just come stay with me.' He lived in Nashville. I thought lockdowns would be a few weeks, but they lasted much longer. I was at my brother's home, tiptoeing around, but I didn't care about my job. I posted even more than normal, traded stocks, made a lot of money shorting and longing COVID. Semiconductor shortages happened. I was obsessed with posting. Someone I argued with on the internet doxed me, revealing my anonymous identity. I stopped posting for three weeks, then thought, 'Why do I care?' I started posting under my real name and made a blog, SemiAnalysis. On my 24th birthday, I posted two blogs. They weren't great, but they were the best stuff you could find on the internet about semis. I kept posting, got consulting business. In 2020, I was crashing out, didn't know what to do. I packed my truck, bought a tent, and drove around all the national parks in America. Two or three days a week I'd stay in a random motel for $30 a night, working on stuff. On weekends I'd read textbooks in national parks or listen to audiobooks about semiconductors and AI. I got way more educated over six months. I was alone the whole time, posting blogs. Everyone was like, 'What the hell are you doing?'
Pre-Starlink or the very early days of Starlink?
Pre-Starlink. It was very much like, 'What are you doing?' I traveled around Latin America for a year with a friend and then with my ex. From 2021 to 2024, I've been completely homeless since mid-2020, traveling to every conference in the world. I go to 40-plus conferences a year, no matter where in the supply chain. I went to one conference and thought, 'Wow, this is amazing.' You get to talk to experts, and they're happy to talk because in semiconductors everyone's a boomer and they don't see young people excited about it, so they're really happy to tell you stuff. You just have to ask.
Was there a part of the supply chain or one of these conferences that particularly changed your view of the semi-world, or that you feel is particularly underrated?
Trade shows and conferences range widely. The ones I have the most fun at include NeurIPS because it's 20,000 AI researchers in my age range, and it's a lot of fun and you learn a lot. There are also parties. Then it ranges to random chemical conferences in Japan with 300 Japanese dudes, 20 guys from ASML, 20 from TSMC, 20 from Intel—the only ones who speak English. Everyone else speaks only Japanese, but it's still interesting. I have a skill set of bonding with anyone regardless of background. The most interesting conferences are often the really big ones because that's where the biggest stuff is happening. But the niches that are really exciting are SPIE conferences—super deep in details. The first time I went to SPIE Advanced Lithography, I didn't understand 90% of what I heard. I read a lot, and the next time I understood half, then 75%. Even now, I still don't understand everything. Whereas at NeurIPS, you can get a mapping of everything pretty quickly. Some parts of the supply chain are so arcane and technical that it takes a long time to even understand what's happening. You go to a conference to understand the research, but what you really care about is how that research intersects with technology and how it differs from today. None of the research papers tell you what's happening today, so you ask people, build contacts, and learn about the supply chain—like which company supplies which, even if it's not publicly stated, or how much a chemical costs and how much a tool uses.
You hear horror stories of a chemical having a shortage that totally threw off part of the supply chain, and it turns out there are only three companies in the world that make that chemical.
My favorite one: I learned from a Japanese guy at that Japanese conference, in very broken English, that his father worked in the industry in the 1980s. The only factory in the world that built a certain chemical burned down, causing memory prices to double or triple. I thought, 'Wow, not too different from today.'
Not at all. Crazy. Um, inference going to be the biggest market on Earth, biggest market beyond Earth—agree or disagree?
Obviously, the use of tokens is going to be the biggest market, and the value created from tokens is going to be the biggest market. I think tokconomics—the use of tokens, adoption of AI—is the most important thing happening. Inference, whether open or closed models, will be one of the biggest markets in the world, much bigger than oil, many percentage points of GDP.
What you've done with InferenceX is industry standard. Say a word on why you started it, what it does, and what people misunderstand about performance benchmarking on inference.
SemiAnalysis does a lot of research for institutional clients and subscription products, but also things that would just be cool to figure out. We've done a lot of GPU benchmarking for training and inference performance. We saw that inference benchmarking was point-in-time—you test, release, and it's outdated because models change all the time. Every week there's a new model, and software layers like PyTorch, vLLM, SGLang update twice a week. Performance changes constantly. New inference optimizations keep driving efficiency and cost down; we've seen model cost drop for equivalent quality by 60x a year. To stay on top of that, you need living, breathing benchmarks that run constantly on the latest hardware and models. We embarked on a project with buy-in from the ecosystem. CoreWeave, Crusoe, Nebius, Oracle, Microsoft, Amazon, Google, and OpenAI contributed compute. We worked with SGLang, vLLM, and now Radix and InRact. Nvidia, AMD, Google, and Amazon are collaborating. We have over $50 million of hardware donated; once we add TPUs and Trainium, it should be over $100 million. About 15 different chip types run these benchmarks every day on the latest models from Moonshot, Alibaba, Chinese labs, and US open-source models like GPT-OSS and Neotron. We run them in an automated fashion on dedicated servers, sweeping across many configurations and optimization types. All results and configurations are public. We created the Pareto optimal curve because people often compare suboptimal points. We open-sourced containers for the optimal points across the interactivity vs. batch size curve. Anyone can go to InferenceX, download the optimal point, and run it. They can check daily or auto-download the most optimal point for their model, and their inference performance will be near peak.
Is that curve the most important curve in your opinion? The throughput-interactivity curve?
Yes. Most things in hardware, infrastructure, model, and application layers are downstream of that curve. Is it something that needs to be super fast with low latency? Then I make batch size very low and use techniques like speculative decoding or multi-token prediction. Or is it batch processing a ton of documents where I don't care about speed? Then I pack a bunch of users and don't use those techniques. Right now, AI infrastructure is one-size-fits-all, but over time we'll have batch workloads and instant response needs. The whole curve will matter. We see this with Anthropic's Claude Code Fast mode costing more than regular mode, and OpenAI's priority queue.
Sorry, dumb question. How does cost factor into the chart? If I have a batch size of 100,
And I can do 10 tokens per second per user, so a thousand tokens per second total. That's one side of the curve—super slow. The other side is 500 tokens per second for one user. There are points in the middle that are more Pareto optimal. The average person wants 50 to 100 tokens per second. So the curve shows a thousand tokens total per second versus 250, depending on how many users I batch. Some workloads want the 4x cost decrease because the same hardware can do a thousand versus 250, and some users will pay 4x more because they care about time. If you had to guess, what percent of inference compute will happen in space in 10 or 15 years?
This is a tough one. You choose the time frame.
Not investment advice. Thank you. I don't think space data centers will really matter in the next 3 to 5 years. But in 20 years, I think the vast majority of compute will be in space. The real factor is the cost of building power on terrestrial land and how much power you can do there. My views on how many gigawatts or terawatts are devoted to inference is a crazy curve for me personally.
What's your forecast? How many gigawatts?
By 2030, just OpenAI and Anthropic will have over 100 gigawatts combined. Add Meta, Google, and so on. It's a humongous amount of compute dedicated to inference. By 2040, it'll be terawatts. The curve of productivity we're going to get is huge. So inference deployments will be huge. By 2040, probably more than half of incremental compute will be in space, but by 2030, it's sub-1%.
Do you think intelligence per watt has been increasing? There's still a giant gap between where we are and human biology. Do you think we'll close that gap, and where will the gains come from?
It depends on what you're doing. A TI-84 is way more intelligence per watt for math than us, and it's 30 years old. For general intelligence, InferenceX measures power and cost of hardware. We offer cost vs. interactivity and power vs. interactivity. Intelligence per watt has been increasing. We've seen a 60x cost decrease for the same benchmark level, and intelligence per watt has improved closer to 40x. Some efficiencies are non-power ways, but there's been a huge improvement annually. I expect that to continue. We're many orders of magnitude away from the human brain, but that doesn't matter because we can devote a lot of power to computers. It's much easier to power computers than human brains—we have sickness, disease, food preferences, sleep.
Uh, yeah, exactly.
Let me ask one more question on the general theme. In terms of intelligence per watt or per dollar, there are three levels of input: hardware improvements, low-level systems optimizations like kernel-level matrix multiplication libraries, and high-level model algorithmic improvements. It seems to me that in the last three years, most gains have come from hardware and some from the model level. Do you agree? What does the future look like? Is there juice to squeeze at the kernel level?
Yeah, Sean, I completely disagree with you. Great, that's why I'm asking.
One way is to look at these three layers. From Hopper to Blackwell over the last three years, there's roughly a 30x improvement on DeepSeek's most optimized deployment. But over the last three years, we've had way more improvement in intelligence per watt, a lot coming from the model layer. Three years ago it was GPT-4; now a smaller Qwen model with 27B parameters and 2B active is way better. So huge improvement on the model layer, sizable improvement on hardware, but the co-design layer is what's important. If you look at DeepSeek's architecture, it's co-optimized. The shapes of experts in DeepSeek V3 were optimized for Hopper, and V4 for Blackwell and Huawei's chip. TPUs are objectively amazing chips, but they suck at running DeepSeek, while they're great at other models that don't run well on Nvidia. There's deep optimization in shapes, network IO, collectives, arithmetic intensity of attention—all co-optimized between model, hardware, and infrastructure software. It's hard to disentangle the gains.
Yeah, DeepSeek got huge efficiency gains from co-optimization or kernel-level memory optimization.
Yes, it's kernels, but you build the hardware architecture for the chip. The shapes of all the experts in DeepSeek V3 were optimized for Hopper, and V4 for Blackwell and Huawei. TPUs are great for other models but not DeepSeek. There's co-optimization across shapes, network IO, collectives, arithmetic intensity. It's hard to disentangle.
Do you think China has done this a lot better than the West in the last few years? DeepSeek was one of the first models to really do this.
I don't necessarily think so. The West doesn't tell people what they do. OpenAI didn't disclose how sparse GPT-4 was or its shape size. GPT-4 is roughly the same size, slightly smaller than DeepSeek V3, and came out a little earlier.
So is your view that all three things have been happening simultaneously at roughly the same rate, and the biggest gains come when you co-optimize?
I would say there have been more gains on the model layer than on the co-optimization or software infrastructure and hardware layers. But there have been innovations on every layer. The biggest gain and the beauty of the best labs is when they co-optimize all three. Anthropic uses many kinds of hardware; they train on TPUs and inference on Trainium and GPUs. They've optimized their hardware, model, and everything. OpenAI's prior models were optimized for Hopper, now more for Blackwell. Google optimized Gemini 2 for TPU v6e, Gemini 3 for TPU v7. When you pull that model and run it on old hardware, it's not great. Co-optimization is the most important thing—software-hardware co-design. The real breakthrough is when you leapfrog a few layers and co-optimize, turning 2x here, 2x there, 2x there from multiplicative 8x into 100x because you've optimized across all three layers. That's what's exciting at labs like Nvidia, which co-optimizes from the model layer down to silicon, or TSMC, which co-optimizes from fabrication all the way upstream to customer designs.
