Hello everyone. Thank you so much for coming, and I think I made it by one minute, right? So not too late, not too late. So I wanted to talk through, you know, sort of from silicon to systems and what we do at SemiAnalysis. So SemiAnalysis, we're now 110 people across the world, 15 countries. We focus on tracking the AI infrastructure supply chain from fabrication equipment all the way upstream through to the testing of models across evals and such. We focus on having a high percentage of the company being engineering backgrounds. So people who have worked all the way from the equipment companies like ASML, Applied Materials, Lam Research that make the equipment that makes the chips, all the way through people who have worked at, you know, at fabs like R&D at TSMC, R&D at Intel, up through people who've designed chips at companies like Nvidia and people who've done chip design at other companies, through people who have deployed infrastructure at Google, Amazon, Microsoft, people who have worked on compilers, people who've worked on models, as well as people who have built data centers and power plants. And so we try to be very technical, but also keep track of the financial side of the industry as well. So bridging from, you know, the systems to technical to financial to the whole co-design stack. You know, we try to go to every conference in the world. So if you don't see us, please let us know and we'll go to that conference. But ultimately, you know, the idea is to be able to bridge across everything. So what do we build? We have a bunch of different products. You know, we have the various data products that cover different parts of the stack, all the way from tokonomics, which is the economics of the token economy, all the way downstream through wafer fabrication equipment and everything in the middle, right? So we have dedicated teams focused on each aspect, whether it be networking, optics, data centers, power, equipment, foundry, etc. Furthermore, we have a variety of benchmarks. So we have InferenceX, which is a completely open source benchmark I'll talk about a bit more, AgentX as well, EndpointX and SandboxX, which are testing inference endpoints and sandboxes. And then we also do various stuff on the side of evaluating models. A lot of our work is open source, so we try and give away as much as possible for free. We also do ClusterMax, which is a ranking of various clouds. So there's over 323 providers, of which 77 we actually were able to review, and 19 get medallions. Most of them do not pass all of our standards, whether it be lifecycle, security, network performance, storage performance. So we test for all these aspects. And we also provide a variety of different data sets. And so that's who we are and what we build. Most recently, I think our coolest thing that we've done is, you know, a lot of this was in the digital realm, but we actually built a reverse engineering lab in Oregon. So, you know, it's many thousands of square feet. We have various equipment in there, over $10 million of equipment, various scanning electron microscopes, TEM, FIBs, etc. And what we do is we reverse engineer the chip. So we'll buy a chip or delayer it layer by layer, take high-resolution photography of it, look at the architecture floor plan, and get all the way down to the transistor level. So on the left side, you can see a comparison between Intel's 18A and Samsung's 2nm SF2. So both 2-nanometer class technologies, both gate-all-around transistors. We just recently posted the new TSMC 2nm from the iPhone or from the Mac, the new M6 Mac. So we're reverse engineering all the way from the transistor level, looking at the materials, characterizing them, looking at EDS heat maps and things of that nature, all the way upstream through to the microarchitecture and again performance, volumes, units, etc. And so that's what we do as a firm. I did want to talk a bit about InferenceX today. I think it's, you know, something probably our most impactful benchmark that we released. It's completely free, open source. In InferenceX, we're testing Nvidia's GPUs. So we have seven different kinds of GPUs: H100, H200, B200, B300, and the GBs. And then also Vera Rubin. So we're testing and benchmarking those every day on various open-source models. You can see all the various models on the left: Kimi, DeepSeek, Minimax, GLM, Qwen, as well as some other models that I did not name. We also have AMD, three different GPUs from them. Soon they'll be adding MI450, which will come later this year. And then also Google, we have TPUs v7s and v8s next year. We've tested OpenAI's internal silicon. We're the only ones who were allowed into their lab to be able to test it, which was pretty sick. And they let us say whatever we wanted about it. So that was pretty fun. And then in addition, we also have a few different kinds of accelerators coming. Trainium we're in progress on, and we have racks of Trainium 2 and 3. We're also working with a few of these smaller startups like Cerebras and others to test their chips on and get them integrated into InferenceX every night. You know, GitHub Actions runs these benchmarks. You can see the configuration. You can see the most optimized configuration, download that container and run it on the hardware that you have access to. So it lets you compare performance across the chips as well as download them. In addition, we, you know, have something called AgentX. But it's backed by the whole ecosystem, all the way from folks like OpenAI, all the open-source companies, all the hardware companies, and a number of hyperscalers have donated hardware to us, contribute code, etc. So it's a really awesome open-source initiative. We hope you guys can participate as well. You know, feel free to make a PR on anything that you think could be better. Again, it is free, open source, and it's fully funded by us in the industry, and it's a money-losing enterprise for us, which is fun. But, you know, what the way people used to benchmark models was they'd do fixed input-output ratios, right? Oh, 8,000 tokens in, 1,000 tokens out, or 1,000 tokens in, 8,000 tokens out. That's a very, like, contrived way to do it because no one's usage is actually done in that way. So instead, what we've been working on has been with agentic coding. What we launched earlier this year was AgentX. We took many millions of dollars of Claude Code traces, right? You know, you use Claude Code, it's going to spawn a bunch of sub-agents, it's going to do a bunch of tool calls. As it kicks these off, we gather all this data and we created a representative data set which we open-sourced. So it will have hundreds of parallel agents running doing different sorts of tool calls. You know, when you're doing a tool call, you're not generating tokens. It's stalled, and you should, you know, to optimally use your hardware, you have to merge in other users who are, you know, in your sort of continuous batching. What we've seen is the majority of traffic now is agent-based, right? Not chat-based. And so, you know, over 60% as recently as June 26, but it's even higher now. So most of the API model usage is these agentic frameworks like Codex, Claude Code, these sorts of harnesses that run and spawn many sub-agents and do many tool calls. And so what that is, is the official thing that we open-sourced is 8,300 sessions that were sort of picked out of the full data set of many millions of dollars of spend, of real spend, by the way, of Claude Code across our company as well as many other companies. And it includes many anonymized sessions. The context length is very long. As you know, in agent sort of workloads, the P50, which is 50% of sessions of agent streams, are 80,000 tokens long, the context length is, but as you get to the P90 and P99, it's 272K and 675,000 context length. And so these sessions are replayed on the hardware, and so that you can get an accurate sort of view of what the performance is. And in that, Nvidia sort of leads currently with their performance. You know, there's a huge difference in performance across Nvidia versus AMD on SGLang at, on Qwen 3.5 at 90 tokens a second, you can see a huge performance difference. It depends on what point on the curve you choose. So any company is going to pick a point. If you look sort of on the left, your left side of the curve, the difference is much smaller, you know, between, say, an AMD and Nvidia, but as you go out to the right side of the curve, the difference spreads quite a bit, right? So if you're talking about really low rates for inference, right, sort of, you know, 10 tokens a second, really slow, you can read, you know, average person reads, you know, 50 or 100 tokens per second is the, or per, um, um, 10 tokens a second is not really 50 to 100 words per minute, which 10 tokens a second is not fast enough. Um, you get the performance difference is quite small across different hardware. Um, generally depends on what the hardware is, but AMD versus Nvidia, the difference isn't that huge. But as you go to 90 tokens a second or 150 tokens a second, which is what you want for very interactive workloads, right? Engineers don't want to be waiting all the time for their AI models to respond to them, um, or any other workload that you're doing, um, the cost efficiency grows quite a bit. So GB300 versus AMD's MI355 is a 5x difference on GLM 5.3 at 150 tokens per second in terms of cost per token. And so, of course, GB300 is more expensive than MI355, and yet the performance difference is so much more that it makes up for them having a 5x difference in cost per token. And then, and then, and we do all the leading techniques, right, like KV cache offload and all these other things to integrate it in there. What's interesting is, you know, in most of the chart curve, Nvidia is faster, but actually on the far right tail of it, right, super high tokens per second, we've found that in GLM 5.2 to in GLM 5.3, AMD with TileRT, which is this group of researchers in China who have open-sourced an inference library for super fast inference, they actually are faster than Nvidia, you know, significantly so, right? Nvidia's curve doesn't even get to that point on the chart, right? You can't even get 470 tokens per second on Nvidia hardware today from any of the engines that we test, which include TensorRT-LLM from Nvidia with Dynamo, right, SGLang, vLLM, as well as TileRT. You can't even get that speed from Nvidia, but with AMD you can. And so it's really interesting that there are use cases where AMD can do things that Nvidia cannot. And so, you know, we use vLLM for the pre-fill portion of the workload, right, where you're processing the prompt, and then we used TileRT for the decode portion. And when you're doing decode, that's where that 470 tokens per second per user is very important. And so it's really important to be able to recognize it's not just the hardware, but it's the software, it's the system, it's everything that you have to aggregate together to get the peak performance per watt, performance per dollar for your specific workload. If we go look at Rubin, though, Rubin is, you know, very new. It's just come out. Most people don't have their hands on it. We've been able to measure Rubin at its pretty immature early state. You know, just starting to ship. Software is just starting to, in the open source, just starting to integrate it. It's got 7.2x tokens per megawatt versus GB300, which is a drastic difference. So, you know, when, you know, generally in the past we've been used to as a community, oh, 2x performance increases per generation and it costs 50% more, so it's 50% better performance per dollar or performance per watt. But now, you know, given this co-op optimization that's happening across models, hardware, and how fast everything's moving, you're seeing gen, we're seeing gen-on-gen the performance increase is significantly larger. The table on the left shows depending on whether you're like 75 tokens a second, okay, it's 50% better tokens per watt, all the way to if you're 150 tokens per second, it's, it's 7.2x tokens per watt. And then once you go out to the tail of like 200 tokens per second, it also compresses back down to 2.7 tokens per watt. But what's interesting is if you look at all the services like OpenAI, Anthropic, etc., they try to serve, you know, in that 75 to 150 tokens per second regime. So the range at which your user needs faster tokens will define how much faster or how much better the new hardware is. So if you're running at slow speeds, potentially Rubin is not as worthwhile as if you're running at very high speeds. The other thing is we were able to integrate TPUs into InferenceX as well recently. So this is on fixed 8K1K because it's still pretty early, but we're upstreaming the TPU AgentX, you know, these agent replays pretty soon. So it'll be open source pretty soon, but TPUs have a pretty big advantage versus GPUs. So you can see at the low throughput regime, it's not that important, right? GPUs are just as good. GPUs are very good at throughput. But at the high end of the curve, where you're running very fast tokens per second, TPUs are actually much faster and much more cost efficient, which is an interesting optimization of the hardware and software. And this is by the same FP8, you know, same precision, same sequence length, same everything. We try to control for every variable. And these performance numbers move every day, right? Because they get run every night and new software gets updated pretty frequently. Whether it's a new SGLang or vLLM version, a new driver version, a new CUDA version, whatever it is, those updates often affect performance, which is why we run it constantly every day. And the whole point here is, if you're running any of these models, you should just download the optimized container from us on the open source on the GitHub and see what's best. The scope for TPUs moving forward is pretty exciting. You know, that comparison was at FP8. We're adding FP4 pretty soon. You know, using various things such as aggregating, serving, and single token prediction. Changing to, you know, using multi-token prediction with disaggregated prefill and decode. And using AgentX, which is that agent replay system that I mentioned earlier. Once we integrate all these, there'll be even more apples-to-apples comparisons. And then serving software, right? TorchTPU is coming. We've been working with Google on releasing that, which is PyTorch for TPUs. And we've been working with Meta as well and the PyTorch Foundation using Pallas kernels. So the TPUs will dramatically improve performance as Google open sources this software that is not currently available to external users. And so with that, we'll have, you know, with TPU v8, which we expect to have pretty soon in the coming next six months or so, with disaggregated prefill, decode, multi-token prediction, all these things will be integrated into the open source repo. And so, you know, that's sort of what's going on with InferenceX. And the next things that we're working on are, you know, full-scale release from preview to official results on Rubin, MI455. AMD has committed to letting us run these benchmarks on their extra wide rack. If you've seen the rack on the show floor, it's like super wide. Running it on that rack in some of their data centers pretty soon. Google's work, we're working with Google on the next generation TPU which comes out soon. And then Cerebras, we've been working with them as well. And then Amazon's Trainium 3. So all five of those are in flight. And only because the community here, you know, the Open Compute Project and many other places help us with this. You know, it's not just us working on this, right? It takes a whole community of contributors. So that's all for sort of InferenceX and what we're doing on that side, but happy to answer questions about anything.
