Today I'd like to talk about SemiAnalysis's Inference Max. We launched this benchmark last week, as well as talk a little bit more generally about the firm. So first off, I'm going to talk about what we do and then we're going to talk about Inference Max. What we do, we're a company. There's 44 of us. Slides a little late already. Across the world. So we've got people who've worked all the way from ASML and TSMC and Intel all the way up to places like Tesla and Nvidia and Microsoft over their careers. So most of the team is either engineering background or finance background. And what we do is we have a lot of reports publicly that you may have seen. But we also have a lot of industry data that we sell to companies throughout the industry. And so many firms here purchase some of this data whether it's all the way from the accelerator and fab side all the way up to token usage and demand. And we also have some free tools online, right? There's a tool called Fab Map. You can track every FAB in the world. It tells you what capacity, what node, etc. We have wafer calculators. We have various other tools that are coming out. And the one that we want to talk about today is the Inference Max which is a new tool. It's completely open source and it's free.
So what's important is that inference is becoming increasingly specialized, right? You're well past the world where one chip does everything. It may be the same type of chip but you're splitting it out into different groups whether it's prefill only chips, i.e. processing the prompt, or decode which is doing the autoregressive generation of tokens. There is various offload work to move KV caches off of GPU memory into whether it be CPU memory or storage in other nodes in between turns of a conversation. And then there's various aspects of mixture of expert models and expert parallelism that's making inference systems be, you know, the standard unit for an inference deployment being hundreds of GPUs instead of a single node. And then there's all these different things about traffic and so these are some of the key trends in inference.
But benchmarking this is very, very difficult, right? It's broken. There's a lot of lack of transparency in what the benchmarks are. Often times a vendor will just quote one number, cherry-picked and compare it to another number. They're often out of date. They may be months and months out of date. They're updated infrequently again, right? Very like maybe quarterly or semiannually or maybe just on launch. And then there's no progression, right? They won't show, hey, what's new hardware doing? Hey, how did new hardware start off versus now? If you tried to use Nvidia's Blackwell 6 months ago, the numbers were not amazing, right? But now they're actually amazing. So how did that progress over time? Same with AMD, right? If you go back a year ago, it wasn't nearly as good as it is today for the existing hardware. These benchmarks are also extremely hard to replicate, right? They're often a hyper optimized solution that only works for one use case, right? Whether it be long context and short output, they'll only choose one point along the entire curve. And you don't know if that point on the curve is extremely low tokens per second per user. Sometimes these benchmarks are, hey, the user is getting a token every 10 tokens a second, which is not usable. If you use ChatGPT, it's like 100. So they'll cherry-pick where they are on the benchmarks. And then the model weights will be private in many cases and it'll be tied up in spreadsheets.
And so, Inference Max is aiming to change all of that in the benchmarking space. So, it's completely transparent in that it's completely open-source Apache 2.0 license, so anyone can take it, fork it, replicate it, do whatever they want with what we've done. It's using state-of-the-art techniques, right? So, we're using including prefill-decode disaggregation, right? Where there's different nodes doing prefill versus different nodes doing decode. There's wide expert parallelism, right? So you have many GPUs working on a single problem rather than a single node. And then it's updated nightly with the latest software, right? So this is the biggest deal about AI, right? Is models change every 3 to 6 months the best model is changed in the open source. And on the closed source, the same thing happens. The software changes multiple times a week, right? vLLM, SGLang, the two most popular inference libraries, TensorRT-LLM as well, all change twice a week they have an update. Furthermore, PyTorch has nightlies where they change the code. You have driver updates constantly. So these things change every day and the performance changes every day. And so this benchmark is run every single day and all the results are publicly posted.
And why this is important is you can see the progress of hardware over time as the software stack gets more optimized. When new hardware is released, the software stack is not as optimized and so your performance is not great but over time that really improves. And this is really important for vendors who are not Nvidia especially because then their software is significantly behind and it may not take them a few months to get to peak performance. It may take them a year but people are referencing performance from a year ago or comparisons between providers end up being, hey, these numbers are from a year ago whereas these numbers are from today. And so it's important to show the progress of hardware over time and by benchmarking every night you get to see that. It's also very easy to replicate, right? Again you can just branch it and run it yourself. The configurations, all the software is there, easy to use. And then it's easy to analyze as well, right? There's a nice GUI. You can also download all the raw numbers like before, but there's a nice GUI that charts all the different numbers.
And so that's what Inference Max is, right? It is an open-source automated benchmark that runs on tens of millions of dollars of hardware every single day. Today we do six different GPU types. Nvidia's GB200, B200, MI355, H200, MI325, H100, MI300. And soon Nvidia will be contributing B300, GB300 to us so that we can run this benchmark nightly. As well as some other clouds. Furthermore we have Google TPU and Amazon Trainium coming soon as well and that will cover over 95% of the flops deployed in the world of AI accelerators. And we would like to introduce more hardware over time as well. But today we have tens of millions of dollars of hardware of the current and last generation GPUs.
The reason, as I mentioned before, the software updates every single day and so being able to know what software performs the best at what point along the curve is also really important because inference is not a unipolar workload. Some users may have really long context inputs and very short outputs for document summarization, others may have the exact opposite for agentic or reasoning tasks. And so we don't choose a fixed point, we sweep across the entire curve of batch size, throughput, parallelism types and different types of workloads. Right? So today with Inference Max v1 that is released, it's been running for a few weeks. The results are all online on the GitHub as well as through the GUI on inferencemax.ai. We run three of the most popular models, right? Llama 70B, GPT open source and DeepSeek. And so these three models are the most popular but we will change these as new models come to market and they gain a lot of traction. So very soon we'll be adding DeepSeek 3.2 with sparse attention. It's a different flavor of their existing model. It's just started picking up steam. But it is a little bit different, right? As I mentioned, we have B300, GB300, TPUs, and Trainium also coming. And then lastly, we'll have performance evaluations of the quality, right? Because it turns out outside of even quantization, right? Which we run different quantization formats, 4-bit, 8-bit, etc. Outside of quantization, there's also model quality difference between different hardware because there's certain optimizations the vendors can do that will reduce quality but will give you more performance. And so we're also adding that on Math 500 and GPQA Diamond which are the two leading benchmarks for quality in the model space.
You know, we have a lot of support from industry leaders across the space. AMD, Nvidia, OpenAI and the list goes on and on. So I'll sort of pause a little bit to let you read the quotes but AMD and Lisa Su is very supportive of our effort and they've given us, as I mentioned, many GPUs to help us run this and they dedicate engineering time and everything gets open sourced. They're not paying us to do this, they're just contributing hardware but we get AMD hardware from other clouds as well. The same for Nvidia, right? Nvidia is contributing a lot of GPUs. They're not paying us for this, it's all open source. And we do get Nvidia GPUs from other clouds as well who are contributing. We've got support of folks like OpenAI. So Peter, he's the one who decides all the capacity planning for OpenAI. And he's in charge of Stargate. And so he also is supportive and we've got many other vendors including Oracle, including Microsoft, including Together and many, many, vLLM, PyTorch, there's many, many folks. And so if you read the report we have all the quotes listed there. I don't want to say I've named everyone but we've got more than a dozen firms sponsoring compute, engineering, quotes about what they find useful out of Inference Max.
And so let's talk about how Inference Max works. Right. So we have these GPU resources that are given to us by various clouds as well as the hardware vendors. And then we have our open-source GitHub profile where through GitHub Actions we execute this benchmark using Docker and Slurm, launching server-specific scripts which then sweeps across all different parts of the benchmark. These scripts are launched, they run. If something fails, we retry, right? Because hardware can be unreliable. And all of the, we create a database of different types of benchmarks that we run, right? Hey, we're running tensor parallelism 1 and pipeline parallelism 8 or hey, we're running tensor parallelism 8, pipeline parallelism 1. And we sweep across all the different parallelism configs as well as different batch sizes, different, hey I'm serving 8 users concurrently, hey I'm serving 32 users concurrently, hey I'm serving 64 users concurrently. And this runs constantly. So GitHub has helped us a lot. There were some challenges with how we have one of the largest fleets of GPUs that are running nightly actions via GitHub Actions. So GitHub has helped us a lot in adding some features that help us because we're running hundreds of GPUs.
And so what is this Pareto optimal frontier? You see there's a lot of dots along this curve, right? There are many inference configurations that are sub-optimal. And so when you look at companies doing marketing, they often choose a suboptimal configuration for their competitor and then an optimal configuration for themselves. That's logical, right? Everyone does this. That's what marketing teams are for. And so you have to sweep all of these configurations because in one setting, perhaps it's reasoning workloads, that configuration is suboptimal. But then in document summarization, that configuration may be optimal. So you have to sweep across every type of configuration so you know what the Pareto optimal is. And then because we have these Pareto optimal curves for what is the most efficient configuration for inference, people can also just go to Inference Max, go to GitHub and download the exact configuration and say, hey, this is what we're going to use for our own production-scale deployment. So it's not just for comparing hardware, right? It's also for saying, hey, I've got GPUs. I want to run this model. What's the most optimal configuration? Because it's not as simple as, oh, I'm going to download vLLM or I'm going to download SGLang. There's a lot of flags you have to configure. And so you have this Pareto optimal curve. And now if anyone is running a suboptimal configuration, they would know that because we sweep across with hundreds of GPUs all the different configurations on different workloads. And the Pareto optimal is maximizing token throughput for that given latency regime and it's optimizing your revenue as a result, right? Because whether you're selling tokens or using tokens in your own company.
And why this is, the other thing about Inference Max that's really important is we're not just throughput for GPUs, right? We're also doing power and cost. And so why this is important is because inference is always a trade-off even once you have this Pareto optimal curve, is interactivity versus throughput. So interactivity is, hey, I'm in a race car, I can go really, really fast but only one person sits in a race car. Or I'm in a bus and I can transport 30, 40 people but I'm going pretty slow relatively. So Inference Max aims to show and publicly release, hey, what are the optimal configurations along the entire curve. And so then, depending on what use case it is, maybe it is batch processing of documents overnight, you don't care about having a fast user experience, you care about riding the bus, right? Just give me the lowest cost possible. For another configuration it may be, hey, I have software developers who are making $200, $300,000 a year and actually what matters for their productivity is really, really fast response time. And so in that use case you want the race car for them. And so this is where we're discussing this is on TCO. And why does TCO matter is because it's your cost. 20% of your cost is your electricity, your data center real estate roughly. And then the rest of the cost is that hardware at least on a standard Nvidia deployment, your GPUs, your networking etc. And so we account for all of this. And even if you have 20% lower power and data center cost, it ends up being only a 4% difference on your TCO. So there's a lot of, hey, maybe if this new hardware consumes 20% more power it may not actually matter for the final TCO calculation if the hardware is half as much. So this is why it's really important to frame this in terms of performance TCO and performance per watt.
And so Inference Max, the first results, there's a full report that you can go through. There's also a live dashboard where you can go through all the results. I'm going to be cherry-picking here obviously. But there's certain interesting results. The largest model we're running today is DeepSeek R1 and GB200 has a huge power efficiency advantage, right? Everyone here understands the challenges of running and deploying GB200. There's a lot of challenges with the backplane. There's a lot of challenges with liquid cooling, the power density. But it turns out at certain interactivity rates, i.e. tokens per second per user, it's 10x more efficient per watt, right? Compared to H200. And even compared to B200, you can see on the chart, it's still quite a bit more efficient. So it's really, really important that these sorts of optimizations are understood because for certain interactivity levels it's really important. So if you're doing the race car, you have that software developer, you need to have extremely fast performance per user. And for other use cases, hey, actually it turns out B200 is just as good as GB200 if you're say doing batch processing of documents overnight.
In another model, GPT open source, it's the model that OpenAI released publicly. In document processing scenarios, right? So we do different scenarios. We do document processing which is 8,000 context in, 1,000 out. We do chat, which is 1,000 in, 1,000 out. And then we do reasoning, which is 1,000 in, 8,000 out. Across all these different types of use cases for the models, we see that in some cases AMD actually does have a publicly usable open-source implementation that beats Nvidia even, right? With the MI355 versus B200, which is a surprise, right? Most people are like, oh, maybe AMD is better performance TCO or hey, actually AMD software is really hard to use. But this is publicly usable software. You can just download the configuration from our GitHub and run it and it's better performance TCO than the B200 in certain use cases. So you have to look through and see what's your use case, what's useful. Once you start getting to really high tokens per second per user, then it switches back to Nvidia is actually better. And so there's a lot of nuances in this on a performance TCO basis.
Another thing is that as you have different quantization, there is significantly different performance TCO across chips. So at some points actually, hey, H100 versus B200 there's not a humongous difference but at other points you can have a 10x performance TCO benefit from going with a more expensive Blackwell server than a cheaper H100 server. So these sorts of decisions are really important. And FP4, 4-bit floating-point formats are difficult to run. And so we're thankful for our contributors at vLLM and Red Hat for helping us get this implemented and we'll see more performance coming soon from them. So because it's running every night, the maintainers of this open-source software are also contributing heavily to making sure 4-bit works properly. So in some cases for some hardware 4-bit works great right now but for others it doesn't and that's on the public open-source software. But now everyone can see where performance is and actually see where they need to be measured to improve. What gets measured improves.
And then we also do performance on a watt basis. And we also do performance on a regular basis. So when we look at the previous slide, this is cost per million tokens. But when we look at this slide, it's just performance. If we don't divide by TCO, then it looks like the performance of B200 is actually 15x that of H100. Versus the performance TCO is only 10x, right? Because that's the cost differential between an H100 and a B200. And we show the pricing that we use for TCO, which accounts for networking, accounts for GPU cost, accounts for power, accounts for data centers. That's all detailed also on this site and on the report that we released. All of this being free again, right? And we see again, different configurations are what's really, really important.
I'll skip that slide to say that speed is the moat here, right? There's continuous improvement and that's what's really, really important. Over two months, the performance of AMD dramatically improved because they went from, you know, software problems are always a thing, to over a two-month period, they've dramatically improved. We see the same thing across Nvidia's Blackwell, right? Maybe Hopper is more stable. It's only improving slowly, but when you look at the newest hardware, it's skyrocketing in performance. And sometimes when you buy hardware, it's not about, hey, what's the performance I can get six months from now? It's about what I need today. And for other times, it's like actually I do care more about what I'm going to get over 6 months to 5 years out rather than today. And so that's really important is to see the speed and the moat. And so I think given I'm almost out of time, I will stop there and not talk about our future plans. Oh, I have five more minutes. Okay. I'd rather take questions anyways. If anyone wants to have a question, there's microphones to the left and right.
Future plans. Okay. I'll go back to this slide, right? What is our future plans? The future plans are adding more models, right? Because new models will come out, whether it's an Alibaba model now dethrones DeepSeek or Kimi model dethrones DeepSeek or Meta releases a new model, whoever it is. As these new models come out we'll add these. New hardware, right? TPUs, Trainium, new Nvidia named hardware, that's the other thing. And then more parallelism configurations as inference continues to get more complex on parallelism. Today only Nvidia can use these wide expert parallelism configurations and primarily on the GB200 although it can be implemented on other hardware. But as other vendors are able to adopt it we'll have that be added because these new parallelism configurations enable a lot more performance, TCO, performance per watt. And then the other thing is model quality evaluations, right? Math 500, GPQA. And so that's our future plans and it's primarily dictated based on, you know, we have a team of engineering internally. But also there are engineers all across the ecosystem whether it be Nvidia, AMD, vLLM, SGLang, PyTorch that are helping us. And so because it's open source, anyone can fork it, anyone can contribute code, submit a PR, we can implement it. It is really engineering limited and then it's capital limited. And then because it's free and open source, we don't make money off of this.
Your cluster max... I'll repeat the question. Yeah, your cluster max rating system has network performance criteria. Does the network matter for max? Yeah. So in cluster max, networking performance, we measure clustering performance. This is where we measure performance and quality and usability and operations and security across different clouds. Inference Max is a quantitative benchmark on performance and TCO and power. And there, networking performance does impact performance of the LLM for large models when you're doing multi-node inference, right? And so that matters on the scale-up domain. Nvidia has a switched scale-up, right? Where they have the NVSwitch whereas AMD current generation does not and that does impact performance, we see that. And then if you go multi-node you also see that performance impact. Whereas next generation, AMD will finally have a switch and that will help them a lot but then there will be other optimizations. So networking performance is extremely important.
Will you be allowing vendors to run this before? So vendors can run this in-house whenever they want. The code is open source. Go download it. Go fork it. It's just Apache 2 license. It's very, very easy. Now to be publicly demonstrated, that is a different thing, right? And we want to add vendors, right? It takes a lot of hardware to run these benchmarks nightly which is the criteria for us. So you can run it and you can go point in time you can say, hey, here's our performance compared to Inference Max, great. But for us to actually include it on the public dashboard we have to be the ones running it. So just donate the compute to us and then we have to have, obviously there's going to be a lot of engineering work between vendors and SemiAnalysis on making sure that performance looks good and everything. And then the TCO and power is not just provided by the vendor, it's actually measured. So vendors can be added to this but it is a bit more of an arduous task. If you want to run in-house, fork it, go ahead.
Hey, nice work, Dylan. So for all the open source models, you have very good answers for us. Any comment about the frontier models? So the challenging thing is frontier models are closed. How can we run them? Right. Thankfully some of the open source models especially, as we continue to move forward architecturally, are closer and closer to the frontier models. When we look at DeepSeek versus GPT-4o, these models are actually, and GPT-5, these model architectures are not that different. Likewise when we look at Kimi K2, right, which is another model from Moonshot, that model is also very similar to frontier models. So it is challenging to say frontier models will run this way but the open models are approaching architecturally the closed models. Now performance-wise in terms of evaluations there is a difference. One last question. Plans to extend this to training? No, we're focused primarily on inference with Inference Max. Training is a much more, you can't run training benchmarks on hundreds of GPUs. Those are irrelevant, right? We're focused on inference benchmarks.
Awesome. Thank you so much everyone.
