All right, welcome back. SemiAnalysis weekly podcast number two. Today, Doug and I are joined by Cam Kishi and Brian Shan. We are going to talk about Inference Max. Guys, welcome to the show.
Hey, what's up, Jordan? Nice to be here.
We're covering, I think, all three major time zones right now, right, in the US. Yep. We're missing Mountain Time maybe, but we got Pacific, Eastern, and Central representing.
Doug, welcome back, man. You just got off the train.
Yeah, I just came in, came down from Boston, took the train, took a lot of calls on the train. Now, here we are having our weekly.
How long is the train ride from Boston, New York?
Four hours, dude. It's not... That's not the... Okay, it's honestly, and this is completely off topic already, it's just nice. You don't have to go through the airport security and stuff like that. You could usually work on the train. That's my favorite part. The Wi-Fi is okay.
Do you buy the first class on on Metro or whatever?
Yeah, dude. It's uh it's business class, bro. We spent it. So, um yeah, but let's go. Here we are. Uh Cam and Brian uh and also Jordan are all part of the team that does the beautiful work at Inference X, NotMax. Um and I think today that that's what we were going to go over, right? Am I am I off base?
Yeah, absolutely. Yeah. So, obviously version two of the InfMax article came out um earlier this week and in the I guess previous edition there's uh there's been a lot of updates. So, uh first of all is the name change from Inference Max to Inference X. Um shout out HBO Max for that one. And then the uh the other like more interesting changes are I think really focused on on two things. One is the dashboard itself which anybody can log into and see. You can go to inferencemax.ai or inference.smos.com.
inferencex.com, not AI, right?
Um any of them work and you can actually see the results for yourself. We can talk about these what these results actually are. Um, and then the the second part of it is the the code. In other words, like it's an open repo where anybody can see where you can run this stuff and uh you can fork it and use it for yourself. So, yeah. Um,
Guys, can you start by giving like more background on why we're putting out this 2.0 release and like what are the major changes from the original Inference Max and now moving to Inference X.
Yeah. Yeah, definitely. So I I think originally with with Inference Max 1.0 we the goal was to, you know, benchmark a ton of of different GPU SKs on a on on various different uh production workloads. But uh for the last 3 months we we mainly do been uh benchmarking just single node deployments on of deepseek and GP2oss and a few llama llama 7db uh models and I I think in reality big labs, neoclouds, uh and like serverless providers are using more state-of-the-art techniques for inference such as uh disaggregated prefill, wide expert parallelism, uh speculative decoding and in order to have a more nuanced and and robust benchmark, we wanted to include those. So since the 1.0 release, I I think that's been the main primary focus is just go going at it and trying to add all the disag uh benchmarks. And I think it's going to it it the results really show a better picture of what of what uh Frontier inference is like uh in production. So
And specifically that means the Deepseek R1 model was added. Right.
Right. Right. Deepseek was there in V1 but it was only single node. Uh so now it's multi-node. We have we're testing the entire NVLink domain. Uh multiple nodes of MI355 with with their Polar nicks and and yeah it's pretty cool.
Yeah. And maybe explain that as well because um there's like a whole section. So the first part is the models and the um software configuration on the runtime to enable features like speculative decoding that you were saying as well as the pre-filled decode disaggregation. But the second part is the hardware. So the first release did not include GB300. It didn't include B300 which are both new GPUs from Nvidia being included in 2.0. And the major results maybe improvement would be showing how big of an improvement the GB200 or GB300 is actually bringing not with the G not with the gray CPU but with the NVL domain. Right.
Right. Right. Yeah. Yeah. We can get into this deeper later, but specifically with Deep Seek, you can run expert parallel, which is effectively a form of parallelism for LLM inference where you have uh for DeepSec, it's 256 experts. Um, and you can basically allocate or disperse these experts across a bunch of different GPUs. So, when you're doing this on one node, you're running EP8. So, you have 256 divided by 8 experts on each GPU. And when you're running uh you know that across like eight nodes of of B200 say right uh you're dispersing over 72 GPUs but it's going over infiniband right as the as the scale out because um yeah but when you're running on GB you know you have NVLink domain and it's like 72 GPUs acting as one as they say right and then expert parallelism is just so fast and we can go into the those results later but it's just very clear how how that frame mogs you know all of the uh y
Frame bogging. Yeah. Yeah. Okay. Let me actually go into the results right now because I think this is probably the uh the most key thing for those watching. I uh I've got um the inference max version 2 article on screen. This is the first major screenshot and I think it's the biggest takeaway that we had um from these results which is on a direct comparison of like the max performance for DeepSeek that you can get out of an H100 with all of the tunings and like you know improvements that have come when compared directly to GB200 and GB300 at low interactivity levels. So low speed per user, you get 20 times more throughput. In other words, you can support 20 times more users per GPU or more requests per GPU. And at high interactivity levels, it's just doing things that H100 can't even do. So 80 to 100 times more interactivity uh sorry more throughput per GPU at an interactivity level of 100 tokens per second which to be clear we are seeing used today right like there are models available like codec spark from openai uh as well as
I thought the codec spark by the way was is is it on server or no I and product spark is on cerebrus. We don't know what um exactly what the uh opus uh 4.6 fast is being served on right now. We got our suspicions about tranium and TPU because that's what anthropic uses for inference compute.
They have GPUs too, but they have GPUs too. So
Yeah, but the I mean the point is that those fast the quote unquote fast models are being served at interactivity levels above 100 tokens per second per user from what we can see right now. whereas the standard models are more around 40.
So it's a like it's a real requirement for people to want to serve models at 100 tokens per second per user or per request. And um the fact that even at that level of interactivity, the GB200 or GB300 is showing some 20 to 80 times more throughput is just such an obvious statement [laughter] that there have been incredible improvements generation over generation here. So
Sorry to summarize framework, you know, oh my god, other than the awesome work that the imprintex team has done, um I guess I I you know, since I like to, you know, semi analysis has a lot more people than just like Dylan or me or even Jordan, I just guess I would love to like talk about like what's the whole process in like, you know, doing all this. I think people, you know, they we put out the the article, it's super effing long, and people are like, "Wow, that seems like a lot of work. Too long, didn't read. That's good for you." Or, "I'm sorry," or whatever. You know, they send the meme back. But like, um, like I I just don't feel like people appreciate the scale of what's happening and the tests that we're doing every single night. Um, and yeah, I think Cam and Brian can definitely offer a really cool insight into it. Um, would like to hear, you know, like what's it like, you know, making the spaghetti yourself, I guess. Yeah, Brian, I'll let you take that one, man.
Wait, is did Jordan just disappear?
Oh, yeah. Jordan comes Jordan will come back when he comes back. Don't worry about him.
Oh, sure. Yeah. So, actually we work quite closely with AMD and Nvidia engineers who help us find uh recipes for these kind of setups. Um because optimizing such a task is not easy like there are many environmental variables available. There's many settings to tweak. We work very closely with both companies to try and showcase their hardware to the best to the best of its potential basically.
Yeah. And I I definitely think um something that people uh accuse us accuse us of is favoritism. But I think more than anything else, we really just try to make sure this benchmark is a leave a living breathing benchmark. And what what's shown each night is effectively the best results, right? um they work with us and they put like you know they put up numbers right and these benchmarks have massively improved partially because you know people at AMD people Nvidia both teams have been working really hard on increasing throughput and yeah I don't think there's um a favoritism more so that we just want to be the best benchmark and see the most token throughput so um yeah
Yes exactly and that's why we are like uh currently trying to do day zero support for some of the new Chinese models Quen 3.5 Miniax Kin 2.5. Yeah, we really want to track how these uh AMD and Nvidia actually improve their implementations for these models throughout time. Yeah.
Yeah. I I think that's a good point. And like a lot of the benchmarks that currently exist are kind of just one point in time thing. And not only one point in time, but one single point on the prao frontier, right? And like Brian said, we're really focused on trying to show that this is a very hard problem to solve and a lot of uh people work very hard to improve it and push the frontier forward iteratively, you know, week by week. So I think that's kind of one of our our northstar goals.
What was this cam? Like I know you have experience doing inference benchmarking before infertex came along. So like when you're saying this runs nightly, like what was it like before that leading question?
Yeah. So in in my previous experience, I think Brian has some too with with other consortiums of of inference benchmarks. Uh won't name names, but um I think it's really uh benchmark maxing is is a word that comes to mind, right? Like uh you know, it's important to understand that just for those who don't have the background, right? Uh I I just I'm going to share my screen real quick if I can. Screen here. Okay, cool. Yeah. So like um the idea here is like if I look at GB200, right? Um this is an entire like curve of different uh configurations that you can tune and serve at. So you know I don't know if clog code wants really fast a fast mode, right? And we could talk about this more in detail, but you know, they're they have uh lower total system throughput, but they're serving users at 140 tokens per second, right? So each user receives tokens very fast. But if you want to like make the most out of your system and have total system throughput that's very high, you're going to be over here or somewhere in the middle. So really, there's no one-ize-fits-all solution. And I think other benchmarks for inference kind of assume that as a prior where hey we just want to maximize the total throughput and advertise that as a single number and really the problem of inference is like so much more nuanced than that. So I think that that's kind of the difference there. Another difference is like you mentioned we we run nightly or weekly you know on a cadence at pace with what VLM SG lane tensor RT you know all of the serving runtimes release at instead of you know every 6 months or every you know two quarters or or yeah every quarter whatever you know so the release cadence is is you know just constant and that's the that's the second chart in the article is showing the improvement in performance for uh rock cam right on the Deep Seek model which is pretty significant like it almost doubles in the span of I think 3 months or something like
Yeah. Yeah. Like we can look at this right now, right? So, uh MI355 I think is you're talking about MI 355 multiode here, right?
Yeah. So you can see that the first uh and really one of the first public recipes that AMD has ever released I think to my knowledge was you know January 8th and then the most recent one is this January 17th one. And you can see right that um so this is so by the way square is FP8 circle FP4 but you can see let's just show FP8 for now. You can see right that um they they're just like continuously improving. This isn't a good example, but they're they're the the curve is just being pushed forward like multiple times in a span of like one or two months, which is impressive in and of itself because this just goes to show that it's not just like hardware improvements, right? Like hardware is, you know, released every two years, maybe every year and a half.
These are just like engineering challenges, right? reshare your screen here. I think would help uh for the recording. But um
No, you were sharing, but it was frozen. So,
I can bring it up on the uh on the other screen. Um there you go. Yeah.
Yeah, that's much better. Yeah. So, for those viewing, basically Cam had a uh or those just listening, basically Cam's got a graph from the Inference Max, the Inference X dashboard with four dates on it, January 8th through February 17th. And it shows an improvement um from about a thousand tokens per second at 20 token per second a thousand tokens per second of throughput per GPU at a 20 token per second interactivity level all the way up to 3,000. So a triple roughly triple the performance in one month after optimization on the software level which to go back to the kind of original motivation for answering this question is like when there's a benchmark that is tied to first a specific interactivity level that only accepts submissions every 3 to 6 months and does not get updated over time. You can't actually see these improvements for a long time. You can't see them three times within a month. And it's hard to give feedback or get feedback when you're doing development in that speed, let's say.
Yeah. Uh and one more thing on this note of the the difference is you know we we really um enjoy our partnerships with AMD and Nvidia on on an engineering level and we want to show them in the best light always like for both for both companies but also with other benchmarks you you can it's very much more like supplier like company ccentric right so like if a company submits a benchmark and they they see that they're losing to someone they can unsubmit the results right here uh we don't really allow that, right? So, so we are in an independent like we're an independent like entity here. We're just performing the benchmarks with their help. So, we're going to show the results no matter what and we're once they're up, they're not going to be taken down, right? So, I think that's like kind of an element of transparency that we have over other existing benchmarks.
Yeah. Awesome. How about um that process of working together, right? like there's uh there's been huge improvements in the software stack um tied to the new hardware, right? Maybe you could walk through the improvements that you've seen roughly like one at a time and how that how that comes to be cuz like let's take prefilled decode disagregation to start right if there's a framework that's developed or paper really blog post that's written like just serve that explains the theory on how to do this it takes a while for that actually to show up in software that anybody can download and run And there's only a limited set of people in the world that have a GB200 NVL72 rack to test these optimizations on. Right. So, how do you go through that? Like, how do you go through doing that testing and working with Nvidia or AMD to get the latest software?
Yeah. So, I mean, Brian, you you want to you want to talk about this?
You can go ahead first. I I mean I mean the the the fact of the matter is like most of the inference level optimizations that are being done are are by those AMD and Nvidia engineers right so all credit to them where credit is due like the Dynamo team the tensor RTLLM team the uh the AMD distributed inference team like they're amazing and they are extremely smarter than I am and they they are making these like kernel level improvements um like every day and I think this is where like these iterative improvements come from, right? They have their own testing clusters, their own testing environments. Um, but you know, we we try to help them and understand their optimizations. But, uh, yeah, I mean, all credit to them in that regard, I think.
Yeah. Especially like in the early stages for the recently released models. We also try to work with them to discover like new flags or search for recipes online that other people have found or uh test those recipes other people have published.
Yeah. I mean Brian knows this like it it's kind of like a puzzle because you have these engines like VLM, SG Lang and TRT that over the last few years have just blown up in size, right? and you know every time you add a new feature maybe there's like a new feature flag like environment variable that you set for like a backend for for a mo model or whatever you know I don't know like there so now there's like a thousand different environment variables you can you can set and like finding the right combination of which to use for what deployment is like really a a challenge that requires a very deep understanding of all these different frameworks. So yeah.
Yeah. And going to another issue is like some of these optimizations work by themselves, but when you combine them, the effect isn't really replicated.
Yeah. So it's not as simple as just uh combining good flex. It's about finding flexs that work good in combination.
Yeah. Brian, I think you should talk about more about the problem. Yeah. AMD's composability issues.
Yeah. So that's one issue that we discovered uh with AMD is that they have many flags that work well and they say work well by themselves. They say this flag will boost boost performance by this much but when we actually combine them they don't really work well together. Uh I don't remember the exact flex that do this but yeah that's one big issue that AMD have and specifically AMD has.
Yeah. And like at a high level, this can look like, you know, when you have FP8 inference, right? Like just plain single node FP8 inference is actually on par with that of of B200, its counterpart. But then when you go, you know, FP4, FP4 inference is like significantly worse than B200. And then when you add disagp it's like significantly worse and even you know [snorts] like way like worse than the last step right so so it's just about this like composability of software uh is kind of like an issue for AMD at the moment but I I think they can do better
Yeah. Can you um can you talk about some of the analysis like we've talked a little bit about prefill decode disagregation YDP these are relatively complicated to understand cuz I think you need to understand a little bit about how a transformer model works to be able to like understand the concept. But one that's probably pretty simple is MTP like multi-token prediction here. And the chart that shows how much that can reduce cost is like uh a pretty pretty clear obvious one that that horizontal bar chart that you made, Brian. So maybe you'd be willing to like walk through what sort of uh improvements MTP has brought and just explain like as as an example of one of the optimizations that you could turn on what MTP is.
Yeah, sure. So MTP is actually a recent type of speculative decoding. Uh previously we actually used smaller draft models to produce these tokens but uh deepse has actually popularized this method of using MTP heads within the model um that are pre-trained along with the model for this uh multi-token prediction and basically these optimizations in general uh exploit this uh memory bound issue with decoding because during decoding are actually producing logits for uh previous tokens as well, not just for the next token. So, speculative decoding makes use of these uh logits to verify past tokens. So, there's no actually there's actually no drop in performance uh in evaluation performance sorry as our charts on GSM AK show. And basically this is basic almost a free optimization in the sense that it provides a lot of per performance gains at almost no cost to accuracy.
Yeah. And just to be clear, GSMAK is a benchmark grade school math. It's like simple math questions and you validate that the model can still perform all those uh reference examples. It can still hit the reference level of performance on that evaluation.
Yep. um I can share my screen and show that that chart but the results. So I maybe one of the things that semi analysis that we've spent a lot of time doing is actually um showing the like uh analysis beyond this right so MTP makes things faster but what does this actually mean for users and the result is clearly uh cheaper tokens either pro the provider can increase their margins because they can serve the tokens for cheaper at the same price or they can pass those savings on to uh their customers, right?
Exactly. Yeah.
Also, maybe dumb question looking out, how would we cuz cuz I I wonder how um the realized uh cost differences are going to be seen. Like I've been wondering this like for example is this just like going to look like a better cash hit rate or is this going to truly just be like they're going to eat the margin and you know your end customer will never really see the the difference. I would love to hear what you guys like how how do you think it how do you think M MTP and also I mean I guess the question is is are these are these changes even in production today right because it's like relatively new.
Yeah, MTP is is specific to DeepSeek and yes, it's definitely in production. Other speculative decoding methods are are also used in production. I mean, like Brian said, it's a bit more nuanced than this, but it's basically a free optimization in terms of accuracy because of the way the probability
distributions work. So yeah, they're definitely being used in production. And then I really have no idea how the savings will be used, right? I mean, I'm not like a finance person, you know, I don't know the economics of this, but yeah, I mean, as Jordan was just showing on his screen, it's like significantly cheaper in some scenarios, like by like 50%, right? Or more. So I'm assuming some of those savings will make it to the end user, but I'm not entirely sure.
Yeah, the chart that I was just showing on screen showed that the B300 with Dynamo and TensorRT-LLM runtime was serving the DeepSeek model at 68 tokens per second per user, a reasonable speed at a price of 25 cents per million tokens.
Which many people can compare to an existing price that they might spend on a model, right? People will be familiar with a model that might cost a dollar per million tokens or $15 per million tokens. Opus is at that level for the 4.6 model. Depends on your blend of input-output tokens. But we're making an assumption here. I think 3:1 input to output ratio. And we just price it that way for input and output, right? Combined. And so when you turn MTP on, it goes from 25 cents per million tokens to 5.7 cents per million tokens. 80% decrease in the example sharing, right?
And so you're talking about a free optimization for sure. But
I think the key point to tease out about why this is cool that Inference X does this is that you can just see the results for both. You want to see it with MTP on, you want to see it with MTP off. You want to see the y-axis with throughput per GPU or cost per million tokens or throughput per GPU per all-in utility megawatt if you care about power. We have all of that analysis done from a basis of our data that we have in terms of how much servers cost, how much power costs and things like that.
So maybe we can talk about more of the metrics or the analysis that you guys did beyond just running it and seeing the output on throughput. It's like actually drawing the conclusions of how much this is going to cost people.
