All right, for this next panel, this is one I'm particularly excited about. We're going to be discussing the current state of benchmarking in the industry today. And we have three very special guests. First is Mr. ML comments himself, Mr. ML Perf, David Caner. Come on up. Give him a round of applause. After him, we have Dylan Patel, who's the CEO and chief posting officer at SemiAnalysis. Come on down. And then last but not least, we have Micah Hill, the CEO of Artificial Analysis. Come on out. Give him a round of applause. Thanks, everybody. Everybody have mics. You have your mic? Do you need a mic? Yeah. Okay. Awesome.
Benchmarking. Benchmarking is one of those controversial things. We as a cloud hate it because every customer, especially on the AMD side, every customer who comes to us, the very first thing they want to ask is can we see benchmarks. They want to see what, you know, they're comparing obviously what Nvidia is doing and how does it go up against AMD. And we hate that question. The reason why we hate it is this is how it is when it's not in production and those two things are usually different. And so it's been a hot topic. My team is like, oh, we hate when that question is asked about benchmarks, but we know that it's something that's really important for the industry. And so what I'd like us to do first, I'd like you guys to briefly introduce yourselves and give a little history on how you got into where you're currently at, your current thing, whether it's ML Common, ML Perf, and then why benchmarking is so important today. And we'll start with you, David.
So I had been running a website on computer architecture and microprocessor design and consulting for the industry for a long time. I actually wrote the patents for a company that makes very small chips called Cerebras Systems and I was very curious about the software side. And there was a public call for people who are interested in benchmarking and interested in AI to show up and that was the first meeting of what would become MLPerf and I was elected to be group secretary. So the joke is I started out as secretary alam madman and then they made me secretary Allahabankiun. And actually Greg who was our prior speaker was at that meeting with me. And so that was sort of my trajectory. It was very much one of curiosity and seeing the impact of bad benchmarks in other fields like mobile where there are some chips that have some of the processors are designed specifically with portions only for benchmarks, not real applications, and seeing that as sort of wasteful and distorting the market. Whereas a good benchmark really drives the whole industry forward. Right? It helps to focus the concentration on what is important and helps to deliver more value to everyone and find problems, mistakes, issues along the way.
When did you start ML comments into that?
So the project 2018, so and ML Commons came into being in 2020 and so we had two years as very collaborative sort of open-source existence.
Um, yeah, so I have a blog or had a blog called SemiAnalysis and we do a lot of other stuff now. When I was a teenager, I would like argue on forums about CPU and GPU benchmarks for gaming and mobile phone benchmarks and, you know, see all of the issues in there. You know, started doing consulting and, you know, the blog and analyst stuff and at some point started hiring folks that actually knew how to test stuff because the saddest thing about, you know, vendors is that they love to lie. All their quoted specs are lies, impossible to achieve. Whether it's Nvidia or AMD, neither of them you can ever hit their peak flops. They're just lying. And there's a number of other ways that they lie that get more complex. And so what sucks is no one, you know, there really isn't anywhere in the community where independent third party sort of benchmarks of things come, right? So some of the initial benchmarking we did was just showing, hey, like yeah, these flops are lies. You cannot, if you just take a 8K by 8K by 8K gem, very common shape, and you run it on the matrix multiply unit of AMD and Nvidia, you get like 50% sustained performance on Nvidia and you get like 30% sustained performance on AMD. There is no functional way to get anywhere close to their flops that they advertise. And stepping forward we started doing this with training and inference benchmarks as well. And ultimately we do a bunch of microbenchmarks of chips as well but most of the focus of the technical team at SemiAnalysis has been over the last year roughly has been InferenceX which we launched about 8 months ago and we've, you know, I think that's the whole point of that is to show vendors, make vendors stop lying and they can put out their lies but the moment they put out their lies people can just check the real numbers and see that that doesn't matter and index on the real numbers. So I think that's sort of the reason why I think benchmarking is so important is to check against the monoliths like Nvidia and AMD and others who lie.
Cool. Hey, so I'm Micah. I got into this in a kind of different direction to the way that the others here did. So I in like 2022, 2023 was trying to build a completely different AI startup and was building a thing, like starting to use the LLM APIs, became convinced that OpenAI was right and LLMs were the future. My co-founder and I started Artificial Analysis to publish independent benchmark numbers for the language models for the APIs and how much they cost. We figured that someone's got to go hit these things every day. Someone's got to see how fast they are, and then put all that data together so that developers can see how to do the trade-offs between which models to use, which infrastructure to use, how to bring that together in applications. Honestly, part of the reason is just that I was lucky to have very high inference bills in early 2023 before that was like a big thing. And so was obsessed with trying to find the minimum viable model for each task within an application to serve stuff. So at Artificial Analysis now like what we do is benchmark all the models, benchmark the inference APIs and more and more have like tried to take this full-stack approach to trying to understand from the tasks that we want to get done and the agents that are going to do them through the models that are doing them through the infrastructure down to the chips how we can test all of that together. I'd say that to us like what makes our benchmarks great is trying to make them as useful as possible for developers and companies who are making decisions. So that's why it's very important to us to get our intelligence index out for every model but be able to show that in context of all the trade-offs that developers and companies have to make when they make those decisions.
That's good. The AI benchmarking world is evolving from the time it started to today. And I'd love you guys to speak on kind of how it began and maybe we can start with you David about how it began and how, you know, two or three years ago and how we see it today and where it kind of needs to go moving into the future.
Yeah. So I'll take us back maybe more than three or four years, but the genesis of MLPerf, it was certainly not the first AI benchmark. We started with training and there were several other efforts before but many of them they did not, none of them took root. None of them really had the right ingredients to be a good benchmark. Taking into account accuracy, having a thoughtful approach to architecture neutrality and inclusion, right? If you want a benchmark to measure across all the different solutions, they all need to run and it needs to not advantage the incumbent. You need to actually have a coherent thought about accuracy. And I remember back in the day people would say, 'Oh, you know, my training speed is this many images on ResNet at batch one because at the time this particular company thought that batch one was important.' And some of the key ingredients to having a successful benchmark was actually figuring out, okay, what should we allow people to vary? What should we fix? What is truly important? Like the important thing for training is you've got to hit the target accuracy. You know, a wrong answer doesn't actually count. You know, some of the other things that we decided to leave implementation flexible was precision and so forth. Many people stress about a particular precision but that's an implementation choice. At the end of the day you care about what's the actual accuracy and how fast do I get the answer? What's the cost? And so those were the nice simple days compared to now. I think one of the things that to me is very exciting is we've now seen the shift to inference becoming the dominant workload and the complexity there is huge. It's rich. Right? I like to say that inference is a lot like salt. Very few people eat just salt. But if you sprinkle a bit of salt on almost anything, it gets better. Right? That's what all of our cookbooks teach us. And so we're going from a world where AI in general is sort of a more specialist thing. You have, you know, AI infra teams towards a world where AI is going to be everywhere and everyone needs to think about it in sort of every application. And that creates just a huge opportunity. It makes measurement more challenging because now you're starting to think about hybrid workloads but it's also much more valuable, right? We have a broader audience of folks to deal with. You know, Micah said at the time what triggered him was developing applications and trying to figure out what was the right tool to use. A year from now there will be many, many, many more Micahs who are struggling with that question which I think makes the importance of getting good benchmarks that are informative and relevant even more critical.
The evolution of benchmarking, where have you seen it, like where do you see it today?
Yeah, I mean I think the reason we did our InferenceX benchmarking is that vendors just when they lose they don't submit, right? And that's the challenge is they don't like to show when they lose. And then furthermore the numbers they do show often are something that you can't actually run, right? You are like, oh cool, that's awesome performance, let me go and try and do that. You try and get any framework out there whether it be open source stuff like vLLM or closed source stuff like TensorRT-LLM with all of Nvidia's flags and such, there's no way to get the performance that vendors like to claim. And so our whole thing there was well how do we actually have a benchmark that represents what people actually get on performance. And furthermore with software changing literally multiple times a week, right? PyTorch has nightlies, vLLM has nightlies, CUDA drivers update constantly, you just go through the whole list, there's dependency changes on the stack almost every day if not every day. And so you end up with what is today's performance is a question that was unanswered. And so that's what we started working on with InferenceX. How do you measure today's performance on today's software on a variety of chips on a variety of models not at the whims of vendors backing out or not, right? And if a vendor loses in one scenario, tough luck. Hopefully they got optimized for that. And so that's the whole point of InferenceX. And it being, you know, and once you have this entire curve of batch versus throughput. And any point on the curve a user could want, right? Someone might want low batch, like David said, most people don't, but some people might want low batch sizes. And that was the problem with most benchmarks is at the time was no one was even doing high batch sizes. And the flip side is some people may show only their super, super high batch workload, but then ultimately if you actually try and use a super, super high batch size inference server, the speed is so slow that users can't use it. But it turns out there's offline use cases. So you have to show the whole curve of, you know, low batch size to high batch size and the sweet spots in the middle and then make it so people can select which one they actually want, what speed, tokens per second, what cost trade-off they want to do and be able to run it, you know, be able to grab the image and run it. And that's sort of where we come out with InferenceX is what can a real world user utilize today and how does that change over time because a week from now things will change. As an example, when GLM came out, from, I believe, Zhipu AI, they're the only public AI lab in the world actually, interestingly, in China. But they're a public AI lab, GLM-4, leading edge model, at least at the time of release. Kimi's a little better now, but it came out. We implemented it day zero. Nvidia and AMD were both pissed, but who cares? And then within weeks, you know, even within days, performance started rising because they were able to upstream performance improvements. But for someone who on day zero wants to run this brand new model, how do you go do that, right? And what performance can I get and what can I expect? And three days later, the performance is way better. But that's the point. We want to measure what's actually happening. So I think benchmarking is useful insofar as much as it is useful to the end user. Yeah.
Yeah. I 100% agree with that. Right. Like I think this point, there's an interesting point here just about who the benchmarks that each of us are publishing are for and that actually has changed a lot over the last several years because I completely agree, GLM-4 comes out, great model by the way, if you want to use it like you need to know how it's going to perform on the different ways that you're actually going to use it and no one's excited about wanting to hype those numbers on day one whether it's the chip companies, the software companies, even some of the like serverless inference providers. No one's going to have the best thing, but if you're going to switch to it, you care about that. Anyway, there's a big picture point that I just want to make here that there is an interesting thing that this question about like where is AI benchmarking going is like shockingly broad now at this point because what we're talking about is testing what these like weird black box models actually do, what they're capable of. There's no like we don't even know like what the specs are in any of the frontier lab stuff anymore. But even for the ones where like you can look at all the details of the open weights models, none of that really matters like what you care about is the stuff that you can test about what can this model do, how much is it going to cost, how's it going to perform in different ways. So a fact that I think is quite interesting is that we have this chart of our intelligence index versus the cost it costs us to run all of the models in that intelligence index. Right now, there is a 1,000x range on the x-axis between the most expensive models and the cheapest models that we evaluate on the site. You can actually go even cheaper than the far left hand side of that on the x-axis if you're self-hosting and prepared to be like on the part of the curve where you're throughput maxing and don't care at all about interactivity, love a good high throughput, high batch workload. And so that's just a spread of things to think about that is not that normal in most markets for there to be a clean curve of basically substitutable products over a 1,000x range and how much they cost. I don't think that's going away anytime soon. If anything, we might add an extra order of magnitude to that as like very tiny models on the edge that can do a surprising amount, keep getting better, and as at the very big end, we see the 10 trillion plus parameter models come out of a whole bunch of labs pretty soon.
Yeah. Yeah, that's really good. I want to touch on something that you brought up, Micah, about, you know, who are these benchmarks for? And the, you know, who is the audience that these are for? Are they the researchers, the buyers, the investors, the engineers? Is it possible for a benchmarking firm to go after all of them or do you guys have a specific focus that you're trying to target with your benchmarks that you're going after? I would love to hear kind of who is the audience and if it's unique, I would love you to expound on that.
All those of you all do. This is a kind of interesting question actually for all of us.
I think you enumerate the audience is quite big, right? I think it's everything from investors to journalists to all levels of the supply chain from developers to silicon engineers working on next generation architectures. And there is a real tension in terms of who do you want to serve, who can you serve? What can actually be accomplished? Because some of them have conflicting needs or desires. Ultimately I think the key to a successful benchmark especially given the changes in AI which I sort of sum up as being like a combination of broader audience and greater velocity, right? It, you know, in the last two years we're seeing every two weeks a new model, a new capability come out. It hasn't, you know, 2018 to 2022 was not like that. It's how do we drive the information out there in a way that people can use it and I think once upon a time it was sufficient to have, hey, okay we've got source code available, right? That's fine for AI engineers who are on infra teams who are used to thinking about that. There are a lot of people who need information that, you know, they're not going to go spelunk through GitHub. And ultimately buying is a critical decision and part of it is how do you help make that decision? How do you help inform people about what they wish to buy? How they might wish to tune it. Part of it is also a good benchmark should hopefully reveal and enlighten what are the tips and tricks that the community is going to use to drive up more performance. So those are just a few of the things I think about.
Would you say MLPerf does it have a spec like I know you guys started pretty broad early on but do you have a specific audience that makes the most sense for you guys to?
