Derek Dicker & Vik Malyala | Beyond the GPU

Same here. Thank you.

Absolutely. The only way that happens is by not just having a relationship between the two of us, but all the way down to the engineers thinking through what's coming next, aligning our requirements, designing products together, co-innovating together. It's been a fantastic relationship so far.

I think the most important thing is it's two engineering-centric companies that are collaborating and co-developing platforms and solutions together. AMD is focusing on the silicon. Our role as Super Micro is to see how best to design a platform or a solution that takes advantage of every one of those features, making it available for customers. So basically making it real at a server level for the customers to use. It goes very deep from the very beginning. From the first generation when AMD introduced EPYC, it was all about maximum number of cores, maximum memory you could pack in a system, and it had 128 lanes of PCI Express, which was unheard of. It started with that and we never looked back. Every generation there is tremendous improvement, generation over generation, features and functionality get added at the silicon level, and to support that from Super Micro at a system level, now at a rack scale level or a data center level, we are working towards getting the right platforms out together.

Your perspective on demand. Common knowledge, I think you talk to anyone in the industry, whether it's CPU or GPU or anything that connects all these things, everyone is saying they're sold out for the next several quarters. Take a look at the demand, it's insatiable. In the case of AI, it all started with training clusters, massive training clusters being built. But in the last 8 to 10 months, we see the demand coming up a lot more with inferencing, specifically agentic AI. The RAG models were there, GenAI models came up, and now we're talking about all these agentic AI workloads that are completely transforming the requirement for all customers. Traditional workloads are great, but in addition, people are trying to adopt AI into their workflows to improve efficiency and productivity for their companies. That is driving the entire decision-making process. The idea is to take something they feel comfortable with and use it to enhance their workloads to get better business value. That is driving the demand, along with efficiencies. As people start deploying these, they are deployed at scale, not just one or two, but clusters of them. Now people need to figure out how to optimize their cost function and total cost of ownership, bringing out the efficiencies. So a lot of these things are happening and that's driving the demand.

Absolutely. I was trying to see where to start. But what is happening is it's expensive. Anything with AI is an additional expense on top of whatever infrastructure spend people already have. Initially, the easiest way is to go to a public cloud service provider and take these tokens and start using them. Everyone is happy until they see the bill. Then all of a sudden, what do they do? There are definite benefits, so people are trying to figure out how to bring it on prem. That means they need to retrofit data centers. So what happens? If you take a look at the amount of power available per rack, that is being revised as they want to bring AI infrastructure in. Once that is taken care of, they can bring very high-density compute right next to it. So we are seeing that we can make the best use of available power and space by bringing both AI infrastructure, which is GPUs, as well as compute. We see people are more receptive to the idea of using liquid cooling. When they bring liquid cooling, they can increase the density like anything. We're talking about 40,000 plus cores per rack for CPU, not GPUs. When was the last time you heard about that kind of density at a rack level for CPU?

I think to add to that, GPUs are the most expensive part and we want to keep the GPUs busy doing the actual work. Anything that can be done to support the GPUs and make sure they are fed properly is where the compute comes in. With agentic workloads, a lot of these things can be done on a CPU far better than on an expensive GPU. That's number one. Second, so many small batches are coming in with these agents running. With more processor cores packed in, like 256 cores per CPU, with a lot of memory and IO bandwidth, a lot of stuff can be done. I was reading about the total latency impact of running these things in CPU architecture. You're talking about 50 to 80% of the latencies of these agents can be addressed from a CPU point of view. So not just for the premise that CPUs were used in the past, now we're talking about a lot of agentic AI workloads, whether it's database lookups, tool callings, all the orchestration. All this is happening on the CPU. CPU counts are increasing considerably in inference-specific workloads for enterprise customers. Coupled with properly balancing with the GPU architecture, that is what is going to provide the right value for customers.

I think that's the reason the CPU to GPU balance comes into picture. You take a look at the latest generation processor. Now you're talking about PCIe Gen 6, 16 channels of memory with 8,000 plus MHz memory. So now you're talking about how much memory bandwidth you can get, what kind of IO you can get because of faster PCI Express, and how many GPUs to a specific CPU we are mapping. If that memory and storage is not sufficient, what other storage you can add to create a more balanced compute, GPU, storage, and network with very fast interconnect, like InfiniBand for example.

One of the things that working with AMD, as they define the processor SKUs, in a given thermal design you can pack fewer cores with higher frequency or more cores with less frequency, how memory mapping should happen, how MCM designs fit into workloads. That's something we always look at as we design our platforms. We would not know whether people would be focusing on these systems for using accelerators, storage, network, or high-performance compute. But what we do know is by giving those options in the most optimized manner and being able to build these systems not just at a system level but conceptualize at a rack level, how we bring a cluster and deliver it, that's the main emphasis. It becomes an easier transition for us as we start looking into the AI-centric workloads. What kind of GPUs you are using, how many GPUs, what it takes to keep the GPU busy, and what new workloads you want to run. Traditional workloads are not going anywhere. They are still running, but the same systems can be tuned a little further in terms of system configuration to run all these new agentic and inference workloads. That's where the difference is. No one has a crystal ball. If that were the case, people would have talked about agentic-optimized CPUs five years ago. Why is it only being heard now? Because things are changing. We just need to be ready with the right tools to deliver the solution at the right time. I would add one other thing. The magic of a relationship like this is where we can communicate together on things we see coming, but be able to pivot, execute, and make modifications. The beauty is that Super Micro has a great set of technologies that allows them to build different types of systems within our silicon architecture. Our engineers have thought forward to what we may want to modulate on. That's where the chiplet architecture comes in really well at a CPU level. We can adjust solutions out in time.

I think you hit the nail right on the head. When we started looking at Venice, one of the things that we started looking at is what are the kind of environments that it is going to be deployed in besides the traditional way. For example, we have standard 1U, 2U pizza boxes with a single or dual socket that's been there like our cloud DC hyper these kind of platforms. Then we extended that into a dense compute like FlexTwin, which is liquid cooled platforms and a 2U4 four-node kind of thing. And then we have integrated switch into that for those who wanted to have a dense compute but in a smaller form factor like a SuperBlade. And we went, the kind of product line extends one after another. But then we also have seen people are using either air or liquid because things are being now dealt at a rack scale, especially in a larger deployment arena. So what we did was, what is the power delivery going to be in these kind of systems? Traditionally, you have like a 60 amp or 100 amp power drops coming in, and then you put PDUs and then you put all these systems together. That's all great, but now we started looking at what is that we can do with an OCP enhanced or OCP inspired designs where like OV3 rack with power delivery using the DC bus bar. We have kind of transformed the product line not just for the traditional ones but also to take advantage of these things. So things are being considered at a rack scale or a data center scale. What is that we need to do in a given operating environment and how do we basically pack as much punch in a smaller footprint? You kind of figure out where it would be needed and how do we get the most optimal solution there. So that's the thought process and we've been working with AMD on how best to put these things together so that it will be relevant for customers.

Well, so to start with, your workloads are changing. Some of these could be requiring more memory. Some of them could be using, take advantage of faster memory. Some of them could be requiring very fast interconnect as systems cannot just handle all the workload in a single system and it needs to be looked at multiple systems clustered together. So the idea behind this is to give that option for the customers and see how different applications will behave and make sure that we are ready for that. Let's get into the joint engineering because a lot has to happen between when you get to high yields and an OEM ready to a deployment. So what is the engineering that occurs between your two companies in between that cycle?

So I think what we are seeing is like silicon is getting hotter and we need to figure out a way to optimize the thermal design. So we need to pick the right type of form factor and the solutions associated with it could be air cooled or could be liquid cooled because not everyone has liquid cooled in their data centers. So we need to have platforms that can accommodate and provide the highest performance using air cooled and what does that system look like? So thermal design becomes very important for that. And the second part of it is the interoperability. So obviously people are not just going to just use that for a specific workload. Some put accelerators in it. These accelerators could be inferencing or network optimization or any of those things from a security point of view. All these things need to be validated within those designs. And the networking options whether people want to use Ethernet or InfiniBand or whatnot, how best you actually put it together, what kind of storage you want to be supporting in that because now it's not just about the capacities but it's about the form factors too, E1S, E3S, and you're talking about U.2 to U3, and if somebody wants to use Compute Express Link as a part of these devices, and if somebody wants to try UALink as a part of these systems, all these things we start looking at and say, okay, is AMD going to support it and if so what needs to be done, what are the BIOS optimizations that need to come up with, and what kind of firmware updates that we need to make the system run operate in a certain way. All these things are the ones that we work hand in hand with AMD. And the designs of the systems are also a function of what we discuss with AMD and also what we discuss with customers. If a customer says, 'I'm looking for a single socket and I'm looking for one DIMM per channel,' that's quite a bit different from 'I'm looking for the most dense kind of compute doing blah blah blah.' I think the idea again here is talking to customers and getting a pulse of that and looking at what kind of technology that we need to make sure that we'll be validating because we don't know whether it's going to be popular or not but we still need to make sure that the platform is designed, otherwise we'll be late for the market. We want to be as a launch partner, we want to bring as many products, the broadest product portfolio that we can bring, air or liquid, rack scale or a single node. All these things is what we are looking at to bring, and in order to do that we need to have every bit of the design checked twice. What Supermicro does and AMD engineering comes in and like, 'Hey, if you do this, is the signal integrity going to be okay? Is it going to work on this? What needs to be done from our AGESA code for you to do a code drop for your BIOS?' and things like that. A lot to be done and a lot is being done prior to launch and after launch. I think we're going to continue to refine to get the best tuned solutions for our customer.

Time to value. I think the whole idea is that you want to take systems that are fully validated and to the extent that one can, we run synthetic benchmarks and we run ISV qualifications. Ultimately, it's the customer that is going to take it and use their applications, their workloads to go do something with it. We want to minimize that pain by bringing, like, okay, time to value or time to online, make the systems, bring them to the customer as quickly as possible so that they can have a time to market advantage and they can see the benefits quickly.

I call it like A plus A plus A. Like, we want to use the AMD EPYC with Instinct as GPUs as well as the connectivity coming from either Polar or Nyx or you're talking about Volcano going forward. Why that matters is because I just want to make sure that interoperability is fully and thoroughly done, and it's easiest when it is under a single roof. So AMD has in optimizing on their end and we validate from our front. We still give the flexibility for customers to choose their own architecture for the overall solution. But the idea again here is to validate all these things together and make it readily available for the customers to deploy. Again going back to our initial thoughts, if you're putting Helios, that's fantastic, but what is the right size of compute that you want to put in front of it and what is the connectivity that we need to be putting, what network topology that we need to bring in order to get the maximum throughput from this one? So you can throw the most expensive equipment and you can actually connect them with the fastest fabric and then you can give all the power and cooling you want and you can get the best performance, but CFOs are not going to like it. The idea here is again giving a balanced architecture that is well orchestrated and how we can provide the value to customers, and that's what we are trying to do, especially when working with AMD on this compute plus GPU together to provide an optimal solution whether it's training or inferencing.

So the good thing is that I think dense compute is getting more and more common mainly because of how the industry is trending. If we take a look at our existing MI355 type of systems, Instinct 355 series, we are able to put like 8 to 10 systems in a rack getting to like 150 kilowatts per rack currently that people are already using in the data centers, so it's not going to be a total surprise to start with. But at the same time, when you're deploying it at scale, often times a traditional data center may not even fly, and we are getting strong customer demand on these platforms and they know what this platform is going to look like, and based on that they're actually retrofitting their data center or building a brand new data center. And since we are getting that pulse, what we are also able to do is like how we can optimize these compute platforms to go along with it so it doesn't look like an oddball. So we want to make sure that the compute systems are also thought about at a rack scale, think about the power delivery, the cooling, and everything so that way you don't have to start creating parallel environments with different dynamics in it. So that's something that we are working with the customers and customers are quite open and happy about it because now we are really thinking in lines of what they are trying to do and taking their feedback and start putting in our platform designs. So is it difficult? Yes. And is something that needs to be done? Absolutely. Are we doing it? Absolutely.

Yeah. I mean, I'm not a golfer, but you know you don't take a putter for a driver. So it's the idea again here is to give the optimal solution for the customer. So if people are doing small scale inferencing workloads, I mean the Instinct MI350, 355 and also the PCIe version the MI350P, these things are going to just fit fine in an aircooled environment for customers. If they want to go with liquid cooling in a small scale, they can still do the MI355 that can do that. And as for people looking for the best in class and the beast, if you want to have it, Helios is there. So the idea is whether it's air cooled or liquid cooled, rack scale or at a node level, could be like 1 GPU to 8 GPU type of systems, we want to give them the option, and that's what the industry wants and that's what we are trying to do so that we don't want to slow down the innovation, we want to accelerate together.

Where is that crystal ball? [laughter] I always...

But you're right. I think it's a very good point because traditional workloads they are going to evolve. People are trying to figure out a way to optimize that to the extent possible because now all of a sudden, let's take an example of like two generations ago systems where if you want to run an application that requires a certain memory footprint, it may not even fit in a given system. So now they need to look at how it's going to run across four or five systems put together and connect with whatever the network at that time. So now look at 16 channels of memory, 256 cores in a single system put in a dual socket, you are able to get like what, 512 cores or maybe double that number of threads and whatnot. The entire architecture can be redefined in terms of how the software is going to run. So the idea again here is if people are running certain things, they can do at least the same if not multiple times better traditional workloads, but at the same time when they are doing that, they want to take advantage of what is happening with AI. Every company is looking to see what is that they can do to improve the decision-making, how do they run agentic workloads, and how do they improve the customer experience? Think of customer support calls or an internal knowledge base, database queries and whatnot. Every one of these things will be done far better either you use a traditional workload as is or you optimize it further using some agentic workloads. Both of them can be supported with the same system. So it's prudent for anyone to take a look at what they are trying to do and how they can take advantage of these new systems that are bringing the value to them.

Right. I think going back, I was thinking through and one of the things that we have seen is especially the enterprise customers or the people who have already data centers, they are struggling to see how to ensure that they are able to take advantage of the latest gear that is coming in. What Supermicro is doing is to figure out a way to lessen the pain. We have started investing in so-called data center building block solutions or DCBS, where we are going to help them to retrofit their data centers by bringing liquid cooling into that, and could be like water towers and things like that, or could be power shelves that need to be put in this or change the power delivery to the systems. This way, one doesn't have to go and build a brand new data center, but can take advantage of all the latest within the existing data center by making certain modifications. If someone wants to start brand new, we can certainly help them on that front as well. But to just kind of wrap the thoughts on that one, if the customers want to take advantage of the latest, they are not alone. We can actually certainly help them on a system level as well as the deployment level.

Balanced platforms and architecture is the key. What we are doing together is whether you want to use the standard compute, you want to use accelerated compute, how to put these things together to make sure that it meets or exceeds someone's expectations. And in order to do that, one has to do a proof of concept quickly. We can give as much information possible that we have already done together. But ultimately, it boils down to customers using it as a POC and very quickly roll it into production. And that's what I think people have to start with.

Thank you so much for the opportunity. It's been great. Thank you.