Well, gentlemen, we're here at AI Infra 2026 and I'm delighted to be hosting this panel with some luminaries within this space. We have Sid, the CEO of d-Matrix, Anton, the VP of product with SambaNova, and Naveen, the CEO of Encharge. Welcome, all three of you.
Absolutely. So I'm sure we'll have some really interesting conversation. This AI sector seems to be really quiet these days. So not a lot going on, right? [laughter]
Well, for the last few years, the AI infrastructure strategy for most data center operators and hyperscalers was simple: buy as many processors as you can. But as we move deeper into the inference era, power consumption, memory latency, context window expansions, and rising unit economics are forcing an approach to shift. What is the single biggest inflection point you're seeing today that proves custom silicon is becoming critical for the next chapter of AI infrastructure?
So Justin, thanks for having me. Okay, the way at least we see it is, there were really two watershed moments in the last four years, call it, around AI inferencing specifically. You know, training has been around for a while, right, starting 2015, and that was a world in which GPU compute dominated a lot, right. Then we had the ChatGPT watershed moment, end of 2022, and we really, for the first time, saw the transition from pre-training, training to inference, right? People first started worrying about what does it take to deploy these models at scale and the unit economics of running those and efficiencies around that, etc. But then there was another watershed, there were multiple other smaller ones, that's, you know, going back to how boring life is right now. But there was a rather major watershed in at least my mind, which was the launch of agentic, end of 2025, when, you know, things like Codex and Claude Code and then OpenClaw came around at the same time and it unleashed the powers of agentic coding and the agentic frameworks matured, right, and people just for the first time understood what agentic can do for you, right. And so the last 10 odd months have been all about agentic and that has driven diversity of compute because what happened is the demand for compute just shot up. You see token consumption in the last, you know, 10 odd months, it's just completely different trajectory compared to the months before it, and that forced the fact that you can't be in a homogeneous computing world anymore. You need to move to a world where there are other options. You saw TPUs proliferating. You saw other XPUs proliferating. You saw CPUs making a comeback. You saw, you know, all kinds of compute kind of, you know, working together. We saw it at d-Matrix for sure. You know, hey, can we work together with GPUs to augment GPUs for specific applications. Latency became a huge deal because as these agentic applications took off, cybersecurity agents took off. I mean, the latency becomes like the central theme because, you know, machines interacting with machines, pro coders who want to interact with the tools in real time, cybersecurity agents trying to shut down rogue agents before they can do too much damage. I mean, everything is around, you know, how quickly can we, you know, stop some of this stuff or interact with some of this stuff, right? So latency became like the central theme and that drove a different type of compute which is more latency sensitive. And so throughput was important, is important, will be important. GPUs are great, HBM-based solutions are going to be great because we had a huge uptick in demand, those solutions will do well, but on that Pareto curve now suddenly there are other points on that Pareto curve that are also becoming equally important, which is driving diversity of compute. Not everybody cares just about throughput. Some people care about latency, and some people care about TCO, and some people care about energy efficiency. And so there are all these points in the Pareto which is driving a diversity of compute, which is great news for everyone here, right? I mean, this is exactly the world we all want to live in, and that's the way it should be. I think everyone should thrive in a rising tide, right? So, no, very exciting.
Yeah, I think very similar perspective. Agentic AI just has increased orders of magnitude the number of tokens being consumed within, you know, a 9-month, 9 to 12-month period. And we're absolutely in a capacity-constrained world as a result of that. I think the only remarks I would add on top of what Sid said is the reasons for these bottlenecks. Obviously, demand is going to be impossible to fulfill probably for the next number of years at least. But to what extent can we fulfill demand, it really comes down to how do we utilize infrastructure, data center infrastructure that exists today. A lot of the new-fangled data center projects are running into lots of issues, logistics issues, construction issues, legislative issues. So being able to utilize existing brownfield infrastructure and capacity does help to unlock a lot of those things that are inhibiting our ability to really provide more inference capacity. As well as that, and the problem there for GPUs, especially the next-gen GPUs, is more and more Nvidia and AMD in particular are going to these extremely dense form factor racks that require very, very sophisticated liquid cooling systems and a lot of specific data center requirements. So they can't go into these existing data centers. So being able to augment capacity by having systems that can, I think, is very, very important, a big focus for us at SambaNova. And as well as that, just utilizing the energy that you have and being able to generate as many tokens at the points in the Pareto curve that are interesting to customers, which more and more, to Sid's point, is moving further on the x-axis, lower latency, faster speed, models are getting bigger, context length is getting bigger. How do you maintain that speed and still give good throughput per watt?
Yeah. So I think, you know, look, there's this explosion in the number of tokens that we need and the sheer volume, and that's kind of what both Sid and Anton talked about. But I think the other really critical inflection point here that makes, you know, sort of the way your hardware operates in this era of inference really critical is we've had this explosion, most currently because of, you know, kind of agentic frameworks, but what you see is like literally every layer of the stack is limited by what the hardware can provide it. And I think that's the important point here, right? If you're a hyperscaler, you know, you just need to be able to provide enough capacity under some hard power constraint. If you're innovating at the level of, you know, some new AI service or new AI application, you know, your ability to derive margins from your innovation strategy and innovation cycle is limited by, you know, kind of what are the margins and costs that are derived from that hardware. So every level of this is now coming down to, you know, fundamentally what the hardware can provide, and that means we got to squeeze the most efficiency out of it. And the industry knows that, and the industry has recognized and honed in on that bottleneck. So those architectures, you know, sort of that can give us the most efficiency and the ways of using those architectures that can give us the most efficiency are going to be the critical enabler, driver, unblocker of innovation critically at every layer of the stack. So I think that's the, you know, the key here. It's both that explosion in capacity that's needed, but also that bottleneck that is limited by what that hardware level, that infrastructure level can deliver.
Absolutely. Well, I'm happy to hear all three of your comments because they seem to really be resonant with what we're hearing out at the summit today. So, Naveen, we titled this session the hardware panel because customers today don't just want silicon. So, selling chips is no longer enough. They want deployable capacity, working software, compilers, and systems that integrate seamlessly on day one. How has the burden of building full-stack hardware systems changed what it takes to bring AI hardware to market today?
Yeah. So listen, I think, you know, at the end of the day, this thing that we're doing is only useful if we can deliver the full solution. And so that's kind of where your question starts, right? I think what's interesting is, and this point was made before, there's a lot of diversity in the ways we're deploying AI and, you know, who is deploying it and how they operate and so on. So even, you know, providing that, you know, full stack and what the usage experience through that needs to look like actually has some diversity to it, right? If you are hyperscalers, you are now building in very custom ways and very, you know, sort of tuned ways the deployment of this both at the hardware level and at the software level. If on the other hand, you know, you're some sort of enterprise on-prem trying to figure out how to stand up a system, the way you want that to look and the ways you'll approach that and the requirements you have also look a little different. And it goes on and on for all of the other kinds of deployments, you know, all the way from appliances you have on your desktop to things that are running at the extreme edge. And so I think what that means for a hardware company is you need to take a strategy, either you're going to do this deployment and you build to that. And that means not just the hardware and the software architecture, it also means the way you work together with those partners to deliver the solution. If you choose to be a company that's a little bit more horizontal, then you got to figure out what is the architecture that allows you to address, you know, kind of at the right point in that, you know, kind of verticalization towards the solution. What part are you going to own and what part are they going to own? But I think all of this starts by having a hardware and software architecture that really has its eye on the full solution and, you know, sort of can provide, you know, sort of that full experience from optimized, you know, kind of execution to optimized deployment to optimized accuracy and application, you know, sort of attributes or requirements.
Yeah, I mean, I think it's hard to decouple the microarchitecture from the system. I think your system choices depend a lot on your microarchitecture. For us, for instance, a big focus is the fact that we can parallelize models across many chips. We don't need to do that. We do it because it allows us to push further that Pareto frontier on speed and latency. So that, by virtue of being able to do that without the need to put as many chips in a rack as possible, it allows us to make some interesting decisions at the system level, which again allows us to go into some of these, you know, brownfield data centers, stay air-cooled for now, although, you know, we'll have dense liquid-cooled racks in the future as well. But it allows us to make some of these decisions at the system level that we couldn't otherwise, and it opens up a lot of opportunities for partnerships. Particularly, you know, we talk about it very openly how we are doing disaggregated inference with Nvidia GPUs today and with Intel, who are a very, very strategic partner for us. But as well as that, there's a whole ecosystem around all of this. There is ODM who bring a lot of value. There is a lot of software enablement that the community is rallying behind, and by not adopting those technologies, by not plugging into them, you risk being left behind because that's the expectation from users. So I think the good thing now is, you know, everything does start with the microarchitecture and that ultimately drives a lot of the decisions available to you.
But from there, I think it's all about how do you openly embrace this ecosystem that's being created, because walled garden just doesn't work. It's not an effective strategy whenever we're in a supply-constrained world and everybody needs to figure out how to work together.
Absolutely. That seems to be a very intentional decision that Rodrigo has made very, very early on. How does that ecosystem embrace really permeate throughout the organization?
It permeates down because we live and breathe it every day. We know that we can't do everything. We're a fairly small company and we need to invest our resources as effectively as we can and win together with partners. And I think we have always embraced this, and now Rodrigo is as committed to it as ever, if not more so. And the big catalyst obviously was our Intel partnership that we announced, and we've really come to realize just how beneficial it is to have these deep, entrenched partnerships, and that's motivated Rodrigo to go and figure out, okay, how do we find other ways to plug into this ecosystem and embrace it.
Yeah. So, I mean, couldn't agree more, right? This is an ecosystem play at the end of the day, right? You know, silicon is a big part of what all of us do here, right? I mean, I'll just tell you our story. We started 2019, you know, we came from a background building really good silicon, right? And the world was very different back then. I mean, this concept of an ecosystem didn't even exist because people didn't know what the ecosystem was. Inference didn't exist, right? In 2019, spent so much time just educating people on what inference is. The training ecosystem did exist, right, and mostly Nvidia at the time, but everything changed in the last few years, right? So the biggest transition we had to make as a company was to go from a chip-scale company to a rack-scale company, like, and we had to do that, you know, in the span of like two years, right? And that was a very steep climb for the team, and, you know, we did a couple of acquisitions along the way because I realized, you know, it's impossible to organically grow into that type of company just by hiring your way out of it. You have to acquire, you have to partner, right? So we've been actively, 2026 has been a year where, you know, I did two acquisitions on software systems. We acquired a company called Giga IO, we acquired a company called Walaroo, to change the makeup of how we approach this problem. But partnerships are super critical. I mean, we just announced a big partnership with Nvidia just last week around NVLink Fusion. Prior to that, we had talked about a partnership with them on disaggregated heterogeneous inference around inference clouds like Parasail and Gimlet that we're working with actively. So, you know, take older Blackwell, Hopper, Ampere GPUs, turbo boost them with our, you know, solution, like you retrofit them and suddenly you got a life extension on those, right? So it becomes an ecosystem play, it's beneficial to, you know, older GPU fleets, but the new partnership that we announced is the forward-looking one, right, which is around NVLink Fusion where we're saying, okay, moving forward, we need to plug into an ecosystem that is already fairly broadly deployed, that has gone through the learnings, that is, you know, validated, deployed, operational at scale with a, you know, robust supply chain to back it up, right? We can do that all on our own, but that's going to take us years, right, as a company. So why reinvent the wheel when there is somebody out there who's willing to kind of, you know, allow you to stay in their house and, you know, kind of focus on what you do really well, and this is truly a, you know, joint development partnership. So they are helping us with the rack, we are helping, you know, we're building the trays, and it's really a highly intertwined co-development partnership where a lot of our system and chip resources work with their software and system resources to make the solution possible. The product would still be a d-Matrix product. So I think we like the fact that they gave us that flexibility. They said, 'Look, you own the product, you own the customer, right? d-Matrix does, you, you, we'll help you put it together, but you have to take it to market.' And, you know, once you find customers to deploy it, we'll be happy to come in, Nvidia engineering will be happy to come in and help you deploy that, service the product, operate the product, triage any field issues if anything came up. You know, there's a whole triaging agreement that we have with them, right? R&R document that clearly outlines who's going to do what when something goes wrong, right? But you need these, you need these as a startup if you want to really scale into a world that is moving as fast as it is and is moving quickly to rack scale because the unit of compute is now a rack, right? It's not a chip anymore. We had to make those adjustments as a company and we did. And, you know, again, everything here, you know, both Anton and Naveen said is absolutely true. It's not just, you know, the chip. Chip innovation is important, that's your foundation, that's where your differentiation comes from, but then how you take it to market is through an ecosystem, through a solution.
Well, absolutely. And I was thinking back, the first time we met was at the AI hardware conference, and that just shows you it's now the infra conference because the ecosystem has grown. And Dave Patterson over at Google would chuckle to hear what you had to say because just yesterday he was talking from the stage about how he's frustrated because he thought that the old Blackwells and stuff he would be able to just get, and now he can't because people are repurposing them, and the length of time, the lifeline on those is much longer than expected.
Anton, I think we'd all agree that the fundamental bottleneck in AI today is moving data. And each of your companies has taken a very different architectural approach to break the memory wall. Let's examine the differences. What is a core principle of your architecture that solves the data movement problem, and why do you believe your approach is a right path for next-gen workloads?
Yeah. So I think this problem ultimately comes down to you have models and they're big, but they're also dynamic. You know, models, more and more the concept of a mixture of experts models inherently means that you don't know which bits of the model you're loading at any given point in time. And you have the KV cache, the data, and that is now becoming dynamic as well because the similar concepts of sparsity are being applied to the KV cache as well. So really, how we see this is, okay, we have this amount of memory bandwidth, it's because fundamentally it's about how you utilize your memory bandwidth. We have this much bandwidth in this size of a system. For us in SN50, it's 256 chips. So in aggregate, this is what we have. How do we use it as effectively as possible? There's a term that Databricks coined called model bandwidth utilization. It's different from memory bandwidth utilization. Memory bandwidth utilization measures just what goes over the wire. And there's a lot of spill that goes over the wire. There's a lot of useless things that go over the wire. Model bandwidth utilization is just about measuring you have this much bandwidth. How much of that is being used to load the model and to load the data? And that's the metric that guides everything that we do. So our architecture allows us to overlap the loading from memory, the compute on the chip, and the communication between the chips. That is the fundamental innovation of data flow. That's, you know, from gen one, we're on our fifth gen, we've taped out sixth gen. That has held true throughout all of the generations of our chip. Our thesis on our chip hasn't changed at all in that respect. So if you're able to do that, if you're able to use all of that memory bandwidth, you're able to have larger and larger systems, and you're able to hide all of the overhead with these systems communicating with each other in these dynamic workloads, then that's the answer. Now on top of that, the other answer is you need to give your systems enough bandwidth. You need to be able to innovate on the memory, and HBM is one solution. There are other solutions that are coming to market that are very, very exciting and interesting that provide more bandwidth, but it only matters if you can use it, and you can use it at scale.
