We're connecting chips that have never been connected together before because no one has taken chips from vendor A and vendor B and plugged them together and orchestrated a single workload across them.
Hello everyone. Today we have special guests from Gimlet Labs. We have Natalie and Belt here and we're going to talk all things heterogeneous silicon and rethinking the data center. So let's start, Natalie, with you. Our audience probably doesn't know you guys or Gimlet, so tell us more about you.
Not yet. Yeah, there you go. My name is Natalie. I'm a co-founder of Gimlet. Gimlet, we can go more into it, but what we are is we're an inference cloud built for agents. And one of the key aspects about our technology is that we've built this inference cloud across heterogeneous hardware. And we can get more into that, but we think that that is going to be the future of inference.
Nice. Exciting. And Belier, who are you and how did you get to Gimlet?
I'm Belier. Nice to meet you everyone. I joined Gimlet roughly five months ago. Before joining Gimlet, I've been at Intel. And Gimlet was one of my portfolio companies that I've been working very closely with. I'm amazingly excited about what we are building at Gimlet. So I ditched my corporate job and jumped onto the startup again and trying to build a very exciting business. It's been an amazing five months so far.
Nice. Exciting. So, okay, I'm just going to try taking us through some of your slides that I clipped from blogs and found online and whatnot to just get your reactions and have you talk through for our audience kind of in real time what you're trying to do, what problem you're trying to solve and why it's important. So, you know, I thought this was compelling. You know, I've written a lot about this shift from a one-size-fits-all GPU to multi-vendor, multi-silicon environments. So I was excited when I saw that you guys are thinking about this too. Natalie, tell me, make the case for heterogeneous infrastructure.
Yeah, we'll do. So I guess just starting with the context that we're all probably familiar with. It feels like every day you hear an announcement that one of the large frontier labs has made some kind of compute deal for capacity with some chip vendor, whether it's like Trainium, AMD, TPUs, Nvidia. And I think another piece of news you hear, it feels like almost every day, is that there's a new accelerator company that just launched. They have amazing performance on inference. They're designed for inference specifically. And I think that what we're basically seeing in the broad context is that everyone's extremely capacity constrained at this point, trying to scale out their inference. They're trying to improve the performance of their inference. They need as much compute as possible and then they also need specialized compute potentially to make it even faster. So then how does that all fit together? And sometimes people ask, oh, is this new chip going to be the GPU killer or something like that.
So the way that we see it at Gimlet is a little bit different. We think that all of these options are really great for different purposes and that's important because agentic inference is not a uniform workload. Different parts of it have different compute needs and different bottlenecks. And so when you think about a really, really large scale workload that you need to be very fast and efficient because we're pouring trillions of dollars of capex into it, then you want to start thinking, okay, how can I optimize my attention of this model? How can I optimize my speculative decoder or my tool calls? Each of those components actually benefits from a different type of hardware because it has different trade-offs. So what we see is that the industry is moving toward a heterogeneous stack for inference in order to meet the performance needs.
Yes. So yeah, I found you guys had this slide here and this feels like it's exactly what you were saying, which is breaking down the workloads that are running at scale. It used to be there was a time when it was kind of like let's accelerate everything and we're not sure what the dominant workloads are. So like a GPU can run high performance compute or scientific compute or AI. But now obviously all of the inference that's happening is really about like LLM inference, these few frontier labs at scale. And so now you can, it feels like, start to take that inference workload and ask what is the right silicon for this workload. But maybe to the point of the table which I'll have you talk to here is like what are the different parts of that workload? What are their system requirements and how might those actually fit onto different hardware even?
Yeah. Yeah, for sure. Like I think that one thing about GPUs is they're incredibly versatile. So we definitely think they're going to be an important part of the inference stack. When you look at this table, we have it broken down by different very, very high level phases of inference, kind of showing like the resource needs for each of them and how they vary and how you actually can't have one chip that is optimal for all of these. It's just literally not possible. But each of these is a critical stage, right? So how do you solve that problem? Our core belief is that you solve that problem by disaggregating the workload and running each segment on the chip that's the best suited for it.
So the other thing I want to point out about this table is that even this is very, very coarse grained. You can subdivide each of these components into more segments, each of which have distinct bottlenecks from each other. So it's one of those problems that even at this level I think people will benefit from disaggregating, but we're thinking even more so than that, like even within LLM prefill, how can we disaggregate that further?
Okay, and I know we're probably jumping ahead, but say more here. So you're talking about how can we split the workload up ever finer and finer and then apparently in real time being able to sort of distribute that across the correct hardware, right? And you also don't want to indefinitely subdivide, right? Because there is cost between sending the data from one chip to another, but it's about like expressing the workload, finding the optimal points to split it up, and then scheduling and scaling it across the available hardware. Okay. So, do you do that in advance of running it then? Like you look at the workload and kind of figure out where those points are to break it up.
Yeah. Yeah, that's a great question and I think some of the other slides will go into it a little bit more, but the way to think about it is that we take the workload, we trace it. So like if you give us like PyTorch, we'll trace that. It could be something else too and then we'll actually turn that into a graph representation and then our orchestrator and scheduler basically figures out how to segment it into its component parts for further compilation. So we basically trace it. We walk that graph and we understand what's there. We break it up and optimize how we do those splits and then for each of those segments we'll lower it to the target hardware. And so one thing that I also like to point out here is we really work closely with our hardware partners because we're trying to use the frameworks that they have available at the low level, not trying to create a stack that is a programming language for every single chip. So once we have those segments, we'll actually compile them and lower them down to like for example TensorRT on Nvidia or other similar frameworks on other hardware.
Sure. Gotcha. Fascinating. So I feel like what I'm hearing from you is that this is obviously more than just a hardware play, but obviously you're doing a lot in the software stack to really orchestrate is what I'm hearing.
Yeah, that's right. I mean, we think of ourselves primarily as focusing on the software layer. We have to tap into hardware as well because we're connecting these different platforms together. Like we're connecting chips that have never been connected together before because no one has taken chips from vendor A and vendor B and plugged them together and orchestrated a single workload across them. So we end up having to also play in that layer a bit too. But what we're really, really emphasizing, you know, at Gimlet is the software layer for orchestrating across this hardware. And we think that like to the slide that you just pulled up, we think that this is a problem that is going to compound, not ease over time because everyone is still coping from the massive scale up of like simple LLM inference, right? But what everyone's moving to, and you know, we see this with coding agents, is multi-step agents that they're doing searches, they're running things on your machine, they're maybe calling out to other agents. And these are even more heterogeneous than the LLM chat models, which were much more heterogeneous than people really even account for on their own. Once we start moving to background async agents that are all communicating with each other, they're multimodal, there's different model types. I think that the whole problem of the inefficiency of a homogeneous stack is just going to get completely untenable.
Yes. Yes. That makes a lot of sense. We're in a world where we're moving especially to where it's not just the human interacting with the LLM, but you've got agents. The agents are doing different things to your point, calling different models. So there's lots of opportunity to optimize. So again, maybe thinking like this sort of agent end-to-end workflow, like what does that look like from an orchestration perspective? You have an illustration here, but in my head it just feels like very complicated when I'm thinking about agents tool calling and all that stuff. Like talk to me more about this orchestration layer.
Yeah. Yeah. And we touched on it a bit before, but I think that we think about optimizing and orchestrating across an entire agent, not just an individual model, right? And I think that that's what you're saying. Yeah. I think that like these things, we represent them as like graphs in our system, right? And at the end of the day, we don't really care what type of model it is, what things it's doing, as long as we can represent it in our compiler's framework and then figure out what its bottlenecks are and then schedule it on hardware. So like whether it's one model, two models, models with functions, it's all kind of the same in the way that we've designed our system. The important part is that we can trace that entire thing and then split it up and then like this diagram shows, route it to the appropriate accelerator.
And in this diagram you showed GPU, specialized accelerator, so maybe like an SRAM-heavy one, and CPU. Tell me, CPUs are all the talk lately. Tell me how you're thinking about CPUs. Like what type of workloads are you putting on the CPUs?
Yeah. Yeah, that's a great question. And I think that it's been really awesome to see the excitement about CPUs recently because they are a really important workhorse of these agentic workloads because a pure LLM or a pure model, it only has so much capability unless you can actually connect it to the outside world and ability to do general purpose tasks. So I think the most obvious application of CPUs is things like tool calls. But you can also use them for things like smaller models or data processing and other types of things that benefit from the CPU's trade-offs. But yeah, I would say that like tool calls for me are the most exciting thing. And when you actually run that tool call in the same place that you're running the LLM, it really improves the end-to-end latency of the overall agent.
What do you mean by in the same place as the LLM?
Yeah. Yeah. So like for example when I'm using a coding agent today, the LLM is running on someone's server and then it's coming back to me saying to my machine, please look up the contents of this file or please do a web search and then that is executed from my laptop. This introduces like a very network-bound aspect of the workload because it has to constantly jump back and forth between my laptop and where the model is running. And so what I'm saying is that for cases where you can actually run those tools on the server side, you end up with much, much better performance.
Yes. Yes. Okay. So would you say then especially in your architecture, the CPU rack should be like in the same data hall on the same network or is it just like as long as it's off of your laptop and running in the cloud, maybe there's like lower latency?
It depends on the needs of the workload, but what we would generally say is that the way we approach it at Gimlet is we want to connect all of this hardware together through high-speed fabric. And so that's why we're not just saying this data center is for hardware A and this data center for hardware B, but we're actually physically connecting these racks together. So yeah, in general, like I think that it's better the closer it is.
Sure. Totally. The reason why we want that proximity is actually latency because there is a big demand for really fast tokens and higher user interactivity. This today usually comes at the expense of a throughput hit. And in a world where everybody is power constrained, capacity constrained, people have to make really hard choices whether am I going to have a throughput hit but offer high, low latency tokens or am I just going to optimize for throughput? By putting these different types of hardware in the same data center, interconnecting them, we're trying to give customers a solution that actually expands that barrier where they can make these choices with less of a tradeoff on either end.
Yes. Yes. Okay. Interesting. So at the end of the day, if we want the as fast tokens as possible, you're saying we should disaggregate the workload and put it on the right silicon for that shape of the workload. And that also we need a network and I think this slide talks about it. We need a high-speed fabric and ideally you would have all of the hardware that you're scheduling across sitting on the same fabric to reduce latency.
Yeah, that's right. And I mean I think that for some types of disaggregation this matters more than others. So for something like prefill-decode disaggregation you might be okay with a hop because that's only happening one single time between the ingestion of the context and the outputting of the first token, then hop to emitting every subsequent token, but for more fine-grained disaggregation it becomes more important.
Sure, that makes sense. Okay. So at a high level, we've talked through some of what you're trying to do, which again reflecting back for listeners, you're saying, "Hey, what if we built an inference cloud for agents where actually inside the data center there's lots of different kinds of hardware, and we'll write a software stack that's like an orchestration stack that looks at the workload, figures out where's the right place to break it into like little subtasks, if you will, and then we will give it to the correct hardware, whether that's CPUs or SRAM accelerators or HBM accelerators and we'll have it all on a high-speed fabric so it can all communicate really well." So I guess that leads to the question then of who's this for? Who are the customers and why is your cloud going to be compelling for them? So Belier, I'll hand it off, like educate us on the customers.
I put our customers maybe in a couple of big buckets. The first bucket is frontier labs in my mind who are making all these contracts with many different silicon vendors. But again everybody is power constrained, capacity constrained today and as we talked about they're all trying to solve the problem of how can I provide the fastest tokens which is better user experience without compromising my throughput or getting as much throughput as I can from my existing investment. This is a never-ending problem as the baseline keeps moving and the capacity constraints become more and more of a bottleneck for everyone. That's one bucket. The second bucket of customers we get a lot of interest is from sovereign cloud vendors who are interested in supply chain diversity who are putting together multiple of these contracts in place but lack the capability to be able to serve them at scale. Bringing up a new hardware vendor is a lot of work. Porting one same workload from Nvidia to AMD to Matrix, it's a lot of work. What we're talking about is not saying that we will take your workload and we will port it. What we're saying is you shouldn't be worrying about these different hardwares and porting your workload to each and every one of them separately. You should just make an API call or you should have an intelligent software stack if you're deploying our software stack in your data centers that actually takes your workload and figures this mixing and matching algorithm itself rather than your engineers trying to write kernels for each and every one of these hardware. This also creates a big bottleneck for them to get these new emerging architectures because who's going to write those kernels for those? It's pretty hard. The third set of customers are what I call the up-and-coming AI natives who are buying tokens at scale. Eleven Labs, Notions, Gleans, Harveys of the world. Companies who are building the next generation diffusion models, very latency sensitive. They're amazingly constrained what the current infrastructure is offering them, which is we have a good enough product, but it doesn't give the latency or the fast tokens that you need to be able to innovate the next frontier tier of user experience. The first two buckets for us is a combination of both us deploying our software in their existing data centers. The second, third tier of customers is mostly around customers who buy tokens at bulk from our new cloud infrastructure.
Okay, this is super interesting. So I want to unpack this and go into each of them. So let's start with sovereigns. So sovereigns, what I heard you saying is like, hey, you're a sovereign, you're standing up your own data centers, you're going to buy from different vendors over time so you can have that supply chain diversity. And then you've gotten yourself into a situation where you already have different hardware, but now you're stuck with like, oh man, that has increased the amount of software engineering we have to do because now we have to decide maybe manually or something which workload goes where and we have to write optimized kernels to run on the different hardware. And so it's like a software burden on maybe a customer who doesn't have a huge, huge software team. So you guys can come in and say, "Hey, we'll take a look at your hardware and we will help you orchestrate across that hardware." Is that kind of ultimately?
That's ultimately what we're trying to go for.
Okay. Okay. Okay. So that makes a lot of sense. So you're kind of like you're the software engineering team that they need.
The software platform that they need. Right. Today most of these infrastructure is being set up as a bare metal as a service infrastructure which has its own challenges, right, as we discussed from a software engineering perspective. What we're offering them is not a set of software engineers. We're doing this work for our own neo-cloud offering anyway. We are building this orchestration stack in deep partnership with those hardware vendors for our own business. What we're offering them is a ready-made platform that we can deploy in their existing data centers for them to very quickly get to market with the existing investments that they're making, but not only time to market but also get better throughput, better capacity and better user experience from that as well because all the sovereign clouds also, they just don't want to build this for the sake of building it, right? They also want to be at the frontier of the innovation as well. If you look at Europe, there's a lot of government funding that's going in this area for them to be part of the innovation ecosystem. Same in Middle East, same in India, same in Asia, right? How can you give them an offering that actually helps them get there faster, differentiate them is another part of the equation. And the other one is they are very keen on supply chain diversity. Having an Nvidia and AMD doesn't solve the problem, right? There is a lot of hardware innovation that's happening outside of US as well that also has the same issues. If you look at Korea, there's really interesting chip companies that are coming out of Korean ecosystem that we're talking together right now and they're also thinking through how can these type of emerging hardware architectures can also be consumed without the software burden because being able to do this kernel engineering, the software model porting is a lot of work.
Sure. Yeah. Fascinating. I didn't think about the point that like we tend to focus on American chip companies, but there's actually other chip companies elsewhere. And so, not only for sovereigns, can you solve the like, hey, you don't have to worry about software, our platform solves that for you. And then I did like your point, which by the way, we will make sure that it's highly optimized. So, it's not just that you got it to run, but we're going to optimize it for you. But then on top of it, yes, you can as a platform, you can take on the burden of getting comfortable and making sure that you work with all sorts of vendors from different countries because that makes sense for you as a platform and then that's something that you can offer to all of your customers.
