We're recording this less than 12 hours since OpenAI actually announced Jalapeno at Hot Chips, and I've got a flight to catch in about 4 hours from now to travel across the globe. But this couldn't wait, so we wanted to talk about OpenAI's Jalapeno inference chip as soon as possible and break it down to the level of understanding we have, from the conversations we had with people at Hot Chips. So let's get into it.
Hello, listeners. We've got another Semi Doped coming for you today. So like Vic said, Jalapeno and OpenAI's talk was so awesome, we just wanted to get straight into it as soon as we could. Obviously lots to digest. This won't be comprehensive, but I think we both have a lot of interesting takeaways as we watched it. And I'll just say that I personally thought it was the best talk at Hot Chips that I saw. I'm watching it remotely, and the first thing that came to mind was like, did they name this chip Jalapeno because they knew they were going to launch it at Hot Chips?
Yeah, that was so funny because everybody in the volunteers and organizers there were wearing these yellow shirts with red jalapenos on them. And I'm like, wait, is this like a self-organized advertisement for OpenAI Jalapeno? Like everywhere, the subliminal messaging across the whole conference was these chili peppers. And like, okay, I don't know if it's a coincidence or plan, but it's pretty cool.
Totally. Yes. 4D chess if that was the plan. That was genius. But okay, so let's get into it. So, you know, I just took some of their slides and a few screenshots from SemiAnalysis, who had an amazing article. So go read it if you haven't. Props to SemiAnalysis and OpenAI for benchmarking this chip together in advance, so that SemiAnalysis had a great article to drop. But let's start talking through some of the slides from the talk and pull out our insights. And I thought what would be really interesting to start with is sort of the vision that was set by the OpenAI team, which was Richard Ho, Ravi, and Chris. Richard was previously with Google working on TPUs and has a long and interesting legacy. And then Ravi is the chip architect, and Chris was kind of like the software co-design guy. All of them were very interesting. And early on, they said that really the two metrics that they were designing for, which I put on this slide, are the user experience, which they defined not as the time to first token but really like the time to last token, the end-to-end latency. Ultimately it's not like how quickly does this start processing, it's just how quickly does it get done, right? And then also the energy per request, so how much energy did it take to do that. And what I thought also was interesting was they basically said, look, those two are at odds with each other. You can always go faster if you use more energy. Therefore, we ought not to report standalone numbers, but we should always try to show Pareto Frontier curves whenever possible to show you as we dial one knob how it impacts the other. And so I kind of liked that because right away they're saying like, we're not trying to game this. We're going to show you the full curves.
But the other thing that stood out to me right away was like, hey, wait a minute. This is really interesting. This is a model lab whose customers are AI users designing chips, and they are designing first and foremost for the end AI user experience and not cost. And I thought that was interesting. I've seen a lot of presentations from merchant silicon vendors, and they are selling to the AI labs or to the hyperscalers, but ultimately they have a different set of customers so to speak, and that can impact even design decisions. So traditionally you hear a lot of merchant silicon vendors right away talking about designing for TCO, which makes a lot of sense. But I just thought it was interesting here that we've got a different team designing the chips that is closer to the end user, and therefore they were actually thinking about you and I using AI more than they were thinking about the buyer of the silicon. So I just wanted to pause there. I thought now that we've got a model lab kind of vertically integrating and designing their own chips, it's interesting to see how they might make design decisions differently than a silicon vendor designing the chips.
So, Nvidia and Jensen actually have said this before. If you're choosing an architecture for a chip that is based on making the chip cheaper to make or something like that, it's a bad decision. You've got to make the best performance decisions, period, because we are not in the era of trying to save money in the making of a chip, but in the running of a chip. The tokens per joule, which is a nicer way of saying tokens per second per watt, which is like too many division symbols there, but yeah, watts and seconds can be mangled to form joule. So tokens per joule. Joule is a unit of energy, watt is a unit of power, right? So the tokens per joule is a very important metric. That is the ultimate cost of ownership because you are going to put in energy and you're going to get out tokens, right? You're going to put in joules, you're going to get out tokens. So the energy per request is one important metric, but I don't think the concept is new. You know, people have been putting tokens per second per megawatt on these interactivity curves that you see all the time, and I'm sure it's there in this presentation too, all the time. So it's always like this is the y-axis in most curves. And then some form of latency or how the user is served. So on these charts we will see it's usually tokens per second per user, right? That's the x-axis in most of these interactivity curves you see. But yeah, so their idea is just essentially to bring on the TPU team and vibe code a chip in 9 months. Okay, that's their whole play.
Yes, totally. And I think maybe last thought here, there was a quote I think from Ravi as he was talking about that. It said that there's a market for tokens. It wants them fast, it wants them cheap, and making things go fast gives us joy. So I thought like, oh, cool. What a cool team to work on too. So then they had a slide where the OpenAI team gave major props to SemiAnalysis's inferenceX and said, hey, we benchmarked against inferenceX, we think it's a fair public power-normalized comparison. So again, saying we want to be as transparent about how our system performs as possible. And a couple things they pointed out is instead of picking a particular model that fit well, what they liked about inferenceX was there's multiple open-source models. So they can be tested on something that's small, something that's big, something in the middle. And they also liked that at the end of the day, all that really matters is that user experience from the entire system. So they thought this was a fair way to capture and compare end-to-end latency and how the whole system does. So tune your hardware, tune your software, tune your networking, whatever you need to tune to make the best experience possible for users. I will point out, you know, some of the feedback that happened in conversations since where people pointed out that Jalapeno was tested with pretty small input and output context length. So you know, I think there's probably questions on, okay, that's cool, that's awesome, but would love to see how it performs with like a million input context length. Did you talk to anyone on that topic?
No, actually. People were generally quite happy, but not amazed. I was more towards the amazed side. I mean, I was obviously happy about seeing the chip come out or whatever. But to me it's just like the performance. I read the SemiAnalysis article at least halfway even before I attended the talk, and it's a very performant chip. To be clear, I think even SemiAnalysis says that they have not run any agentic workloads on it, so they also have the AgentX platform that it has to be benchmarked against at some point. I'm sure they'll get there, but inferenceX is a good start. So throughout the conversation, I was trying to figure out what are the parameters here that would break the Jalapeno chip. Like, are they cherry-picking, let's say, I don't know, the input sequence length and output sequence length here that is like 8k and 1k is one metric, but I'm just always trying to think of like, is this a cherry-picked use case, you know? So I'm trying to be critical the whole time, but it's really hard to tell because if you think that this is an easy case, then you're like, oh wait, what's the number in the other case? So it's hard to figure out the whole thing.
But one thing that is very useful is this slide. Yeah, actually you should just go there. Yeah, because if anybody's listening to this, we are actually talking through the slides on YouTube. So if you want to look at the slides and see our annotations on the slides, we have them all marked up. So you know, we could obviously try to talk through some of the stuff you're seeing. So if you're watching this or listening to this while driving, shouldn't be watching while you're driving. But yeah, we'll try to explain some of these things. One thing that people immediately thought when Jalapeno came out is like, oh, it's optimized for the OpenAI models, but they actually showed that it's not so. And they actually ran GPT-OSS, which is like a narrow small model, all the way to a Kimi K2.5 model, which is a much different shaped model. So it's not designed for that. And SemiAnalysis points out that in fact they were able to play Doom on it. You know, I guess it could do other things than inference, but I'm not sure how Doom plays into this. I guess, yeah, cool, it can do other things is all I take away.
Yeah. Yeah. No. Okay. So you make very interesting points, which is like we want to ask what all is this chip good for and where does it kind of break and are they cherry-picking something here? And I think the point of this slide was the OpenAI team saying, look, we made a sort of generalized inference chip. So it is an inference chip. It is focused on inference, not training. It is focused on LLMs, not convolutional neural network inference, right? But it's general enough that it can run a small model, it can run a medium-sized model, it can run a huge model. And so I think, you know, as you're trying to think about all the different corner cases that it can and can't handle, I think probably the only knob that we would like to see more is really that context length, because I think right here they're showing, you know, hey, we can run these different models. And to your point, it's not just specifically the OpenAI models, but more broadly it can run any open-source model. Now, of course, right away you have to ask, oh, interesting. They're obviously as engineers trying to show that it's flexible enough to handle innovations that come down the pipe in the future, whether it's algorithmic innovations or harness innovations or what have you. But right away, if you're thinking from a business perspective, you're like, wait a minute. This can run other models. That's super interesting. What is the implication for Nvidia's or AMD's general-purpose GPUs, which kind of make the argument of like, hey, they can run all sorts of different shapes of models, it's not set in stone. And then when the ratio of this to that changes, oh no, the hardware is dead on arrival. And then even the implications for Google TPUs, which Google TPUs used to counter-position against GPUs to say like, hey, we are more specific for matrix multiplication and data flow with a data flow architecture, but we aren't as narrow as like it only works on a particular model. And so here is OpenAI coming in and sort of saying like, hey, we're fairly generic too, but of course still really focused on large language model inference.
Yeah, do you think just like Google TPUs, they weren't selling these initially to the general market, they were doing it for themselves. Do you think OpenAI is going to start selling these chips? Clearly, it runs any model. So do you think they're going to start selling it to everybody?
Yeah, that's a good question. On the one hand, you could say no, they shouldn't. They should use their vertical integration as a means to compete against Anthropic, against Google, have better margins, move faster, control their costs, make a better user experience, and so on. Of course, what gets complicated is if you're like on the finance team, you're like, okay, that's great, except this is so expensive to build and run these things at scale. Wouldn't it be nice to offset these costs by being able to rent them to others or sell them to others? So for example, I kind of think of the analogy of foundries and IBM, where it's like, hey, for the longest time, of course, Intel wanted to use their foundry captive. They wanted to keep it to themselves and take all the advantages for their product team saying like, hey, we're on the new node sooner than everyone else and our product team can benefit from that, and that'll ultimately benefit our business. Why would we share this node with AMD or a competitor? But then of course the cost of a fab just keeps doubling and doubling, and at some point you know you have to like amortize those costs. And so now they made this chip with a small team and they did it quickly, but obviously still the cost to tape it out and ramp it up, we're still talking hundreds of millions of dollars or whatever, and so that would be the question is like.
Is there a reason where they would want to amortize those costs and rent it or sell it to others but do it in a way that maybe doesn't lose a competitive advantage? So what I think the first thing is like, what about enterprises? Like, don't sell them to or rent them to Anthropic, but what about enterprises who want to run on-prem or run cheaper? Could you sell them to a neocloud, have the neocloud rent it to enterprises? And of course it doesn't have to be bare metal rentals, it could still just be like, I want to run OpenAI inference as a service, I just want the cheap version. This is what AWS is doing with Trainium. It's like, hey, you know, we can run our models, whatever models at 30% cheaper or whatever because it's running on our XPUs and not on GPUs with the various middleman margins. And so yeah, I'm just thinking out loud here, but I think there could be ways that OpenAI could pursue this line of business if they wanted to. Time will tell.
I think I'm going to go at this a little differently. I think that OpenAI's unique advantage here would be to make a chip specific to their architecture. I'm not exactly sure why they are showing that this works with every other model unless they want to go and sell this to other people. Like otherwise, in the spirit of extreme co-design, you want to get, let's say there is a way, and we'll get to some numbers here, but let's say there's a way to make this chip perform like a Cerebras-class chip because you can essentially finely tune the hardware for the actual software you're going to run on it and the actual workloads that you're going to run on it. Because this is the information that Nvidia does not have. Nvidia does not sell these tokens or anything like that. Anthropic and OpenAI have these chips that can be finely tuned to what their models are supposed to run. So if you can make extreme performance just by co-designing with it, they should do it because they even have all kinds of statistics and data and things like that that nobody else has about how their tokens are being used. And so they should use that to their unique advantage. I mean, I don't see why it's a bad idea to design Jalapeno only for OpenAI models and finally get performance up to Cerebras levels.
Okay Vic. So to the question of should they design even more specifically for their own models or should they keep it general, because to your point they know their models and they know what's coming down the pike better and therefore they would have that advantage of we are vertically integrated, we can co-design with our model team and run out ahead, and that could be our secret sauce is just knowing what the model team's doing and designing to that. I thought I included this slide. There was a fairly related point made by Ravi I believe during the talk where he said at design time there's two costs that we have to consider. Marginal cost, which is hey, how much will it cost to include yet another feature? We've got a laundry list of features. How many transistors or how much floor plan area would it take to include that? And opportunity cost, which is if there's user demand that you can't supply because you didn't have the feature, then too bad, you're out of luck. And he said, you know, the opportunity cost is generally much more than the marginal cost, which he said there's like a regret factor there. So ultimately they have a laundry list of features. They want to include them all and you have to think about like, ah, which ones are we going to regret if we don't do that. Now of course you take all these features and you try to fit them into your floor plan and then it's like, oh wow, we have way too many features and there's no way we're going to make this fit in the chip. So you just have to like start cutting stuff and eventually you just need to tape out the chip. So that's where the roadmap comes in. You say, ah, maybe we can get this feature in later. And so my thinking and question is like, I wonder if there are ways to still have a sort of general design, if you will, meaning it'll work for any transformer-based LLM, but yet also be designing for particular features for your model team in the chip, maybe that you aren't going to talk about, but that.
So this is good because I think by keeping it relatively general, I mean, they don't have to run Kimi on it, having generality will help them develop their own future models on it. So I can kind of argue against my own argument on that front.
The second thing that somebody in the audience actually asked is, hey, what was the hardest decisions you had to make on this chip, you know?
So I like that question because it really opens up the door to a lot of answers. Of course, Ravi was very measured in answering and he said the hardest decisions were the ones that we had to decide what to cut from the chip. Like we wanted to make all these cool things but you can't, you know, so how do we decide what to cut? So that's the hardest decision.
Exactly. Yeah. Which I thought that was an insightful question. And then I think Richard also pointed out with this next slide like, hey, but guess what? We have a roadmap of chips. We've got, I'm holding Jalapeno in my hand. Our Gen 2 is already approaching tape out and we're thinking about Gen 3. So I'm sure it's a little bit of kind of just scheduling and prioritization of like we really want this feature, but these things are not going to make Gen 1, but how can we get them in Gen 2? And so maybe.
I like the approach that this is our first take. And they made this chip in a really short span of time. It is about 9 months from first RTL to tape out, which is an incredibly short period of time. For anybody who's worked on chips would know that this is ridiculously short. Usually these things used to take 2 to 3 years at the very minimum, and it involved a lot of manual work. It seems like AI has really helped them. And I was like, wait, is this a way of saying that you should use more AI because it kind of is a proof of concept that more AI is better, and it helps OpenAI's business by showing how the sausage is made, you know, to use AI to make chips and then chips to make AI, you know. So I was like, yeah, it's cool. I don't know. It's cool if people use AI to make a chip. Yeah.
Oh, totally. Totally. And yeah, SemiAnalysis I believe made the point of like, hey, most people, their first chip, it's like almost throwaway. It's like proof that they can do it and they learn a lot through it, but where they want to compete is on chip two or three or four. Yeah.
So it's very interesting that they came out swinging with chip one. Let me actually go back a couple slides though because, you know, okay, here's a screenshot from SemiAnalysis. This is they've got Jalapeno, which is the purple line here that's out on the Pareto, has set a new Pareto frontier. It's with DeepSeek R1 671 billion parameters. So not the biggest model but a fairly decent sized model, and you can see that at given ISO interactivity it's much higher throughput and it can also reach much further out in interactivity than a GPU. Now, I think to be fair, some of the pushback was that Jalapeno will have HBM4 and a lot of these chips that it was comparing to, the Blackwell, is on HBM3E I believe. And this, you know, MI355 as well. And so the fair comparison is going to be Jalapeno against Vera Rubin and Jalapeno against Helios. But I just wanted to make the point that this is their first chip and it's obviously instantly competitive. And so I think that's also what's impressive. Not only was it fast but it is competitive as well. So that was that kind of medium model when they showed like, look, we can do models for, you know, from left to right on the spectrum. This was that point in the middle.
They also showed when they run a small model, GPT-OSS 120 billion parameters. Probably what really stood out and has very interesting implications to sit and think about for a while is, hey, look, Jalapeno can reach out to above 1,000 tokens per second per user, and that's the SRAM territory. And so, you know, GPUs just can't get there. And so then that's where GPUs said, okay, fine, we will, you have Dynamo will split the workload to prefill and decode, and we'll run the decode on these SRAM chips, Groq, or it could be Cerebras. And now all of a sudden Jalapeno is coming in and saying, wait a minute, which this is the same argument by the way that AI ASIC startups are making, which is like, hey, actually, what if you can have one chip that can do both high throughput at a lower decode speed but also reach those thousands of tokens per second. This is compared to Blackwell, of course. So if you put the Rubin system with the LPU, you know, basically the Groq chip, you can get this Pareto curve too. So it's not like there isn't hardware out there that can't do this. It exists in different forms. But it's nice to see that this chip in a small model can do over a thousand tokens per second per user.
Yes. Which again is interesting. It will be interesting to see how far Vera Rubin can get by itself on this because then you have to ask yourself like, oh okay, well you can get it with Groq, that's nice, but remember how many racks of Groq it takes, you know, of LPUs. So then you have to ask, is the cost worth it and probably the power consumption worth it to have an NVL72 rack and nine Groq racks or whatever it is, or could you just have a rack or two of Jalapenos?
So Jalapeno's benefit really seems to be the token throughput per watt. It is actually very energy efficient in generating this kind of performance. That's one of the key takeaways especially if you normalize it per watt. It's a 700 watt TDP chip. Which compared to the Blackwell, you know, the Blackwell GB200 is over 1,200 watts. So that's a big difference in the power consumption. It's a very energy efficient chip as well.
Yes, totally. Which matters. And that was another point why they liked SemiAnalysis inferenceX was it is power normalized. So it's not like, oh, just use more power and you get more tokens or higher interactivity. Well, that's kind of cheating. So, power normalized. Okay, one other thing. So, back to this slide that had their roadmap. At the bottom of the slide they called out, you know, hey, thanks to our incredible partners, especially Broadcom and Celestica. So I thought it'd be worth talking through really quick. Who are some companies that are helping and participating? And if you're thinking about like what happens if this goes really well for OpenAI and as literally one of the two biggest consumers of compute, if their own chips seem to fit their needs very well and they eventually spend more and more of their gigawatts or megawatts on their own chips, who might be some of the other supply chain vendors who will benefit? Obviously Broadcom, and then there's some other ones as well. Do you want to talk through these quick?
Yeah. So the CPU is an x86 CPU. People believe it's a Turin class CPU, which is a good CPU. I'm sure the Venice will do better if they want to do AMD. There are Intel
Alternatives. So, there's a lot of CPUs out there, but this is a Turing class CPU, which is a good one. I mean, it's a good CPU for sure. It's built on TSMC N3 node. Um, and I think they have the N3P versus N3E, uh, which are slight variations of the same process node but for different power optimization, uh, and speed. And I think the biggest speculative thing is that, uh, this uses HBM from Samsung, HBM4 from Samsung, and seemingly has a slightly better, uh, you know, speed per pin compared to, I don't know, like SK Hynix perhaps. Uh, so that's maybe giving some more boost to the performance, but that's entirely speculative.
Yes, and thanks to SemiAnalysis. You know, they talk about this, uh, it was SemiAnalysis speculated that Celestica was the partner here at the system level design. So, you know, I think you kind of think of them as like the ODM partner helping build out the full system. Um, sort of like how AMD acquired that company I'm blanking on to help them with their Helios rack design. Um, Celestica partnering here and then Broadcom. So, Broadcom, um, obviously is like, uh, the partner in helping the ASIC get built. Sometimes that can be like all the backend design. It's not quite clear, but there was mention, um, in the SemiAnalysis article and sort of a little bit in the Q&A, um, about interface IP and IO chiplets. So, I think in the Q&A there was mention of most of the chip is designed from scratch. I think this was, uh, Richard talking, the compute die I think only has some interface IP and then there's also an IO chiplet as well. A lot of the interfaces are existing IP, um, the rest of it was all fresh RTL. So basically maybe Broadcom is also the partner on the, like, IP, the IO chiplet, but then obviously as we'll get to when we talk about the networking, um, they're participating with their Tomahawk, uh, switches. So Broadcom sort of benefits from potentially IP and IO, but then definitely the switches.
Um, so let's get into the networking because obviously that's an interesting piece. So let's start talking about architecture. So when Chris Liry came up and talked about the architecture, he set the stage of like, and I think kind of implicitly in my head I'm thinking like, oh dude, like all day Sunday Monday was about HBM, the memory bandwidth wall and everything. And then he came up and said, hey, actually, uh, memory bandwidth is a limiter, but we are not fully utilizing and hitting that ceiling where literally if we don't get more memory bandwidth we're dead. Because he's like, if you run the math, and the math is here on the screen, um, we should be getting 1 to 2,000 tokens per second per user without speculative decoding, and with it we should be getting 5 to 10,000 tokens per speculative per user. We're nowhere near that. So if you just pencil out like what is the HBM ceiling, we are not there. And then the questions are why are we not there and how can we get closer to it. Um, and one of the really interesting things that, um, Chris hit on and I wanted to zoom in on was he talked about KV cache, and I have a couple quotes, I'll read them, and how KV cache is very important, but done wrong you're shuttling all this KV data around, and then also maybe you have like copies of it everywhere to try to like not have to move it around. And so, um, the question is could you think differently about it to overcome some of those trade-offs? And so, uh, let me read these. So he said, in agentic inference the largest and fastest growing data structure is the KV cache. We do not believe that the long-term solution involves moving large KV states. Disaggregated prefill is a solution for GPUs because GPUs need to build a large batch to make sampling efficient. This ends up requiring a lot of prefill replicas supplying a sampling replica. The cost of losing KV locality is also replicating these weights in every single HBM. Um, so, you know, obviously pointing out like, hey, the way that GPUs work, uh, there's some dirty tricks that they have to do to make it work. And obviously, you know, we talk a lot about how like, oh, prefill and decode and disaggregation is the way, and then here they're kind of questioning like, is it? And he, uh, Chris went, or let me pause. Do you have anything to say there?
Uh, that's a very interesting calculation essentially, that the fact that, hey, yeah, this, uh, if you have the total aggregate bandwidth, if you take all the HBM4 chips, uh, across 128 chips, which I believe is what goes into their rack, you get like one petabit per second. And then, uh, at a 4-bit, you know, FP4, if you have a 1 trillion parameter model and you have, you know, let's say 0.5 terabytes of data, you should be getting about 2,000 tokens per second, but even just from HBM. But like we are not getting 2,000, uh, tokens per second from HBM because that's the whole thing about why, you know, Cerebras is using SRAM and LPUs are using SRAM and all that so that you can get it faster, you know, above 200. I mean, Cerebras actually in a separate discussion we can probably have promises like, you know, 4,000 tokens per second because of SRAM, but the complexity of their system is enormous. This is a good argument saying like, hey, why are we not even using HBM to the max? And you can see people like Nvidia are pushing the future per pin lane rate, per pin rates from, you know, 10 GB per second to like 16 Gbit per second. And I saw some charts in the memory, uh, talks at Hot Chips that HBM5 will reach more like 23, 24, you know, Gbit per second. But are you actually using all that bandwidth and does it show up in the token count? OpenAI says no, it's not. We are not using the whole capability of even HBM. So why don't we do that? So I like that approach very much.
