We're recording this less than 12 hours since OpenAI actually announced Jalapeno at Hot Chips, and I have got a flight to catch in about 4 hours from now to travel across the globe. But this couldn't wait. So we wanted to talk about OpenAI's Jalapeno inference chip as soon as possible and break it down to the level of understanding we have and from the conversations we had with people at Hot Chips. So, let's get into it.
Hello listeners. We've got another Semi Doped coming for you today. So like Vic said, just Jalapeno and OpenAI's talk was so awesome. We just wanted to get straight into it as soon as we could. Obviously lots to digest. This won't be comprehensive, but I think we both have a lot of interesting takeaways as we watched it. And I'll just say that I personally thought it was the best talk at Hot Chips that I saw. I'm watching it remotely and you know, the first thing that came to mind was like, did they name this chip Jalapeno because they knew they were going to launch it at Hot Chips?
Yeah, that was so funny because everybody in the volunteers and organizers there were wearing these yellow shirts with like red jalapenos on them and I'm like, wait, is this like a self-organized advertisement for OpenAI Jalapeno? Like everywhere, like the subliminal messaging across the whole conference was these chili peppers and like, okay, I don't know if it's a coincidence or plan, but it's pretty cool.
Totally. Yes. 4D chess if that was the plan. That was genius. But okay, so let's get into it. So, you know, I just took some of their slides and a few screenshots from SemiAnalysis who had an amazing article. So, go read it if you haven't. Props to SemiAnalysis and OpenAI for benchmarking this chip together in advance, so that SemiAnalysis had a great article to drop. But let's start talking through some of the slides from the talk and pull out our insights. And I thought what would be really interesting to start with is sort of the vision that was set by the OpenAI team, which was Richard Ho and Ravi and Chris. Richard was previously with Google working on TPUs and has like a long and interesting legacy. And then Ravi is the chip architect and Chris was kind of like the software co-design guy. All of them were very interesting and early on they said that really the two metrics that they were designing for, which I put on this slide, are the user experience, which they defined not as the time to first token but really like the time to last token, the end-to-end latency. Ultimately it's not like how quickly does this start processing, it's just how quickly does it get done, right? And then also the energy per request, so how much energy did it take to do that? And what of course I thought also was interesting was they basically said, look, those two are at odds with each other. You can always go faster if you use more energy. Therefore, we ought not to report standalone numbers, but we should always try to show Pareto frontier curves whenever possible to show you as we dial one knob how it impacts the other. And so I thought I kind of like that because right away they're saying like, we're not trying to game this. We're going to show you the full curves. But the other thing that stood out to me right away was like, hey, wait a minute. This is really interesting. This is a model lab whose customers are AI users designing chips and they are designing first and foremost for the user, the end AI user experience and not cost. And I thought that was interesting. I've seen a lot of presentations from merchant silicon vendors and they are selling to the AI labs or to the hyperscalers but ultimately they have a different set of customers so to speak and that can impact even design decisions. And so you know traditionally you hear a lot of hyperscalers or sorry merchant silicon vendors right away talking about designing for TCO which makes a lot of sense but I just thought it was interesting here that we've got a different person, a sort of team designing the chips that is closer to the end user and then therefore they were actually thinking about you and I using AI more than they were thinking about the buyer of the silicon. So I just wanted to pause there and I thought now that we've got a model lab kind of vertically integrating and designing their own chips, it's interesting to see how they might make design decisions differently than a silicon vendor designing the chips.
So, Nvidia and Jensen actually has said this before. If you're choosing an architecture for a chip that is based on making the chip cheaper to make or something like that, it's a bad decision. You've got to make the best performance decisions period because we are not in the era of trying to save money in making of a chip but like the running of a chip, the tokens per joule, which is a nicer way of saying tokens per second per watt, which is like too many division symbols there but yeah watts and seconds can be like mangled to form joule. So tokens per joule. Joule is a unit of energy, watt is a unit of power, right? So the tokens per joule is a very important metric. So that is the ultimate cost of ownership because you are going to put in energy and you're going to get out tokens, right? You're going to put in joules, you're going to get out tokens. So the energy per request is one important metric but I don't think the concept is new. You know people have been putting tokens per second per megawatt on these interactivity curves that you see all the time and I'm sure it's there in this presentation too all the time. So it's always like this is the y-axis in most curves. And then some form of latency or how the user is served. So on these charts we will see it's usually tokens per second per user. Right? That's the x-axis in most of these interactivity curves you see. But yeah, so their idea is just essentially to bring on the TPU team and vibe code a chip in 9 months. Okay, that's their whole play.
Yes, totally. And I think yeah, maybe last thought here, there was a quote I think from Ravi as he was talking about that it said that there's a market for tokens. It wants them fast, it wants them cheap, and making things go fast gives us joy. So I thought like, oh, cool. What a cool team to work on too. So then they had a slide where the OpenAI team gave major props to SemiAnalysis's InferenceX and said hey we benchmarked against InferenceX we think it's a fair public power-normalized comparison. So again saying we want to be as transparent about how our system performs as possible. And a couple things they pointed out is instead of picking a particular model that fit well, what they liked about InferenceX was there's multiple open-source models. So they can be tested on something that's small, something that's big, something in the middle. And they also liked that at the end of the day, all that really matters is that user experience from the entire system. So they thought this was a fair way to capture and compare end-to-end latency and how the whole system does. So tune your hardware, tune your software, tune your networking, whatever you need to tune to make the best experience possible for users. I will point out, you know, some of the feedback that in the conversations that have happened since where people pointed out that Jalapeno was tested with a pretty small input and output context length and so you know I think there's probably questions on okay that's cool that's awesome but would love to see how it performs with like you know a million input context length. Did you talk to anyone on that topic?
No actually. People were generally quite happy with this but not amazed. I was more towards the amazed side than you know I mean I was obviously happy about seeing the chip come out or whatever. But to me it's just like the performance. I read the SemiAnalysis article at least halfway even before I attended the talk and it's a very performant chip to be clear. I think even SemiAnalysis says that they have not run any agentic workloads on it so they also have the AgentX platform that it has to be benchmarked against at some point. I'm sure they'll get there but InferenceX is a good start. So throughout the conversation I was trying to figure out what are the parameters here that would break you know the Jalapeno chip like are they cherry-picking let's say I don't know the input sequence length and output sequence length here that is like 8k and 1k is one metric but I'm like I'm just always trying to think of like is this a cherry-pick use case you know so I'm trying to be critical the whole time but it's really hard to tell because if you think that this is an easy case then you're like oh wait what's the number in the other case so it's hard to figure out the whole thing.
But one thing that is very useful is this slide. Yeah actually you should just go there. Yeah because if I mean if anybody's listening to this we are actually talking through the slides on YouTube so if you want to like look at the slides and see our annotations on the slides we have them all marked up so you know we could obviously try to talk through some of the stuff you're seeing. So if you're watching this or listening to this while driving, shouldn't be watching while you're driving. But yeah, we'll try to explain some of these things. One thing that people immediately thought when Jalapeno came out is like oh it's optimized for the OpenAI models but they actually showed that it's not so and they actually ran GPT-OSS which is like a narrow small model all the way to a Kimi K2.5 model which is much different shaped model. So, it's not designed for that. And SemiAnalysis points out that in fact they were able to play Doom on it, you know. I guess it could do other things than inference, but I'm not sure how Doom plays into this. I guess Yeah. Cool. It can do other things is all I take away.
Yeah. Yeah. No. Okay. So, you make very interesting points, which is like we want to ask what all is this chip good for and where does it kind of break and are they cherry-picking something here? And I think the point of this slide was the OpenAI team saying look we made a sort of generalized inference chip. So it is an inference chip. It is focused on inference not training. It is focused on LLMs, not convolutional neural network inference, right? But it's general enough that it can run a small model, it can run a medium-sized model, it can run a huge model. And so I think, you know, as you're trying to think about all the different corner cases that it can and can't handle, I think probably the only knob that we would like to see more is really that context length because I think right here they're showing, you know, hey, we can run these different models. And to your point, it's not just specifically the OpenAI models, but it's more broadly it can run any open-source model. Now, of course, right away you have to ask, oh, interesting. They're obviously as engineers trying to show that it's flexible enough to handle innovations that come down the pipe in the future, whether it's algorithmic innovations or harness innovations or what have you. But right away, if you're thinking from a business perspective, you're like, "Wait a minute. This can run other models. That's super interesting. What is the implication for Nvidia's or AMD's general purpose GPUs which kind of make the argument of like hey they can run all sorts of different shapes of models it's not set in stone and then when the ratio of this to that changes oh no the hardware is dead on arrival. And then even the implications for Google TPUs which Google TPUs used to counter-position against GPUs to say like hey we are more specific for your matrix multiplication and data flow with a data flow architecture but we aren't as narrow as like it only works on a particular model and so here is OpenAI coming in and sort of saying like hey we're fairly generic too but of course still really focused on large language model inference.
Yeah do you think just like Google TPUs they weren't selling these initially to the general market they were doing it for themselves. Do you think you know OpenAI is going to start selling these chips? Clearly, it runs any model. So, do you think they're going to start selling it to everybody?
Yeah, that's a good question. On the one hand, you could say no, they shouldn't. They should use their vertical integration as a means to compete against Anthropic, against Google, have better margins, move faster, control their costs, make a better user experience, and so on. Of course, what gets complicated is if you're like on the finance team, you're like, "Okay, that's great, except this is so expensive to, you know, build and run these things at scale. Wouldn't it be nice to offset these costs by being able to rent them to others or sell them to others?" So for example, I kind of think of the analogy of foundries and IBMs where it's like, hey, for the longest time, of course, Intel wanted to use their foundry captive. They wanted to keep it to themselves and take all the advantages for their product team saying like, hey, we're on the new node sooner than everyone else and our product team can benefit that and that'll ultimately benefit our business. Why would we share this node with AMD or a competitor? But then of course the cost of a fab just keeps doubling and doubling and at some point you know you have to like amortize those costs and so now they made this chip with a small team and they did it quickly but obviously still the cost to tape it out and to ramp it up we're still you know talking hundreds of millions or whatever and so that would be the question is like is there a reason where they would want to advertise those costs and rent it or sell it to others but do it in a way that maybe sort of doesn't lose a competitive advantage. So what I think the first thing that I think is like what about enterprises like don't sell them to or rent them to Anthropic but what about enterprises who want to run on-prem or run cheaper could you sell them to a neocloud have the neocloud you know rent it to enterprises and of course it doesn't have to be bare metal rentals it could still just be like I want to run you know OpenAI inference as a service I just want the cheap version this is what AWS is doing with Trainium. It's like, hey, you know, we can give you we can run our models, whatever models at 30% cheaper or whatever because it's running on our XPUs and not on GPUs with the various middleman margins. And so, yeah, I'm just thinking out loud here, but I think there could be ways that OpenAI could pursue this line of business if they wanted to. Time will tell.
I think I'm going to go think go go go at this a little differently. I think that OpenAI's benefit or unique advantage here would be to make a chip specific to their architecture. I'm like I'm not exactly sure why they are showing that this works with every other model unless they want to go and sell this to other people. Like otherwise in the spirit of extreme co-design you want to get let's say there is a way and we'll get to some numbers here but let's say there's a way to make this chip perform like a Cerebras class chip because you can essentially finely tune the hardware for the actual software you're going to run on it and the actual workloads that you're going to run on it because this is the information that Nvidia does not have. Nvidia does not sell these tokens or anything like that. Anthropic and OpenAI have these chips that are, you know, can be finely tuned to what their models are supposed to run, you know. So, if you can make extreme performance just by co-designing with it, they should do it because they even have all kinds of statistics and data and things like that that nobody else has about how their tokens are being like, you know, what kind of workloads and know-how, you know, their models are being used. And so they should use that to their unique advantage. I mean like what I don't see why it's a bad idea to design Jalapeno only for OpenAI models and finally get performance up to Cerebras levels.
Okay Vic. So to the question of should they design even more specifically for their own models or should they keep it general because to your point they know their models and they know it's coming down the pike better and therefore they would have that advantage of we are vertically integrated. We can co-design with our model team and run out ahead and that could be our secret sauce is just knowing what the model team's doing and designing to that. I thought I included this slide. There was a fairly related point made by Ravi I believe during the talk where he said at design time there's two costs that we have to consider. Marginal cost which is hey how much will it cost to include yet another feature? We've got a laundry list of features. How many transistors or how much floor plan area would it take to include that? And opportunity cost, which is if there's user demand that you can't supply because you didn't have the feature, then too bad, you're out of luck. And he said, you know, the opportunity cost is generally much more than the marginal cost, which he said there's like a regret factor there. So ultimately they have a laundry list of features. They want to include them all and you have to think about like ah which ones we going to regret if we don't do that. Now of course you take all these features and you try to fit them into your floor plan and then it's like oh wow we have way too many features and we actually there's no way we're going to make this fit in the chip. So you just have to like start cutting stuff and eventually you just need to tape out the chip. So that's where the roadmap comes in. You say ah maybe we can get this feature in later. And so my thinking and question is like I wonder if there are ways to still have a sort of general design, if you will, meaning it'll work for any transformer-based LLM, but yet also be designing for particular features for your model team in the chip, maybe that you aren't going to talk about, but that...
So this is good because I think by keeping it relatively general, I mean, they don't have to run Kimi on it, having generality will help them develop their own future models on it. So I can kind of argue against my own argument on that front.
The second thing that is that somebody in the audience actually asked, "Hey, what was the hardest decisions you had to make on this chip, you know?"
So I like that question because it really opens up the door to a lot of answers. Of course, Ravi was very measured in answering and he said the hardest decisions were the ones that we had to decide what to cut from the chip. Like we wanted to make all these cool things but you can't, you know, so how do we decide what to cut? So that's the hardest decision.
Exactly. Yeah. Which I thought that was an insightful question. And then I think Richard also pointed out with this next slide like hey but guess what? We have a roadmap of chips. We've got I'm holding Jalapeno in my hand. Our Gen 2 is already approaching tape out and we're thinking about Gen 3. So, I'm sure it's a little bit of kind of just scheduling and prioritization of like we really want this feature, but these things are not going to make Gen 1, but how can we get them in Gen 2? And so maybe...
I like the approach that this is our first take. And they made this chip in a really short span of time. It is about 9 months from first RTL to tape out which is an incredibly short period of time for anybody who's worked on chips would know that this is ridiculously short. Usually these things used to take 2 to 3 years at the very minimum and it involved a lot of manual work. It seems like AI has really helped them. And I was like, wait, is this a way of saying that you should use more AI because it kind of is a proof of concept that more AI is better and it helps OpenAI's business by showing how the sausage is made, you know, to use AI to make chips and then chips to make AI, you know, it's so I was like, yeah, it's cool. I don't know. It's cool if people use AI to make a chip. Yeah.
Oh, totally. Totally. And yeah, SemiAnalysis I believe made the point of like hey most people their first chip it's like almost throw away. It's like proof that they can do it and they learn a lot through it but where they want to compete is on chip two or three or four. Yeah.
So it's very interesting that they came out swinging with chip one. Let me actually go back a couple slides though because you know okay here's a slide screenshot from SemiAnalysis. This is they've got Jalapeno which is the purple line here that's out on the Pareto has set a new Pareto frontier. It's with DeepSeek R1 671 billion parameters. So not the biggest model but a fairly decent sized model and you can see that you know at given ISO interactivity it's much higher throughput and it can also reach much further out in interactivity than a GPU. Now, I think to be fair, some of the pushback was the Jalapeno will have HBM4 and a lot of these chips that it was comparing to the Blackwell is on HBM3E, I believe. And so this, you know, MI355 as well. And so the fair comparison is going to be Jalapeno against Vera Rubin and Jalapeno against Helios. But I just wanted to make the point that this is their first chip and it's obviously instantly competitive and so I think that's also what's impressive. Not only was it fast but it is competitive as well. So that was that kind of medium model when they showed like look we can do models for you know from left to right on the spectrum. This was that point in the middle. They also showed when they run a small model GPT-OSS 120 billion parameters probably what really stood out and has very interesting implications to sit and think about for a while is hey look Jalapeno can reach out to above 1,000 tokens per second per user and that's the SRAM territory and so you know GPUs just can't get there and so then that's where GPUs said okay fine we will you have Dynamo will split the workload to prefill and decode and we'll run the decode on these SRAM chips Groq or it could be Cerebras and now all of a sudden Jalapeno is coming in and saying wait a minute which this is the same argument by the way that AI ASIC startups are making which is like hey actually what if you can have one chip that can do both high throughput at a lower decode speed but also reach those you know thousands of tokens per second.
This is compared to Blackwell of course. So if you put the Rubin system with the LPX, you know, basically the Groq chip, you can get this Pareto curve too. So it's not like there isn't hardware out there that can't do this. It exists in different forms. But it's nice to see that this chip in a small model can do over a thousand tokens per second per user.
Yes. Which again is interesting, you know, can it will be interesting to see how far Vera Rubin can get by itself on this because then you have to ask yourself like oh okay well you can get it with Groq that's nice but remember how many racks of Groq it takes you know of LPUs. So then you have to ask is it is the cost worth it and probably the power consumption worth it to have a NVL72 rack and nine Groq racks or whatever it is or could you just have a rack or two of Jalapenos?
So Jalapeno's benefit really seems to be the token throughput per watt. It is actually very energy efficient in generating this kind of performance. That's one of the key takeaways especially if you normalize it per watt. It's a 700 watt TDP chip. Which compared to the Blackwell, you know, the Blackwell GB200 is over 1,200 W. So, that's a big difference in the power consumption. It's a very energy efficient chip as well.
Yes, totally. Which matters. And they another that was another point why they liked SemiAnalysis InferenceX was it is power normalized. So it's not like, oh, just use more power and you get more tokens or higher interactivity. Well, that's kind of cheating. So, you know, power normalized. Okay, one other thing. So, back to this slide that had their roadmap. At the bottom of the slide, they called out, you know, and hey, thanks to our incredible partners, especially Broadcom and Celestica. So, I thought it'd be worth talking through really quick. Who are some companies that are helping and participating? And if you're thinking about like what happens if this goes really well for OpenAI and as literally one of the two biggest consumers of compute, if their own chips seem to fit their needs very well and they eventually spend more and more of their gigawatts or megawatts on their own chips, who might be some of the other supply chain vendors who will benefit? Obviously Broadcom and then there's some other ones as well. Do you want to talk through these quick?
Yeah. So, the CPU is an x86 CPU. People believe it's a Turin class CPU, which is a good CPU. I'm sure the Venice will do better if they want to do AMD. There are Intel...
