I'm like, Jensen, why are you signing on a memory wafer? We need that stuff. There's been a lot of noise about how Micron's out, it's not going to be on the Vera Rubin, and I was just like, oh look, Jensen must not have gotten the memo. He must not have gotten the memo that Micron is not designed in.
Hello listeners and welcome to another SemiAccurate podcast. I'm Austin Lines with Chip Strat, and with me is Vik Shaker from Vik's newsletters. So, Vik, what's going on? What's up this week?
A lot of things, right? This was the GTC conference week and OFC, which is optical fiber conference. So, after Nvidia's announcements and all the stuff that happened in the optics world, I don't know. I'm kind of like, seriously, I'm filled up, my mind is filled up with all this news, and I've come to the point where this week I've actually made a bunch of tweets that made no sense at all. Like I'm just tweeting stuff that makes no sense. Like I don't read through stuff fully, and I just say this, and then I'm like, what am I doing, man? Like calm down. I think my brain is fizzling out.
Totally. That's right. Let's see. So much to say. Yeah. So much to stay on top of. And yeah, I was at GTC and then I was trying to keep up with what was going on in OFC, and then I was just like, never mind. I can't do both, you know. I'll just have to read it next week. But today I'm not reading any of it. It's Friday and I'm just going to go outside because it's nice here.
Yeah, that's nice. Yeah. Then I mean, you just got back from California, right? Yeah, yeah. Which it was hot there. So, when I left Iowa, it was like 7 degrees Fahrenheit, -17 wind chill. I had to park on the outside of a parking ramp at the top. It was like freezing. And then when I got to California, it was like 88 degrees when I landed. They're having like a heatwave. And then even when I got home, it was like 78 here and my kids were sunburned from playing outside. I heard Austin, I was like, oh, it's spring. Actually, my day-to-day temperature in Bangalore is around 80 to 85 degrees, and if you're at, you know, I think you're easily at -15. I don't know. Yeah, yeah. Yeah, we have 100 degrees just between this podcast between us. That's hilarious. That's so good. Yes, I'm not used to it. Yeah. But okay, so yeah, where should we start? Let's start not Nvidia. What do you think?
Yeah, so I have to tell you this first. Like I've heard so many people right now saying, oh, I spent so many tokens in like agents and this and that. And then Nvidia comes and says like the guy Jensen in the Allen podcast, I think he says, if you're a $500,000 engineer working for a company, I expect you to spend at least $250,000 in tokens. I'm like, wait. That's not going to go down well. I was like, you kind of imagine going to your boss and says like, yeah, I want to spend 250k in tokens. I'm like, I thought AI was supposed to make things cheaper. What's going on?
That's hilarious. Yeah, you know, so I got some interesting thoughts on this. So, I actually was in an industry analyst Q&A earlier before Jensen said that the podcast thing. And I asked him the question because he had said in the keynote, hey, you know, I think that we'll soon get to a point where a token budget is a differentiator for employees. And so I said to him like, hey, look, I agree because to me, I think that agentic AI is the killer app that we've all been waiting for for generative AI, right? When gen AI first came out, it was just a chatbot. Everyone was like, okay, this is cool, but what's the killer app? I think agentic AI is, and I think like for me specifically, it's coding, but I think it'll be more broad than that. But maybe it will just be coding. Maybe it's just like everyone can code, you know.
So, anyway, so I said to Jensen, I was like, hey, I actually believe you that once you go agentic AI, you'll never go back. So, if you're at a company where they're just like, ah, sorry, we've got strict token budgets, you can't spend much, and then some other company says like, dude, you can go to town, cloud code all you want, like you'll never go back. You'll be like, that's amazing. But to your point, I started thinking about it and I was like, well, wait a minute, if you give people actual token budget, like let's say $10,000 a month. We'll just use round numbers. Well, what if I'm super productive and I'm going to town and I blow through it in a week? Now, am I going to just not get to use cloud code for the next 3 weeks until it resets, right? And so the incentives are maybe a little misaligned. So, what I said to Jensen was, are companies in this situation going to buy tokens for every employee, or are they going to buy token generators? Are they going to buy the hardware itself? Because then you essentially have like unlimited tokens, if you will.
Now, I was also hoping that that would open the door to talking about like on-prem inference, like the Dell ProMax with the GB300 where you can literally like air-cooled, plug it in at your office, and have the data center GPU because there's I think a lot to explore here and a lot more I want to hear from Dell, which is like, hey, what if for a team, instead of saying, yeah, we're going to spend a million dollars for this dev team in tokens per year. What if you're just like, we're going to get these are like $120,000 machines. What if you're like, we're going to get like four of them, and you can all remote in and share them and go to town, and I think it's got like 768 gigs of memory or something. Now, not all of that's HBM, but like hey, run big open-source models, do what you can, you know. Maybe that's actually going to be a lot more cost-effective than people just getting these crazy token budgets where once you've spent it, it's gone. And it's a whole op ex versus cap ex type conversation stuff. Like I just feel like CFOs are going to really want something other than, oh great, now I got to spend an extra 200 grand per person with OpenAI, for example.
Yeah, that's the way I think it's going to go, and the op ex cap ex thing is obviously very important. But you know, even when you deploy on prem, I don't think you're going to have virtually unlimited tokens or anything because, you know, there's a token per second generation thing for a given quality of hardware, and then there are so many engineers, so it's going to be divided up at some point. Just think of it as like EDA licenses, right? You sign up for 20 licenses. And when somebody is using it all up, I have done this personally, I go to somebody and say, hey, can you please get off the license if you're not using it because I want to run something. And then they do, and then that's how it goes. So, this will be on a similar note, I think. Not exactly the same, but I think it'll be like, yeah, you've used up your token budget this week. Can you chill? Can you like do something else and maybe next week get back on it, right?
Sure, sure, sure. Yeah. I see your point. And that would be the trade-off is like, well, this machine can generate a certain number of tokens per second for a certain number of concurrent users 24/7, but now the onus is on the employees to have to kind of manage that and share it and whatever. Yeah, it's a resource. It's a hardware resource like any other that has to be shared. Right? You know, we used to have these like big computers where I used to do a lot of like electromagnetics work, which is like heavily computational. And yeah, so that was how it was. Like you submit a job to it, and then you hang out or whatever, and then it gives you the results back. Or think of it like LSF clusters. Like you submit a job, you don't see the machine, right? And when the simulation is done, it gives you the results. It'll be like an LSF cluster of tokens, right? So, you're going to be using from this cluster instead of compute, which is kind of what it does anyway, LSF. Load sharing facility, I think it's called. So, it's going to be like that. It's going to be a bunch of servers that every big company at least will deploy. And smaller companies will deploy smaller versions of the same thing.
Yeah, exactly, exactly. But the nice thing is you have a fixed cost. You know exactly what you're going to spend as opposed to just like APIs where everyone can, they don't have, you kind of don't have to share in some respects when you use APIs, but you could go to town, you could blow through things or blow over and spend too much, you know. Yeah, and nobody wants to send their tokens anywhere else outside premises. Some of these industries are very, very sensitive to this, and you just can't do it. It's very because even if, how do you know that like your tokens don't like go somewhere else in the cluster and somebody else does something because you're using a bunch of CPUs and GPUs in a facility, data center somewhere. Totally. What is a guarantee that your bits did not go to your competitor's bits which are using GPUs from the neighboring rack? Like that kind of security guarantees are coming into newer Nvidia GPUs. They have been talking a lot more about security guarantees. That's coming up. It's going to be a thing. But a lot of companies will not like that, right?
Totally. And you know, Jensen, when I asked him this question, he spoke to it, and he sort of said like, hey, I think that there'll be a hybrid. Like there'll be times when you want the biggest, best frontier models, and it's going to have to run in the cloud, whether it's your own cloud or, you know, someone else's cloud. But there's to your point, there's going to be sensitive data. Also, he said he believes in like a lot of local fine-tuned models where a company will say like, hey, we have our bespoke data, we've trained our model for our particular use cases, let that run on prem and handle and then any of that stuff, you know? And so I definitely do think we'll get to a world. Of course, then there's lots of questions around like how's all this orchestration going to work and who's going to build it? But hey, now that we have AI, maybe it can get built easily and quickly.
Yeah, now that you mentioned that like these fine-tuned models, I have to mention Matthew Berman. I don't know if you've seen his YouTube channel. It's pretty amazing. Yeah. And yeah, he has all these amazing videos and he mentioned this one thing that he's working on. He said, "Yeah, I had my open claw agent has all of my information. It has all of my memories, everything I prefer, don't prefer, all of that stuff. And all of that has like the outputs from a frontier model. So he's like, I'm going to take like a lighter model, like an open source model, smaller one, and I'm going to train it just on my inputs and outputs that I have already stored in my open claw environment. Nice. And that's going to save my token cost enormously." Nice. He said he's going to like look at it and like post a future video or something. It's very exciting. So I decided after watching Matthew Berman, I'm going to like take this open claw thing seriously because Jensen was also like so emphatic about it at GTC. I'm like, "Okay, fine. It's time. Okay, let's..." I played with it a little bit earlier on my laptop and then, you know, all these security things. So I didn't, I just quit it there. But now I deployed it on... I have a little home server that I run where, you know, I keep my own files. I run my own little cloud with all my family photographs and stuff like that. So I deployed it in a docker container within my home server and I put in like a reverse proxy to some kind of like an open claw dot something dot something, you know? So if I go there, I can go to my dashboard now, whatever I want. And before anybody listening to say like, saying like, "Hey, why are you reverse proxying it to an URL? That's like the worst thing you can do to agents." To be fair, this is entirely within my local network. It doesn't go outside. It's not exposed to the internet. If it is, it's through Tailscale. So I have a VPN Tailscale that's going. So I do have security protocols in place. But yeah, I set it up and I bought a bunch of credits for like Claude. And I'm like, "Okay, fine. Let's do this. Let's set this up." And I've been playing with it.
That's amazing. You'll have to share what you do with me sometime. I'd like to set up something that's same. One of the things that I really want to do because I like my mantra right now is ABC, always be Claude-ing. And so like I always just want to check on my jobs, my agents, and then just because usually it's like, okay, they're good on their own, but then after a few minutes they'll just ask for something. And I'm always like, "Yes, yes, yes." You know, like even if I tried to give it like disable all things, never ask me anything, just go to town. Like it seems to always still have to ask me. And so I want to just be like from my phone be able to like check, you know, go to that URL, check on the dashboard, or have a terminal or something and just be like, "Oh, yep, keep going. Keep going." You know? Wait, doesn't it, didn't you set it up with like Telegram or whatever? WhatsApp? Well, there you go. That's a great idea. I think that was like one of the most initial use cases that like got everybody hooked on this because you could just like hook it up to WhatsApp. And then you could chat with it on your phone like messaging. Oh, beautiful. And it's like keeps doing whatever you want on the phone. I think it's only like quite easy. Well, it's solved. It's a solved problem then. Yes.
Yeah. Anyway, I don't know if I'll continue this docker container approach or maybe I'll get... I have an old Mac M1. So maybe I'll like set it up on that. I don't know. But yeah, I think it's kind of cool. I'm going to agentify this thing because like I told you earlier like I'm overwhelmed with the news. I can't take it. Like I have so many subscriptions on Substack that I have to read. With the news, I'm kind of done. I'm overwhelmed. So let's go agentic on this one and we'll talk about it at least briefly on the podcast. I'll tell you how it's going.
Sounds good, man. Sounds good. All right. So okay, on top of all the conferences, there was also Micron earnings. Did you pay attention to that at all?
A little bit, I think. I didn't do too much of it because as I said, I mean like memory now, I mean okay, Micron's doing great. That's the takeaway I have. They showed a lot of earnings and their stock is up I don't know some 60% this year. Amazing. And they are supplying HBM4 to Vera Rubin. Which is something you posted on X. And I was like, "Wait, what? That's cool." And Jensen's like signing a wafer and Sanjay Mehrotra's holding the wafer. I'm like, "Jensen, why are you signing on a memory wafer? We need that stuff. Stop. That's way too expensive to be signing, dude. What are you doing? People need that, Jensen. That's hilarious. That's good. Yeah, yeah, I posted it because, you know, there's been a lot of noise about how Micron's out, their HBM4 is not going to be on the Vera Rubin. And I was just like, "Oh, look. Jensen must not have gotten the memo. He must not have gotten the memo that Micron is not designed in." Somebody paid up for the wrong thing. No, I'm kidding. Yeah, I think maybe there was so much news about this couple of months ago that Nvidia is not going to be choosing Micron and they have 0% share. And a couple of people like on Substack wrote about it and pushed back and said like, "No, we don't believe that's the case. Maybe SK Hynix is going to have the lion's share of it, but it's not zero. I mean, I think it's a smaller percentage." I still don't know what the percentage is, to be fair. And then there was this argument that even if Micron was out of HBM, it's not a bad thing because have you seen DRAM prices recently? You know, it's like easier to make DRAM. Just make DRAM and then sell it. You can like ship so many bits. You don't need to stack them, package them, test them. HBM is a nightmare. I know you have like a lot of margin for selling HBM. It's a high margin, high dollar count product, but just ship DRAM and you're going to be fine. Is what I read from some other Substack. And I was like, "Yeah, I kind of buy that." Yeah. So anyway, turns out they were in. They're in.
Well, and Micron had a press release. I think it was maybe it was part of their earnings or whatever, but they had specifically said that their HBM4 has been designed in. Now everyone's asking, "Okay, but does that mean you're qualified yet?" But no vendor has yet said they're qualified. So I think Micron has publicly said probably everything that they're allowed to say, which is "Look, Jensen signed our thing. You saw it. It's public and we are designed in and we, you know, have the right pin speeds. So, you know, we can't say anymore, I'm guessing. But all signs point to they're going to play this ball here." Yeah, the pin speeds is an interesting thing because glad you mentioned it. The JEDEC spec for HBM4 is I think about 8 GBPS per pin. I hope I got that right. I don't know if it's 8 GBPS. It's okay. So anyway, let's at least look at random numbers. I probably got the units all wrong. But then the news is that they have been pushing the JEDEC speed per pin higher and higher so that Nvidia can like stay ahead of AMD. Whose MI450 I don't know is arguably really competitive or better, I don't know, than Vera Rubin. We have to see. But they wanted to get ahead of AMD and they've been pushing all these HBM vendors to make faster and faster memory. Now one of the things is like it's gone up to 11 GBPS, right? I mean, that's the kind of speeds that like SK Hynix is reporting. Oh yeah, we did 11. In one of the earnings calls, I think not this one, but the last one or one before that, the detail was that the base die, the logic die that sits below all these DRAM chips in a HBM, was actually designed with a memory node. So a memory node is like a one beta node or a one alpha or a one gamma. These are how these memory chips are actually designated. It's not so much as a two nanometer or three nanometer. Those are like logic nodes. So memory is like a different process technology because you've got to make those very tall capacitors that sit on a transistor for the 1T1C cell that goes into the DRAM, right? So the 1T1C cell requires like a unique amount of engineering. So the entire process of making memory is entirely very different on a wafer level than logic chips. This may not be like news to some people listening to this, but if you ask me the same thing two years ago, it would have been news to me because I thought maybe everything is like a chip, right? Whatever. So memory chips are very different. The thing is these memory chips are typically not designed for speed. I mean, they are designed for memory density. They are not designed to be the fastest transistor ever made. That is what logic chips do. That's why you go from a three nanometer transistor to a two nanometer transistor and you get all these speed gains and you know, this is how you know, chips have been scaling forever, right? They get smaller and faster and all that. Memory nodes are not built for that. So the whole argument was Micron HBM used a memory node for its base die and therefore they are struggling to meet the speeds that is being demanded by Nvidia. And so that was like another case against Micron. But I think they repeatedly came out, "No, no, we are fine. The speeds are fine." And then there were like questions about like, yeah, really? How many speeds are fine? One out of 10 are fine or five out of 10 chips are fine? That's you know. That's right. All valid questions. I mean, these are like legitimate good questions that have to be asked. But this is what is behind all of this stuff. Right now we see that yes, it is being designed into Vera Rubin. I still don't know what the speeds are. I don't know whether they're different from SK Hynix. So yeah, you know, take all of this with a grain of salt. Whether you're like in the engineering world or in the chip world in general or whether you're an investor like I think so if people had sold off on the 0% content in Vera Rubin, they would have I don't know lost out 60% share growth now, yeah. So yeah, you make a good point. Look, it's very nuanced. Just because you can hit those speeds, how many chips can hit it? And then of course, you know, how are you yielding? Which is asking the same thing. So essentially like how expensive are your chips going to be can hit that 11 gigabits per second per pin or whatever. So yes, lots of nuance, lots of noise. It'll be nice once these things just ship and then the companies can just tell us. Yes, we, you know. Yes, eventually we'll know. But the whole point of the investor game is to kind of figure this out early. Correct. Yes, that is 100% correct. And I think so another thing that stuck out. So obviously Micron, you know, crushed their earnings. They beat on all fronts. I think they even surprised investors with just how much the revenue was. One of the things that I did notice was there are lots of investments in new capacity that'll be coming online in like 2028. So there's a DRAM fab, I believe, in Taiwan that they have acquired and are kind of pulling that forward. There's a new NAND fab in Singapore and I think they're ramping up fabs in Idaho and New York. And so of course, naturally, I think in memory there's always going to be this tension of like, great, you're winning and you're reinvesting in more capacity, which is good because AI demand will continue to increase, capex will continue to increase, but there's always the caution that people have of like, oh, this feels cyclical in nature and so it feels like you're doing what always happens, which is you're winning and so you're investing in more capacity and eventually we're going to be flooded with more capacity and what's that going to do to prices? But do you have any reactions on that front on the capex investments?
