I'm like, Jensen, why are you signing on a memory wafer? We need that stuff. There's been a lot of noise about how Micron's out, it's not going to be on the Vera Rubin, and I was just like, oh look, Jensen must not have gotten the memo. He must not have gotten the memo that Micron is not designed in.
Hello listeners and welcome to another SemiAccurate podcast. I'm Austin Lyons with Chipstrat, and with me is Vik Sekar from Vik's newsletters. So, Vik, what's going on? What's up this week?
A lot of things, right? This was the GTC conference week and OFC, which is optical fiber conference. So, after Nvidia's announcements and all the stuff that happened in the optics world, I don't know. I'm kind of like, seriously, I'm filled up, my mind is filled up with all this news, and I've come to the point where this week I've actually made a bunch of tweets that made no sense at all. Like I'm just tweeting stuff that makes no sense. Like I don't read through stuff fully, and I just say this, and then I'm like, what am I doing, man? Like calm down. I think my brain is fizzling out.
Totally. That's right. Let's see. So much to say. Yeah. So much to stay on top of. And yeah, I was at GTC and then I was trying to keep up with what was going on in OFC, and then I was just like, never mind. I can't do both, you know. I'll just have to read it next week. But today I'm not reading any of it. It's Friday and I'm just going to go outside because it's nice here.
Yeah, that's nice. Yeah. Then I mean, you just got back from California, right?
Yeah, yeah. Which it was hot there. So, when I left Iowa, it was like 7 degrees Fahrenheit, -17 wind chill. I had to park on the outside of a parking ramp at the top. It was like freezing. And then when I got to California, it was like 88 degrees when I landed. They're having like a heatwave. And then even when I got home, it was like 78 here and my kids were sunburned from playing outside. I heard Austin, I was like, oh, it's spring.
Actually, my day-to-day temperature in Bangalore is around 80 to 85 degrees, and if you're at, you know, I think you're easily at -15. I don't know.
Yeah, yeah. Yeah, we have 100 degrees just between this podcast between us. That's hilarious. That's so good.
Yeah. But okay, so yeah, where should we start? Let's start not Nvidia. What do you think?
Yeah, so I have to tell you this first. Like I've heard so many people right now saying, oh, I spent so many tokens in like agents and this and that. And then Nvidia comes and says like the guy Jensen in the All-In podcast, I think he says, if you're a $500,000 engineer working for a company, I expect you to spend at least $250,000 in tokens. I'm like, wait. That's not going to go down well. I was like, you kind of imagine going to your boss and saying like, yeah, I want to spend 250k in tokens. I'm like, I thought AI was supposed to make things cheaper. What's going on?
That's hilarious. Yeah, you know, so I got some interesting thoughts on this. So, I actually was in an industry analyst Q&A earlier before Jensen said that podcast thing. And I asked him the question because he had said in the keynote, hey, you know, I think that we'll soon get to a point where a token budget is a differentiator for employees. And so I said to him like, hey, look, I agree because to me, I think that agentic AI is the killer app that we've all been waiting for for generative AI, right? When gen AI first came out, it was just a chatbot. Everyone was like, okay, this is cool, but what's the killer app? I think agentic AI is, and I think like for me specifically, it's coding, but I think it'll be more broad than that. But maybe it will just be coding. Maybe it's just like everyone can code, you know.
So, anyway, so I said to Jensen, I was like, hey, I actually believe you that once you go agentic AI, you'll never go back. So, if you're at a company where they're just like, ah, sorry, we've got strict token budgets, you can't spend much, and then some other company says like, dude, you can go to town, cloud code all you want, like you'll never go back. You'll be like, that's amazing. But to your point, I started thinking about it and I was like, well, wait a minute, if you give people actual token budget, like let's say $10,000 a month. We'll just use round numbers. Well, what if I'm super productive and I'm going to town and I blow through it in a week? Now, am I going to just not get to use cloud code for the next 3 weeks until it resets, right? And so the incentives are maybe a little misaligned. So, what I said to Jensen was, are companies in this situation going to buy tokens for every employee, or are they going to buy token generators? Are they going to buy the hardware itself? Because then you essentially have like unlimited tokens, if you will.
Now, I was also hoping that that would open the door to talking about like on-prem inference, like the Dell ProMax with the GB300 where you can literally like air-cooled, plug it in at your office, and have the data center GPU because there's I think a lot to explore here and a lot more I want to hear from Dell, which is like, hey, what if for a team, instead of saying, yeah, we're going to spend a million dollars for this dev team in tokens per year. What if you're just like, we're going to get these are like $120,000 machines. What if you're like, we're going to get like four of them, and you can all remote in and share them and go to town, and I think it's got like 768 gigs of memory or something. Now, not all of that's HBM, but like hey, run big open-source models, do what you can, you know. Maybe that's actually going to be a lot more cost-effective than people just getting these crazy token budgets where once you've spent it, it's gone. And it's the whole opex versus capex type conversation stuff. Like I just feel like CFOs are going to really want something other than, oh great, now I got to spend an extra 200 grand per person with OpenAI, for example.
Yeah, that's the way I think it's going to go, and the opex capex thing is obviously very important. But you know, even when you deploy on prem, I don't think you're going to have virtually unlimited tokens or anything because, you know, there's a token per second generation thing for a given quality of hardware, and then there are so many engineers, so it's going to be divided up at some point. Just think of it as like EDA licenses, right? You sign up for 20 licenses. And when somebody is using it all up, I have done this personally, I go to somebody and say, hey, can you please get off the license if you're not using it because I want to run something. And then they do, and then that's how it goes. So, this will be on a similar note, I think. Not exactly the same, but I think it'll be like, yeah, you've used up your token budget this week. Can you chill? Can you like do something else and maybe next week get back on it, right?
Sure, sure, sure. Yeah. I see your point. And that would be the trade-off is like, well, this machine can generate a certain number of tokens per second for a certain number of concurrent users 24/7, but now the onus is on the employees to have to kind of manage that and share it and whatever.
Yeah, it's a resource. It's a hardware resource like any other that has to be shared. Right? You know, we used to have these like big computers where I used to do a lot of like electromagnetics work, which is like heavily computational. And yeah, so that was how it was. Like you submit a job to it, and then you hang out or whatever, and then it gives you the results back. Or think of it like LSF clusters. Like you submit a job, you don't see the machine, right? And when the simulation is done, it gives you the results. It'll be like an LSF cluster of tokens, right? So, you're going to be using from this cluster instead of compute, which is kind of what it does anyway, LSF. Load sharing facility, I think it's called. So, it's going to be like that. It's going to be a bunch of servers that every big company at least will deploy. And smaller companies will deploy smaller versions of the same thing.
Yeah, exactly, exactly. But the nice thing is you have a fixed cost. You know exactly what you're going to spend as opposed to just like APIs where everyone can, you kind of don't have to share in some respects when you use APIs, but you could go to town, you could blow through things or blow over and spend too much, you know.
Yeah, and nobody wants to send their tokens anywhere else outside premises. Some of these industries are very, very sensitive to this, and you just can't do it. It's very because even if, how do you know that like your tokens don't go somewhere else in the cluster and somebody else does something because you're using a bunch of CPUs and GPUs in a facility, data center somewhere. Totally. What is a guarantee that your bits did not go to your competitor's bits which are using GPUs from the neighboring rack? Like that kind of security guarantees are coming into newer Nvidia GPUs. They have been talking a lot more about security guarantees. That's coming up. It's going to be a thing. But a lot of companies will not like that, right?
Totally. And you know, Jensen, when I asked him this question, he spoke to it, and he sort of said like, hey, I think that there'll be a hybrid. Like there'll be times when you want the biggest, best frontier models, and it's going to have to run in the cloud, whether it's your own cloud or, you know, someone else's cloud. But there's to your point, there's going to be sensitive data. Also, he said he believes in like a lot of local fine-tuned models where a company will say like, hey, we have our bespoke data, we've trained our model for our particular use cases, let that run on prem and handle any of that stuff, you know? And so I definitely do think we'll get to a world. Of course, then there's lots of questions around like how's all this orchestration going to work and who's going to build it? But hey, now that we have AI, maybe it can get built easily and quickly.
Yeah. Now that you mentioned that like these fine-tuned models, I have to mention Matthew Berman. I don't know if you've seen his YouTube channel. It's pretty amazing. And yeah, he has all these amazing videos and he mentioned this one thing that he's working on. He said, 'Yeah, my OpenClaw agent has all of my information. It has all of my memories, everything I prefer, don't prefer, all of that stuff. And all of that has like the outputs from a frontier model.' So he's like, 'I'm going to take like a lighter model, like an open source model, smaller one, and I'm going to train it just on my inputs and outputs that I have already stored in my OpenClaw environment. Nice. And that's going to save my token cost enormously.' Nice. He said he's going to like look at it and post a future video or something. It's very exciting.
So I decided after watching Matthew Berman, I'm going to take this OpenClaw thing seriously because Jensen was also so emphatic about it at GTC. I'm like, 'Okay, fine. It's time. Okay, let's.' I played with it a little bit earlier on my laptop and then, you know, all these security things. So I just quit it there. But now I deployed it on, I have a little home server that I run where, you know, I keep my own files. I run my own little cloud with all my family photographs and stuff like that. So I deployed it in a docker container within my home server and I put in like a reverse proxy to some kind of like an OpenClaw dot something dot something, you know? So if I go there, I can go to my dashboard now, whatever I want. And before anybody listening says like, 'Hey, why are you reverse proxying it to a URL? That's like the worst thing you can do to agents.' To be fair, this is entirely within my local network. It doesn't go outside. It's not exposed to the internet. If it is, it's through Tailscale. So I have a VPN Tailscale that's going. So I do have security protocols in place. But yeah, I set it up and I bought a bunch of credits for like Claude. And I'm like, 'Okay, fine. Let's do this. Let's set this up.' And I've been playing with it.
That's amazing. You'll have to share what you do with me sometime. I'd like to set up something that's the same. One of the things that I really want to do because my mantra right now is ABC, always be Claude-ing. And so I always just want to check on my jobs, my agents, and then just because usually it's like, okay, they're good on their own, but then after a few minutes they'll just ask for something. And I'm always like, 'Yes, yes, yes.' You know, like even if I tried to give it like disable all things, never ask me anything, just go to town. Like it seems to always still have to ask me. And so I want to just be like from my phone be able to like check, you know, go to that URL, check on the dashboard, or have a terminal or something and just be like, 'Oh, yep, keep going. Keep going.' You know?
Wait, doesn't it, didn't you set it up with like Telegram or whatever? WhatsApp?
Well, there you go. That's a great idea. I think that was like one of the most initial use cases that got everybody hooked on this because you could just like hook it up to WhatsApp. And then you could chat with it on your phone like messaging. Oh, beautiful. And it keeps doing whatever you want on the phone. I think it's only quite easy. Well, it's a solved problem then. Yes.
Yeah. Anyway, I don't know if I'll continue this docker container approach or maybe I'll get, I have an old Mac M1. So maybe I'll set it up on that. I don't know. But yeah, I think it's kind of cool. I'm going to agentify this thing because like I told you earlier, I'm overwhelmed with the news. I can't take it. Like I have so many subscriptions on Substack that I have to read. With the news, I'm kind of done. I'm overwhelmed. So let's go agentic on this one and we'll talk about it at least briefly on the podcast. I'll tell you how it's going.
Sounds good, man. Sounds good. All right. So okay, on top of all the conferences, there was also Micron earnings. Did you pay attention to that at all?
A little bit, I think. I didn't do too much of it because as I said, I mean like memory now, I mean okay, Micron's doing great. That's the takeaway I have. They showed a lot of earnings and their stock is up I don't know some 60% this year. Amazing. And they are supplying HBM4 to Vera Rubin.
Which is something you posted on X. And I was like, 'Wait, what? That's cool.' And Jensen's like signing a wafer and Sanjay Mehrotra's holding the wafer. I'm like, 'Jensen, why are you signing on a memory wafer? We need that stuff. Stop. That's way too expensive to be signing, dude. What are you doing? People need that, Jensen.' That's hilarious. That's good.
