Last week, Qualcomm said agents will run on your phone, your glasses, and your earbuds. But chip companies don't decide where inference runs. App companies do. And the biggest app companies of our time right now, OpenAI and Anthropic, well, they only get paid when it runs in their cloud. So, who will actually make local AI happen? Maybe the company that makes its money on ads and already ships Snapdragon in its glasses, like Meta or Google, but do they have the moxie? Let's get into it.
Hello everyone, I'm Austin Lyons. Welcome to another Semi Doped episode. I write Chipstrat, a semiconductor and AI infrastructure newsletter for investors and tech executives.
Hey, I'm Vik Sekar. I've worked in the semiconductor industry for about 20 years, mostly on engineering. I'm the founder of Semi-Exponent, where I do independent semiconductor research, mostly focused on AI and infrastructure. And I also write Vik's newsletter on Substack, where I just explain how things work and what matters to the markets. Um, so that's what we bring to the newsletter. Check us out on our Substacks, where we write like a whole lot of stuff every week.
Yes, totally. Uh, Vik, I know you wrote some stuff last week, but let's not talk about that. Um, let's talk about our new intro. Tell me about our sick new intro.
Yeah, that was, uh, by the way, uh, all done with, um, Opus 5.5. And you know, uh, we have our podcast brand document, and all I did was like, I gave it a brand document, uh, a PDF, and was like, just make me a motion graphics intro. And our branding document has basically the theme it's based off of, like, uh, the field and the single point of singularity. And our designer put in these like phrases in there. So it just used that and built the whole thing out. And I was like, 'Ah, this is insane.' And I told it like a few other things, like make it 6 seconds, make it like 30 seconds. And I tried different things and I was like, 'Wait, wait, the logo stays for too long. Make the logo stay for lesser time, make the animation slower.' It's insane. That's what it came up with.
And then, uh, I used, uh, you know, Suno AI, which is like a music generation thing. Uh, and if you get the paid subscription, uh, to the tool, you can actually generate commercial, uh, music. You can use that in podcasts and, you know, other things to make money off of. Uh, not a sponsor, but they could very well be, I suppose. Uh, really, like, it's just so, so cool. Like, I kept trying different things. I said, 'Hey, make some like jazz fusion.' I'm like, 'No, this sounds like elevator music.' And I'm like, 'I don't know. Like, I want something exciting.' And I told it like, build it up and like give it something in the beginning that's repetitive, but like I want a climax finish and then goes to slower things where, you know, we can talk over it and things like that. I don't know, it all works just fine. And then our editor takes it and, you know, runs with it with the magic there. So all this is really nice. AI pro AI.
Dude, so, so good. So amazing. I love that the Suno, like, you still had to have taste. Like, you were able to use Suno to generate the music, which is incredible, um, and really speed things up as opposed to, you know, what would you have to do previously? Contact someone who knows how to do this or figure out how to make music yourself, you know, with your computer. Um, but you had the taste to say, 'Oh, you know, here's what I'm listening for. I need it to build. I want there to be a hook. I need it to be catchy.'
Yeah, so yeah, clearly I am a musician and like I have recorded music and I kind of know what to say to it. So that little bit of domain expertise actually helps, uh, to tell it what to do. Otherwise, it's very impressive. I think most people can do something with it.
Awesome. I love it. Okay, so let's talk about what else is new in this episode. Um, so we have a presenting sponsor.
Yes. Awesome. We have Crusoe. This is like our first episode with a presenting sponsor. We're very excited.
Very excited. Really, uh, stoked to partner with them. And hang tight and you'll hear our first ad read and you'll get to learn more about them. But, uh, it's going to be fun and you're going to learn a lot about them. There's so much to learn about Crusoe and, um, we're going to do fun ad reads every single week and they'll be different. So stay tuned, listen to them and check them out. And, you know, they sat down with us and we spent a lot of time talking about all their value props and their interesting technology from top to bottom. So you'll get to learn about that too over the next couple months. Um, but with that, let's get into it. So, I was at Snapdragon Summit last week in Maui in Hawaii. It was awesome. There was a tropical storm incoming, which I didn't even know, but some other analysts are like, 'Hey, like my mom or my wife called and said they saw on the news that there was a tropical storm that was going to hit Hawaii.' I was like, 'Oh man, really? I'm just too busy enjoying the beach to even notice.' But we got out before it was a problem. Um, but yeah, let's talk about Qualcomm. So I think that, you know, of course everyone may have seen, uh, I did an interview with Durgham which was really interesting and talked about high bandwidth compute and that was mostly because Durgham was there and I was super excited to talk about it. Um, but the name of the game for Snapdragon Summit, my takeaway was Qualcomm really talking about a vision for agents running anywhere and the fact that, um, their Snapdragon, so they have different sort of flavors of it that can run in watches, in augmented reality glasses, uh, on your phone, on your car, in your laptop. And so naturally Qualcomm as a hardware vendor, their vision is agents that can run anywhere.
Some time ago, if you had told me like agents were going to run on the edge, I think the perception I would have had is LLMs are too big. Like, how are you going to run a frontier like LLM that is intelligent on your phone? And clearly we will never have, I don't know, like a terabyte of memory on the phone to run a trillion parameter model. And clearly scaling laws still hold, which means the bigger you make a model in terms of its parameters, the smarter it seems to get, and we still haven't seen the end of that. So I would have laughed at the idea of, 'What? No way we can have that running on our phones,' right? But in talking with you just before the show and also, uh, reading a little bit about this stuff and the event that Qualcomm just had, it seems like there is a different way to think about it. Um, agents could be a portal into our world of AI, and we'll clarify what that means. It's very interesting.
Yeah. You know, you make a really interesting point, which, like, go back a couple years. Uh, remember like Intel, uh, Pat Gelsinger, he talked a lot about AI PCs and there was this big push. Yeah, we're going to have these AI PCs. They've got NPUs and everyone's like, 'Oh, how many TOPS does your NPU have? Mine has 35. Mine has 45. Mine has 55.' And it was all about like, um, the idea was running AI locally, but at the time it was LLMs. And first of all, there were barely any open-source LLMs when that started. Um, and then, you know, when they came out, it's a 70 billion parameter and it barely fits on your laptop and your NPU really can't chug along and do too much. And so everyone's like, 'Dude, what are we doing here? Like, this is going to be slow. What's the use case?' Um, and ultimately AI PCs were kind of a dud. Everyone just ran their AI in the cloud. Um, and I think now we are getting closer to a world where there will be useful AI, and we can get into this, that could run at the edge and it doesn't have to be an LLM. It doesn't have to be a frontier trillion parameter LLM. Um, but we can get into that. I will say, uh, you know, we're still in this transitory world where, and we'll get into this, where you could see a world that will run AI at the edge and yet there's still talk about like, oh yeah, we can run all of this, uh, agentic AI, for example, Claude and Codex and, you know, OpenCloud, Perplexity, uh, you know, and a lot of that is still running in the cloud. So there's still a little bit of, you know, we're envisioning a world to come, but we're still talking about apps that actually run in the cloud. But, um, you know, obviously I think Meta comes to mind, uh, as a truly agentic AI and then where that could go is pretty interesting. But yeah, I'll hand it over to you.
The app thing is very interesting. So recently I have gotten, uh, very much, let's say, AI-built, to say not that I wasn't already, but the reason is given the Substack and now I'm, you know, running, I've been talking to prospective clients about Semi-Exponent and institutional research for financial firms and things like that. And then we have the podcast, right? And then we have so many like podcast briefings and recordings and back and forth of documents. It's a large universe that we deal with, really, you know. And so I have been thinking this needs to be more agentic AI for me. I really need the help. Like, uh, I could go off and hire people and all that, but I was like, okay, let me see what parts of this. It's not like human taste isn't important. It is. And we do have people who work with us on the podcast. We do need people. I'm not saying we're going to replace everything, but I'm like, just let's, what can AI take care of for me? Because there's like a minimum requirement for some things, the MVP in my life. What can it do? So the app itself, the phone has become a very big part of my day. And the reason is I have ChatGPT. I have Claude too, and this happens both places really, but in ChatGPT I have now scheduled tasks. So what it does is like every morning it reads all the emails and because I'm on a different time zone, it gives me a briefing as soon as I wake up in the morning. So when I'm making coffee, uh, literally ChatGPT reads out to me like whatever has come to me or I read it out or whatever, it doesn't matter, but it's all in one place. It's become an integral part of my morning. And so yes, AI doesn't run on my device, but it's become a portal for me to handle tasks during the day. And this is kind of what Qualcomm has been saying, uh, except that their view is that a lot more inference runs on the device. Maybe we are not just there yet, but I can see where we are headed because it's become so important for me that every hour, right? I don't want to miss emails and all that because some emails require me to respond immediately. So I have another scheduled task that, uh, you know, reads my emails and tells me every hour. So a notification comes up on my phone: 'Hey, somebody's asking you to do this.' Here's another great example. I messed up and I always get time zones wrong and anybody who's like worked with me on this, I always get rescheduling invites in the calendars and I always get, I can't do math in my head, I guess. Um, so it sent me a notification. It says, 'Hey, your email says that you're going to meet with this person at this time, but your scheduled, uh, you know, meeting is one hour late. Are you sure this is what you want to do, like, or do you want to move it?' I'm like, 'Oh my god.' Like, I told ChatGPT like, 'Remember to put that on my to-do list and put it on my task calendar because it's hooked up to Google, right?' And so it made it, uh, you know, it's put on my to-do list and I, you know, I just have to go and send, you know, reschedule it. I could ask it to do that too, but I'm just not interfacing with other people yet because I don't want it to like keep sending stuff to other people. Could be done, uh, automatically. But you see where this is going, right? Like I know inference doesn't run on the device yet, but on the cloud it's still useful. Edge devices are being useful to me already.
Yeah. Yeah. I love that use case. Good job, ChatGPT, by the way, for keeping Vik straight and figuring it out.
I wouldn't even have showed up to this podcast on time. I would have been one hour late.
This episode of Semi Doped is presented by Crusoe. Crusoe builds AI infrastructure from the ground up: power, data centers, and the cloud on top. Crusoe Intelligence Foundry is their managed AI platform, and a nice feature in there is serverless inference. Now if you're going, 'What in Jensen's name is that?' it means that there are no GPUs for you to provision and no clusters to manage. You simply pay per token. If you're running open models like GLM, DeepSeek, Kimi, or Qwen, Crusoe's managed inference is fast. Nearly 10x faster time to first token and 5x the throughput of vLLM. Their memory-aware caching layer routes requests to maximize cache hits. Since it works with the OpenAI SDK, switching over to Crusoe's API is quick and easy to get it running. All you have to do is pick a model, grab an API key, and start sending requests. If you sign up with the link in the show notes, you get $5 in free credits to try Crusoe Intelligence Foundry. Think of it, five whole bucks, so you can just burn some tokens just for fun. And some of these open models are quite inexpensive, so you'll actually have a ton of fun with it. Check out Crusoe and see their serverless inference on their managed AI platform. You really got to find out this for yourself. There are a whole lot more features in this that I want to try out for myself, but serverless inference, I found, is the best place to get started. It's a great way to support the show and try out Crusoe's Intelligence Foundry. The link is in the show notes. And now back to the show.
Previously, when everything was just running ChatGPT, chatting with ChatGPT or chatting with Codex, still everyone is still focused on like, oh, it's got to be Fable, it's got to be the frontier model. Um, what I think has been really interesting lately is Meta and Instinct, like we've talked about Grokbot and these agents where you realize, one, I could see this going to actual like normal consumer adoption because it's so simple to use. You know, there's billions of people that have, um, Meta's family of apps installed and it's very simple for them to get up and running with Meta. And, um, by the way, when they sign up for Meta, they get a dedicated, uh, what is it, like two CPUs, a dedicated cloud VM. It's like two CPUs, 8 gigabytes of RAM. Um, I think I even saw maybe like 100 gigabytes of storage, or, uh, maybe that was Tom's Hardware speculating or something, but clearly like quite a beefy computer for every single person. And so on the one hand, this is like crazy powerful that all these people can spin up agents that can actually go do useful stuff. And by the way, um, my use case for Meta, because I was like, 'Okay, I want to try this out. What am I going to do?' And I've got all this like electronics and old recording equipment and stuff like this sitting just next to me, uh, in my office, and it kills my wife because she's kind of a minimalist and she's like, 'This junk is sitting here. Do something about it.' I'm like, literally... She's like, 'It has so much value. It's just sitting there, just sell it on Facebook Marketplace, you know?' I'm like, 'I just don't have time. I just don't have time to take pictures, write out all the listings, look up how much everything should cost.' And then I was like, 'Wait a minute. This is perfect for Meta. I'm just going to take pictures of everything and tell it, bro, just post this on Facebook Marketplace. I don't care what the price is. You pick a good price. You make a nice listing.' And I just gave it a bunch of pictures, and it actually did an awesome job. It like looked up what everything was called and wrote like really great descriptions, figured out the prices, and posted it and everything. And so I'm like, 'Dude, that is like truly useful work for me.' And I even may make some money out of it. And but even more importantly, of course, I'm going to make my wife happy. Uh, so, you know, happy wife, happy life. Um, but, you know, here's the interesting thing. The more it... so I was like, oh man, I'm using those, uh, you know, all this stuff that was provisioned for me in the cloud, but then I stopped using it and I'm like, is it just sitting there? Are they sharing with other people? But I was thinking like, man, if I get this just cranking all the time for me, Meta is so handy, so convenient because it's on my phone. And so I was actually checking it when I was in Hawaii. You know, people were responding like, 'Hey, can I buy this?' So I was telling Meta like, tell everyone like, 'No, I'll get back to you when I'm back from vacation,' but don't tell them I'm gone. I don't want them to rob my house, you know. So just tell them like, 'I'll get back to you this weekend,' you know? And so Meta is doing that for me. And I just thought like again, hey, some of that stuff, here's my phone right here and it's pretty simple. Like they're paying a lot of money in the cloud to reserve this infrastructure and use it for me. But like, couldn't some of this very simple stuff like sending someone a message, like couldn't that actually happen on my phone? So it just got me thinking like, hey, maybe app companies who end up spending a ton of money in the cloud, like maybe Meta, maybe they will be the ones who are incentivized to actually push a model down to my phone and try to orchestrate and have some of it run locally when it can.
Yeah, that's very interesting, the fact that, you know, you mentioned that everybody gets a virtual machine in some cloud instance somewhere. Actually, I sat down, uh, the Substack post this week was all about estimating how many CPUs are going to be deployed assuming, I don't know, a billion VMs being in the cloud, like how many CPUs do you need? Because I think there's like a lot of excitement about CPU requirements. So I calculated all this out. It's very interesting actually because people can share CPUs, you know, everybody doesn't need to get a dedicated VM. Like, you know, when it's not posting your, uh, Facebook Marketplace ad, like it could go and help Vik like get his calendar right. Like the same CPUs, why should it just chill out? So this kind of stuff is there. So the second thing is like what we don't have yet, I think, is, uh, we've got 8 GB of RAM on this thing, right? And we've got, uh, some computing power, like two vCPUs worth of computing power right here, got some storage. I mean, I don't know how much is in this thing, maybe not 100 GB. Like I need space for apps, or do I? We'll get to that soon. Do I really need apps anymore is a big question. So yeah, like although there's like all the CPU deployments on the cloud and everybody's excited about the prospect of CPU, I'm like, look, my phone is already my portal to the AI world. It's happening right now. It keeps buzzing about all the scheduled tasks I have.
You're putting up stuff for sale. Everybody's happy. Things are getting done. You're making money.
But then what's my CPU doing here? Nothing. So I'm like, why don't we now start using this as our edge computing device where your two VMs and 8GB of RAM live, so that, you know, you can just talk to this device and it has everything. It knows everything about you. It knows where you are. It knows how much you walked in a day. It could even listen to you. It knows who you're talking to. Uh, all your devices connect to this, like your watch can connect to it. Your, you know, your AirPods can connect to it. Everything is phone-centric. So we have to reimagine the AI on the edge world in this framing.
Yes. Yes. I totally agree. And it's all about all that context too. And that's kind of the point you're making. It's like you've got your watch, it's got your health data, your steps data. Maybe for me it logs my Strava, whatever. Um, you've got your microphones, you've got your phone, which has not only of course your calendar and whatever, but it's got like the pictures that you've taken. So there's all this just local context on your watch and your glasses and your earbuds and your phone. And so again, um, the idea, let's say it's just Claude and OpenAI, the idea that you would take everything local, send it up to the cloud, have it compute in the cloud, send it back down feels a little crazy. And of course, in their world, and remember, uh, OpenAI and Claude, uh, OpenAI and Anthropic, these are like two of the fastest growing companies ever. Great companies, uh, amazing app companies, whether they meant to become that or not in the first place because they're both kind of research companies. Um, their business model is we make money when we do inference in the cloud. And yet we are saying actually there's a world where we would really like to just run this stuff locally because all of our context is here. You could have, of course, there's privacy if you can just filter through things and take only what you need here and don't send up all my photos to the cloud. Don't send all this information to the cloud just so that you can compute it there and send it back down. Like why not do it here? And what's interesting again is Meta stands out as a company who their business model is ads. They just want to help me and have the opportunity to show me ads and just be part of my life essentially. Now whether you like ads or not is a whole another conversation. I think it can be useful. Um, but there's a disincentive of course for the model labs to want to come help me at the edge. But Meta's like, 'Bro, I want to help you at the edge. You've already got our apps installed. Like, let me help you.' And so Meta has already made open models. Meta has, um, I think they released Meta Llama, which is in this, uh, direction of running things locally, orchestrating, running on the edge. So it's very natural actually for Meta to say, 'Hey, I've made Meta. It's super helpful for you. It's really expensive. You've got this, um, cloud or this little VM here with these CPUs and the 8 gigs of memory, which is a very great point. It's just sitting idle. We can give you more Meta if we can just run some of it locally.' Like imagine I hit my token limit and they're like, 'Yeah, didn't that suck? We can make your tokens go farther. Let us run some. Will you give us permission to run it on your phone when possible?' And I would say, 'Totally.' And this doesn't even necessarily mean that it has to be the stuff like you said that requires that sort of system to deep thinking, deep research. You need a frontier model. There's lots of stuff that can be done at the edge that aligns with just helping my little agent making some quick decisions and helping me with my life, looking through my context or whatever before deciding what needs to go to the cloud.
Yeah. So that's the whole point because look at the app, like you mentioned the Strava thing, um, and you run a lot and, uh, you keep track of your steps. You even posted on X and all that and you were like, when you were in Hawaii, you were like, 'Look at this, I have 20,000 steps today,' you know, all that stuff. Clearly it all runs on an app, but do you really need that app? You know, there's a fitness app sitting on your home screen occupying like this space and you have to find where it is or you have to type fitness and then you have to click on the app and it shows you the steps. And why do you need that? All you have to do is ask it how many steps and it has a screen. That's all it needs. It's just a screen. Everything that happens on a phone can just happen. You don't need an app for it. Like, and this is a point that Ben Thompson recently made on Stratechery, uh, which we are both big fans of that newsletter. He's like the OG newsletter guy. I think at some point everybody who started a newsletter wanted to be like Ben. So exactly. So he made the point that there is no need for apps anymore because everything can be built and destroyed on demand. It's very simple to do. Anybody can like build an app for their own use cases and change it out if they don't want to. Some things may not even need an app like this fitness example, right? So where are we going in this world of like devices? Like what do devices need to have? Do we need to think of them as computers or what is its function in the world of AI are really deep questions that we have to think about because that defines what edge AI is. Another point that Ben made, I want to mention this and I'll ask you what you think because it's a very important point. He's like, look, now there is no reason for anybody to code. There's like literally no purpose. It is a syntax that nobody understands with this like braces and like colons and like indenting or whatever. We don't need to deal with that mess, okay? Like you just ask it, it writes the code, it gives you, you don't worry what happens in the background. Now we have news and we have these CPUs and all that. These agents can even control your computer, like ChatGPT has computer use. So now agents control your computer too, like so you don't need to even use your computer. The computer is meant for your AI to use, not for you, right? So, you see where I'm going with this? Like...
