Last week, Qualcomm said agents will run on your phone, your glasses, and your earbuds. But chip companies don't decide where inference runs. App companies do. And the biggest app companies of our time right now, OpenAI and Anthropic, well, they only get paid when it runs in their cloud. So, who will actually make local AI happen? Maybe the company that makes its money on ads and already ships Snapdragon in its glasses like Meta or Google, but do they have the moxie? Let's get into it.
Hello everyone, I'm Austin Lyons. Welcome to another Semi Doped episode. I write Chipstrap, a semiconductor and AI infrastructure newsletter for investors and tech executives.
Hey, I'm Vik Sekar. I've worked in the semiconductor industry for about 20 years, mostly on engineering. I'm the founder of Semi Exponent, where I do independent semiconductor research, mostly focused on AI and infrastructure, and I also write Vik's newsletter on Substack, where I just explain how things work and what matters to the markets. So that's what we bring to the newsletter. Check us out on our Substacks, where we write a whole lot of stuff every week.
Yes, totally. Vik, I know you wrote some stuff last week, but let's not talk about that. Let's talk about our new intro. Tell me about our sick new intro.
Yeah, that was, by the way, all done with Opus 5.5. And you know, we have our podcast brand document, and all I did was like, I gave it a brand document PDF and was like, just make me a motion graphics intro. And our branding document has basically the theme it's based off of, like the field and the single point of singularity. And our designer put in these like phrases in there. So it just used that and built the whole thing out. And I was like, ah, this is insane. And I told it like a few other things, like make it 6 seconds, make it like 30 seconds. And I tried different things and I was like, wait, wait, the logo stays for too long. Make the logo stay for lesser time, make the animation slower. It's insane. That's what it came up with.
And then I used, you know, Suno AI, which is like a music generation thing. And if you get the paid subscription to the tool, you can actually generate commercial music. You can use that in podcasts and, you know, other things to make money off of. Not a sponsor, but they could very well be, I suppose. Really, like, it's just so cool. Like I kept trying different things. I said, hey, make some like jazz fusion. I'm like, no, this sounds like elevator music. And I'm like, I don't know, like I want something exciting. And I told it like build it up and like give it something in the beginning that's repetitive, but like I want a climax finish and then goes to slower things where, you know, we can talk over it and things like that. I don't know, it all works just fine. And then our editor takes it and, you know, runs with it with the magic there. So all this is really nice, AI pro AI.
Dude, so good, so amazing. I love that with Suno, like you still had to have taste. Like you're able to use Suno to generate the music, which is incredible, and really speed things up as opposed to, you know, what would you have to do previously, contact someone who knows how to do this or figure out how to make music yourself, you know, with your computer. But you had the taste to say, oh, you know, here's what I'm listening for, I need it to build, I want there to be a hook, I need it to be catchy.
Yeah, so yeah, clearly I am a musician and like I have recorded music and I kind of know what to say to it. So that little bit of domain expertise actually helps to tell it what to do. Otherwise, it's very impressive. I think most people can do something with it.
Awesome. I love it. Okay, so let's talk about what else is new in this episode. So we have a presenting sponsor.
Yes. Awesome. We have Crusoe. This is like our first episode with a presenting sponsor. We're very excited.
Very excited, really stoked to partner with them, and hang tight and you'll hear our first ad read and you'll get to learn more about them. But it's going to be fun and you're going to learn a lot about them. There's so much to learn about Crusoe and we're going to do fun ad reads every single week and they'll be different. So stay tuned, listen to them and check them out. And you know, they sat down with us and we spent a lot of time talking about all their value props and their interesting technology from top to bottom. So you'll get to learn about that too over the next couple months. But with that, let's get into it. So, I was at Snapdragon Summit last week in Maui in Hawaii. It was awesome. There was a tropical storm incoming, which I didn't even know, but some other analysts are like, hey, my mom or my wife called and said they saw on the news that there was a tropical storm that was going to hit Hawaii. I was like, oh man, really? I'm just too busy enjoying the beach to even notice. But we got out before it was a problem. But yeah, let's talk about Qualcomm. So I think that, you know, of course everyone may have seen I did an interview with Durge which was really interesting and talked about high bandwidth compute and that was mostly because Durge was there and I was super excited to talk about it. But the name of the game for Snapdragon Summit, my takeaway was Qualcomm really talking about a vision for agents running anywhere and the fact that their Snapdragon, so they have different sort of flavors of it that can run in watches, in augmented reality glasses, on your phone, on your car, in your laptop. And so naturally Qualcomm as a hardware vendor, their vision is agents that can run anywhere. Some time ago if you had told me like agents were going to run on the edge, I think the perception I would have had is LLMs are too big, like how are you going to run a frontier like LLM that is intelligent on your phone? And clearly we will never have, I don't know, like a terabyte of memory on the phone to run a trillion parameter model. And clearly scaling laws still hold, which means the bigger you make a model in terms of its parameters, the smarter it seems to get, and we still haven't seen the end of that. So I would have laughed at the idea of, no way we can have that running on our phones, right? But in talking with you just before the show and also reading a little bit about this stuff and the event that Qualcomm just had, it seems like there is a different way to think about it. Agents could be a portal into our world of AI, and we'll clarify what that means. It's very interesting.
Yeah. You know, you make a really interesting point which like go back a couple years. Remember like Intel, Pat Gelsinger, he talked a lot about AI PCs and there was this big push. Yeah, we're going to have these AI PCs. They've got NPUs and everyone's like, oh, how many TOPS does your NPU have? Mine has 35. Mine has 45. Mine has 55. And it was all about like, the idea was running AI locally, but at the time it was LLMs and first of all there were barely any open-source LLMs when that started. And then you know when they came out it's a 70 billion parameter and it barely fits on your laptop and your NPU really can't chug along and do too much. And so everyone's like dude what are we doing here? Like this is going to be slow. What's the use case? And ultimately AI PCs were kind of a dud. Everyone just ran their AI in the cloud. And I think now we are getting closer to a world where there will be useful AI and we can get into this that could run at the edge and it doesn't have to be an LLM. It doesn't have to be a frontier trillion parameter LLM. But we can get into that. I will say, you know, we're still in this transitory world where, and we'll get into this, where you could see a world that we'll run AI at the edge and yet there's still talk about like, oh yeah, we can run all of this agentic AI for example, Claude and Codex and, you know, OpenCloud, Perplexity, you know, and a lot of that is still running in the cloud. So there's still a little bit of, you know, we're envisioning a world to come but we're still talking about apps that actually run in the cloud. But, you know, obviously I think Muse comes to mind as a truly agentic AI and then where that could go is pretty interesting. But yeah, I'll hand it over to you.
The app thing is very interesting. So recently I have gotten very much, let's say, AI-built, to say not that I wasn't already, but the reason is given the Substack and now I'm, you know, running, I've been talking to prospective clients about Semi Exponent and institutional research for financial firms and things like that, and then we have the podcast, right? And then we have so many like podcast briefings and recordings and back and forth of documents. It's a large universe that we deal with really, you know. And so I have been thinking this needs to be more agentic AI for me. I really need the help. Like I could go off and hire people and all that, but I was like, okay, let me see what parts of this. It's not like human taste isn't important, it is, and we do have people who work with us on the podcast. We do need people. I'm not saying we're going to replace everything, but I'm like, just let's, what can AI take care of for me? Because there's like a minimum requirement for some things, the MVP in my life. What can it do? So the app itself, the phone has become a very big part of my day and the reason is I have ChatGPT. I have Claude too, and this happens both places really, but in ChatGPT I have now scheduled tasks. So what it does is like every morning it reads all the emails and because I'm on a different time zone it gives me a briefing as soon as I wake up in the morning. So when I'm making coffee, literally ChatGPT reads out to me like whatever has come to me or I read it out or whatever, it doesn't matter, but it's all in one place. It's become an integral part of my morning. And so yes, AI doesn't run on my device, but it's become a portal for me to handle tasks during the day. And this is kind of what Qualcomm has been saying, except that their view is that a lot more inference runs on the device. Maybe we are not just there yet, but I can see where we are headed because it's become so important for me that every hour, right? I don't want to miss emails and all that because some emails require me to respond immediately. So I have another scheduled task that, you know, reads my emails and tells me every hour. So a notification comes up on my phone. Hey, somebody's asking you to do this. Here's another great example. I messed up and I always get time zones wrong and anybody who's like worked with me on this, I always get rescheduling invites full in the calendars and I always get, I can't do math in my head I guess. So it sent me a notification. It says, hey, your email says that you're going to meet with this person at this time but your scheduled, you know, meeting is one hour late. Are you sure this is what you want to do, like, or do you want to move it? I'm like, oh my god. Like I told ChatGPT, like, remember to put that on my to-do list and put it on my task calendar because it's hooked up to Google, right? And so it made it, you know, it's put on my to-do list and I, you know, I just have to go and send, you know, reschedule it. I could ask it to do that too, but I'm just not interfacing with other people yet because I don't want it to like keep sending stuff to other people. Could be done automatically. But you see where this is going, right? Like I know inference doesn't run on the device yet, but on the cloud it's still useful. Edge devices are being useful to me already.
Yeah. Yeah. I love that use case. Good job, ChatGPT, by the way, for keeping Vik straight and figuring it out.
I wouldn't even have showed up to this podcast on time. I would have been one hour late.
That's hilarious.
This episode of Semi Doped is presented by Crusoe. Crusoe builds AI infrastructure from the ground up. Power data centers and the cloud on top. Crusoe Intelligence Foundry is their managed AI platform and a nice feature in there is serverless inference. Now if you're going, what in Jensen's name is that? It means that there are no GPUs for you to provision and no clusters to manage. You simply pay per token. If you're running open models like GLM, DeepSeek, Kimi or Qwen, Crusoe's managed inference is fast. Nearly 10x faster time to first token and 5x the throughput of VLM. Their memory alloy caching layer routes requests to maximize cache hits. Since it works with OpenAI SDK, switching over to Crusoe's API is quick and easy to get it running. All you have to do is pick a model, grab an API key, and start sending requests. If you sign up with the link in the show notes, you get $5 in free credits to try Crusoe Intelligence Foundry. Think of it, five whole bucks, so you can just burn some tokens just for fun. And some of these open models are quite inexpensive, so you'll actually have a ton of fun with it. Check out Crusoe and see their serverless inference on their managed AI platform. You really got to find out this for yourself. There are a whole lot of more features in this that I want to try out for myself, but serverless inference, I found, is the best place to get started. It's a great way to support the show and try out Crusoe's Intelligence Foundry. The link is in the show notes. And now back to the show.
Previously, when everything was just running ChatGPT, chatting with ChatGPT or chatting with Codex, still everyone is still focused on like, oh, it's got to be Fable, it's got to be the frontier model. What I think has been really interesting lately is Muse and Instinct, like we've talked about Grokbot and these agents where you realize one, I could see this going to actual like normal consumer adoption because it's so simple to use. You know, you can, there's billions of people that have Meta's family of apps installed and it's very simple for them to get up and running with Muse. And by the way, when they sign up for Muse, they get a dedicated, what is it, like two CPUs, a dedicated cloud VM. It's like two CPUs, 8 gigabytes of RAM. I think I even saw maybe like a 100 gigabytes of storage or maybe that was Tom's Hardware speculating or something, but clearly like quite a beefy computer for every single person. And so on the one hand, this is like crazy powerful that all these people can spin up agents that can actually go do useful stuff. And by the way, my use case for Muse because I was like, okay, I want to try this out. What am I going to do? And I've got all this like electronics and old recording equipment and stuff like this sitting just next to me in my office and it kills my wife because she's kind of a minimalist and she's like, this junk is sitting here. Do something about it. I'm like literally, she's like, it has so much value. It's just sitting there, just sell it on Facebook Marketplace, you know? I'm like, I just don't have time. I just don't have time to take pictures, write out all the listings, look up how much everything should cost. And then I was like, wait a minute. This is perfect for Muse. I'm just going to take pictures of everything and tell it, bro, just post this on Facebook Marketplace. I don't care what the price is. You pick a good price. You make a nice listing. And I just gave it a bunch of pictures, and it actually did an awesome job. It like looked up what everything was called and wrote like really great descriptions, figured out the prices, and posted it and everything. And so I'm like, dude, that is like truly useful work for me. And I even may make some money out of it. But even more importantly, of course, I'm going to make my wife happy. So, you know, happy wife, happy life. But, you know, here's the interesting thing. The more it, so I was like, oh man, I'm using those, you know, all this stuff that was provisioned for me in the cloud, but then I stopped using it and I'm like, is it just sitting there? Are they sharing with other people? But I was thinking like, man, if I get this just cranking all the time for me, Muse is so handy, so convenient because it's on my phone. And so I was actually checking it when I was in Hawaii. You know, people were responding like, hey, can I buy this? So I was telling Muse like, tell everyone like, no, I'll get back to you when I'm back from vacation, but don't tell them I'm gone. I don't want them to rob my house, you know. So just tell them like, I'll get back to you this weekend, you know? And so Muse is doing that for me. And I just thought like again, hey, some of that stuff, it's here's my phone right here and it's pretty simple. Like they're paying a lot of money in the cloud to reserve this infrastructure and use it for me. But like, couldn't some of this very simple stuff like sending someone a message, like couldn't that actually happen on my phone? So it just got me thinking like, hey, maybe app companies who end up spending a ton of money in the cloud, like maybe Meta, maybe they will be the ones who are incentivized to actually push a model down to my phone and try to orchestrate and have some of it run locally when it can.
Yeah, that's very interesting, the fact that, you know, you mentioned that everybody gets a virtual machine in some cloud instance somewhere. Actually, I sat down, the Substack post this week was all about estimating how many CPUs are going to be deployed assuming, I don't know, a billion VMs being in the cloud, like how many CPUs do you need because I think there's like a lot of excitement about CPU requirements. So I calculated all this out. It's very interesting actually because people can share CPUs, you know, everybody doesn't need to get a dedicated VM, like, you know, when it's not posting your Facebook Marketplace ad, like it could go and help Vik like get his calendar right, like the same CPUs, why should it just chill out. So this kind of stuff is there. So the second thing is like what we don't have yet I think is, we've got 8 GB of RAM on this thing right and we've got some computing power like two vCPUs worth of computing power right here, got some storage. I mean I don't know how much is in this thing, maybe not 100 GB, like I need space for apps or do I, we'll get to that soon, do I really need apps anymore is a big question. So yeah, like although there's like all the CPU deployments on the cloud and everybody's excited about the prospect of CPU, I'm like look, my phone is already my portal to the AI world. It's happening right now. It keeps buzzing about all the scheduled tasks I have.
You're putting up stuff for sale. Everybody's happy. Things are getting done. You're making money. But then what's my CPU doing here? Nothing. So I'm like why don't we now start using this as our edge computing device where your two VMs and 8GB of RAM live so that, you know, you can just talk to this device and it has everything. It knows everything about you. It knows where you are. It knows how much you walked in a day. It could even listen to you. It knows who you're talking to. All your devices connect to this, like your watch can connect to it. Your, you know, your AirPods can connect to it. Everything is phone-centric. So we have to reimagine the AI on the edge world in this framing.
Yes. Yes. I totally agree. And it's all about all that context too. And that's kind of the point you're making. It's like you've got your watch, it's got your health data, your steps data. Maybe for me it logs my Strava, whatever. You've got your microphones, you've got your phone, which has not only of course your calendar and whatever, but it's got like the pictures that you've taken. So there's all this just local context on your watch and your glasses and your earbuds and your phone. And so again, the idea, let's say it's just Claude and OpenAI, the idea that you would take everything local, send it up to the cloud, have it compute in the cloud, send it back down feels a little crazy. And of course, in their world, and remember, OpenAI and Claude, OpenAI and Anthropic, these are like two of the fastest growing companies ever. Great companies, amazing app companies, whether they meant to become that or not in the first place because they're both kind of research companies. Their business model is we make money when we do inference in the cloud. And yet we are saying actually there's a world where we would really like to just run this stuff locally because all of our context is here. You could have, of course, there's privacy if you can just filter through things and take only what you need here and don't send up all my photos to the cloud. Don't send all this information to the cloud just so that you can compute it there and send it back down. Like why not do it here? And what's interesting again is Meta stands out as a company who their business model is ads. They just want to help me and have the opportunity to show me ads and just be part of my life essentially. Now whether you like ads or not is a whole another conversation. I think it can be useful. But there's a disincentive of course for the model labs to want to come help me at the edge. But Meta's like, bro, I want to help you at the edge. You've already got our apps installed. Like, let me help you. And so Meta has already made open models. Meta has, I think they released Muse Glimmer, which is in this direction of running things locally, orchestrating, running on the edge. So it's very natural actually for Meta to say, hey, I've made Muse. It's super helpful for you. It's really expensive. You've got this cloud or this little VM here with these CPUs and the 8 gigs of memory, which is a very great point. It's just sitting idle. We can give you more Muse if we can just run some of it locally. Like imagine I hit my token limit and they're like, yeah, didn't that suck? We can make your tokens go farther. Let us run some. Will you give us permission to run it on your phone when possible? And I would say, totally. And this doesn't even necessarily mean that it has to be the stuff like you said that requires that sort of system to deep thinking, deep research. You need a frontier model. There's lots of stuff that can be done at the edge that aligns with just helping my little agent making some quick decisions and helping me with my life, looking through my context or whatever before deciding what needs to go to the cloud.
Yeah. So that's the whole point because look at the app like you mentioned the Strava thing and you run a lot and you keep track of your steps, you even posted on X and all that and you were like, when you were in Hawaii, you were like, look at this, I have 20,000 steps today, you know, all that stuff. Clearly it all runs on an app, but do you really need that app? You know, there's a fitness app sitting on your home screen occupying like this space and you have to find where it is or you have to type fitness and then you have to click on the app and it shows you the steps. And why do you need that? All you have to do is ask it how many steps and it has a screen. That's all it needs. It's just a screen. Everything that happens on a phone can just happen. You don't need an app for it. Like, and this is a point that Ben Thompson recently made on Stratechery, which we are both big fans of that newsletter. He's like the OG newsletter guy. I think at some point everybody who started a newsletter wanted to be like Ben. So exactly. So he made the point that there is no need for apps anymore because everything can be built and destroyed on demand. It's very simple to do. Anybody can like build an app for their own use cases and change it out if they don't want to. Some things may not even need an app like this fitness example, right? So where are we going in this world of like devices? Like what do devices need to have? Do we need to think of them as computers or what is its function in the world of AI are really deep questions that we have to think about because that defines what edge AI is. Another point that Ben made, I want to mention this and I'll ask you what you think because it's a very important point. He's like, look, now there is no reason for anybody to code. There's like literally no purpose. It is a syntax that nobody understands with this like braces and like colons and like indenting or whatever. We don't need to deal with that mess. Okay? Like you just ask it, it writes the code, it gives you, you don't worry what happens in the background. Now we have news and we have these CPUs and all that. These agents can even control your computer like ChatGPT has a computer use. So now agents control your computer too, like so you don't need to even use your computer. The computer is meant for your AI to use, not for you, right? So, you see where I'm going with this? Like,
So, what is the device supposed to do now?
Oh, good. Okay. So, this is a great setup. I have lots of thoughts. So the idea that we don't need to code, I would say baloney to that, although many people don't need to learn. But, and by the way, I got that, Ben lives in Wisconsin right now and I was listening to this guy Raz, How I Built This, with the guy who started Culver's, which is like, you know, if anyone knows, it's like a fast food restaurant. It started in Wisconsin and it's so funny because the guy's so Wisconsin and instead of swearing on the episode when he was like really angry he said baloney to that and I was like, oh, that's so good, I have to use that.
Yeah, thank you. So thank you. I wanted to swear but I, you know, I don't want to get that explicit rating on our podcast with our sponsor. But okay. So, as you said earlier, why could you make such a great little hook and little jingle with AI using Suno? It's because you had domain expertise because you've made music manually the hard way. Therefore, you knew how to prompt the AI. I think obviously with the best coding, it's still going to be people who have written code and understand how to write good code and what good code looks like. And actually, and we'll get into this a little bit later, there's this new model
that came out recently, Jev, and I won't get into it right now because then I'll get too distracted from the point that you were trying to make. Um, but Jev, what what's really cool is it can you can use it to take in like the world's input. Um, so you could give it free form text or you could give it a a chatbot conversation or a phone call or whatever. And then you can have it do simple things like classification, like, oh, here's what I I'm a bank. Here's what the person said. Should it go to fraud department? Should it go to the payment department? Whatever. It's literally just like a little these are little like classifiers or little switch statements or if statements, if you will. Um, but it can work crazy fast and it's trained like an LLM on the on the world's um the internet's corpus. Um, so it's very intelligent. And anyway, these these are like new programming constructs that will actually go into code. And if you like zoom out and you think like, wait a minute, today we're in a world where the LLM just writes all the code end to end. But if you actually try to stick an LLM call in your code, it gets really funky because these LLM calls are like they take a long time potentially. They're not very bounded. They're not typed. So you don't quite know what's going to come back. It's just a string that comes back and you have to parse it and figure out. And if you even if you ask for structured output, so it comes back as like a JSON or something, it it's still like very slow and it's like inefficient. So anyway, I say all this to say there's going to actually be new ways of programming which the models aren't even trained on yet. So, I think there's all sorts of reasons why having domain experience as a programmer. Sure, for the Ben Thompsons, they don't need to learn how to code. Um, but for folks like you and I, like knowing how to write RTL or knowing how to code will help to obviously be able to prompt it better. And then again, as we continue to even invent new ways of programming, someone's got to prompt the AI to do that to train it on it. But to the question of do we even need apps? So, yes, we need to program. We need people who have domain experience in programming so they can they can do it better and more efficiently. Do we need apps? Okay, this is actually really interesting and I won't bury the lead. I agree with you that we don't need apps. And let's think from first principles. Why do we have user interfaces in the first place? At the end of the day, let's take Excel for an example. Dude, at the end of the day, you could just have a database and a prompt and like store data in a database and promp use SQL and just say like, hey, uh, whatever. I'm trying to um, calculate my mortgage. like you could put you could either write all this stuff in the command line or you know put a bunch of data in a database and pull extract it out and make calculations and everything could be the command line but obviously that's like a a very difficult thing for people to learn. So we make these user interfaces like Excel where it's like oh it's very visual and you can you know um have little if statements and summing and stuff like that and it's super simple. But the problem with all user interfaces is someone has to decide what the rest of us are going to be forced to use. And that's why um user interface designers are amazing because they're almost like psychologists. They can think really well about like what's the best experience and what are people going to try to do and what are they going to expect? Are they going to expect this button and that button? But still there's that new level. Yes, you didn't have to learn how to code, but you had to learn how to use the UI. So you still have to learn this paradigm that's forced upon you. And we've all been in UIs that are terrible where you're just like, "Dude, what the heck? This why is this button here or why or whatever and it just sucks." And ultimately, what that user interface is a slightly simpler way for us to tell a machine what to do, right? And so if it's not telling it with programming, it's telling it with some user interface that someone created. And by the way, it's never perfect. Like um for me, I I've got this like um very niche thing where I I have hearing aids and I and I can connect them via Bluetooth, but it's not normal Bluetooth. It's this other thing. And for me to access it, I have to go into the accessibility menu on Mac and scroll down and click on this thing and do this pairing. And then there's not like an easy disconnect. If I want to switch the headphones, I just have to forget it every time. Super cumbersome. But Apple has given me no other simpler way. And they don't give me the ability to make my own little like app on top of it where I could just I literally just want to push a button and say connect or disconnect, right? So I'm forced to live inside the paradigm that Apple made. And I, hey, I'm glad they made it a couple clicks instead of having to write code, but it still sucks. I would love to just use human language and say, "Disconnect my hearing aids." Right. And I think that's the point you're making and that Ben Thompson is making too, which is like at the end of the day, we all just want to talk to the machine, to the agent, to the computer, to the phone, tell it what we want, have it give it back. And of course, the beautiful thing and and Ben Thompson talks about this about like generating UI too is like if it gives it back to you in text or in uh speech, but you're like, I'm visual, create a nice visualization for me, it can just on demand create a visualization for you. And so I, you know, I definitely think that's the world where apps are going. But I think to your point that can impact like how we interact with things at the edge and make the edge much more interesting if we sort of expand our mind to say how could we interact with AI? How could we control computers? How could we revisualize and rethink things?
