Stanford AI Club ft. Modal: Scaling the Future of AI Compute

Yeah, thanks for having me.

Yeah. I grew up — I mean, I'm super old. I grew up programming on a Mac Plus. I started coding in the early '90s, using a tool called HyperCard, if anyone knows. It's like a black and white thing. Started building just like because I was bored at home. Just like started writing code, and I think at some point in '97 started writing, I think, JavaScript, maybe Perl, C++, a bunch of random stuff. Went to this like special math high school where I got into competitive programming, did IOI and some other stuff, and then I went to university. I kept doing some competitive programming, and I actually started physics. I wasn't really sure what I wanted to do. I think that's like kind of a consistent pattern in my life. Just kind of keep doing things, kind of going with the flow. But a bunch of my friends from — we had like a bunch of teams doing competitive programming at KTH in Sweden where I studied. A bunch of them, they ended up going to this weird music startup, and I was like, 'Yeah, what is this?' Like, I don't know, like maybe I should join because all the smart people I know was joining it. So I ended up joining Spotify in 2008 originally. I actually did my master's thesis there as a kind of an intern about music recommendations, and then I joined full-time in 2009.

I spent seven years there. It's actually kind of a longer story, but yeah, I spent seven years there mostly building the music recommendation system, kind of scaling that up to a team of about 25. At some point, I actually left Spotify because I was kind of bored and wanted to do something else. And I got a job offer from a hedge fund in New York. And this is 2011. So I moved to New York, joined a hedge fund. Loved New York, hated my job. So then I went back to Spotify actually again. So I worked at Spotify two times, in total seven years. But yeah, I worked at Spotify for three, four years in New York, and yeah, built the music recommendation system. The first version, I should say — I mean, I left Spotify 11 years ago. It's been a long time. And then left Spotify because I wanted to do something new. Started at a company called Better, which is a weird roller coaster. Was the number 10 employee. When I left, it was 10,000 people. Quit during the pandemic, started hacking on stuff, and eventually I built what's now Modal. That was my life story.

Yeah, I mean, this was like pre-cloud, and you know, there was like Linux. It wasn't that much. But by the way, I think that's true for any point in time. Like in 20 years, like there's going to be some new framework, and people are going to be like, 'How did you survive without, you know, Crab Core? Like how did you even write code without Crab Core?' I don't know if it's going to be called Crab Core, but so I think that's true in general. Like, you know, back then, you know, we had Python, and like I think scikit-learn maybe came out around that time. TensorFlow — like no, there was like Theano at some point came out 2012. But yeah, I mean, obviously we had to do a little bit more ourselves, and it was kind of annoying to write. And part of like that actually was what ended up informing Spotify — or sorry, Modal. It was the time that I spent at Spotify building the music recommendation system. I kind of felt like I had to spend 90% of the time like not building the actual like machine learning stuff, but like the infrastructure underneath. I open-sourced a thing called Luigi, which was like a workflow scheduler, and I built a vector database that I open-sourced. And it's like a lot of sort of infrastructure wasn't quite there. So that was actually in part what informed me later to build something like Modal. In a way, I kind of wanted to build a tool that I always wanted to have building in the machine learning land. So in a way, it's like the infrastructure now that I'm building is like kind of selfishly something I always wanted to have.

Go work with smart people, and you'll learn a lot. I think that's ultimately the most impactful thing I've done in my career is just, you know, working closely with — working at fast-growing startups too. I think it's always been to me — I mean, a startup degenerate. I only worked at startups my whole life. I interned at some big companies, but didn't like it. I don't know. I just like kind of like be part of like working together with smart people and building cool stuff. I think that's always like the best way to learn stuff.

Yeah. You can spin up an EC2 instance. I love AWS. Like I've used AWS since 2008, 2009 or something like that, and it enables me to build like a lot of stuff. But I mean, the developer experience is always like trash. And I think especially when you're like iterating fast and trying to build stuff, like you want like fast feedback loops. You want the ability to like spin things up and run it in the cloud. So anyway, so what does Modal do? So part of it is like it solves the developer experience, and that's actually where I started. I wanted something that was like very interactive. I can just write some code, launch it in the cloud, run on a bunch of GPUs, scale out, like all these things, right? What Modal does today is more than just the developer experience. We also focus a lot on just the productionization and running things at large scale in the cloud. And this is, you know, something that kind of we realized over time is like there are all these companies out there — or like we sort of built for it, but we didn't quite build for like Gen AI. But anyway, there's a lot of companies out there that are building AI applications, right? And whether that's like training or inference or sandboxes or many other things, right? Like so many cool like AI applications right now are being built. They all need infrastructure. And so what I think of like the vision for Modal is like we just want to build that infrastructure so no one has to think about the underlying stuff. People shouldn't think about containers or, you know, EC2 or AWS or writing YAML or Kubernetes. Like they should focus on what makes the beer taste good, like the sort of building the machine learning algorithms and the cool AI stuff, and then let us take care of the infrastructure. That's kind of my vision for Modal, whether that's like, you know, training or inference or anything else, many more things in the future hopefully. So, you know, obviously very large market, and we're building more and more tools, seeing a lot of traction. So I think, you know, my vision has always been in the end, you know, anything you need, you just come to us and we just like take care of it. You never have to go to AWS or something like that.

I think, I mean, we're an infrastructure company. So first of all, I think if you're building infrastructure, you shouldn't be afraid of like going deep, right? Like it would be a different thing if it were an application company. Like if you're like Spotify or whatever, like is it the right thing to build your own file system? Like no, like that would be a stupid idea. And I think a lot of, you know, VCs also when I told them like we're going to build all this like low-level stuff, they're like, 'Wait a minute, why aren't you just using off-the-shelf tools?' But I think inherently when you're building infrastructure, you should build all this stuff underneath. And one thing I think it's like maybe like kind of a cultural thing at Modal, and I always felt is like all these tools out there, like they're just built by humans, and like you can respect them for what they do, but you don't have to use those tools. And if a human built it, you can probably like build it in a different way. So like whether that's like Docker or Kubernetes or whatever it is, like these tools are complex, but you know, it turns out like a good engineer can sit down and kind of understand the essence of it and build the same thing, you know, in a different way and maybe not have all the different functionality, but like, you know, fundamentally like it's all doable. Like we don't go down and try to rebuild like Linux. I think something like that would be insane. But like a lot of stuff, like it's actually I think we're very good at identifying problems that are hard but not like mega hard. And like those problems are kind of good things to focus on because people are kind of afraid of it. They're like, 'Oh, like Kubernetes, like it's so complex.' Whereas like we were like, 'No, like we're going to build our own thing, like we can do it.'

Totally. I mean, the fact that we built our own file system, part of that was just I wanted a super fast feedback loop. And so that was initially driven by like the ability to just take code on my local laptop and just like execute it in the cloud in a way where like I feel like it's like super fast. One thing I've learned, I've written code for I don't know 35 years or something, is like developer experience or like developer productivity, a lot of it comes down to this like super fast feedback loop. If something takes a minute, like you just like lose focus, or at least I don't know, maybe I have ADD or something. So like having these like fast feedback loops, I think it's just so critical. So we spent like an absurd amount of time like, you know, with the file system and like lazy loading and content addressing and like making sure that containers can start in the cloud and just like get the data it needs like very fast and cache that. That ended up later like turning out to be super valuable for like basically building serverless functions because now you can boot containers like extremely fast and you can scale up and down very fast. And so what we realized is like that kind of unlocks a different way to think about running code in the cloud because like we can do these, you know, very heavy like compute-intensive workloads in a fully serverless way. And it's not just that, we also have CPU snapshotting now, we have GPU snapshotting. The fact that we can enable this like fully consumption-based model where like you don't have to reserve like set aside like a static number of pods or whatever, and we can just handle these like very bursty workloads. A lot of that is like because we invested in like very deep infrastructure and built it all ourselves. There's many more things, like we built our own resource manager. So like that now enables us to run capacity all over the world, and we're working on this like low-latency tier. So we can like run things like kind of like a CDN, like close to customers, and kind of just like route intelligently to like wherever it's like closest to you. So we get low latency. Developer experience, of course, they already talked about that. Yeah, we can manage all the capacity way more efficiently because we have like this unified system that like optimizes capacity at any point in time. So like having all that control and like building all the custom stuff, like it really to me — it was a longer route, it took a long time to lay the foundation for that, but now that we have it, it's like an enormous benefit for us as a company.

I think it's really just time, right? Like you can often ship something that doesn't feel magic like pretty quickly, but like making something feel like kind of magic for a customer, I think that always like takes a lot of time. Like you need to kind of rethink a lot of the underlying infrastructure. But by the way, like one thing I think is like kind of fascinating, Spotify, like the magic moment for me was like, you know, I'm sure you all use Spotify, or most of you, you should all use it. Like you just double-click on a song and like it just like starts streaming. That was like insane. The first time I experienced it, it was like, 'Wait a minute, this music is like in the cloud and it just like streams.' So like I think it's like finding those like magic moments and like, you know, making it like conceptually it's like very straightforward. It just kind of feels like the music is there. But like when you think about all the technology, at least back then, that we had to put into that at Spotify, and I think similar like at Modal, like making that sort of feel magic, that's often a lot of technology that you have to invest in. Other things, you know, around like the SDK, making it like simple, kind of matching people's like conceptual model how to think about things. It's very easy to start to like ask customers like, you know, what do they need? They want to customize in a million different ways. Suddenly you end up with an SDK that has like 2,000 different flags, different settings, different options. But often you can kind of rethink that from first principles. Okay, what are the trade-offs? You want to trade off like cost versus latency, whatever. And then like kind of change how you like manifest that to the customer and optimize it ourselves. But these are hard technical problems. But like once you nail that, you have that magic experience, like it takes a lot of time to do that. But fundamentally it's like so much better.

I mean, we were always like focused on inference. I don't know if it's like super different for us. I think it's helped us a lot the fact that like the inference market has gotten like a thousand times bigger in the last few years. But like we never really did training. We're working on training right now. So I think training is something we hopefully can launch in the next maybe few months. I don't know, like take it to GA maybe a little longer. But yeah, inference has been huge for us. And then sandboxes and a bunch of other stuff like we — batch jobs is something we do a lot too. Yeah, like I wrote a blog post about this like a couple years ago how I thought inference is going to get huge. And I don't know, it's kind of obvious when you think about it because training costs money. You don't make money from training. You make money from inference. So it's obvious like in the end like inference has to be much larger than training.

I don't really think about it like — I think when you start a company, you kind of have to draw like certain — kind of have to like draw like a segment and say like this is the part of the market that we care about. Like we said what we care about is like running stuff in the cloud. You know, there might be people who want to run stuff, but by the way, I think people like kind of phrase these things as like one versus the other. I think it's probably true that like local inference or like on-device, like edge, whatever, is going to grow like 10x, maybe 100x over the next like few years. I also think it's true that cloud inference is also going to grow 10x, maybe 100x over the next few years. So I don't think it's like one versus the other. I think both things are going to grow like enormously. So like, you know, we're in the space of like cloud stuff, like we do cloud really well, like that's what we focus on. We don't really do stuff — we work with robotics companies and stuff like that, and you know, sometimes they, you know, they're running models for like self-driving cars. Like obviously they don't want to control a self-driving car on Modal, you know, which is fine. Like basically what they do then is, you know, they run some simulations or training or whatever in Modal, and then they take those models and they deploy them to the self-driving cars. Luckily, I mean, there's a lot of ways like containerization and like you can, you know, models are pretty, you know, model artifacts are pretty easy to port from one runtime to another. So it sort of works, but you know, we're firmly focused on the sort of the cloud. That's what we care about.

I don't think so.

I think the cloud is just so big. I mean, AWS — like look at AWS, I guess they tried, they have like Outposts and like a few things, but like it's just so big, like the cloud. I think we're still early with the cloud, by the way. I think the cloud's going to get even bigger, like 10x.

I don't know, in the next 10 years maybe.

I think it's like semi-intentional. I think part of it just reflects the fact that I only worked on consumer products and in particular Spotify. I think Spotify like basically zero marketing or like sales — or I mean, I guess there's like sales for like ad sales, but I mean Spotify just built like a really good product and then just like people just wanted it. In fact, actually, Spotify is like funny, it's like an absurd product because like it obviously had like very clear product-market fit from day one, but like it was very unclear if it's going to be a viable business. Luckily it turned out to be a viable business. But anyway, like the way I started building Modal, I came from a consumer background. So for me it was all about like I want, you know, people to feel that magic moment as soon as possible. Just get them through the onboarding process. Make it super dumb simple to like set up the thing, get started writing code in the cloud. Like just making them feel that thing. And I mean that's not like I think how a lot of software gets built, especially not in the B2B space. But for me it was like a very natural way to do it, and I don't think I reflected that much on it like when we built it. I just wanted people to pick it up, and then like the number one channel was like me like just posting stuff on Twitter all the time, just like 'try it out, like it's really cool.' And that's how we got a lot of customers, and I think because it resonated with people because I think the product was quite good too. And so what we call, you know, PLG, product-led growth, has always been like a big part of our growth strategy, whether intentional or non-intentional, I think kind of somewhere in between. But it's worked out really well for us, and I do think when I look at like, you know, very successful businesses in general, like there is often that sort of element, like whether that's like AWS or Stripe or Twilio or Datadog or like a lot of these companies, like they often start out as like having strong adoption from like the grassroots sort of bottoms-up adoption and then layering on sales. So that's what we've been doing in the last couple years, like, you know, adding more of like an enterprise sales touch, which is something I don't really have like a background in, but like, you know, I've come to appreciate like whatever MEDDIC methodology or whatever. There's all these, you know, sales books that you can read if you have a, you know, problem falling asleep. So I sort of come to appreciate like, you know, part of the deal like doing big enterprise contracts and stuff.

I talk to customers all the time, and I think that's super important, and I listen to their concerns, and I listen to what they tell me we should build. I don't generally build those things like because I think often like you need to take your customers very seriously when they, you know, like when they express like, you know, 'this is hard to do, why couldn't you do this or that,' whatever. But like those customers, like they're just going to come up with like the simplest possible solution, and that's like very rarely the correct solution. So I think it's really important to understand their pain points, like why, you know, what is the thing that's like hard for them to do, but then actually don't really listen to their proposed solutions. They're often like not that good. Which is like I don't, you expect that either, right? Like I think, you know, like they haven't spent as much time thinking about Modal as I have. So like, you know, so I think it's very important to talk to customers all the time and understand their pain points. But I think at the end of the day, it's my job to then like synthesize all these like pain points and like come up — or not just me, my, you know, the whole company — and then figure out like what to build for them. And that's often very different than what they're asking for.

I don't — I think it kind of comes pretty naturally. You just have to like ask, you know, it's like the five Ws or whatever. You just have to go back to the root cause. So what is it they're struggling with, our customers? Like what are they actually trying to build? Is this good? Like when you like — what is, you know, like you write up like a proposed SDK and you look at it, it's like 'this looks kind of ugly,' you know, like then you're like, 'Huh, like I don't think a customer would understand this.' And then you iterate a little bit on it, and like eventually you land with a solution that's like, 'Yes, this looks very clean. I'm not embarrassed of like going back to the customer and showing them this.'

Yes. But I'm drawing a blank right now. I know there's like a couple of features we built that like didn't really take off. I mean, I guess like notebooks is like a lot of people asked for it. I guess it's like sort of taken off, but not really from like a business point of view. I don't know. There's like probably a few other things we built that didn't really take off. It's weird though, like you never know. Sandboxes, we put it out like three years ago. No one really used it, and then like it exploded last summer. Although sandboxes, that was not something customers were asking for. That was just an idea we had.

First was like vibe coding. So it's like Lovable and a bunch of customers started using it for vibe coding, and then it was like reinforcement learning, and then it was background agents, and now there's like a bunch of other stuff. So that's been kind of a crazy last sort of nine months I think with sandboxes, which is now like a quarter of our revenue, growing super fast too. Kind of unexpected honestly, but I don't know, it worked out. It took a few hours I think for Rockshot to build the initial feature.

Then we put it out. I mean, we obviously like iterated. Now that we have a lot of traction, like now we spend a lot of time like making it better all the time.

Yeah. I mean, inference is probably like 70% of our revenue. It breaks down though. It's like it's actually not as much like LLM. Like LLM is maybe a third of that. So it's like maybe LLM inference is like 25%. Then it's like other modalities like audio, video, music — Suno is a big customer — RunGen, like video, like those modalities, like it's probably like half our revenue. Then there's like sandboxes a quarter, and then like 5% like random stuff like there's batch jobs that we actually have a lot of, computational biotech too, which is kind of cool. Physics, we have some people using us for weather forecasting, which is kind of cool. So yeah, I mean, it's — which is always like to me that was always the goal, like we always wanted to build something very general purpose. The fact that you can use Modal for all these things to me is like makes me so happy that that started happening for real in like 2025 last year, you know, that we had like a lot of customers in so many different verticals. I think it's just going to get more. I'm really interested going into gaming, finance...

Like whatever, I think there's so many compute-intensive things, and I want Modal to be this platform you use for all of that.

I mean, I think weather forecasting is super cool because it's so unexpected, and maybe because I have a physics background too, I find this super fascinating. I mean, we have the whole spectrum. A couple of days ago I was going from one meeting to another where in the first meeting they were using Modal for face swapping, and then the second meeting they were using Modal for solid-state physics, like optimizing materials, and then I went to another meeting and they were using Modal to cure cancer, and okay, that seems worthwhile.

So I like that it's a whole spectrum, and I love Suno too. It's the whole spectrum from very serious use cases to very... my kids love Suno too. A lot of scatological humor in their songs. It's kind of fun.

So partly because it's music, because I know that space a lot. But I think Suno to me is an example in a way. The reason why I like it so much is that this whole company could not have existed before Gen AI, right? There's a lot of companies now that are adding a bunch of random features, sometimes they're good, sometimes they're bad, but Suno is fundamentally a company that was completely enabled by Gen AI. And that to me is kind of cool. It's technology enabling something that fundamentally couldn't have existed.

I don't know.

Yeah, and that's a terrible mistake. Unforgivable. I think honestly, today the LLMs are so powerful, it's kind of like I almost feel bad in a way, but you almost don't need to train models initially. Just use prompts and get started, and then later when it starts to take off, then you can go and rethink.

I don't know, it's just kind of remarkable to what extent LLMs can be used for almost everything. Like even in weird domains like robotics or biotech or whatever, you often could just start with prompting and get pretty far, and then kind of go back and rethink. So I don't know, that's one. But of course, in other domains you kind of have to train your models.