Can AI Infrastructure Work Like Magic? Erik Bernhardsson, CEO, Modal

Thank you. It's exciting to be here.

Yeah, it's like the data influencer. That's the thing.

Yeah, so Modal makes it easy to build, scale, and deploy applications in the data, AI, machine learning world. So basically, you can think of it as like we take — you write a bit of Python code, and we take that code, we stick it in a container, we execute that in the cloud in a way where you don't have to think about infrastructure, you don't have to think about GPUs, you don't have to think about, you know, the cloud vendors. We just magically make that work and scale it to very large, you know, thousands of GPUs potentially.

Yeah, and by the way, I think it's funny because it's like the typical VC, like, you know, what's the market map? And my reply is like, there is no — like, we defy it. Like, we do all of it.

No, I don't think it was the nonsense. I think I was just pointing out how much stuff there is. I love your market map, by the way.

It's still in there.

So there's obviously the clouds, right? Like AWS, GCP, Azure, etc. Oracle, I should, you know, shout out to Oracle. Actually, I like them a lot. We use them, big fan. So they, I think, you know, have been there for a long time and have been doing a phenomenal job building, you know, sort of primitives for running stuff in the cloud. More recently, there have been a bunch of sort of newfangled — I call them like the alt clouds — CoreWeave, Lambda, what are the other ones? There's like FluidStack, there's Vulture, there's a lot of them, right? Like they sort of, you know, as it turns out with like GPUs, what people really care about is like, just want a box with a GPU in many cases. And so, you know, that's something that's like a little bit more like of a commodity and I think a little bit easier to build. And arguably, you know, they would probably disagree with this. So I think we've seen a sort of, you know, renaissance of like — or like not a renaissance, but like a bunch of new sort of cloud entrants, like, you know, sort of competing with the hyperscalers, focusing more on just like only GPU availability. And so those, I just — I'm a developer, if I want access to a GPU, instead of buying my own GPU, I go to those clouds, and there will not be much else, but there will be what I need. It's like, you know, they provision a box with a GPU and you can SSH into it, right? So it's like sort of like you get a GPU on your desk. But so those people are in the sort of data center business, meaning that they will have the GPU farms and the electricity. Yeah, it's like sort of like almost like real estate, right? Like they own sort of, you know, real estate and they rent it out. Arguably sort of, you know, WeWork, you could say, if you want to be, you know, a little cynical. But like not a ton of value like higher up in the stack, which is like, you know, I think where we play is, you know, slightly higher up in the stack. I think like if you go up in the stack, you know, start looking at like more like applications or infrastructure to service, there's like a few different providers. There's, you know, first of all, there's a whole bucket of all the LLM inference providers, you know. So there's OpenAI, Anthropic, you know, then there's like a few ones focusing on open source models. So Fireworks, Together, I think Anyscale to some extent, Replicate, etc. So those I would put in the LLM bucket. And then it's like a slightly adjacent bucket, I would say, of like more like AI as an API. And actually that, you know, probably more like where I would put Replicate. I also put Fal in there. I think we occupied a slightly different bucket, which is more like you're not using a model through an API, you write code. Like we're all about like running — taking users' arbitrary code and running it in the cloud, right? Like we're a serverless compute platform. So we take your code, we execute it in the cloud. Like you can run whatever code you want. In many cases that involves a GPU, in many cases not. In that space, I think there's like less of an obvious — I would say like Baseten, maybe to some extent. It's like somewhat similar to us. Like they focus only on model inference. We do model inference. Model inference is probably the majority of our use case, but we also have many other use cases. Like we have people using us for batch, we have people using us for, you know, large-scale like video encodings or building web services or all kinds of other stuff. And then I don't know, what other boxes should I talk about that we didn't cover? I think that's pretty much it. You know, at the very beginning, there would be just like the pure hardware, which, you know, people can still use directly obviously without any of those cloud solutions, but that's a whole different level of effort.

So I think it's sort of interesting to see, you know, for LLM, I think like the reason why it's happening for LLM is like it's like a kind of well-defined API. It's like text in, text out, and it works really well. And it's sort of, you know, a few big models like dominate everything. The spaces where we tend to compete and where we have most of our customers is like more like very custom models, like people training their own proprietary video models, image models, like audio, things like that, or having very custom workflows. And there's like less of sort of like, you know, power law there. Like there's like, you know, thousands of models. Like everyone has like their own slightly different models. And that's where we focus is like if you have, you know, your own custom model, you know, you're probably really good at like training models and building applications, but we handle all the infrastructure. So, you know, all like autoscaling and all the, you know, making sure utilization is maximized and all these things.

I mean, I don't know, like I don't know if I necessarily believe in it, but I certainly bet on it. Like, you know, I think there's some sort of, you know, existential risk that there's like such a good multimodal model that people just use it for everything, right? It's like a model to rule it all. I don't know if that's true. Like, I tend to think it's probably less likely to happen than other people, just because like, you know, having seen so many like custom, very, you know, all the idiosyncratic, you know, idiosyncrasies around like building like custom models. Like let's say you want to, you know, build like a voice cloning thing, right? Like, you know, probably like a multimodal model is not going to care about all the specific things around that, right? Like, I don't know, I'm a big believer that like, you know, in the long run, like people will train a lot of different custom models, even though like the multimodal models may handle even more stuff. I mean, the pie is growing, I guess. Like it's like a general statement of like software engineering in the last 30 years, but like the pie always gets bigger.

Yeah, so I've done a bunch of different things, but in particular, I was at Spotify for seven years and built a thing called Luigi, which was a workflow scheduler. So the sort of problem I tried to solve at Spotify was we had a lot of different batch jobs and you ended up with like a very complex graph. You know, you need to run like hundreds of different jobs in certain sequences and sort of very parametric ways. And so I built Luigi to sort of, you know, help us figure that out, like how to execute all of it in like correct order. Open sourced it, a bunch of people started using it. I think, you know, this is like 2011. Then I like stopped really caring about it in 2015, like Airflow kind of took over. And then like later now there's like Dagster and Prefect and Flyte and a bunch of other ones. I mean, I still think it's a good idea and like kind of, you know, it related to like how I ended up starting Modal. Like part of, you know, how Modal came to be is I actually started looking again at workflow scheduling in late 2020 when I realized I wanted to start something or build something. And I started thinking a lot about like, you know, what makes a good workflow scheduler. I even built one. I have some like janky code in my laptop. I realized at some point like a workflow scheduler is only as good as like the sort of underlying compute layer. Like because at the end of the day, you need like to sort of, you know, you can't just like describe the work you need to do. You also need to describe the code. You also need to write the code. And if that like integration point is bad, you know, if it doesn't, you know, if you have like kind of a, you know, like a system where you have to like define the workflow over here and the application code over here, like you're never going to have a great experience. So actually, I'm just going to stop working on workflow scheduling. I'm going to focus all on the compute provider instead. And what makes a good compute provider? And I realized what I want is I just want something that like lets me define application code in Python because, you know, I was very focused on like data, AI, machine learning applications and let this infrastructure handle all the rest, like all the scaling, all the, you know, provisioning, all the resource management, all that stuff. And, you know, I just got obsessed and carried away. And, you know, three, four years later now, you know, I have a company around it. But as sort of the genesis of Modal, so it kind of started with Luigi in a way.

No, I don't know, but that sounds good. No, I feel like we had a conversation many years ago and I probably said something like that. You tell me. But yeah, I do seem to remember that like at the beginning.

Yeah, like late, early 2021 really. That was the time when what was top of mind, I think, for a lot of potential customers were more like data infra and it was pre, pre, pre. So it's a little bit different. And yeah, I think that's always been the ethos of the company is like I kind of at the end of the day just want to build better tools for data, AI, machine learning teams. And what we actually build, I'm actually somewhat agnostic, you know, at the end of the day. Like I want to be a platform that helps people build those types of applications.

I mean, to some extent. I mean, a little bit. Like I, you know, like I don't know, like AI is like, you know, has like ups and downs. Like I think we had like whatever, like what, like, you know, four AI winters or whatever. Like now, like AI is like a great term. I feel like, you know, 2020, AI was like a bad term. You know, I used to say only like data. I'm just going to focus on data. You know, there's been so many other like terms coming and going. Like to me, it's all the same. It's like data, right? Like you're working with data and like sometimes you're training models, sometimes you're doing, I don't know, like data pipelines and batch compute. But like at the end of it, it's data.

Totally.

So I think the core of it is really just like compute and running containers. That's where we started. Like we always had a vision of building a platform, but like the core of it is like we want to make it easy to run code in the cloud. And, you know, it turns out that like inference is such a killer use case for that, that, you know, effectively spent the last like three, four years now like entirely focused on just like how do we take user code and execute it in the cloud in a way where like, you know, we handle the scaling and all the infrastructure. We kind of let everything else like sort of wait until we really nail that experience. How that breaks down is under the hood. And we realized in order to deliver that experience, we had to kind of go deep in the layer of like the infrastructure and throw out a lot of existing stuff. Like in particular, one of the things I always cared about is like developer experience. Like I wanted to have, you know, like almost like build a tool. I always wanted to have, through my own experience like building these types of things, I didn't invent it, but like at Spotify, I built a music recommendation system. So it's a big part of like, you know, my life used to be like, you know, iterating and like shipping and deploying things and like, or just running, you know, various types of batch jobs and like, you know, deploying and or debugging things. And at the core of that, like one thing I realized, for instance, was like if you don't have like a good, super fast feedback loop, like you can never make it. Like so much of developer experience comes from having like a super fast feedback loop where you can like take code and execute it in the cloud in a way where it almost feels like it's local. And if you solve that problem, you solve a lot of different problems. Like you make engineers more productive because they can run things quickly. You sort of kind of, you know, bridge the entire gap. You almost like, you basically get rid of the sort of, you know, what I think was like the gap between local development and cloud development, which I think with AI and machine learning has always been a big challenge because, you know, you end up having like different environments, like different toolchains, and then, you know, you end up having to take like research code and like repackage it and rerun it, you know, in order to like get it out there in the cloud. So if you get rid of that like distinction between local and cloud and just turn it into like one environment, you always run things in the cloud, you solve a lot of different problems. But in order to do that, you have to make it fast. You almost have to make it feel like you're developing locally when you're running things in the cloud. So what we realized, like in order to do that, you have to solve a number of core technical challenges around container starting. So like what I mean is like you have to take code and you have to, you know, ship it to the cloud and you have to start that container. And ideally you want to do that in like a couple seconds at most because that's like, you know, sort of a human reaction speed. It's like kind of feels snappy. And then we started looking at, okay, well, you know, what actually happens typically in a system when you do that is, you know, you deploy something to Kubernetes and then, you know, you have to pull down a Docker container image and that then has to start up. And all of those like steps are super inefficient. So we built our own. We threw out Kubernetes out the window. We threw out Docker out the window. We built our own file system in order to optimize for how container images are distributed. We built our own scheduler in order to maintain this pool of workers and make it possible to start tasks very quickly. We built our own container image builder because we had our own container image format. So, you know, which is kind of a big effort.

I mean, it was like, you know, like, yeah, too long. And this was during the ZIRP era. So like, I feel like, you know, VCs are like, yeah, cool. I mean, like people were like asking like, why do you really have to go so deep? And, you know, but fundamentally, I think, you know, we felt like, you know, we're going to solve this problem well. So we spent basically like, you know, a year and a half, two years, two and a half years, like almost like in a cave, just like building a product that like no one had used, which is maybe a bad idea. And then like when we actually like kind of emerged out of that cave, you know, it turned out like we didn't really have like an obvious use case. So we struggled for like, you know, six months or a year or so to try to find a use case. And then luckily all this like, you know, Stable Diffusion came out in like summer of 2022 or something like that. And a lot of people started coming to us and like, actually, this kind of makes sense. Like you have this like serverless GPU thing. You can sort of, you know, and realize like this makes a lot of sense for us to focus on. And then we started seeing actual like more like meaningful like traction with like users.

They're like all over. So we use, and this is not secret by the way, we use AWS, we use GCP, we use Oracle. We have a number of alternative cloud providers now that we're also using, and we're doing a lot of different regions. We have a bunch of GPUs in the US, we have a bunch in Europe.

Yeah, totally. And for a few different reasons. Like one is just capacity management. Like, you know, from day to, you know, from time to time, like capacity varies, right? Like the sort of cyclicality, if you look at like Europe versus US is actually kind of interesting. You see like a lot of, you know, when, when sort of, you know, when it's like early morning, like you see like, you know, US is not, you know, high utilization, but like Europe has high utilization and vice versa. So you can play all these games kind of just like aggregating capacity across different regions. Pricing is the other obvious dimension, right? Like we actually have a system that continuously looks at cloud pricing and solves an optimization problem to sort of, you know, figure out how do we allocate capacity across like, you know, a hundred different regions in order to deliver the capacity we need to our customers. Because we use a combination of reservations, but also a lot of like spot and on-demand capacity, which means like, you know, we rely on the cloud's ability to scale up and down, which, you know, because it's a sort of supply-demand matching problem, right? Like we have volatile demand coming from our customers, which means like ideally we also have, you know, the ability to sort of match that with, you know, on-demand capacity on the other side. Which is also like why I think we found a good product-market fit with inference, because inference is inherently sort of unpredictable, right? Like you don't know necessarily, you know, when you're deploying something, what the usage is going to be. There's going to be a lot of daily sort of, you know, variation. It's going to be a lot of like weird spikes, you know, when something goes viral, etc. And a big part of the pitch of Modal is, you know, you can use us, we'll just scale automatically. You don't have to worry about, you know, setting up, you know, making a sort of cloud commitment, especially not, you know, three-year or, you know, one-year reservation. You can just like, you know, deploy things on Modal. And then if you one day need a thousand GPUs, we can typically get you a thousand GPUs like pretty quickly, like talking minutes.

Yeah, you can definitely like take a Hugging Face model and deploy it. That's what a lot of people use Modal for. I think where Modal really shines is also like people training their own models, like having custom. One example I always bring up is like, you know, amazing use case, I love the product, is Suno, which is AI-generated music. And, you know, they have a big cluster, they train their own models outside of Modal, and then they use Modal for the inference side. So, which means, you know, like we run, you know, generate like all the sort of generation of AI-generated music happens on Modal, like very large scale, right? Like we, you know, millions and millions of music generated on Modal, sort of full cycle. It's actually funny, you know, like when I talk to them, I end up talking a lot about like licensing and stuff like that, music licensing, because I happen to know a lot about that. But yeah, so what I love about that is, you know, they can do what they're good at, which is to, you know, build an amazing user experience and a user application and train models. And for all the sort of infrastructure, you know, mess, they can effectively outsource that to Modal. And, you know, that happens to be something we love and we can do really well. So this is sort of, you know, I think that's sort of comparative advantage of, you know, specialization in that sense.

We are interested in training. We're looking at it. I think training like very large scale, like people are very price sensitive, transactional. Like I think you have like very different sort of requirements. Like typically like infinite bandwidth and like with very high interconnect, it puts a lot of different, you know, constraints on the infrastructure that we don't support today. It's something we're interested in, like, you know, looking at down the road, but it's not like a super high priority for the business right now. I think like another use case that we do support, it's actually like a lot of like pre-processing for training. So a lot of users use this for like, you know, let's say you have, you know, millions and millions of audio files or video files and you want to extract some training data from that. Like typically, you know, that requires a sort of batch job that like, I don't know, extracts features from that. And that's something that Modal does really well too. It's like very high, like bursty, you know, batch parallelism, just like fan out, like parallelize, run this over millions and millions of audio files or something like that, and then you're done, right? So it's sort of the ability to be elastic and just like grow very quickly.

Because everything in Modal is usage-based, so you just pay for like the exact time your GPUs or CPUs run.

I think fine-tuning is like, you know, a year ago I would have said like fine-tuning is like absolutely critical. I think now the models have gotten so good that like, you know, this is a question of like, you know, is fine-tuning even needed there? I think fine-tuning has a lot of different, you know, use cases. I think in particular if you like very domain-specific models, let's say you're like focused on like financial data or, you know, or document extraction, something like that, I think fine-tuning absolutely makes a ton of sense. And so we have a lot of users using Modal for that. I think beyond that, like, I don't know, like fine-tuning is still to me like a little bit like an unproven thing, but clearly there's like certain places where it does make sense.

Exactly. Yeah. And also like a couple years ago, you know, a lot of companies tried to train their own LLMs. Like that to me makes almost no sense, right? Like, you know, MosaicML and stuff like that. So, you know, so yeah, like I agree with you, right? Like, you know, it's obvious that like in the long run, like inference spend will dominate just for like the obvious reason that like that's where you make the money, right? Like, you know, when you're training a big model, that's like, you know, you're spending a lot of money training that model. How do you recoup that? Like it's typically through inference, right? If you look at OpenAI, any of these businesses. So to me, it's very clear that like maybe in the past we've had like 80-20, you know, spend GPUs, you know, training versus inference. I think in the future it might be the other way around. Like, you know, much more, you know, more dollars will be spent on inference than training.