Today I'm chatting with Erik Bernhardsson, founder and CEO of Modal. Modal developed a serverless cloud platform for AI, machine learning, and data applications. And before that, Erik worked at Better.com and Spotify, where he led Spotify's machine learning efforts and built a recommender system. Well, Erik, thanks so much for joining me today on No Priors.
Yeah, thanks. It's great to be here.
So, if I remember correctly, you worked at Spotify and helped build out their ML team and recommender system, and then were also at Better.com. What inspired you to start Modal, and what problem were you hoping to solve?
Yeah, so it started at Spotify a long time ago, 2008, and I spent seven years there. And yeah, I built the music recommendation system. And back then, there was like nothing really in terms of data infrastructure. Hadoop was like the most modern thing. And so I spent a lot of time building a lot of infrastructure. In particular, I built a workflow scheduler called Luigi that basically no one uses today. I built a vector database called Annoy that, you know, for a brief period people used, but no one really uses today. So I spent a lot of time building a lot of that stuff. Then later at Better, I was a CTO and thinking a lot about like developer productivity and stuff. And then during the pandemic, I took some time off and started hacking on stuff. And I realized I always wanted to build basically better infrastructure for these types of things like data, AI, machine learning. So pretty quickly I realized like this is what I wanted to do, and that's sort of the genesis of Modal.
That's cool. How did that approach evolve, or what are the main areas that the company focuses on today?
So I started looking into, first of all, just like what are the challenges with data, AI, machine learning infrastructure, and started to think about it from a developer productivity point of view: what's a tool I want to have? And really, a big sort of challenge is like working with the cloud is arguably kind of annoying. And as much as I love the cloud for the power it gives me, and I've used the cloud since, you know, way back 2009 or so, it's actually pretty frustrating to work with. And so in my head, I had this idea of like, what if you make cloud development feel almost as good as local development, right? Like have these fast feedback loops. And so I started thinking about how do we build out, and realized pretty quickly like, well, actually we can't really use Docker and Kubernetes, so we're going to have to throw that out, and probably going to have to build our own file system, which we did pretty early, and build our own scheduler, and build our own container runtime. And so that was like basically the two first years of Modal, just like laying all that foundational infrastructure layer in place.
Yeah. And then in terms of the things that you offer today for your customers, what are the main services or products?
Yeah, so we're infrastructure as a service, which means like on one side we run a very big compute pool, like thousands of GPUs and CPUs, and we make it very easy to get, you know, if you need a hundred GPUs, we can typically get you that within seconds. So sort of one big multi-tenant pool, which means like capacity planning is something we kind of take, you know, it's something we solve for customers. They don't really need to think about reservations. We always provide a lot of on-demand GPUs. On the other side, there's a Python SDK that makes it very easy to build applications. And so the idea is like you write code, basically like functions in Python, and then we take those functions, turn them into serverless functions in the cloud. We handle all the containerization and all the infrastructure stuff, so you don't have to think about all the sort of Kubernetes and Docker stuff. And the real killer app, as it turns out, like we started this company pre-GenAI, but as it turns out, the main thing that really started driving all the traction was when Stable Diffusion came out, and a bunch of people came to us and like, hey, actually this looks kind of cool, like you have GPU access, it's very easy to, you know, you don't have to think about spinning up machines and provisioning them. So that was like our first sort of killer app, was like just doing GenAI in a serverless way, with the focus of diffusion models. Like now we actually have a lot more of different modalities. Like a lot of usage is still like text-to-image, but we also see a lot of audio, music. So one example of a customer I think is super cool, building really amazing stuff, is Suno, which does AI-generated music. So they run all their inference on Modal, very large scale. There's a lot of customers like that sort of dealing with like, you know, building cool GenAI models, in particular I would say in the modalities of like audio, video, image, and music, stuff like that.
That's cool. And I think Suno's using all like Transformer backbone now for stuff, right, versus a diffusion model based thing?
So I think it's a combination of both. I'm not sure though.
Yeah, I think they talk about it publicly, that's the only reason I mention it. But you wrote in October this post, I think it was called 'The Future of AI Needs More Flexible GPU Capacity'. And in general, what I've heard in the industry is that a lot of ways that people use GPUs are really going to be wasteful. And so I'm a little bit curious about your view on flexibility around GPU use. How much is actually used versus wasted? How much optimization is left, you know, even just with existing types of GPUs that people are using today?
Yeah, I mean, GPUs are expensive, right? And I think it's sort of kind of as like a paradox. It's like means that cloud, you know, a lot of the cloud capacity, it's like the only way to get it is to sign long-term commitments, which I think for a lot of startups is really not the right model for how things should be. Like I think the amazing thing about the cloud was always to me that like you have on-demand access to like whatever many CPUs you need. But for GPUs, like the main way to get access has been over the last few years, due to the scarcity, has been to sign long-term contracts. And I think fundamentally that's just not how startups should do it, right? Like, you know, and that kind of, it's been sort of supply-demand issues. But just, you know, looking at the CPU market, the fact that you have like instant access to thousands of CPUs if you need it, like my vision has always been this should be the same thing for GPUs. And that means, you know, especially as we shift more to inference, I think for training it's been sort of less of an issue because like you can sort of just make use of the training resources you need. But for inference especially, you don't even know how much you need, right, like in advance. Like it's very volatile. And it's a big challenge that we solve for a lot of customers is we're fully usage-based. So when you run things on Modal, we charge you only for the time the container is actually running. And that's a massive hassle for customers traditionally, like doing the capacity planning and thinking how many GPUs, and then having this issue like either you over-provision and you're paying for a lot of idle capacity, or you under-provision and then you have, you know, when you run into capacity shortage, like you have degradation in service. And so whereas with Modal, we can handle these like very bursty, very unpredictable workloads really well because we basically take all these user workloads and just run a big pool of thousands of GPUs across many different customers.
Yeah, one of the things that always struck me about training is, to your point, you kind of spin up a GPU cluster, you run a huge supercomputer, right? And then you run it for months in some cases, and then your output is a file. And that's literally what you've generated. You know, it's kind of insane if you think about it. And that file in some sense is a representation of the entire internet or some corpus of human knowledge or whatever. And then to your point, with inference, you need a bit more flexibility in terms of spinning things up and down. Or alternatively, if you're doing shorter training runs or certain aspects of post-training, you may need more flexible capacity to deal with.
Totally. And that's something we're really interested in right now. Like traditionally, most of Modal has always been inference, like that's been our main use case. But we're really interested also in training. So in particular, like probably focus more on these like shorter, like very bursty sort of experimental training runs, not the like very big training runs, because I think that's a very different market. So that's like a very interesting thing.
How do you think about meeting people's end-to-end needs? So I know that there's a lot of other things that people do. There's, you know, a lot of people are using RAG to basically augment what they're doing, or, you know, there's a variety of different things that people are now doing at time of inference in terms of using compute to, you know, take different approaches there. You know, I'm a little bit curious how you think about the end-to-end stack of things that could be provided as infrastructure and where Modal focuses or wants to focus.
Yeah, totally. I mean, our goal has always been to build a platform and cover like the end-to-end use cases. It just turned out that inference was, we were well positioned to focus on that as our first killer app. But my end goal has always been to make engineers more productive and focus on what I think was like the high-code side of ML. Like where I think our target audience tends to be more like sort of traditional like ML engineers, like people building their own models. But there's many different aspects of that. There's like the data preprocessing, then there's the training, and then there's the inference. And that is actually probably like even more things, right? Like, you know, having feedback loops, RAG data, and like, you know, online ranking models and all these things. And so my goal for Modal has always been to cover all of that stuff. And so it's interesting, you see a lot of customers now. We don't have a training, but a lot of customers use Modal for batch preprocessing. So they use Modal to, you know, maybe they're training a video model, so maybe they have like petabytes of video, so then they use Modal actually maybe with GPUs even to like do feature extraction, and then they train it elsewhere, and then they come back to Modal for the inference. So for us to do training makes a lot of sense. And in general, I think there's a lot of, you know, makes a lot of sense to sort of build a platform where you can handle the entire sort of machine learning lifecycle end-to-end, and many other things related to that, also the data pipelines and nightly batch jobs and all these things.
Yeah, I mean, what you describe is a pretty broad platform-based approach. I think there's a handful of companies who are sort of in your general space or market. How do you feel the Modal differentiates from them?
I think first of all, we're cloud-native. Like we're just like cloud maximalists. Like we went all in and said like basically we're going to build a multi-tenant platform that runs everyone's compute. And the benefits of that are like very tremendous because like we could just do capacity management much better. And that's one of the ways we can offer like instantaneous access to hundreds of GPUs if you need to. Like you can do these like very bursty things and we just give you lots of GPUs, right? I think the other benefit or the other sort of differentiation is to be very general-purpose. We focus on sort of what I think, as I mentioned, like high-code. Like we run custom code in our containers in our infrastructure, which is a harder problem. Like containerization and running user code in a safe way is a hard problem. And then dealing with container cold start, and like I mentioned, we have to build our own scheduler, we have to build our own container runtime, and our own file system to boot containers very quickly. And I think so unlike many other vendors, like they're only focused on say inference or maybe only LLMs, our approach has always been to build a very general-purpose platform. And sort of, you know, in the long run, I hope that that sort of manifestation will be more clear because I think there's many other products we can build on top of this now that we have the compute layer sort of kind of becoming more and more mature.
When I talk to large enterprises about how they're thinking about adoption of AI, many of them already have their data on Azure, GCP, or AWS. They're running their application on it. They've bought credits in the marketplace. They want to spend residual. They've already gone through security reviews. You know, they've kind of done a lot, and they worry about things like latency or pings out to other third-party services versus just running on their own existing cloud provider or their hyperscaler that they work with, or set of hyperscalers. You know, many of them actually work across multiple. How do you think about that in the context of Modal, in terms of your own compute versus hyperscalers versus, you know, the ability to run anywhere?
Yeah, totally. And of course, there's also sort of security compliance aspect of this. Like I think, you know, it is a challenge. I look back at when the cloud came, and I remember back in like 2008, 2009, and the cloud came, and my first reaction was like, how the hell, like why would anyone put their computer in someone else's computer and like run that? And I think, you know, to me that was just like insane, like why would anyone do that? But over the next couple years, I realized like actually kind of makes a lot of sense. And I think now even like among enterprise companies, like there's a lot of recognition that like, yeah, actually probably our computer is more safe in the big hyperscalers. And in a similar vein, I remember talking to Snowflake back in say 2012 or something like that, and they had a sort of similar approach where like they basically said like we're going to run databases in the cloud and it's not going to be in your environment, you know, or maybe in your environment, but like we're in infrastructure as a service. And I thought that was nuts. And then obviously like I think Snowflake now is a very large, you know, public integrated company. I think they showed that like infrastructure as a service makes a lot of sense. And so I think there is a little bit of resistance to adopting this like multi-tenant model, but I think, you know, when you look at like security and adoption of cloud, I think we have a lot of tailwinds blowing in our direction. I think security is moving away from sort of network layer into application layer. I think bandwidth costs are coming down. I think there's a lot of tricks you can do to minimize bandwidth transfer costs. You can store data in like R2, for instance, which has zero egress fees. It's something that I think is realistically going to, you know, mean we're going to have to push a lot, but I think there's so many benefits of this multi-tenant model in terms of capacity management that to me it is very clearly like a big part of the future of AI is like running a big pool of compute and slicing it very dynamically.
You mentioned earlier that one of the things that really caused early adoption of Modal was Stable Diffusion and sort of these open source models around image gen. Are there any open source projects or models that you're seeing be very popular in recent days or in the last couple months that have really started taking off?
That's a good question. I think if anything, it's actually been a little bit of a shift towards more like proprietary models, but like proprietary open source models, I guess. So like Flux, I think most recently has been, you know, a model that's getting a lot of attention. I'm personally very interested in like audio. I think audio is like very underexplored. I think there's a lot of opportunity for open source models in that space, but I don't think we've seen anything really cool yet.
What else do you think is missing in the world today in terms of AI infrastructure or infrastructure as a service?
So I mean, I'm very biased, but I think Modal is missing, like basically a way for engineers to take code and run it. Look, I'm very bullish on like, you know, code and like people wanting to write code and building stuff themselves. I think outside of sort of LLM space, which is like a very kind of different world in my opinion, I think there's always going to be a lot of applications where people want to train their own models, they want to run their own models, or at least like run other models but have like very custom workflows. And I just don't think there's been a great way to do that. It's like pretty painful to do that. And so I think that's pretty exciting. I think on the storage side, there's some other really exciting stuff. Like we haven't really touched storage at Modal. Like we focus very much on compute. So I'm personally very interested in sort of vector databases, like how's that going to evolve. I don't think anyone really knows. I'm pretty interested in like, you know, more efficient storage around training data. I'm also interested in like, I guess another thing I'm very fascinated by right now is training workloads. In order to train large models efficiently, you have to really spend a lot of money and time setting up the networking. So one of the things I'm really excited about is what if you don't, you know, what if we can make training less bandwidth-hungry? Because I think that would actually change a lot of the infrastructure on training, where you can now like kind of tie together a lot of GPUs in different data centers and not have to, you know, have this like very large data centers with like, you know, InfiniBand and stuff. So that's like another sort of infrastructure thing I'm looking forward to see more development on.
