Erik Bernhardsson, Modal CEO on AI Infra, Data Compute & Scaling GPUs

Hi. Thanks for having me.

Yeah. Yeah, likewise.

Yeah, I think in particular I learned most of it through Spotify. That was my sort of formative experience. I was there for 7 years and I started straight out of school and my sort of spent, you know, I was number 40 when we joined and I think we're 2,000 when we left. So, that was really kind of where I cut my teeth, like building systems, like hiring people, building teams, like learning how to work with other people. Um, and it was a crazy startup growing very fast and so now I consider myself just like a startup degenerate. I don't think I could ever work at a big company. Uh, I sort of wouldn't even know how to do it.

Yeah, so I joined Spotify to build a music recommendation system and eventually like, you know, basically like what we realized like after I joined was like we needed more help with like business intelligence. And then but eventually I came back to music recommendations and built it. The way it works is it is basically like that the idea is like called collaborative filtering and the idea is like build a bunch of algorithms that have no idea what, you know, what music is about. Like, we're just looking at trying to find pattern in what people consume. We look at consumption patterns and there's a few different ways you can do it. Like, one way you can do it is like you could look at a playlist and you could look at like what are tracks in those playlists and you can almost like think about it as like a LLM or like a language model task, like NLP task, like natural language processing where like you have a playlist and then every track in that playlist is like a word and you basically try to predict like what's the next track in that playlist and learn a statistical model based on that. Although you don't care about the words really. Like there's no correspondence. Like every track is just its own like unique token. Uh, and by doing that you end up like basically like learning a bunch of vector representations for every track and every playlist and every user and every artist. Uh, and then you can, um, and it's like a high-dimensional vector. Not as high-dimensional as many LLMs. Like we used 40 dimensions at Spotify. It's like kind of enough. Uh, and then you can look in this space, this 40-dimensional space, and, um, you can, uh, look for like nearest neighbors basically. And that's so you can search the space for like, it's basically like the idea is like if two points are close in the space, uh, they're probably very similar.

Not at all. I mean, I learned a lot at Spotify, more about like building consumer products and I think that was very helpful in building teams and all kinds of stuff. Uh, and I don't know. Better was like kind of, I wanted to try something new and it was very different and I ended up staying there six years cuz the company was growing like crazy. Then I left and like right after I left it sort of collapsed a little bit, but now it kind of came back more recently. Um, so it was a very different experience. In hindsight I don't know if like it was like the perfect place for me, but I learned an enormous amount of stuff and was very useful in terms of, you know, building culture and like building teams and growing very fast and raising money and board meetings and all kinds of other stuff.

Um, yeah. Yeah, I started building Modal because a lot of what I realized about Spotify, about the infrastructure was that like as I was trying to build all these complex applications, I ended up spending way more time on just like building the underlying infrastructure. In particular, actually at Spotify I built this open source framework called Luigi for like workflow scheduling, but I also built a lot of other stuff. And it felt like kind of weird. I was like spending way more time on like the underlying infrastructure than the actual like building the applications on top. So at some point I realized, and this is like, you know, when I started Modal like many years later, that, you know, someone should just build that infrastructure, right? Like all these companies, Spotify's like there's nothing unique to Spotify about this infrastructure. They just want to run like big machine learning or AI or data jobs. And, you know, it's just kind of annoying to do that. So someone should just build this like infrastructure layer that lets people run stuff in the cloud and not have to think about, you know, provisioning, load balancing, scaling, containerization, all that stuff. Uh, so we built that.

Well, I mean I think a lot of what I realized is like a lot of the traditional infrastructure that existed at that time, Kubernetes or going back even further is like things like Hadoop or whatever. A lot of that infrastructure wasn't just, it really wasn't well suited for these new types of AI workloads. You know, when you start to have to deal with large models and GPUs and, uh, you know, more like fast feedback loops for developers. And so what we realized is like you kind of need to build something new, right? You know, and that's when we ended up spending two years almost in a cave like just building all this like foundational technology. Uh, our own file system and our own container runtime and our scheduler and all these things. Uh, and it was really to serve, you know, what we felt like was a big gap in the market.

No, it's all good.

Um, yeah, so I mean, if you look at like Kubernetes for instance, like a lot of it is built for back-end applications. So it's more like kind of CRUD apps, you know, apps mostly waiting for databases or coordinating with other systems and it works pretty well for that kind of stuff. Like developers can like test those locally pretty well and then when they're done, then they deploy it on Kubernetes and it runs in the cloud and stuff like that. When you're developing AI apps, like first of all, you need GPUs and so most developers don't have GPUs locally, so they need to run that in the cloud somehow. And like using Kubernetes for that is just kind of annoying. You have to build a Docker container, push it to the cloud, run it in the cloud, get results. Like it's just very slow feedback loops. Uh, so that was like the biggest thing we, like the big problem that we started focus on was just like, how do we take code on a developer's laptop and like run it in the cloud, you know, and do that in a way where it feels like you're running things locally. Cuz if you can get the cloud experience to feel like you're developing locally, then you solve a big part of the problem with, you know, building software and running it in the cloud. Later we realized that this is also extremely valuable if you're doing things like inference cuz if you can figure out a way to start containers very fast, and it's kind of the same problem, uh, then you can scale up and down very fast, too. And so you build a serverless system that, you know, can, um, uh, that I think has a lot of benefits, especially when you're dealing with these like very expensive GPUs. Uh, so those are some of the inspirations for why we decided you kind of need to start over. You can't take the existing stuff and build some of the stuff on top of it. You kind of need to start over and build a whole new infra stack.

Yeah, I think there's a couple of things that are different from us. I mean, first of all, we're different than the cloud vendors, right? Like we're kind of one layer up. Like we use a lot of different cloud vendors. We use AWS and GCP and Azure and Oracle and many other ones. And they're great at like physical infrastructure, but they're not really, I think of there's like a higher, that's something, there's a gap higher up in the stack. And I think they recognize there's a lot of opportunity to innovate on that. And so, you know, we consider a lot of them partners. And, you know, we might use other cloud providers in the future. Uh, then I think, you know, on the higher level stack, like I think there's like a couple things that differentiate us from other vendors. Uh, so, I mean, first of all, one thing is like we're very multi-purpose. Like we're very, you know, we don't just power LLM inference, we're not just inference, we also power training, fine-tuning, batch processing, code execution, and many more things to come. Like we have storage primitives and, you know, networking primitives and all these other stuff. So we aim to be, you know, way more of a general-purpose system that you can compose a lot of different building blocks. So it's basically very general. Uh, the other thing is developer experience. I should have started with that cuz that's actually the biggest thing is we have way better developer experience. Like we really spend our, you know, an enormous amount of time like obsessing over Python SDK and making it easy for developers to iterate fast and, you know, write natural code. And then the third thing is like we spent so much time innovating in the core infrastructure layer, like building all these like low-level primitives. And I frankly, when I look at like anyone else in the landscape today, there's no one else that like goes as deep as we do and kind of rethink, you know, container infrastructure and file systems and all kinds of stuff like that.

Thank you.

I mean there's no doubt that a lot of it ends up in, you know, Jensen Huang's pockets eventually. Uh, if you think about like our, like where we are in the stack, uh, I mean we're not a software margin business, but we have like relatively okay margins. Um, I mean we buy GPUs, right? Or like we rent GPUs. We don't actually buy them. We just rent them. And then, you know, in a way like we add a lot of software on top of it and that enables us to charge more for the customers. So our customers are application developers, right? Uh, and so our margins aren't software margins, but they're like quite okay, right? And then, and so when you think about the money we raised, it's only just going to go towards GPUs cuz that's what our GPUs are revenue for. So the revenue is what we use to pay the cost, that's the cost, right? And I think that's quite important. You know, you can't, you know, um, fudge, you know, I think there's maybe companies out there like, you know, using that, the, you know, raising money and then buying GPUs and that's okay if you maybe trading a model cuz that's a capital expenditure then. But for us like, you know, running GPUs, that's the cost of goods and so it goes into our margin. So, you know, and since we have positive margins, it's our customer revenue that funds the GPUs. Whatever we use the money for, it's primarily labor. I mean that's the thing that costs a lot of money. Like hiring engineers, building out a world-class go-to-market team, then of course a little bit of other stuff like office space and whatever. Uh, but the big thing is really humans and, uh, you know, building an engineering team and, and that's, we think that's the right thing to do. Like we think it's fine to, uh, burn money, uh, in the interest of, you know, investing in R&D, uh, we think those will pay amazing dividends over the next, you know, whatever 20 years. And so that's fundamentally, I mean, I think not different than any other VC-funded startup is we have positive gross margins, but we're burning money overall, but we're also investing a lot in R&D and hiring a lot of engineers and we're building some amazing features that we think will help us get, you know, the next few customers.

Yeah, I think maybe dichotomizing a little bit, but like we found it like kind of useful to think in terms of like two different profiles. Like, one is sometimes we just go and hire like super, you know, cracked people like straight out of school. Like, just like super smart people that are like quite, maybe quite junior in their career. Like, we don't really care. Like, they're just like smart and like able to write a lot of code. And then on the other side of the spectrum, sometimes we need like people who really understand something very specific, you know, right around like distributed systems or Linux kernels or GPU performance. And in that case, we hire a little bit more like senior. We're talking like 5, 10, 15 years experience. Uh, in practice, it's a little bit more complex. There's other roles, too, but I think those are like the two main buckets of labor that we hire.

A lot of it so far in the last couple years has really been inbound and I think of it as like founder-led marketing. I spent a lot of time in my early days at Modal like just tweeting and talking about the product with people and that was enough to like drive a lot of inbound. Uh, and I think we've gotten to a really good point. I mean, we're doing $60 million in annualized revenue and that's like without almost doing any sort of sales and marketing. We started building out a marketing team over the last year or so and we're still like relatively small. Now we're starting to see a lot of success also with like more like traditional enterprise marketing. So we're building that up very quickly. We're also building up our marketing team. Uh, but I think it's sort of coming from it from a little bit of a different angle. We have like a very solid foundation of product-led growth. And I think it really reflects the product being just like, you know, having amazing developer experience and being very good at like self-service. And we have many examples of customers signing up themselves, putting in a credit card, starting to write code, and like over time spending, you know, hundreds of thousands of dollars a month on Modal or even more. Uh, barely without talking, I mean at some point we reached out to people spending a lot of money. But fundamentally we're talking about people who they didn't need selling in order to use Modal. The selling was like they tried out the product and they liked it. And having that like really strong product-led foundation I think has led us to have like very capital efficient growth and fundamentally this reflects that the product is good.

Well, I think in terms of personas it has to be machine learning engineers. Like people building machine learning applications who need to run models or train models or run big data processing jobs. In terms of companies, I think it kind of reflects a little bit of like the adoption pattern of GenAI. Like initially when we started a couple years ago, started seeing traction, it mostly came from like GenAI native companies. So these are companies like, I don't know, Suno does like AI generated music and similar companies like doing generative media or like things that are like kind of new. Over time we've seen a lot more traction also coming from like established companies. Like, you know, you're talking about, you know, public companies or digital natives or enterprise companies. And I think it reflects the fact that they are now starting to put a lot of this stuff into production. Uh, and that's often when they come to us for help cuz running this stuff in production is hard. Couple other verticals we've done really well is like live coding platforms more recently is like seen a huge surge and we power code execution. Uh, we have like a primitive that sandboxes for running untrusted third-party code. Uh, it works very well for that. Uh, and then another vertical is actually computational biotech where we see a lot of traction right now. We have a lot of customers using us for protein folding or modeling binding affinities or some really cool stuff. I mean like we're talking about like developing drugs that will save people's lives and cure cancer or whatever. You know, it's like kind of a fun, you know, not as fun as, you know, it's awesome to see that as opposed to like a lot of our other customers. Not to cheer you up but there's a lot of sort of face swapping and like, you know, fun kind of stuff. But like there's also the sort of serious stuff. So, we power a very broad swath of very different types of users.

I mean, some of them, yeah, yeah, yeah. Like and, you know, not to say we, I mean we obviously don't have customers spending that much. Like our total revenue right now is maybe $20 million. But like, you know, but it's growing extremely fast and like I think over time like we should be able to capture a lot of those customers. There's an enormous amount of, there's an enormous willingness to spend, you know, if you're developing new drugs and you can, you know, use computers for something maybe traditionally you'd have to use clinical trials for or, you know, physical, you'd running stuff in a lab. Now you can simulate using computers and so there's a lot of willingness to spend money on this. There's a lot of money in drugs.

It's a mix. I mean, some of them have come inbound, some of them have been outbound. Um, I mean, I think that's like something where we're starting to see a lot of traction in that space now. And it sort of reflects that these companies are now putting a lot of stuff in production and wanting to scale up. And so, yeah, that's why we're investing very hard right now in an enterprise go-to-market team. And that wasn't necessarily the case two years ago, but now we feel like, you know, it's time to. And that's part of why we're raising money, too, is, you know, these deals, they take a lot more hand-holding. Like, you need to deal with infosec questionnaires and, you know, understanding org structures and spending time like, you know, uh, so doing solutions engineering or other things. Uh, but on the other hand, there's way more upside also in those deals, we think.

Yeah. Yeah, I mean, a lot of it has been bottoms-up at the smaller companies, of course. And I mean, or like honestly, I think the bottoms-up and top-down is kind of the same thing cuz like typically, what you have at these companies is like you have a CTO who's still writing code, also has a credit card. So, the buyer and the, you know, decision maker and the end user is the same person. At a later stage company, those are maybe three different people, so it's a little bit of a different thing. Uh, in that case, you know, often the machine learning engineers, they might be the champion, but separately you have a platform team who's maybe more of the like end user like in the sense that like they will like bring that, you know, maintain the system. And then you have of course the CTO who's maybe overseeing everything and then there's like separate like a security team that has to approve everything. Uh, and so you need to spend time with all of those personas and make sure they feel good about Modal.

Uh, I wish, I wish it was that much time on building and selling. I feel like as a CEO I end up spending also a lot of other time. I mean like something like recruiting, right? Like, you know, I spend a lot of time recruiting people. Like I think it's so important to spend time like cuz like humans are the root of everything. Recruiting is a big one. What about the, the time like generally involved in like the system. I almost think of it as like, I mean it's a very technical product so it's like it's basically a product, you know, looking at the behavior of the system, looking at the SDK, like trying to understand, you know, from a user behavior point of view like how it works. I do spend a lot of time selling. Spend a lot of time like talking to customers. Um, spend a fair amount of time also in like investor relationships and like, you know, there's always, there's always like, you know, the existing round but then there's the next round and, you know, building up like proactively like relationships with investors and stuff like that. And of course there's always like random like legal stuff like, you know, I spend a lot, like spend too much time like optimizing like option stuff and taxes for the employees. Like something I spend way too much time on. Uh, we really need to hire a general counsel I think over time. Um, what else do I spend time on? I'll spend a little bit of time on like cost structure, like understanding our cost and how we can optimize our cost. But yeah. Yeah, I don't know. I mean, like I don't know. Unloading the dishwasher. Like I don't know. There's just so much stuff in a startup. Like someone has to do it and like I don't know. I feel like, you know, if you're not willing to jump in and do almost anything at a startup, like you probably shouldn't be the CEO. I think that's the beauty of the startup also. It's like, you know, I like, everyone kind of jumps in and is ready to do almost anything. Whatever it takes to make the company successful.

Yeah, so first of all, on pricing, I think it's actually relatively straightforward. I mean, our underlying cost is like per GPU hour or like per GPU second. And so I think it makes sense just from like a cost alignment perspective to manifest that to customers. I think the other thing is like that's just what customers are used to cuz that's what everyone else uses. So the fact that we're like, you know, usage-based, I think it's kind of natural function of just being in the same space. We are a little bit different. We're serverless and so I think like one thing we kind of struggle with is just like maybe explaining to people like, well, you know, these are like you only pay for the time the GPU is actually running as we scale up and down. You never pay for idle capacity. You don't have to make reservations, right? Like, you know, and then most people understand that there's a lot of differentiation and it's worth paying extra. We often talk about like our, you know, effective cost is often much lower even though our list price might be a little bit higher. Um, so, uh, and then so all the other stuff I think is, um, you know, the pricing structure. We do have team plans and we have that and other stuff. I think that's going to evolve over time. Uh, I think we're probably going to figure out like, you know, other ways to charge people. But like, you know, when you're talking about things like our back or audit trails or logging and stuff like that, a lot of that is harder to charge on a usage base, you know, in a usage-based way. So my feeling is like you kind of have to just bundle all of that and sell it as like an enterprise tier and then you have to charge it in some other way and that's, you know, usually like a team cost or per seat base or something like that. Um, what was the other question you were asking? You were asking, uh...

Yeah. Yeah, I think that's hard. I think...

I think, I think like the end state profit margins, if you look at, you know, state of the art cloud vendors, like you can get it to like 50%, you know, and that's where like EC2 is for instance in AWS. And so, you know, we see a lot of ways we can optimize our margins. Like, and like fundamentally, we're not going to have software margins on this business. Like that's just not who we are. Um, but we think we can get the steady state margins to a 40, 50, maybe 60% just on the compute itself. And then, you know, much like AWS has EC2 and many other services, we also think there's going to be many opportunities to build a lot of stuff on top of it. And we already have things like notebooks and other things. So, we think sort of the, you know, compound margins, like the blended average of all of that, we think we can get it to I don't know, 60%, you know, it is maybe even higher if, you know, in the very long run. Oh, you asked me something about, uh, the terms of the revenue. Basically, we have very low working capital needs cuz almost all of our costs are like on a, you know, net 30 basis. And we also, uh, you know, our customers pay on a net 30 basis. So, so there's very little working capital requirement in this business.

Uh, I mean, there's so many things here that I don't know if anything is missing. Um, so I don't know, man. I think there's just so many... I'm trying to pull up like a similar one. If you ever see the... Am I able to share?

Uh, okay, I have to go to system setting. No, never mind. I was going to share the map a16z publishes every year. Someone said this earlier, too. So...

Yeah. Thousands of boxes. It's like 20 pages, basically. I mean, like I feel like if anything there's like too much stuff here. Like everyone's like trying to solve like point solutions. And that's like sort of my, like, you know, my gripe with... Oh, yeah, there you go. Like that's my challenge with the system, with the whole thing is like everyone's just like solving like a thing, but like they're not going after the bigger issues. Like they're trying to replace like a point, you know, solution with another point solution. I kind of feel like maybe we're like crazy, but like we're trying to rip out the entire infra stack and replace, you know, AWS. Yeah, no, I'm not AWS, but like we're not, we're trying to replace like, you know, 50 of these logos with Modal. Like I don't know. Or, you know, I mean, not replace cuz they're probably going to still be around, but like we're trying to do, you know, 100 of these things, right? Like we're not trying to do like one of these things.

I'm sorry, what do you mean? Like in terms of like...

Oh, I think that's almost like, that's incredibly hard cuz you can only do that much right now. Right? Like we're not AWS. We don't have like, you know, 50,000 engineers. We have 50...

So the sequence here is very hard, right? But like we started out building like an underlying, you know, compute layer. And I think that's increasingly more and more mature. There's a lot of stuff still to be done. We're working a lot on lowering latency, for instance. But like having kind of put that in place, I think there's a lot of opportunity now to kind of do a lot of high stuff up in the stack.

So, I think, yeah, I mean, you know, like in the long run, like we always wanted to cover the entire AI life cycle, right? Like all the way from like, okay, I want to train a model, then you have to like first get the data, then you have to, you know, run a bunch of data prep on that data, then you have to train a model, then you have to deploy that model, and then you have to deal with, you know, the model in production. So, you want like observability, you need feature stores, you need a, you know, so that's just like one thing, right? Like that sort of end-to-end like AI stuff. And then there's a bunch of stuff around it like experiment tracking, notebooks, like, you know, data munging, like analyzing large data sets, like querying stuff, you know, real-time streaming.

Like I think, and the benefit of like the approach we're taking is like I think it's very strong integration advantages of like, you know, I hate the word synergies, but like that's basically what it is, right? Like building A and B, you actually get like way more value than like A and B in isolation because you have like very strong integration points, especially if you own the compute layer. And that's I think, you know, something a lot of startups don't. It's like because they don't have the compute layer to monetize. So, we can build all these other, you know, all these products and maybe some of them, you know, doesn't actually drive revenue, but it drives revenue indirectly through the compute layers. So, like I think that's the benefit of having this like kind of platform approach where, you know, you make money through the compute, but then you build all these other stuff and it all sort of, you know, becomes this like multi-product platform that like really solves all your AI application concerns.

You hear us. I think in order to be, you know, successful, you have to decide pretty early on like a persona and like focus on that. And our persona is very much developers. So, we're proudly high-code. And I think it's, you know, I think a lot of people get pulled into sort of a no-code direction. But if you look at actual AI spend, it's like mostly controlled by engineers in the end. That's not entirely true. I mean, there are like very successful consumer products. I mean, ChatGPT being an obvious example, right? But, you know, and we're probably not going to power ChatGPT, but like we might power like some a lot of the chat applications in the future. So, you know, we're going to be that. My view is like we're going to be there somewhere in the stack and there's going to be developers deploy stuff on us and, you know, of course somewhere else there's going to be consumers using the product because you need an end user. But we want to focus on the developers because at the end of the day they tend to control most of the spend when it comes to AI or infrastructure as a whole.