Hey everybody, welcome back to the New York Stock Exchange. We're here in the Buttonwood podium overlooking the options exchange. My name is Dave Vellante and you're watching the NYSE Wired and theCUBE's Mixture of Experts series. Erik Bernhardsson is here. He's the CEO and co-founder of Modal, a company that does serverless compute and really cloud innovator, startup. Super psyched to have you here, Erik.
Yeah, it's great to be here. Thanks a lot for having me.
You're very welcome. Why did you start the company? What's Modal all about?
Yeah, it's a long story, but I think the shorter version is I have built a lot of AI products in my career so far. In particular, I was at Spotify for many years and built the music recommendation system and realized at some point that a lot of traditional infrastructure just isn't built for AI and for a lot of these newer types of applications. And with infrastructure, I mean this sort of software layer sort of sitting under the applications. You have the applications, and this is what the machine learning engineers are building, and these could be things like training models, or deploying models, or building vibe coding platforms, or computational biotech, or other things. They only need infrastructure, right? So that infrastructure layer today is often built for very different types of applications. So we realized at some point we should just build a whole new layer that is more geared towards these types of new applications and rethink a lot of the assumptions and focus on the needs of these new applications in terms of scaling up and down, or working with large models, working with GPU capacity all over the world, et cetera. So that's why we started Modal.
So interesting. I mean, I never studied Spotify deeply, although as a consumer and a user of the service, I always felt like there was some really smart people behind the scenes. You were obviously one of them building what I refer to as data products. I mean, essentially, you had data sets and you would serve up services to consumers like myself that were really well-thought-out, and the infrastructure underneath was something that you probably didn't get off the shelf, as you had to build this, right? Is that what you're implying? And then now what you're doing is bringing that to the masses? Is that...
No, that's exactly right. So a big part of what Spotify focused on was, how do we make it easy for people to know what to play? You have 10 million tracks, where do you start? And so my team specifically focused a lot on music recommendations and built a lot of features around that. And my experience building those products was because you didn't have the right infrastructure, you ended up spending way more time building the infrastructure than actually spending time building the applications. Then I realized, 'Someone should just build this layer.' And I'm sure Spotify isn't the only company, so I'm sure there's 10,000 other companies out there who also need this infrastructure layer. And that's what turned out to be quite true. We realized so far there's a lot of companies out there who need better infrastructure, especially now that we're switching to GPUs and very large models and LLMs and all these things. You especially need more infrastructure.
So there are a lot of recommendation engines out there and a lot of really crappy ones. You think about retail, a lot of times, you would get these kind of conditioned response type of experiences, which were terrible. You got four choices. Which one are you? None of the above. Spotify obviously, and TikTok obviously have very strong, whether it's algorithms or whether it's infrastructure. But now, we're entering sort of a new era beyond things like recommendation engines where we're doing so many more interesting things with AI. What's your vision in terms of supporting some of those new applications and capabilities?
Yeah. I mean, first of all, it's all switching to GPUs. We started playing around with GPUs in 2014 at Spotify, which was quite early, but most of this stuff wasn't running on GPUs, and I don't know if it's still running. I left a while ago, so that's a big change. And then the models have gotten very big, so that's another big change is we're working with these very large LLMs these days. There's obviously generated media, which kind of completely changes the game. So you have algorithms that can generate music these days, generate video, generate images. I mean, all these things are just a complete game changer for a lot of companies. And we're starting to see that. We're starting to see a lot of companies adopting these things. Initially, we saw a lot of early stage companies adopting these things, but what we see today is a lot of demand is also coming from later stage, sometimes public companies, to try to deploy these things, and that's when they often come to us, because they need help with that.
So interesting. So 2014, you had experience with GPUs. So I think that kind of brings you into the CUDA era and the programming of those GPUs. And again, I come back to this notion of mainstream. So our vision is, we wrote a piece a while ago, why Jamie Dimon is Sam Altman's biggest competitor. The thinking there was there's all this wonderful proprietary data sitting inside of places like JPMC. We use that as a metaphor for enterprises, and we've also written about Uber for all. Uber had to build its own infrastructure like Spotify had to build its own infrastructure. So Uber for all is, again, a metaphor for solutions or capabilities that allow enterprises to actually build this stuff much more cheaply and faster without having to hire thousands of engineers. Is that what you're bringing to the table? And what other piece parts do you see emerging? Or is it part of your TAM expansion strategy to build those other piece parts?
Yeah, I mean, Modal focuses specifically on these machine learning problems. So our consumers are people who build the models, but I think there's a larger trend here, which you're talking about, which is I almost think of every company becoming a tech company, and also the tech companies becoming just normal companies. Spotify, for instance, today, are they a music company, are they a tech company? I don't know. It almost doesn't matter, right? They started out being very much a tech company, but the truth is thousands of other companies had similar problems like music distribution and working with large data sets, and those used to be unique things that Spotify had to solve in an idiosyncratic way in their own ways. But today, I think those are just products you can buy from a lot of vendors. And so I think there's an emergence of so many different infrastructure vendors and so many different tools, and I think there's also ... You look at the existing companies that previously weren't tech companies, they're buying all these vendors. So I think it's a normalization of tech, and I think you're going to see the same thing with AI, who's an AI company, who's not an AI company. I don't know. Everyone is an AI company. Everyone is going to need a lot of these things. Look at traditional companies, like you mentioned JP Morgan, how many engineers do they have? 100,000. Absurd number. They're going to need a lot. They're going to have to also buy a lot of this infrastructure. And so there's a role for many different parts of that stack, and we play into one of the lower layers.
We focus a lot on running code, but there's many opportunities also higher up in the stack simplifying how to work with AI. And maybe you don't even have to think about the models and maybe you can just point and click. We're not a point-and-click system, we're not a low-code or no-code system. We're very much a high-code system. We focus on the software engineers, but it's all part of the same trend, I think, of the democratization or broadening the footprint of technology and AI and the cloud and maybe other things.
But your basic premise is you shouldn't have to have a PhD in AI infrastructure in order to deploy AI. You're making that simpler, even though it's full stack.
Yeah. No, I think that's right.
You're targeting... So what is broken about today's model generally and maybe the cloud model specifically?
Yeah, yeah. And by the way, I will say that PhDs are actually our best customers, even if they are among the few people who understand the cost of this stuff, they're also tired of bad infrastructure. So we sell to the whole spectrum of machine learning PhDs all the way to just any sort of developer.
But the point is you don't want them wasting their time building infrastructure.
Exactly, exactly. They should focus on what they're good at, which is to build amazing models. Right? So going back to your question, what's broken with today's infrastructure? A lot of today's infrastructure is built for very different applications, built for backend applications. So it means you have kind of smooth workloads. They're very IO-bound in the sense that they often just coordinate work between many other systems like maybe databases, et cetera. What I'm talking about are things just to drop some names here like Kubernetes or Docker, et cetera. And that's been the dominant development paradigm and the dominant infrastructure for the last 10 years or so. But when you start to look at a lot of these new applications, you run into a lot of different problems. First of all, you have a lot of GPU capacity, or actually I should say GPU scarcity. So you have different regions all over the world with different capacity, and often when you start to scale up, you run into constraints. Maybe you run out of GPUs in one region, so you have to go to a different region, or you need to run things close to you so you have certain latency constraints because speed of light is a real thing for a lot of real-time applications.
Another thing is these models are very big. So building applications that have to load up these models, load them into the GPU and do that fast and scale up reactively to user workloads, you need to really think deeply about how do you build infrastructure that can do that, especially when you look at inference, and we're very focused on inference historically. We've had a lot of use cases in inference. And when you're doing inference, fundamentally, you don't know what the user demand is. The user demand goes up and down, and so scaling up and down with these workloads is a real challenge. And so we basically decided at some point in order to support a lot of those things, we had to build basically our own stack. We decided we can't build a new stack on top of an old stack, so let's throw out the old stack, let's build our own file system, let's build our own container runtime, let's build our own scheduler and many other things. And we had to do that in order to be able to support these new types of applications.
The other thing I think is very broken also is developer experience. So it wasn't as much of a problem in the old paradigm, but when you're starting to look at these AI researchers and how they're operating and how they're trying to leverage the old infrastructure, they're running into a lot of challenges. Just being able to take models and get it out in production and putting it in front of users, it's a real challenge for these developers. So we spent a lot of time super obsessed with like, how do we make it fun for engineers? How do we make them fast? How do we really improve the speed to market? Because I think that's almost as important as, or more important maybe than everything else. There are so many companies out there where speed to market is really the thing that matters. So by making engineers 10x more productive, you're also gaining a massive advantage.
So let me ask you, follow up on that. So if I think about the cloud today, I've got access to all these primitives and services through APIs.
I got Elastic Beanstalk, I've got Kubernetes, and you're saying this is just too complicated today. It requires too much brute force to actually scale, and you're orchestrating that in a much simpler way and streamlining that experience.
Yeah, that's exactly right. So first of all, we're not trying to compete with the cloud vendors. We are customers of the cloud vendors.
Yeah, you're running on the cloud.
Yeah, we're running on the cloud and we're super happy. We consider them partners. We use a lot of different cloud vendors, AWS, GCP, et cetera, Oracle and Azure and many others. They're very good at delivering physical data, like running physical data centers and delivering compute capacity through an API.
You need compute networking there.
Exactly. The lower stuff, right?
What I think there's a gap is slightly higher up in the stack, and that's really most visible in developer experience, like working and building these AI applications. You can build them straight on the clouds, but it's pretty hard. And even if it's hard, I mean I know how to do it. It's also annoying. It takes a lot of work. Because we think a lot of the primitives that they offer are somewhat crude, and you have to spend a lot of time configuring them and putting them together. And so I don't think that the right building blocks are there. So we think that there's a massive opportunity slightly higher up in the stack to weave together all that physical capacity and offer a different set of building blocks that is more geared towards these new applications. And by doing that, we can let engineers iterate much faster and not have to worry about a lot of the underlying physical infrastructure.
So you would use a Modal container versus Kubernetes, is that right?
That's right. Yeah. We replace Kubernetes from...
Compare the two. To a lay person, what's the difference in terms of the developer experience?
Yeah, yeah. So the number one thing is, like you said, the developer experience and the ability to iterate. So one of the really cool things we can do is when you're writing code in Modal, you can iterate almost as if it's like running locally. So you can have code on your local computer and you can write it, and then you can hit enter, and we can trigger that code running in the cloud within about a second. So we have that superfast feedback loop. Whereas traditionally, you would have to build a container, push that container to the cloud, trigger that container, go download logs. It is a very slow set of feedback loop, many minutes or even hours, sometimes days. So the most important thing is having that superfast feedback loop. The other thing is making everything programmable. So everything in Modal is in code. We focus a lot on Python, because that's just how machine learning engineers think. Although we have other SDKs, we offer Go and TypeScript as well. And then kind of rethinking the fundamental primitives. We think more in terms of here's the function, deploy that function into the cloud, and don't have to think about necessarily scaling and containers and all these things. We sort of manage that fully automatically for you. So it's just one tool that handles all of that stuff, gets it into production, handles all the scaling up and down. In a way, using better abstractions so engineers don't have to think about all the containers and writing YAML and configuring things and running a bunch of different commands, and it all kind of just works in one single command.
And you do that function experience, you do that using existing serverless functions from AWS, Google, and Azure, or it's your own...
Okay. So essentially you're using the clouds for the core infrastructure.
The security that they bring around the S3 bucket or whatever it is, and then you build on top of that?
Primarily the compute primitive, so EC2 in case of AWS. Running physical nodes in the cloud, that is the main thing we use. And then everything above that, we basically built our own stack.
Okay. And then your developers would tap whatever other services they need in storage or that.
Exactly. Yeah, we also have built our own storage primitives, and that runs on top of S3 and R2 and Cloudflare. But again, we sort of abstract away the complexity of that. For them, it's just a CLI or a Python SDK.
And you mentioned latency before. Do you run on-prem as well in a hybrid model?
