Great, okay, so I'm Erik Bernhardsson. I'm the CEO and the founder of a company called Modal. We're based here in New York. I've been working at this for a few years now. Like Matt said, I have a background at Spotify, where I built a music recommendation system, among other things. I've done a lot of AI. I was a CTO at a company called Betterment — a different story — but started Modal a number of years ago because I wanted to build better infrastructure for data, AI, machine learning, and in particular, it's like what I wanted to build for is like what I think of as like high code. So this will be a little bit maybe a different presentation because I'll do some live coding to show you what you can do with Modal.
You can think of Modal kind of as like two different things. One is an SDK in Python where you write functions that call each other, and you define the environments and you set up GPUs, etc. And then we run all of it for you in our cloud. So we maintain a very big pool of thousands of GPUs across many different clouds. We manage all the scaling, we manage all the infrastructure for you, so you don't have to think about provisioning, capacity management, etc. And then we charge it in a fully serverless manner. So what that means is we charge you only for the time the container is actually running, which means, you know, you only pay for, you know, effectively utilization, right? You don't pay for idle GPUs.
Customers pick Modal for a few different reasons. Really, the core thing is developer experience. Hopefully, I'm able to get that point across in a couple minutes. I already mentioned usage-based pricing. We have a lot of GPUs. If you need 100 GPUs or even 1,000 GPUs, you can usually get that within minutes or even seconds sometimes, and you don't have to think about infrastructure. So if you want to scale up and run a lot of stuff at a pretty big scale, Suno is an example of a company that runs things at very large scale using Modal. They do AI-generated music, pretty cool. And Substack is also using Modal, a bunch of other companies. Ramp, actually, there's a lot of other applications, but I would say like in general, like we've seen most traction like gen inference, kind of actually less than like LLMs. LLMs is very API-driven. We focus more on people who want to run their own code, so it tends to be more like custom models for image, audio, video, music, etc. We also see a fair amount of use cases in biotech, LLM fine-tuning, and all kinds of other stuff. So pretty wide range. It's a very general-purpose platform. You could run almost anything in Modal.
Okay, so here's a very simple Modal app, and hopefully, you know, actually, like a question: how many of you like writing code actively in Python? Like, so, okay, so this is, you know, hopefully a majority, like this is somewhat familiar syntax. So this is a little function, doesn't do a whole lot, and then we call it basically like there's a function here that's called that returns a square of a number, prints some stuff to standard out, and then we call it with 42. And so if you run this thing in terminal like this, basically what happens is we take this function, we stick it in a container, we execute that container in the cloud, and as it's executing in the cloud, it streams back. And so when I'm running this, I'm not actually running code locally. I'm actually running — I mean, a little bit of code locally — but like the actual container runs in the cloud. And so the idea with Modal is like you can effectively run code in the cloud, but in a way it almost feels like it's local because it's so fast. And that includes something as basic as, you know, so basic as like let's say we want to edit the code and rerun it. Like I can just run it, right? I don't have to like build a Docker container and push that to, you know, AWS and like rerun it and like download logs, whatever. Like, you know, it actually prints the — here you can see it prints the whatever I edited. And this extends to other things.
So let's make it a little bit more interesting. Let's say we want to run this on H100. So we're not going to use this H100, but we're going to run it on a container that has access to an H100. Maybe it takes a little longer because, you know, but we have a big pool of H100s. So effectively now we ran something that has access to an H100, and we can even try to use this H100 if we want to, because Modal lets you install any dependencies or any — you can effectively, you can build any image really. Like you can give us a Dockerfile, we can build that for you, or you can just in Python define in code say I want to install, start with the slim, and then pip install torch, image equals image, and now we can import torch here. torch.cuda.is_available — I think that's the function. Does anyone know? Let's see if this runs. Now we're going to run on — we're going to build this image actually. This might take a few seconds because the torch is actually, I think, a gigabyte large. So I don't know if anyone has like a good joke about GPUs while we're waiting, but this is going to run, and we can build images very quickly in the cloud. We then cache these images so you don't have to rebuild this. If I run this the second time, like this whole, you know, part where it's outputting a bunch of yellow text is not going to have to run. I'll show that in a second what it looks like.
And but yeah, as I mentioned, you can give us a Dockerfile. You can also start with an image that exists in Docker Hub. You can install drivers, you can install binary packages, whatever, right? So we basically let you do anything with these images. And very large images can be built and can be used very efficiently in Modal. I'll talk about that in a second, but we built our own file system, among other things, in order to store these images very efficiently and also boot these images very fast. I can maybe show you something else. Sorry about that. Let's install numpy instead. numpy is much smaller. So the idea is you can, yeah, again, like we'll install numpy, we'll just build this and install it, and hopefully this will take a few seconds at most. See, six seconds, much faster. Thank you all. Right now we're running this container, we're importing numpy in it, and if you run it again, we're not going to reinstall numpy. We're running this, you know, just booting up immediately and running this.
Sorry, I'm going to run it again. Let's make it a little bit more interesting. Let's actually do — Modal lets you basically do sort of batch fan-out parallelism. So let's say, you know, we have this function that computes the square of a number, not super interesting, but you could imagine doing something more compute-intensive. Let's say we want to batch over a bunch of inputs, so we want to map over 10,000 inputs, and remember we're running these on H100s. So, and in fact, let's slow it down a little bit just to — let's run this. Now we're going to fan out and parallelize over 10,000 inputs, and we're going to execute those on H100s. And so, you know, we're already firing up a bunch of H100 containers. We're already running 22, 25 containers. We're processing a bunch of these outputs. So a very common use case for Modal is to do sort of like data prep or like batch processing and fan out and run like very compute-intensive operations, video transcoding or FFTs or things like that over very large datasets using, you know, a large pool of GPUs or other things. Running 70 H100s right now. This will cost us about a few hundred bucks an hour, so I'll kill this.
You can do a lot of other stuff with Modal. So just to give you one idea, you can take these things and you can deploy them, and, you know, if you do that now, you can actually — sorry, not function — we can look them up from any Python process. So we can take this function and look it up from this Python process, and then we can call this, and it's not doing a whole lot. It's just sleeping for a second and returning a square of a number. But this, for instance, you can basically like deploy these functions and use that for inference. So that's a very common use case. You can have these, you know, complex models. But I mean, if this is a little abstract, I mean, like, you know, just to show you like what people can — you know, we built this little demo the other day just to give you an idea of like things you can run on Modal. So here's like maybe a more visual thing. You know, that maybe here's basically like Stable Diffusion SDXL Turbo running on Modal. So the idea, like, and this is just a few lines of code. There's basically like a Python shell around this GPU model.
Yeah, we built — in order to do this, we built our own container runtime, we built our own scheduler, we built our own file system, built our own container. We're basically spending a lot of time in the infra layer at Modal, and which is something most people don't like, but we love that stuff. We're obsessed with it. We spent basically the last three years like optimizing container cold start across different regions, and I'm not going to go into like deep details, but basically if you like look at what Docker does, it's incredibly inefficient. There's a lot of ways to optimize it, which we did. We eventually got rid of Docker, we eventually threw out Kubernetes and started over and built all that stuff ourselves. There's all these tricks you can do basically building, you know, using FUSE or using what's called a content-addressed file system in order to optimize image retrieval and storage. We maintain a very large pool. So this is just one example. We run AWS, we run GCP, we run Oracle, and a bunch of alternative clouds. We run, I think, 40, 50 different regions, many different GPU types. We solve a mixed-integer programming problem every few seconds, every minute or so, to basically like size our pool up and down in order to match supply and demand, because we have, you know, a lot of different users running a lot of different things, and so we need to fulfill their capacity demands.
Yeah, pip install modal, and you get, you know, $30 free credits to play around with it. Very easy to get started. If you're familiar with Python, you can deploy things and run GPUs in the cloud. If you're a startup, we have credits programs, so you can get 10,000 or sometimes even 25,000 credits. Cool, awesome. Seven seconds left. Thank you so much.
