Welcome to Runtime 2026. It's pretty cool to see all of you here. Got a really cool lineup. Got some really good sessions. Going to talk a lot about this new thing called AI. We're going to talk about infrastructure and some really cool applications. And there's some insane, cool builders in this room. Some people that I'm obviously like, every day I feel like kind of grateful to work with some people like you building some really cool applications, hopefully on top of Modal. And we're going to see some really cool talks about computational bio, robotics, how to buy and sell GPUs, math, and a lot of other stuff.
And so hopefully this is a time for you to meet some new people, learn some new stuff, maybe meet some old friends, and really sort of share some notes about, I think there's a lot of stuff going on in this world, and I think you all feel it. There's like a whole new set of capabilities of building with software. And hopefully you get some takeaways from how to get that. So I wanted to start off with talking a bit about some of these applications and how I think it's very clear in the last few years we've seen some incredible new capabilities, a lot of it built by people in this room. So I'll start by showing some of those things and talk a bit about what's going on.
So you got companies like Runway doing things like generative video, like basically generating videos that look real. Companies like Ramp, and Cognition, and Nightshift and many other ones, really changing how we build code, how we build applications and how we write code, and I think you all feel it. Like, the craft of writing software has obviously changed in many different ways in the last few years. Robotics. Got some incredibly cool robots that can do things like folding laundry and walk and do other cool things. So, with AI, we've helped, you know, could train models that start to get to human level dexterity and all these things. Companies like Oura helping us live healthier lives.
And speaking of health, a bunch of companies like Chai and Achira and other ones using AI and computational methods to discover new drugs. And I genuinely look forward to taking AI-generated drugs, actually. Like, there's so many cool things that can help us live longer and, like, I'm old, so, you know, I think a lot about that. But so what I think is incredible about a lot of these things is that none of these things could've been built just a few years ago. So I think there's a lot of software basically like that exists today that, you know, we've unleashed and unlocked like a whole new set of capabilities. This isn't necessarily like the first time in the history of software we've been doing this.
Like, I've been coding for... Actually, I reflected on it the other day. It was 34 years. It's kind of crazy how long I've been coding. Started in '92. So there's been a lot of these shifts, like, you know, in the last 40 years. We've seen things like, you know, the web and cloud and big data and mobile and other things. Although I do think this, like, last few years, like with all the gen AI stuff or whatever you wanna call it, it's kind of incredible like what we've unlocked. But there's been many cases in the past we've also sort of unlocked. And when I started coding, when I started working professionally, this was 2008, 2009, the cloud was new. So I think there's like certain things we can learn a little bit from that.
The cloud kind of forced us to rethink a lot of how to build software. You had to deal with, you know, distributed systems and sharding and consensus. You had to deal with, you know, complex cloud stuff. Like, we have to come up with a bunch of new tools. Like, you're running into frictions with the old software, and you have to come up with a bunch of new tools, like Kubernetes and Docker and Terraform and a bunch of other stuff, in order to kinda take advantage of that new technology. And I think that's like very clear throughout the history of software. You kinda have to come up with the new tools 'cause you run into limitations with the old tools. You run into friction points.
And so some of the friction points today with AI, or GenAI, you know, you have to deal with like the fact that GPUs are not abundant, and you can't just scale up a million GPUs wherever you want, whenever you want it. You have to deal with like very large models that take a long time to load into memory. You have to deal with like crazy stuff like LLMs outputting code that you have to execute dynamically or compile. And so there's all these things, and you kinda wanna do that without, like, you know, just taking the old technology and turning it into a bunch of like YAML and Dockerfile hell and, you know, trying to build, you know, for this new stuff on top of the old stuff. I think every sort of new shift like this requires a new set of infrastructure, and we have to go very deep.
I mean, you know, obviously you hopefully know about Modal. Like, we think a lot about what does that new infrastructure look like and how do we kinda rethink, and we threw out a lot of the existing stuff and had to build some very deep things around file systems and container runtimes and a lot of this stuff. And in general, like, I think a lot about, like, you know, what are all these building blocks? The beauty of software is, like, you have infinite composability of, you know, you take these building blocks and you can build... It's like Lego. You take the building blocks and you can build whatever you want, and I think that's a very beautiful thing about software. But those building blocks have to be really good, and they have to be very general purpose. They have to be battle tested. They have to, like, sort of snap to each other in a nice way.
And so I think a lot about what those building blocks are. We've been working on this for a number of years. So I wanted to start by introducing a bunch of new building blocks that we've been working on. So I'm gonna go through a bunch of cool stuff we're working on in order to sort of help you build, really help you accelerate the pace of development. 'Cause I think just like, you know, I showed you a bunch of applications, like four years ago, I don't think we could've imagined a lot of those things. I think it's really hard to imagine in like two years all the software we're gonna build. But I do think, like, if we get these building blocks, you know, then that helps accelerate how we get to this new world of like new capabilities and software.
So I'll talk about inference for a second. Inference has always been the core of what we do. It's our biggest use case. And we've always had a lot of customers, you know, figuring out, like, you can use Modal for GenAI. Initially, we saw customers like, you know, Suno using us for AI-generated music, and over time, customers figured out how to use Modal for LLM inference, deploying SGLang and vLLM and all those things. But a somewhat ironic thing is, like, we actually never had an inference product. Like, we never had something. We had the Modal functions, and Modal functions is like an example of these building blocks that I think were always like fairly good and sort of generalizable, and you can use it for a very wide range of things.
But, you know, it's 2026 and there's a lot of more cool things we can do, and especially if you go higher up in the stack. It turns out there's a lot of interesting things we can do building higher-level features specifically for inference and in particular for LLM inference. So here's one thing we've been launching: shared endpoints. A lot of customers, talked to a lot of customers, and they wanna use the open-source models, like the really good things like Kimi K3 and DeepSeek 4.1 and GLM 5.3 and all those ones. And, you know, you could always do that using Modal. You could, you know, stick it in a container, deploy SGLang and vLLM. Somewhat complex, and for a lot of customers a little bit of overkill.
So we're now launching shared endpoints, fully token-based pricing. You don't have to think about infrastructure. We got all the latest and greatest models. We have to build a lot of stuff underneath. So sort of this is like an example of a building block very high up in the stack. But we have to think a lot about everything from like KV caching to routing and load balancing and HTTP and a lot of these things. And of course also things like the models themselves and how do we optimize this. We train very custom draft models. We constantly keep optimizing things. So these models will just get better. I mean, the shared endpoints will just get cheaper and faster over time as you use them.
And another really cool thing is we already have a lot of customers for this, you know, customers like PostHog and Fathom and Snorkel using this for coding agents and audio transcription and a whole bunch of other stuff. So some of the stuff we've been working on, for instance, is kind of exploring the Pareto frontier, you know, doing parameter sweeps and looking at like, you know, what are the best set of parameters and how can we train the best draft models? And really kind of going into like LLM performance tuning as like a science. So shared endpoints, that kind of represents like a happy medium. We try to optimize for a lot of different use cases at the same time. Shared endpoints, you know, like some customers use it for very high throughput stuff. Some people wanna use it for low latency. We can't optimize for everyone at the same time. So shared endpoints is a little bit like sort of a happy medium that works for like most use cases.
But we also have a lot of customers who want, you know, they wanna optimize for their specific use case. And so we're launching dedicated endpoints as well. So dedicated are single-tenant endpoints with all the sort of benefits that you like about Modal, auto-scaling, no commitment needed. You can just, you know, sign up and start using it. We built a CLI and a UI that you can deploy this in a few seconds. But all the sort of benefits of, you know, higher level things to get a lot of observability and other things out of the box. And we have to kind of rebuild a lot of the stack, and I'll talk about that in a second, like all the KV caching and stuff.
So dedicated endpoints are for you customers who want... You know, you wanna customize the performance. You wanna maybe, maybe for security reasons, you want single tenancy, and you wanna go into like deeper parameter tuning. Or maybe bring your own weights actually, 'cause a lot of customers are fine-tuning their model, and they might wanna bring their own weights. A really cool thing about dedicated endpoints is that it's not a black box. When you deploy things from Modal, like we work very closely with vLLM and SGLang. In fact, a bunch of them are here, and they're gonna help you with performance tuning. I think we have a Genius Bar later you can talk to them about.
So it ships with great performance out of the box, but you can peek into the box, and it's all your code. And you can take it out and run it elsewhere and tweak it and change it. And so I think there's a little bit of a different approach, is, you know, really leaning into open source and working with some incredible people building some incredible technology out there. Get full observability out of the box. I think this is an example of, like, the fact that we're building higher up in the stack. We can sort of take all the metrics and stuff that comes out of the engine and just plot it inside the Modal dashboard. So you can track things like time to first token and throughput and other things directly in the stack, as well as all the stuff you always had before, like GPU temperature and things like that.
So we already have a lot of customers using this. Cognition, and by the way, Scott Wu's coming up next. It's gonna be really cool. Reducto, DoorDash, using these in production today. Endpoint candidates. So I talked a bit about, you know, sort of the science of performance tuning for LLM inference. So endpoint candidates basically systematizes that and creates like a better way to try different variants. Like, you can basically roll out a new variant and kinda run an experiment in production, and if it works, you can promote it to run the main traffic. You can roll back. You can try different things.
So it's basically a way for you to, like, try different parameter sets and really kind of, you know, exploring the frontier of, you know, LLM serving. And something that's an incredibly cool thing about this is, like, this is a step towards, we're working on this with a number of design partners, doing this fully automatically. So while you're sleeping, the model will just propose new candidates, roll them out, test them in production. Things just get cheaper and faster over time. It's very cool. We had to build a lot of low-level routing and load balancing and all kinds of stuff underneath, especially with the KV caching for LLM inference. It's a big part of achieving good performance, so we built a whole routing system underneath. This is what powers all of these things.
Works really well with autoscaling too, and I think that's something that we've always been very good at. So the sort of scalability, scale up and down, and introducing new replicas. Model service is something that underpins all of these things. And so far I talked about LLM inference, but model service is actually a building block that you can use on its own for other use cases too. So it turns out Modal functions, again, I mentioned you can use them for a lot of different things, and people kinda figured out how to actually create, like, an insane amount of different types of applications based on Modal functions. But Modal functions were kinda built for sort of functions. Like, they're like inputs and outputs. For a lot of things you actually want something more like HTTP streaming. You want maybe low latency, you know, bidirectional streaming and things like that.
And so for LLM inference, for voice agents, for video models, we saw a need to build basically, like, directly for HTTP, and we're launching model servers, which basically makes all this like way much faster. So now you can just take anything, any code that runs an HTTP server and just deploy it on Modal, and you get really good performance out of the box, and we manage everything. We had to build our own custom routing and load balancing layer in Rust to reduce latency. Sticky sessions. When you're dealing with applications like, let's say you have a voice agent and you have a customer on a phone call talking to someone, and you have some model in the background tracking that. You might have essentially like a stateful actor.
Sticky routing makes it possible for you to sort of connect to a consistent backend and manage that state in a consistent way. So we're launching this to make it possible to build even more complex applications that require more complex routing. And we already have a lot of different customers and a lot of different people. So I just wanna call out all these people here, Inco on the slide, Typesafe, DoorDash, Infract, SGLang, and Runway. They're all gonna be here today and talk a lot about inference and pushing the frontier and doing some really cool stuff.
All right, let's talk about training. Turns out you can't do AI without a model, but how do you get the model? And you have to train a model. And you actually don't have to train a model. I lied a little bit. Like, you can take models that are open source and use them inside Modal. And a lot of people have done that for a long time, obviously, or sometimes you can just use them through an API. But a lot of people have actually been training models inside Modal for a long time too. And there's like at least two use cases that a lot of customers have been doing, which is post-training, typically LLM models. You take an open source LLM model, and you post-train the model using, you know, SFT or reinforcement learning. But a lot of customers are actually pre-training a model too.
And to be clear, like, I'm not necessarily talking about LLM models. I'm talking about things like, you know, biology models or robotics or things like, you know, recommendation systems or audio models or things like that. You could actually do that really well on Modal in certain ways. Like, 'cause we have this function primitive and you can use that up to eight GPUs. You could deploy things and run it for twenty-four hours and train a model. It works sort of okay. However, you can only do it up to eight GPUs. So that was the limitation. We started seeing customers who wanted to train on way more GPUs. So we actually started working on this product called Multi-Node Training almost two years ago, actually more than two years ago.
It's been a very complex product to build 'cause we had to figure out a lot of very complex things like how do you scale up and schedule a lot of nodes at the same time in the same zone? How do you get them all interconnected with RDMA? Which, you know, all these training methods, they need like very high bandwidth to do the gradient updates. And how do you deal with the security? Like, how do you isolate that networking? And those turn out to be very hard problems. But we solved it, and we're now introducing Modal Clusters. So just one single line, and you get, you know, fully interconnected with RDMA a bunch of different nodes that can all talk to each other. And it works pretty much out of the box.
Like, you can do pre-training on Modal with all the benefits that you always liked about Modal. The sort of instantaneous scale up, the fast feedback loops, the pay-as-you-go, no capacity commitment required, but now with multiple nodes. It's very cool. So we already have a lot of customers using this. We've had some design partners for a while. Some of those customers are here. 1X building robotics models, Decagon training customer support agents, and Runway training video models. And over time, we've also been working with certain customers that are doing more complex things, like reinforcement learning is a very popular thing to do.
Very hard, very annoying to do because you have a bunch of infrastructure challenges, but also a lot on the machine learning side, like figuring out how to do the rollouts and how to update the weights and all these things. So we are launching managed reinforcement learning training. Comes with a bunch of recipes, comes with a UI, a bunch of building blocks that are battle tested. We worked with a number of customers doing these things. I was very excited about that. It's gonna help a lot of our customers doing more complex post-training. Another thing we noticed, we talked to a lot of different customers, is they really liked the Tinker style model of, you know, using an API to manage all the infrastructure, and then they just provide a number of methods to compute gradients.
So we're launching an Async RL SDK, and basically the idea is, like, it's a Tinker-compatible SDK for post-training. And one way I kinda think about this is, like, it's sort of a layer cake where you have all these building blocks, and you can use the underlying things, you know, like Modal Clusters in themselves, but you can also use the higher level things like the RL training framework if you want a higher level service. As we already have a lot of customers doing this with us, Ngram, Sierra, Decagon, and Cognition, among many others. And there's gonna be some really cool talks later today about that as well.
All right. Sandboxes. A lot of stuff going on in the sandboxes. We're actually gonna kick off a little demo, so we'll get right into it. And, you know, the marketing team was very annoyed when I insisted we should do this. This is actually a bunch of sandboxes running a model, like, live. It's not recorded. We're gonna see how far we can push this. Can we get to a million? Let's see. We've been running this a couple of times in the past few days and, you know, the system has handled it quite well, but hopefully it handles it really well today too. While it's running, all these sandboxes are running a model, like, live right now.
We started working on sandboxes back in 2023, and the observation back then was a bunch of people were starting to use LLMs to generate code. And we realized they kinda need a way to execute that code and, like, you know, how do you execute code you don't necessarily trust in an isolated way? So we launched sandboxes and started seeing some traction early on, but it really took off about a year ago with Sonnet 4.5. Finally, LLMs could generate, like, whole apps. And so that's when it really started exploding. We've always been really good at scale, like, we're aiming for gigas and really good at reliability and performance and scale and these things.
But the scale that we started seeing people running kinda shocked us, like, you know, and we had to re-architect a lot of the system and kinda rebuild a lot of the architecture to handle even more scale. So I think with Modal, like, one of the things I've always been proud of with sandboxes is, like, we're really good at this insane scale. As you can see, we're running 755,000 sandboxes right now. And, you know, that's something you could always trust us to do, like anything related to performance and scale and other things. Let's see if we can make it to a million. We've always been pretty good at... There's a lot of other stuff too with sandboxes in particular that matter besides scale, but that was always like the core thing that we're very good at. Very close now. There we go. A million sandboxes.
And people do this, by the way. Like, we have a bunch of customers running hundreds of thousands or millions of sandboxes in production. So pretty cool. But scale is just one of the things that matter in sandboxes. It's also, you know, about offering building blocks and, you know, agents are getting more and more complex. They're sort of pushing the boundaries for what they need from infrastructure. We built our isolation mechanism based on gVisor initially, which is a Google product. But it turned out, like, gVisor is basically like 95% compatible with Linux and it gives you almost full compatibility. But for certain things it's not quite there.
There's like things, you know, like running Docker or messing around with firewalls or installing kernel modules, all kinds of crazy stuff. And some people actually wanted... It turns out a lot of people actually needed that. So we're launching VM sandboxes today. And VM sandboxes basically gives you a full Linux sandbox. So all the things you like with Linux, you can connect to it, you can poke around, it behaves just like Linux, and you can now give it to your agents. Very cool. And we already have a lot of customers who've been doing this as design partners in the past. Linear, Snorkel, and Lagora among several. So looking forward to see what you can build with that.
Another thing we started seeing with sandboxes was customers needed more capabilities in terms of coordinating sandboxes, like, and things, you know, like dealing with complex secrets injection schemes, or maybe co-locating multiple sandboxes 'cause they wanna have a harness running close to the sandboxes because they need low latency communication, or a bunch of other things. So we are now launching sidecars, which is basically the idea is, like, you can spin up sandboxes and you can have them run on a single worker, and, you know, get like extremely good latency overhead and do things, you know, like secrets injection and complex things and other things.
So those are some of the building blocks, inference and sandboxes and training. I am very excited to see. Hopefully some of these building blocks will help you build this, you know, new set of capabilities with software and over the next few years, hopefully I'll... I'm excited to... Maybe next year I'll see you up on stage, like, talking about some of the stuff you built with this thing. So anyway, we have an extremely good lineup. Again, super impressed with all of you. You inspire me every day to, like, build better building blocks. You... Actually, you do all the cool stuff. Like, I do the boring stuff of, like, providing the infrastructure for you 'cause I feel like you all are, like, kind of pushing the frontier in ways that are far more creative in certain ways.
But so just wanted to thank you all for coming and really want you to enjoy the rest of the day. We're gonna have some really, really cool talks from many other people. Thank you.
