Distributed systems to AI platforms with Mark Russinovich & Ion Stoica | BRK227

Good afternoon.

And I am Ion Stoica.

I hope so.

Yeah, so actually, starting around 15 years ago, at Berkeley, we started to develop quite a bit of systems, initially in the domain of big data. And started initially with Mesos and Spark. Then after that, Ray, and more recently, vLLM and Golang. As a high level, we're always trying to understand and figure out what is a problem you try to solve, and also a little bit careful about the trends. A problem you have today, is that problem going to be bigger tomorrow than today? Spark was scaling up data analytics and making it more interactive, and much better support iterative patterns like machine learning. Then Ray, complex scaling, complex AI workloads like reinforcement learning on heterogeneous hardware back then. And of course, vLLM is trying to exploit the power limits inherent of the GPUs. But then the memory became a bottleneck. So, trying to alleviate that bottleneck. The one thing I want to say, because we are at the Microsoft conference here, is that I'm very happy that some of the things we've done has impact also with our collaboration and partnerships. As you know, Spark and Databricks, great partnership with Azure Databricks. And actually, today, we also announced a partnership between Azure and Anyscale, and Anyscale is around Ray. So, yes, very happy about that and very much looking forward to this partnership to grow.

Yeah. So, you know, on my side, you, Mark, you have so many contributions. You wrote Windows Internals, then you architected Azure infrastructure, which I did use. And then you are now building and architecting AI infrastructure for Azure, and security are leading. So, one question I have, you've seen all these phases, and you've been in distributed systems for a long time and security. So, what is new and what is old? What still applies and what is no longer applying?

Yeah, and what about the fundamentals still remain? But again, to be more precise about concepts like consensus, consistency, distributed scheduling, which of these concepts remain more or less the same, and which are changing and morphing to support new requirements?

I think it very much depends. Certainly, I also see a serverless trend. I think that quite a few systems become more stateful. That is because the trend of bringing the state in and saving the state if needed for the next operation, in some cases, it's too high. And it's not only the state, loading the state and saving the state, it's initializing. If you are talking about initializing the GPUs and so forth, it can take a long time. So this forces you to be more stateful. The problem with that is that it complicates, like we all know, the fault tolerance.

Great question. It very much depends on where you are coming from. So you're coming from Spark, big data, it's about MapReduce plus plus. It's bulk synchronous processing, which is a pretty simple model. You partition the data, you have stages, at every stage you apply the same task on different partitions and do a shuffle between them. What happens with, for instance, you talk about KV cache and inference, things are becoming so much more coupled, tightly coupled. If you think about mixture of experts and then you have the KV cache and you need to distribute it across different GPUs on the same node and even across nodes, everything becomes much more coupled and more difficult because of that. Now, if you come from maybe HPC, maybe you can say, "I told you so." But even there, with HPC, it was much more — you don't have such a tightly coupling between the data and the compute. Because even there, you don't have a lot of data to start with, because you do simulations. So I think that's what it is. And this creates a lot of challenges. And even when you talk about load balancing, things are changing so much more quickly if you think about expert parallel load balancing, because you can change from layer to layer when you go in the forward pass to do inference to do serving. And the other thing is that things are evolving so quickly. By the time you build something, you have new requirements. It's never ending.

So, we have this, I remember like it was 10, 15 years ago when people started cloud computing and so forth, and people were saying the data center is the new computer. I'm just curious, is that abstraction still actuality because now you can have training spanning different availability zones and things like that. I'm curious about that.

Yeah, it's fascinating.

My favorite services.

Well, if you talk about serverless, it's kind of people speak about it synonymously with containers. Serverless equals containers. And when people think containers, they think the original serverless containers, which were like Lambda or Azure Functions, which are short-lived, and so people conflate serverless with short-lived applications or pieces of code, when in fact serverless doesn't have an opinion about what code runs in these containers, and so you could put SQL Server inside of a container, and it can run for years. And it also can be monstrous. It can have terabytes of memory. So I think for me, I separate the two. And I say serverless is you don't worry about the infrastructure, it is containerized for the most part, although we talk about lightweight containers now and Wasm sandboxes which are also a form of serverless. But that's where I think the whole trend — if you take a look at Azure's architectural evolution, we're moving everything to serverless. We're moving the whole infrastructure to containers that are on a serverless infrastructure that is doing the scheduling and placement for the job, which is expressing constraints.

So now continuing on that note. We have our infra, we build these infrastructures mostly to serve requests initiated by people directly or via BI tools and things like that. But now we are moving rapidly to an age of agents, to having agent loops instead of just requests as a unit of work. How does this impact, do you think, about the architecture stack?

So, yeah, I think we are early. The one thing I see is a constant increase in complexity. Even the definition of agents, what is an agent? Well, was an agent an LLM invoking a tool? But now it's much more than that. You have a tool, you have an entire harnessing, which can be very complex. So now, and then you are talking about some kind of continual learning. People talk more and more about whether this happens in context, continuously evolving the prompt. And in some cases even updating the weights, kind of like RL. So, you need something to support very diverse and complex applications. You need to do, for sure, inference, then basically running arbitrary application because that's part of the harnessing. And then maybe you are also going to do some kind of training if you want to update the policy. And when you have the models, of course, the models are going to be co-located — it has to be co-located because if you want to update it, or it can just be something running outside, it can be like OpenAI or a new model that's released by Microsoft. So it's so complex. So you need to have something which is very flexible to support these workloads. And yeah, I think that we'll see. I assume that in time, things will stabilize one way or another, but we are very early.

That's a great question. And just to be clear, one of the reasons they are non-deterministic is because, obviously, the LLMs themselves are non-deterministic. If you send the same prompt over and over again, you shouldn't expect the same results. Actually, in some cases, you rely on that to get diversity in order to get better solutions. So I think there are a few things to consider. One, first of all, what is your output? And in some cases, like with coding and so forth, the artifact can be still deterministic. You use this to create, to optimize the system, to synthesize even a system, and that one is deterministic. You can have harnessing tests and so forth to check, and then you can have metrics to hill-climb. If you want to optimize for throughput, for latency, the output is still deterministic. I think now this doesn't answer the question about how you debug your evolution systems or agentic system. And unfortunately, there, you have to presumably record the inputs and outputs and the traces and then try to replace the traces to find what the problem is if there is a problem.

I think clearly it will happen. I think there are three levels of optimizations will happen. At the top level, there will be many algorithmic optimizations still. So I assume that a few orders of magnitude improvement over the next few years will come from the algorithms. What I mean by algorithms, if you look at just inference or training, right now, of course, the workload changes, you have agents, you have much larger prompts or outputs, the context is much larger. You see a lot of optimizations. And architectural optimizations, software architecture, like sparse attention, different kinds of attention. You see different layers, different attentions. And now you see even within the same layer different attentions. Even maybe you'll have per-token learned attentions and things like that. So that can give you a big improvement. Of course, this comes with a lot of complexity. Then, obviously, you are going to have GPUs, the accelerators, which are going to still make progress, 1.5 or 1.6 every year improvements. And that's coming, again, with complexity because it's not like Moore's law used to be. The instructions and architecture are more or less the same. How you get these improvements, a lot of them is by adding new instructions which are targeting particularly new segments of the workload. I expect you are going to have more and more instructions in the GPUs to target this kind of sophisticated attentions. And then, from the inherent, this will go a little bit slower. You are still going to have smaller transistors, you are going to be more power-efficient, power per flops will improve and things like that. But these are multiplicative. So when you multiply them, you can get massive gains in the next few years.

I think that there is some stack which is used by companies. At the lower level, Kubernetes or Slurm. Under the hood, if you want to move, because everyone wants to have the freedom to move across regions or from region to region, maybe you have something like SkyPilot. On top of that, you have to have some framework like Ray or Pathways or something like that, which helps you to distribute these AI workloads; then obviously you have PyTorch and so forth. So I think you can put together this stack, and then you have vLLM, or Golang, whatever, TensorRT-LLM as an inference layer. And then post-training, you have WELL or SkyRL, and things like that. So you can create a stack, and a pretty good stack. The question is about what will happen and how this stack will evolve because the one thing I see in terms of complexity increasing is that we used to build — one of the things you teach students, and we learn ourselves, is how do you build a complex system? You build it in a modular way, with components with nice, clean abstractions which abstract away the implementation, so you can evolve each of them independently. Now, the problem with that — and this is when we are talking about these layers and all these components — as we know, it also comes with some overhead. And it used to be perfectly fine to spend a few percent, 10% overhead to have this nice, evolvable system, because we could be even faster because we are going to evolve these systems quickly. But now it's so much breaking the layering, and it's called cross-layer optimizations. This is what I see. It's like, in some sense, all these layers are going to get crossed and broken. And I see, you can do maybe in an orderly way, but I don't know where you are stopping. Is this collaboration between Ray and Kubernetes, with you guys? But what is happening here is that something like Kubernetes wants to know more information about the applications from Ray to make more intelligent decisions, and something like Ray, which is up there, wants now to have more control to tell Kubernetes what to do. So you can allow that. But I don't know when you stop because, of course, you are no longer going to have these clean, narrow abstractions. I think that's very interesting.

So, one question I do have for you, we are talking about the workflows, agents, and so forth. Do you think about a boundary, two different kind of patterns of workloads, or do you think they are —

And workflows. Traditionally, what we call them.

You can have the agent synthesize that workflow.

Yep, that's what I mentioned about. So the other thing is that we used to have training and serving separate deployments and things like that. But now with this post-training, starting even earlier from reinforcement learning with human feedback, and now it's continual learning and things like that, it seems that it's a blur out there. So what should the platform look like —

Yes, of course.

Absolutely.

Yeah, thank you. And we haven't touched on —

On confidential computing. So before going, I know this is a topic dear to your heart, Mark. So can you say something about security governance and so forth?

So now you can, please.

Let me just wrap this up. And I think there were some more topics that we can discuss. And maybe we can take some questions from the audience. But confidential computing, for those of you that aren't familiar with it, it's hardware-based protection of data while it's in use. So the hardware itself is encrypting memory and creating a privileged boundary around the data such that nothing else on the system can get access to what's inside of that enclave, they're called. And it comes with a cool property the hardware provides, which is an attestation, where it can measure what's inside the enclave and then the application in that enclave can offer that attestation to another system that then can look at it and say, "Oh, your compute inside of an enclave, I trust that enclave, I trust your configuration, I'm going to give you a secret," typically a key, which allows you to go and process that data in the clear. And it's protected against so many of the threats that not having this enclave protecting that boundary would expose. And when it comes to agents, having an attestation of what is the agent, what is its provenance, do I know that the data that I'm giving to the agent is protected with the highest assurance as possible inside that enclave. You're going to see a trend with AI and confidential computing that I think is going to really drive forward confidential computing. Confidential GPUs from NVIDIA, confidential GPUs from AMD, and even Microsoft's Maia processor.

Yeah, that's fascinating.

And me as well. But let me just give... So, I think that... My TLDR is that now we have very powerful coding agents and so forth. And many of you, I'm sure — how many of you use coding agents? Everyone, right? I do think that — actually, the biggest challenge is that it's about how do you know that the code which is generated is going to work correctly for some definition of correctness. It is doing what you hope it to do. I think that's the key. And the more code it generates, the harder it is to do that. And I think this was also true before. Everyone here probably developed a production system or has been involved in developing a production system. Where do you spend most of the time? You don't spend most of the time in prototyping or writing the code; you spend the time in debugging, then maintaining the code. It's easily 10 to 1 effort. So, in some sense, what these coding agents are doing is going to commoditize that writing the code. But the bottleneck will be more and more in verifying. And, yes, you can use these agents to say, "Hey, check that, is that correct?" But it's very hard. Some agents are lazy. So, without you understanding the code, at least at this point today, and having continuously testing it and enhancing the test, I don't think you can trust that code is doing what it's supposed to do. Because that kind of reward hacking is very different from what you can think about. Let me give you just one example. I have so many examples about that, but I'm giving you the simplest one. You mentioned earlier about load balancing. So, say you want to develop a load balancer algorithm using the agents. And we did that a few months ago using EPLB, Expert Parallelism Load Balancer. And what you want to do is the metrics, you want to maximize the load you are going to handle. Now, one solution that we've seen is, because that's a metric the "hill climbs", the load, is just dropping requests. You wouldn't think that when you tell someone, even when you write the PRD about the load balancers, "Hey, you shouldn't drop the request." It's understandable. But these models are going to do that because they are going to maximize the things you told them to maximize, in the simplest and easiest way. So that's what I'm saying. And there are many things which can pass all the unit tests and so forth, and then when you look deeper, there are things which are, in a way, overfit to your test you couldn't imagine. That's what you still need to, I think, at this stage. I don't know what will happen in a few months, in one year, but right now, you have to be able to specify. If you didn't specify something, you can almost guarantee that the systems will find something around it. And there are only two places a specification comes from. One is the training data set. For instance, one year ago, when you generated code, you had a lot of vulnerabilities. Now, you have fewer vulnerabilities when you generate code with agentic. The other one is from the prompt. You have to specify what it should do or not. If it's not in the prompt or it's not in the training data set, it's going to find a way around.

Yeah, so that's the main bottleneck.

So you'll answer it after I answer it.

I think that from the theoretical perspective, it's very hard for me to see how it's going to be ever solved. And fundamentally, because we know from Gödel that no axiomatic system can be complete.

So that tells you something. I think that when we look at other techniques to do it, like for instance, assume that you have a formal specification. You can write in Lean, on Coal, and so forth. So from that, if I have the formal specification, then you can generate the code, and then you can generate the proof that the code satisfies the specification. So that is good, and you can see that. The problem is that obviously then you need to write that specification, which is not easy. And even when you write the specification, it cannot — if it's incomplete, you don't specify a property because, in the case of the load balancer, I don't write, "Oh, you shouldn't drop the packets." So maybe if there is a conservation law there, you can argue that you should put it in. But there can be assumptions, because any specification has to have some assumptions to make the proof. But you are not going to imagine all of the assumptions.

Almost, almost. Even that is not. Let me give you an example — even that is not. I remember we had this peer-to-peer court system and so forth. I was a young faculty there, still implementing all of these things. I thought it was bulletproof. And we were testing it on Planet Lab — twenty years ago, Planet Lab, an internet infrastructure where each organization has servers, and you deploy applications on the servers and then you communicate. And it breaks. And I don't know how it breaks. It has to work. And when we looked, it was an assumption we missed. The assumption we made implicitly is a kind of transient connectivity. If A communicates with B and B communicates with C, then A should communicate with C. And in the Internet as a whole, because it's not behind NATs and so forth, this has to hold. And it should have held, except that one university misconfigured their servers, which blocked some communication. So that's what I'm trying to say. It's extremely hard.

We have a biased audience here.

Thank you very much.