Good afternoon, everybody.
So it's good to see you all here at Build. I'm Mark Russinovich.
And we've got kind of a fireside chat today, but it's a fireside chat where we don't have a moderator. We're moderating ourselves. So, we're going to be talking through a bunch of questions related to AI, AI infrastructure, models, architectures for AI, and it should be a pretty engaging conversation.
Yeah. So, I think we can start by talking a little bit of our background and why we're up here talking about these subjects. So, Ion, you've got a background where you've had some amazing impact on the industry in data analytics, in AI, in your role as a professor, in your role as a startup founder, Ray, vLLM, Databricks, Spark. Why don't you tell us a little bit about your background?
Yeah, so actually, starting around 15 years ago, at Berkeley, we started to develop quite a bit of systems, initially in the domain of big data. And started initially with Mesos and Spark. Then after that, Ray, and more recently, vLLM and Golang. As a high level, we're always trying to understand and figure out what is a problem you try to solve, and also a little bit careful about the trends. A problem you have today, is that problem going to be bigger tomorrow than today? Spark was scaling up data analytics and making it more interactive, and much better support iterative patterns like machine learning. Then Ray, complex scaling, complex AI workloads like reinforcement learning on heterogeneous hardware back then. And of course, vLLM is trying to exploit the power limits inherent of the GPUs. But then the memory became a bottleneck. So, trying to alleviate that bottleneck. The one thing I want to say, because we are at the Microsoft conference here, is that I'm very happy that some of the things we've done has impact also with our collaboration and partnerships. As you know, Spark and Databricks, great partnership with Azure Databricks. And actually, today, we also announced a partnership between Azure and Anyscale, and Anyscale is around Ray. So, yes, very happy about that and very much looking forward to this partnership to grow.
Yeah. So, you know, on my side, you, Mark, you have so many contributions. You wrote Windows Internals, then you architected Azure infrastructure, which I did use. And then you are now building and architecting AI infrastructure for Azure, and security are leading. So, one question I have, you've seen all these phases, and you've been in distributed systems for a long time and security. So, what is new and what is old? What still applies and what is no longer applying?
Yeah, well, that's a good question. I think one of the things that still holds, and this holds, I think, you'll even say for infrastructure and systems, is the fundamentals are still in place. I think if you take a look at fundamentally what's changed, it's these non-deterministic systems that are now carrying out workflows that they create on the fly. And that introduces a bunch of new risk areas. In some sense, humans are non-deterministic agents creating workflows on the fly. But humans you can hold accountable. Also, humans, I think in general, are more reliable than the AI systems we have today, and certainly, when it comes to humans, we designed our systems around human speed and scale. And when you have agents now that are intelligent running around, they can perform lots of damage in a very short period of time. So I think if we get into it a little bit later, we'll talk about some of the things like prompt injection risks around that, risks around jailbreaking, risks around hallucination; all of those things can cause workflows to go awry and for the system to not behave as you intend.
Yeah, and what about the fundamentals still remain? But again, to be more precise about concepts like consensus, consistency, distributed scheduling, which of these concepts remain more or less the same, and which are changing and morphing to support new requirements?
Yeah. Well, and I think when we talk about AI and running these agent workloads, and you talk about the LLMs and hosting the LLMs, the things like the consensus around the infrastructure, knowing what is available and online, things like load balancing still matters. Locality matters still. So you want to route requests, especially when you talk about KV caches, which is the context. You need to make sure that the requests are going back to the same systems. Otherwise, you're going to be wasting compute and bandwidth as you're moving things around. So all of those things still apply. I think, and this is an interesting question, I think, for you. Are these systems stateless? Do you consider them stateless or stateful?
I think it very much depends. Certainly, I also see a serverless trend. I think that quite a few systems become more stateful. That is because the trend of bringing the state in and saving the state if needed for the next operation, in some cases, it's too high. And it's not only the state, loading the state and saving the state, it's initializing. If you are talking about initializing the GPUs and so forth, it can take a long time. So this forces you to be more stateful. The problem with that is that it complicates, like we all know, the fault tolerance.
Yeah. So when it comes to these distributed infrastructures for AI, which include the GPUs, high bandwidth interconnections, the KV cache, when you look at distributed systems through its evolution, what assumptions change with an architecture to support this kind of computation?
Great question. It very much depends on where you are coming from. So you're coming from Spark, big data, it's about MapReduce plus plus. It's bulk synchronous processing, which is a pretty simple model. You partition the data, you have stages, at every stage you apply the same task on different partitions and do a shuffle between them. What happens with, for instance, you talk about KV cache and inference, things are becoming so much more coupled, tightly coupled. If you think about mixture of experts and then you have the KV cache and you need to distribute it across different GPUs on the same node and even across nodes, everything becomes much more coupled and more difficult because of that. Now, if you come from maybe HPC, maybe you can say, "I told you so." But even there, with HPC, it was much more — you don't have such a tightly coupling between the data and the compute. Because even there, you don't have a lot of data to start with, because you do simulations. So I think that's what it is. And this creates a lot of challenges. And even when you talk about load balancing, things are changing so much more quickly if you think about expert parallel load balancing, because you can change from layer to layer when you go in the forward pass to do inference to do serving. And the other thing is that things are evolving so quickly. By the time you build something, you have new requirements. It's never ending.
So, we have this, I remember like it was 10, 15 years ago when people started cloud computing and so forth, and people were saying the data center is the new computer. I'm just curious, is that abstraction still actuality because now you can have training spanning different availability zones and things like that. I'm curious about that.
I mean, I think if you take a look when that statement was made, it was when a data center would hold all the compute that you could imagine in the data centers at that time. Even if you take a look at the evolution of Azure as a hyperscaler, the availability zone became a unit of compute, but even in an availability zone, we have multiple data centers typically. It's not just one data center per availability zone. Then you have a region, you have multiple availability zones. So even for traditional IT workloads or serverless computing, your application can be spread across many data centers. And now, you mentioned distributed training and AI training, and that's where you run into these AI super factories as we call them, where you've got massive datacenters just filled with GPUs specifically to host that training workload. And for that, we passed the line several years ago of a single datacenter being enough to hold the scale of the AI supercomputers that we're building now to train these frontier models. And then over the last couple of years, we've crossed a line of even one region of datacenters is not big enough to train these models. And now we have these Fairwater systems that we've launched, these AI super factories where there are multiple Fairwaters in Wisconsin, multiple in Atlanta, and they're connected with a dedicated network so that you can have a job that spans the GPUs across both regions with a very direct cable between them doing RDMA across the two regions.
Now, going back again, we discussed a little bit about serverless.
Yeah. So, what does it mean when you have today's long-lived agents, which can go hours, sometimes days?
Well, if you talk about serverless, it's kind of people speak about it synonymously with containers. Serverless equals containers. And when people think containers, they think the original serverless containers, which were like Lambda or Azure Functions, which are short-lived, and so people conflate serverless with short-lived applications or pieces of code, when in fact serverless doesn't have an opinion about what code runs in these containers, and so you could put SQL Server inside of a container, and it can run for years. And it also can be monstrous. It can have terabytes of memory. So I think for me, I separate the two. And I say serverless is you don't worry about the infrastructure, it is containerized for the most part, although we talk about lightweight containers now and Wasm sandboxes which are also a form of serverless. But that's where I think the whole trend — if you take a look at Azure's architectural evolution, we're moving everything to serverless. We're moving the whole infrastructure to containers that are on a serverless infrastructure that is doing the scheduling and placement for the job, which is expressing constraints.
So now continuing on that note. We have our infra, we build these infrastructures mostly to serve requests initiated by people directly or via BI tools and things like that. But now we are moving rapidly to an age of agents, to having agent loops instead of just requests as a unit of work. How does this impact, do you think, about the architecture stack?
Well, I think we're still at the early days of this, actually, because just in the last few years, we went from chat to now agentic systems, which are this loop where the agents are talking to other systems as part of their computation. And they might even be talking to other agents. And so when it comes to how you define an agentic application that consists of these things, I think we're still in the early days of specifying the application model for these. And a lot of it is still ad hoc. But we're getting towards what is an agentic application? What are the components? How do you place them for efficiency? How do you automatically deploy them instead of taking piecemeal and deploying and connecting them together after the fact? So in my opinion, I think that we're still early. But I'd be curious to hear about what you think.
So, yeah, I think we are early. The one thing I see is a constant increase in complexity. Even the definition of agents, what is an agent? Well, was an agent an LLM invoking a tool? But now it's much more than that. You have a tool, you have an entire harnessing, which can be very complex. So now, and then you are talking about some kind of continual learning. People talk more and more about whether this happens in context, continuously evolving the prompt. And in some cases even updating the weights, kind of like RL. So, you need something to support very diverse and complex applications. You need to do, for sure, inference, then basically running arbitrary application because that's part of the harnessing. And then maybe you are also going to do some kind of training if you want to update the policy. And when you have the models, of course, the models are going to be co-located — it has to be co-located because if you want to update it, or it can just be something running outside, it can be like OpenAI or a new model that's released by Microsoft. So it's so complex. So you need to have something which is very flexible to support these workloads. And yeah, I think that we'll see. I assume that in time, things will stabilize one way or another, but we are very early.
Yeah. So one of the things that I'd be really curious to hear your thoughts on, in the systems that we've designed, they've largely been deterministic. You design the system, you've got a workload, you optimize for it. These systems are inherently non-deterministic, as we talked about. And so how does that change the way you design a system?
That's a great question. And just to be clear, one of the reasons they are non-deterministic is because, obviously, the LLMs themselves are non-deterministic. If you send the same prompt over and over again, you shouldn't expect the same results. Actually, in some cases, you rely on that to get diversity in order to get better solutions. So I think there are a few things to consider. One, first of all, what is your output? And in some cases, like with coding and so forth, the artifact can be still deterministic. You use this to create, to optimize the system, to synthesize even a system, and that one is deterministic. You can have harnessing tests and so forth to check, and then you can have metrics to hill-climb. If you want to optimize for throughput, for latency, the output is still deterministic. I think now this doesn't answer the question about how you debug your evolution systems or agentic system. And unfortunately, there, you have to presumably record the inputs and outputs and the traces and then try to replace the traces to find what the problem is if there is a problem.
So when you talk about an AI stack, you talk about GPUs, network, you talk about different types of accelerators. Where do you see the optimizations happening, the most important ones? Or have we reached some limit someplace where there's the bottleneck that we need to focus on?
I think clearly it will happen. I think there are three levels of optimizations will happen. At the top level, there will be many algorithmic optimizations still. So I assume that a few orders of magnitude improvement over the next few years will come from the algorithms. What I mean by algorithms, if you look at just inference or training, right now, of course, the workload changes, you have agents, you have much larger prompts or outputs, the context is much larger. You see a lot of optimizations. And architectural optimizations, software architecture, like sparse attention, different kinds of attention. You see different layers, different attentions. And now you see even within the same layer different attentions. Even maybe you'll have per-token learned attentions and things like that. So that can give you a big improvement. Of course, this comes with a lot of complexity. Then, obviously, you are going to have GPUs, the accelerators, which are going to still make progress, 1.5 or 1.6 every year improvements. And that's coming, again, with complexity because it's not like Moore's law used to be. The instructions and architecture are more or less the same. How you get these improvements, a lot of them is by adding new instructions which are targeting particularly new segments of the workload. I expect you are going to have more and more instructions in the GPUs to target this kind of sophisticated attentions. And then, from the inherent, this will go a little bit slower. You are still going to have smaller transistors, you are going to be more power-efficient, power per flops will improve and things like that. But these are multiplicative. So when you multiply them, you can get massive gains in the next few years.
Okay, I have another question for you. Because you've been involved so much with open source in AI infrastructure and analytics, do you see a canonical open source-based stack? And do you think it will converge or there will be diversity?
I think that there is some stack which is used by companies. At the lower level, Kubernetes or Slurm. Under the hood, if you want to move, because everyone wants to have the freedom to move across regions or from region to region, maybe you have something like SkyPilot. On top of that, you have to have some framework like Ray or Pathways or something like that, which helps you to distribute these AI workloads; then obviously you have PyTorch and so forth. So I think you can put together this stack, and then you have vLLM, or Golang, whatever, TensorRT-LLM as an inference layer. And then post-training, you have WELL or SkyRL, and things like that. So you can create a stack, and a pretty good stack. The question is about what will happen and how this stack will evolve because the one thing I see in terms of complexity increasing is that we used to build — one of the things you teach students, and we learn ourselves, is how do you build a complex system? You build it in a modular way, with components with nice, clean abstractions which abstract away the implementation, so you can evolve each of them independently. Now, the problem with that — and this is when we are talking about these layers and all these components — as we know, it also comes with some overhead. And it used to be perfectly fine to spend a few percent, 10% overhead to have this nice, evolvable system, because we could be even faster because we are going to evolve these systems quickly. But now it's so much breaking the layering, and it's called cross-layer optimizations. This is what I see. It's like, in some sense, all these layers are going to get crossed and broken. And I see, you can do maybe in an orderly way, but I don't know where you are stopping. Is this collaboration between Ray and Kubernetes, with you guys? But what is happening here is that something like Kubernetes wants to know more information about the applications from Ray to make more intelligent decisions, and something like Ray, which is up there, wants now to have more control to tell Kubernetes what to do. So you can allow that. But I don't know when you stop because, of course, you are no longer going to have these clean, narrow abstractions. I think that's very interesting.
So, we've got about five minutes left, I think.
So, one question I do have for you, we are talking about the workflows, agents, and so forth. Do you think about a boundary, two different kind of patterns of workloads, or do you think they are —
