How to Turn Research Into Billion-Dollar Companies, with Ion Stoica

Thanks for having me. Really appreciate it and appreciate the nice introduction. The only caveat there I will say that actually the students invented all of this stuff, so I think they should take the credit here. Yeah.

So I think, you know, the first observation I have is that things in academia, from some there are differences but all of those are similarities between academia and industry. So one of the similarities that in both cases what you want to do is like solve particular problems, right? You know, maybe sometimes very rarely problems which haven't been solved before, but in most cases you want to solve existing problems which are kind of important better. Right? So then, you know, some of the projects in academia inherently are going to solve some of the problems and, you know, solve a bunch of problems. And some of these problems are going to be more relevant for industries than others. And if you are lucky to solve a problem which is relevant for industry and you have a reasonable artifact like a system to do so, then it's going to get some adoption because people are going to use it to solve that particular problem maybe in a better way. And if that system or artifact gets more and more popular, then you are presented with several options. And the options come in the form of, you know, are you going to, because people when they use these systems and they are serious about betting on the systems to be part of their infrastructure or something like that, the one question they are going to have is like, okay, so what will happen with this system after the students graduate and so forth, right? It's like, can I bet on it? Is it going to be still there a few years from now? Who is going to maintain it? Who is going to add new features, evolve it, and so forth? So that's kind of the question and then you, on this, you know, people who were involved in the project, started the project, there are a few options. So one option is about, hey, you know, you decide that okay, it's not worth it. So you're going to drop the pet project. Whatever happens, it happens. The other option is that maybe you are going to donate the project to another company to continue develop it. Or the last one is to start a company, right? So, in many of these cases, you know, it's a project like two companies. That kind of was the process, right? Started to work on a project. The project was fortunately solving a problem people care about. The system we built or the artifact we built to solve the problem, you know, got traction, was good enough for people to use it and becomes more and more popular. And now you are confronted with these kind of questions about what to do. And in quite a few of these cases even was to start a company, which was a lot of times the decision was driven by the students themselves. I did help with that. So, but that's kind of in short. The one thing I want to make sure and to be very clear about, all these systems we built are open source, right? We are coming from Berkeley and as you know that Berkeley had this a long time tradition in developing and pushing open source. All the way from, you know, Unix, you know, FreeBSD, right? Unix before Linux. You know, from networking stack for the internet. And like TCP/IP, the first database, relational database Ingres and many more.

I think, you know, that's how to, so first of all, both Inferex and Radix Arc are early on. So, there are still a lot of things to figure out about the business. So that's number one. In terms of your question, it's about remaining open source. I do think that they are going and I do hope they are going to remain open source for the foreseeable future. And I think one of the reasons for that is that it's a competitive market and environment. So I think that alone it should encourage the open source motion. And, you know, it's like it's even, you know, it's like vLLM, it's SGLang, it's TensorFlow and Arc, you know, Nvidia's offering as well, TensorRT-LLM. And yeah, so I think that's enough to drive the open source for quite a bit.

Yeah, that is a good question. So one thing, yeah, we actually when we look at some of these problems and when we started some of these projects, one of the things we did consider about the trends to see whether the trends are going to be very important, the relevance of the problems you're going to solve, right? It's like if say you focus on solving a particular bottleneck, if that bottleneck disappears because of the trends, then your solution always will be no longer relevant. So I think that's kind of an important aspect. So now I think that taking a step back, I think that the biggest challenge I can see is that the sheer complexity of the stack has grown order of magnitude compared to what it was 15-20 years ago. Just to be very, you know, and again, instead of, I'm describing the problems here. Again, with the problem and requirement things. But when it's this Spark, it was, you know, starting in 2009, so I don't know, like 2010. So, it was 16, 17 years ago. When it did that, you know, it was still complex because well, you are talking about distributed systems. Distributed systems are more complex than single node or single systems. Fundamentally, because you need to deal with failures, communication, and many other things. But still, if you are talking about the environment, you know, the clusters you are deploying the systems on, we are talking here about Hadoop, Spark. They are homogeneous clusters. So, more or less the same nodes. And then each node has Intel CPU or maybe AMD. That was the biggest delta back then. And maybe you have hard drives and they are connected by the Ethernet, right? That's kind of what you have. So then on top of that, you are going to have this kind of Hadoop and Spark, which are also pretty, so to speak, pretty, they are distributed, but they are pretty regular parallels. Right? This is data parallels. And you have this kind of stages and each stage consists of tasks, which each task by itself is basically doing the same kind of work for different data partitions. You partition the data and then you shuffle between the stages. And now fast forward today, and it's so much more complexity at all levels. You know, it's, we think about the clusters. Clusters are not only distributed, you know, you have not only you have huge heterogeneity, right? You don't have only CPU and GPUs, but if you look at the GPUs, accelerators, and so forth. Like the last time when I checked, like vLLM was supporting hundreds of different types of GPUs and accelerators, right? Then it's not only that, it's also the networking. The networking is far more complex. You have RDMA's and InfiniBand, and things like that. And then the hierarchy, the topology of the cluster, network topology is very important, right? Because now you are going to have not only kind of server, you connect servers, you can plug, you know, rack level servers, like NVL72, you can have then pods, right? This kind of a lot of racks, which is tightly connected together. And then you look at the applications, the applications are, you know, workloads are so much more complex, right? It's not some kind of regular parallelisms, but you have nested parallelisms, you have so many things which because they are driven not only about training or, yeah, you have post training and inference. If you look at the inference, how complex it became, right? You have different kind of parallelisms, seven levels, seven different kind of parallelisms, like data parallelisms, tensor parallelisms, context parallelisms, sequence parallelisms, expert parallelisms, and so forth. So you have all this complexity and then things are moving so fast, right? Because every year you have a new model, GPUs from every major provider. Like look at Nvidia, right? And these are pretty different, they introduce new instructions. As the bottom line, actually, because you ask about this, it's not the challenge, it's not only one particular gap that people try to optimize the entire stack, right? It's like usually when you build a complex system, what do you do? You divide it in kind of some components, modules, have clear interfaces between them, and this allows you to conquer complexity because now you can worry about developing these components more or less independently as long as you provide the same, maintain the same interface between them. But now that typically when you do this modularity, it comes with overhead. You pay the overhead, some overhead, right? And people are not willing to pay that overhead because, you know, you train a model it just costs billions of dollars. So then what before was okay if a few percent you pay for overhead to have this kind of nice modular system which kind of is more evolvable, it's easier to reason about it. Now, you know, 1% if it costs you 1 billion dollars to train, this is 10 million, right? So then they break the layering in all these solutions, cross-layer optimization. So it's so much more complex. So I think that conquering that complexity, I think that's kind of it. That's kind of is a key. And even in the case of one workload like inference, that alone is so complex.

I think one, of course, one answer is going to be like hopefully, you know, things will stabilize and then we are going to have more time to think about what are the right abstractions, what are the right components. The other answer is about, you know, you need to figure out maybe other ways to develop this kind of software, to develop the stack, right? So, that's why you are talking about and we've been looking at using AI to optimize and even synthesize some of these components in the stack. Hopefully, that will be much faster, right? Because even though if developers cannot keep up with all this kind of complexity and the speed of evolution of the ecosystem, then, you know, hopefully AI can help you to, like that's a promise, right? It improves productivity of the engineers, so you can do things much faster. So, maybe you can keep up. So, I think these are the two approaches you need to think about at all. Of course, you know, and evaluate different approaches in order to solve the problem in this case.

Yeah, yeah, yeah, yeah, yeah.

I think obviously on one hand is that they're going to become better and better, right? We know that. On the other hand, I think they are relative terms, I do expect they are going to struggle more than maybe say like application level code. And there are several reasons for that. One, there are fewer examples. Like think about distributed, how many distributed systems are out there, right? Versus applications like web application or something like that is like a lot more. The second thing is the code, the logic is much more complex, right? If you talk about the infra. Because again, you need to deal with faults, right? To tolerate failures, recover from failures. If you are talking about the data, you talk about consistency, different consistency levels. And the third thing is just much harder, you know, takes much harder to test. And because eventually you generate code, you need to evaluate, you need to test it, right? And there are two reasons it's harder to test. One, it's because the sheer space for testing is much larger, right? Because again, you have different dimensions. Like I just mentioned a few of them. Data consistency, reliability, actually number of nodes as well. You have, you know, and each of these new dimensions which you do not have necessarily, like for instance failures. If for, you know, the application you are using now is not that complicated to handle it, right? Because it's this kind of, it's called this fate sharing. If it fails, it fails for all of us. And basically we restart it, right? But in distributed systems it's much more subtle. So again, so this means that the number of cases you need to test for is exponentially bigger. And the other one is also takes more time, right? You need to set up the system, you need to start the nodes, you know, start, so all of this takes more time. So, I think for that reasons, I do expect in relative terms, this model to struggle more with this infrastructure applications than, you know, use case in applications. And but the other thing is about for any kind of applications, you still need to make sure and I alluded to this, it's about how do you make sure that the generated code is going to align with your intent. It's going to do what it's supposed to do. And I think this is going to be for a while one of the biggest challenges, right? Because you see like, you know, it's like in some sense, if you look about how you do create, how do you solve, how do you, when you are talking about a problem, you're going to create, to talk about an artifact, typically what you have there are three phases. One, you come up with the idea about what you want to do or you formulate or you maybe or you pick the problem you want to solve, right? Then you come up with the solutions. You write the code or maybe it's a math problem, you come up with the proof. And then you can, or the idea for the proof, and then you check that what you've, your solution really solves the problem, right? And what these models do, they commoditize the middle part, coming up with the solution, generating a solution, like generating code. But then you are going to be left still with trying to figure out and of course you can use LLMs to try to help you to figure out whether the code is correct. But at the end of the day, it's like I think it's going, it's very hard to outsource the entire verification to LLMs, right? Because, you know, if you write a piece of code, it's not the LLM that's going to be liable if you have a problem. It's you are going to be liable.

Yeah, I think again, they are going to become better and better, but for some application, here it's a little bit like kicking the can down the road. What I mean by that is that okay, you are going to have verifiers verifying that the solution which is generated is correct, but how do you know that verifiers are going to handle all the corner cases, okay? Maybe you can have a verifier for that verifier. But you see it's no longer, and then if you start, it is like actually also, and you know, Andrej Karpathy, I think he said you can outsource your thinking but not understanding, right? So, if you don't, you end up with code which kind of is going to pass your tests, right? And then you are going to deploy it, and then it's going to fail. What you are going to do, right? It's like, oh, I am going to add this other test and so forth. But you know, it's like you need to have at least for a while, I don't know what will happen in 1, 2 years, 3 years, but you need to, I think that for the applications infrastructure you care about, you need to understand them, right? No matter how many files you have and so forth. Because the way, here is another way to think about it, at least how I think about it. You obviously need to be worried about this reward hacking, right? What reward hacking means is it means that in our case, like the code or the application is not going to behave in our case like you'd expect. Right? And let me, you know, this is maybe too abstract, but let me give you some examples. And sometimes these kind of reward hacks you cannot even think about them, right? So, if you cannot think about them, maybe it's very hard to also figure out or even for other verifiers to think about it because the verifiers are also trained on the data produced by humans mostly. Also now you have a lot more synthetic data. And we've seen a lot of these reward hacks in our research. Say you want to build a load balancer, right? The simplest one. So, you get a bunch of requests and you have a bunch of servers and you want to redirect the request to different servers so that, you know, each server is going to keep, you know, a little bit more or less the same. So, then you don't have, you can serve as many requests as possible, right? So, if you serve as many requests as possible, if you can, you know, distribute them evenly across the servers, right? Because if you have a server which is, you know, overloaded, then you are going to have the latency and the service for the request you sent to that server, okay? And, you know, it's in one case the solution, and one of the metrics here you want to maximize the number of requests you can handle. Right? So, for instance, one solution in one particular case, it was for the solution to maximize the number of requests it can handle by dropping them, okay?

So, now, I'm giving these examples because it's one of the things you don't think about, because if I'm asking someone to design a load balancer to maximize the load it can handle, I wouldn't probably even think about, hey, you shouldn't drop the request, right? Because it's obviously that's the wrong way to do. And there are many, many, many examples like this. So, that's what I'm trying to say is that how do you, you don't know what you don't know, right? It's like that's kind of the thing.

I think there are at least three things. So, one is about how you consume this kind of hardware resources. And you can have reservations, you can have on demand and so forth with the, you know, the reservation is cheaper to reserve this GPU but if you need to then on demand but you need to keep them utilized, right? So, I think and then the second thing is about what the framework you are going to use to support the workload on this, you know, hardware, right? And obviously if you want to handle on demand, then you have a more dynamic framework, you know, you can acquire new load. You can do fundamentally auto scaling, right? But the third thing I want to make sure is to say irrespective. If you think about and you try to oversimplify it. If you want to reduce the costs, if you want to even improve your availability, I think the name of the game you want to maximize the utilization of your most expensive, most scarce resources in this case of the GPUs, period, right? And if you do that, in order to, sorry, if again, if you do that, you reduce the cost, right? Because you can do more with less. Also increase availability, right? Because maybe what before I had to do is 10 GPUs, now I use only four GPUs. So, six GPUs I can use for something else. But what this means you need frameworks and you need systems which can handle heterogeneous hardware, right? Like for instance, if I have things like batch inference, right? You know, you do some data pre-processing, some data post-processing. So, and when you do that, those things you can do on CPUs. So, basically, in that particular case, I need a framework in which I can auto-scale whatever the number of CPUs so that I fit enough data to the GPUs to saturate them, right? And also to get quickly data out of the GPUs, right? To not waste the resources, the time on GPUs, compute on the GPUs, right? So, I think that's one of the very important aspects to handle heterogeneous compute. And this is going given more like given for inference, like it's like you can have for multimodal, you can have or even prefill disaggregation, you can have different characteristics for kind of GPUs for prefill, and then token generation, output generation. You can have even different GPUs, like for instance, if you have video, right? Because you can do decoding and coding can happen on cheaper GPUs, while if you have in the middle some transformer doing inference, then you want to deploy it on a much more powerful GPUs. So, I think that's kind of the name of the game. At the high level, the name of the game, you want to maximize the utilization of your most powerful resources, and typically means, especially for more complex workloads, that you need to support heterogeneous compute because this will help you to use cheaper compute, cheaper instances, to saturate your most expensive instances.

Yeah, that's a very hard question.