All right, thank you. We have Ion Stoica, faculty at UC Berkeley, adviser to many in the audience and many of the famous people that we've had pass through the VIP lounge in the last two days. Co-founder of Databricks, co-founder of Anyscale, co-creator of Apache Spark, Apache Ray, working closely with the Lamarina team, advisor to creators of Alimsis, which led to Vakuna and Chatarina and SGLang. So a long list of accomplishments. Often people think of you as the most impactful researcher in systems and AI or AI systems these days. So that's Ion Stoica, also one of my official co-advisors at Berkeley. So I have a bias here. Where should we start? I think we can talk about two things. You want to talk about evolutionary systems or evolutionary techniques? Evolutionary algorithms.
Let's hear. What are these evolutionary algorithms?
You can start with anything you want.
Oh, I want to start with that one.
I think there are a bunch of papers and work, and one particular one was Alpha Evolve. It's an AlphaStar paper that came earlier this year, and they show that they use this to solve a pretty impressive set of problems, both algorithmic and systems, like improving some scheduling algorithms. I'm not sure if Borg or something like that, and you know, 1% or 2% whatever, but it's impressive because it's doing something useful for what we've been doing. At Berkeley, you remember systems research. Yeah, system research. You remember about meas? I haven't forgotten it. Deep scars, I mean wonderful memories.
Yeah, so actually over summer, many people in the audience, there is an open source version of it called Open Evolve, and we tried it in a seminar we had. We ran it in a seminar to try it, and people for different research. One PhD seminar where different PhD students were trying this in weekly meetings. And different people tried it on their projects.
It's like a parameter optimization technique. Using parameter optimization.
It's not parameter optimization. It's basically you are going to change the code, right? Using evolutionary techniques to improve the performance.
Does it use generative AI also?
It's using obviously large language models.
Right. So basically the way one way it's working: you give a simulator code. You give the code, you say what you want to happen, you say which code is implementing what you want to change, for instance load balancers or routing or whatever, and then you are going to iterate on that. You are going to get a solution. You are going to use a simulator to run on a bunch of benchmarks to evaluate it, and the LLM will look at the results. You tell what good results are, what you want to optimize, and then you iterate by reusing some of the old solutions and always generating hopefully new solutions.
You ask the model to improve the code in a given way. You kind of give it some instructions, you see it improve. How do you remember how do you do the research? Right? You do research the way PhD students do research.
Iteratively, right? It's like this is exactly trying to emulate, right? You want to improve some system performance, and typically you can do it either in the real system, but that sometimes is too expensive. So you have a simulator. If there is no simulator, you write a simulator.
I remember writing those.
You remember, right? And then now you are going to start to apply your ideas to implement in the simulator different algorithms, you know, tweaks and so forth. And what do you do? You have a bunch of traces. You run that simulator on the traces. You look at the results.
Good. I mean, if you know the right directions, you iterate until you get some good enough results to publish, right? That's what you do.
So you also iterate in something like gradient ascent. And this is just an alternative.
This doesn't change. It's like an optimization technique where you're iterating to find the model.
Doesn't change the model, only the code.
Only the source code. So you're iterating on the source code that the simulator code.
Which is part of the prompt.
Right. Yeah. Which is what the model is generating and then iterating on the source code using the same prompt each time.
Using the same prompt, but you can also change the prompt like in Japa. But in particular, the prompt is going to change by taking into account the previous solutions and their feedback. I've spoken with Lakshia and Delara about Japa and all their optimizers. So that's another area in which evolutionary technique is showing up. But now it seems like there's this category that you are thinking about.
Yeah, Alpha Evolve more general. Alpha Evolve, or at least Open Evolve, the way the prompt evolves is more structured.
We just mostly the prompt evolves by adding new solutions, and that's kind of the way, and it's working. The point is that the excitement comes from it working better than expected.
Okay. Wow. And just to zoom out a little, it is doing what PhD students do.
Exactly. Which is maybe one of the more exciting things. I can't remember, I think you might have told me the seminar was one you already ran or are going to run, about a variety of AI techniques that researchers might use. How are researchers using AI to do better research? And are there agents or other frameworks that are making progress at doing research themselves? That has come up in a few different ways in my conversations so far.
Totally. So this is an evolutionary technique being one that is working.
One of showing progress and promise.
Yes. Yes. Were there others that are showing promise that the students? I think it's always there.
Okay. RL. Okay. Yeah. People mostly know they are related. But let me tell you what the exciting things about this one are, at least in my opinion. First of all, it seems pretty robust. People at least during the summer started with different models, different kinds of settings, and many of them got good results. Good results meaning improving the performance of the state-of-the-art or whatever the work they were doing in the research. The second thing is it's pretty cheap.
Yeah. It's only a few tens of dollars.
And it's pretty fast. We did in general 100 iterations.
100 iterations. And you get progress in 100 iterations, which is minutes.
Yeah. In each iteration, depending on the system you are using, you do an inference on the LLM to generate a new solution, you have to evaluate it, and then that's an iteration.
That's like a minute or a few minutes depending on.
Depending on the simulator, actually.
That can be the bottleneck. The other thing is that you can use the largest language models because you don't need to change them.
Don't need to modify them.
You can use the proprietary like Gemini 3 or whatever.
Absolutely. And the other thing is the solution is interpretable, right? Because you got the solution is a code. You can actually see if there are new algorithms. You can see what.
Because the artifact is the code itself and the simulator that's being evolved. So that's why in some sense there is some similarity with reinforcement learning. In both cases you have a model and you have a prompt, right? And you generate a solution, then you evaluate. With reinforcement learning you do your rollouts, get the trajectory, and use the trajectory with feedback to update the model. In this case, use the feedback, which is a result on particular solutions, to update the prompt by maybe incorporating that solution or other solutions in the next prompt. Right? So in one model, in the other one prompt. Probably you can mix them.
But in Open Evolve or Alpha Evolve, it's actually the code, not the prompt.
But the code is part of the prompt.
Oh, I see. Got it. Yeah. And versus like Japa, where the prompt is being updated and may or may not include code.
Yeah. But you can do code as well.
Yeah. Got it. Cool. So it's like RL in that sense, but is an alternative to RL right now.
And has shown a lot of promise.
Yeah. So that's pretty much at a high level what it is.
Do you see this evolving into a more serious research agenda with a lab, or is there a lot of low-hanging fruit to continue to pursue? Obviously from research, what I like about this direction is that you try to solve real problems. It's like problems you need for your research.
So we're actually giving it the problem I care about as a researcher and seeing if it makes progress on what I'm doing.
Right. It's nice because it's AI systems and so forth, so it solves system problems.
What are examples of the problems the students were giving it? I was building a cluster scheduler.
Actually, the prompts in Open Evolve and so forth are pretty clear. You just describe the problem. Say, okay, this is a problem. But what is one of the problems? When I was doing my problem, I was building a cluster scheduler.
One of the problems is expert parallel load balancing.
Load balancing for GPU clusters.
Experts on the GPUs, right? On process of experts, CPU.
Oh no, okay, yeah, mixture. Yeah.
Right. So you want to have a balancing factor which is good. So you have good load balancing, and because the problem is very dynamic, because experts, each token each layer, you can change what experts are going to.
This is for distributed inference, right? So you're deciding when you do distributed inference for mixture of experts models, you need to schedule more GPUs under some experts if those are getting more.
Experts are lightly utilized, you're going to give them more of them on the same GPU. If an expert is very highly utilized, maybe you want to replicate it on different GPUs. So it's scheduling for resources for inference dynamically based on the workflow.
Yeah, mapping, assigning experts to the GPU, and then obviously you have to do the routing.
And there are two things which matter: how well you load balance, because for high throughput, the better you balance, the higher throughput and lower cost. And the other thing is how fast you can rebalance, because it's a very dynamic environment. These are two metrics. For instance, we did that for DeepSeek, and DeepSeek has an open-source implementation, but we also had access to the proprietary implementation from DeepSeek.
You had access to their proprietary implementation.
Yeah. I mean, we have a postdoc working at DeepSeek and like that.
