What's new in Ray and Anyscale: KubeRay Connect, Native RDMA, and more | Ray Summit 2026

Good morning and welcome to Ray Summit 2026. A year ago when we were all here, we saw the hockey stick growth of Ray downloads and celebrated it passing 30 million downloads in one quarter. By the end of June, it had reached 170 million downloads — 5x growth in one year. And this incredible growth is fueled by one common thread: more organizations evolving from simply consuming AI to building their own. And that shift from consuming AI to building AI is what is fueling the growth of Ray. Over the next couple of days, we'll focus on the evolution of Ray: how we are making it more reliable to meet the needs of the scale, how we are making it scale, how we are making it easier to integrate, and most importantly, making it work for your unique needs.

You heard last year from Cursor about how they use Ray to build Composer, their custom coding model. Since then, Microsoft's MAI Thinking One, NVIDIA's Neotron 3 Ultra, and ZAI's GLM 5.2 have all published reports on how they use Ray to build these models. You'll hear from some of them today and tomorrow. And this trend of building AI is not limited to frontier labs. All the robotics companies out there, physical AI companies, biosciences, fintech, and many other industries are all embarking on this trend of building and owning their AI. Here's a few examples. Laya Sciences, and you'll hear from them today, is building scientific superintelligence using Anyscale and Ray to accelerate the pace of scientific discoveries. Tar Robotics is building autonomous trucks and unifying auto-labeling, training, reinforcement learning, and simulations in a common stack. Torque Robotics is building — I already talked about — Bedrock teaches excavators to move Earth on their own, closing the loop from field data to retrained autonomy. And Coinbase runs its production fraud and credit risk models on Anyscale and Ray, powering its platform in real time.

Ray is increasingly becoming the backbone and the framework of choice for curating and pre-training custom models with multimodal data, post-training open models with reinforcement learning, and embedding inference directly inside the products. But the journey to custom intelligence doesn't end with inference. The next step is to close the loop and feed the next iteration so models can improve continuously. This continuous learning loop is something we are seeing across the board, and you'll hear this as a theme not only at this conference but as a trend for the next year in AI. And this evolution is happening at a breakneck pace, and to drive that pace takes a community of builders beyond those of us here at Anyscale. More than 1,500 developers have contributed to Ray, representing these amazing organizations and many more. We are very grateful, obviously, for their contributions, but also invite those of you who want to participate in building Ray to be a part of it.

This community exists because Ray has been open source from day one. And last year we took that commitment further by donating Ray to the PyTorch Foundation, giving it neutral open governance, and that matters now more than ever. Ray is co-evolving with the ecosystem, and open source is at the heart of this ecosystem. We believe an open-source stack will dominate AI-native computing, similar to what happened with the cloud and the internet eras that came earlier. Ray, along with PyTorch, open models, Kubernetes, vLLM, and other great open-source tools, are coming together in a stack that drives the majority of AI compute spend today. However, the stack is incomplete without also tightly integrating with the hardware it runs on — CPUs, GPUs, TPUs, and many other accelerators that are hitting the market every day. You'll hear a lot more from us and other leaders about how Ray is integrating more deeply with the AI compute ecosystem.

And this is also the area where Anyscale's business strategy is evolving rapidly. Give me a couple of minutes here to update you on where we are with the business. As demand for AI-native computing continues to skyrocket, compute availability has become a big challenge. This is why we embraced last year a multi-cloud strategy to deliver Anyscale across all the major providers of AI compute, as a way for you to deploy Ray workloads wherever you have compute capacity. Since then, in all our conversations with you, we also heard clearly you want to see compute built into the Anyscale platform and to provide a full-stack solution that is vertically optimized. So in addition to having a multi-cloud platform, we already started delivering compute built with Anyscale to several customers along with our existing cloud and NeoCloud partners. But this is also why we chose to come together with Nscale to craft this vertically integrated AI-native computing stack that is jointly optimized across every layer of the stack, from power all the way up to tokens. Many of you here are probably wondering what this means for Ray and also for our multi-cloud strategy. Rather than just hear only from me, it is my pleasure to also bring Josh Payne, founder and CEO of Nscale, to share his thoughts here with you. Josh, welcome to Ray.

Awesome. I remember distinctly the meeting I had with your team here at Ray Summit last year and could not have imagined you'd be here sharing the stage with me. So it's fantastic to have you here. Let's kick it off. When we met and since then, how do you see Anyscale and Ray fitting into Nscale's vision overall to create this vertically integrated cloud?

Awesome. That's fantastic to hear. Now, you heard us talk about Ray donated to the PyTorch Foundation last year as an important step towards open governance. What does it mean from Nscale's commitment to open-source standpoint?

More resources are always welcome, Josh. Thank you. And as Ray adoption continues to grow, what do you see as the opportunity for the ecosystem and the developers building on top of Ray to take advantage?

And to wrap it up, any final words for all of us here?

That's exciting to hear, Josh. Yes. Give it up for Josh. [applause] I think the opportunity to build a generational platform with the new requirements, bringing software and hardware together in a new stack, is one for the ages. I'm looking forward to doing it with you. And talking about a generational platform, let's bring up Robert, creator of Ray and co-founder of Anyscale, to share his thoughts here as well. [applause] [music]

As Keith showed, the advantage our customers have been able to achieve through these learning loops. This is about taking custom models, adapting them with proprietary data, embedding them directly in products, and continuously improving them. And thanks to GLM 5.2, the broader world is realizing the potential of this picture. But this is actually something that this community, many people here, have been building for years. And so I think as a group, as a community, we're quite familiar with the value of this picture. So I'm going to talk a little bit about what it takes to actually build one of these.

Every workload in this diagram is powered by massive amounts of compute. Compute alone is not enough. To turn raw compute, raw GPUs, into a working learning loop requires a platform. And as the complexity of the hardware has grown and as the importance of the learning loop has grown, this platform has become essential. The platform needs to solve two broad buckets of problems. The first is what I call workload challenges. Think about everything that one user needs to run one workload on one cluster. This is about: how do I scale my data processing pipeline to petabytes of video data, right? How do I take my training job and make sure it's fault tolerant and it recovers really quickly from failures? How do I maximize throughput for my inference pipeline or inference deployment? These are the types of challenges you have to solve to run a single workload. And then there's everything around the workload — all the problems that arise when you have many users, many workloads, many clusters. These are the platform challenges that also need to be solved. So we're going to cover both of these, and I'm going to start with the workload challenges.

And to illustrate this, I'm going to describe the software stack that makes up an individual AI workload. So in the software layers, the most familiar is the training and inference engine — think PyTorch, JAX, vLLM. This layer is responsible for running the model efficiently on the GPU, right? Squeezing the most performance out of the underlying hardware. Under that you have the distributed compute layer. This layer is responsible for solving the distributed systems challenges of scaling. Think about process lifecycle management, process placement, coordination, resource management, failure handling, data ingest, and data movement. These are the types of challenges solved here, and that's where Ray sits. Now, these two layers together are sufficient. You can use them to build and scale any AI workload. But we've increasingly been seeing a third layer of higher-level workload frameworks emerging, building on top of the lower layers, and these include all of the RL frameworks — think about Verl, Nemo RL, Sky RL, Slime, Miles — includes data frameworks, inference frameworks, and together these different layers form the software stack for running one individual AI workload. Now, if you're wondering where is Kubernetes, that will come soon once we get to the broader platform challenges around the workload. Right now, we're just talking about a single workload.

So to try to bring this picture to life, I'm going to walk through a concrete example. Okay, I'm going to choose a specific workload, and I'm going to focus on reinforcement learning because so much of the progress in AI recently is due to reinforcement learning, and a larger and larger fraction of compute is going to reinforcement learning. So I'm going to walk through the anatomy of a reinforcement learning workload, talk about what it takes to run and scale it, and why it's actually hard from a distributed systems perspective. So broadly, there are two different parts to reinforcement learning. There's training — just training the model — and data generation, which is using the model to generate new data to then update the model. And to break it down a little more, the way data is generated is through an agent coordinating closely with an environment, a reward model, an inference server. And these processes, actually these groups of processes, operate in very tight coordination. So the inference server will generate an action which will be sent to the environment. The environment will execute the action and create a new state of the world which will be sent back to the inference server, and they work in a tight loop, and rollout data is generated that way. And then when the rollout data is generated, it can be scored by a reward model and then shipped back to training to actually train the model.

And so let's talk a little bit about this flow of data. So if you look at the training side, suppose I have two data-parallel replicas, two FSDP replicas. Now, as I'm shipping the rollout data from the data generation component to the training component, it may seem like a simple point-to-point transfer — I send the data, batch it together, feed it into training. But remember, rollouts are sequences. They have different lengths. And so when you batch them together, you can end up with a lot of padding. And that can decrease the training efficiency. And so even this little step of moving data from point A to point B actually requires some global coordination to intelligently pack the rollouts together into batches and shuffle them around to minimize padding. So it looks a little bit more like this. Right? Now let's look in the other direction. I'm trying to describe some of the distributed systems challenges that arise when scaling reinforcement learning. So looking in the other direction, I've now updated my model. I want to ship the weights back to the inference server to then use it to generate new data. Again, it looks like a simple point-to-point movement of data, but training engines and inference engines shard the weights differently. So the weights actually need to be resharded as they're moved. And not only that, when they arrive at the inference component, inference itself is not just one thing. Often inference can be decomposed into a prefill stage which processes input tokens and is more compute-bound, and a decode stage which generates output tokens and is more memory-bandwidth bound. And interestingly, to get good performance now, each of these — you have two different groups of processes, each with their own compute resources, each with their own hardware requirements. To get good performance, you actually need to tune the ratio of prefill workers to decode workers. And how do you do that? The appropriate ratio there depends on the sequence-length characteristics of your query — so like input tokens versus output tokens. And that kind of makes sense because prefill handles the input tokens and decode generates output tokens. And so you end up tuning the ratios of these different process groups. But when you're doing RL training, you're often training on a mixture of different problem domains. And each problem domain may have different sequence-length characteristics. And so to really get the best performance, you may actually want multiple inference deployments, each tuned to a different profile of sequence-length characteristics, right? And of course, if you have multiple inference deployments, that complicates your weight sync, right? That complicates your routing strategies, that complicates your failure handling.

Okay, so what did we just illustrate? We just talked about some of the major components and movement of data between different processes in an RL workload, and we mostly described the distributed systems challenges that arise from coordination across different components. There's actually a large set of challenges we didn't touch on at all, which is managing the complexity within each individual component. And we didn't talk about it, but each individual component has its own failure-handling strategies, right? Its own hardware requirements, its own topology requirements. Training will run on, you know, reserved GB300s where you have to reason about rack topology. Environments will run on sandboxes on hundreds of thousands of CPU cores. The reward model may be bursty and require more elasticity. The agent can be bound and really need to be heavily multi-threaded. So each component has its own unique set of challenges and tricks to make it scale.

So all of this complexity — some of which we talked about, but a lot of which we didn't — is why we need this type of picture. It's the value of this software stack. It's the reason that nearly every RL framework uses this stack, builds on Ray to manage the complexity of running one single workload. And all of this, again, has been in the context of what it takes to run one individual workload. So next we're going to move on to talk about the broader platform challenges that arise when you have many workloads, many users, many clusters. But before we do that, Richard and Edward are going to talk about some of the latest developments in this stack, what we've been working on in these layers. So to do that, please welcome Edward to the stage. Thank you.

Reliability improvements and investments have come largely from making the system more memory aware. This means that Ray 2.58 offers things like new flags to enable on-demand batch sizing, providing default values for memory, and also providing better worker isolation. These changes have eliminated 90% of out-of-memory issues and also spilling in internal workloads.

At the same time, users continue to push Ray Data to much larger scales. We've optimized the Ray Data execution loop for this particular matter, and Ray Data can now scale to effectively 5,000 GPUs, maintaining high utilization, which is 10 times higher than the original 2.54 baseline.

And finally, in collaboration with the NVIDIA QDF team, we've added support for GPU-native operators in Ray Data, improving cost on certain operations and certain workloads by up to three times. All these improvements and investments make Ray Data more reliable, more scalable, and more hardware aware.

And these same three themes become even more important as we move from preparing data to training models. Training is the most capital-intensive part of the learning loop, and these training runs are getting even bigger and even larger. But the newer hardware that we're getting is becoming more fragile. Software remains our best bet in bridging this widening gap.

In Ray Train, we've invested significantly in making large-scale training more resilient. Ray Train now natively supports TorchFT, which provides first-class support for fault-tolerant large-scale PyTorch training. In partnership with Google Cloud, we've also added elastic training for JAX on TPUs.

And finally, in partnership with NVIDIA, we're also designing a native health check framework native to Ray Train that will integrate with existing monitoring systems like DCGM. This is an active area of investment, and we're working actively with the community on this, and we're excited to make this available in the future.

Now, training itself is also changing. Post-training is becoming a critical workload that companies are using in order to improve model quality, reduce costs, and own their own intelligence. We've invested heavily in SkyRL in collaboration with the Sky Lab at UC Berkeley. SkyRL is a full-stack post-training library that is built on top of Ray, vLLM, and Megatron.

And one critical operation in post-training frameworks is syncing weights between the inference engine and the training engine. This is where Ray comes in. Last year, we announced Ray Direct Transport, or RDT.

This year, we're excited to announce that we've integrated RDT into vLLM and SkyRL, as well as SGLang and Megatron, as a weight sync engine. RDT can handle the synchronization of 235-billion-parameter models in 5 seconds, which is 10 times faster than the NCCL baseline in vLLM.

RDT enables post-training workloads to be more efficient and more resilient to hardware failures, and we're excited to see this component adopted across the post-training ecosystem.

Now, post-training also creates another scaling challenge, which is to run large numbers of isolated environments alongside the model. So, in conjunction with Google Cloud, we've developed and we're excited to announce Ray Sandboxes as part of Ray 2.58. And our initial implementation, which is backed by gVisor, can scale up to 100,000 sandboxes in 30 seconds. We're excited to invest more in this as we go on.

And finally, we're continuing to improve SkyRL at scale. We've validated recipes for long-horizon post-training of 100-billion-parameter models such as Qwen 3.5 and GLM 4.7. And we're lucky, extremely lucky, to work closely with a growing set of design partners including Lila Sciences and more, of whom you will hear from later at the conference today.

Now, finally, the last workload in the learning loop is inference. If training is the most capital-intensive, inference is the most business-critical. Ray Serve provides application authors with not only the flexibility and scalability of Ray, but also the capabilities needed to handle serving in production.

And over the past year, we've pushed Ray Serve substantially on scale and performance. We've made major changes to the proxy architecture and the serve controller. These changes improve both the per-node throughput and overall cluster scalability, and together this improves overall system throughput by 40 times.

In collaboration with the Google Cloud team, we've also optimized the full stack across Ray Serve and vLLM, integrating HAProxy, enabling direct streaming, and also landing a new Ray executor backend in vLLM. These improvements improve throughput on decode-heavy workloads by up to 25 times, allowing Ray Serve to match native vLLM implementations.

And finally, we've integrated NVIDIA Dynamo's KV-aware router, which enables Ray Serve to incorporate live KV cache events across the cluster to make better routing decisions.

I'm also excited to announce that Ray Serve LLM is GA as of Ray 2.58.

I've covered a large number of announcements, but if you want to dive into details, I really recommend you taking a look, checking out the deep dive sessions, the breakout sessions after the keynote.

So, there's a clear takeaway as we wrap up. As the AI revolution continues, reliability, scalability, and hardware awareness continue to increase in importance. But as Edward mentioned, we're going to be continuing to invest in these directions as part of the Ray journey ahead.

Now, Edward and I have talked a lot about the Ray workloads, but there's a whole new set of challenges as we continue and have more users scale out to use Ray and have Ray support as part of the platform. And to share more, I'd like to welcome Christian, Field CTO at Anyscale.