Robert Nishihara, thank you for chatting with me.
Yeah, thanks for having me.
So, you've been coming to NeurIPS for a long time, since before it was called NeurIPS, I think. When did you start coming and what's it been like watching it grow up?
Well, when I first came to NeurIPS, it was a 400-person conference. Today, I think it's 30,000 or something massive. It was a small, very warm community where everyone knew each other. It was always located somewhere you could ski because the organizers loved skiing. For the workshops, they would have a huge break in the middle of the day so everyone could go skiing.
This was around the advent of deep learning. It exploded over the next one or two years to a 10,000-person conference, and tickets would sell out overnight. You just couldn't get tickets. The conference organizers weren't able to scale fast enough.
Yeah, because they had the constraint that it had to be located somewhere you could ski, so you can't just sit anywhere. For a number of years, it was just in Montreal over and over, and it was freezing. Were the early days, pre-GPT breakthrough, a scrappy, contrarian mentality? As a PhD student in systems, I did some AI and ML libraries, but it wasn't a hard contrarian bet. There were big systems that had changed the world, but AI hadn't fundamentally changed the world like it has now. Have you seen a cultural shift from 'we've made it' to that core group of 400 still getting together in a little room here somewhere?
It definitely feels different. Your point about it not having changed everything yet—this is something Yann would always say: there's no big AI company. You'd look around and there wasn't a big new AI company the way there were big new internet companies. But now there are. Ion always says that if we had started Databricks a year early, we would have been too early, and if a year late, too late. A great company is going to happen; it's just whether you're the one. Same with Perplexity—there were two decades of dead bodies of companies that tried to disrupt search, and Perplexity made the bet at the right time.
I think that's a big factor. We see this with Ray. When we started Ray, if you were training a single model on a small model on a single GPU, like SGD or logistic regression, you don't really need Ray. So you were making this crazy idea of trying to sell everybody that the future is scaling AI, and they're all like, 'I'm pretty much fine here with my CPU.' Or it seemed like a niche thing. They'd say, 'Maybe Google needs that, but does anyone else need distributed computing for AI?'
You called it early and being the first.
We would always reason from first principles: do we think AI is taking off? Yes. Do we think there's a growing need for scale? Yes. Do we think that introduces hard systems and infrastructure challenges? Yes. And do we think one of these years the doors are going to blow off? And now they have. Congratulations, that's awesome.
Yeah, it's funny. We didn't know if we were too early or too late or right on time, but now looking back, we know.
You went to Berkeley, got your PhD not too long after I got mine. I was in RAD and AMP Lab; you were in RISELab. I worked on the tail end of AMP Lab, so we overlapped. You were a machine learning PhD student with Mike Jordan and also worked with Ion Stoica. I built a project called Spark, focused on data, and that built up to some of the insights you all had with Ray. Maybe you can tell us what Ray is and the history of how Ray has had impact. I know it was used most recently with Cursor's coding model and for a bunch of other iconic AI stuff. So tell us how Ray has evolved, what it is, and where you see it going.
I started doing a PhD in machine learning, doing RL research, optimization algorithms, deep learning training. I never set out to build infrastructure; I didn't think of myself as a systems person. But it became clear early on that we had no other choice. AI had different requirements. We weren't going to take off-the-shelf systems built for previous eras and retrofit them for AI. AI needed something new. At the same time, we had all seen what happened with data infrastructure with Spark and Databricks, and we thought AI infrastructure seems like a 100x bigger problem and a bigger opportunity. So we got started.
Congratulations. So you bet very early on the growing need for scaling compute for AI. That was the premise of Ray: you need distributed computing for AI. You started the open source project. It was an interesting time because most people doing AI did not need to scale; you had a lot of recommendation models trained on a single machine. Early on with Ray, we worked with early design partners like Ant Financial, Uber, Pinterest. It's really over the last year, with the explosion of not just generative AI but reasoning models, reinforcement learning, very powerful multimodal and open-weight models, that the need for scale has exploded, and the value of AI for businesses has exploded. We've seen Ray adoption explode similarly. You mentioned Cursor used Ray to build Composer, their frontier coding model. xAI uses Ray for all their image and video processing. Companies like Thinking Machines, Runway, Physical Intelligence—all different domains use Ray as part of their tech stack for scaling compute-intensive AI workloads. I think one of the things you see is a renewed emphasis on infrastructure for AI, not just hardware but software infrastructure, because all these companies care so much about velocity. If you're building a model, you don't want to be a model-building company and have the eighth best model. Or if you're an established business, you don't want to be disrupted by new AI companies. So everybody's concerned with velocity, and one of the key things you need to move fast is great infrastructure that's reliable and enables productivity. The trajectory with Ray has been that the problems Ray was designed to solve are really the problems everyone has today.
Specifically, if you boil it down.
Reliability at scale is the core thing, but I would say there's a huge amount of complexity in AI. A lot of that is related to scale, but a lot is just growing complexity. You have more heterogeneity of hardware accelerators, mixed CPU and GPU compute, TPUs, many other hardware accelerators. You've got more multimodality of data: you still have tabular data and log data, but also tons of image, video, audio, and robotic sensor data that people didn't do much with before because it's hard. But now that's becoming possible. So growing complexity in terms of scale of compute, scale of model size. In the past, when you would deploy models, you'd put the model in a container and replicate it. Now a single model will span multiple nodes, with very complex sharding strategies for different layers. You might replicate attention layers and shard expert layers, and a single query can get routed in complex ways through different GPUs. You have more agents to wrangle. So there are many different dimensions of growing complexity—managing greater scale but greater complexity in many dimensions.
Yeah. Maybe you can dive deeper on one thing: is Ray a generic enough layer of infrastructure? Are you building other infrastructure next to Ray to handle such a diversity of needs? You were talking about inference, training, managing agent processes that would be CPU in addition to GPU. Is Ray evolving to handle such a wide diversity of infrastructure needs?
