All right, so welcome to the Generative AI in the Real World podcast. Today we have Robert Nishihara, co-creator of Ray and co-founder of Anyscale. So Robert, welcome to the podcast.
Thank you so much. Good to see you again.
So we'll talk obviously about Ray. For people who are not familiar, Ray is an open-source project that came out of Berkeley that has become the default AI compute engine, the default foundation layer. I've been writing about Ray for many years, and recently I wrote a series of posts about how I'm observing companies building custom AI platforms, and a common substrate seems to be Ray across many companies. The reason I bring this up is because Robert then has a great perch because he gets to work with a lot of companies that are on the cutting edge of AI but are also tackling some of the hardest problems in AI. So I wanted to pick his brain about some key AI trends. The first thing is about data. We hear that people are running out of data, the teams training these foundation models are running out of data, but also that the data tools themselves are lacking because everyone is focused on training and not enough on the data tools. So give us, Robert, your high-level view of the state of affairs as far as data in AI.
Yeah, well, the importance of data in AI is something that many people have identified and have known for a while, but this is actually a really big paradigm shift in how the field of machine learning has been thinking about AI. If you go back a decade to the ImageNet benchmark, ImageNet is one of the famous computer vision benchmarks where you have tons of images and you try to build models that can classify the images into different categories like cars, cats, dogs, trees, and so on. A huge amount of the progress that this benchmark drove in AI was really around building better model architectures, trying to innovate on the optimization algorithms to take your ImageNet dataset, train a model, and ultimately do better on object recognition. But the interesting thing was that in this benchmark, the dataset was considered static. You take your ImageNet dataset, split it into train and test data, then train your model. All the innovation is on the model architecture. Now we're in a different paradigm where the model architecture is not completely static; there's still innovation there, but Transformers are very common. There are a few standard model architectures at this point. The optimization algorithm is much more static; we're all using variations of stochastic gradient descent. And where all of the innovation is happening is on the data side: acquiring different sources of data, evaluating which sources of data make sense to train on, curating the data, generating synthetic data, extracting, filtering out low-quality data. So I actually think that especially given how important the quality of the training data is to impact the quality of the resulting model, the way people are thinking about curating and acquiring training data has really shifted over the years. Back to ImageNet, the kind of data preparation or processing that people did was very limited; you might crop the images, scale them to augment your dataset. Now you're using AI to really process and curate the data. That's the big shift. For example, imagine you're working with a bunch of video data. You want to filter out low-quality data. How do you do that? That's an AI task. Or you're working with an autonomous vehicle company; you have self-driving cars with millions of miles of driving, tons of video footage, and you're trying to find the most interesting data to train on. A lot of the data might just be driving down an empty road for a long stretch, nothing interesting happening. Some of the data is a pedestrian doing something crazy, something you've never seen before. Not all data is created equal; some data is far more informative or valuable than others. So a lot of the process is taking all of your data and extracting the most informative, most interesting, highest quality data to train on, and that is very much an AI task.
There's a lot here. And Robert, for the most part, a lot of the tools, at least in the resurgence of data engineering tools five years ago, were centered around pipelines and orchestration tools for structured data mainly in data warehouses and lake houses. So now we have unstructured data, but increasingly, as you point out, multimodal data. So what's the challenge as far as the tooling around working with that same data and trying to do the same sorts of data processing and data prep that people are accustomed to?
Lots of companies have lots of data, and they've acquired all of this data and are storing it because they believe they can get insights out of it. They want to get value out of it, make decisions. The way they get value out of the data today is by running SQL queries. They'll run SQL queries, run some simple analytics to learn about their data and explore it for structured data. Not because they don't have unstructured data, but because you can't really touch that with your SQL queries. Your SQL queries are not going to operate on PDFs or videos or images or arbitrary text documents. But the reality is that what you can do with SQL queries is quite limited. The amount of insight you can get — imagine I'm a company trying to get insight about my sales pipeline. The kind of SQL queries that are easy to run are things like asking which companies have been in certain stages of the pipeline for certain durations, pre-built fields and categories that I've already thought about. But all of the insight is in the notes that the sales representatives are taking, in the videos of the recorded meeting calls, all of these kinds of things, and you're not getting any value out of that today. So I think the way that people will get insights out of data in the future is with AI. They'll have AI read the data, reason about it, and draw conclusions. That is a very different paradigm. So I imagine data processing shifting from being a SQL-centric workload to an AI-centric workload. And if it's an AI-centric workload, it's going to be a GPU-centric workload, or perhaps mixed CPU and GPU workload. The tooling around workloads that are very GPU and AI intensive and very data intensive is practically non-existent, or it's very immature. This tooling for multimodal data processing is a challenge.
Yeah. In a future pipeline, you can imagine — I'll draw a few examples. I have a piece of text, then I remove personally identifying information, then I correct grammatical errors or translate foreign terms, then I embed. So there's this complicated pipeline that is not really going to be addressed by the data engineering tools from the structured world.
Yes, you're going to have AI to do that task that you just described. And then secondly, as you point out, in certain parts of the pipeline maybe you can use CPU instead of GPU. So to say that we're now in the AI world, it's unstructured, I'm only going to use GPUs for data processing — I don't think that's going to be economically feasible.
No, I don't think that's feasible.
Data processing is not just running inference with an LLM or with some model. There's lots of other regular processing. If you have video data, you might want to decompress the video, re-encode it in formats that are better for downstream consumption, find scene changes between different clips, do transcription, run a vision language model to generate text descriptions, run a lot of classifiers to annotate the video and extract structured information. Some stages will be GPU bound, some will be memory bound because videos are massive, some will be CPU bound. You're going to want to move toward an architecture where you can disaggregate these different resources. If GPUs are the bottleneck for a certain stage, you can scale those up. If CPUs are the bottleneck, you can scale those up. And of course you want to be able to stream all of the stages of processing together so that you can keep your GPUs busy the whole time. You don't want to just do one stage, wait for it to finish, then do the next, and so on. All of these things are areas where the tooling has a long way to go. And in the types of data you're describing — a billion images or a few hundred hours of video — it's almost assumed that you will have to go distributed and scale out. There is no choice but to really scale the computation. The data is not going to fit on a single machine, and you're not going to be able to process it in any reasonable amount of time.
So everything you've discussed so far around the data challenges around multimodal data — is this a real problem today, or are you forecasting for the future?
This is something people are running into today, and a lot. In the past, we were really mostly getting value out of structured data because we had the tools to process and query structured data. We weren't really thinking about what we could do with unstructured data, with multimodal data. Videos — I keep mentioning video as an example because it's a type of data where there's so much value, and yet we are not even scratching the surface of what we could be doing with video data. Previously, you couldn't get much value out of video data or images, so people weren't collecting it as much because there was less incentive. Now that generative AI is really unlocking the ability to get value out of all these different types of data, we're going to collect a lot more data because we're getting more value out of it. When we talk to companies, many of them are looking at the scaling laws, seeing that if they train a bigger model with more compute and more data, they can get better results. That's almost a formula for getting better results. There's subtlety to it, but it gives you a roadmap. So all of these companies are saying, 'We need to enable training on 100x more data,' because that's the path to getting better results. The challenge is that you didn't build your internal systems to handle 100x more scale. You don't build that far in advance. So now, to enable building the next model and shipping the next AI feature, all of these ML platforms and ML infrastructure are on the critical path for building and shipping that model. And it'll be on the critical path for the next model after that because you need even more data. So we're finding a lot of responsibility and stress on these ML infrastructure teams who are now on the critical path for delivering these kinds of results.
Actually, as I was asking you that previous question, I started remembering at the last Ray Summit in September in San Francisco, there were two talks that caught my attention. One was from ByteDance, where they were processing petabyte-scale data, mainly audio and video pipelines. The other one was from Pinterest, for model training for their large-scale recommender system. Again, they had to do this dance between CPUs and GPUs. But I guess my question is, both companies I just named are super tech-forward companies, digital-first companies. Do you imagine what you're describing is something that regular enterprises will also have to grapple with, maybe not today but soon?
There are always companies that are early adopters. When it comes to AI, a lot of the market is still very early. A lot of companies are very much in the exploratory phase of figuring out how to use AI in their business: should they build a chatbot, build internal copilots, how do they really incorporate AI in a native way into their products? Companies like ByteDance and Pinterest are very much at the cutting edge, at the frontier of running AI in production. They've been doing this for a long time. If anyone knows how to work with multimodal data, it's going to be these companies because of the nature of their products. That said, every company, or nearly every company, has a lot of data — its own data, data about its specific business and how its customers use its product — that they can use to improve their platform. So this is something that every company is going to want to do: get insights out of data, use data to make decisions. The value is there, the potential is there. The challenge is just the tooling and infrastructure to really make that possible.
There's another interesting angle around data at scale that ties into this new world of really large-scale AI models: experiments. For example, you're building an application, an agent, or retrieval-augmented generation. The reality is you're probably going to have to play around with different knobs and run experiments. Some of those experiments may start early in the pipeline, like how do I chunk, embed, and pre-process my data? You can probably use the default settings, but you're likely going to be leaving a lot of insights on the table. So all these data tools that Robert is describing seem to play into that: if you want to do AI well, you have to be able to run experiments, and data processing and preparation is part of experimentation.
Absolutely. The customization is not just at the level of the model, fine-tuning the model, or the prompt. Those are important things that require experimentation, but there are so many decisions to make at every stage of the pipeline, exactly like you said: from what data to collect, how to segment or chunk the data, how to embed it, what embedding functions to use, how to do retrieval, what to stick into the context if it's a RAG application, how to rank the context, what model to use, how to fine-tune it, what data to use for fine-tuning, whether you fact-check the outputs of the model. There are so many decisions to make. In order to really iterate quickly, you need to be in a position where you can try out different choices and evaluate how well they work. That means we often recommend companies over-invest in evals early on, do all the steps by hand, evaluate not just the end-to-end thing but individual stages of the pipeline, because that's ultimately very important for moving quickly as you develop.
And if you don't have the right computational foundation, these experiments will actually be impossible to pull off.
