Great to be here with you, really excited to chat a bit.
Yeah, likewise. Maybe we should start with some intros.
Robert, do you want to start us off? So I'm Robert. We got our start, as Ben said, building Ray, building open-source tools for scaling compute-intensive AI workloads. We observed over the past decade that AI has just been growing more and more computationally intensive. Everything used to run on a single machine; today that looks very different. So we started building open-source tools. There are obviously a lot of hard systems challenges when it comes to scaling training or inference or other data processing across thousands of machines or many GPUs, and we wanted to try to provide better open-source tools to make that easier to do. Ray has come a long way. We spent a lot of time investing in building it and continue to do so. Now it's used by companies like Uber, which trains all their models, Pinterest runs and trains all their models, and all across the tech industry it's really come a long way. So that's a little bit about me, and we started Anyscale to commercialize Ray.
Cool, great to be here. So my name's Mark. I'm VP of Research and I lead an organization called Frontiers at OpenAI. For those of you who don't know, Frontiers' mandate is to really figure out the missing paradigms on the way towards AGI. There are a lot of companies which have figured out pre-training or post-training and are expanding on those paradigms, but there may still be some things that are missing. For me, the two big themes of work here are first, multimodal, and second, reasoning. We've been doing a lot of pushes in these directions. I joined OpenAI about six years ago, spent half my time there as an IC and half my time leading organizations as well. One cool thing is Robert and I go way back. I think we've been friends for a decade since college.
High school, yeah. I'm curious to get your take on what are the biggest breakthroughs that you've seen over the last decade. And before we dive into that, you should also say you and your team are responsible for a lot of models that people here know about. You want to mention a few of those?
Sure, sure. Yeah, so I mean, multimodal, we've worked on it for a long time. I've led efforts around DALL-E 2, DALL-E 3, the recent voice experiences that you may have played with, GPT-4, the multimodal aspect of that.
Awesome. But yeah, so we've been working in AI for a long time. And of course, year to year sometimes a lot of the things look similar to the previous year, but over the span of a decade things have changed a tremendous amount. One of the biggest changes over the past decade in AI—I don't know how many of you took an introductory machine learning course in college or something like that—but the course often starts out by framing machine learning as consisting of supervised learning, which is like learning from labeled data, and unsupervised learning, which is like learning from unlabeled data, and reinforcement learning. The courses would often say, hey, we know how to do supervised learning. Here are a bunch of algorithms like decision trees and support vector machines and stuff. And unsupervised learning, we don't really know how to do that. And they would talk about K-means clustering and various other clustering algorithms. And that has totally changed. I would say one of the biggest differences now is that we do know how to do unsupervised learning.
Yeah, totally agree. I mean, I think unsupervised learning used to be this kind of pipe dream. Starting around GPT-1, GPT-2 times, that's when we kind of realized data is one of these big ingredients here. You don't have to play by the rules of academic datasets where you kind of say, oh, here's the dataset that I have to train on and get good performance on. It really opened the door for just collecting as much data as you could get and finding a lot of alpha in the datasets and really going and trying to learn almost the broadest distribution of tasks, by which kind of it follows that any specific task is just a small consequence.
Yeah, and the way people define tasks or think about progress in AI has changed a tremendous amount. I'm sure a lot of people here are familiar with the ImageNet benchmark. It's a computer vision, an object recognition benchmark which drove a lot of progress in computer vision over the years. The sort of way people went about research was you have this dataset of objects and labels, you divide it into your training data and your test data, and that's static. You don't change the dataset. And what you do as a researcher is you try to design a better neural network architecture or optimization algorithm to train on that training data and then hopefully you get a better result, a better accuracy on the test data. That did drive a huge amount of progress, but it's also very different from how people approach it now. Today, the dataset is the key variable and the optimization algorithm and neural network architecture are a little more static.
Yeah, I mean, I would say all of these things are still extremely key. You have to work on your data stack, you have to work on your systems, you have to work on your optimization stack. They're all kind of key ingredients. And I know actually Anyscale's done quite a bit of work here. How do you kind of think about helping people scale their models?
Well, one thing that we've seen is just a huge increase in the engineering effort behind AI. And of course, you guys at OpenAI really identified a lot of these empirical scaling laws. The idea that by putting in more compute and more data and training bigger models, you can get better results. And so for a lot of the companies that we work with, they're looking at those results and seeing that if we want to, the formula for getting better results is to train on more data. And their North Star is to enable training on more data. But of course, if you're trying to scale up the data by an order of magnitude, you're going to break a lot of your systems and there's a huge amount of engineering work that goes into enabling that.
Yeah, I think one thing Robert touched on is just one thing that's really important these days is engineering in a way that it wasn't important maybe a decade ago in research. And really this stems partly also from our belief in scaling laws. If you believe in scaling laws, I think you believe in scaling and you really do have to build out an engineering stack. Part of the science there is also being able to extrapolate from what you see in small models to behavior in large models. And I think there's been a lot of interesting and exciting research on that front.
Can you say more about that? We talked about the engineering, but really the science behind it.
Exactly. So I mean, OpenAI has done a little bit of work in this vein too, right? Just kind of being able to set hyperparameters and do a lot of experimentation at small scale and really being able to see that same behavior carry forward to larger scale. A lot of this has to do with how you parameterize your neural networks. I think there's also been a lot of intuition of like, how do I initialize my networks or what kind of behavior do I observe in terms of like activation growth or different parts of the network, like what's healthy behavior. The more you can kind of build intuition at small scale, the easier it is to train your larger scale models.
Yeah, and this is sort of a natural progression of this area of hyperparameter optimization or hyperparameter tuning. Back in 5 years ago, 10 years ago, when people thought about this idea of hyperparameter optimization in machine learning, the research was more about doing random search versus grid search versus Bayesian optimization and other techniques. And the discussion was less about how do we extrapolate from a small compute budget to a large compute budget.
Yeah, so that's progressed quite a bit. And I would say research today is just really hand-in-hand with engineering. It's like when you're kind of in the business of scaling models, it feels like every order of magnitude you scale, you're going to hit some kind of research bottleneck, some kind of engineering bottleneck. And it presents problems on a lot of different fronts that you have to solve. And that's really the excitement of it. There has to be some kind of technical breakthrough you make, there has to be some kind of breakthrough you make on engineering your systems.
Yeah, so we talked a little bit about progress that's happened over the years, but what do we not know how to do today? What are the open questions?
Well, I think I may have a very biased take here, but I work on multimodality, I work on reasoning. In terms of multimodality, I think there's still a lot of open questions. Multimodal is one of the things over the last five to 10 years that's really started working well. If you look at generative image models today, generative video models today, they're working quite well. And I do think there's kind of this dual way to see the open problems in multimodal today. One is you can kind of take language models and be like, we've figured out so much machinery here, how to scale it, how to train these models, and how do we kind of fit multimodal into this language paradigm. That's like one way to frame the problem. Can we take multimodal problems and try to fit them into this Transformer stack? Another way to see it is in the multimodal space things are still very much up in the air. There are models like diffusion models, GANs are still fairly popular and still quite efficient. And is there any way to kind of bring language over and kind of fit it in this multimodal framework? And there are two kind of exciting lenses. I think there's this back and forth bridge of ideas and that's been very exciting to work in.
Yeah, so when I think of multimodal data, I think of text, images, video, and audio primarily. Is that how you think about it? Are there other important ones?
Yeah, I mean, that's a really good question. I guess, yeah, I do think of these as kind of the juiciest, biggest modalities: text, images, video, audio. Certainly there's other modalities out there. There's been a lot of work on 3D, we've done some of that work as well. You can imagine kind of even like actions as modalities or something. But yeah, I do think of these as kind of the big ones.
So since you're in charge of multimodality, are you going to solve these modalities like one at a time or are you going to just figure out a general-purpose way to incorporate new modalities?
It's also a very good question. So yeah, I think when I think about multimodality, even though I think they can share a lot of structures in terms of how you model one modality versus another, there's also fairly important differences between the modalities as well. When you look at something like audio, there are different demands. When you look at our advanced voice mode experience, really the expectation from the user is that you want to be able to speak to it with real-time interaction. And that could be very different from something like image generation where really what you care about is a very high-quality image and you're willing to wait for that. So I do think even though you can share a lot of structures, it does still kind of bear fruit to think about these modalities separately and to kind of design different stacks for them.
Yeah, and you mentioned incorporating other modalities into the language framework or incorporating language into the others. You're singling out language here. Is that a fundamental thing or is that just where we've made the most progress so far?
Yeah, so I mean, I do think it's mostly the latter. We've kind of converged on a pretty fundamental recipe for training and scaling language models. And it does feel like there are different flavors of that, but people kind of do understand this recipe and it is kind of interesting to just see if you can fit things into that recipe. But I'm actually kind of curious for the dual problem too. Just like when you look at industry, what are the challenges that people are coming to you with? Are they multimodal challenges? What kind of problems?
