Greg Brockman on Founding OpenAI and Systems for AI | Ray Summit 2022

Good to be here.

In 2015, we had this dinner down in Palo Alto, not too far from here, with a bunch of people, some of my co-founders. We had this question: is it too late to start a lab trying to build artificial general intelligence? It felt like AI is clearly going somewhere, maybe you can even get to the point someday that you can build a machine that's as smart as a human, and it's clear it's going to change everything. It was a question of: do you think that people in this room could start a company to try to make it go well, or is it already in the hands of the players that are out there? I think we all kind of looked at each other and said it's not obviously impossible, maybe it could be done. So the next day I was full-time trying to put together a team and make it happen. That's our mission. Our goal is to try to make artificial intelligence advance in a way that's most beneficial to humanity. We have a research company that we've been pushing forward this technology and deploying it to lots of users.

I think that large models are one of the most surprising things in terms of what they can do over the past five years. In some ways, it's not too surprising if you rewind to the beginning of this whole deep learning wave, back to the 60s and 50s when people were first starting to think about building neural nets. It's always been about bigger computers and bigger networks. But the thing that's been really magical is that if you take a neural net and train it on huge amounts of data with huge amounts of diversity, you basically imagine training a model on all the text data on the internet. Then it can actually learn to generate text that is almost indistinguishable from human-written text, and use it for tasks that any sort of language-based task your company might have. What we've seen is that if you train on a very wide dataset rather than a narrow one, you end up with a model that's more capable towards any of those individual tasks than if you had just done the narrow thing.

There are two classes of model right now that we've deployed, and other people have related models as well. There's the GPT family where you pre-train on text, so you literally just say 'here's some text, predict the next word,' and then you're able to do any sort of classification, text, joke writing is starting to work. Actually, at a wedding I was at recently, half the speeches were co-written with our GPT models.

There's some good ones about how many OpenAI engineers it takes to change a light bulb.

Exactly. The second class that people are starting to see that's really taking the internet by storm is text-to-image generation. The thing that's so interesting to me is that at the end of the day, we're just having almost the same neural network architecture. There are some differences, one uses diffusion, one uses an autoregressive objective, but it really is just that you show it a bunch of bits and say 'please could you output a bunch of bits.' This is just a bunch of examples of humans doing it, and now do the same thing yourself. That task of being able to predict what comes next is something we've been able to make succeed. So now you can ask for a dog playing chess on the moon, and you'll get a picture of a dog playing chess on the moon, and it will get all the context right, like the shadows and the dog on the moon probably needs a space suit. All these concepts that we always wondered how you're supposed to get inside a neural net are emerging from having all this experience baked into the model.

We have an Instagram, you should check it out. It's pretty fun, there's just all sorts of variety in there.

I always gravitated to infrastructure because you get this leverage out of building at this lower level. If you're the person who supports everything, you can claim credit for all the great things people do on top of you. At OpenAI, when we started out, we looked around at the open source ecosystem rather than having to reinvent all the wheels we had to at Stripe. There were things like Kubernetes and Terraform that made it so we could just move faster, we didn't have to build a bunch of management for the server infrastructure, building on top of AWS and all that. So there's a question of at the actual application scheduling layer, what do you do? We first had a bunch of projects that were handwritten, you have a Redis and you write your own little protocol for moving things back and forth. Then we rediscovered MPI, a piece of technology and protocol that's existed for decades and is used in a lot of HPC. MPI for a long time became the way we would have processes coordinate, launch them all, and have them talk to each other. At one point we ended up with a system where we had MPI for some core stuff, but then you really find you want to flexibly talk to your model, make requests to it, run some inference on this pass, do some training on that pass, hook up the result from this model to that model for reinforcement learning. So we built another layer on top of MPI that we called Rook, where you basically had a pickle TCP stream and then you start doing things like broadcasting a request to all the workers. You can see there's a lot of complexity that was homebuilt, and all these features we never added, like being able to make multiple requests at once, we had no story for that.

It was the bare minimum amount of investment we could make and still not be unhappy. It was definitely a decent amount of work. It's one of those things where if something's not your core competence, if you really just want to be training the models, you're thinking 'why am I shuffling around the bits and dealing with a TCP stream with pickles in it?' I just want to send this request from this model to that one, I want it to happen.

That's not a burning passion.

We were doing a big rewrite of all our infrastructure, really trying to go through all the details and see if there were better ways to architect. When it came to the system I described, we wanted to wire processes, have them launch, know if they broke, be very flexible, get back exceptions, all those things that sound like table stakes. We did a survey of what's out there, looked at Dask, Ray, probably half a dozen other projects. Ray was by far the winner.