First Public Talk on GPT-3 by Tom Brown

Language models are two-shot learners, popularly known as GPT-3. I was one of many authors who contributed to this, and this project was a huge undertaking with almost all of the OpenAI team and all the sub-teams within OpenAI. So I think all credit goes to these people, and any mistakes during this talk I take full responsibility for, so bear with me.

So first off, what is this talk going to be about? Basically, I'm going to talk about what led to GPT-3, what the actual GPT-3 paper is about, and I'm going to talk a little bit more about what might happen next. David and Adam gave me permission to wax philosophical about what might happen next and further into the future, so if you're not into that, just feel free to mute and drop off. I won't be too offended. But if you stick around for that, I think it could be entertaining too.

Step one: what led to GPT-3? Well, unsurprisingly, GPT-2. You can tell from the number. But there was a particular thing about GPT-2 that was kind of interesting, which was that GPT-2 talked about this zero-shot behavior. So you might recall this figure from GPT-2, which showed four different tasks that were being done zero-shot with GPT-2. This was pretty interesting and cool, but something that careful readers might have noticed was that the baselines here were things like embedding the nearest neighbor for a translation, and you all know about translation—that's not a super strong baseline. And if we look over at summarization, you might see random three. For people who work on summarization, random three is you're trying to make a summary of a big block of text, and random three is the cutting-edge technique of taking three random sentences from the whole long article and just saying these are the summary. So GPT-2 had some pretty bad zero-shot performance. I think this was something where, if I talked with Alec about GPT-2 when he was writing it, he said that this was actually something he really struggled with writing this paper, because Alec is the machine learning engineer par excellence, and he is known for making things really great. I think he really loves squeezing every last bit of performance out and doing really great on a task. So it was hard for him where GPT-2 was starting to do some of these things, but it was just really terrible at them. So GPT-2 wasn't really about being impressive on these tasks; GPT-2 was about making some predictions. In particular, it pointed out that yes, it's still terrible at these tasks, but it's going up and to the right, so it's starting to get better. It was kind of making this rough prediction that if we continued to make these models bigger, then they would get better at some of these tasks. It didn't have a lot of evidence behind that, but it was pointing at this and kind of waggling its eyebrows suggestively.

So this was kind of what GPT-2 was about. And after GPT-2, as we were starting to get involved in the beginnings of GPT-3, a different group was putting together these scaling laws for neural language models, which I'll just refer to as scaling laws. It really was about trying to make scaling laws. So it's a bunch of physicists who are like, okay, machine learning seems like it has a bunch of different people who are mathematicians proving things very concretely, and a bunch of people who are engineers who are going to try to throw things at a wall and see what works. What we're going to try to do is kind of bridge a middle ground where we can't actually put together a super concrete theory about how things work, but we can gather a bunch of evidence and try to fit lots to it and try to make some sort of theories that have some predictive power, even if we don't know why it is that these theories work. So kind of where GPT-2 made rough predictions, the scaling laws made incredibly detailed predictions. I was only kind of tangentially related in this, but I was an end consumer of it, where many of these plots kind of informed how we ended up doing GPT-3.

Just walking through these a little bit, I like to think about the scaling laws paper from an economics point of view, where it kind of gives a production function for a Transformer. So this is like, if you have some amount of... yeah, this is like what are the inputs to a Transformer that result in a certain loss. They study these by breaking it up into the different components that you need, like parameters, data size, and compute. Here are three of the main plots from the scaling laws paper. As you can see, this parameters plot is showing how loss scales with parameters as long as you have enough data set size. For this, it's trying to represent the infinite data set size limit and when you have infinite compute. So if you just train a model for as long as you want with a given amount of parameters, what is the loss that you'll end up with? I think pretty astonishingly, they just found these incredibly good fits for a power law. As you all know, a power law is just a polynomial where you have exactly one value raised to a power plus some constant. The parameters seem like they fit well. The data set size again, this is if you have as big a model as you want and as much compute as you want, what loss can you get with a given amount of data? Then they put those together to also give, if you have a given amount of compute and you can make your parameters any size you want and you have as much data as you want, what is the efficient way to train a model given some compute? They found that this also followed this really consistent power law. So that was one kind of prediction that they made.

Another prediction that they made was, just by plotting the loss of each token, they noticed that yes, these things were going down for any given model size. So this is the smallest model. To some degree, this is basically what ended up in GPT-3 being the in-context learning. The model is picking up on a pattern as it goes along. I think this was not really surprising to people, but it was something that hadn't been studied very carefully. Also, they showed that this happens more for the larger models, and the larger models actually sweep under the big models, so the amount of in-context learning that happens for these larger models is also more than for the small models. Then I think the other thing that was quite surprising and interesting about this was that scaling laws fit the loss over seven orders of magnitude, and seven orders of magnitude is just a crazy large amount. I think that was something that was quite surprising. If you put GPT-2 here, it strongly implied that we could continue this out at least another half of the graph maybe and expect it to continue to be mostly straight.

Okay, so that's kind of what led to GPT-3. There were basically those three predictions that were made. Then what GPT-3 was about was validating those three predictions. Just to spell this out, I see there being three main predictions that GPT-3 was about validating. The first one was that the loss improves with the model size, so that was the last thing shown in the scaling laws paper. Basically, we have these very straight lines where the loss will improve with model size or amount of compute that you have. Also from the scaling laws paper, in-context learning improves with size, so you get more in-context learning or meta-learning as the model scales. Finally, a lower loss is correlated with better performance on many downstream tasks. This was something that the scaling laws paper didn't look at, but that was part of the waggling suggestive eyebrows of GPT-2, which is basically we had looked at a bunch of different tasks, and for all the ones that we'd found so far, if you did better on the pre-training loss, you also did better on the downstream task. So kind of these three predictions having been made and proven out enough, or proven out to some degree, was enough for OpenAI to be willing to invest in another two orders of magnitude in compute to do this. So the bottom of this little red block is GPT-2, and the top is GPT-3.

What we found was, looking at prediction number one, yes, as we scaled up, this is the same plot from the scaling laws paper with GPT-2, but now this is the GPT-3 version, and GPT-3 more or less follows this curve even though we've added another two orders of magnitude. Actually, this is three orders of magnitude of compute, two orders of magnitude of model size. I think going into this, my personal guess was that there was maybe a 60% chance that this would continue to follow this curve. I think the physicists had more trust in the law of straight lines, so they were more optimistic about it. But I think it was worth putting that bet into it based off of the evidence that they had before. So yeah, the first prediction was yes, that loss does indeed improve with model size and it continues up to the scale of about GPT-3.

The second one was that the scaling laws predicted this in-context learning to improve. The way we looked at that is we did this on just a ton of different tasks. You can see here that as the models get bigger, the spread between the few-shot, the zero-shot, and the one-shot starts to increase. So that's that one. But one thing you might notice is that this plot is kind of a weird plot. This is an aggregate plot of 42 different accuracy-denominated benchmarks. The original version of the plot that we had also had BLEU score confusingly averaged into the accuracy. I'm glad that we cut it to just our 42 accuracy-based benchmarks. But this is kind of a weird way of showing this information. By page count, the majority of the GPT-3 paper is just plot after plot showing each of these things for individual tasks. There's a large appendix about this, and I think that was also a large amount of the actual work that we put into this paper, just running all of these different tasks, kind of following the text-to-text format that T5 pioneered. What we found there was that really, for most but not all of the tasks, lower loss really did yield better performance on these tasks. So okay, those were the main three predictions that GPT-3 was about validating.

Now I'm going to get to the fun part, which is talking about some limitations of GPT-3. There's too many to fit on one slide, so this is part one. First thing is that it's incredibly expensive and inconvenient. The main innovation here was just making the model very, very big. The two implications of that are you have to spend a lot of money to train it, you have to spend a lot of money to do inference on it, and you need a crazy distributed system to run it. All of that is just very raw right now. It also had a quite outdated architecture. We were working on this as T5 came out, and I think T5 is the best evidence that our model actually is using a totally outdated architecture and that it would do much better if we improved the architecture. I appreciate all the help that Colin and Adam gave over email as I was trying to figure out ways to improve this. T5 did a great job of sharing best practices with the community. Finally, GPT-3 is just really bad at NLI and kind of NLI-like stuff. You can see here it's just flat, random guessing. I don't know why that is, but that's that. All these things I think will take work, but they definitely seem solvable.

So let's get to limitations of GPT-3 part two. Here I'm going to start to get a little philosophical. GPT-3 is totally not interpretable, so we can't figure out what's going on inside this crazy black box that we have. It's not robust; it gives completely miscalibrated predictions, and if you get it out of a domain, if you get it going down a weird path, it'll do very weird and unexpected things. Of course, it retains all the biases of the data where it's been trained on the auto-regressive language modeling task, and if you want to use it for something else, it's not going to be into that. What it wants to do is just predict the next token correctly. As you might notice, yes, these are all limitations just of deep learning in general, and unfortunately they probably won't be solved by GPT-4.

This brings me to the future section, which is just what might happen next. I'm going to talk about what might happen in the next five to ten years. The short answer is that I don't know. What I'd like is for all of you guys to make your own predictions. This is something that I found useful to do every couple of years as an exercise: just maybe set aside two hours, get a cup of coffee, quiet space, and just write down what you think will happen with AI. I think that's actually something that is pretty rare for people to do. Then I think something that's even rarer is to write down what would change your mind, because then you actually have a trigger to notice if something actually different or weird has happened with AI, and that maybe it would be time to actually update what you think is going to happen. If you do that, just email me what your predictions are at [email protected]. I'm just very curious about this, and I will guarantee that I'll read them. I don't know if I'll have any useful feedback, but I really love thinking about what different people think is going to happen because I think there's a ton of uncertainty.

I think one thing that I am pretty certain about is that the stakes are becoming high. It seems like there's way more resources now being pushed into AI, and it seems like AI is likely to permeate every aspect of our lives within the next five to ten years, unless something greatly updates what I think. So I guess this is what, to paraphrase Uncle Ben, with great parameters comes great responsibility. I think that we have a chance as researchers now to do some things to help out. One of those things would be what I described before: make your own predictions, come up with your own ideas about what you think might happen, share those things, and be willing to update them as things change. I think we're kind of living in the future a bit, and we have a vantage point where we have some perspective on what types of things might happen that other people won't. I talked about three big limitations that I don't think will be solved by GPT-4. I think those are things where I'd like to see more people working on them. Since those are such big limitations and they're slowing down everybody, if anybody makes progress on them, everyone would be useful to just snatch them up and immediately use them. They're the most obvious public good, where they would have a huge impact on the world. Finally, just keep lines of communication open between these different research groups. I think that helps both from spreading what the good ideas are faster, and it also helps in terms of if there are new things that we're worried about, it's a way of making sure that people know about what's going on. I've personally been super grateful for all the help from Adam, Colin, and the rest of the T5 folks as we were putting together this project, and I'm grateful for all of you guys for inviting me here to talk. So thanks. I'm Tom.