Hello, I'm Andrew Maine and this is the OpenAI podcast. Today our guests are OpenAI's chief scientist Jakub Pachocki and Szymon Sidor. We're going to talk about measuring AI progress, how you determine AGI, and where the next breakthrough might come from.
The model was able to correctly identify that didn't make progress on the problem.
We started asking very very seriously the question like are we ready as an organization for incredibly fast-paced progress? When we think about how we shape our research program at OpenAI, we seek to create intelligence that's very general.
I want to first start off by understanding your roles. So, Jakub, you're the chief researcher, chief scientist at OpenAI.
Okay. What does chief scientist mean?
So, the primary thing I'm responsible for is setting the research road map for the company. So deciding what is a technical path we are going to bet on and what is the underlying long-term research that we're going to pursue.
So how about you Szymon? What do you do?
Yeah, I mostly do IC work. I try to maybe sprinkle of leadership somewhere in there. I try to do whatever is most useful.
Now you two knew each other before working at OpenAI, right?
Yeah, we went to the same high school.
I think we became best friends after we left. I think coming to OpenAI is the kind of emotional experience that forms bonds. I think in high school we were more like colleagues.
What kind of high school produces guys like you?
So, well, yeah, we went to this high school in Poland. I think we were both drawn there by this computer science teacher, Mr. Ryszard Szczerbiński, who's had a great track record before we went there of bringing up computer scientists, programmers with this big focus on programming competitions and kind of pursuing excellence in this one field. So I think that was a very formative experience and a great mentor for us.
Yeah. Definitely. I think Jakub was really going deep on programming. I think it went way beyond typical high school curriculum like there was graph theory, matrices and all sorts of stuff like that. I actually hope that maybe with ChatGPT it's a little bit easier for people now to do this kind of deep dives because without the right mentor and without a lot of work it's kind of hard to replicate that experience. I've been using it to explain things like the Monty Hall problem where you have to choose which door and you go into ChatGPT and you say make a graphic interactive version of this and all of a sudden you can see it. It can show you the different solutions if you do one thing or the other. I think it's one of these things where I'm excited about the ability not just to explain in text but to build multimedia to do things. And it does get into the area of there's not really a measure for that. You know that it's a use case you go this didn't exist before and we talk about AGI but we kind of have very loose definitions and whatever and I'd love to hear how you would describe it both from a technical and also like a lay person's understanding of it.
Yeah. Well, maybe to address the point about teaching, this sort of better explanation of some concept or teaching chosen method is definitely a powerful use of ChatGPT and I think works well with a teacher like our Mr. Szczerbiński. But at the same time, I think the thing that he was able to provide was more like emotional support and space, which I think it will be hard for AI to do alone.
That's a great point. I think that gets lost a lot because I sometimes hear people talk about, oh AI will replace education. And for me, I had teachers that maybe their facts weren't always right, but their heart was there and they cared and they answered questions for stuff. And so I think that's a good point that these are companions to that and I think that a teacher using these tools can be a more capable teacher.
But so on the subject of AGI though, I want to hear first like give me your technical definition of it or actually not a technical question give me like how you would describe it to like if you were talking to a younger sibling.
Yeah. A few years ago when we would talk about AGI, you know it felt like the technology deep learning has incredible promise but at the same time the concept still felt a little bit abstract and far away. And so I think whether you talk about human level intelligence, ability to converse naturally, ability to solve math problems or pursue research they all kind of felt like in the same space. I think as technology has progressed now we see these are actually quite distinct capabilities and I think we pretty clearly are at the point where the AI is able to converse naturally on a wide range of topics, it is able to solve math problems. I think getting gold medal at IMO is something we've long discussed as a milestone on the path to AGI and that happened. I think solving all the problems on the national math olympiad is actually a little bit harder and another milestone on the path there but I think increasingly we see that these pointwise measures are less adequate and so we turn to thinking about what is its actual impact in the world. For me personally, the thing that I think about when I think about how AI progress really impacts the world meaningfully, I first think about its potential for automating the discovery and production of new technology. I think we tend to associate new ideas, fundamental technological progress with just human ingenuity. And we measure our progress by these major milestone inventions and technological revolutions. And I think it is just hard to internalize like this is possible to automate most of this process. It is possible to have a big computer that is coming up with ideas that fundamentally change our understanding of the world. And I actually think that is not that far away. And so thinking about what separates us from that and what are the consequences of such technology is my first thought.
I just ordered a little Mac Studio because I want to take the open source model GP-DOS and I want to just let it run non-stop because that idea just the idea of letting it generate and do stuff 24/7 is fascinating to me. But you're talking about a scale of basically automating science at a huge scale. And so what kind of discoveries what kind of things do we think might be the first things we could see from that?
When we think about how we shape our research program at OpenAI, we seek to create intelligence that is very general. We drive towards this automated researcher as a priority but we don't really think of it as let's take this specific domains and let's deploy this technology there. I think that is a way to make faster pointwise progress. But I think the potential for the really big discoveries and the most meaningful technology advancement comes from this generality.
Although still I think we see the technology is easier to apply in some domains than others. I think especially in places that combine a large amount of reasoning with a lot of domain knowledge and intuition seem very amenable to these systems. I think in particular we see pretty incredible results on medicine which is very encouraging. I have high hopes about that. Yeah I think naturally being a company of AI researchers we think a lot about automating our own work. I think it is also kind of a, you know I don't think it's a, if AI can indeed reach a point where you can automate AI research and that is probably a very important thing to automate. And similarly thinking about how it can help with automating research on AI alignment and safety.
I'm obviously impressed by the IMO results. I mean I was actually about to add that in the past when we were talking about the IMO with Jakub that was a few years ago and we were still trying to even figure out what our definition of AGI might be. One concept we were considering is something like solving all the problems on the math olympiad and why did that feel appropriate is just like okay if you have a model with such superior mathematical reasoning, then it should be able to disrupt a bunch of different domains that can be mathematically modeled. Right. I mean I'm in general just, I think maybe this podcast is just a good opportunity to share a little bit more of an inside perspective. I was just astounded by the progress. I think so sometimes I see those headlines where people say that the economic impact of AI is only like 3% or 5% and those headlines are often accompanied by comments like well so AI is slowing down or people are overhyping AI so much and it's only like 3% so what's up with that. And when I see headlines like this, I remember maybe 10 years ago I was working on natural language processing with deep learning and back then it just didn't really work. Like I remember Jakub once came to test one of the technologies we were working on and that was trying to detect sentiment of sentences and he was trying this movie is bad correctly classified as negative, this movie is good correctly classified as positive and then he would say this movie is not bad and the model is like oh negative.
Right. So that was 10 years ago. And since then we slowly started solving tasks like this, solving tasks like decide is this word a noun a verb that was like sentiment neuron, then we had GPT-1, GPT-2 started producing a paragraph of text that made sense right that was such a breakthrough right now it feels so simple but back then it was such a breakthrough then we had GPT-3, GPT-4. GPT-4 was to me like my personal AGI moment because it would sometimes say things that surprised me and I was like can this model actually surprise me right it's still back then like ChatGPT for my personal use kind of felt a little bit more like a nuance and maybe slightly better Google but what's the big deal and then suddenly we get to deep research and this can actually answer questions to really rarely make things up. That felt useful. And then finally now we have models that can compete in programming competitions which was very hard earned for me personally and even more so for Jakub obviously and the pace of progress just from the perspective of somebody working on this technology is absolutely amazing. So when you see that 3% like I raised you like 10 years ago if you had to quantify it, it would probably be like 0.00001% or something, right? So really I think those numbers need to be put in perspective, right? And there is no reason not to believe that in a year it will be 10%, in two years it will be 20% and so on.
Yeah, I've heard it said that if you looked at a graph of the economy from let's say the worldwide web early 90s forward and you said point to the internet happening to the economy you can't find the point there's no point you go oh okay Tim Berners-Lee announced this whatever and I think AI is a lot like that where people go oh we've only measured this one our measures are hard it's hard to know who's using it how they're using it and you brought up a very good point too about if you've been following it for a while I remember training a very simple next character predictor on my computer and it was terrible, right? One, I'm using a small computer, but even then and then you got the sentiment analysis, you're playing with BERT and it's kind. But then GPT-2 comes out and I read every single output on GitHub. Every single output GPT-2 came out because I'm like, there is something going on with this. And that's how I ended up working at OpenAI was because I was this obsessive person about that. Then with access to GPT-3 kept saying, oh, this is really this path that's moving forward. But it's kind of crazy now because if six weeks go by and a benchmark hasn't been broken, people are like, oh, we hit the wall. We hit the wall. And I would say part of the problem though is that benchmarks in some ways feel like you'll see modest improvement on them. I've heard some of the benchmarks have problems. Some of them actually have wrong answers and it's impossible to get 100% if you answer them correctly. But also, we talk about the term internally. I've heard people talk about this as saturation. Do you want to talk about that?
Yeah, I think there's a few issues that we're hitting with benchmarks right now. Yeah, I mean a pretty clear one is saturation and that is just the models genuinely reaching a point where for the standardized forms of measuring intelligence or ability they are at human level for a lot of them. You know if you're able to perform amongst the top on these very hard high school competitions where we have the best competitors from around the world it just becomes quite hard to have this very constrained measurement. Previously when we were looking at GPT-1, GPT-2, GPT-3, GPT-4 scaling paradigm the benchmarks were really just measuring the rising of the tide. I think now the field has developed a lot of more data efficient ways to train for specific abilities right doesn't mean you train on these benchmarks but you can train models that are disproportionately good at math compared to their ability to write for example and so they will do better on math benchmarks but it's no longer as representative of their overall intelligence in other topics. I think these two issues combined, yeah, I think we really have to think about the reward utility of these models and especially their ability to discover new insights.
Yeah, I guess that's a thing that sort of gets overlooked is that you can build a model that's a really good test taker, but that model may not really be that useful for work. Ideally, your model should score well on tests, but just because a model got these scores doesn't mean you're going to find it personally useful. And I certainly think that's a challenge we're at now where when people say is a model good or bad. It's kind of like saying you're trying to create a blanket assessment when there's 100 different use cases for it. You know, is a model good or bad? Maybe it's great at creative writing, maybe it's bad at math, maybe it's great at math and bad at creative writing. And that becomes a really big challenge. And we've talked about this with like one for math, international math olympiad and these kinds of metrics. Why are they important? Why is it important to put it into these sort of human level competitions?
I think the reason we've been excited about these competitions like the international math olympiad, information olympiad is that they are a pretty interesting example of a test that is constrained, doesn't require that much knowledge but really tests your ability to think about a problem hard for an hour or two or three. And we have very good evidence these problems are hard. You know there's a lot of people that try to solve them and compete at solving them and it matters to them. So yeah, so I think this is and for models that excelled at knowing a lot of things but not necessarily thinking very hard in the past that really seemed like the right milestone to be working towards.
Now I'm so I understand it the model that scored gold medal level on that wasn't using like a calculator. It wasn't using other tools. It wasn't using some of the frameworks. It was doing it purely through reasoning.
Yeah, that's right. For the International Math Olympiad, yeah, the model was not using other tools like yeah.
And again, and it was like two years ago, you asked it to multiply two four-digit numbers, it would fail.
Yeah. But definitely for this kind of contest, it really is of course in a limited domain of math, but it really is about fairly creative thinking, not about applying a formula.
I guess that's part of the challenge though is that once you start moving outside of math, it gets to be harder. You can start to come up with things like humanity's last exam, which I think is a pretty neat test, but you find that certain models after they learn a certain kind of tool use kind of figure out maybe sort of how to solve these problems better. And I would wonder what kind of benchmarks are we going to need? You know, what are you looking at to say, okay, this is how I can kind of get an objective measure of a capability. One thing that surprised me in the past, I was talking to one of our co-workers here Anna Makandu and I was telling her about IMO. I was excited about some progress and she's like what's IMO and that kind of was a very in front of the moment for me because I do realize that some of those benchmarks we kind of live in a bubble a little bit.
Mhm. For me that competition feels important and especially the computer science counterpart IOI because it was a big part of my life and so is true for many co-workers here. But actually for an average person working in other field or maybe not as interested in mathematics or computer science, maybe they're interested in history or something that the...
Anna speaks like five languages too. So I could see for her a different metric based on that would be interesting.
Yeah. So I think one thing that it's not a perfect metric but at least helps keep us honest and helps keep us escape the bubble is just ChatGPT right because everybody uses ChatGPT and they use it for all sorts of use cases. And obviously there's a lot of pitfalls to using that as a metric, but at least it avoids that particular problem where there are just some things that I'm more familiar with and other people might appreciate other things and this gives you a very wide coverage.
Yeah. And in there too, you have subsets of users, people who are building GPTs and doing more complicated stuff. You mentioned before too the fact that the model will reason longer and that seems like a very interesting way to evaluate capabilities.
Yeah. And I think this is also maybe one challenge with focusing on only usage of ChatGPT and broad adoption of AI as the metric of progress. Like I think this hasn't really happened to a very meaningful extent yet but I think it will start happening pretty soon. We should be able to use vastly more compute than a user would normally be willing to buy for themselves to produce technology artifacts that are useful to a lot of people. And I think that for me will be a very important measure of progress.
Which of these wins were the most surprising to you?
