AGI progress, surprising breakthroughs, and the road ahead — the OpenAI Podcast Ep. 5

The model was able to correctly identify that didn't make progress on the problem.

We started asking very very seriously the question like are we ready as an organization for incredibly fast-paced progress? When we think about how we shape our research program at OpenAI, we seek to create intelligence that's very general.

Chief scientist. Yes.

So, the primary thing I'm responsible for is setting the research road map for the company. So deciding what is a technical path we are going to bet on and what is the underlying long-term research that we're going to pursue.

So, well, yeah, we went to this high school in Poland. I think we were both drawn there by this computer science teacher, Mr. Ryszard Szczerbiński, who's had a great track record before we went there of bringing up computer scientists, programmers with this big focus on programming competitions and kind of pursuing excellence in this one field. So I think that was a very formative experience and a great mentor for us.

Yeah. Well, maybe to address the point about teaching, this sort of better explanation of some concept or teaching chosen method is definitely a powerful use of ChatGPT and I think works well with a teacher like our Mr. Szczerbiński. But at the same time, I think the thing that he was able to provide was more like emotional support and space, which I think it will be hard for AI to do alone.

Yeah. A few years ago when we would talk about AGI, you know it felt like the technology deep learning has incredible promise but at the same time the concept still felt a little bit abstract and far away. And so I think whether you talk about human level intelligence, ability to converse naturally, ability to solve math problems or pursue research they all kind of felt like in the same space. I think as technology has progressed now we see these are actually quite distinct capabilities and I think we pretty clearly are at the point where the AI is able to converse naturally on a wide range of topics, it is able to solve math problems. I think getting gold medal at IMO is something we've long discussed as a milestone on the path to AGI and that happened. I think solving all the problems on the national math olympiad is actually a little bit harder and another milestone on the path there but I think increasingly we see that these pointwise measures are less adequate and so we turn to thinking about what is its actual impact in the world. For me personally, the thing that I think about when I think about how AI progress really impacts the world meaningfully, I first think about its potential for automating the discovery and production of new technology. I think we tend to associate new ideas, fundamental technological progress with just human ingenuity. And we measure our progress by these major milestone inventions and technological revolutions. And I think it is just hard to internalize like this is possible to automate most of this process. It is possible to have a big computer that is coming up with ideas that fundamentally change our understanding of the world. And I actually think that is not that far away. And so thinking about what separates us from that and what are the consequences of such technology is my first thought.

When we think about how we shape our research program at OpenAI, we seek to create intelligence that is very general. We drive towards this automated researcher as a priority but we don't really think of it as let's take this specific domains and let's deploy this technology there. I think that is a way to make faster pointwise progress. But I think the potential for the really big discoveries and the most meaningful technology advancement comes from this generality.

Although still I think we see the technology is easier to apply in some domains than others. I think especially in places that combine a large amount of reasoning with a lot of domain knowledge and intuition seem very amenable to these systems. I think in particular we see pretty incredible results on medicine which is very encouraging. I have high hopes about that. Yeah I think naturally being a company of AI researchers we think a lot about automating our own work. I think it is also kind of a, you know I don't think it's a, if AI can indeed reach a point where you can automate AI research and that is probably a very important thing to automate. And similarly thinking about how it can help with automating research on AI alignment and safety.

Yeah. Yeah.

Yeah, I think there's a few issues that we're hitting with benchmarks right now. Yeah, I mean a pretty clear one is saturation and that is just the models genuinely reaching a point where for the standardized forms of measuring intelligence or ability they are at human level for a lot of them. You know if you're able to perform amongst the top on these very hard high school competitions where we have the best competitors from around the world it just becomes quite hard to have this very constrained measurement. Previously when we were looking at GPT-1, GPT-2, GPT-3, GPT-4 scaling paradigm the benchmarks were really just measuring the rising of the tide. I think now the field has developed a lot of more data efficient ways to train for specific abilities right doesn't mean you train on these benchmarks but you can train models that are disproportionately good at math compared to their ability to write for example and so they will do better on math benchmarks but it's no longer as representative of their overall intelligence in other topics. I think these two issues combined, yeah, I think we really have to think about the reward utility of these models and especially their ability to discover new insights.

I think the reason we've been excited about these competitions like the international math olympiad, information olympiad is that they are a pretty interesting example of a test that is constrained, doesn't require that much knowledge but really tests your ability to think about a problem hard for an hour or two or three. And we have very good evidence these problems are hard. You know there's a lot of people that try to solve them and compete at solving them and it matters to them. So yeah, so I think this is and for models that excelled at knowing a lot of things but not necessarily thinking very hard in the past that really seemed like the right milestone to be working towards.

Yeah, that's right. For the International Math Olympiad, yeah, the model was not using other tools like yeah.

Yeah. But definitely for this kind of contest, it really is of course in a limited domain of math, but it really is about fairly creative thinking, not about applying a formula.

Yeah. And I think this is also maybe one challenge with focusing on only usage of ChatGPT and broad adoption of AI as the metric of progress. Like I think this hasn't really happened to a very meaningful extent yet but I think it will start happening pretty soon. We should be able to use vastly more compute than a user would normally be willing to buy for themselves to produce technology artifacts that are useful to a lot of people. And I think that for me will be a very important measure of progress.

I think we definitely anticipated getting to this point when we saw the reasoning models starting to work. At the same time, I think this recent set of things is very impressive. I think maybe out of those I think IMO came a little bit sooner than I expected. IMO gold again like I think IMO problem six will still, IMO has all the problems require creative thought and some new insights I think but typically there's this proverbial problem six that requires very out of the box thinking and it's really usually outside the typical domains of the other problems and so in the past we were actually drawing a boundary between getting a gold, solving these other problems and actually considering solving all the problems and in particular problem six. So it was pretty hilarious in some way to see ourselves and also Google DeepMind at the same time like oh yeah we solved problems one through five perfectly and we didn't make any progress on problem six. I think that kind of makes that challenge pretty clear.

Yeah, the model was able to correctly identify that didn't make progress on the problem.

Oh, so I think in the past few weeks actually our models have performed incredibly well in three competitions. So we talked about two of them which is IOI and IMO. There is also this competition that is open to everyone not just high schoolers called AtCoder. It's a very prestigious, high-quality competition organized in Japan but open to competitors worldwide and in this particular contest it was about longer horizon heuristic problems where you're given only a single problem you have 10 hours to solve it and so you have competitors racing to figure out the best approach to this difficult optimization problem. So it's a bit different because there isn't a single correct solution. There isn't a single pattern to follow. These tasks are extremely diverse and you can focus on the single task for 10 hours. And so we entered our model into this contest. And to me this had a little bit of a personal significance. I used to be a very engaged competitor in the past in these more short form, closed form contests like IOI. And my friend Siho who also works at OpenAI excelled at these long duration contests and when we worked together he would mock me a little bit that my sort of contest would be automated long before his because they are longer duration, require more focused work. And turns out in this contest in Japan, Siho was actually one of the top contenders. And so I was watching this live stream watching our model race with Siho throughout the competition. In the end, our model actually got second place and Siho won. So he alone stood in the way of his prediction not coming true.

I think one thing we always try to not underestimate is the importance of scaling. I think even as we look at these reasoning models it's not like the previous scaling paradigm of pre-training has vanished right like I think we will see these things compound and I think there's also new directions that we can move in in particular we were talking about extending the horizon that these models can plan for and reason in. And I think if you look at it from the perspective of just compute spend we say okay yeah we went from GPT-4 was doing some out of compute for every answer to GPT-5 pro which maybe uses I don't know like 10x, 20x I don't know some non-trivial but in some ways not that impressive amount of compute more and can produce much better answers. I think on the scale of what amount of compute would you be willing to spend on a problem that actually matters to a lot of people right on progress on a medical research question, progress on developing the next generation of models right these are incomparably larger amounts and so I think that question of model persistence and ability to work for a very long time on a focused problem is a pretty clear next step.

So I was talking about automating research. You know my picture of how that would actually look like is imagine a company of very capable researchers and engineers that is largely automated right and now again that is something that will interface with the world in all sorts of ways. It won't be just a black box like it will talk to people. It will take in inputs. It will run experiments. But I think having this sort of potential for developing new technology and other kind of artifacts, code bases, designs I think can radically accelerate the pace of technical progress. So I think that is something that we will feel and we need to do a lot of work to get it right from a technical and from societal perspectives. But I think that is kind of where I turn to. I think we should also expect a lot of progress on the actual interfaces that we interact with.

You know we see ChatGPT can feel quite humanlike. We can form attachments with it. I think as it becomes more persistent as it becomes capable of expressing itself in different forms and texts I think those effects will become stronger and again that will be something I think will become a very big and important conversation.

I think there's definitely like we are in a place where there's very tough trade-offs where there's such clear economic, personal value you can extract out of having the model have access to a lot of your data. At the same time, I think we are not at the threshold of robustness where we can fully trust these models to not be exploited by someone trying to exploit them.

Yeah it's definitely a big problem I think we as a field will have to iterate on.

High school students today. Oh, yeah. That one is also a great question, right? Because I hear a lot of what I consider misinformation on that online. So, you should absolutely learn to code. Like one skill that is at premium and will continue being at premium is to have really structured intellect that can break complicated problems into pieces and that might not be programming in the future but programming is a fine way to acquire that skill. So are other domains where you need to think a lot. So don't let people tell you that you should not learn to code.

Well, you know, thinking about how I thought about things in high school, I think it's pretty incredible how many perceived constraints are not actually there when you really think about it. You know, maybe the first revelation for me was like, hey, if I really am passionate about this computer science stuff, I can actually spend a bit more time on it at the cost of maybe spending a bit less time on other 12 subjects in school. But then somehow it was a big revelation to me that actually I can go and study in the USA at some point and that's not really something that seems obviously accessible. And spending some time here in Silicon Valley and seeing how people are willing to really attack these big problems with ambition and the belief that you can actually make a meaningful positive change in the world. Yeah, I think has been incredibly inspiring and it's something I cherish about this community.

I think there's a couple books. I remember my, it's actually very hilarious thinking about it now. I didn't really connect the dots, but my dad gave me this book once when I was like 15. I was pretty unsure what I want to do. It was a Polish version of a book by some author I didn't know. Called Hackers and Painters. Yeah. So it was actually Paul Graham. So I guess like this community. So I found that pretty inspiring.

Yeah, took me a while to become convinced that deep learning is more than a fad. Because we don't really understand the underlying optimization and I think this kind of has been the story of our research here making progress on these questions about how it really works but it really is like a physical phenomenon in some way and to a classically trained computer scientist that was a weird thing to accept.

That was before AlphaGo.

Who knows?

Has been more useful.

Mathematics or theoretical computer science? Either of those.

Thanks Andrew.