I'm Natalie, your OpenAI Forum Community Architect. I'd like to begin all of our talks by reminding us of OpenAI's mission, which is to ensure that artificial general intelligence benefits all of humanity. To conclude our speaker series for the year, we're hosting one of our favorite all-time guests, Professor Terence Tao, and two of my very inspiring colleagues at OpenAI, Mark Chen and James Donovan. Terence Tao is a professor of mathematics at UCLA. His areas of research include harmonic analysis, PDE, combinatorics, and number theory. He's received a number of awards, including the Fields Medal in 2006. Since 2021, Tao also serves on the President's Council of Advisors on Science and Technology. Mark Chen is the Senior Vice President of Research at OpenAI, where he oversees advanced AI initiatives, driving innovation in language models, reinforcement learning, multimodal models, and AI alignment. Since joining in 2018, he has played a key role in shaping the organization's most ambitious projects. Mark is dedicated to ensuring AI developments benefit society while maintaining a focus on responsible research. Finally, James Donovan leads science policy and partnerships in global affairs, focusing on how our models can be best used to accelerate scientific research and commercialization. He came to OpenAI having been a founder, VC investor, and partner at Convergent Research, where he helped launch multiple moonshot science organizations, including Lean, a formal proof assistant for complex mathematics. Please help me welcome our special guests to the OpenAI Forum.
Hey, fantastic. Thank you so much, Natalie. I really appreciate the introduction. Is yours, James?
Thank you so much. What an honor to be here with such great minds tonight. Before we get going, I do just want to give a big thank you to Natalie and team for organizing all of this. It's no easy thing to get so many people together and run it as smoothly as she always does. It's a great honor for me specifically to be here to talk to you both, so thank you for finding the time. And just as a sort of general note, though this is the conclusion to one year's Forum events, it is the beginning, as always, of the next year running, where we'll have a theme focusing on science and how our models intersect and accelerate science, hopefully safely and equitably for the wider world. So to get going, both, I wanted to start just by getting a sense of, maybe first Terence and then you, Mark, what are the most interesting questions that you're focused on in your research today, and why is it important that we try and solve those questions?
Okay, well, there's lots of technical math questions that I would love to solve. I think more relevantly for this meeting, I'm really interested in how we can just rework mathematics from the ground up and how we can use all these new tools to really collaborate in ways that we couldn't do before, to do mathematics with a scale we couldn't do before. And I think it could be a new age of discovery. Like right now, mathematicians, we work on individual problems at a time. We spend months working on one problem and then move on to the next. With these tools, we could potentially, you know, just scan hundreds or thousands of problems at once and do really different types of mathematics. So I'm really excited about that possibility.
Cool. Yeah, I mean, one of our big focuses over the last year has been reasoning. So, you know, since GPT-4, we've kind of shifted our focus slightly. I think GPT-4, for all intents and purposes, is a very smart model, contains a lot of world knowledge, but it's also stupid in many ways too. It gets, you know, tripped up by simple puzzles and oftentimes very reliant on the prior. Like if it has some kind of prior knowledge of how a puzzle should shape out, it often kind of makes that same kind of pattern matching mistake. I think these pointed to us really to a deficiency in the model's ability to deeply reason. And so we've been focused on developing what we now see as the O-series of models. So these are models that are more like system two thinkers than system one thinkers. I think they less often kind of give the intuitive, fast response and spend some time kind of reflecting on the problem before producing a response. I think just to highlight two other problems that are key to our research agenda: data efficiency is certainly one of them. I think we care about how to ingest all of the data in the world, including non-text data. And third is a very practical problem: just how do we create, you know, intuitive, delightful experiences for our users.
Yes, it's true. I mean, that last problem is maybe a little beyond the world of math specifically, but is a critical one, that kind of human-computer interface question here. I do, Terence, want to ask you specifically about the O1 models as Mark has outlined them. But before I do, you just mentioned a potentially new type of maths. At various times, you've spoken about, you know, maths at industrial scale. You've also talked about different ways of cooperating in maths. Would you mind unpacking that for us a little bit?
Sure. So like, math has always been perceived as like a really difficult activity, and it currently is, for many reasons. But one of which is that it's relying on one human or maybe a small number of humans to do a lot of different tasks to achieve a complex goal. You know, if you want to make progress in mathematics, you have to first come up with a good question, and then you have to find the tools to solve it. Then you have to learn the literature, you have to try some arguments, you have to do the computations, you have to check the arguments, make sure it's correct. And then you have to write it up in a way that can be explained, and then you have to give talks and you have to apply for grants, and there's lots of other different things you have to do. And these are all kind of different skills. But in other industries, you know, we have division of labor. You don't rely on one person. Like if you're making a movie, you don't have one person produce the movie, edit the movie, act in the movie, and get financing for the movie, and so forth. You have different roles. But we've not found a way to decouple all these tasks in mathematics until recently. I think now that we have these tools, in principle, you could have a collaboration where one person has the vision, one person or maybe an AI does the computations, and then there's another who writes the paper, and so forth. And so you don't need one person to be expert in all aspects. So I think a lot of people are discouraged from doing mathematics because they look at all the different things, the checklist of what you have to do to be a good mathematician, and it's really daunting. But maybe there are people who are good at looking at data and inspecting patterns and then asking an AI to check, you know, can you confirm that this pattern exists? Or maybe they're not very good at finding the right questions to ask, but they can work on some very narrow, specific piece of a larger project. So I think these tools allow the discipline to be decoupled in many ways, to be made more modular. And so some tasks be done by AI, some tasks by humans, some tasks by maybe like formal proof assistants, some tasks by the general public. We have citizen science in other disciplines. We have amateur astronomers who discover comets or amateur biologists who collect butterflies. And we don't really have a way of utilizing amateur mathematicians outside of some very small, sort of fringe projects. So there's a lot of potential, and I think we have to throw a lot of things at the wall and see what sticks.
Terence, I have a quick follow-up question for you. I'm curious, like AI aside, what the maximum number of humans up to date that have been kind of able to collaboratively work on a single math project or, yeah, do you think there's like an upper limit here?
Right. So in practice, the limit is around 50 or 60. It's really hard past that point. You have to check each other's work, and also just getting everyone in the same room and so forth. There are a small number of projects which have many authors, but for example, proof formalization projects, where a big proof gets formalized, that's one of the few tasks in mathematics that we already do know how to crowdsource and split. Because you run it all on GitHub or something, and all the contributions are verified because they're in this formal language such as Lean. And so these can have 20, 30 authors. Lean has this thing called Mathlib, it's a library of all undergraduate mathematics. It's never been officially a research project, but I think technically it has like thousands or at least hundreds. So yeah, but it's only really in formal mathematics that we're really seeing the large collaboration so far.
Fantastic. And I do want to echo that shout-out to Lean, they're doing some really incredible work, and I think we might have a few members of the Lean team on the call today. As you were unpacking that, Terence, it sounded like your default assumption was that humans will still divvy up tasks, they'll still understand enough about the process to decide who's doing what. Where my first question for you would be, do you think therefore there'll be different roles that emerge for mathematicians, different specialties that they adapt? And I'd flip it over to you, Mark, to say whether you think that's likely to always be humans or you see a world in which the O-series itself is breaking down problems or the O-series models.
Right. So I see software engineering as kind of a template for where math might go. You know, so in the past, maybe there's one heroic programmer who did everything, the same way that mathematicians sort of do everything now. But now you have project managers and programmers and quality assurance teams, and so forth. And so one can imagine doing that right now. So I'm involved right now in several projects which are collaborative, and they involve both a theoretical math component and a formal proof component, and people are also running various code algorithms and so forth. And it's already specializing the way that I was expecting. So there's some people who don't know the math but they're very good at formalizing theorems, it's almost like solving puzzles to them. And then the ones that are good at running GitHub and doing all the project management and just making sure that all the backend runs smoothly. And then there's people who do data visualization and so forth. And we all coordinate. So far it's been mostly humans and sort of more old-fashioned AI type, like automated theorem provers and often just running Python code or something. But I think it's a paradigm in which AI would slot very nicely in once they get good enough.
Yeah, no, that makes a lot of sense to me too. I feel like today I almost kind of treat AI as a co-worker in many respects. You know, there are things that I don't do very well that, you know, I can farm off to an AI. I'm only kind of conjecturing here because I'm not a mathematician, but in terms of where AIs might be strong in helping to solve mathematical problems, could be first just like recognizing patterns, right? Machines are fairly good at this, especially if there's a lot of kind of data or just a lot of stuff to sift through. And then I think from identifying patterns, you can start to form conjectures. And I think they might have a unique strength in doing that. Again, coming up with proof strategies, I think, Terence, this is something we talked about last time. I think humans today still probably have a better intuition for what the right steps forward are, but maybe a blind spot when it comes to one particular step. And I think last time we mentioned there's like some generating function approach that a model suggested in one of the kind of toy problems you were trying to solve, and that actually turned out to be not a terrible idea in that situation. Also maybe just like verification, models might be able to kind of verify certain steps that you're pretty sure are right but you just want to get another pair of eyes on. And maybe also kind of like generating counter-examples too. I think if there's something where you just want to think of a lot of potential ways that a theorem could be false or something, a model may be able to exhaust that a lot more efficiently.
That makes a great deal of sense. You've both mentioned in your answers the role of theorem provers and formalization more broadly. Is it fair to say that you both think that that is a necessary intermediary layer between doing the maths and using LLMs or equivalent technologies for largely?
Yes. I mean, the proof has to be correct. And the thing about math proofs is that if you have 100 steps in the proof and one of them is wrong, then the whole proof can fall apart. And AI, of course, makes all these mistakes. There are types of mathematics where a positive failure rate is acceptable. Like Mark said, like finding patterns, finding conjectures, it's okay to have an AI that is only correct 50% of the time if you have some other way to check it. And in particular, if it tries to output an argument, it's a very natural synergy to force the AI to output in something like Lean, and then if it compiles, great, if not, it sends back an error message and it updates its answer. People have already implemented this and they can prove kind of maybe short proofs on the level of an undergraduate homework assignment can be sort of done by this iterative technique. Definitely not at the point where you can just ask it a high-level math question and it outputs a huge proof. I mean, okay, AlphaProof can do it with three days of compute, but it doesn't scale. Yeah, for some soft things where a positive error rate is acceptable, you won't need the formal proof assistants. But anything really complex where one mistake can propagate, it's basically indispensable.
Yeah, I think at OpenAI, you know, in various times in our history, we've focused more or less on formalized mathematics. And I think today we do a little bit less. I think primarily just because we want to explore reasoning in full generality. And we do hope that kind of like reasoning that you learn in fields of computer science are fairly similar to reasoning that you learn in fields like math. So yeah, definitely understand the advantages though of doing formal mathematics.
I'd quite like to come back to that architecture of the theorem prover and maths AI and see whether or not that's true for other domains of science as well. But one question I have before that is that even in the training process, there's probably a lot of incorrect ways of solving things that don't get into the training models, because mathematicians on the whole don't publish incorrect things, and that's true for science more broadly. Do you both think that would make a big difference? Is that a sort of cultural norm that we should be trying to push, that people do publish failed answers?
Separately from AI, I think that's a good idea. It is hard to encourage that. People don't like to admit their mistakes. But yeah, this could really be great training data for AIs. When I teach my classes, sometimes the classes I get that are most effective are accidentally when I prepare a proof and I give it in class and I screw up, the proof does not work, and I have to fix it in real time. And the class sees me try various things, oh okay, what if I change this hypothesis, might try this example. And I've gotten feedback later that those were the most valuable classes that I have taught, and it was because I made mistakes. And this is data that largely, I think, people like you, they just don't have access to. I mean, this is, I think, in fact, many experts in a domain have the expertise built on decades of mistakes that taught them what not to do, the negative space. There's beginning to be, I think, as we move to a more formal environment, like right now we are formalizing sort of proofs after they're done, we will eventually get to the point where we will formalize as we go. We will maybe converse with an AI while we think about math and we'll try to formalize the steps as we go. And then maybe it doesn't work and you have to backtrack and so forth. And that will sort of naturally create some of this data that we don't have right now.
Out of interest, a lot of mathematicians talk about the beauty of a theorem and the kind of eureka moment when everything fits together and can be expressed elegantly. Is there a chance we lose that kind of cognitive process when using tools like these?
I think a similar situation came up when calculators became ubiquitous. People said, you know, now that you don't have to do arithmetic by hand, you lose your number sense. And to some extent, this is true. I would imagine that a mathematician from 50 years ago is much better with getting number sense from direct calculations. But you also get a different type of number sense from just playing with the calculator. So I think there'll be a different type of beauty standard. I think there will be some computer-generated proofs that are also really elegant and amazing in a different way. But I don't think the AI paradigm will take over completely for many decades. I think mathematicians are somewhat slow to, you know, we still use chalk and blackboard, you can see. So there'll be people who will still craft really wonderful proofs. I think there'll be a class of mathematicians who will take AI-generated mathematics and convert it into something much more human. I think that would be a common thing to do in the future.
Out of interest, Mark, when you hear an answer like that from Terence, do you put a lot of thought into thinking not just how do you make reasoning high quality, so accurate, but also how a human can work with the kind of outputs and that side of the equation too?
Yeah, yeah. So I mean, I think when you think about RL, right, it's like also just kind of incentivizing the model and having the model learn from its mistakes. So that like highly resonated with me. Yeah, and I do think that's how you develop robust and strong reasoning skills, right? You can't really just be shown a lot of examples of accurate reasoning because there's so much negative space in mathematical reasoning. I do think models will become helpful, much more useful. I'm quite an optimist on this. And in terms of kind of like the impacts, right, it's really kind of interesting to hear about, you know, not so much that like people will lose a sense of like aesthetics or intuition, but maybe like kind of develop new abstraction layers and kind of new abstractions and intuitions will form out of that. And that seems interesting and quite likely as well. So yeah, no, that'll be cool to see, and especially if it happens fairly soon.
Yeah, it's a really interesting line to follow, just in the sense of, I think in my own world of biology, the assumption tends to be these models will find patterns across things that were otherwise seen as unrelated and you'll find all these underlying unity across things. But that's kind of on the idea there's lots of low-hanging fruit we just haven't noticed. Whereas I think for things like maths and parts of physics, the refinement is almost in the way that that activity is done, and that we feel like that might be fundamentally different. And I wonder, Terence and Mark, whether you think it will have an implication how we educate people in maths, and in particular support people who are going to do frontier maths research.
Well, yeah, I mean, of course students are already using large language models to, you know, most obviously help their homework, but also get a second perspective on a topic. Educators are actually, you know, we're also figuring out how to integrate large language models into our teaching. So like one thing that's become increasingly common is to present some math problem or problems, give GPT's answer to it, and say this answer is wrong, please critique it. Or have a conversation with the AI and actually teach it to fix the answer. There was actually one class where they made a group project where the teacher handed out a practice final for the class and they said, okay, try to train an AI with prompt engineering and data analysis to actually figure out, and generating synthetic versions of the final exam, how to most efficiently teach an AI to solve the final. And they did. So they had one group do prompts and one group to do benchmarking and so forth. But it also forced them, you know, for example, to generate all the data for like to generate synthetic exams, they really had to understand the class material to do it. So it was actually kind of an excuse to really delve deep and learn both the class material and how to use these AI tools. So we'll find innovative ways to combine these two.
Yeah, I guess kind of some people, they point to, you know, fears, right? It's like if you have too much of a dependence on AI systems, right, like do your skills erode or kind of do you have like less insight? I'm actually very curious for Terence's thoughts on this. But yeah, I think while he's figuring out his light, maybe I can...
Okay, yeah, it did give a very dramatic...
So I quite enjoyed the lone genius in the dark, you know, out of the cave.
Yeah, well, so what was your question again?
Yeah, just like, do you think there's any truth to like reliance on AI tools kind of leading to maybe like less skill in general in mathematics or maybe loss of insight or something like that?
Well, it will be a transfer. I think we will use some skills less often, but we will develop other skills more often. So there's an analogy of chess. Chess is now essentially a solved problem, but people still play chess quite a lot. But the way they practice in chess is quite different now. They experiment with different moves and then they ask a chess engine, you know, is this a good move or not? And so for example, the theory of chess theory is flourishing. There are lots of century-old maxims about what part of the chess board is good to control and so forth that are actually being reevaluated now with humans asking chess engines various questions. And that is a different way of getting intuition about chess rather than sort of the standard sort of just play lots of games and read lots of textbooks and so forth. So yeah, it will be a shift, you know, it's a trade-off, but I think a net positive.
Yeah, I think when people ask me also about just like, you know, how should they adapt to this coming out, I still think like largely there's no need to suddenly abandon studying any particular subject, right? I think really people should be kind of embracing AI and just seeing how it can make them more efficient. Like in math specifically, right, like it could help you with a lot of tedious computations, you know, if it's some kind of routine thing that you already know kind of inside out and you can just have the model kind of carry out the manipulations or something like that. I still think there's just a lot of value in just very deep understanding of a subject. Yeah, even in machine learning today, right, the people who are affecting the biggest change are the people who just very deeply understand the math or the systems. And I think that will continue to be a very big lever. Also just like focusing on abstractions, I think like humans do have a particular aesthetic that's just tied to the core of mathematics. And I think because other humans are judging that aesthetic, models may have a more difficult time kind of emulating that when it comes to defining the problem and having taste. Yeah, and of course, you know, math is just a good skill to have. I think it's very transferable, it teaches you kind of robust reasoning, and I think people who are mathematicians are just very adaptable in general. So definitely no reason not to invest heavily in math.
It's an interesting point, actually, Mark, when you talk about the aesthetic of maths. We're getting a little abstract, but it is possible that the way that we conceive of maths is somehow tied to the way that we experience reality as humans, and that if you had models doing very sophisticated maths, you know, we might get to a point where it is exceeding the ability of humans to verify or even make sense of in our context. Do either of you see that as a possible future anytime soon, and if so, how would you react to that?
Well, I mean, it's already the case that mathematicians sometimes produce enormous proofs that no one person understands. We already use a lot of computer assistance. There are some proofs that require, you know, that have like terabyte-long proof certificates because there's a massive SAT solver calculation or some big numerical modeling or something. And then there's also proofs that are built upon like a tower of hundreds of papers in literature, and we're taking these previous results as black boxes and no one person understands everything. So we're already to some extent used to this. And mathematicians, we can cope because it's a, we have this language of abstraction, you know, and we can sort of compartmentalize a complex proof and you just need to understand one piece and you just trust that either a computer or human understands the other pieces and it all works out. So this will keep happening with this. So we will have big complicated arguments where part of it is going to be AI-generated, hopefully formally verified too. I mean, it's a trend, it's just accelerating a trend that's already been happening. I don't see it as a real phase change.
Yeah, yeah. A lot of the worries I have are similar, just like you could have some error that propagates or other people build on top of some result and you're just kind of building on some faulty mathematics, right? Especially if the volume of kind of new computer-generated insights increases. I mean, one thing that we worry about a lot at OpenAI is this more general problem of scalable oversight. And the idea is just kind of like when a model spends a lot of time, let's say like thinking, and it comes up with some kind of fundamental insight that it's thought a lot about to arrive at, how do you know that the model didn't make a mistake? How do you know it's right? How do you trust it? And fundamentally, it's just a very real problem that felt fairly theoretical maybe a couple years back, but I think today models do have that capability to solve very hard problems. And so, you know, how do we vet and trust that the problem came up with the correct answer? Well, math is the one place where we have a shot because of we have the formal verification that can also be done in an automated way.
No, indeed. And you would hope that progress there unlocks progress across all the other sciences ultimately, right? We can find a way to derive from those mathematical proofs down into physics, chemistry, and so on. Terence, there are quite a few people in the room today who are working practically in math, for students or otherwise. So I have a few very practical questions. Maybe not a phase change using AI or AI-related tools, but there are some cultural elements of maths practically that might change. Some of the unique things are maths competitions, and I know you were in Bristol not long ago on exactly that theme. Do you see like the actual ecosystem of maths changing to accommodate LLMs, and if so, how?
It will. It's hard to predict exactly how it will. I think there'll be new types of mathematics that are not popular now because they're just technically infeasible. So in particular, experimental mathematics is a very, very small segment. Maths is like 95% theoretical, which is unusual among all the sciences. Usually there's a balance between experiment and theory. But experiments are hard, you have to be really good at programming, or your task has to be sort of simple enough that you can automate it with a regular piece of software, which is within the skill of a math addition to program. But with AI, you could do much more sophisticated explorations. So you might want to, traditionally you might study one differential equation, but you might ask an AI, here's an analysis of this differential equation, now repeat the same analysis for the next 500 equations on this list. And this is something that you can't really automate right now with traditional tools, because you need the software to do some understanding of what the problem is. So I think the type of mathematics will change. I think we will be, there's already a trend to become more collaborative and that will just accelerate with AI. But I think at least for the next decade or two, we'll still be writing papers and refereeing and teaching and so forth. I think it won't be a major change to say that we will use more and more AI in our work, just like we're already using more and more computer assistance in our work in other ways.
Yeah, and I think just a point on the competitions, I think I can speak more to programming competitions, but I don't know if they would fundamentally change too much. I think at least most people I know who kind of do that, a lot, you know, it's just very fun to do. I think kind of even beyond kind of the technical skills that you gain. And cheating will become a problem, that's maybe the one...
Yeah, exactly. Yeah, I mean, I think that's also just a very deep question, right? It's like even like how do you interview people when the models can solve very, very difficult problems. So yeah, but I do think, you know, contests, a big part of the reason people do it is because it's just fun. And I think the analogy to chess is a good one.
Yes, so cheating is definitely one element of this. But I guess the less deliberate or trying to break the rules elements is maybe attribution. You know, what happens in a world in which we have you potentially large parts of formalization being done by LLMs, or even novel ideas emerging from LLMs because of a combinatorial approach. Can you both envisage a world in which we are attributing breakthroughs directly to LLMs themselves, and what might that mean?
Yeah, this is going to be a big issue that we have to face. I think it's already the authorship model that we have of papers where there's like, you know, so in the sciences we have maybe one lead author and then a whole bunch of secondary authors. And so mathematicians, we don't do that yet. We still like order alphabetically by last name and we haven't really, we have largely sort of ignored the question of who did what and we just say, oh, we all contribute equally. I think we're going to have to be more precise about attribution in papers in the future. So there's already a trend where at least in the sciences where you write a paper, there's some section on author contributions, you know, who did what. And if it's a GitHub, you can look at the GitHub commits and this also gives you some data. So and then maybe there'll be some way to automatically, you know, inspect data and somehow summarize who did what. Yeah, so once half the commits are done by an AI and so forth, yeah, there was a question like do you actually promote the AI to a co-author or do you at least put the acknowledgements? We don't have the norms for this yet. We'll have to work it out. There'll be some test cases and some controversies and we'll eventually work out something that works for everybody. But yeah, I don't have the answers for that one.
Yeah, I do think there's also this related, not exactly the same style of issue, of just kind of access. I think if models continue to contribute large chunks of proofs, like are the people who have more access to compute, are they in an advantageous position when it comes to doing mathematics? Yeah, definitely something to kind of think through. And yeah, I don't quite know how to like follow that train of thought quite yet. But yeah, it's definitely a hard problem.
Yeah, it's going to be interesting to see. I mean, you already get maybe more on the creative side of the world questions about attribution and ownership. But it'll be interesting as it gets more and more involved in science about intellectual property and how we think about the R&D cycle in such a world. On that kind of topic of applied use of maths or science more broadly, for those who are not themselves mathematicians, we've spoken a lot about the act of maths changing and why that's important. If we were, ignoring the mechanism for how to achieve that, if we were to get to a place where foundational maths was being meaningfully accelerated, what would you expect to see happening in the world? What does that unlock for the rest of society?
Well, I think it could increase citizen participation in mathematics. One could imagine, you know, for example, people debate about, you know, is the Earth round or flat? It's amazing how this is still... but you know, with an AI, you could actually start constructing models and you can say, okay, suppose the Earth is flat, what would this sky look like and so forth. And right now you need quite a bit of math before you can figure out like how much things would change. But you can imagine with these tools that it could actually just create a visualizer for you and you can see, oh, this is what this theory of the universe would look like. And so I mean, I think it could really connect mathematics to a lot of people who currently feel excluded from it because of just the sheer technical skill needed to do anything in the subject.
Do you think it is a prerequisite that we get better at doing this kind of maths in order to use AI in other applied scientific applications? You know, is it a prerequisite for accelerating engineering or physics? And a question for you as well, Mark, whether you see that as a necessary first step.
Well, I mean, so much of the science is already math-based. If you don't understand the math, you can't model accurately without the math. And yes, certainly on the back end, I mean, if you want to train the AI, you need a lot of math. I mean, it's possible that we could enter a world where you could be a biologist or whatever and you could ask an AI to run a statistical study or something and you don't need to know the fine details of exactly what the parameters are. And if the AI is reliable enough, it could actually do all the math for you. And so it could make the math optional to do science in a way that it isn't right now. So it could work both ways.
Yeah, I mean, I think I trust Terence the most on the implications of having accelerated math progress and what that means. I think really as a researcher, and just speaking on behalf of a lot of the researchers here, I think the most exciting applications of our models are when they're used to accelerate science. I think really like trying to provide this kind of very general-purpose tool that experts can use in their daily lives to just accelerate their work. I think across other sciences, right, like we've seen people in material science, people in healthcare kind of already use the reasoning models and have testimonials to the fact of like, hey, you know, this is almost like some undergrad that I can kind of give tasks to and they can come with fairly coherent analyses of certain situations. Or kind of like Terence said, like a lot of people will be like, hey, you know, here's a scenario, can you kind of do some calculations and what would the implications of this scenario look like? And I think people have found it fairly effective in those situations.
No, absolutely. I mean, I suppose where my mind is going is that very rapidly you hit a world in which a very small number of people could actually vet whether or not the answers you're being given are correct. And perhaps the structure of theorem proving plus an ever more sophisticated LLM in math is the only way, if you actually get a scalable verification solution to that problem. And so in a way, we always have to have formal mathematics at the top and then everything else is derived from it. Given that that's a potential future and some of the other themes we've spoken about, Terence, do you have advice for young mathematicians on where they should be focusing and the kind of questions they should be tackling?
I think, yeah, my advice is they have to be flexible. That I think mathematics is becoming, you know, more technology-infused and more collaborative. And maybe 50 years ago you could specialize in one sub-sub-subfield of mathematics and barely even interact with other mathematicians and you could make a living out of that. And that's basically not so feasible now. I think math is part of a much larger ecosystem, which is a healthy thing. And with AI, it unlocks much broader collaborations than previously thought possible. You know, you could collaborate with scientists in a domain in which you really have no expertise, but the AI can help you get up to speed at a basic level and serve as sort of a universal translator between scientists. So yeah, just be open-minded and recognize that these tools also have limitations. You know, and you can't just sort of blindly use these tools. I mean, you still have to build up your...
own human skills so that you can supervise the AI. Yeah, it isn't a magic wand. Yes, I don't think even we of OpenAI would be encouraging folks to use it without quite a heavy bit of expertise and oversight.
Maybe a similar question for you, Mark, but slightly different, which is just based on the trajectory that you're seeing, what skills would you encourage students to be picking up now to be able to make the most of these models over time?
Yeah, I mean honestly, like technical fields, we still need technical experts in technical fields to essentially who can kind of synergize with the tools very well. I love the idea, just the general advice to stay flexible, right? And to show kind of like AI research a little bit, I think it'd be very helpful for people in just a variety of fields to at least kind of understand the basics of, you know, how neural nets work, how they're trained, you know, what their dynamics are like, and just like what their limitations are, right, as an implication. So I think just the more that people kind of play around with it and yeah, just understand how you can accelerate them, I think the more effective they will be. I do think there will be a kind of multiplier on everyone's efficiency maybe a couple years down the line, right? And that multiple hopefully will be like, you know, significantly greater than one. But I do think people who effectively leverage AI tools will be by and large more effective than people who are just kind of blind to it.
Yeah, that certainly resonates. I wonder if the key question has become less 'will they be useful' and more the speed of their evolution. In some ways, Terry, you've been on the inside watching as these models get better at different moments in time. And I do hear, you know, recently the performance on IMO and silver level, though accepting that there was a little bit of jigging going on to make that happen. Have you been surprised at the rates of progress?
It's been sort of both exceeding and also underperforming my expectations. Yeah, so it seems like in any task for which you can generate data of similar tasks, so you know, for example, the IMO thing, you know, DeepMind generated a lot of synthetic proofs, actually a lot of synthetic failed proofs, that was actually part of their secret. So a lot of tasks which I thought would not have been doable for several years are now done. On the other hand, every time you sort of go beyond the sphere of where there's data and you go into a research-level problem where there's like only 10 people in the world have really thought hard about this question, it's still, you know, the AI tools are still not being so useful. So I had this project I'm still running right now where instead of proving one big problem, we're proving like 20 million small mathematical questions. And I thought this was a task in which AI would be ideal for, you know, because if they could handle some percentage. But it turned out that, you know, all these questions that this project studied, you know, maybe 99% could be handled by kind of more traditional computational brute force methods, and 1% which was quite hard and quite human intervention. The AI that have been tried, they could recover much of the 99% of the fairly easy problems, but they didn't really contribute to the hardcore of the really challenging questions. So that could just be the nature of the state of the technology today. Yeah, I don't see... there would have to be quite a few more breakthroughs, I think, before you see them sort of autonomously solving these research-level questions.
Yeah, I think just to speak to one anecdote in my mind that speaks to this kind of like, you know, impressive and at the same time, you know, room to go kind of angle. I think, you know, we participated in the IMO this year as well with our o1 models. And I think on one hand, you know, it did take them a lot of samples per problem. I think like we announced in our blog post like you need 10,000 samples per problem to extract kind of gold medal level performance from the model, which feels like a lot. But at the same time, it's like just incredible to me that it can do this at all. And you know, some of these are very like anti-pattern style problems. And so it's like somewhere in there. And I think I'm just really excited by kind of really getting that capability out.
Yeah, it's funny that it always feels a little intellectually unsatisfying when you feel like you've almost cheated in a way because you've reconstructed the problem. But then I zoom out and I wonder how much of scientific progress is just lots of that stacked together and then it creates a paradigm shift that in retrospect seems very clever but was actually just little things together to some degree.
You know, the joy of programming is exactly that, when you redefine a problem such that it can be solved first, necessarily first principles working your way through it. It does raise the question for me though, which is maybe what we're talking about here is that we're teaching the models to reason in a specific way and that category of reasoning works well for some type of problem. Do we think that, and maybe starting with you Mark and then on to Terry, do you envision a world in which one class of models does lots of different types of reasoning simultaneously, or is it more like a world in which you have sort of individual models doing different types of reasoning that comes together? And then for you Terry, what kinds of reasoning would you need to see to think that you could unlock using AI some of the more challenging, the smaller subset of questions that currently they struggle with?
Yeah, I mean, I do think there's beauty in just having one model that can kind of reason across a bunch of different domains. I think when you try to hook up a lot of complicated systems, like you make a lot of design choices and I think, you know, simplicity is really one of the key mantras in AI development. I do think like, yeah, you could set up structures, of course, of AIs that collaborate in a certain way and that's also very exciting, right? Like, you know, could we build out this model of, you know, like you're a specialist here and you're the PM of this math project and you know, you're the proof writer and you're checking like the 10,000 cases or something like that. Yeah, I think like that's also a very interesting paradigm to take.
Well, I mean, I definitely see AI problem solving as a very complementary way. It's a very, you know, a very data-driven way of solving problems. And as you said, you know, for certain tasks it's actually much better than humans. A lot of perceived difficulty, what we're learning actually is that our perception of difficulty of certain tasks has had to be recalibrated because we just didn't try to use a data-driven approach to solve certain class of problems. But some problems are genuinely hard. Without, I think, I mean, you know, in math there are even questions that are undecidable, you know, that no amount of data can actually solve certain problems. We can actually prove that they can't be proven. But yeah, so I think, I mean, this is not really the AI strength, but you know, if you want an AI to really compete on solving math problems the way that humans would, they need to reason in data-scarce environments where, you know, there's a new mathematical object that you're studying and you know five or six facts about it, you know, a small number of examples. Maybe there's a very vague analogy with some other mathematical object that's already out there and you have to just extrapolate from a very small amount of data what to do next. And this is something that AIs don't excel in. And maybe it's entirely sort of the wrong, I mean, I think trying to force AIs to do that is like using the wrong tool to achieve a task. I mean, this is something that humans are actually really good and efficient at. It's all the brute force checking and case analysis and synthesizing, you know, finding the patterns that they're not so good at. So I don't know, it may be a mistake to think of intelligence as sort of a one-dimensional scale and which one's better, AI or humans. I think they really should think of it as complementary.
Yeah, I do hope if we're successful in our research program that we'll have very data-efficient reasoners too. So hopefully we can prove you wrong, Terry.
Okay, glint in Mark's eyes as you were talking. I see itching to jump in there. Okay, well, I would love to be proven wrong, yeah.
We're coming up to the end of our time both, so maybe to end this and as a way of tying this all together, if you were both, you know, tomorrow appointed to be Vice Chancellor of a university given some meaningful budget, what would you set up to make an effective, in your case Terry, math department, in your case Mark, maybe broader science department, and what infrastructure would you be investing in to really take advantage of these new technologies?
That's a good question. I can imagine having some centralized computer resource to run local models that you can tune yourself and so forth. It's a little hard, I mean, the technology is changing so fast that an investment in any specific hardware or software now may not be so important in a few years. Yeah, so some, certainly, some location where you can bring together lots of people from different disciplines and to figure out ways to use these technologies together. I mean, we're already developing lots of these tech hub type things already. So yeah, but I think it has to be very free-form and because the technology is so unpredictable. But yeah, we need the different departments to talk to each other and see where the synergies are.
