Terence Tao and Mark Chen - Fireside Chat with James Donovan - IPAM at UCLA

Well, thank you very much. Before we get going, just a massive thank you obviously to the institute for hosting us today. Beautiful space. And also for all of you for turning up. I know you're not here to hear me talk, so I won't talk for too much longer. So really just to say also a massive thank you to you both for coming. It's rare that you get two such great minds in the same place. So we really appreciate the time going into this.

It's our third time.

Yeah. And a little pattern is starting to build up. And maybe actually that's a good place to start that conversation. You guys had a conversation almost a year ago or so to the day and at the time Terry I think your prognosis for where GBT was for mathematics was something like a very ineffective grad student.

Which remained with me because I'd heard that feedback myself as a human being.

So it was a clear benchmark.

Guys for Yeah. Okay. Yes. That was my opinion.

Why don't we start with how you think things have changed since then and then Mark hear your side of the story.

Okay. Yeah. So a lot has happened in the last year. That's not just in AI but yeah so these tools have definitely become a lot more powerful. I think there are now capabilities that basically are now normalized and like we just use them all the time. So deep research tools. Literature search has become really really good. It has surpassed traditional searches. Code generation of course is the big thing but as a pure mathematician I'm not as heavy user of code but it has changed the way I approach a math problem. I will plot something, if there's an inequality I think is true I will ask an AI to try to prove it or disprove it. I already use it. If there's a lemma that I don't think I know how to prove but I just can't be bothered doing the pen and paper calculations. I will just outsource it. I've not yet found it to be useful at the deepest level of when I'm trying to solve a problem and with pen and paper or with a colleague I can't sort of interact with it on a conversational level quite at the level I need yet. But maybe in the future. But I think also socially I think we're beginning to, the mathematician community as a whole is beginning to understand that these tools are here to stay and we have to actually start adapting how we do our research. So certain things that were very tedious and maybe we would force our graduate students to do, you know, we can offload to AI and this opens up lots of new ways to do mathematics, lots of research projects especially at scale that we just could not dream of doing. So while I think we can use AI to assist our current workflows, it's a little bit awkward still to do that, but I think there'll be much more mileage out of creating new workflows which are optimized for AI. It's like when we invented the automobile, we started changing the way we built cities to, you know, and of course you could say that maybe not all the changes were good, but yeah, we're sort of in this intermediate stage where somehow our roads are still for people on horses and we now have automobiles.

So, would it be fair to say that we've got to the point where occasionally helpful collaborating? But maybe more interesting that all the bigger open spaces, how you change the way you do maths with these tools coming. Mark, would that be true to what you're seeing and what you're building for?

I mean I'd like to come to maybe first proof and transition as we go into more front-end mathematics, but maybe to stay with the capabilities right now. I think often the Erdos problems are seen as a way of getting a litmus test for where the models are. And that maybe is a representative set and that some of those problems are maybe not as complex as other parts of those problems and were designed that way and that you might say that the success of the models has been in doing lots of the easier ones quickly versus necessarily moving towards the kind of ACON level problems. Is that a fair depiction in your mind of where they are today?

Yeah. So I've been heavily involved in tracking the progress on the Erdos problems in particular. So I mean and it is basically largely true what you say these problems range widely in difficulty. There are some that we desperately want to have solved and they've been worked at for decades. You know I have papers you know making tiny progress on some of these problems. And to date AI has not really helped with the ones that we've already poured a lot of attention to. But there was this very long tail of problems. Erdos posed a thousand problems. They weren't all winners. But he understood that the important thing was to stimulate discussion and interest and he kind of knew that the problems that were going to be important would take on a life of their own. But those are as long unexplored problems where there's maybe almost no follow-up literature. And that's where the AI tools have really made a lot of spectacular progress. You maybe 20 30 of these problems have been solved with fairly minimal human supervision by these AI tools. And we were able to verify them too often with some other AI tools in the form of verification. And we've kind of worked out a kind of a workflow for doing this without being overwhelmed by AI slop incorrect solutions or whatever. So yeah it is a new capability that we hadn't before. Because we can now attack attention bottleneck problems. And so I think what this suggests to me is that we need to start creating more and more broad challenge sets of problems for AI tools also but the general public. So actually in the same period many of these early problems were also solved by amateur mathematicians sometimes with AI tools, sometimes without. The same mechanisms that the same kind of workflows that enable AI to be successful also actually enable amateur mathematicians to be successful as well. So I foresee a change in our culture where instead of only working on a small number of really hard problems and not sharing a longer list of other things we'd care about, we all start as mathematicians, we all start releasing problems of things that we want to get answers to. And 100 problems and maybe this AI can solve 10% of them and maybe this other high school student can solve another 5%. But this we can get a much much more community-driven way of doing mathematics. So I think this is what the only problems I sort of get an early harbinger of.

Yeah. And it's maybe interesting to contextualize that other domains of science at least in my own world of biology in which the number of people collaborating in any given paper is just exponentially risen over time. And that seems to be the trajectory that science is much more of a team sport. Maths is maybe and to some degree of physics the outlier in that domain. When you're thinking about this Mark is it always just a question of how smart can we make the models and the ever more difficult questions they can answer or is it also a question of how do you empower humans to work collaboratively on these problems?

I think it combines very well actually. So I think what AI will enable is finally a way to use division of labor which is something that like all industry you know since the industrial revolution like every industry has managed to become more efficient through division of labor except mathematics. So you know so to do mathematics traditionally there's several different tasks to you know there's problem generation there's strategy generation and then strategy selection among all the strategies you generated and then execution of the strategy verification of the strategy communication of the results. And so you basically need we've trained our mathematicians to sort of be somewhat good at each of these tasks. So we specialize in field but we have to have some idea where problems come from what are good problems what are good strategies you have we have some technical skill we have to verify and we have to explain now some mathematicians are better at some of these than others and so we have been able to benefit from collaboration because of this. But we can't really specialize the same way that in the sciences you know you can have you kind of have technical staff and you can have people who are project managers and things like that. So but now with AI and other modern collaboration tools and formal verification it has become possible to run math projects where individual participants specialize in just one of these areas and maybe there's some gaps you know among your collaboration no one knows how to do the technical thing but AI can plug in some of the gaps in a collaboration. But you still need the humans because the AI performance is very very jagged. So maybe some of these inputs can now be automated but if you automate that too much you know for example if you can automate strategy generation okay but you can't automate verification then you just get this whole hundreds and hundreds of possible strategies but anyway AI generated that you can't deal with that but if the verification also keeps pace then suddenly you have a new style of doing mathematics that could be extremely effective.

So we've seen with the problems that you get an AI to try to solve a problem. It will first thing you do is go to the problem website, look it up, say, it's an open problem. It's too hard. I'm not going to try. So just say do not use the internet. I try to solve the problem yourself. It's actually pretty easy.

It's good to know that it's, you know, Frontier Research is actually just about coaxing the models into behaving the way you want to. That vision right now is probably quite a compelling vision for this room and beyond where we're sort of saying the technology fundamentally empowers more people to collaborate on these problems. But is this just a stepping stone to a world, Terry, in which you're only collaborating with many AI agents and slowly but surely they come to dominate the sphere?

I think yes and no. I mean I think the type of math that we do today might slowly kind of move in that direction. But there could be very new types of doing math that we can't even envision right now. Which I think so math is infinite and the difficulty levels are unbounded there even problems in math are unsolvable we know they're unsolvable okay there's an asterisk but I don't want to talk about it okay you know so like there's certain things that even the most powerful AI we can't there's even right now there's certain cryptographic challenges that yeah I mean AI cannot you know mine all the bitcoin right whatever. So I think there will always be a frontier. And I'm pretty sure just because how complimentary human and AI at least current generation LLMs are with human skills that the best combination is always going to be a complex combination of humans. But the nature of the combination may change over time.

So let's assume even just philosophically there is this frontier beyond which at least the current paradigm of AI wouldn't be able to cross and some centaur like human AI collaboration is needed getting to that frontier in your mind Mark is that a question of much smarter RL training or is it actually just a question of raw computation you know if I could give you an infinite amount of compute today would you be able to accelerate your way to that frontier?

The term hill climbing, if I've learned anything working with the research team, is that we always must find and define a hill to climb. And perhaps that's where these two worlds come together as defining the right hill. So maybe if we focus on that for the next segment. First proof would seem like an obvious example of where we've tried to codify a hill to climb. In your mind, Terry, is that representative of what you're thinking could be the new emergent math to come or is that the final form of this more classical math that AI has been working on?

There'll be a spectrum. Yeah. So, first proof is a very interesting experiment. And the proofs that the various people with AI tools generated were quite good. What we saw actually that there was a definite verification bottleneck. So we had a lot of proofs generated of some were terrible, some were quite good, some were similar to things in the literature, some were similar to the proofs that the authors themselves had. There was a couple which were actually different from the official proofs. And so that was interesting. But to evaluate carefully exactly how novel and how interesting each proof was like you don't have a way of doing that effectively. So I think that the first proof team are going to create a more structured competition later where they will have some mechanism for verification. So we in order to take full advantage of the new capabilities that AIs have we do need to create challenges that are easily verifiable. So somehow the level of automation and AI power that you can profitably use before it becomes slop is roughly proportional to how stringent your verification is. So yeah so I think initially you're going to see a lot of progress in areas either which are sort of elementary enough that they can be relatively easily formalized so combinatorics I think you're going to see. So the early problems definitely fall in this category there's some numerical type challenges where you want to find a configuration mathematical object that obeys certain properties once you have the objects verifying them are very easy. We saw some examples actually here in physics. There's some similar type problems. I think there we will see a lot of progress. But there are other parts of mathematics where the objective is not to find an object that obeys a certain property but to find a good overarching theory to explain something or a good definition and those we have a lot harder time to verify like you know if you want to propose a new conjecture or a new strategy to answer a problem to you know maybe AI can generate a hundred of these possible strategies but only a human expert can verify or can give an informed opinion. And so that will be a bottleneck. So even if AI drives the cost of sort of creating solutions down to zero there are still other huge bottlenecks which we didn't which were not front and center in our minds. But yeah I think we also need to become very much better at stating goals precisely. So AI is almost too good at fulfilling a goal to the letter. So you ask I want to solve this problem. I want a proof of this theorem. And maybe a future just runs for now. Here's your proof. But actually what you wanted was you wanted people to work hard to fail to find examples to connect to other literature and to communicate all the partial results. And that was actually the value of solving a particular problem. And there is a danger that if you specify your goal to an AI too narrowly, you miss out on most of the benefit. So we'll have to be more careful about goal specification.

Yeah. Right. Although it may sometimes be useful to turn that off, you know. So I mean I've worked sometimes on a problem where I know too many techniques. Okay. And there's a powerful technique I know will solve the problem, but it requires a lot of technical skill to use and so I do it and I solve my problem and then I publish or something and then someone points out actually you had to use this much simpler tool. You have a much simpler proof. Yes. I do worry a little bit that sometimes having access to every single technique known in the literature is not necessarily the best way forward. But having a diverse array like multiple AI tools and I think there will still be people who will take pleasure and pride in sort of doing things old school and finding sort of more human ways to solve problems too.

Well I wonder if that pattern even plays the case study you gave Mark. Where it's true if we could go back in time just before a particular paradigm shift in whatever domain of science and then see whether or not the model would predict it that could be one verification tool. But I guess the kind of Kuhnian vision of paradigm shifts could also mean that in fact there is a future paradigm shift to come that would invalidate the prior paradigm shift. So you don't actually want the model to guess the previous one because it might take it off a pathway that doesn't get to the next one as you go. And it throws into relief I think this question of verification validation it's both a philosophical and a practical question of all the domains you could argue that maths and disclaimer here I did work a little bit on lean so big shout out to that team has the actual capacity to do automated verification in a way that very few other domains do not perfectly and not without its own drawbacks. Is your instinct that that structure where there'll need to be a sort of separate validation tool will need to come into existence for all the other domains of knowledge that we want to work on that it will mirror what's happened in maths or it will need some other type of paradigm.

I as I said I definitely believe that there's an upper bound on how much AI you can inject into a workflow before it becomes a net loss that it is causing more errors and problems than it is solving. And one of the biggest upper bounds is the ability to verify. So yeah so in math I think we have the best shot at getting really high levels of automation of being able to effectively use excuse me high levels of automation in a way that you couldn't do in less verifiable domains because we have a high verification bar at least for the specific task of proving things which is not the only thing that we care about but and proving things that we've already specified we want to prove. Yeah but although even for verification does have weaknesses the language itself can be exploited by malicious agents. So and so an AI may sort of you know attempt to be helpful and try to prove as many things as possible just secretly add some axioms to the formal system and things you can try to shut them down but if the AI is too powerful actually yeah I mean at some point you actually have to sort of limit how capable your AI is or you know or have periodically humans involved in the loop. So you know there are other in the other sciences you can do some of this. So for example numerical simulation can be used as a verifier in some cases but again you can't rely on like if you say what you want to model the weather and you have a supercomputer that predicts the weather and you have trained an AI to mimic the numerical simulation it is possible that at some point they will just exploit some feature of the numerical simulation that is not part of the ground truth so it will work up to a point and then it will stop. So we do need to get a lot better at knowing the limits of our verifiers. So you know a lot of verification systems that we have, they work just fine if they're used non-adversarily. But you know if you're training an AI specifically to maximize output based on using this verifier, it will find the exploits. AI is so good at that. Yeah, it's a ruthless cheater. So we do have to be aware of that and just because a human verifier surpasses all human tests it may not be suitable for AI use.

That makes a lot of sense and intuitively to AI cheating the easiest way to make something measurable is to design it to be measurable from day one from step one. Mark do you think like that when you're trying to make the models ever smarter? Are you thinking in terms of first principle what would have to be true to be measured as being smarter or do you rely purely on generalization to try and get ever smarter models?

Beyond the structural nature of maths and therefore the ways that you can formally verify. Is there some other practical benefit to pursuing ever greater capabilities in that space or is it in your mind really more just equivalent of an eval?

Do you want? Well, I think one positive feature for using math as a test bed for other use cases is that so you know we had this quote earlier today of Vladimir Arnold that mathematics is a place where experiments are cheap. It's also the place where failure is cheap. So you know it's related you know so you know if you're an engineer and you ask to build a bridge and the bridge collapses that's an expensive mistake. You know if you're a surgeon and you asked to and you cut the wrong thing that's an

In math, if you try to prove a theorem and your proof strategy doesn't work, that's not an expensive mistake. So I think we have this freedom to fail, which is more so than in other disciplines. And because of this, we have a culture of learning from our mistakes a lot more than in other disciplines. So it's a relatively safer place to experiment with AI than, let's say, bridge building or heart surgery.

Yeah, it makes a lot of sense. I mean, I even wonder if you can push that, and Kevin, your work touches on this. Assuming a world in which the models may be discovering things that really are beyond the frontier of human knowledge, or even the ability for a human to really conceptually follow, presumably there needs to be some way to re-represent those findings into a logical chain that at least we can follow the steps, if not the actual constituent parts. And that maths, more than any other domain, seems to have invented a workflow that accommodates that. Certainly compared to, say, biology or chemistry, which doesn't have as much by way of kind of formal axioms that you can work against. So potentially it's a necessary precursor to any truly frontier science advancement to have this capability. Regardless though, it does, to your point, Terry, suggest that the way that we think about doing math in the future will change, that we're emphasizing maybe creativity, collaboration, different skills perhaps than what have happened the last 100 years. Does that filter then through into how you teach maths?

Yeah, it's... and so open problems still, how to... yeah, so in the very short term, some things have had to change. So yeah, like homework, weekly homework assignments have been the first casualty. But you know, I think we can push our students to do more ambitious things now. So I switched much more to a project-based type of assessment. In smaller classes, you can do some oral assessment. The skills we need to teach will be different. So validation, independent, the ability to independently verify AI-generated output will become essential. Softer skills, how to work with people, we have not been uniformly good at that in the past, but we'll have to get better. The pace of change is such that education systems are not catching up as rapidly as... but I think by necessity we'll be forced to. I mean, with COVID, for example, we did do some emergency changes to our curriculum and it kind of worked. It was not a great experience. So hopefully this time we can do a bit more planning. But it'll be on that level of change, I think.

Yeah. So definitely I've noticed homework scores going up and in-person scores going down. Not so much... it's not at a collapse level. I mean, I don't have hard data. I do get a sense that the weakest students are using AI to sort of get to a median level. And the brightest, brightest students generally tend to avoid using AI because they're worried about... they can notice that using it too much atrophies their skills. The weakest students, I think, they feel like they have less to lose. But which... so it in some sense is an equalizing feature in that respect. And yeah, I mean, once you have a certain level of expertise, these tools are great. And so maybe the equilibrium is to actually discourage their use or use them in specific ways. I can certainly see homework assignments of the future where the solution is not the point, because anybody can enter in, but for example, what prompt did you use to get to the solution? And that might be the more interesting assessment tool. So yeah, we have to figure it out. And actually, it's important, just like AIs will optimize whatever reward function, the reward function we give to our students will actually make a lot of difference. Yeah, we have to think this carefully.

Yeah, it's... in some ways the extreme cases are easy to understand. Total cognitive offloading and therefore no learning is occurring, etc. Maybe the more nuanced though would be something where it is a productive use of AI, and I think you used this metaphor in a recent interview, which is to say you've been helicoptered to the destination as opposed to taking the scenic route there. There's something about the change in workflow corresponds to something at the cognitive level for humans, and that we don't know yet what it may be that you lose if you do that, but we need to be alive and alert to it. Do you have a thesis about what we might lose if we start doing that?

Um, I think we will see empirically pretty soon. So yeah, I think we just need much more awareness of all the different facets of research or any other task. So somehow... so I said before AI allows for a decoupling of many things, which can be good for divisional labor, it's more efficient. But it does mean that goals that... where previously it was okay to set very fuzzy goals because any human attempt to reach these goals would sort of also hit all the nearby goals as well. So yeah, as I said, with an AI, you know, if you want to go see a nice waterfall or something on a mountain, you take a hike and sometimes you see some interesting wildlife or you get a glimpse of an even nicer location that you might want to go to someday. And maybe you meet some other hikers and you have conversation. There's serendipity, which just naturally happens. And so in the past we would just say it's a good idea to go visit this waterfall, but we didn't sort of unpack that carefully enough to say why we do that and what are the actual values, what are the actual benefits. And so but now we have this alternate way to get to these waterfalls. As I said, you can get an AI helicopter to drop you off there. And so yes, you get your little Instagram photo, but maybe that's not the only thing that you wanted. So yeah, I think unfortunately we're going to have to learn this by experience. It's hard to... I mean, you can talk romantically about the journey and things, but I think it's only when we see what happens when we don't have that, that we really understand what we're missing.

Yeah. I mean, I should say just today we released our learning outcomes measurement suite, which is how we use the models to assess whether humans are learning when they use them. So agreed as sort of a live research question. But I wonder, Mark, for you, serendipity and the idea of an inexact answer from the model in order to create space to explore, is that a quality of the models that you're interested in exploring? Is that a model behavior question, personality question? How do you grapple with that?

Yeah, it's much harder to do reinforcement learning on collaboration. Like, how do you score how good you're vibing with your co...

I agree that it might not be as difficult as you imagine. There are probably quite clear biological signals of what vibing looks like in the real world that you can bring back into the machine world.

There we go. I'm glad that I've got a quote that will be kept after this. I will open up the floor for questions just after this. I'll ask one more just to give you all time to think about it. Perhaps we will meet again in a year, and I'd be interested to know what your predictions look like for where we'll be in one more year's time.

I really hope we're going to see a lot of new types of mathematical projects that are kind of challenge-based, where, you know, so like first-proof type things, where some group of mathematicians, for example, will create a really good set of problems that they would like some proportion of these problems solved, and they have a very good gradation of difficulty, they have a very good verification protocol, and they would just open it up to the community. And so just sort of taking full advantage of, well, not just AI but also just like the internet and Metcalfe's law, you know, if there's N people who can produce problems and people that can solve problems, then there's N squared possible connections. And mathematicians have been very bad at using sort of making this large-scale network. So I think we'll see a different, almost like a marketplace style of doing mathematics. And there I think we'll see AI shine. So this is one thing I want to see, and maybe in a year we'll start seeing that.

Fantastic. There was this metaphor in software development of bazaars and cathedrals. A bazaar being a self-organizing thing that springs up and is very diverse, a cathedral being one great mind architects and is therefore very elegant. And that the idea perhaps will be that in math we get both those phenomena occurring, and hopefully that is a flourishing. So I do want to open up to questions. Does anyone have a burning question?

Yeah, I wonder whether any of you can talk about the word models which instead of predicting the next token, it predicts the next state. And what I'm reading about the V-JEPA is that it can actually self-correct things because the hallucination can be avoided. Is that all true? Because whatever I could run on my Mac, context model is just a hello world thing. So I don't know much about it.

For the benefit of those watching, the question was about how do we feel about world models? Are they a paradigm shift, and would they be specifically useful for maths and solving hallucinations?

I think it's potentially a very promising alternate direction. LMs are great, and in some ways they're too great, actually, in that we've kind of routed our entire AI infrastructure around making the LMs as powerful as possible, and it could crowd out some other very complementary ways to create AI assistants that are jagged in a completely different way. So I definitely support research into world models. I think for a long time they will underperform the LMs because we had just, because of all the momentum and infrastructure, have... it's like we have built our cities around the automobile and gasoline, and we have this entire infrastructure, and it's actually making it hard for alternate transportation to break through. But there are definitely people who are pushing that, and I wish them a lot of luck. Yeah.

I wonder on that if there is a middle ground, in as much as you can construct RL environments that are based on the laws of physics or follow the laws of physics, which to some degree confers the benefits you otherwise get from a world model to an LLM or anything else. So perhaps it's an intersection rather than two alternate pathways.

Yeah. So AI in science, in many fields, well, is effective when it's predicting very well. For example, protein folding, predicting weather accurately. But in mathematics and in theoretical physics, we're asking for something different, right? We want to kind of understand, get a formula, get a proof.

But is it... do you think it's potentially too limiting? Will it be easier to get an AI that will tell you, I have a proof of theorem, hypothesis, but your brain is too narrow to understand it. So I can teach other AIs about it and do more progress with it.

Yeah, that's definitely my relationship with AI already. So, some of us have hit that frontier.

Again, for the benefit of those watching, the question was in some domains of science, I think we're satisfied with pure simulation. If it can do the thing, we consider the thing to be proven even if you can't formally verify it. So, if it can predict weather accurately, even if we don't know how, we're kind of happy with that outcome. We hold in maths and physics to a different standard, which is that it must be verifiable in the way we already discussed. Is that somehow limiting or a mistake to put that restraint?

Yeah, I think there'll be different types of mathematical tasks that we don't do nowadays which we will entrust AI to, and they could be quite complementary to the task of getting a formal proof of a problem. To give you an analogy, so in chess nowadays, all chess players train using these chess engines. And one thing the chess engine does is it gives you the score at any time in his position. You know, white is three points ahead or whatever. And it's a really great signal to train human chess players. You get instant feedback. Oh, that was a really bad move. I'll try and do this instead. I could imagine an AI which, you know, a human is trying to do a proof and like every time you say I'm going to try to prove a contradiction, your score goes down. Okay, that's a really bad idea. Okay, you should back up and do something else. So, maybe a Swan electric shock. So, that's a pro... the pro model. Yeah. Okay. So, yeah, a good math teacher can do that, but yeah, but maybe so we have to be creative. The type of tasks that maybe AI could help with that we just don't think about today.

So on that, can you get anything out of the space stuff, like see what the intermediate might think?

These two questions may link up in as much as for interpretability, the way that we collapse the latent space does limit the kind of associations that could come out of the model. At what point do you decide that it's better to maintain the latent space because of the theoretical new connections it could make at the cost of interpretability, or do you just think that we need a different interpretability paradigm that doesn't require that kind of collapsing?

Well, ideally we should have a diversity of models. So maybe there are some applications where you just want the answer, you don't care about interpretability, and then you just turn the dial one way. Okay. But there are other applications where you really want to see the process, you really want to see a human-readable chain of thought or whatever, and you turn the dial the other way. Yeah.

Yeah. I mean, it does feel intuitively like having to compress things into language does come at a cost. So even alongside a plurality of models, you might also want a plurality of expressions or verification methods so that you don't somehow force it into a shape that doesn't make sense. Sorry, please go ahead.

I have a question about attribution and incentives a little bit as we look to this kind of future of science. One thing with AlphaFold, you know, sort of the world, I think largely thinks like AI came and solved that problem. And of course, AI in some sense did, but it was sitting on the protein data bank and decades of effort. And then you see the protein data bank loses its funding immediately after and various things like that. And I'm just curious that, you know, as we think about like these large-scale math problems and all of the human effort that's going to need to go into producing the sets of problems and verifying them and etc., it does seem like there are for many of these things, theoretical physics problems, all of them, that there's a lot of danger that in some sense while AI was maybe like the critical enabler that made something happen, it didn't happen on its own, it's really this ecosystem. And that so on one side it's how do we control that narrative, and the other part is in some sense like for OpenAI and for the big companies have a lot of responsibility in some sense about how they navigate that. And I'm just curious about, you know, how do we avoid it going in a bad direction.

Um, well, yeah, this is an important point. A partial solution, so as I said, I do envisage the rise of like challenge problems where people will create these data sets of tasks that they want solved, and there it's kind of win-win because the people who create these data sets, they will get some fractional problems solved, which is what they want, and then but these data sets could be very useful to calibrate AIs. And so there are some cases where it can be win-win. But yeah, there are definitely cases where people have built a data set at great expense for not for this reason, and then it gets absorbed into various AI. And yeah, I don't know how well we can track that. Yeah, so it leads into like intellectual property law, and it is a very tricky problem. Okay, which I will toss to you to deal with.

And they come somehow like humans, experiments, and all this is not so. With the math problems, what we've seen is that there'll be times when there's an open math problem that no one has looked at, and then some AI solution gets a solution, and then this hits social media, AI solved an unsolved problem. Okay, and in many, many cases, you know, like 24 hours later, someone often armed with one of these deep research tools uncovers that this result was already proven by a very similar method in the literature. And we can't say for sure whether the AI used that solution or indirectly was aware of it. But it happened so frequently. I mean, we have this whole table of contribution. It's like the whole section we just... this... and so to some extent, we at least have the capability to detect at least some of this, because we also have these research tools. So we can recover attribution sometimes. It's not perfect. But yeah, it could be that the same technology that allows us to use the literature to solve problems can also use the literature to attribute solutions.

Yeah, that's a very fair point. I understand the incentives. I do hope, you know, anyone here at OpenAI can verify this, but like we care a lot about integrity. I think, you know, at least I would very much strongly fight for the correct narrative there.

Maybe just to add a flavor from my previous life, maybe one paradigm shift is that the general public tends to underestimate the degree to which the progress of science relies on improving tools fundamentally. You need better microscopes, so on and so forth. These are not glamorous jobs. It's not typically what theoretical scientists like to work on. And that's a narrative we should be saying, is that by building ever better tools for science, that's what enables humans to accelerate the entire field, often, you know, really big steps forward. AlphaFold is really a tool. It's not per se scientific research, though it does do some research component things there too. And if we can stay on the emphasis of it being a tool, that might also flow back to your point about where does the investment flow in society, because it's to the things that need to surround the tool to make it effective, but ultimately in service of the human researchers or scientists that will use the tool to solve the problem. We are just on the edge of time, so I'll take one more if that's all right.

Yeah. So I think one of the really interesting things about this conference is, you know, it's accelerating math and physics using AI, and I think there's a certain amount of kind of synergy that comes from like being able to solve math and then how that can help solve like other physics problems. So I'm curious to hear your thoughts on like additional synergies you see coming out of, you know, OpenAI through you guys kind of tackling math, physics, and then kind of what else you guys see coming out from here.

Yeah, and say that IPAM, this institute, I mean our entire core mission is basically to find these synergies. We, you know, Institute for Pure and Applied Mathematics is basically in the name. Yeah, we bring together events like this, you know, where different communities talk together. Yeah, so, and a lot of it is this serendipity. I mean, we have some idea, you know, we don't just randomly smash together fields. Okay, but yeah, we do pick ones where we do believe there will be a lot of unexpected fruitful collaboration.

I'm glad you leave randomly smashing together particles for the physicists. That's a good moment to close it. And also to plug that Kevin is about to give a lecture which covers many of these questions, i.e. what needs to be true to accelerate all parts of science and the tools to build for it. So I hope you'll come to that. But thank you all so very much.