Hi everyone and welcome back to the Information Bottleneck podcast. And today we have the pleasure to host Daniel Litt, who is a professor at the University of Toronto. Hey Daniel, thank you for joining us.
Yeah, it's my pleasure. So today we're going to talk mostly about AI and math.
You're a professor, um, that you, you worked on math for, for all your academic life. Um, so I, I want to start like, uh, maybe at the beginning, right? Or like maybe like a year ago you had this bet that AI wouldn't write top tier math papers by 2030.
So this, yeah, so it was March 2025, so I guess about a year and a half ago, and I bet Tomasoglu, that, uh, Tomay Vasogu, that, so the specific, the precise bet was that there would not be a paper autonomously produced by AI for a budget of $100,000 that in my judgment could have been published in the Annals of Mathematics, which is like one of the top math journals, in 2025. So I think actually strictly speaking the bet has not resolved, like right now the models cannot literally do this. Um, I think there are some results they've proven more or less autonomously, uh, which could get in there, but like they actually can't write a paper. Um, like the, the papers suck. Like you would not be able to get them published. But yeah, I, I just had to concede the bet because like the whole point was, you know, of course you can, you can try to, you can try to look, you know, analyze technically the bet, but clearly like I morally lost the bet, like they're, they're doing interesting math.
So what, what has changed, right? Like what actually, like now let's say AI models can do that they didn't do like a year ago or a year and a half ago.
Yeah, yeah, it's kind of interesting. I mean, I think like my expectation at the time was that like there were like a lot of things you needed to write, uh, like, do high quality mathematics, and it seemed to me at the time that the models were like getting good at some of those things and not getting good at others. Um, so this was, I guess, March 2025 was maybe around when o3 came out, and the models were like start, starting to be able to do like technical things like somewhat reliably. Um, but like, you know, kind of just didn't really have a good understanding of, of a lot of basic objects. Um, and they were like very uneven within different parts of math, uh, or like between different parts of math. What I think, uh, I was just kind of wrong about at the time was that like I thought you needed like all the skills that people have in order to do high quality math. And like instead I think what's happened is the models have become like really superhuman at some things. Like they like know, really know everything in the literature pretty well, for example. And they're actually like still way worse than us at some other things. Um, but they, they're able to kind of, uh, substitute some of that, like, like the skills where they're superhuman for some of the skills where they're not so good. Um, so yeah, I think where I kind of, uh, was wrong and like I, you know, I've kind of learned a lot about what it takes to do high quality math is just like, you know, just by being very, very superhuman at some, some kind of bundle of skills, which is like a relatively small, I think, portion of the bundle of skills that people use when they do math, you can still actually produce pretty high quality.
So what, let's try to like decompose it, like what models are really good in math and what they are like not so good, and like, and do you think they're like getting improved like also?
Yeah, I think they're definitely... Okay, so like there's like a bunch of things I think I do when I do, when I do math. Like, okay, first of all, like I have some knowledge of what people know and like basic techniques or whatever, I know a bunch of examples. So there the models are like super, super good, like they, they really just know everything. Um, not necessarily at a super high level, but you know, they're kind of morphing.
Let's just a second. So like if I'm taking like existing like math paper and I'm telling like the model to reproduce it or like to prove the thing that's like already in the, in the paper, it will, it will do it.
Well, they'll just even like know what paper you're talking about and like be like, 'Oh, well that's this, this result.' And I mean they won't even have to reproduce it. Like it's just very, very familiar with the full literature. Um, yeah, but yeah, you, you know, if you take, uh, well you can ask about like papers that are not available, like can I prove a lemma in one of my papers in progress, we can talk about that, but I think that's kind of a different, different question. Um, okay, so that's one thing, like just like broad knowledge. Uh, another thing is just like, um, you know, reasoning ability. Uh, it's actually not clear to me how the models compare to humans in reasoning ability. Um, my guess is like probably pretty comparable, but...
Yeah. So like you're trying to, you're trying to get from A to B and you try to, you try to figure out like a path, you know, some, some, some steps you need to take. So you maybe you need to prove some intermediate results or something on the way and you need to like reason to figure out what those results are and like how to prove them. So the models are like pretty good at this, but it's like not an even comparison, right? So like when I try to go from A to B, I do some like huge amount of reasoning to find like, find what the intermediate statements I might need are. And like 99% of the time a bunch of those are in the literature, right? So I'm doing a huge amount of reasoning that like ends up being going into some literature search, and the models don't have to do that because they know the literature super well. So, uh, yeah, so I think that, you know, there's like when humans do mathematics, they, a huge amount of expertise and, sorry, a huge amount of reasoning is spent like rediscovering things that people already know. Um, so yeah, then there's a bunch of other sort of more fuzzy skills like, um, you know, a lot of math is like philosophy, like often when I'm like trying to figure out what might be true, there's like an analogy I have between two, two situations, and there I've just like never seen the models do that. Um, I'm not saying they can't, but like, you know, maybe there's like signs of this happening at some very low level. So there's, yeah, some kind of more fuzzy like creativity skills that like I think, you know, we don't really see from the models in other domains either so much, and those are also relevant to math. Um, I mean I think one issue is like the tasks we ask the models to do in math are like typically of the form like, here's a conjecture which is like some precise statement, like prove that conjecture. That's like not most of what mathematicians do. Like of course, like that's kind of the thing we measure. Like it's like a high prestige activity or whatever. Like you're, you know, working and working and then you like prove some old, old conjecture and like you get a nice paper out of it. But like most of what I do is like more like I have some kind of fuzzy, you know, some object I would like to understand and I'm like just even trying to figure out what's true about it, and that I just don't know how to measure. But like when I ask the models to do it, you know, of course it's hard to, to say precisely whether the output is good or bad, but it's like I don't like the output typically, like it might, my judgment is that it's bad.
But let's try to like, to try to quantify it, like what you actually don't like it. Is it, is it not true because like the derivation are not true, or is it not like, you don't like it because like it's too verbose, like do like what exactly, or like maybe it sounds like AI generated?
Not getting at anything fundamental. Like it's, you know, maybe I want to, you know, I'm trying to say like, okay, well here's some phenomenon I know about in situation A, like I don't know, I'll just, I'll say some words to give you a precise example. So the behavior of the homology of a family of degenerating varieties, some object I like to think about, okay, and I would like to understand what's happening in situation B. So instead of replacing with some other invariance, the fundamental sort of something I like to think about. And so, okay, you say like, oh, I would like to find the statement analogous to such and such, and like, you know, the models will produce some output and it's like just like clearly a bad answer to the question, and they should know that. Um, like it doesn't actually get at anything fundamental. They'll like make some assumptions that are way, way too, way too strong to be interesting. Like they'll try to say something true maybe, but at, at the cost of like, you know, like sometimes I don't know would be a better answer.
So, so you, you've talked about, um, the jagged frontier, the tendency of models to be, you know, ultra superhuman in one place and maybe not so much in other places. Um, it appears further, at least in my opinion, that the rate of AI development progress almost seems like it's accelerating, you know, at least locally. I would argue certainly since 2022. What do you think is the endgame of all of this? Like, do you think that most theoretical mathematics, uh, that a human could even conceive of will be solved relatively soon? And do you further, you know, think that these apparently superhuman, uh, math systems will create basically miracles? Like what I'm trying to understand is, is like, is it mostly just going to be theoretical math that gets solved or are we going to get room temperature superconductors and all the other cool stuff that we would hope from solving all of math?
Okay, so let me just say, so mostly what I've been saying in the last 5 minutes is like a description of current capabilities. My expectation is that the models will get robustly superhuman at math and like at intellectual labor in general. It's, you know, even in the sort of, uh, the bundle of tests I've described where they're like not that good, I think there's some weak evidence that they're getting like a little bit better, and you know, as you say, the kind of the historical trends have been that if, if you see them getting a little bit better at something, then probably they'll get a lot better soon. Okay. And, and my guess is that that's like just the case across like all intellectual labor. Although like to be clear, this is like a pretty low certainty prediction. Like I, you know, I think it's more likely than not, but you know, I wouldn't bet the house on it. Um, okay. So, uh, yeah. Will we see miracles? Like, will we solve math? Like, just no, I think. Uh, there's just no, no evidence for that. And I think that there's like very good reasons to think that like even just basic mathematical questions a person might ask can get arbitrarily hard. Um, so you know, the models will get to some level, but it will still, my expectation is it will be very easy to find questions that they make essentially no progress on, even questions of the form like, does this particular polynomial have an integer solution? Like those, that kind of question can get arbitrarily hard.
But do you think like models will be also good in suggesting these questions?
Eventually, yeah. I, that's some, it's an area where they're relatively weak now, but again there's like signs of improvement. Like right now, if you just like tried to run the models in a loop where you said like try to ask some good questions and then resolve them, like you'd basically get garbage. Like they would try to solve a bunch of open conjectures and then like once they get stuck on whatever list of open conjectures they find, they would try to, they would start generating questions, and like the questions they generate would be like pretty low quality in, in like my judgment or the judgment of most mathematicians. Uh, but yeah, they're getting, clearly getting better at asking questions, and so like yeah, I expect them to get good at it eventually. Um, what's a little bit relevant to the future of mathematics as a human endeavor is like, will not just like, will they get good at asking questions, but like will they ask the same questions I would ask? And that's much less clear. Like that, that's, you know, I have my own personal preferences, I'm not ChatGPT, like probably we keep asking different questions, and it's like very easy for me at least to generate interesting questions. Uh, so it's sort of hard for me to imagine a world where they're just like going to have already asked everything I would ask, even if they get good at asking questions.
But okay, so maybe it's a good time like to talk about like the role of mathematicians, right? Like in the sense that like, okay, so if, if like AI can ask the question and, and answer these questions, um, why we actually care about like human mathematicians in the sense that, okay, so like you can, you can, I agree that like it may be that like you will ask like other questions, right? But like why we, why we care about it? Like why your questions like are more important than other questions?
Well, they're not necessarily more important. Um, I think they might be more important to me. But yeah, of course, it's very reasonable to ask like, why do we want human expertise in anything? Uh, like why do we want, you know, quantum chromodynamics, as you know, that's some fundamental question about the universe which thus far has zero applications, but nonetheless maybe we think we want people to know about it. Maybe let me give you another example. Uh, so telescopes are superhuman at looking at the stars, and astronomers are like not inventing stars, right? They're like discovering something that exists. Um, but nonetheless, like maybe we want people to know about like the structure of the universe or something like that. Okay. So now let's imagine a world where we have superhuman AI. So it's like asking questions that in my judgment are good, and then sometimes resolving those questions. So like the output of that activity is like a lot of PDFs or something, right? Um, okay. Uh, who cares, right? Like, why do we want more PDFs is the question I would ask you. Like, my hope is that we would want those PDFs because someone can read them, uh, and someone is reading them. Uh, and so like, uh, you know, okay, presumably some of the PDFs will have applications. Those might be important for other reasons. And like, well, I also think we probably, if we're kind of going to take a result produced by an AI that at least right now we have no guarantees that it's like behaving well and, and sort of make it, uh, instantiated in the real world, like probably we want people to understand that. But even for like just an answer to some fundamental question in number theory or algebraic geometry or whatever, like about, you know, some basic concepts of shapes and numbers, like my view is that these outputs are worthless if no one can understand them. So like we want a reservoir of human expertise like just so we can actually engage with like whatever great results are being produced.
And do, do you think like, like some people say that like at the end we want that mathematicians or like math in general like generate tools, right? That like the, the, let's say the final goal is not only like to, to, to prove something, but also like to generate like more tools that we can use them in, in other like fields or like other domains. Do you agree with...
Yeah. In, in fact I would argue that the fundamental goal of mathematics is not really about proof only. Uh, like, okay, so one thing we definitely want to know is like what is true, right? So, uh, you know, we ask some question and then like we want to know the answer for sure, and it's like really exciting to find the answer. But like actually the questions we ask are like usually operationalizations of something that is a bit more fuzzy, right? So like the, the actual thing we want is like understand like what numbers, like what numbers are or what shapes are or how they behave, and then like we try to ask that question in like a really precise way, and then we prove it to ground what we're doing in truth. But like the, the usually the question is like meant to measure like how much do we understand this object, and a lot of the time you kind of only know if the answer has really helped you understand it post hoc, right? So like for example, um, let me, uh, yeah, like the Riemann hypothesis or something, it's like a famous conjecture in number theory, and like let's imagine we had an alien come and give, or whatever AI come and give us some kind of zero-knowledge proof of the Riemann hypothesis, okay? So this would give us like a yes or no, but like the Riemann hypothesis is also true for a reason, and like we want to know the reason. Um, so okay, so sometimes a proof will tell you the reason and sometimes it won't. So of course for the Riemann hypothesis, it's, we don't, we don't have a proof, so we don't know what, what a proof will tell us. Um, but yeah, it's actually kind of common to have some situation where you like think that knowing the answer is going to like definitely improve your situation, your understanding of a situation, and then you get the answer and you're like, 'Oh, well that didn't really help that much.' And then you have to find a new question or, you know, maybe find a different proof of the same question. Um, and that's like a lot of what we do is like, or at least a lot of what I do is like actually, you know, you know, you get the answer and then you're not done, like there's now a thousand more questions that you have to ask. And like, you know, even if an AI produces a proof, like okay, right now the expositions are, are very bad and you have to work really hard to understand the proof. Even if it produces a really beautiful proof that's perfectly exposited, you actually still have to work really hard to understand it. Like this, you know, it's maybe you guys have read math papers and it turns out that's like a hard thing to do even if you're a professional, um, even if they're like beautifully written, and then like you're just not done, right? Like so maybe your, your understanding has increased a small amount, um, and then okay, well then you keep, keep working to understand it. Maybe like let me give you an example. So I think the first interesting result, uh, in my view that, uh, the, that AI systems proved autonomously was this resolution to the unit distance, uh, conjecture. Um, so Erdős had some conjecture about like, you know, what's, you know, if you, if you arrange n points in the plane, what's the maximum number of pairs that you can arrange to have unit distance one apart from one another? And so he conjectured it was like n to the 1 plus o of 1 log over log log n or some, okay, some very, very slowly increasing function. And then the models slightly improved that. Um, so they, they found some sequence of constructions where you got n to the 1 plus epsilon for some very small value of epsilon. Uh, okay, we don't actually know what the truth is, right? Like even once we have this great new result which has been very generative and led to a lot of other kind of interesting constructions, uh, like we still actually what we have is an upper bound of n to the 4/3 and a lower bound of n to the, you know, 1.02 or something. Okay. So like here we very slightly increased our understanding, and now okay, we're almost back where we started, like there's just this huge gap between what we know and what we want to know.
Do, do you think like there is this, like you just, you mentioned like bounds, so like there is this like, I don't know, claim that like if you actually, if AI will be so good, so we can actually like make our bounds so tight, right? That like they will actually like be practical, uh, in real life. Uh, do you agree with that? Uh, I, I mean...
Okay, there's a couple things I kind of, uh, where, yeah, where I'm skeptical that we, we see the kind of improvements you're talking about. Okay. Like I, I definitely think AI systems will produce results that are useful. So like the, that's maybe the weak version of your question, and that I think absolutely will happen. Okay. So there's, um, two thing, two places where I think I, I kind of generally have disagreement with this view. So one is like, so Alan asked about miracles. Um, I, we might see some very impressive technology. I don't think we'll see like, I, I don't think there's good evidence we'll see like miracles. Like it's definitely not the case that like any question you ask the models will be able to answer. Um, okay. So there's two reasons I think that. So one is this, uh, fact I mentioned that like, you know, just basic questions can get arbitrarily hard. And this is like, that's not just like an article of faith. Like if you just try to, try to make sense of what hardness means, there's just a pretty easy counting argument that, that will tell you that like you should be able to find extremely hard questions very easily. There might be some reason like interesting questions are systematically easier, but empirically like that doesn't really seem to be the case. Um, okay. Then the other question is just like, let's observe how progress happens in mathematics but also outside. I think what's really happening most of the time is like we have some body of knowledge and then like people are slowly pushing that outward, and then every once in a while like opportunistically you kind of have some new result that lets you like, what, resolve some basic question or like find some new understanding. But it's not like, uh, you know, if you had, you know, if you asked von Neumann every question, you know, without him knowing anything, he would be able to answer it. Like he's still sort of using the existing body of knowledge and like maybe pushing it out epsilon in each direction. And okay, if we look at what the AI systems are doing now, they're really doing the same thing. So like even with the very, very best results they're producing, like you usually see that they're using very recent work of typically of human mathematicians. Uh, so we saw this with Navier-Stokes, uh, we saw this with, uh, arguably the unit distance problem. So there was this quite recent work of Mehta, Ramakrishnan being used. We saw this with like this production of like construction of a non-solvable group. So like we really see that AI systems at least right now are acting in this kind of same opportunistic way that like human mathematicians have acted. Uh, they're, they're sort of, you know, still just pushing the boundary a little bit. Um, and you know, so maybe we'll start pushing the boundary a little bit very quickly and then we'll see some rapid progress. That's not so clear to me. Uh, that's like kind of what my expectation is. But like I don't think we're gonna end up in some world where like you just ask any question which, you know, requires you to take, you know, a thousand brilliant leaps of insight and then the box and just get there. Like just math is some hugely high dimensional space. You just can't... I, I don't, you know, it's very hard, hard to find your way around a thousand dimensional or million dimensional space.
So, so, um, I, I have some questions that are hopefully going to go more towards the math side. And, and, um, so I, and again my math background is basically the minimum needed for a computer science degree, right? But I'm fascinated by, uh, intuitionism and mathematical constructivism, and basically people who don't regard the law of excluded middle as okay, right? Like because frankly my own personal experience is I always thought that, that it was obvious to people that, uh, the law of excluded middle did not, you know, match, you know, direct experience, but of course apparently this is a common thing in computer science departments and not math departments. Anyway, what I'm curious is how do you think, um, AI systems like today, uh, interact now with these mathematical epistemology debates? Um, I, I think today my understanding is that most mathematicians view intuitionism and constructivism as valuable but still kind of heterodox and not...
Maybe, maybe explain first, Alan, like what does this...
Well, well maybe Daniel should explain it because he'll probably do it more correctly.
Yeah, I could. Well, I, let me just say in terms of culture within the math community, there are people who worry about foundations and like basic questions of like, not like logic like you're asking. The vast, vast majority of mathematicians just don't worry about this stuff at all. Like in principle, uh, right, so maybe let me say like how math is supposed to work and how it actually works. So like in principle the story you hear of mathematics is that there's like some sequence of axioms. So most people think about like the Zermelo-Fraenkel set theory axioms, maybe plus the axiom of choice, and then when you prove something you're supposed to start at those axioms and apply some deduction rules until you get to the theorem you're proving. Okay. And then there's various sets of axioms, there's various like allowed deduction rules. So you mentioned the law of the excluded middle. That's, that's a rule you might accept or not. And if you don't accept it you might be able to prove fewer things. Okay. And then people like sometimes argue, you know, which set of axioms should we start with? Uh, you can, there's a, a cool, I'm not an expert in set theory, but like for, there's this really cool paper by Penelope Maddy called 'Believing the Axioms' where like she tries to explain like why she thinks some collection of axioms like is actually a reflection of truth in some way. Okay. In practice, this is not how mathematics works. Like if you ask the, the, the modal working mathematician list the axioms of Zermelo-Fraenkel set theory, they will not be able to do it. Okay. And, and to be fair I am such a person, I cannot list for you right now the axioms of Zermelo-Fraenkel set theory. Um, and in fact like most results you would want to prove are not really dependent on the axioms. Like for any sort of reasonable collection of axioms that people actually think about, you'll be proving the same results. And so, so my view of this is that like this sort of idea of like form, like kind of formal mathematics, uh, kind of starting from axioms and applying deduction rules, that's sort of a way of like trying to ground ourselves in truth. Like, like we're supposed to in principle be able to do this, but actually we're doing something that's like kind of closer to physics. So like I'm trying to understand like what is a shape, and I have some intuitive notion of what a, what is a shape, and like, you know, then one can ask questions about that intuitive notion, and I'm trying to like probe those questions in some way, and I need some way to stay, to like make sure I'm not bullshitting. So like I still try to prove my things are correct and like try to live in some world where I can, you know, persuade myself that those proofs in principle could be formalized in whatever set theory or some other foundations. But like this is just, it's just like a way of grounding ourselves in truth. It has nothing like the day-to-day impact of like which axioms we choose and which whatever, whether you're intuitionist or constructivist or, or, you know, whether you accept the law of excluded middle, like basically is irrelevant to most mathematical practice.
I, I do want to just quickly note that the, the type that I do care about is infinite set theory a lot in my personal reading, and I claim that those axioms are especially important there. But anyway...
No, not honestly, not really, yeah.
Mainly on things like, I guess the one thing would be on, um, Cantor's diagonalization argument, and if you look into the whole debates about like can math be formalized, Hilbert and Brouwer and all that, that's where they started complaining, and Cantor being a particularly controversial figure. That's what I'm thinking of when I cite all these. Again, you're...
There was definitely like a big debate about this in the early 20th century. Um, yeah, I think even then it was like probably not super relevant to like the modal mathematician who was like whatever trying to understand differential equations or something like that. But yeah, I mean, you know, there have been many foundational crises. Um, I think now this intersects a little bit with like existing mathematical practice and, and AI because, um, you know, uh, it's now in some cases practical to actually formalize one's arguments. So, um, you can write, take a, take your paper and ask, uh, an AI model to formalize it in, in, most people now are using Lean, which is some kind of, uh, it's called a proof assistant. So it's a way of like formally checking your proof is correct. And there's other, other, um, competing languages like Agda and Coq and so on. Um, but, uh, yeah, so, so these are not, uh, it's not literally the same thing. Uh, so for example, uh, there's no proof that Lean is sound, and in fact many people, there have been many errors found in it recently and patched. So, um, like you can ask, well, how much of a guarantee is a formalization to begin with? You know, I think it's like pretty good evidence but not, not a perfect guarantee of correctness. Um, but yeah, I mean, this is sort of interesting for practice now because we have some cases where the models have like produced some argument with an accompanying Lean formalization. Uh, so that's like very, very good evidence the argument is correct that we get before anyone actually understands it. So it's kind of an interesting sociological development.
So I want to talk a bit about like how you actually like evaluate, um, or check this proof, right? So if, if human, like in the past, right, like when humans actually like wrote this proof, so we, we, we assume that humans can go over them and check them, right? Um, sometimes, right? Like this was our assumption at least. Yeah. Um, but like today, right, like so tell me like now if the model like generate very long proof, do you have any way to, to check it?
Yeah. So okay, maybe let me just say what the pre-existing situation was. So it's very hard to check papers written by either human or AI. The primary evidence previously we had that a paper was correct was that a human wrote it, and then historically the author has been responsible for correctness, and this works like pretty well. So I would say like basically every paper that a human has ever written has lots of errors in it. Typically those errors are pretty minor, and like they tend to shake out in the wash, like in the sense that, um, you know, if a result actually gets used, it's, it's, it's checked and like sort of stress tested in various ways. Like a wrong result is like super useful because... Do you know about the principle of the explosion? So it, like a false statement implies everything. Um, so, uh, it's in some sense it's kind of, we expect false statements to get caught. Um, and I think, you know, I found lots of wrong papers in my, my career, like not, not my own papers but other people's papers. Um, this has been very, you know, it's very annoying, like, you know, you try to prove something and you like, oh, you try find some useful statement in literature and it turns out it's, it's false. Um, so yeah, so, so that was the situation until recently. Okay. So the models now can produce, uh, very long arguments. Um, sometimes they're right. Uh, they're not always right. I think they're relatively reliable now. So if the models produce a relatively short argument, so let's say, you know, 5 to 10 pages, and you ask them is it correct, it's like quite likely correct. Um, they're not perfect at this yet. Um, so I have seen even with the latest models, uh, kind of sort of significant errors. Um, the, the way those errors often arise is that they will, you know, the models will rely on a wrong statement in existing literature. That, that's kind of a typical, uh, typical kind of error. Um, right, so then, uh, you know, you can ask, uh, well, what if there's a Lean formalization, uh, or whatever, some, some kind of formalization. Um, so often the, I mean, I think those are much more reliable. There's a lot of, uh, uncertainty in general about how to understand them. So, so for example, um, it might be, I, I think a Lean argument or a formal argument is very good evidence that the statement as it has been formalized is true. Um, it is quite hard to tell sometimes that the formalization of a statement actually agrees with the informal statement it's supposed to, it's supposed to capture. Um, so for example, if someone were to write down a formal statement of the Hodge conjecture in Lean, it would probably be many thousands of lines at least, and it would be very hard to check that it's kind of actually faithfully formalizing the Hodge conjecture. And I'm not sure I would believe it if the model, uh, produced a formal statement of the Hodge conjecture. Like I'm pretty sure that that conjecture has not been formalized in Lean by, by human authors. And yeah, it may, you know, something... Well, I guess we'll get there when we get there. Um...
But, but at least like, like in, in my perspective, like the, the, the major, the big problem is like the intent. Like in the past, like we assume that like humans, again most of them, right, like most of the time, like they have, they are coming like with good intent, like they, they don't cheat, right? And they don't like try to... Yeah. Um, besides some of them, but, but like with models we don't have this actually. Like it can be... So I, I, I just like someone, and on Twitter I just mentioned like there are for example like, um, some models will find kernel, um, bugs in Lean, right? Lean is like a, a language, programming language, and so there are some bugs inside it, and you can, and models, right, they can try to check if they, how, you give them, I don't know, 24 hours, like over a week to, to, to prove something, and they try again and again and again, at the end they don't care, they will try to find, and like if there is a bug and they can cheat and, and prove this with a bug on the kernel, in the kernel, they will do it.
This is definitely right. So yeah, so first of all, yeah, the models will definitely try to cheat in informal proofs. Like if you, you ask them to prove something, they will often kind of like be at a key point if, if they... And you know, so this is, I should say this really only happens at the edge of capabilities. So if they can prove something, like they definitely will do it, like they, they, they won't lie just for fun. But yeah, so as you say, if you run them for 24 hours on goal mode or something, they'll, they may try to trick you at the end. So either in informal proof or in Lean, right? So they might, you know, try to exploit some kind of kernel bug that I think there are examples where maybe this has been seen, although it's kind of unusual, or they might try to misformalize a statement or something like this. Uh, so yeah, this happens, but it's not as bad as you might think. And the reason why is we can just like ask them, did you formalize this faithfully? And like usually they'll say no. I mean, like there, there's no, the models don't coordinate with each other to trick you. So, um...
But do, but do you think that this is something that like there are a lot of discussion about alignment these days, right? And safety, and like what will happen when the model will be too smart and like will cheat and hide their channel slot and all these things, right? Um, do you think it will happen if it will happen also, um, in math?
I mean, we definitely see examples of it, but yeah, all I'm saying like this, you know, the models definitely are, do not behave in the same kind of good faith that like we have human mathematicians do. Uh, but because they, you know, different instances of the model don't coordinate with each other, it's sort of not such a big deal, like as long as you actually, actually ask them to, to check their work. So yeah, this is actually like a real failure mode we see. Like we, there are people who are like trying to go through the whole literature and like just produce a paper on every conjecture, and like a lot, when someone does this, like a lot of those outputs are wrong, even though the models are really reliable, and it's just because they're like trying to elicit some output and like don't care, care if it's true. Like I think this is an example of human misalignment. Um, and like if you take those, those outputs and just put them back in the model and go, is this correct? Like usually the model will know they're not. So not always. The models don't always find their own errors, but it's, it's, um, you know, you can really do a lot to kind of reduce this kind of failure mode if you're just, you know, have, you know, even the tiniest amount of due diligence with not everyone using the model to do math.
So, so you don't think we need a new, uh, ways to, to review and check, um, paper?
We, we definitely do. Uh, I mean, but the main issue is like not correctness. Like I think that we, you know, for formalization, you know, it's not practical always, like, you know, this whatever formalization for Fermat's theorem maybe cost somewhere between 300,000 and a million dollars, and like I don't have that kind of money to burn on, on formalizing math, but like in a year probably it'll be much cheaper, and I think that will provide like really good evidence of correctness, like way, way better than we've had in the past, you know. So just comparing, you know, my, my guess is that like basically every paper written at least 2 years ago has lots of minor errors in it, and some of them have lots of major errors, and you know, that's something that's going to get a lot better. Where we need, we're going to have to think about how we do review is just like we also want to like learn something from the outputs, you know, besides binary yes or no, and like probably there's going to be in the next few years more papers produced, like more mathematical PDFs produced than have been produced in the entire history of like human mathematics, and so like right now our existing systems are definitely not robust to that. And so, you know, we want, we'll want people to engage with some portion of that, and figuring out which portion and how to get people to engage with it is like I think a pretty serious problem the, the profession has to, has to figure out.
So how, how big of a problem is there with slop in the sense of like repetitive overused phraseology possibly leading to diversity collapse in the kinds of problems that systems would work on, uh, the ways that they prove results, the, the results themselves, etc.? Like do you think that humans even well after AI systems are super, superhuman at math will retain value as diversity generators?
I think that's a possibility. Um, yeah. So I think we definitely see signs of this now. So there are multiple examples now of, of different teams putting out like proofs of the same result due to whatever GPT-5.6 or GPT-6 or Claude like more or less simultaneously. Like the, the models seem to be just doing the same things basically. So even like different models, so ChatGPT and Claude, like the collection of, of, um, results they can prove seems to be like basically the same. So I don't know if you remember...
Why? Do, do you have a guess why?
Um, it's a good question. Uh, and partly it's like a lot of the, you know, there's certain things they're very good at, like the, you know, the, the main thing the models are really, really good at is like taking a bunch of known techniques and then applying them in a technically strong way. And this is, uh, something where like I think a lot of conjectures are accessible this way. That's like a good portion of what human mathematicians do. Um, but you know, sometimes you need to develop a new technique. I'm not saying the models can't do that. Like I think there are examples of that too, but they're way, way weaker at that. And so, you know, the, the sort, the strengths the models have just like play to like just this having this super broad knowledge and like willingness to grind and not getting tired. And so the results they're proving are like the ones where those skills are really useful and like where the, where the skills they're kind of bad at are, are not as useful. Um, but, and also what I'm describing is like kind of a feature of human mathematics also, right? Like the models are being used by human mathematicians or people at the labs who might be mathematicians, and like so there's some collection of people who are like trolling through some collection of conjectures and not adding anything new and just kind of...
pressing the button and you know those those groups of people are proving the same conjectures because just they're doing the same thing and not adding any kind as you said cognitive diversity.
