Daniel Litt | Working with LLMs to do high quality math

Thanks so much for the invitation and for the introduction. It's really nice to be here. I went to college here, I guess, like 20 years ago now. And I'm really happy to be back and I hope that many of you stop by my office and come to talk about math. I'm interested in algebraic geometry and number theory and other things. I'm also happy to talk about AI, but I personally much prefer talking about math. So hopefully you'll hear me talk about that some other times, too. All right. So I want to start by explaining why this talk is going to be out of date in probably a couple weeks, just by kind of briefly reviewing a bit of the trajectory of AI and mathematics over the last few years. So I first started trying to get LLMs to do math in April 2022. So this is before GPT-3 was released to the public. OpenAI felt it might be too dangerous for public release. So there was a period when it was embargoed and the only way you could use it was through a video game called AI Dungeon, which was a Dungeons and Dragons game where the LLM would tell you a story you kind of interacted with. Did anyone here try AI Dungeon? A couple people. Yeah. So I mostly role played as a math professor, which I'm very familiar with, and would try to get the models to do math. And so here I tried to get GPT-3 through AI Dungeon to add 8 + 11. And there must have been something in the system prompt that told it not to do math because at the time the models were extremely bad at math. They couldn't reliably add two-digit numbers because very consistently if you tried to do this, you'd say, 'I typed 8 + 11 into my calculator,' it would say the calculator is broken and refuse to output a number. Okay, so in late 2022, GPT-3 was publicly released and I think this is my first time sort of successfully trying to get it to prove something. So I got... so here you can see the text that is not highlighted I typed, and then the text with a green highlight was typed by GPT-3. And so I got it to at least state the hypothesis of Fermat's little theorem, albeit with an extra hypothesis, and then with some hints I got it to produce what's almost a proof. Has anyone spotted the error in the proof? Okay, there is a minor, minor error, but it's close enough. You'd give this probably nine out of 10 if a student wrote this on an exam.

Okay. So here's where the models work. So I think the base case is okay. There's some issue. Okay, let me not get into it. Yeah, the models can now prove Fermat's little theorem. It's not the point. All right, so this is about where the models were in March 2023. So this is a problem from GSM8K, which was sort of a benchmark consisting of math problems you might give to elementary school students. So the nature of the problems is, you know, there's some word problem involving cookies or rabbits or something like that. Maybe if you unwind the word problem you have to take 12 * 4 - 7 or something like that. And the models could not do this until GPT-4, which essentially saturated it. So GPT-4 was not getting 100% on this benchmark of problems for elementary school students. It was getting like around 90%.

Okay, then December 2024 is when the first reasoning models were released to the public. I think they were available internally several months earlier, but o1, which I think was the first publicly available reasoning model, was released then, and that model did like reasonably well on the AIME, which is a benchmark modeled after the AIME contest for high school students. I know I took it in high school. It's like one of the, maybe it's like the second step on your way to qualifying to the IMO. And you can see these are like fairly tricky problems. And o1 again didn't quite get 100%, but it was getting around 75%. We don't know how much that cost, but I imagine it seems likely that it was in the hundreds of thousands of dollars at the time. And here the AIME was actually, I think this was the first time, my understanding from talking to people at the labs, this is like when the labs kind of realized that training on math was going to be a useful thing for them, and they kind of discovered that training on the AIME let them develop techniques that would generalize to other things that they might have thought were more economically valuable than pure mathematics, although, you know, some problems are worth a million dollars, so maybe some of them have economic value. Okay, so now July 2025. So that's a little more than a year ago, an internal model at OpenAI and also I think at DeepMind performed at a gold medal equivalent level on the IMO. So here's a problem from the 2025 IMO, which I don't know, I probably couldn't solve this problem in a reasonable amount of time. I imagine many of you could. January 2026 was I think the first time the model solved an Erdős problem. So probably all of you know this, but Erdős was a very prolific mathematician who proposed many interesting problems. Some maybe not so interesting, but some very interesting. Thomas Bloom collected those problems on a website called erdosproblems.com, and I think he definitely did not intend it, but it became a sort of locus for where AI companies and also just people playing around with models started trying to use it to solve problems. And here I think this is the first one which arguably received a novel solution. So before this there were a number where the models kind of like extracted something from the literature that immediately implied the desired statement, or you found a proof in the literature already existing that Thomas had missed. But this one, it was some kind of relatively short argument on the order of, I don't know, 10 lines. Okay, February 2026 was the first instance of the first proof benchmark where mathematicians, including some in this room, I think, took lemmas from their own papers and project progress and challenged the models to solve them. And again, an internal model at OpenAI with some maybe questionable amount of autonomy, and then also an internal scaffold at DeepMind, the Alpha scaffold developed by Tony Fung, got on the order of six or seven of these problems correct, whereas publicly available models got nothing too correct. Okay. May 2026 I think is the first interesting result due to an AI. So an internal model at OpenAI, which we now know to be Astra, resolved a unit distance conjecture. And so I actually, I think I was the first person outside OpenAI to see this proof. So what happened was the model told them, as I understand it, they were trying many conjectures on the model. The model told them it had a proof and then they couldn't check it was correct. So they needed someone who was kind of familiar with the theory of class field towers due to Golod-Shafarevich, and they asked me to check that part of the proof. And so that was, yeah, that was my connection to this story. And here's I guess some visualization due to Alberto Lozano-Robledo of the construction, although you can't really visualize the construction that the models produced because it's sort of some infinite tower and it only actually beats the record once you go to some situation where you have 10 to 100 or 10 to a thousand points.

All right. So then now we started in July, we started to see proofs of famous conjectures drop via tweet. August 1st, OpenAI released solutions to 10 pretty well-known substantial problems in several fields, and then yesterday they released a solution to the Navier-Stokes Millennium Prize problem. Okay. So I always start my talks with some kind of story like this because I think it's really important for us to think about the trajectory here. Like my expectation is that this trajectory will continue. There's no reason to think we're at the top. And so anything I say is going to be out of date very soon. I think the structure of the profession is going to have to be very dramatically rethought quite soon. I think there's a lot of discussion happening here, but I really encourage you to think about this. Like I think, you know, this is really exciting from the point of view of mathematics. I think we should be ecstatic for mathematics, but I think it also means that mathematical progress is becoming disconnected from mathematical expertise, and so if we care about maintaining human expertise, we have to think about valuing other things than papers. Okay. So I want to try to talk a little about that. You know, I think, you know, I've only started to come to this quite recently, so I'll still tell you about some theorems I proved. But, you know, probably that's something that will be, I think we might try to emphasize a little bit less in the future. And I really, it's not entirely clear to me what that will look like. I've been trying to think about it hard. But, yeah, I encourage you to think about it, too.

Okay. Okay. So maybe let me just give you a brief overview of my view of how LLMs are changing mathematics so far. So I think the biggest increase, the biggest change we're seeing is just like there's now a huge amount of attention available. Like the primary bottleneck I think to solving many of the problems we've seen LLMs solve is just that no one looked at them or maybe the right person didn't happen to look at them. We're seeing a lot of last mile results, so including arguably some quite substantial ones. So results where there were ideas and deep work done by mathematicians and then AI kind of took the last step. I think not all of the results have this flavor, and if you kind of think about the trajectory I saw, like even if you think some of these substantial results have a last mile flavor, you should not expect that to last long. I think we should expect extremely deep work to come from the models very soon. So there is some higher quality autonomous work already. And then there's a lot of talk about kind of methods of getting the models to do autonomous work. So ways of scaling up the amount of test time compute, auto research loops and trees. I'm not going to talk about those. So this talk, for the rest of the talk, I want to talk about non-autonomous work. So there's also a lot of talk about formalization, which I'll discuss a little bit at the end of the talk. But I want to talk mostly about how LLMs can be used non-autonomously to help produce high quality informal mathematics. And let me just say a little bit about what I mean by that. So I think Gowers pointed out to me that of course it's possible for the models to produce autonomously high quality mathematics, or at least mathematics that like, you know, clearly would have been very, very, very significant and very highly rewarded a few months ago or a year ago. So I think we need to think a little bit about what high quality means, and to me at least part of being high quality means that it produces human understanding. So a paper that no one has read, no matter how significant, cannot possibly be high quality. So that's why I kind of want to insist that at least some of this work has to be non-autonomous.

Oh, but yeah, so that's just in this talk. Yeah. So I'm not going to talk about formalization except very...

Yeah, of course. I don't mean like non-rigorous, although I also think that's very important to do non-rigorous mathematics. Okay. So my plan is just to explain a bunch of experiments trying to use LLMs in my own work and kind of think about some lessons from that. But maybe before I do that, are there any questions or complaints? I know that this is kind of a fractious topic. So if you want to complain, I encourage you, you can also do it after the talk. Yeah.

That's a great question. I mean, so okay, so I should say I'm writing an essay on this topic. This is a little off topic for the talk. So maybe I won't say anything except to say I'm happy to discuss after, and to say that I'm writing an essay, or I've written an essay which I will post next week, which is an attempt to answer that question. So if you will wait a week, I will have a maybe better thought out answer to you. But yeah, I think it's a very good question. Any other questions before I move on? So I'm going to start talking about math in a, or like my, no, I shouldn't say my math, math that I'm involved in, for the rest of the talk.

Okay. So let me first give you an example of what I consider low-quality autonomous work. But this is an experiment I did, and I think many people have tried it. So I asked Codex to spin off five sub-agents to each solve an open problem in algebraic geometry. And it did. So I should say, so of these five, this was a few months ago, so two of them were wrong, and then three of them with a little back and forth answered, you know, somewhat interesting open questions in algebraic geometry. So these would have been publishable papers a few months ago, and they wouldn't have been at the level where it was worth my time to write them up well, but so this is something you can do. You asked what should graduate students do to be marketable on the job market. They should not do this. So I think this is, if we don't kind of radically reform the profession, I think this is the winning strategy to play the slot machine for theorems. And you know, sometimes those theorems will be significant. I think that's going to destroy whatever human expertise we have in the long run, like that's just not how we can train high quality mathematicians. So I really encourage you to do this, not to do this. And I want to point out I didn't pick the problems, like here we're basically like, you know, trying to sweep low-hanging fruit, and like maybe we also want to try to achieve some deep understanding or something like this. And I think this is just not a reasonable way to achieve kind of deep understanding of anything interesting in mathematics.

Okay, so let me start with something that I think is a little bit more interesting. So this was a question of Serre. So he asked, does there exist a motive whose motivic Galois group is a simple group of type G2 or E8? And he called this question 'puhazardous,' a little bit dangerous, or more dangerous than the previous question he asked at least, which I did not include. Okay, so the question is, do there exist motives with Galois group of type G2, and then later authors have understood this as including the other exceptional simple groups F4, E6, E7, and E8. The reason this is a little bit hard is that if you try to make things out of curves, you can get exceptional groups, whatever a motive is, and you know, this is some way of associating to an algebraic variety a semi-simple algebraic group. If your algebraic variety is a curve, no matter what you do, you don't get one of these, and so you need to do something a little bit higher dimensional. And the theorem, okay, the theory of motives is partially conjectural, so one has to make sense of the question, but at least for some kind of reasonable operationalization of this question of Serre, the theorem is yes. So you can achieve each of these groups as motivic groups, and this is due to the work of many, many people whose names you can read there. So I don't know if any of them are in the room, but maybe I'll just briefly remark, so F4 is not quite in the literature, but you can extract it by combining work of Galkin, Lyubich, and O. And figure. Okay. So yeah, so you should think of this maybe as some version of the inverse Galois problem, which asks can you find every finite group as a Galois group over Q. This is asking can you find every exceptional algebraic group as a motivic group. And here's the classification of simple Lie algebras. Okay. So there was a version of this question that was still open that I kind of got curious about. I'll say a word about why soon, but it was can you make families of motives with exceptional G. So if you just stare at the previous constructions, this was known except for E6, and the construction of E6 due to, oh sorry, due to Boxer, Caligari, Emerson, 11, Metapust, Para, Petrius only gave some isolated sporadic examples of E6 motives. And so, okay, so the theorem I want to talk about, which had some AI involvement, was that the answer is also yes for E6, and it was a cool construction, which was really what I liked about it. It was kind of an excuse to play with cubics.

Okay. So just a fun, the reason I kind of got curious about this is that so I was invited to come here as the Benedict H. Gross visitor, and when I was in college, Benedict H. Gross asked me a question which was some version of this for E7, which is still open. And I kind of have, I'm going to try to solve it this semester. So E6 was supposed to be the warm-up. That was why I kind of got curious about this. Okay, so here's the theorem. It says that whatever these E6 motives are, you can make them using some kind of classical construction involving cubic three-folds. So for the experts in the room, these E6 motives appear in the theory of cyclic étale covers of the Fano surface of lines on a cubic. If you don't know what that means, don't worry. I just want to make some points to the process of generating this theorem. But I want to point out, so actually this theorem, I guess the paper came out in April, the work was done by maybe December or January, and this was like before the models were able to do algebraic geometry in a reasonable way. So we didn't actually use AI to prove this theorem. So what actually happened is we started using it after we proved the theorem.

Yeah, you can understand it that way. So yeah, so if you look at the...

That's right, okay. Sorry, I really love this example. I think it's some kind of very beautiful geometry. Maybe I can't resist saying, so there's some, the E6 sort of comes from the Weyl group E6 monodromy of the 27 lines on a cubic surface, and somehow related. Okay. Okay. So I want to talk about how AI ended up being used here. So when we wrote the paper, there were some lemmas that I really didn't like. So I'll talk mostly about this first one. So in order to prove this, we had to understand something about the cup product on the cohomology of the Fano surface of lines on a cubic three-fold. This is a map from a 55-dimensional vector space to a 51-dimensional vector space. And to prove the theorem I enunciated on the previous slide, we needed to show it had rank at least 50. So okay, if you have a 55 x 51 matrix and it's rank at least 50, it has rank 50 or 51. Okay, and we didn't know what the rank was. Okay, so this is all we needed to prove, but I was extremely annoyed that we didn't know the answer. Okay. And then well, we managed to show with some help from the models, which I want to explain how this happened, that the rank was actually exactly 50. So in fact this matrix never has full rank. Okay. So I just want to discuss this as a little case study because I think it was a fun example of a kind of genuine collaboration between humans and the models. So we wanted to prove, what we needed was this lower bound of 50. So we proved that just by hand. And okay, so as of 2026 no model could do this. With some hand-holding you could get the state-of-the-art models at the time to do it. And now of course they can do it with no trouble. And with the upper bound of 50, which is what we proved with LLM help, you couldn't elicit this result just by asking the models the question. So what we did is we actually computed a bunch of examples and we came up with some kind of explanation for why the rank might be 50. So we wrote down some elements that should be in the kernel, and well at the time, okay, so here's, sorry, here's the statement, there's some explicit section cutting out the divisor of lines of the second type which explains the kernel of this map. So that's lines whose normal bundle is non-generic if you know what that means. Okay, and at the time Gemini Deep Think was one of the state-of-the-art models for math, no longer true, given this version of the conjecture gave an essentially correct argument which was just a direct computation.

Okay, so the reason I wanted to make this case study is that now the models can prove the original statement I gave with no problem. So you ask ChatGPT, compute the rank of this map. It will produce the grossest possible argument, which is like a 10-page representation theory computation to show that the map has rank 50. It won't find the better statement. And I don't know how to elicit that from the models. So I mean, I'm not saying the models can't find the better statement or that in six months or one month or one week or for a billion dollars you couldn't get the models to do it. I don't know what question to ask to get them to do it. And crucially, like the word 'better' or 'more interesting' is like not something I know how to operationalize for the models. The models will have their own notions of what's an interesting question, what is a better formulation. Like you can see them as they work on a problem, they'll try different things and have some fairly human approach to find the right statement, but it's not the statement that you like, and I think it's often not the statement I like. And you know, even if the models have good judgment and good taste and ask good questions, which I think will happen, it's not your taste. So my view is that this is kind of an important lesson. Okay, more upbeat. So one of the sort of challenging parts of being an algebraic geometer is you think about shapes all the time, but you never get to see them. So I love algebraic varieties, but it's very unusual that I get to actually look at one if it's not like a Riemann surface. And now it's actually very easy to visualize things. So this is the Fano surface of the real points of the Fano surface of lines on a cubic three-fold, and I got Codex to produce this with like a couple hours of work. And I think it's, you know, a very nice benefit that we can actually now, you know, this probably would have taken me a month six months ago, and now it takes a couple hours, and it's, I know, just being able to kind of experience objects a little bit better is a very pleasant activity. If you go to this link you can rotate this around. The gray line is the divisor of lines of the second type. It's just very fun. Okay. So I'm going to move on to another project where it was useful. Before I do that, are there any questions?

So the target was 51-dimensional. So it turns out this map, we kind of guessed...

Yeah.

Yeah. So that was the point. The sort of better statement we found was interesting.

So that's the, yeah, it was really fun. It's quite easy. I mean, you know, I attributed the result to Deep Think. I think, you know, I could have done it, but...

Yes. So it's cubic three-folds. Yeah, so I do have analogous constructions for E7. I don't have E8 yet. The E7 constructions are pretty horrible. So I don't have a beautiful construction for E7 yet. And so actually that's one version of this question I'm trying to find, is find a beautiful construction for E7. Yeah, I'm very happy to discuss it. Yeah, there are some, it seems like there are some serious obstructions. Like, but yeah, maybe I won't get into it in AI. I'm sure I will talk about this result sometime in the next semester, but I'm also happy to discuss it. Any other questions?

This is actually from Paul.

So I checked. So there were, I produced this, this produced five arguments. Two of them were wrong. Three of them with some back and forth were correct, and yeah, I emailed the authors for the corresponding papers. It was not something where, you know, these were kind of throwaway conjectures that no one had really spent much time in. I mean, I think probably all of us have now had this experience of, you know, answering one of these throwaway questions, getting, getting just asking. Okay. So I want to talk, oh, any other questions before I talk about another project?

Yeah. So this is of course why I didn't, you know, I think these things, there needs to be another place to post them. I think GitHub is a reasonable option, but I mean, I also think it's sort of hard for me to imagine how our existing system of disseminating papers and PDFs can survive. I think it just, we need to, yeah, save the arXiv is something I would love to do, but I'm not optimistic about it.

Okay, I want to talk about another happier thing than the fate of the arXiv. So this is a fun question that was, I think, actually a relatively early example of using computers to do math. Relatively early meaning like 1999, which I guess is not that early, but early for me. I was 11. So there's a space that has been studied for a long time going back to work of Marone in the late 19th century, which is the character variety of the projective line minus a bunch of points. So in down-to-earth language, this means you take a bunch of conjugacy classes in some group, think like SL(n,R) or SU(p,q), and you look at tuples of matrices whose product is the identity, where the i-th matrix is in the i-th conjugacy class up to conjugation. Okay, so these are representations of a free group where you impose some conditions on the conjugacy classes of the generators and their product line. Okay, so this has been, I think, most studied in the case of four 2x2 matrices in SL2R. The group G is SL2R, and well, depending on what the conjugacy classes are, when you graph that variety, it's some real surface, it looks like this. Okay, and I think the first time anyone graphed it, even though it had been studied since, I don't know, the 1870s, was in 1999. Benoit and Goldman made this picture using a 3D graphing calculator, which is maybe one of the precursors to AI. And they noticed this bean. Okay, and I think they were a little surprised. The reason why is that SL2R is a non-compact group. So there was no reason to expect a compact component in this variety. So they kind of asked, why is it there? Okay, and so there's some more general question. You can take a, this is what you get if you take representations of the fundamental group of a surface with n punctures, a sphere with n punctures, and you fix the conjugacy classes of the loops around each puncture. You can ask this for a general surface with n punctures. You look at representations of its fundamental group where you send the loops around each puncture to a fixed conjugacy class in some real group. And some question which has been studied a lot in, well, at least is related to questions that have been studied a lot in dynamics, is what are the compact components of this? So there's been a lot of work studying like the action of mapping class groups on these compact components, but even the compact components themselves are not well understood.

Okay. And so there was beautiful work by Daron and Fathi which kind of explained all these compact components for SL2R. And then I asked my student Charlie Wu to kind of study these compact components, and I gave him, I suggested the following thesis problem, which is like, have we already found all of them? So there was this kind of beautiful construction by Daron, Fathi, and Tolman, which in his first paper he kind of put on some very uniform footing, and then I asked like, was that everything? Okay. And then the dream of every thesis advisor happened, and he came back to me two weeks later with a counterexample. So he found some other compact components, a new construction, and his construction kind of made it clear what all of these compact components, like what the right classification is. And so, okay, and one nice consequence of this is that he showed there's never any compact components if you look at character varieties of surfaces with higher genus. So this is really a property of the genus zero situation. I think I have greater than one here. I think there's now, he's improved the theorem, so it's greater than or equal to. Okay, so here's a, sorry, this is useless, but this is a picture of a compact component he made. Just looks like a bean. I mean, that's what they all look like. Yeah. So if they're surfaces, the compact components are always spheres, it turns out. So yeah, so he kind of did some classification in terms of objects called variations of Hodge structure. And these are really cool things that we have had a lot of fun thinking about, but at the time he discovered them, which was like fall 2025, like it was really hard to find them. So you had to do some very involved kind of guess and check, and each step of the guess and check was some very long homology computation. This was kind of an artisanal process. So I think each new example would maybe take him a week, and if I was less lazy it would probably take me a week too. Okay. So here's the definition. We had to find some variations of Hodge structure which have this cohomological property. If you, I'm going quickly through this because I don't, you know, this isn't a talk on algebraic geometry. But okay, so the question I asked myself in winter 2025, which is I think when everyone discovered vibe coding, was like, can you actually find these on a computer? Okay. So, sorry, I should have updated this slide. 12 months ago, I would never have tried this. Six months ago, I did try this. And yeah, so I've coded some tools to do scientific computation with parabolic Higgs bundles, and we found lots of examples. Okay. And the code is low quality. I've tried to improve it with existing models. It's still low quality. Probably I will try to do it from scratch, but it's, this is, yeah, I...

I mean, it's actually quite challenging to do this. Like, I really encourage you to — like, scientific computing is now something you can really do. I think, like, there are a lot of fields where none of us touch a computer, or it's very rare for us to touch a computer, and like, I think that many more of us should. It's just become much easier. So like, you can actually compute with objects that I would not have thought you could talk to a computer about. The downside is that it's pretty hard to tell if what your code is doing is correct.

Um, so, okay, probably for those of you who are serious coders, this is obvious, but like, I had to learn about unit tests. So I worked out a lot of examples myself and then made sure that the code that was produced was getting the same answers on those examples. Like, you know, there are ways of checking if code is likely to be correct without actually reading it, and that was a new skill I had to develop. But yeah, mostly what I want to stress here is that this is like working out examples, or thousands of examples, or millions of examples, is now a possibility in many areas where I think it probably would not have been worth our time a year ago.

Are there questions about this?

Uh, I mean, yeah, that's a good question. So, well, first of all, it's just very — my experience was that it was very hard to figure out the organization. Like, the models like to produce, you know, hundreds of Python files. To actually do a computation, you type in some command and then it has 23 flags. It's not a problem for a model to remember 23 flags in each of the things. Yeah, so it was just not really designed to be human friendly. Of course, I could have given better specifications and tried to get something, but, you know, there's some trade-off between getting something quickly and getting something usable.

So the other reason is that, you know, it's low quality because I didn't read it. So it's low quality. You know, my understanding of the code is low quality is another way of what I'm trying to say here. So I think this is kind of a trade-off that I don't really know how to think about well yet. And, you know, it's the same trade-off we're facing in a lot of our work with LLMs, is that there's a — you know, you can now to some extent trade your understanding for a better result, or faster results at least.

Um, okay. Um, maybe I will skip this one because we're running a little low on time. Um, okay. Well, I'll just say something. Yeah. No, I'm going to skip this one, I think. Um, okay. Sorry. There's some cool pictures. Okay.

So I want to talk about just some other activities that I think are things that we might find valuable aside from like trying to prove theorems in this — you know, now that we have our computer friends, or friendmies, maybe. So I think one thing that I've gotten very interested in recently is math that is easy to check. So, you know, it's now I guess possible to formalize arguments for hundreds of thousands or possibly millions of dollars in some cases, and then we can trust the results if you trust the formalizations of statements. If you don't have that money to spend — or okay, I think if we wait a year, probably formalization will be much cheaper, and so this is maybe a short-term problem.

You might ask, well, like, how can we check if a result we get from a computer is correct and worth looking at before actually reading the proof? Right? So I don't know, I don't know about you guys, my experience over the last year has been that I've been asked to read many, many LLM-generated proofs, and it's very hard to know in advance whether that's worth doing. So here's a project I was involved in, which is the Frontier Math open problems benchmark, which was a collection of problems which were designed to have the property that it's easy to get evidence that an answer is correct. So not necessarily a formal check, but evidence.

So maybe let me just mention my favorite example of this is — well, one of the ones I proposed to the benchmark was finding a polynomial whose Galois group is M23, which was recently solved, not autonomously by AI. My understanding is it's been a collaboration between human and AI, and some beautiful work by, among others, people in this room.

Another one that I quite like is — so there's the Hadamard conjecture. So a Hadamard matrix is a square matrix whose entries are ones and minus ones so that any two distinct columns are orthogonal to each other. An example is if you take the character table of Z mod 2 to the n, that gives you a 2 to the n by 2 to the n Hadamard matrix. It's known that if you have an n by n Hadamard matrix with n bigger than two, then n is divisible by four, and the conjecture is that for every n divisible by four there exists a Hadamard matrix, and this is open.

Well, one way you could try to get evidence for this is by asking the models to write a computer program that rapidly produces a 4n by 4n Hadamard or 4k by 4k Hadamard matrix. So I think the way this was operationalized in the benchmark was to find the five missing cases we knew where n was less than 2,000, and that was done recently by an internal model at Anthropic. We still do not have a proof of the Hadamard conjecture. So I think this problem in some sense remains open, like it's still quite easy to verify if a, I don't know, 2500 by 2500 matrix is indeed a Hadamard matrix. And so, you know, I think this is a good way to get evidence if one is making progress on the conjecture.

Okay. Another thing that I think has increased importance is problem lists. So we saw earlier that ishros.com attracted a lot of effort. My view is that there's now quite a lot of resources pouring into AI for math. Much of it is pouring into stuff that is not particularly interesting. Much of it is interesting, although too. I would personally like it to produce things of scientific value. And so let me — I want to explain why I encourage you to produce problem lists. So first of all, you might be able to convince someone to spend a million dollars to solve a problem whose answer you want to know.

But second of all, I think by producing a list you kind of commit yourself to actually reading and understanding the solution. And I worry that a lot of these problems are going to be solved and, you know, someone will put a 160-page PDF on their, you know, their company's web page and then no one will read it. And I think it's nice for you to say, like, I actually care about this problem. I commit myself to understand the problem in advance. I think that's like a healthy thing for the community. So, okay, I have problemslike.com as opposed to problems Erdős likes. Although I had a huge amount of help from Thomas Bloom setting up this website. I think so far there are 15 problems on it. I was on vacation for a while and didn't add any, but my plan is to add one or two each week. I think two have been solved so far, with answers sort of implicit in the literature, but interesting enough that I kind of was excited to see them solved. And then, well, this one I think is very hard. This one, if you solve this one, I will take you out to a very nice dinner. Okay.

I expect the answer to this one is yes. Yeah. So this is some version of the p-curvature conjecture for maybe nonlinear differential equations.

Yeah, I'll talk a lot — I'm sure I will give several talks related to the p-curvature conjecture this semester. That's my jam. Okay. So I want to talk a little bit about using LLMs for exposition. So I think there's a general consensus that it is not a good idea to have an LLM write your paper, which I agree with. LLMs are right now much worse at writing papers than humans. I expect that won't last long, but that's the case now. But I nonetheless think they can be used productively. So I want to give a couple examples.

So first of all, you can just ask the models, give me line-by-line comments on your paper. I think this really, really improves exposition. I think — well, I'll discuss some numbers in a bit — but every paper any of us has ever written has had lots of errors in it, and the models are now very good at catching those errors, and I think we owe it to our readers to use them to do so. So here, this was kind of fun. The models take you pretty seriously. So here the models gave me comments on every line when I said line. So line 533 was fine. Yeah. Okay.

So I decided to be the change I want to see in the world and run all my papers through — well, I used a scaffold called Refine to review my papers, which is just a way of scaling up test-time compute. So, you know, you can of course ask Codex or ChatGPT, read my paper and give me comments. It will do an okay job, but you can make it a bit better by spending a bit more. And Refine generously donated some reviews to me. So I had to review all my papers. And my co-authors, like Aaron, were very gracious in letting me post the information. I think it was a little unexpected to get this onslaught of slop from me.

But yeah, so I got about 400 comments on my papers. About 98% of them seem to be correct. So I checked that by having an LLM review them and then randomly checking a large portion to see that I agreed with what the LLM was checking. Most of them were quite minor, so you should think like typos, but I agreed with 98% of the ones I checked. And there were four theorems — sorry, five theorems with missing hypotheses across these papers. Four of them in one paper, which was my thesis. But nothing too serious, luckily. You should think like I forgot to say a variety was integral or something like that. You know, well, I think the worst was I wrote the degree of a field extension instead of the separable degree. So you have to fix these things. And then there were some definitely more wrong lemmas than that, but the proofs could be salvaged.

So yeah, so they're now also largely LLM-generated also, but again I have checked the serious comments. Okay, I want to talk about briefly auto-formalization. So as we now know, anything can be — I think if you can formalize the proof, you can formalize anything. So it seems to me that any correct result in the literature can probably now be formalized if you have a million dollars to spend. Yeah.

Oh, they're on my website. Sorry, for the papers where my co-authors have given me permission to post them, they are on my website.

My plan is to update the papers eventually, but yeah, I figured I would just post them quickly and then try to get to the — yeah, yeah. But yeah, eventually I hope to update the arXiv ones too. It's a lot of work.

Other questions about this? Yeah.

I think that if they did the complex structure on S6, yeah, which included formalizing some version of the Poincaré conjecture in dimension six, I think it's — they did the Navier-Stokes. I think if you can, I mean —

I would be very interested to —

Uh, yeah, well, you know, it seems like the models get better as you pay more. So maybe the question is not can you do it, but how much would it cost?

Yeah. Well, yeah, well, maybe let's discuss afterwards. My expectation is that if not now, then quite soon.