It's actually like unbelievably simple. Like it's kind of crazy that people didn't find it. So you take linear... so here's the map. You take linear polynomials and quadratic polynomials. So that's, you know, a one-dimensional space times a two-dimensional space and you multiply them. So that's a map from a three-dimensional space to a three-dimensional space.
And then, okay, so inside of that, you delete some stuff from those spaces and then that's the example. So it basically is just the multiplication map and you have to delete some things, for example, to make the derivative injective. So you delete the locus where it's not injective. And that's it. So really extremely simple.
Welcome to Begin Proof. I'm Slater, partner at Bain Capital Ventures, and today I'm talking with Daniel Litt. Daniel is a mathematician at the University of Toronto where he works in arithmetic geometry, and today we'll be talking about AI-assisted mathematics from the point of view of a mathematician who's working at the research frontier. Well, Daniel, it's been sort of a big news week in math.
So for the audience, or however I'll say this, we're recording this after the rest of the podcast that Daniel and I did before, you know, because of the update of the Jacobian conjecture falling earlier this week. So Daniel, I'm really excited to get your reaction to that. You know, I thought one fun way of opening this would be kind of like, you know, let's put the veil of ignorance back on and say, you know, sort of before knowing that this counterexample existed, you know, what were sort of both sides of the intuition? Like why would you have intuition that this might be true on one hand, like what are sort of the reasons why something like this might be the case? And then, you know, also maybe why you might have some intuitions that it's false. Like what were, you know, some reasons why it wasn't sort of like totally clear?
Yeah, that's a good question. So I think, first of all, I tried to... so this is in my area. It's like a conjecture in algebraic geometry. I definitely knew about the conjecture. I would say it's not like my style. I maybe spent a few minutes thinking about it and didn't get anywhere over the course of my career. There's been a lot of work on it, so it's definitely not an attention bottleneck conjecture. My sense is that there was no consensus about its truth value.
The main reason for believing it to be true, I think, is probably just that no one had found a counterexample over a long time, so it's kind of anthropological. There's better evidence in the case of polynomials in two variables, so there it's known up to degree about 100-something, and the new counterexample doesn't actually disprove that case. So you can actually, if you go back, if you just search the literature, which I did after the conjecture was resolved, like you can find a lot of people saying there's very strong evidence in dimension two, who knows what happens in dimension three and higher, and indeed it turns out there's a counterexample in dimension three and higher.
So I think the reason to believe it's true was basically just no one had found a counterexample. The reason to believe it's false is like there's just no reason for it to be true. It's sort of one of these things which I think is like sort of an isolated conjecture in algebraic geometry, like it's not part of a big program. That, you know, sometimes you can get evidence for a conjecture, like the Riemann hypothesis or something, like I'm very confident it's true because it fits into this like really complicated and very self-consistent picture that makes a lot of predictions and many of those predictions have been verified. The Jacobian conjecture, it actually was known to be like equivalent to or like follow from a bunch of other conjectures in different areas. So another reason it's nice to have it resolved is like that resolves a bunch of those conjectures, but it's sort of, yeah, I don't know, it doesn't fit into a program. So I think it was maybe post hoc, it was kind of hard to get evidence for it or against it.
Yeah, that's the kind of thing I was wondering about. So, for example, I had a friend who worked on like discrete harmonic analysis, and you know, there's sort of all of these known results in sort of normal continuous harmonic analysis, and like that would give you some intuitions that like the discrete versions like maybe should be true, and you know, but also there's unique features of the discrete case, and like that would also give you some intuitions about like why some things should be false. You know, of course, you actually have to come up with an argument.
That's actually a really good point. So in fact, there are versions of the Jacobian conjecture that were known to be false. And my understanding, and this is not something I'm really an expert in, but my understanding is that the counterexample Fable produced is like kind of closely related to a counterexample someone else, Fushkin, I think, had produced to a variant of the conjecture.
We haven't seen the, you know, the chain of thought or the chat session or whatever, so we don't really know whether it relied on those ideas. But yeah, I think you could try to get evidence that it was false by looking at these sort of related conjectures that do appear to be false or are known to be false.
What is your intuition for the proof? So, as you already alluded to, the original tweet was sort of, you know, very playfully cryptic. You know, you could verify these polynomials, you know, sort of have the desired property, but then, you know, you don't sort of have intuition for that. And I think a couple of explanations have come out now. You know, Terry Tao had this kind of analysis-flavored one. Somebody else on Twitter, I think, I think it was Andy Jiang.
Yeah, gave the polynomials to ChatGPT and like found a pretty nice geometric description.
Yeah, what's your intuition for it? Like now, as you understand it now, like what sort of is your intuition?
It's actually like unbelievably simple. Like it's kind of crazy that people didn't find it. So you take linear... so here's the map. You take linear polynomials and quadratic polynomials. So that's, you know, a one-dimensional space times a two-dimensional space and you multiply them. So that's a map from a three-dimensional space to a three-dimensional space. So, and then, okay, so inside of that, you delete some stuff from those spaces and then that's the example. So it basically is just the multiplication map and you have to delete some things, for example, to make the derivative injective. So you delete the locus where it's not injective. And that's it. So really extremely simple.
Why it works, I mean, I don't know. It works because you check it works. So it's not totally clear like how you would come up with it if you like decided, well, I'm going to sit down and disprove the Jacobian conjecture. I think that's still a bit of a mystery to me. But I don't know, maybe you just try a lot of things. It's kind of crazy to me. Like this is a map that has like probably, you know, a million pages written about it. It's definitely like a very well-known object in math, in algebraic geometry. We like did not discover it because of this proof. It's kind of crazy to me that like no one noticed that there's a counterexample to the Jacobian conjecture sitting right there. But yeah, so I would say post hoc there's a very, like, quite beautiful and simple geometric explanation for why it gives a counterexample to the Jacobian conjecture. Somehow the only not totally obvious thing from what I said, like once you know what to delete, is like why that this... so there's a source and target of the map, they're both isomorphic to just space C3. It's not sort of totally obvious why the source is isomorphic to C3. It's sort of an exotic isomorphism, but it's like if someone tells you what I just said, like they're going to finish the, you know, you're going to finish the proof in like a few minutes probably.
Yeah, I wanted to ask about that specifically. So, you know, there's this style of blog post that's sort of like you could have invented X, like Goran has this interesting thing of like you could have invented Transformers, and it's sort of a just-so story where if you had the right ideas at the right time, you know, you could sort of follow these logical steps. And I was reading, you know, Andy's argument, and I'm like, okay, I can imagine considering this map, that makes sense. And then I can imagine saying, yeah, I've sort of got to delete the places where it's singular. You know, that's okay. You know, and then the sort of thing is, okay, so, you know, I'm...
you delete a hyperplane tangent to but not oscillating into the small diagonal.
Yeah, you sort of like choose the hyperplane, you know, sort of carefully, and like I guess once you've had that intuition, it's clear that like the target space is going to be A3, or like that's not that crazy, but it's sort of the thing that I have like zero intuition for is like you can choose it in a careful way where the preimage is A3, and like maybe to me that's the part where I have like the least intuition for why this should work.
Yeah, that's also the part like I don't know how to tell a just-so story there. I think so, one reason, one way you might have come up with it is look at this prior counterexample and like try to massage it in some way. Like I wouldn't be surprised if that's what happened. Yeah, I don't have a just-so story there. I mean, I think that it also could just be like you fiddle around with that map, and you know, it's a map whose target is P3, so it's very easy to arrange that by blowing up something you have on space, and then you just fiddle around until you win. I mean, one nice thing about the models is they have a lot of time, they have a lot of energy to just try stuff. So yeah, I don't know. It would be good to, it would be interesting to know like, you know, what number thing this was that Fable tried, was like the thousandth thing to try or the third thing.
Yeah, or even kind of like the edit path of like, you know, you try out a few things, but what are you sort of trying to fix about, you know, sort of the early trials to get to this one? Why do...
Yeah, I would love a story like that. I don't...
Yeah, I'd be really interested in that too. Why do you think that this was missed for so long? You know, I think that this proof would have been totally... I think this problem was posed in 1844. There's no modern machinery to this argument. You could have, people would have understood the proof if it was presented next week. You know, why do you think it took so long to find this counterexample?
Yeah, it's really a mystery. I mean, so I think the history: so in 1844 someone asked about the two-variable version, and then in 1939 it was asked about by maybe in all dimensions, and then the sort of general conjecture was popularized by Oehme, but I think again mostly about the two-variable version. So I think one possibility is that just most work was happening in the case of two variables.
I think another possibility is that the people who were motivated to work on the conjecture believed it was true, even if there wasn't sort of a broad consensus in the community it was true. But I don't know, I think it's just, you know, my feeling actually having seen this is that probably there are a lot of conjectures out there which just have simple counterexamples or proofs, and we just miss them because we don't do exhaustive search. Like a lot of the way mathematicians work is not like you pick some conjecture and then you like just work on that and work on that and work on that. Sometimes that happens, like you get obsessed with a problem and you just try to go deep on it. But a lot of the time it sort of goes the other way. You have an idea and then you apply the idea. It's sort of if you work opportunistically, and so I'm a little surprised that like opportunistically someone didn't find like literally this counterexample. But you know, there's a lot of randomness to like what ideas people have and how they kind of discover that they can apply them. So it could just be that we're very bad at like, you know, we search some very, like some Brownian motion in a very high-dimensional space, and that means we miss a lot of stuff. My expectation actually is the models are also doing some, you know, Brownian motion in a very high-dimensional space of proofs, and they're also missing a lot of stuff. So even probably very simple stuff.
So okay, I think it's interesting that this example and the prior one that we talked about, the Erdős distance problem, you know, sort of the models have succeeded by finding counterexamples, if you will, or like constructions, you know, that humans didn't. And of course the models are going to get better in all sorts of ways. We shouldn't expect them to sort of stop here, but you know, maybe at this point in time they're sort of like counterexample-generating machines. They're extremely good at this. And my question for you is, does that change the way that you personally do research, where it's like, hey, I think this thing might be true, let's just, you know, sort of let a frontier model crank for a while to, you know, exhaustively look for counterexamples? I'm curious if, you know, the kind of counterexample-generating machine, you know, sort of as a distinct thing, if that's changed your sort of research practice at all.
Yeah, for sure. Yeah, I mean, so definitely like literally anytime I have, you know, of course when you try to prove something, you come up with some intermediate sequence of conjectures and you try to verify them, like you try to find a path and then, you know, move forward on the path. And like definitely when I'm doing this now, like if one of these conjectures seems like it's something that like you could imagine writing down an explicit counterexample, I of course ask the models and of course try to find a counterexample myself. I also actually have like picked up a bunch of projects now where this kind of like having this ability is useful, right? So like sometimes you would like to make a construction, and like the way to do that is like you try a thousand things, and you know, previously I would have had to try a thousand things serially, one after another, and now you can try a thousand things in parallel. Like the models actually are still, you know, in the precise area of mathematics I've worked in, but like they're not so reliable like working out examples like this, even the best models, although it's getting way, way better very quickly. But yeah, you know, even an 80% success rate, if you try a thousand things, it's actually kind of, you know, you get information. So yeah, I think, you know, now there are kind of...
there's a style of research which I think has not existed until now, where like the nature of this trial is you like come up with a thousand ideas for how to make something, and then you ask the models to pursue those ideas. And yeah, so that's something I've been like trying to do like as an experiment to get the models to, you know, do something that I find more interesting.
Any other belief updates from this, where, you know, sort of like your predictions or timelines or really like anything that you're betting on about sort of, you know, AI-assisted math has changed, you know, sort of on the other side of this?
I think that I haven't really updated on capabilities. What I have, it's definitely updated me on how much we're missing. Like I think that the big open question for me about the near-term impact of AI on math has just been like how much low-hanging fruit is there? Like how much can you do without like introducing a big new idea? I'm not saying that the models like will not introduce big new ideas eventually, but they're not doing so now. Like right now what they're doing is like, you know, using known techniques effectively. And my sense had been like probably like attention bottleneck questions were not in general accessible to like known techniques with no big new idea. And I think that's just like wrong. Like I think the Jacobian conjecture seems like very strong evidence that actually there's like a lot of known, well-known questions where like maybe even you don't need like 20th century mathematics to resolve them, like maybe it's fine, you know, maybe there's a lot of interesting open questions that you can, you know, find a counterexample to using like ideas from the 19th century. So yeah, so I think that tells me that I think in retrospect that like should have maybe been more obvious, right? There's just like not that many mathematicians and there's like a lot of problems. But yeah, that's been the big update for me.
I'm not asking you to give any scoops on your current research or, you know, things that you're actively working on. So maybe ruling that aside, are there any, you know, sort of problems where you think, oh man, this is the sort of thing that you ought to take this approach for, where, you know, somebody who knows what they're doing should be rooting around for counterexamples now, or even just like, you know, sort of areas if you don't want to name specific problems.
Yeah, that's a good question. Of course I've been thinking about this. I mean, I should say I think the areas I tend to work in, I mean, of course I could be wrong about this, but like I tend to work like adjacent to some big program where we like have in some sense a lot of evidence the conjecture is true. So yeah, the sort of things that I have been trying to get the models to do is, I guess, make positive constructions, not just counterexamples. So I don't know, I can give an example. You said I'm happy to say one thing I'm working on, which is there's a sort of mystery in the theory of Shimura varieties, which is some algebraic geometric object. They carry what are called topological local systems, and they're predicted to be motivic. And okay, there's some class of Shimura varieties called exceptional Shimura varieties where we don't know this is true. So I've been trying to, you know, see if I can prove this. This is definitely something like where I do not expect a model to be able to one-shot an answer. That would be like remarkable. But you know, they're very helpful. So you know, you try to make some construction and then test it has a bunch of properties, and that's something that Codex is now able to do. Like, you know, you make some algebraic construction, you ask it to do a computation about it, and it will succeed. So yeah, that's something that I've been trying to get the models to do.
There's another, I actually find a bit more exciting than the Jacobian conjecture. So there's, Akhil Mathew got, I think, Fable to find a counterexample to an old conjecture of Grothendieck about finite flat group schemes. The conjecture was that a finite flat group scheme of order n is annihilated by n. So this is like analogous to the fact that if you have a group of order n, every element has order dividing n. And so there was now a counterexample to that old question. This is something again definitely a lot of people worked on. I think the broad belief was that it was true. So it's maybe the negative direction might have been attention bottlenecked. And there it seems like there's maybe actually quite a bit to learn about finite flat group schemes from the counterexample. And yeah, I think there's like a, my expectation is that there's like a bunch of old conjectures in commutative algebra that probably will find counterexamples soon too of this nature, where just like the general belief was wrong. So yeah, my instinct is to look in places where, you know, the field is maybe a bit more technical. I mean, of course you see a lot of now counterexamples on social media, random conjectures in graph theory that no one has ever thought about before. But yeah, I think there's probably, you know, in a technical area there's a lot of sort of maybe even guiding conjectures, but you know, if there's only 10 people in the area, the conjecture can be kind of important in some way, but like definitely, you know, the counterexample space will not have been exhaustively searched.
There's one last thing I wanted to ask you about, which is I'm blanking on who it was right now. I was trying to look it up, but, you know, someone was making the argument that maybe the Collatz conjecture, you know, we'll get a counterexample to, and they sort of had...
I think Alex was talking about...
Yeah, you know, recognizing it's outside of your area, what do you think about that?
I don't know, I think that's, yeah, I have no strong opinion. I mean, of course, that's a case where finding a counterexample would really probably require some quite serious new ideas. So it might be true, but I would be a bit surprised to see a model generate a counterexample to a conjecture like that, you know, let's say this year. You know, who knows what will happen next year. Yeah, I mean, I think, but again, I could be wrong. Of course, the premise is that the Collatz conjecture is like well-studied enough that like the sort of known techniques have been exhausted. It's not totally clear that's true. But maybe, yeah, my instinct is probably true in the way that it might not have been true for the Jacobian.
Yeah, I found his argument kind of interesting. It's like, you know, sort of running it through, we should, you know, sort of think as like some type of Turing computation, and you know, yeah, it's sort of an interesting idea I hadn't seen before, but we will see.
Yeah, and one related thing is there's a bunch of conjectures that are known to be undecidable because sort of the process that is being studied is very complete. So sort of deciding the conjecture is like self-referential.
And it's also possible that that's true for Collatz. I think that's some mathematicians believe. I'm pretty agnostic about Collatz. I do think it is possible. One thing about the Collatz conjecture is that it is definitely something where I think most people probably believe it's true. And maybe that's another place one should be looking for counterexamples. Like I don't know, my sense post-Jacobian conjecture is that like there's some weird anthropological effect where the people working on a problem are like the one, you know, like why do you work on a problem? You get obsessed with it and like you have to be excited about it. And it's way more exciting to prove something is true than proving it's false. I don't know. And so maybe like we systematically are missing counterexamples just because the people who work on a problem like trying to prove it, like they're excited to prove it, not to disprove it.
Yeah, yeah, yeah. I mean, it feels in some ways, some counterexamples, you know, you do a lot of theory developing to create, but you know, in some cases, and actually maybe actually the unit distance one is an interesting example of that, where like I don't know how you would do...
Yeah, certainly more like that than...
But sometimes it just feels like, I mean, there's that famous paper where it's like, you know, a theorem of, I forget who it was, but it's like there was this theorem, here's a counterexample, it's like a one-line, you know, paper or something like that.
It might maybe the Euler sum of powers conjecture. Yeah. So there's like a... Yeah. This was sort of very early on in the impact of computers on mathematics, like someone just like checked up to, I don't know, the number 100 or something, and they have a one-page paper where they have a, you know, some sum of powers of three-digit numbers is some other... I mean, it's an amazing thing. I think it's, you know, we're sort of seeing this happen again now, like we now have, you know, a new technology like the computer, and it can now knock out a bunch of these conjectures.
Yeah, I vaguely remember something like this with Mersenne too, where it was like he had some conjecture, and like literally the next one after he stopped was like the counterexample.
Amazing. Well, Daniel, thanks so much for doing this. It was really great to get your reaction to this news.
Yeah, and yeah, thanks for having me back. All right.
Thank you.
So, Daniel, thanks so much for joining us.
Yeah, my pleasure. Yeah, thanks for the invitation. It's nice to be here.
What do you think about the aesthetics of the models? And you know what I mean by that is, you know, at least in my experience, like mathematicians all have kind of, you know, sort of, you could prove the same thing in like 10 different ways, and some of those proofs feel elegant and beautiful and sort of, you know, right with a capital R, and some of them feel like a grind or, you know, sort of definition pushing, and maybe you get less insight out of them. How would you rate the models on aesthetics?
Yeah, so first of all, I'm actually kind of anti-aesthetics in mathematics. Like I tell my students that anything true is beautiful, and you should, you know, prove what you're trying to prove by any means necessary. Actually, and I think the models have kind of a similar philosophy, like they're very happy to grind. Like one way in which the models are superhuman is they don't get tired. And I think that's actually, you can see that kind of reflected in this unit distance problem solution. Like they're willing to try something that I think would to a human mathematician like throw up some red flags as like I can probably get lost in the weeds here. Not saying that a human couldn't come up with this. I actually think like if you ask the right human to try to produce a similar argument, it's quite possible they would. But yeah, it's definitely like kind of a scary, like you have to be willing to grind to pursue this kind of argument. And one, you know, so one strength there is like you can see if you ask them to construct an example of something, they're like willing to try 100 things, and that's something, you know, very expensive for a human mathematician to try 100 things, like in terms of time devoted to the problem.
Yeah, in terms of like the actual arguments they produce, like are they different in some aesthetic way from a human? I actually think the reasoning the models show, like you can read the summarized chain of thought from the solution to this unit distance problem and much more summarized chain of thought from ChatGPT Pro when you ask it a math question. It's different from people in that like they will try more things or maybe be more computational, like they'll, you know, they're very quick to write code to do some computation, but the reasoning overall seems pretty human-like. Like I actually don't think there's a huge aesthetic difference. One thing that is true is that the models I think generally show poor research taste compared to a human mathematician. I'm not saying this is anything intrinsic or won't change in the future, but it's just how things are now. So like for example, I have some project now where I'm trying to produce some construction, and this involves trying like a thousand things where it's satisfying some numerical parameters and see what happens. It's very something where the models are now very useful for like, you know, you could just have them try a thousand things. But yeah, when it comes to, you know, when something doesn't quite work and then you want to repair it in some way, they will often choose some kind of approach to the repair that like has no chance of working or that it's like purely computational with no kind of conceptual, you know, conceptual reason to think it would work. And this is something where, you know, I think it's kind of a blocker for them to currently do high-quality research in this kind of thing. So that's arguably an aesthetic flaw or something.
How would you, I guess what would you want to see to feel like they had better research taste? Would it be, you know, sort of 10 different, you know, ways that you could go about trying to, you know, sort of, you know, prove some bound or something like that? They sort of choosing the one that's more likely to work, or how do you evaluate?
Yeah, something like that. Like just some kind of first-principles thinking. And in fact, you can just like, you can already substantially improve research taste by saying think about this from first principles often. Like, yeah, so...
I mean, I think one reason why people have research taste is because we have kind of very limited attention and limited ability to compute, or, you know, spend a thousand hours on something, like we need something to guide us, and maybe the models need that less. So it's also possible that they like start doing high-quality research without having research taste just by doing like a lot more research. But yeah, I think the way I would measure it is like just seeing, you know, are they kind of understanding the big picture in some way. There's some areas where the models I think are like, some areas of mathematics, I mean, where I think the models are starting to have some kind of big-picture view and like able to do some philosophy about the underlying problem. But it's definitely something where there are areas where they're very weak.
