Learning to Live with AI (with Daniel Litt) - Numberphile Podcast

So, I'm a math professor working mostly in algebraic geometry and number theory. Most of the time I'm at the University of Toronto, though this semester I'm visiting Harvard. And so yeah, like most math professors, I spend a lot of my time staring off into space thinking about stuff. Broadly speaking, the goal of my work is to try to understand something about how numbers and shapes are related to each other. And so in practice that means I think a lot about shapes defined as the set of solutions to a system of polynomial equations, and then try to understand how their geometry is related to the arithmetic of solutions to those same equations.

Yeah. So I am kind of a visual thinker. In practice, you know, most of the objects I think about are maybe very high-dimensional or, you know, hard to draw. So I kind of will often draw a cartoon. Maybe it will look kind of like a bean with some lines in it. And then some aspect of that picture is supposed to capture some aspect of the underlying geometry. It's fairly unusual for me to actually get to see the shapes that I think about, although it's happened a few times recently.

Yeah, I love teaching. So this semester I'm not teaching, but yeah, normally I teach two or three classes a year. So recently that has been upper-level classes, like for advanced undergraduates or graduate students. And so yeah, I would say advising is maybe one of my favorite parts of the job. But actually, my favorite class I've ever taught was an intro proof class for freshmen. So yeah, I do a lot of teaching and I really like it.

That's a great question. So these days I mostly — I think maybe the last 10 of my papers or something were with collaborators. So I really much more enjoy working on math with collaborators. I mean, in practice that means we meet once a week for an hour or two, and you know, a lot of the time in between is spent staring at a chalkboard or working with a pen and paper. But yeah, I mean, I think I'm a relatively social mathematician, and you know, one of my favorite things to do is to talk about math. So that's also a big part of how I structure my own research.

Yeah, I would say usually with other people. You know, I find it much easier to think about math when I'm talking to someone. And so I think the typical thing that happens is, you know, I and a collaborator will work on something for a week and make no progress, and then when we meet we'll both start explaining to each other why we got stuck, and then the process of explaining it will clarify something, we'll break some barrier that seemed to obstruct progress. Definitely there are examples where one makes progress on their own. I have a very vivid memory of having found some error in an argument that I was working on, and then the next day having an hour and a half drive from my home to somewhere else, and about 15 minutes into the drive realizing that I had the idea to fix this error, and then I have no memory of the next hour and 15 minutes of the drive. So it was probably an extremely dangerous trip, but by the end I had, you know, a way to fix the argument.

No, I think I just drove on autopilot. Yeah, there's famous stories where — I don't remember the mathematicians — some mathematician in Chicago would give people rides in his car and then, you know, intentionally let the windshield fog up so he could write on the windshield while he was driving. The bottom line is maybe don't, you know, don't get in a car with a mathematician while they're thinking about math. But yeah, that's another way that someone sometimes makes progress.

Yeah, sure. So if I'm totally stuck, I would say the main things I do are lie down on the couch and close my eyes. So I'm definitely not napping. I'm doing math. I'm working really hard. And the other thing is sometimes I will play like a video game for 15, 20 minutes, and that I find useful.

Slay the Spire 2. It's great. Highly recommend. I'm not sponsored by Metacritic, but...

I got interested — I know, okay, I guess I've been interested in AI for a long time. I think the first time I started experimenting with it was in 2022. So this is before GPT-3 came out. The only way you could access it was there was this video game called AI Dungeon where you would kind of — it was like Dungeons and Dragons except GPT-3 would be the dungeon master and tell you stories. So I typically would roleplay as a math professor and try to get it to do math, just because I wanted to play with GPT-3, but this was the way to get it.

Right. In 2022 it was just not — you know, the level of accuracy was like, you would ask it to add two-digit numbers and it would not succeed. So not super useful. And then GPT-3 came out and I tried to get it to write proofs. And here you think like proofs of very basic things that you might learn in a proof-based high school class or something, like proving the little theorem. And then I kept playing around, and I would say a lot of my — to be honest, a lot of my thinking about this was like, you know, there was a lot of hype at the time, which I think was maybe presaging where things were going, but was not accurate as a description of current capabilities. So, you know, GPT-4 could start to execute code and was a bit more reliable for just doing a computation because of this, but still, you know, I just wanted to understand where capabilities were, and they were way behind where the hype said they were. The reasoning models came out, and I think September 2024 was the first reasoning model released, and that was starting to actually be useful. Like typically you couldn't get it to answer an interesting open question, but you could use it to maybe learn about a subject or do some literature search. And so I think the first time I really got use out of a model for my own research, I guess summer of last year, summer 2025, o3 was sort of the next generation reasoning model, and I used that to learn a little bit about hyperkähler manifolds, which turned out to be useful for my work.

So this is for — I had some projects where I needed to — you know, I was studying some particular class of hyperkähler manifolds which had come up for me, and it was not an area I knew much about. I was vaguely familiar with it, and so I decided to try to learn a bit using the models, in addition to textbooks and papers. And it was pretty fun. I mean, at the time the models were very inaccurate, like they would make lots of mistakes. So it was kind of a way to just keep me on my toes. So they would say something that's like 90% right, and then I wanted to make sure I was actually catching the errors. And so in fact, those two uses — literature search and just learning — are still the primary ways I use the models. I think it's incredibly powerful to have some kind of semantic search engine. So I can say just like, you know, here I have the following idea: has any work related to this been done? And sometimes the models will point you to something useful, even something in a kind of distant area where maybe they were using different terminology or something, it would have been hard to find with Google. So that I find super useful. Until quite recently, there have been very few cases where it's like, I want to prove a lemma and then I ask the model for a proof. So, you know, as early as January maybe we saw publicly available models proving interesting open conjectures. But you should think of that as some kind of buckshot approach — like there were, you know, a million open conjectures and they would resolve some tiny fraction of them. So for any given question you might have, probably they're not going to be super useful. So that's started to change now. So I would say now it's pretty common — like if I want to prove a lemma, then the models can prove it, or even do more than that.

So I don't mean to dismiss what the models can do. I think also, you know, if I have like whatever a famous conjecture like the Navier-Stokes hypothesis, or just like — you know, a lemma is usually a small thing you might prove on the way to proving a bigger thing. So, you know, the bigger thing we might call a theorem or proposition or something depending on the situation, often the models can do that too. I think depending on what area you're in and what kind of mathematics you're doing and what kind of a mathematician you are, that might be a more significant thing or a less significant thing. For me it is pretty unusual for me to find a statement of a theorem or lemma that I want without also kind of knowing how to prove it. So it does happen, but the typical thing I'm doing is I have some kind of concept that's maybe not fully rigorous that I'm trying to understand, or some phenomenon I've observed, and then the process of understanding it often will lead to a statement of a theorem and the proof at the same time.

Yeah, so this has happened with exactly one paper that's publicly available, and one work in progress. So maybe they're interesting case studies. So I can say something about them.

So I had a paper come out in April where there were two lemmas whose arguments were due to Gemini Deep Think, which was state-of-the-art at the time. Oh, sorry, one was due to Gemini Deep Think and one was due to Alpha, which was an internal scaffold at Google DeepMind developed by Tony Fang. And so here actually it's kind of an interesting story. So the main theorem of this paper we had proven in December of the previous year. So this was like before the models could usefully do any kind of algebraic geometry.

And then my co-author and I were kind of unhappy with the statements of some lemmas in the middle.

It's not that we were worried about correctness. It's just that they weren't optimal in some way. So the sort of example maybe is that we had some big matrix which you couldn't really compute in practice, but we knew it was there, and we wanted to compute its determinant, and you know, maybe we wanted to compute its rank. And so maybe we knew the rank was at least 50 or something, and we wanted to know was it exactly 50. So all we needed was that it was at least 50, but it's sort of annoying to just have this obvious question and then try to answer it. And then like, you know, okay, you don't need the answer, but it was still good to know. Okay. And so this was an example where none of the state-of-the-art models at the time could answer the question out of the box. And so we played around with it and we computed some examples and we found some better statements. So it's a kind of conjectural explanation for why the rank of this matrix might be exactly 50. And then given that better explanation, Google Deep Think, Gemini Deep Think, was able to just prove that that explanation was correct.

That's right. That's right.

Absolutely. And yeah, so there's sort of an interesting lesson, which is: so now if you just ask the state-of-the-art models to prove the original thing we wanted to compute — to show the rank of this matrix is 50 — they can do it. So like GPT-6 will produce this fairly horrible 10-page computation doing it, but they won't find the better statement we found, and I don't know how to elicit that from them. So it's sort of, you know, there's something lost there. And in fact, I mean, if you're to be honest, like of course I knew in principle that this sort of 10-page argument could work. It's like basically the grind argument, and then I could not bring myself to do it. So it's great to have a model that can do it, but also you lose something. Like, you know, I don't think we would have as deeply understood the situation had we just asked GPT-6, like, 'oh, go prove this thing,' and accepted the 10-page argument. And that would have been on us, of course. Like, one could still reject the bad argument once you have it and try to find a better one that's more explanatory. But you know, it is something where arguably the increasing strength in the model makes it so you have to be a little bit more intentional about actually genuinely trying to understand what's going on. So that was one example. So the other example of — so this is a project in progress where I've gotten some significant use out of the models — is in some joint work I'm doing with Josh Lam on a conjecture called the Gromov's cat-systolic curvature conjecture. There was some question that showed up naturally in the course of our work, which was related to some things I don't really know that much about. So there's a theory developed by Gromov and Shnirelman about harmonic maps from Riemannian manifolds to affine buildings, whatever that is — some kind of weird thing that's a little bit closer to analysis than I usually work in. And so of course I've been learning that theory, but you know, I did just out of curiosity ask the model this question that showed up, and it gave a really actually quite beautiful partial answer connecting it to some other areas that I don't really know much about, so in symplectic geometry. And so this was really cool. So now I get to learn these other areas, and there's like a huge amount left for me to do because it's only some kind of very partial answer. So I feel like I've gotten a huge amount out of this actually. Like I now, you know, have a strong incentive to learn this other area that I've always wanted to learn about. And so I kind of get to learn all this cool stuff, and there's still a huge amount left to do. So, you know, I just feel like it's opened a door and not closed a door.

You think I'm nerdy, Brady?

I definitely did not see what was coming. It was mostly curiosity. I think I was probably more positive about the technology. Like, you know, I think if you had asked me in 2022, like, where do I think it will be in 20 years, then I would have been not very sure about that and said like, okay, maybe it would be doing very interesting things. I made a bet in March 2025, so not that long ago, that the models would not be doing, you know, very high-quality work get operationalized in some way by 2030. And that was clearly just wrong. So yeah, I was definitely way, way too slow in my estimation of progress.

Yeah. So maybe this was this interview with Epoch, probably.

Yeah. So I think that was December of last year. So in my defense, that was like before they'd done anything interesting. But yeah, it has been crazy fast. I would say I have a general sense of just like, I will usually say things like, oh, maybe by 15 years from now we'll be doing this, which is just meant to be safe. But yeah, it's definitely the case that I underestimated the rate of progress.

Mhm.