Good evening. I'm Leia How, vice president for research and nation initiatives at the University of Toronto. Before we begin, I would like to share a land acknowledgement recognizing the unique and enduring relationship with Indigenous people and their traditional territory.
I wish to acknowledge this land on which the university operates. For thousands of years, it has been the traditional land of the Wendat nation, the Seneca, and the Mississaugas of the Credit. Today, this meeting site is still home to many of those people, and we are grateful to have the opportunity to work on this land.
On behalf of the University of Toronto, it is my great pleasure to welcome you to the St. George campus and tonight's special event, "Making Math and What Comes Next." We are especially pleased to welcome many distinguished colleagues here, and a special guest, Professor Geoffrey Hinton. We also have wonderful colleagues from the broader science, technology, and innovation leaders who are calling this university home, and many members from the broader research and innovation community who have come together for this conversation.
Tonight's gathering is first and foremost a celebration. As you all know, just a few weeks ago, University of Toronto mathematician Professor Jacob Zimmerman received the 2026 Fields Medal. [applause]
Based on the crowd, I think you all know that this is widely regarded as the highest honor in mathematics and often described as the Nobel Prize of mathematics. In receiving the award, Professor Zimmerman became the first scholar based at a Canadian institution to earn this distinction and only the second Canadian ever to do so. It is a remarkable personal achievement and a landmark moment for the University of Toronto and for Canadian scholarship. Congratulations, Jacob. [applause]
The Fields Medal recognizes not only extraordinary accomplishment, but also the promise of discoveries yet to come. It is fitting then that our focus for tonight is not only on what has been achieved, but also on what lies ahead.
Professor Zimmerman was honored in part for his groundbreaking contributions to the André Oort conjecture, a problem that has challenged mathematicians for decades. His work helped resolve a deep question at the intersection of geometry and number theory while opening entirely new directions for future research.
For many outside of mathematics, the details of that work may seem distant from our everyday lives. Yet, fundamental advances in mathematics have always expanded the boundaries of human knowledge in ways that reverberate far beyond the original context. Mathematics gives us a language for understanding patterns, complexity, and possibility. It underpins science and engineering and computing and increasingly artificial intelligence.
That is one reason tonight's conversation matters so much. We are living through a period of extraordinary technological change. AI systems are becoming more capable, more influential, and more deeply integrated into our lives. At the same time, they are prompting profound questions about knowledge, discovery, creativity, responsibility, and trust.
Questions such as: what role will mathematics play in this new era? How might AI transform the way mathematical research itself is conducted? Could the integration of mathematics and AI accelerate the development of more capable machine intelligence? And perhaps most importantly, can mathematical thinking help us address some of the fundamental challenges associated with advanced AI systems, including questions of reliability, alignment, and safety?
These are not questions that belong to a single discipline. They sit at the intersection of mathematics, computer science, engineering, philosophy, and public policy. They demand collaboration across fields and across sectors. That spirit of interdisciplinary inquiry is central to the identity of the University of Toronto. Our greatest contributions often emerge when exceptional people from different domains come together to tackle consequential questions. Tonight's discussion will be a powerful example of exactly that.
It is also, I think, a reminder that universities have a vital role to play in shaping the future. We are not only places where breakthrough discoveries occur. We are places where society can examine the implications of those discoveries, ask difficult questions, and translate discoveries into impactful policy and technology that benefit society. And so, while we are here to celebrate an historic accomplishment, we are also here to explore what comes next for mathematics, for AI, and for the communities that will be shaped by both.
Before I turn the stage over to Jacob, I'm delighted to introduce a video message from Professor Melanie Woodin, president of the University of Toronto.
Hello. I wish I could be with you in person today, but on behalf of the entire University of Toronto, I want to offer warmest congratulations to Professor Jacob Zimmerman, winner of the 2026 Fields Medal.
None of us will ever forget the moment the news broke. It made headlines around the globe: "Toronto professor wins the Nobel Prize of Mathematics." Led by Prime Minister Mark Carney, the whole country cheered the first researcher at a Canadian institution to be so honored. Here at Jacob's alma mater, people across our three campuses were elated, especially in the math department and the Faculty of Arts and Science.
My own inbox was deluged with congratulatory messages for the university, and I was delighted to dive in and answer them all. And I think we also all heard from family and friends who were excited by the news. Jacob, your achievement is still reverberating, as you can see from today's gathering.
You've made us all very proud as your colleagues, students, and friends, and as Canadians. You've reminded us of the importance of fundamental research as an expression of our shared human aspiration to push the boundaries of knowledge and understanding.
And now, in your leave to focus an eye on AI safety, you're setting a great example of how we can apply our learning to the most urgent applied challenges of our time. It's a fantastic record, and as the Fields Medal acknowledges, it's a sign of more great things to come in your brilliant career. Thank you, Jacob, and congratulations once again. Thanks everyone, and enjoy today's continuing celebration of this historic accomplishment by one of U of T's leading lights.
And it is now my great pleasure to introduce our distinguished speaker, Jacob Zimmerman, professor of mathematics at the University of Toronto. So please join me in welcoming Jacob once again and congratulating him on his extraordinary achievement. [applause]
Thank you so much for coming. It's an enormous pleasure to celebrate here in Toronto. Toronto is a very special city to me, and I work into the presentation a little bit of how much time I've spent here and the different phases that I've gone through as sort of the world is changing so rapidly and my life recently changed. I find myself sort of thinking about the ways that I've spent my life in relation to mathematics. Mathematics is something that's been present in my life since I was three, I'm not sure, and my relationship with it has changed a lot.
And I've always been sort of a very goal-directed agent. I've always been doing the thing that I find myself doing without always examining what it is that I'm doing and how it is that I'm doing it. So, I wanted to meditate on this idea a little bit of theory building and problem solving. But rather than posit the two as sort of at odds with each other, I've always been interested in solving problems and accomplishing the goals that I've found presented in front of me. And theory has always been the thing that allows me to accomplish those goals. And so as we move forward in our world, as I move forward, I sort of want to walk with you through what that's looked like in my life and what that might look like next, and hopefully it'll allow me to reminisce a bit about Toronto as well.
Filling a bathtub. I first engaged with mathematics through the riddles that my grandfather would give me. He was my first math teacher. I've talked a lot about this toaster problem because it's such a clean problem to discuss. But I want to talk about another problem that I think is more common, about water flow and filling a bathtub. So concretely, the problem is as follows. You have a bathtub. You have two sinks at either end. If you turn on one sink, it fills the bathtub in three hours, and the other sink fills it in six hours. How quickly, if both sinks are on, do you fill the bathtub?
So, it's a problem that's not extremely difficult by the tools of modern mathematics. But to somebody who's just starting, it seems impossible. How do you start? You know, there's all sorts of things you could try. You can try manipulating the equation. How many bathtubs can you fill in how much time? Do you add the hours together? Can you find a way to add these numbers together? And what I want to sort of point to is that the answer is this concept of speed. As soon as you realize that you can reformulate this as: in one hour, how many bathtubs do you fill? The answer is 1/3 for this sink, 1/6 for this sink. You add those, you get a half. And so then you get that the answer is it takes two hours to fill one bathtub.
So it's quite simple once you get this concept. And thinking about it now, at the time I was like, "Oh, that's really cool." But coming to it now, what I think about it is the theory is a theory of speed. We develop that theory, we build that theory, and once we have it, it allows us to solve the problems of filling a bathtub. And so that's kind of how I approached things early on when I first came to mathematics, that you have your theory, which is really a collection of tools or tricks or ideas, and that allows you to accomplish your goal, which is problem solving.
Now the sort of next major phase in my life with mathematics was definitely high school competition problems. And I started sort of in earnest working on high school competition problems down the street at University of Toronto Schools, which was my first experience with University of Toronto. And I think, yeah, A. Eagle is there in the crowd, who while I was at UTS was kind enough to allow me to attend his calculus lectures. [applause]
And at the time, the sort of ontology of what I was doing is there were these problems. I'll give you one in a second, just if you want to look at the aesthetic of them. But the idea is you had these problems which are difficult. They're extremely elementary. They require very little prior knowledge and they're very elegant. They're very beautiful. What does elegant mean? Well, it's not clear, but it means something about intelligence, about the way we interact with these problems.
And so one example of this is sort of geometry, one of the more aesthetic parts of mathematics that classically we did and we still do, even though it's in some sense a solved problem by the standards of modern mathematics, but provides a rich arena to engage with for especially a developing high school student who's looking to get interested in the subject. So you have sort of, you know, what I will label theory, though some may disagree. You have trigonometry, you can chase angles, you can study how circles work, you can use coordinates, though it can be hard.
And this is something that you sort of learn as a student. You learn this bag of tricks, you learn this theory, and then you apply it to all these different problems. Problems about if you do various things, you get various different points to the same point, or some ratio is always one half, or three points line to line, or whatever else. And this was sort of my experience with mathematics. I would sit in rooms for these math contests, whether it be the CMO, the USAMO, the IMO, for many, many hours, and you had these very difficult problems. You had to just sit there and bang your head against the wall, try to make progress.
It's a very useful experience and I think a very useful ability to sort of sit in a room and not have the answers and despite that engage in the bold task of trying to get the answers. And it's something that I want to advertise, I suppose, in a way when people sort of knock theory and they think, well, there's no more room for theory in our lives. I think there's some form of courage that's represented most by sitting in a room and banging your head against the wall. That is fundamental to this endeavor.
Here's one of the problems, is to draw a pretty picture. Everything here was done by AI. Obviously, I wrote some notes by hand. Then I was like, "Make it pretty." And it did. This is the CMO problem. You got some circles, you draw a line, you got to show some ratios always the same. It doesn't matter what the problem is. It looks really pretty and it's really fun when math gives you pretty pictures to look at. Hopefully that's going to be a constant going forward. That's one of my hopes for the future.
All right. So then I went to the university again at the University of Toronto, just down the street. I think when I started it was at the Sanford Fleming building, though maybe it was Sydney Smith. It's one of those two. I was confused with that. One of the buildings, then we moved to the Bahen Centre. And Galois theory is a subject which, if you don't know what Galois theory is, doesn't matter, it's irrelevant to the story, but it's one of these subjects that really sticks in my mind because it's something that I learned about 12 times before I understood it. I think I had to keep learning and relearning until, you know, my head clicked and went, ah, now I get it.
And it shows a sort of, I think, chain here that I want to emphasize, where you have this theory which is basically algebra, right? You have the tools of algebra, you add numbers, you multiply numbers, we all study adding and multiplying numbers at various levels. And then you have this problem that really crystallizes the theory, which is the fundamental theorem of arithmetic — sorry, that should say algebra, because that one says algebra, that's my bad — every polynomial has the exact right number of roots. And in that problem inspires more theory.
Roots have symmetry, as Galois noticed. Root two and negative root two look kind of the same, and you can study the ways in which they look the same, and then you can think about that for a while, and then to really crystallize the theory, go to the Galois correspondence, which is a thing, doesn't matter what it is, but it crystallizes the symmetries that roots have. And you can go further, there's more stuff you can do. But this chain is one that when you go to undergraduate mathematics you realize has been happening for sort of centuries, and you get to see the final form of it.
You know, generations of getting theories, crystallizing them in problems, using those problems to refine more theories. You have goals that you describe by the problems, and you make the theories to achieve those goals. As a very goal-oriented agent, I really like this process.
So then I came back to Toronto to the same building as a professor. It's the only university where I've ever been a professor, not counting visiting positions. And it's a really nice place. I have an office at the Heron Institute with a rule here. You again, you're in the same building. No, kind of. There's like a kitchenette, and we often show up at like 6 p.m. when no one else is there, and you have all the time in the world to sort of sit and think. It's really nice.
Anyways, the André Oort conjecture is a question that I spent a long time thinking about, sort of one of the main tenets of my career, and I'm not going to talk about what it is or how we got it exactly, but there's this chain here that I wanted to draw, and it's much longer than what's on the screen. There's sort of many more circles here, right? Because we pay attention to individuals doing things. But even before you start doing a thing, there are so many other things you have to learn, so many theories that allow you to even start considering engaging with the problem and achieving your goal.
So in particular, I want to isolate this one over here. There was this idea in the 20th century, I think started in the 1940s or so: when are functions tame? You can draw functions, and some functions are complicated, and some functions are simple, and what does that mean? And people played around with formalizing that intuition, trying out different versions of what simple means and what complicated means. And they finally came up with something: the theory of o-minimal structures, which gave an answer to this question.
A theory that allowed us to then do various other things like the Pila–Wilkie determinant method, definable Chow theorem, whatever other stuff, which then allowed me and my collaborators to eventually apply to the André Oort conjecture. But I want to isolate this process of: in this case we had a very concrete goal, which was this problem that we cared about for various other fundamental reasons. And we went and looked for tools to achieve this goal, and we had those tools. They're right here. All these tools existed, and we used them. We put them together, and we achieved the goal.
And then in another part of my career, I worked on something else called the Griffiths conjecture, which related to all this stuff, but it wasn't enough. The tools sort of weren't enough to achieve the objective, and for the first time in my life I found myself doing something new, which was doing a little piece of theory building, which I was very excited by because it's not something I'd ever done before. And of course there were many people whose footsteps I followed. There was a whole book that I read and tried to sort of copy line by line as much as I could, but I couldn't copy all of it. I had to do some of my own stuff, which was both fun and also kind of a drag.
But eventually you get these nice results that are really cool in and of themselves, and then you have them, and you have these tools, and then we prove this new thing, the Griffiths conjecture. So there's this constant process of you have a goal, you're looking around for tools to make that goal. I think of those tools as theories. That's what they are in our lives to me at this moment.
So the next stage for me, at least, is MACI, the Mathematical AI Safety Institute, which we launched today together with Andrew Rich, who's in the audience here, my executive director, Gideon Roel on the board as well, where we're going to try to bring mathematicians and the mathematical process to AI.
I think it's not controversial to say that AI is one of the biggest challenges in our world today. But I think what's more controversial is to think of AI as a mathematical problem. I think it's often thought of as an engineering problem. People think that theory doesn't really have a part to play. You know, theory is pie in the sky and AI is concrete engineering. You code stuff up, you tune some parameters, you do a bunch of stuff. But the problem is you can only implement and do the ideas you've already had, and you can only fight with the theories you've already developed.
And we have very concrete problems in AI that we don't talk about now because they're too hard, and so we leave them out of the story. But the problems are very real. How do we trust more powerful agents that we're going to get very soon? Arguably we have now. What does learning entail? We're creating agents. They're somehow learning things. They're telling us about them. What is actually happening when they do that? What's happening when we do that? What is a theory of learning? We have a theory of information. We have theories of complexity. What does learning mean?
When agents interact with each other, when they model each other and themselves, what do they do that? How do we best conceptualize that? These are very real problems. We don't currently have the tools. That's our mission, is to make these tools. Thank you very much. [applause]
Thank you, Jacob. Thank you for the wonderful keynote address. You've given us a great deal to think about. And fortunately for us, the conversation does not end here.
I am now delighted to welcome to the stage Professor David Duvenaud, the Schwarz Reisman Chair in Science and Technology, associate professor in the Department of Computer Science at the University of Toronto, and CIFAR AI Chair. David will join Jacob for a conversation that builds on the ideas that we've just heard, exploring what the evolving relationship between mathematics and artificial intelligence may mean for research, discovery, and the future of AI. Their conversation will be followed by an audience Q&A. So, please join me in welcoming David to the stage. [applause]
All right. Testing, testing. Okay, great. So, now we're going to mine your brain. So, just at the very end there. So, yeah, you announced a new institute that you're founding. And your rationale was a little bit glossing over the safety part. Like you mentioned AI is big, math is helpful, AI can help math, but your institute is specifically about safety. Can you — and you've spent your entire career so far working on hard problems in mathematics. So can you maybe answer what convinced you that AI safety is something that's now worth focusing on?
Yeah, it's a great question. So I've been around the sort of AI safety community for a long time. And I was always convinced of the part that if we manage to actually create artificial intelligence, however we define it — it's very much a "know it when we see it" kind of thing, and in my opinion, we see it — safety would be an issue because we're making creatures, systems, beings, whatever you want to call them, that are in many ways more intelligent, and in many ways knowledge is power. This is true. We see that happening. So I think safety is a big issue to be dealt with.
This is something I became convinced about maybe 10 to 5 years ago. I'm not sure when I started seeing the direction things were going in terms of now being a good time. I think there's several factors at play. One is that the world has come to accept that AI is a real thing that's really happening. It's sort of there's been a transition from thinking about AI in the realm of science fiction to thinking about it as something we're really living with and really grappling with the consequences.
And so in terms of why MACI is launching now, part of the reason is just that there's energy around safety now. There's interest now. People understand the underlying concepts. Another, I think, really important reason is that we've been hearing a lot about — I mean, today besides this was a really slow math day [laughter] — but in general we've been hearing a lot of progress in math being done by AI systems, and it's kind of been, in my opinion, thought of mostly and talked about as a curiosity or a milestone, like, oh, what does it mean, you know, now that AI can do things like this.
But in my opinion, the fact that AI can now solve math problems at the highest level — the Navier–Stokes conjecture was announced very likely solved, hard to argue it's not one of the hardest problems in mathematics based on the amount of effort everybody has put into it and the importance it has, it's on some cool lists — so we're now at a point where AI can do and assist with research at a superhuman level. And so the kinds of things that are in scope to attack from a theoretical perspective have really increased and are going to increase a lot more very soon.
You know, there's in some sense it's sort of people say off-handedly everything is secretly math. Like physics is secretly math and chemistry is secretly physics and biology is secretly chemistry and so on and so on. And the reason physics isn't actually often dealt with mathematically is not because that's wrong, but because it's too difficult. If we take the physics problems and actually model them mathematically, we can often do that, but they're too hard for us. And mathematicians are nothing if not pragmatic. You work on what you have and what you can do. You measure yourself by what is reasonable, and what is reasonable is about to increase.
So these questions that I wrote, which might seem very ambitious, I think there's a good chance that they might start being things we can really make progress on.
I see. But is it your argument that AI safety is relatively neglected and that there's going to be a lot of low-hanging fruit, or is it more that just on its own merits AI safety is something that we just need to work on?
I think both. I think there are definitely low-hanging fruits. So I think AI safety has had people working on it, but it has not had the sort of full power of academia looking at it. That's part of the reason why MACI is attempting to be founded at the scale we're going for. We're aiming to bring 100 mathematicians together in the same building to day in and day out drink tea, eat cookies, and think about AI safety. That's a really powerful tool, the tea and the cookies and the safety.
Well, and so this safety, I guess, has been like a broad tent, but I think it has so many different parts that maybe people don't know exactly what you mean, or if you have a specific area in mind, can you maybe talk a bit more about what you mean when you say safety?
That's a great question. Yeah, so safety means a lot of stuff. The broad goal of safety, I think, as an umbrella term makes a lot of sense. It means how do we make superintelligent systems that don't end up doing very bad things that we don't want. That's, I think, one pretty decent description of AI safety. Now, there's lots of parts to that. There's cybersecurity we hear about all the time. There's security from other forms of attacks. There's trying to build guardrails. There's trying to build control systems. There's trying to monitor things.
But fundamentally, there's other goals that we're neglecting because we don't have understanding for it. For example, how do we get any understanding of what an AI system is going to do next? How can we predict what it's going to do? How can we predict what it's going to want? Can we make sense of what "want" means? Right now, the way AI works — everybody says this — AI systems aren't coded. They're grown. They're evolved to various forms of selection pressure. You know, we don't understand how they work any more than we understand how frogs work. We do experiments.
And the idea is if we can make fundamental theoretical advances on those questions, then we can take those insights and try to aim higher with what we're doing right now on safety.
I see. So, so far you're keeping the conversation very concrete and on unobjectionable goals, and I'm wondering — you keep talking about safety, and I'm wondering if you would be willing to speculate and talk just a bit about what are the sort of top scenarios in your mind, or things that you're actually worried about that you hope to ameliorate. Like, are you like, oh no, everyone's going to be hacked, or is it something more extreme than that? Like, what's actually keeping you awake at night on this topic?
Yeah. So, kind of all of the above. I mean, I wrote a paper with Andrew Rich in the audience called "A Taxonomy of Omnicidal Futures," I believe is what it's called.
And maybe it's worth saying that word a little more slowly.
A Taxonomy of Omnicidal Futures. And "omnicidal" is a word that I think we invented, kind of, that we used to mean at least half of all humans die.
Which is a sobering and scary thought to deal with, and we don't write the paper at all to be provocative or to be glib, but I think this is a real possibility that we have to contend with. And the way that we deal with risks and concerns that are actually present is not to flinch away from them, but to look at them directly and try to steer away once we understand what they entail.
To answer your question, what specifically we're concerned with, the paper outlines five or six different scenarios, concrete scenarios. And we are concerned about all of them. And the reason it was written is that people often ask, well, what specifically are you worried about? You know, there's all sorts of vague abstract arguments about systems more powerful than other systems and instrumental convergence and all these words, but what concretely do you think will happen? And of course that's impossible because predicting the future with enough detail is always a doomed endeavor.
But you can write things that become convincing to various groups of people, that people can read one story and think, "Oh, this actually feels plausible. You know, there's parts to the story that I can actually see happening." So on that list are AI agents getting their own goals and as a result doing things you wouldn't want them to do, taking control in various ways. There's bad actors of various levels misusing AI systems.
You were talking about people being hacked, excuse me, which is often referred to or is similar to super persuasion, getting AI systems — which, I mean, we've already seen an attack pretty recently, a cyber attack, a social engineering attack, where an AI system being tested by the UK Artificial Intelligence Safety Institute attempted to trick a human into accepting some malicious updates, into sort of voluntarily giving up control from the system.
It made some update. It was very hard to understand. The human was skeptical, and they tried to convince the human very persuasively, if you read the transcript, that no, it was okay. I mean, this is a form of super persuasion, right? Not super yet because it didn't work. But super persuasion isn't that hard. Humans just aren't that difficult to convince if you know what you're doing. The world is complicated. Humans can't keep it all in their heads, including the ones that are in charge of our security systems.
You can think about governments racing to build AI first, a situation we're unfortunately in to some extent at the moment. I'm concerned about all of these, and I think there's lots of difficult problems to solve, political questions, societal questions, once we get over the hurdle of can we make superintelligence at all that we understand to any reasonable extent. And until the answer is no, unfortunately the specifics of the risks that come next don't really matter. You have to sort of clear that first hurdle before you get to worry about the rest.
I see. So on your last slide, you had a few teasers for directions that you might want to work at, or people might want to work on in your institute. I'm wondering if you can expand a bit and talk about, like, you know, what can mathematics unlock for safety? Are there any particular success stories that you thought were counterintuitively valuable, or any areas that you think, or maybe signposting those where you think there's low-hanging fruit you're excited about?
Specifically in AI safety?
Yeah. So I think the state of the art is like there are research directions and there are good ideas. There's definitely a place to start from, and we also don't have it yet. So the fundamental goal of MACI is to come up with new ideas and new insights and sort of the right direction. Having said that, we're not starting from scratch. People have had good ideas in the past. So a few of the most often talked about methods are: there's something called safety by debate, invented by — I forget all the authors. Geoffrey Irving, I think, Paul Christiano, Dario was there.
And Dario was there. Yes, that's right. Maybe those are three authors, actually.
Yeah. Where they try to invent a protocol by which, if you and I are talking and you're much smarter than I am — you're an AI system, which is literally the case for many of our conversations — you tell me a bunch of the answers to my questions, and you don't have time to explain it to me because explanations are hard, and I just don't know as much as you, and I don't trust you super much. I don't know how you work. I don't know what your goals are. Can we still come up with a way to communicate by which I can trust the answers you're giving me without having to trust you?
And they have some recommendations, some protocols. Those protocols have flaws. They've improved on them. They've gone back and forth. This is something which is — I don't know for certain, but seems like it's actually used at various labs to some extent as a safety tool right now. So it's already practical to some extent, but there's sort of a lot of room for improvement.
And you know, one concrete scenario you can imagine is we invent like the best protocol. You come up with something that's really practical and really trustworthy, you know, it's very close to something in computer science which I know very well called zero-knowledge proofs, where I can convince you of something without revealing my hand. If we can come up with that, we can imagine the future where we interact with AI systems through this protocol. Now, it doesn't solve the whole problem, but it solves enough of it to already give us a much more manageable sort of way forward.
Right. And I suppose you could argue that once we have superhuman AI assistants, we're all going to be in this position sort of all day, every day, where we're talking to our assistant and we don't know how to verify that it's not misleading us.
We already are. I mean, how many times the coding agents ask you, "Is it okay if I do this?" And it's pretty long, and I try to read it, but I'm sometimes in a rush.
Yeah. Yeah. So in that sense it's already a practical problem. Okay. So but then maybe if there's some more pure mathematicians in the audience, like that almost sounds like a boringly prosaic question now that we've connected it to this concrete thing. Like is there anything more exotic that you think might be surprisingly useful, or some more, you know, like the classic number theory was useless until cryptography came along. Is there any areas where you're thinking experts in these areas should apply because it's actually going to be surprisingly related to some of your problems?
Yeah, I mean, it's hard to — I'll answer your question in a second. I'm not going to weasel out of it. But it is hard ahead of time to sort of name a concrete one, right? Because in retrospect, it's always clear which ones are useful, and if I start saying some now, there's many candidates, most of which presumably aren't going to work out, at least until they've had some radical transformation.
There's a theory called agent foundations, which is a collection of results all reasoning about the question of — usually when we talk about game theory, we think about agents making decisions about a world that's external in a way that doesn't involve them. You're playing a video game. You know how the video game works. You get to think about what would happen if I did this or that, and then making decisions based on that. That can be hard. Video games are already very difficult. Nonetheless, we know how to think about it.
The real world isn't like that. I'm thinking about the real world. The real world contains me. It contains you, who is very similar to me. If I'm an AI agent, it contains a bunch of other agents which are not just similar, but in many ways an exact copy of me. How do we deal with that? How do we deal with the fact that if I make a decision, an exact copy of me would also make the same decision? We're at the same time the same agent, but also different agents because we're physically separate in the world. It's a hard question to even conceptualize.
And the thing I want to sort of communicate is that sometimes when something is too hard to conceptualize, people think, well, it's nowhere near ready for prime time. If we can't even think about the right mathematical foundations, how is it ever going to be useful? But it's remarkably quick to go from the right mathematical conceptualization to implementation in a real-world scenario up to scale. And the pipeline between theory and empirics right now that exists in the AI and AI safety community is extraordinarily quick.
Yeah. And maybe that's a good example of an area where there was surprisingly deep math, I think, involved in solving some of the like logical induction, liar's paradox kind of stuff. And then also it was sort of a surprisingly concrete algorithm in the end, although no one's implemented it yet as far as I know. Okay.
Great. Okay. So let's talk about institution building. So, you know, you're founding a new institute, which is always something hard to do. And as far as I understand, you're planning to staff it mostly with like academic mathematicians, but not have it be an explicitly academic institution in like the form of a university. So what are you planning to do differently from a university? And what could universities do to rise to the occasion of like math mattering in this like new important way?
Yeah, sure. I think in many ways what we're doing is not unprecedented because there are organizations that are research institutions that are affiliated with but not directly under the purview of a university. One example is the Institute for Advanced Study near Princeton, where I ran a special year last year actually, and it has the habit of getting academics to run a special year. They bring together around 100 academics, they all think about something for a year or two, and then they go off and they bring the next batch in.
There's the Fields Institute in Toronto, which is structured slightly differently but has a similar principle. There's other institutes of a similar nature. So in that sense, we want to really import academic culture. I mean, I joked earlier about tea and cookies, but it's not really a joke. I mean, having just a collaborative space where people can come and think, and they're incentivized to work together and share ideas together, is a really powerful tool.
The reason it's different from a university, there's sort of two reasons. One is that the world's moving very quickly, and academia, one of its strengths or weaknesses depending on the situation, is that it moves quite slowly, and it's slow to update, and it reasons very carefully about everything. So that's one reason it's being independent. Another reason is that universities are ultimately geared towards education, and it's hard to launch a research institute about a subject that's not yet fully developed and make it an educational institute at the same time.
You know, it's very possible MACI — you know, one of my ideal worlds, my...
Top world is somebody fully scoops MASI and solves AI safety tomorrow, and then this all shuts down and I get to go home and do some more improv. Another pretty good world is MASI succeeds, we get some strong insights and foundations, and then maybe a year or two out, we start accepting students based on those insights. We develop a school and we get some sort of pipeline of AI safety issues working on AI in terms of what universities can do to step up to the occasion.
There's very pragmatic things like make an AI safety department, and it doesn't have to be made brand new. You can just take a piece of computer science and a piece of mathematics and a piece of physics or whatever it is. I don't mean to leave out fields, I just don't know all of them. And have them rename themselves if they want. You are now the AI safety department. And there is interest, like I've talked to many, many people, many of whom are worried, many of whom are wondering how they can use their skills to help. So that would be an extremely sort of quick thing that can be done. I think en masse it would lead to conferences and workshops being done. People can start sharing and collaborating knowledge. I think a lot of fast pieces move into place really quickly once you just make that nomenclature change.
Okay, so we've been talking about professors so far. What about like students or just sort of more junior mathematicians who also want to contribute, but maybe they think like, oh yeah, I don't have like shovel-ready skills, or if they want to contribute to AI safety, sort of where would you advise they think about trying to contribute, or what skills or areas should they develop?
Yeah, that's a hard question. It's hard to give the best answer to the question. There are lots of answers. So there are by now plenty of fellowships they can apply for. There's plenty of educational programs. There's Blue Dot, there's Iliad, there's MATHS. On our website, we have a bunch of links to several organizations that help with this. There are plenty of others that aren't there. There's UKIS, which is a various teaching fellowship. So there's places you can go to apply to sort of quickly get up to speed on what's going on, start contributing. You can also, I think, the immediate answer is just to start educating yourself. And what does that mean? It could mean use a coding agent to do a thing. That's literally how I started. Together with Arul, we achieved one of our, I guess, lifelong dreams. Not a very high priority one, but a lifelong dream nonetheless, of making a computer play this game called Conway football at a superhuman level. We got to a point where we were like, you know what, coding agents are really good. Let's just see if we can make a computer play this game at superhuman level. And we sat down with an AI agent and, you know, we did everything. I mean, we helped a lot, we were very useful, but we didn't code a single line of code. And that experience that students go through now, if somebody were to sit down and use an agent, it's a very formative experience. You learn how to deal with something that's much more powerful than yourself. You learn how to get information back, feedback at a rate that's previously impossible, and you learn how to adapt to that situation, how to be a manager of the appropriate kind, where skills fit in, because they do right now, especially during this transitory period we're in right now. There's many things humans can do that AI can't do. I don't know how long that'll last for, but at this moment it is true. You can read various writings on this, papers, blog posts, again, these courses. There's various startups and organizations that exist that attempt to take AI integrated into the current world. You know that the problems we are concerned about are existential risk and solving these foundational questions. But as you alluded to earlier, once we solve those questions, there's so many questions left about how AI makes sense in our world. And if we don't solve those questions, then a lot of tumult is going to ensue. And that's extremely risky by itself. What happens to the education industry, to the transportation industry, to the material science industry, to all the other industries that I can't name right now because, again, I don't know very many of them. Right now I don't see nearly enough forward planning. I see a lot of let's adjust to the current AI tools, find their disruptors, but it's basically business as usual. Let's plug the AI in and adapt, and not enough things are radically changing. Let's look ahead and see how we can adapt.
I see. So you're saying even if we manage to solve sort of immediate loss of control and alignment to the extent that nothing blows up today, we still don't have a plan for like if we succeed, what are we going to do with these tools in a way that means our civilization still runs according to how we want it to. Is that what you're saying?
Yeah, that's right. And I mean, even there's concrete risks again, but there's also, I mean, you have this work on gradual disempowerment, which is extremely influential, talking about how does society function once humans are no longer able to produce economic value to our world. That's a question, right? It's not meant to be a rhetorical gotcha. It's a literal question that has literal answers that we haven't really found yet. And I think people who think about social structures, who are in the humanities, you know, who are in governance, whatever else, should go and work on those questions and propose concrete solutions.
Right. So you're saying the sociologists in the room should be also freaking out and making their own institutions or new departments or whatever.
Everyone should make an institution.
And that sounds like a joke, but it's not a joke. Please make an institution.
So just one thing you mentioned, you talked about MATHS and Blue Dot and all these different mentorship and sort of onboarding things. So I'm wondering if you can maybe compare and contrast a little bit from the point of view of let's say someone halfway through their undergrad. Yeah. And they're like, okay, I can take more math courses or I can try to get into one of these mentorship programs. Maybe I can do one in the summer. And like, I'm just wondering if you can compare and contrast and say for whom it would make sense to like go this sort of standard math education route or how you can mix in these other elements.
I mean, look, it's hard is the honest answer. I mean, the world is uncertain, and I think when the world is uncertain, you hedge. So one answer is sort of keep your options open, keep engaged with reality the way it is, and adapt accordingly. On the other hand, you don't have infinite time on your hands. So if you spend a lot of time doing everything, you might become a jack of all trades, master of none. Also a risk. I don't have one thing I would tell my PhD students when I was an adviser is that I don't know the way to succeed in math, but I know a thousand ways to fail. And I'm going to try telling you all of the ways that I know so you can avoid those, and then hopefully something good will happen. It's kind of similar here. I mean, one guaranteed way to fail is not to use AI, not to keep in contact with what AI is doing. Another guaranteed way to fail is not to invest in yourself and to sort of take a nihilistic approach that, well, the AI will do everything I can soon, I should stop improving myself. Those are two failure modes in terms of choosing between them. Different people know themselves better and have different skills, and I just don't know the answer, unfortunately.
Okay. Well, speaking of strategizing, maybe we can take a break from AI safety stuff and talk about some Fields Medal gossip. Yeah. Can you... So I saw an interview where you mentioned at some point you just decided like, crap, I'm like 30-odd, if I want to get a Fields Medal, I need to like do X, Y, and Z. Can you just explain a bit about your like thinking and strategizing and how that all played out?
Yeah. I mean, you pretty much nailed it.
I mean, look, I had this experience where I worked really hard in high school to do well at the IMO, and then I did. And then I went to grad school, and I found myself once again at the bottom of a hill with all my previous accomplishments meaning nothing. And then I saw there was another hill to climb, and I kind of decided for my own sanity and mental health that I'm not going to focus on prizes. I'm going to try to do math and enjoy it because I enjoy doing math, because I do enjoy doing math, and it's sort of in the background. Nobody goes into math, not nobody, but almost nobody goes into math without enjoying it. It's so much work. There's not that much reward. You do it for the love of it because you find it so beautiful. And so I did that, and it was going pretty good, and then, yeah, I think I was 36 or 35, and I thought, boy, there's a time limit on this. I got three years or two years or whatever. And I decided to focus on it again, partly because I sort of thought I'd regret it if I didn't, and partly because I thought even if I don't get it, I'm sort of only trying for two years or whatever it is, and then time's up, and, you know, I'll have tried my best. So it's kind of a consider, kind of like a bet in that way. And also, yeah, I think I would have regretted it if I think if I didn't get the Fields Medal after trying, I would have been okay with it because I would have felt that I sort of did my part. I probably also would have needed, you know, some hugs from my wife, but yeah, the motivation was really I'm in a position where this makes sense now, so let's go for it.
Well, how grindy was that? Like, how much less fun was the problems you chose because they might, like, you know, hit this big target versus the ones that you were doing, like, because they were the most fun and interesting?
Aru and I actually finished a project that we started in grad school and we never had finished before that required, it was this annoying combination where we had to like actually think about it and there was a lot of actual computation. It was sort of a computation, but it's brainless. You can just kind of do it and grind your way through, but if you have to really work to do it and then also you have to stay engaged and keep thinking, that's kind of too much. So we just never did it. Every year we were like, this year, oh yeah, we're definitely going to do it. And then as a result, we did it this year. And look, I mean, I'm a firm believer that in my experience, for me at least, things require effort to find the fun and joy in them a lot of the time. Not always, but like often, you know, I don't find the comedy show funny to have given it a chance and I've understood how the comedy works. And likewise with math, like the more time you spend on something, with anything, the more I understand something, the better it is. So it's not like I was miserable these two years, but yeah, I sort of stopped yoloing it for a while and buckled down, and it's more like I neglected the projects that I would just do because I found them fun at the moment and started working on the ones that I thought could actually have some reward.
Okay. So you sort of started saying no to yourself. Well, how did it feel like a year in? I mean, having grown...
Felt terrible. I didn't have anything. I had nothing to show for it. And it's weird, like in math research, the rewards are so sparse that like I've had years where everything I did worked. Like every problem I tried would succeed and I'd be like, man, I'm on fire. And then I've had like years where nothing I try works. And, you know, I'm a grown man. So I'm used to dealing with failure. But like three, four, or five failures in a row, it's rough. You start thinking, boy. So the first year was kind of like that. Nothing was working. And then things started sort of working and I started feeling better.
Well, I'm kind of curious about this dealing with sparse reward. Like, yeah, there's lots of endeavors that are kind of inhumane in the scale of like when you get the reward versus when you did the work. I mean, and you mentioned starting from nothing and having like your previous accomplishments mean nothing, which I think you meant you were referring to like having done well in contests, but no one cares about that mathematically. Okay. Do you think that your life inadvertently or on purpose provided you with a curriculum for learning from sparser and sparser reward such that you could take these longer and longer bets?
That's a good question.
Yeah, I think so. When I went to Princeton, I actually had some visa confusion, so I came late by three days, and so I missed the orientation. And the secretary there, Julair of the Math Department, who kind of runs the whole show, she was like, oh, Jacob, hi. Like, he missed things, but the important part, like, he didn't miss much. The thing to remember is keep working even if you don't know what you're working towards, which is wonderful advice. It's like all advice, it's very easy to give and very hard to do. But it was very orienting, and I keep finding that it's true more and more in my life. I sort of, I'm in these positions where I feel like I'm trying to do something. I don't really understand anything. Nothing's going anywhere. I don't have any complete stories. I feel like I'm wasting my time. And then later things coalesce. It was similar with MASI, which, I mean, it hasn't, we just launched. So it's hard to call it a success at this point. I understand that. But I mean, I sort of, I was in the AI safety. I was looking around. I really wasn't sure what I was supposed to do. I was learning things. They weren't really making sense. I wasn't really understanding the neighborhood. And then sort of slowly pieces sort of started falling into place. And I tell my undergrads this about math. I used to say all the time that like it's not right for everybody just for this reason. Like it's kind of a serious cost that you have to spend months, often years at a time, not really getting any external reward. You have to get really good at giving yourself rewards, coming up with benchmarks for yourself. Otherwise it's not going to be fun.
