Well, thank you for the kind introduction. It's such a pleasure to be here and to, I guess, complete the collection of SGP roles, which is funny to think about. Hopefully, can you guys, I think you can probably hear me okay. I'm not sure if this microphone's not working or not, but I'm pretty loud. Excellent. Okay, my goal is to not throw it at anybody here, and I apologize to the people in the front of the room. Right, so I thought today, given that this is, gosh, I think my 16th or 17th SGP at this point, I would try to pull together a bit of a story to share some of the value that working in the geometry processing community has done for our research team, not just narrowly in the kind of application domains that we usually think of as geometry processing, but many others. So, this was a fun opportunity for me to trace through some of our research, starting for me at the beginning of graduate school, and kind of figure out the long, circuitous thread that I think is pretty typical for many of us in the field, and really I think is exemplary of the sort of research that goes on in this particular area, which is a little bit exploratory, a little bit mathematical, automatically oriented and also application oriented all at the same time.
So, when I was going through this talk, it was kind of fun for me nostalgically. I actually dug through and found the website for my first SGP. Mysteriously, the website is actually still live for this particular one. It seems to be kind of hit or miss for the 2010s. But yeah, going all the way back to that conference, we were already thinking about many of the problems that I'll tell you about today. And really, the talk today is partially a little bit of technical content, and partially my kind of love letters of this community, and encouragement to all the other folks that are working in this area to think pretty broadly about the different problems that we're working on. In fact, I was able, there weren't too many photographs where I could find myself in the photo album for this conference. I was busy being a scared graduate student hiding in the back, but I did find one next to Miri. And in fact, I found 15 years later that she and I are still roughly in the same pose working together. And she's been one of many collaborators in this room that I've had the pleasure of working with for, I guess, 15, 16 years at this point.
So, in today's talk, I thought I would tell you about one cycle that has appeared a lot in the research that's happened in our team, and I think also in a lot of other research groups in this particular community, but one that's maybe a little bit unexpected from the perspective of the typical research that goes on in geometry processing. Which is an interesting relationship between, I guess, what you might call low dimensional geometry, which is certainly the focus at SGP, problems that are in two and three dimensions, involve meshes, CAD surfaces, and so on, where essentially we tend to work on very fine grained and detailed, interesting geometry problems that have the advantage of being easy to visualize and understand on a computer screen. And the effect that working on those problems has had on our team's research and many others, on working on, for example, abstract data sets and high dimensional data, where there maybe is geometry. Geometry, for example, this phrase manifold hypothesis you find throughout machine learning research. It's not obvious that data actually forms a manifold, but that's a debate for another day. And that really our team has been able to make an interesting closed cycle between research in these two different domains, despite typically being pretty separate and considered very different parts of computer science. And so I think this is just one sort of nice thread that I'll try to pull through these different research themes that have happened in our team, and I think are pretty typical of geometry processing research more broadly. Spectral geometry is a very popular topic in this community and also has informed the design of graph neural networks and equivariant structures and so on that are very popular today.
So, today I thought I would tell you about a research story that's sort of related to just one computational problem, which is very close to my heart. And I think you can phrase it as follows. This is called the optimal transport problem, and it essentially asks what is the cheapest way to transfer one pile of mass onto another? This is a problem that's very close to my heart, and I'll tell you about all kinds of details and different places where this problem appears computationally throughout the talk. We'll also make it more formal mathematically in a few minutes. And it's also a problem that I think has appeared and reappeared in this community, in the mathematics community, and many others, because it's just a very natural idea and one that folks have studied, starting, I think, in about the 1700s. Now, these days in my job at MIT, the optimal transport problem is particularly familiar. So here is a photograph of my car in Boston a couple months ago. And sadly, I believe this was actually a day when I had to go to the office. MIT is weird about taking pride in not having snow days, which is not great, but the optimal transport problem, and having worked on this domain for 15, 20 years at this point, actually can come in handy if you live in the Boston area, because sadly, in order to access your car, you need to solve roughly exactly the problem I'll tell you about today. So, here, when I woke up in the morning, the initial configuration of snow was what you saw on the left hand side. There were a few feet. Sadly, just getting to my car was an interesting challenge, but I'll save the discussion of that for another day. And essentially, the transport problem asked the following. Which is, I'm given two configurations of mass, like the pile of snow as I observe it on top of my car, and then the distribution of snow that it needs to be like on the sidewalk, or not on the sidewalk in Boston, sorry, like on people's front yards, in order for me to leave for work in the morning. And the transport problem asks, what is the most efficient way for me to take that mass and move it from one configuration into another? And this problem is very natural mathematically. It dates back at least to the 1700s. I think the earliest version of this problem goes back to Gaspard Monge in 1781. Interestingly, the version that he posed, it turned out to be mathematically not quite ill-posed, but a little bit challenging to work on from a differential equation perspective. So we zoom forward about 200 years to Kantorovich, who showed that it looks closer to a linear program if you relax this problem a little bit, namely if you let snow have multiple targets rather than thinking of it as a single map. And from there, the optimal transport community really exploded, both mathematically and computationally. In fact, one of the really interesting and unique parts of working in this particular area is that computation and theoretical mathematics really moved hand in hand. I mean, the history of linear programming and the history of optimal transport theory are very much entangled with one another, and advancements in one have affected advancements in the other, and vice versa. So, from my perspective, I think optimal transport is a fantastic example of the kind of problem that we love to study in geometry processing. In particular, every single part of this problem is geometric and computational in nature. So, for example, the input to the problem are two different geometric objects. There's the domain itself, like, for example, if I'm moving a car, then maybe, so, ah, I'm very jet lagged, I apologize. If I'm moving snow off of my car, there is the geometry of R3, like the space that I'm moving the snow around in, and also the geometry of the mass distribution itself, like where the snow is located. And then, essentially, in order to solve this problem, we need to solve an exceptionally large linear program, which tells me for every particle of mass where it should go in the target in order to satisfy certain supply and demand constraints. And so, if you kind of dig through the typical themes that we care about in geometry processing research, you'll see that almost every single one of them that you see at this conference shows up somewhere in optimal transport research. For example, there's both smooth and discrete formulations, right? There's a discrete linear program version of optimal transport we'll talk about in a bit, and also kind of a smooth one where you're looking for a mapping that moves one distribution onto another. It's an extremely large scale optimization problem whose objective function is certainly kind of geometrically structured in nature. Approximations for this particular problem. And one of the really beautiful things about optimal transport research that I think is so much fun is that essentially, if you take anybody from any corner of math or computation, somehow you can take the language of whatever that field is and project it into this particular area of math. So, people in differential equations have really beautiful theory of optimal transport that connects it to fluid dynamics, in linear programming and optimization. There's a whole theory of duality that's really critical for solving these problems in practice. And so, essentially, with every new collaborator that we've picked up along the way, we've managed to learn entirely new perspectives on this problem that we were completely unaware of when we were just beginning to play with it at the SGP conference, probably in about what, 2011 or 2012. And so I really think that SGP is a natural home for optimal transport research, among research in many other topics, because the big challenges here are exactly the kind of challenges that are tackled by every person in this room, from discretizing geometry problems to solving variational problems numerically, and trying to do so in a way that respects the structure of the continuous problem that lies underneath.
So, in any event, today I thought I would tell you a bit about my path into research into this particular area. And as with most researchers in the field, I think there's this sort of interesting aesthetic that we all try to pretend like we're on this directed march towards some big research result. But the reality is that we all kind of enter backward and unknowingly into different research areas. And certainly that's the case for our team's work in this optimal transport field. So, I thought I actually took the opportunity to dig through some of our old SGP papers and figure out, like, how did we get into this space? How do we figure out which problems were interesting? And then, how does our work on these initial problems inform what we do today now that we've thought about these problems so much more carefully? So, from that perspective, I'll tell you a little bit of the story. I should be careful to acknowledge, of course, that transport is an extremely popular research area. This is just one theme within this extremely broad area of work, including many people in this room that have studied different aspects of this problem. So, this will be a bit of a self-absorbed talk where I tell you about some of the work that our group has done, but I want to be really careful to, and I'll try to signpost along the way, all of the cool work that many of our colleagues in the community have done in this space as well. So, I'll tell you a bit of a story about our work in this domain, where essentially by accident, through an SGP paper in about 2011, 2012, we entered the optimal transport domain by proposing what I would argue is probably one of the harder problems that one might try to solve in this particular language, and then solving it kind of poorly, realizing that, whoa, wait a second, the interesting structure that's hiding in this problem, even when you reduce to one dimension and strip out all of the hard stuff that we were trying to tackle at the time, is still really hard. Working on that for roughly 10 years, returning to the problem that we looked at in SGP maybe a decade ago, and then still not knowing how to solve it. And so, in any event, I think this is a really fantastic example of exactly the kind of research we can do in this field. We choose really tough problems that are sort of application oriented, sort of mathematical, whatever. We don't end up solving them, but we learn a lot of really interesting and valuable things along the way, both practically and theoretically. So let's get started. And I'll tell you, this is definitely sort of a retrospective, story oriented talk. So we'll go a little bit chronologically in terms of the research that we've done in this domain. And our story in this case really does start at the SGP 2010 conference, where at the time Miri was presenting this fantastic work on Killing vector fields. And it was part of this larger effort that I think was very popular in this community in computer graphics to solve problems that I think most of us would call shape correspondence problems. So here's actually a data set that I wasted a lot of time on in graduate school. Here are two brains, these are extracted from MRI in Stanford's psychology department. I don't remember which one is which, but maybe some of you can tell me by looking here. And in any event, what was really popular in geometry at the time, which was a little bit before machine learning was as popular as it is today, was to try and solve problems in correspondence, where I'm given two pieces of geometry, like two 3D scans, here are these two brains, and the task is to extract a mapping from one onto another. And typically, there's a lot of different desiderata that you might put on such a mapping. For example, maybe you want it to be bijective, like you want your map to be invertible. You'd like it to be smooth, differentiable, to maybe preserve geometric features. So, for instance, there's sort of a feeling in this particular field that maybe the folds of your brain kind of geometrically should map onto each other, at least at a very coarse level. I think one of the interesting and neglected part of this problem, at least when we started to study it, was that there's also very obviously a machine learning aspect here that, for example, folds in your brain clearly don't just come from some simple geometric process, and there's probably some data and doctor input that's required to determine this in a meaningful fashion. So, this is just one of these fantastic problems that just fights back. And in geometry processing, I think we started with a really foundational question within the correspondence world, which was the representation of the map itself. Now, I think at SGP, certainly yesterday and moving forward in this conference, one of the typical data structures we see in this community is a triangle mesh. And that's what we're seeing on these nice rainbow brains on the slide here. And one of the basic questions that we were all thinking about and tackling at the time was what does it even mean to compute a map? So, for example, if these two brains happen to have exactly the same number of vertices, then maybe I think of a mapping as a permutation of one set of vertices onto the other. But there's obvious combinatorial issues with that, especially if there's just different resolutions of data or they're sampled in different fashion. And so there's this entire literature that was just beginning to emerge at the time involving different ways to represent a map itself and these sort of indirect and clever ways to go about optimizing for one. And one of these that I was sort of lucky just to be around, but really the person that pushed this area forward was Maks Ovsjanikov, involved thinking about mapping through this idea of what we called functional maps at the time. And this really exploded in its own right as one of the really popular problems to study in this community and others. The basic idea, in case you're unfamiliar, is as follows. Let's say that I have a mapping from one surface onto another. So, in other words, I have some way to take a point on the guy on the left, and for each point on the guy on the left, my computer outputs a point on the surface on the right that corresponds to it. Then the basic observation of functional maps is as follows. Let's say that I take a can of paint and I paint the 3D model on the right hand side. I put this rainbow pattern here. Back in 2010, for some reason, all of our maps were rainbow, which turns out, I think Miri has pointed out, to be a really poor way to illustrate mappings. But for today's talk, it's OK. So let's say that I paint the surface on the right hand side. Then one thing I can do is take the colors on the right and pull them back to color the surface on the left. So for every point on the left hand side, I follow my map to the right hand side, I look at the color of that point, and I copy it back to the left. So for every map from A to B, there is a functional map from B to A that takes colors or temperatures or whatever and pushes them back onto the original surface. And the nice thing is that this object, this thing that takes functions to functions, is a functional map. And it has a lot of really nice properties. It's linear. For example, if your map is an isometry, it commutes with the Laplace operator. And because of that, Max and then many other people in this community proposed mapping algorithms that sort of took what used to be a combinatorial problem and map it into linear algebra. Now, this was a really big idea, and really, actually, Max deserves all the credit in the world for then taking this basic idea and running with it, including all the way till today, and really making this pipeline, which I think all of the original authors can agree kind of didn't work, and making it work in all kinds of robust and interesting mapping settings. But one of the problems that we tackled at the time is that it's really hard to go backward. So, in particular, if I have one of these mapping matrices, I can write it in the Fourier basis or something and ask for every Fourier function on the right hand side, what linear combination of Fourier basis on the left kind of corresponds to that object? And that is basically just a big matrix of coefficients, which begs the question if I give you a big matrix of coefficients, does there exist a map that explains this thing? And the answer is no for the vast majority of these kinds of matrices. And so that led our team to think about what are representations of maps that are kind of adjacent to a functional map, but maybe a little bit more explainable from that perspective. And that led our team, and I should be clear also a team at Princeton. Interestingly, we had soft maps and they had fuzzy maps, and they came out at about the same time. To propose an object that was pretty close to a functional map, but a little bit more interpretable from that perspective. And the idea of a soft map, which we proposed at SGP in 2012, was as follows. Basically, a map takes a point on one surface to a point on another one. This idea of a soft map or a fuzzy map was to take a point on one surface to a probability distribution over the other one. And so the basic idea was that this is a relaxation of the mapping problem. It allowed us to do things like superposition. So, for example, here's a figure from our old paper where I have a map from a circle onto itself. And maybe I'm not sure if it should be the identity map or if it should flip vertically. So, one thing that I can do with a soft map is just superpose both of those on top of one another. And there's a lot of really nice connections between these objects and functional maps. So, for example, by taking expectations, I can kind of pull a functional map out of one of these soft map matrices. But at the same time, by just looking at the probability values, I can figure out the targets of points, which is something that's a little bit more challenging in the functional maps language. So, this is something we were really excited about in 2012. And we thought, aha, this is some nice connection between all this beautiful linear algebra language and the functional maps world and moving these descriptors around and all these things that we were doing at the time to more classical language where we really can guarantee and understand conditions under which we can extract one of these maps. And so we worked very hard with this team, including Miri, Leo, Adrian, and Andy, to come up with different algorithms for optimizing for these things. And pretty soon we ran into a basic problem that shows up time and time again in geometry processing research. Which is how do you measure smoothness of such a map? So, we all want our maps to be differentiable and smooth and somehow elastic in nature. And so, one of the big questions for these soft maps objects was if I give you one of these mapping matrices, is this thing continuous? Is it differentiable? And how do I understand that kind of thing? So, remember the basic representation here, where I have points like x and y on one domain, and now they map to distributions like these little Gaussians on the right hand side. I'm realizing now, I think my student Alice pointed out that my notation was totally inconsistent in these slides, and then I forgot to fix it, so I apologize. But the pictures are what matters here. And essentially, the question you might ask is: is this map smooth or differentiable? So, in other words, as I'm displaced from y, does the mass on the target move very quickly? Does it split apart? And so on. Because if I can measure that, then I have a nice objective function for actually optimizing for a smooth mapping, which was our goal at the time. Well, if you kind of look at this picture and you think about the motivation for this talk, you can probably guess where I'm going, which is that if you think of these piles of mass on the right hand side as like those piles of snow on my car, then this is a pretty reasonable way to measure how the map is changing as I move from one point to the next, as kind of like little piles of snow that are displacing differentially along the target domain. And so there was this natural connection to this transport problem at the time. Although interestingly, we gave it a different name, which was this Earth Mover's Distance. Conveniently, my PhD advisor and our senior collaborator, Leonidas Guibas, was one of many people to discover and rediscover optimal transport in different domains. And at the time, back, I forget when this paper came out, I should have marked it, I'm sorry. They gave it a different name, which was Earth Mover's Distance, which really concretely connects to the snow, I suppose, which was exactly the same as the transport problem. In that case, they were using it to study motion between color histograms to understand whether images were related to one another. In fact, if you kind of dig through the original paper, you can find a very short acknowledgement that, like, oh yeah, this is related to this transport theory. And interestingly, over time, these two communities have merged, the computer vision and mathematical one. So we thought, aha, we've got this mapping problem, and now we have some yardstick for measuring whether one of these soft maps is continuous, where between every kind of pair of adjacent points, I look at their target distributions, I measure how much they're moving around on the target surface, and if that's a relatively small number, then maybe my mapping is smooth. For those of you who are familiar with the Dirichlet energy in classical differential geometry, it's sort of close to that notion, and it sort of inspired exactly that. And so that led us to this technique for computing mappings at the time, where we combined loss functions from the functional maps area, like trying to preserve descriptor, with this notion of smoothness that came from the Earth Mover's Distance, or EMD. And unfortunately for us, the algorithm, I think, I would argue, really did not extract very practical mappings. In fact, it's really credit to this community that they were open minded enough to let us submit and publish this work. When you look at the mappings we were computing, like this thing on the left, you'll notice that it's hardly a map from points on triangle meshes to points on another, but rather like giant Voronoi regions on one to giant Voronoi regions on the other that were not particularly smooth. And you really have to squint at the figures to convince yourself that the loss is doing anything at all. In fact, I'd say the main thing this led to is this truly embarrassing YouTube video that my funder at the time put online without my permission with these 3D pigs falling into my mouth. But that aside, the really big challenge of this functional, or rather, this soft maps research problem was just the scale of the computation that was involved. Because for every pair of adjacent patches on the left surface, we had to solve a different transport problem to measure the smoothness on the right just to get to the objective function that we cared about for computing one of these maps. So, we had all the kind of interesting ingredients, but we didn't have the computational machinery we needed to actually solve these problems in practice. Now, interestingly, that kind of didn't get in our way, and we actually took this problem and made it even more complicated and asked a question which I think is very typical in the geometry processing community, which is what would happen if only we could make these Voronoi patches smaller and smaller? Like, what is the limit of refinement of this problem? Which at the time was purely speculative. Like, we couldn't solve this computationally, but kind of aligning with the aesthetic of this field, we kind of asked if there was some differential version of the same thing. And that led us to this follow up research paper, which I circled Adrian Butscher's name because I think he really deserved credit here for A, working through the math, and B, kind of having the vision to figure out where all these different equations were coming from. Which figured out something that, again, was another clue to the story of the research that we did in this area, which is that if we take these maps and we make those little Voronoi regions smaller and smaller and smaller, eventually what you recover is something that looks like a vector field problem. Another thing that's very familiar in this area. Which kind of does the following thing. If I'm standing at a point on the source domain, so let's say that I have a mapping from this peace sign to whatever this guy is on the right hand side, and I draw a path. Now, remember that in the soft maps language, a path maps to a time varying probability distribution that's moving along the surface. It says where I think points are going. Then, essentially, there's some notion of a derivative of this map, which is like a little vector field telling me, just differentially, if I solve this little transport problem between two very nearby points on the source domain, where should the mass move on the target? And that led us to this optimal transport theory, which basically zoomed our team forward about 200 years mathematically, sadly not computationally. And gave really nice language for characterizing and understanding these mappings. Now, what we came to understand, having worked on this problem, is that we did not have the right machinery to solve it at the time. I'd say Adrian deserves a lot of credit here for kind of writing a research paper that was before its time. Like, if I could rewrite it now, I think we have the right computational tools to actually solve some of these problems, and I'll call them out later. But unfortunately for us, now we have a vector field problem on the cross product of two surfaces with a different optimization that has to happen on every single edge of this mesh. So, if you open this old paper from 2013, you'll see that the computational experiments at the end kind of say, like, well, we have this nice theory, we couldn't make it work, but that's kind of cool, and here's these interesting open problems. And again, that's not to be critical of this research, but rather, again, to highlight that the open mindedness of this community really kept us in this area and led us to work on these problems over the longer term and solve many of the challenges that were presented in this early set of research. So, why do I tell you this story? Well, essentially, working on these mapping problems led us to figure out that the really big challenge here was just measuring the objective function. Just figuring out is one of these soft maps smooth or not? And in fact, that problem, that underlying thing, is exactly this optimal transport problem with a little bit of added structure that informs the particular version of the problem that we ended up studying, which are highlighted on the slide here. First of all, our domains were curved surfaces, so a lot of the algorithms for solving transport problems were very specific to Euclidean space, and that wasn't going to work because we had these triangle meshes. And moreover, we were interested in differential motion, right? Because, like, typically in a soft map, if I move from one very close by point on the source domain to another, mass on the target is not really moving that far. So it's enough to do kind of a nice Taylor series approximation of the problem rather than solving a giant linear program at every single edge of a mesh. And so, this added structure led us to really study the basic optimal transport problem with these additional desiderata, and that led to a lot of new techniques for solving things in this domain.
So now we'll pause for a minute and I'll fill in a little bit more mathematical detail so we all have the right language to talk about these problems, and then we'll kind of go back to our research problem. And in particular, I should tell you a little bit about the mathematics of optimal transport. And the good news is that even though the notation is absolutely horrifying and hard to read, the high level ideas in transport are actually very simple, and I think really do translate those pictures of my car pretty cleanly to mathematical notation. So, here, rather than working in two or three dimensions, let's just work in one. So, here I've got piles of mass along the real numbers. I have like a Gaussian distribution, rho naught, on the left. And I'm trying to measure its displacement to another distribution, rho one, on the right hand side. And one of the really interesting and useful things about transport specifically is if I take rho one and I just slide it along the horizontal axis to the right. And I measure transport distance. I measure how much work it takes for me to transform rho naught into rho one. That scales with the distance that I move rho one along the horizontal axis. Now, intuitively, that makes good sense. If I scoop up snow and I have to walk 10 feet instead of five feet, then the work I did was roughly 2x what it was before. But actually, it's pretty challenging to write down a distance that has that property. So, for instance, if you take a probability course, probably the analog of transport you've encountered is something called KL divergence. And that does not have this property. And instead, it kind of levels off as rho one moves along the horizontal axis. So that's really the motivation mathematically behind transport problem. And actually solving it in practice, or at least in mathematical theory, translates to this notation here. I put an integral on the screen and I just saw like five grad students look at the floor. It's okay. I'm going to walk you through it, I promise. So here's our optimization problem. Our variable is pi of x, y, which is a function saying how much snow did I pick up at point x and drop down at point y. Incidentally, this is called the Kantorovich formulation of this problem because I'm allowing to split snow apart. The original formulation of optimal transport said there has to be one target y for every x, and that's what caused a lot of problems for Monge back in the 1700s. But in any event, in this particular version of the problem, I have three basic constraints on my transport plan pi. The first one is I can't move a negative amount of snow. I think we can all agree on that. And then the second two are marginal constraints. They say that if I look at every target y for a fixed point x, I should get the amount of snow that's piled up at that point x. So, in other words, if I look at my source distribution, all of the mass has to go somewhere. And then similarly, the third constraint here does the opposite thing. It says for every destination y, the amount of snow that ends up piled at y is the sum of all the snow that came from every source x onto y. So that's what our three constraints are doing. And the integral here is really just a sum, it's just totaling up all the amount of snow. Now, there are many different ways to solve transport plans. So, for instance, if I'm trying to move snow off of my car, I could do something efficient, which is certainly what I was trying to do. Or maybe I drop it on Misha's house in Baltimore and then I bring it back to Boston or something like that, which would be satisfying but not very efficient. And because of that, there's room to solve an optimization problem. Namely, the optimal transport problem says over all of the possible feasible maps, I want to find the one that minimizes some notion of cost. And here, cost is going to be the integral of pi against some c of x y, which is typically like distance. So here, the cost is kind of like the expense of moving things from x to y. That's c of x comma y times d pi, which is basically the amount that I decided to move. The kind of weird d pi calculus notation here is a little bit irrelevant for today's talk, but you can think of it as just multiplying the two together. Any questions about that, by the way? Because otherwise it's going to get really boring really fast. Okay. So, essentially, there are a lot of different choices for cost, and many of them are motivated in different corners of physics, geometry, or economics, actually. One of the very typical choices is something called the Wasserstein distance. I could not find a photograph of Wasserstein that did not have sunglasses on. Which specifically chooses the cost of moving mass from x to y as proportional to distance to the pth power. Interestingly, that number p is going to matter more than we anticipated when we worked on that Earth Mover's Distance problem originally. In fact, in that original paper, we had p equals 1, which turns out makes our original paper kind of ill-posed. But by putting that number to a 2, you get a much better posed problem, which is a little bit subtle. And essentially, I think this corresponds to the version that I've illustrated to you guys so far, right? That the cost of moving snow around is basically proportional to distance. I don't want to carry it too far. OK, and the nice thing about this is that the optimal value from this optimization problem really is a distance on the space of probability measures. In fact, it has this nice property that it lifts distances between points to distances between these fuzzy objects. Now, at SGP, optimal transport has appeared time and time again. So, for instance, our work and many others are kind of solving a very basic linear program version of the problem, where you just take that pi, you discretize it, and now you have this big matrix of unknown values. And because of the inequality constraint, what you need is some kind of LP solver. But there are many different versions of this problem that depend on different structures that you might put on it. So, for example, one of the ones that's very popular in this community is what we call semi-discrete optimal transport. So, I think the original paper in SGP on this topic was from Quentin Merigot. Although, I think there was another year or two earlier, there was some work on power diagrams in the computational geometry community. And in that case, it kind of makes an interesting assumption on the form of your problem, namely, A, that your cost function is from Euclidean space. From like Rn. There's an exception to that, which is the work of Fernando de Goes, but we're not going to go into that today. But B, that at least one of your two distributions is discrete. So think about if I had snow on my car and I had five trash cans that I had to choose among, that would be kind of like the version of optimal transport that you solve in the semi-discrete case. And in that case, essentially the object that you end up getting when you optimize is some kind of Voronoi diagram telling you this region of space goes to that trash can. And so these are fantastic solvers for that particular problem. But for the desiderata that I laid out earlier, sadly for us, they weren't relevant. So we had to go back to the drawing board. And so we kind of did this interesting thing. We worked on this mapping problem. We really couldn't solve it. We just kept stripping away parts of this problem until we ended up at the basic one, which is just how to measure this distance. And that led us into some initial work that took that Earth Mover's Distance problem and just came up with a solver for solving it on a triangulated surface. And in this case, this was for solving the one Wasserstein distance, where the cost of moving mass from A to B is proportional to distance. By the way, one of these kind of corresponds to work, and the other one corresponds to energy, we'll come back to later. So, in this case, it's mass times distance rather than mass times distance squared. So, if you kind of work out your units, you can kind of guess like kinetic energy versus work and so on. But in any event, the beautiful thing about the W1 problem specifically, which was this little toehold we had computationally into problems that this community knows how to solve, is that there exists a differential formulation. This goes back to yet another community that has studied transport over the years, which is transportation. I guess it's unsurprising. But if you look at Beckmann's original work, he was interested in road networks and traffic planning, these kinds of things. And what he observed is specifically for the one Wasserstein problem. You can kind of say, well, if I move mass from A to B, the cost of that is kind of like moving mass a little bit, and then a little bit more, and then a little bit more, and so on. So you can take that big motion and break it up into little tiny pieces. Notice that it's specific to W1.
In quadratic optimal transport, the cost of moving something two times the distance is 4x the cost. So now you need to know the entire pairwise source and target for your mass. But specifically for the W1 problem, there exists this beautiful formulation of Beckmann. And we were sort of in the right place at the right time to just take this thing, use finite elements to solve it on a triangle mesh, and get a nice distance function out. So what did we do? We took this nice distance, which was at SIGGRAPH back at the time. Sadly, we plugged it into our mapping problem and it didn't work. And at the time, we just got frustrated. We proposed a bunch of other applications for this, and that's what's in this old paper. And then we asked the question of why. Like, what goes wrong? Why is it that the W1 distance somehow didn't solve the soft mapping problem we set out to solve? And that turns out to be thanks to a bit of a subtlety in the way that this problem is written down. Namely, it turns out for interpolation, the one Wasserstein distance is actually pretty bad.
So if I want the midpoint between two probability distributions, one way that I can get the midpoint between two points is to minimize the sum of distances between those two things. And here's a kind of a useful analogy, I think. Let's say that I have two points on the plane, and I want the midpoint. I can do that in an optimization fashion by finding the point x whose distance squared to the first point plus the distance squared to the second point is minimized. Exercise left to the reader. However, if instead of distance squared, you use just distance, then any point along the line from A to B is equally good, because the optimal cost there is just the length of the line segment. And so, exactly the same thing happens in optimal transport. So, here I've got two one-dimensional distributions. If I compute the midpoint with respect to W1 distance, I get something which is not what I wanted, right? It's like a bimodal distribution instead of something that slid along the horizontal axis. But if I just change that 1 to a 2 in that p-Wasserstein distance, I get what I want.
The only problem is we didn't know how to solve that problem. So, what did we do? We again invented a different problem that we could solve and solved that instead. And that actually led us into our entry point into machine learning because there's a closed-form formula for optimal transport with p equals two, specifically in the one-dimensional case. And so, together with Raif Rustamov, we proposed a kind of fun machine learning algorithm that was trying to do things like guess histograms of temperatures along a map of the United States. Notice temperature is a one-dimensional quantity, so we could actually solve this problem. To statistical machinery, where once again we set out to solve our soft maps SGP problem. We were unable to do it, but we ended up with yet another interesting piece of research, in this case, in a completely different domain.
Now here's where I think our story is kind of interesting. Because of this research paper, I had the opportunity to go to the International Conference on Machine Learning, ICML, in Beijing. Now, optimal transport these days at ICML is like, as with almost any topic in machine learning, like an entire conference hall with like 10,000 people in it. But at the time, there were two of us. There was me and one other guy named Marco Cuturi, and we had posters next to each other. And Marco and I really bonded at this conference as the two transport guys. He was working in Japan at the time, and he was kind enough to invite me to visit his institution in Kyoto as a grad student. And so, thanks to this chance encounter with Marco, this actually led to a really popular research paper in computer graphics that we all use today for transport problems.
Where essentially, Marco at the time had published kind of impressively this single-author research paper on a state-of-the-art new algorithm of machine learning for computing optimal transport distances that he called Sinkhorn distances, where essentially you take that transport matrix and you add a term to the optimization problem, namely entropy. And then it turns out there's like this three-lines-of-code algorithm you can use to solve for an approximation of transport very fast. Now, at the time, if you look at that matrix, essentially the matrix that he has to manipulate in the algorithm in Sinkhorn, what you'll see is that every element of that matrix looks like e to the minus distance squared, which to my geometry processing brain was like, aha, I know what to do with this thing. Because in particular, in computer graphics at the time was very popular this new algorithm for computing geodesics on surfaces called the heat method from Keenan Crane, Clarisse Weischedel, and Max Wardetzky, which essentially noticed that the heat kernel on a surface, like this diffusion object that we all know and love, that comes from the Laplace operator. If you take the log of that thing, it is exactly distance squared up to a constant.
And so, what did we do? Well, essentially, actually, while hanging out in Marco's lab just for a week, I took his MATLAB code for solving this Sinkhorn thing. I dropped in Keenan's code for solving heat kernels, basically, replaced his matrix multiply with our implicit kind of solve there. That changed the order of magnitude of the efficiency for the transport solver. And that's what led us to this technique for transporting quantities both along triangle meshes and in Rn that we called convolutional Wasserstein at the time. Incidentally, we called it that because convolutional neural networks were becoming popular. And we thought, oh, maybe we can benefit a bit from the keyword search, which was totally the case. But in any event, this algorithm really, this research paper is like 10, 15 pages long because that's the aesthetic at SIGGRAPH. But really, it all fits into this box of about four lines of code, which just took Marco's algorithm, dropped in the heat kernel where he had a matrix that he had to construct, and everything else is basically left exactly the same.
And what this leads to is a very efficient way to approximate transport distances on a mesh, which is cooked out of ingredients that we all know and love in this community. And indeed, this work really took off and has been adopted in a lot of interesting situations I'll mention shortly. Now, I should put a little bit of a pin. Unfortunately for us, there is a cost to using this technique, which is that it works better and better and better as I diffuse more and more. And when I say better, I mean it converges more quickly. But that's also exactly the case where we regularize our problem and move farther and farther away from the actual transport problem we wanted to solve. So once again, we took our problem, we changed it, we didn't solve the thing that we wanted, and it works better as we move farther from the thing that we started with.
I should note, incidentally, that many years later, with my student Sebastian and also Hugo and Ed, finally, many years later, we were able to tackle the problem without regularization on meshes using this kind of nice structure-preserving version. It required all kinds of interesting ideas from fluid dynamics and so on. There's some really beautiful work from Hugo that actually manages to discretize this thing, but sadly, it's still quite expensive computationally. There's still a lot of interesting open problems in this domain. And I should note, by the way, just as I'm trying to put pins in different places where research in our field has affected others, Hugo then took his work on the SIGGRAPH Asia paper, and it led to all kinds of interesting work in theoretical mathematics that was analyzing the soft maps problem, figuring out when it was well-posed, and even worked out the convergence of the finite element scheme that he was using to the true mathematical solution. There's something that at least my research is a bit irresponsible about.
So at this point, we had a useful transport solver in 3D and 2D. And moreover, we had a bunch of applications for this machinery, including this one kind of funny one in machine learning, but not an application back to the original soft mapping problem we wanted to solve. And a really cool thing happened, which is all kinds of people picked up these algorithms and they used them for all kinds of interesting domains. Not just geometry processing and animation, but also domain adaptation, statistics, gradient flow PDE, and many other different problems in the computational world. And so I thought for fun, I would highlight just a few of the applications that some of the students in our team worked on, because essentially we had this new solver for transport problems that fit in like four lines of code, and it was pretty easy to just play with different settings. One of my absolute favorites is we had an intern, Theia, and what she worked on was actually making a piece of DJ equipment that incorporated a transport problem.
This was, I cannot say that we anticipated that when we moved into this particular field, but the basic idea was if you have two audio clips and you want to transition from one into the other, typically if you work on a piece of audio equipment and you use one of these fader things, then it just kind of like fades from one audio clip to another. With transport, you can do it in a way where the pitches actually move around. In parallel with this, we're working on political redistricting, which is a very touchy topic in the United States, continues to be today. And one of the big questions in that domain is how do you compare districting plans? And we had some interns on our team that worked on actually putting a metric on the space of political redistricting plans, like here's ones in Arkansas, based on transport distances, where the cost of the transport distance is itself a transport distance.
We also had one of the early architecture for machine learning on triangle meshes with Miri's student Danielle and Vova Kim, which I think predated MeshCNN and some of these other techniques. Where essentially, actually, I think Danielle gets all the credit in the world for not only implementing all these transport problems, but differentiating through them in order to make a machine learning pipeline, which was particularly challenging at the time because a lot of the differentiable programming tools we have now were not available. Now in parallel with all this work that we were doing, beyond just these two and three-dimensional problems, everybody was becoming excited about machine learning and big data, data science, and so on. And indeed, one of the things that you see in that domain is that we typically use geometric language to describe things like data sets, collections of data sets, Bayesian priors, and so on. For instance, oftentimes we think about them as supported on low-dimensional manifolds and so on.
And so that led the entire transport community to shift its focus a bit from problems in image processing and meshes to problems of very high-dimensional statistics and other domains. And in fact, that's a pretty reasonable transition. So here's one illustration why. Suppose I want to use a computer to do a problem in generative modeling. Like I want to generate photographs of cats. How do I do that? Well, I can go on the internet, I download 10 million photographs of cats. They're actually mostly on this laptop. And now I have my AI model, which I'm going to denote as P theta. Where theta is like the parameters of my neural network, right? So typically, a model for generating an image maybe starts with a Gaussian random variable and then does all kinds of crazy things to reshape it until it eventually makes something more interesting than that.
And one way that I can model this problem is to say, there's the distribution of things that my computer knows how to model. So, in other words, for a particular set of parameters for my neural network, if I just take the entire Gaussian and I push it through this network, then kind of like what is the space of images that I get? Then there's the distribution of images that I downloaded from the internet, and I want those two things to be as close as possible to one another, which is very natural language for optimal transport. In this case, for minimizing transport distance between the distribution of stuff my computer makes and the distribution of stuff that I've collected from the internet. And so, this basic observation led the transport community to connect this toolbox to problems in machine learning involving modeling, measuring distances between distributions, and understanding how well a statistical model fits a type of data.
In fact, my colleague Marco coined this goofy term, where essentially there was this trend in the machine learning community in the early 2010s to just take every problem we could think of in statistics and then just stick a Wasserstein distance somewhere in there and show that it got a little bit better. Now, sadly for us, that wasn't all, it wasn't as easy as just taking this nice convolutional transport distance and then dropping it into this new set of applications, but rather an entirely new set of problems arises when you begin to study transport in the statistical sense. In particular, there's one really famous mathematical expression I've given you on the right-hand side, which has caused us now about 20 years' worth of trouble. Which is the fact that as the dimensionality of your transport problem increases, the number of samples that you need from your distribution in order to estimate a transport distance accurately exponentiates.
And because of that, all these algorithms that we're working on were pretty efficient. That wasn't the issue. The issue was that the size of the transport problem that you needed to solve was way bigger than even what you could do if your algorithm ran in linear time. And so, because of that, we had to kind of tackle all kinds of challenges at the intersection of algorithmic complexity, which is kind of what we were thinking about on meshes, now coupled with sample complexity and statistical complexity, which has to do with how big your data set needs to be before you can estimate these quantities that you care about. So, once again, this transport problem, which began with just trying to make Max's functional maps thing have basically an inequality constraint, has led us to down this completely new path involving sampling, probability, statistics, and so on.
And in fact, this really led to all kinds of interesting ideas at the intersection of geometry processing and statistics. And in fact, if you kind of look at every transport problem paper that you see in SGP and SIGGRAPH and so on, you can kind of map them pretty cleanly onto a lot of the statistical methods that you see in machine learning settings. For example, using entropic barycenters, like this stuff that we've done in graphics to solve problems in Bayesian inference. Using symmetry detection as a way to kind of think about problems involving quotienting out symmetries for problems like inferring mixtures of Gaussians and so on, where you can like swap the labels around. And so we had a lot of mileage over the years of basically taking this nice intuition that we developed over many years of working on this problem in 2D and 3D and just mapping those problems onto these high-dimensional problems that people were really struggling with in the statistical world.
Now, I should pause here to note, unfortunately for us, this idea of Wassersteinization also fell apart a little bit. And in fact, we have a paper with some friends in Moscow who, at the time, were kind of interested in benchmarking different algorithms in machine learning. And they asked a question whose answer was totally disappointing to me, which said, OK, we're really good at solving transport problems, and we can keep doing this. I have a lot of fun working on these problems. I will happily write more transport solver papers. And the intuition at the time was, ah, if I make my optimal transport solver better, then hopefully my generative AI tool will get better as a result. Sadly, what Alex and his colleagues found was exactly the opposite of that, and I think is an interesting kind of direction for future research, which is sadly, as we made the transport solvers better in high dimensions by increasing the statistical efficiency of the tools, using better representations, and so on, the things that resulted actually degraded.
And so I think this is a really interesting case where somehow using transport has led us into this interesting world where it's clear that geometry has a big effect on high-dimensional inference, but it's not clear exactly how. Even though, for example, optimal transport was used to describe the early architectures, for example, for generative adversarial networks like this Wasserstein GAN, it's not clear that the Wasserstein part of that problem is really the thing that makes it work. That aside, because of that issue in part, our team moved on not only from working on transport to other problems in generative AI. And I just wanted to point out that once again, working on this sort of low-dimensional problem in geometry processing led us to study problems that have nothing to do with geometry processing or even with optimal transport, including things like diffusion models and even places where numerical linear algebra might appear in the worlds of large language models and so on, just in the last couple of years.
And so that sort of led our team to where we are today. Essentially, we started in these very low two- and three-dimensional problems where we were inspired by trying to compute maps between surfaces. And now we're working on these problems that are including like political redistricting, MCMC, sampling, Bayesian inference, and so on, all of which fall into sort of a similar language and intuition. So, I've only got a couple minutes left. I thought I would kind of conclude by some of the lessons that we learned working on high-dimensional problems that have now influenced what we do on the low-dimensional geometry side, and then maybe conclude with some high-level thoughts of just what this story tells us about geometry processing research.
For one, I should note that we're continuing to work on that basic optimal transport problem. In fact, just this summer, we'll have a paper that I think you can understand as a follow-up to convolutional Wasserstein distances roughly a decade ago that notices that one of the formulas we were using in that old paper is completely unnecessary. This is a little bit embarrassing to me, but essentially, in that paper, we were solving interpolation problems using barycenters, which involved kind of recomputing a big optimization problem for every frame in your animation. One thing that I noticed last summer was actually if you look at the dual variables to exactly that optimization problem, you can just read off the optimal solution for interpolation, which led to a thousand pages of math notes that my students were kind enough to translate to this cool paper here, showing that you can solve exactly the same problem orders of magnitude faster by just reusing the variables we already had around.
A different lesson that we've learned that's affected a lot of the research that we do today in low-dimensional geometry is the value of convex analysis, and in particular, this notion of duality. So, if you look at the mathematics of this convolutional Wasserstein idea or of entropic optimal transport, basically it all comes from this idea of kind of convex duality. That if you look at an optimization problem in terms of its Lagrange multipliers, oftentimes you can reveal really beautiful structure that maybe was hiding and you don't see when you write down the original thing. On this spree of identifying every possible convex problem we can find in the graphics literature and looking at its dual just in case it has some cool structure. So, for example, two years ago, Oded and I had this paper that took Alex's work on skinning and showed that it has a nice convex dual that leads to a fun optimization algorithm that's not terribly obvious from the primal version of the problem.
Also, our work in this space has led to some really interesting and intriguing work that I don't know how to describe to this audience. So, for example, in optimal transport, I don't mean... All right. In particular, in the optimal transport community, one of the really interesting developments in machine learning is that they took these optimization problems, which are convex and beautifully structured, and they solved them with neural networks. And if you think about it, that really makes no sense at all. You took your problem, and you use this representation, which is wildly non-convex in its inputs, and all the nice structure that you had in your problem has fallen apart. And yet, these neural optimization solvers for transport are extremely successful.
And that's an idea that we've kind of ported back into our work in this field, including problems in computing minimal surfaces, barycentric coordinates, skinning weights, and so on, of using non-convex function representations to solve convex problems in graphics with great effect and with very high resolution. We found that the optimization techniques that appear in the machine learning literature are also very valuable for geometry processing problems. For example, my student Yu found that this Adam optimizer, which is popular for training neural networks, is also quite useful for solving problems in surface parameterization and for skinning. But interestingly, only if you kind of rephrase your problem in a way where the structure kind of looks similar to what you might see in a neural network. Namely, he kind of does a change of variables where like changing any one variable in your problem has this big global effect on your optimization results, which seems to be really critical in this domain.
And we have new tools for inverse problems. So, for example, this summer, Ana and Ahmed will be presenting work on a whole differentiable library. I guess it was also shared at the tutorials two days ago for solving inverse problems built on this machine learning community and toolkit that became popular for automatic differentiation and is allowing graduate students today to differentiate through functions that I think we would never have the audacity to differentiate through maybe 10 years ago when we were working on these problems to begin with. So, in any event, as always, I'm talking far too much. But hopefully, you guys get the basic idea of this story. We started with a problem at SGP. We were unable to solve it.
But working on these kind of speculative mathematical problems led us to think about all kinds of problems in different domains, keeping an open eye to applications in not just 3D graphics, but also in statistics, in audio, in political redistricting, and so on. And in each case, work on these problems really built up both intuition and a computational toolbox that was valuable far outside of the original space of problems that we were thinking about. So, from that perspective, I think there's a lot of mutual benefits that we can have from this community moving, looking kind of outward to computational domains that we might not conventionally think about as geometry processing. For example, to the outside world, we offer tools like thoughtful discretization, structure preservation, and the ability to visualize and think about really hard and complex geometry problems that would be hard to do in high dimension.
And from the outside world, we can bring in a lot of really good ideas to geometry processing research, from scalable solvers to being able to solve problems in geometry where you have messy data and things don't satisfy the nice assumptions that we might want. There are so many interesting frontiers that we can go to from here. Everything from developing interesting neural solvers for these geometry processing problems and explaining why the heck these things should work so well. I think that second one is actually a very interesting one, even for mathematicians that are a little bit allergic to machine learning research, to ask this really fundamental question of somehow taking these beautiful convex structured problems that we care about in geometry research, removing the convexity, and then solving them with automatic differentiation and machine learning style tools.
Seems to be a pretty successful pipeline, but we don't really have an explanation why. Or basic numerical analysis style language for describing, like, if I have a neural network that solves the Laplace equation with a thousand neurons or a mesh with a thousand vertices, which one of these two things do I expect to perform better for that class of problems? Still, there are open problems in the optimal transport world. For example, this interpolation problem that we started with is still largely unsolved unless you add entropy to your unknown variables. And essentially, we can begin to use this as an inspiration to develop an entire geometry processing toolkit that moves beyond two and three dimensions to data manifolds and all kinds of other domains.
So, in any event, essentially, hopefully, you guys get the high-level story that I'm trying to share with you today. That essentially, there's this big unexpected ecosystem that our team and many others have encountered. That essentially, there's a nice virtuous cycle where both research in two- and three-dimensional kind of meshes and geometry processing can inform high-dimensional research in statistics, inference, and so on, and vice versa. And I think this is some really interesting inspiration for work in this particular field. Moving forward, for example, at SGP, we can think a lot about how to bring in all these other communities, the perspectives, the kind of problems that they study to keep the research that we're doing in geometry processing vibrant and interesting as we have solved all the basic and cool stuff in our particular domain.
Now, to conclude, of course, I should thank many of the people that have done this work. My job as a PI at MIT is just to basically promote all the interesting work that our students are doing. I took the union of all of the co-authors that we've had on these many papers, and sadly, it's not a very useful way to acknowledge the fantastic people that we've had the pleasure of working with over the years. But really, most of the credit goes to the fantastic students and postdocs that we have at MIT. And I encourage every person in this room to come by Boston and visit and bring in their own interesting geometry problems for us to study. In particular, I want to highlight this fun image of our team trying to assemble easels for a conference and completely failing, which is always the highlight of my year as a geometry professor.
In any event, I really thank you all for the time, and hopefully, this is a nice kind of story to share the kind of effect that research in this field can have, not just on tools and applications in maybe computer-aided design and graphics, but across a wide variety of different topics in domains that we really wouldn't have anticipated when we were studying soft maps 10, 15 years ago. And I think this is really an interesting and useful story that we should all carry forward moving into our own domains to think about where the work that we can do can affect all kinds of fields that we might not have thought about originally. So, thank you very much for your time. It's, of course, my pleasure to answer any questions, or I don't want to get in the way of your coffee break, too. But thank you very much.
