Oh, thank you. Well, it's a pleasure to see everybody and to speak to you all today. I'm coming to you guys from the computer science department at MIT, and really my main goal hanging out for the week and next week is to learn all the interesting problems that everybody's working on and to hear back all the cool things that we should be working on. As I think probably the most applied person in this workshop, I kind of leaned into that today, and I'm going to show you a bunch of empirical results that we need your help explaining, and then directions to go from there for a particular set of problems in our domain of geometry processing.
So, in case I haven't had the chance to meet you guys in general, I run a big applied geometry group at MIT, and really our goal is to find all kinds of cool problems in numerical methods and pure geometry research and so on, and then try to find interesting applications for them. That's sort of what we're in the business of doing. So, essentially, if you've got interesting hammers and you're seeking nails, it's always a pleasure to learn about that kind of stuff, and please come chat.
So, essentially from a high level, our research group kind of follows this weird philosophy that glues together, I think, two areas that typically are sort of in two different parts of a computer science department. On the one hand, my personal background back in the day was in differential geometry and PDE through the kind of computer animation world and working in physical simulation. And because of that, we worked a lot in, you know, working on geometric data in the sort of conventional sense of like 3D models, meshing, and so on.
A kind of funny thing happened with my own PhD thesis, which is that I was working in optimal transport, which at the time we thought was going to be a really cool numerical tool for computer animation. In retrospect, that was a terrible idea. It turns out that optimal transport, the whole point of that problem is to be as uninteresting as possible, and that's not really what animators are looking for. But it did turn out to be a useful tool in high-dimensional kind of inference-style problems in probability, and that led to this kind of funny structure that we have now with a research group that's sort of 50% low-dimensional geometry, finite element-style problems on meshes, and then the other half of the group is doing kind of high-dimensional machine learning-style problems. And so today's talk, ostensibly, is in that second category, but actually draws a lot of stuff, a lot of ideas from the first. And essentially what I'm going to try to show, which I think is not all that surprising in this community now that people are excited about things like physically informed neural networks and so on, is that there's all kinds of bizarre discretization techniques for geometry problems, variational problems, and so on that I think can be borrowed from this other world of thinking about kind of high-dimensional data that actually sort of suggests interesting tools in this low-dimensional setting.
And so I'm going to show you kind of lots of fun pictures and very few theorems, mostly because I don't have any, to kind of motivate that we can kind of go back and reconsider, I think, a lot of the pieces of the computational pipeline that we've taken for granted certainly in the computer graphics community, and kind of ask like how much of this is just because this is how we know how to solve these problems versus what's truly necessary to get the best possible solution to the problems that we care about.
As I feel again, obviously there's a big like kind of inferiority complex happening here. I feel like I need to keep disclaiming things. This is going to be mostly empirical and very tenuously related to spectral geometry, but I do think that the numerical methods that we're doing here are closely linked to some of the things that show up in that world, and I'm happy to speculate that a bit. In fact, I guess next week there's this workshop on AI and spectral geometry, and I'll try to make that connection a little more carefully.
So in case you're uninitiated, I come from this world of research called geometry processing. In case that isn't confusing enough for you, apparently computational geometry and geometry processing is like one of these funny academia things where these are two fields that I think to any outsider would sound like synonyms, but of course are like, you know, the Jets and the Sharks kind of situation and don't communicate that much. But in the geometry processing world, I think this sort of dominant technique instead of algorithms involves borrowing ideas from partial differential equations and gradient flows and so on and trying to adapt them to practical problems that we face in computer graphics.
So here's a screenshot from a really classic paper back in 1999, which essentially showed that mean curvature flow, which of course is a thing that's been around in mathematics, is extremely useful in a practical setting for like smoothing the 3D surfaces out. So here they have a 3D scan of someone's face. They've done a time step or two of mean curvature flow, and all the noise from the 3D scan is removed. Obviously, for every one of these positive examples in the '90s, I think there were a bunch of examples that didn't work, but this community has really grown and expanded and come up with really beautiful algorithms that are not only adapting these interesting ideas from the sort of pure theory, but also we're trying to make them robust to messy data that we tend to encounter in the wild. The reality is for us that our audience is often people like artists and 3D modelers who couldn't care less about the fact that we're doing, you know, a gradient flow on surface area. They just want to smooth out this 3D surface. And so, for example, like mean curvature flow is well known to develop all kinds of strange singularities, which are fun for mathematicians to study and just really awful for artists. And so there's all kinds of interesting sort of interplay between mathematical modeling and feedback from people that really don't understand mathematics at all to lead to the kind of algorithms that we're studying.
Of course, behind the scenes, as I think we've now seen in almost every talk in this workshop so far, there's some kind of machinery, oftentimes finite elements. And I think the really kind of interesting bias in this particular corner of computation is to use extremely low-order finite elements. In fact, I think typically, you know, for example, I teach a course in this stuff, and we really don't move beyond just piecewise linear elements on triangulated surfaces, even when you're solving kind of high-order problems, which is a little surprising. So we often work pretty hard to contort our problems into some form that is amenable or even just almost amenable to this kind of discretization. And there are some reasons for that that are pretty rational. I mean, typically we're solving these problems on curved surfaces rather than on flat domains, and sometimes the only access we have to that surface is a triangulated model because that's what the artist made. And so it's actually not obvious that the high-order methods and some of these other cool discretization techniques are that relevant, would lead to that much more accuracy because we just don't have any more information than what we're presented with here.
To kind of move from one extreme to the other, on the one side, I would say that the vast majority of research in geometry processing really does use piecewise linear finite elements on a triangular domain.
You might ask, well, what could we do in the sort of wildly nonlinear universe? And in particular, these days, what is, you know, I think maybe the pessimistic way to put it is like what is trendy, but also I think is to put it a little bit more positively, people have had all of these really successful experiments in so many different domains using architectures that are borrowed from machine learning and neural networks, that there's some really interesting questions to be asked just from a pure numerical perspective without any data at all that I think are worth revisiting in this particular domain.
In particular, I promise that this is the first and last time you will see this horrible diagram here. After having been forced to teach undergrad machine learning a few times, I've come to the conclusion that this is just the world's worst illustration of a matrix multiply. But I think essentially the question is, as we move out of the linear regime, both in terms of the individual elements that we're working with and the fact that our function space is itself linear, right? That like we have these coefficients a_k that we're adjusting to basically this wildly nonlinear regime, we lose a lot of structure, but we also gain some structure that people view as very favorable in the machine learning community and in some cases translates cleanly to the problems that we're studying as well. So I thought I would spend a few minutes kind of giving some of my usual high-level intuition for why this is an interesting thing to study and then show you some examples of research projects from our team that are able, at least empirically, to show that this is a pretty successful strategy, especially when you're worried about problems that are concerned not only with solving a numerical thing to accuracy, but more worried with concerns about robustness, working with really messy domains, and so on.
In case you were unfamiliar, here's machine learning in one lecture slide. Essentially, I think one way that you can understand a lot of the modern techniques is that they're essentially fitting functions of a very particular form. This thing is called the multi-layer perceptron for funny historical reasons that we're stuck with now. But essentially, I think you can think of functions in many machine learning pipelines, almost all in some sense that are popular today, as following a very simple formula in terms of how you actually parameterize the unknown thing. Right? So if we go back to the sort of typical parameterization in finite element, maybe I have a bunch of basis functions psi_k, and then the knobs that I get to control are just coefficients in front of those basis functions. What happens in an MLP is a little bit different. So what goes into your MLP, I borrowed this slide from my machine learning course, so it's a feature vector, but really it's just a point. And I'm trying to parameterize some unknown function on R^d.
So essentially, what these function parameterizations do is just alternate between two different steps. The first thing that you do is apply something linear, so you have some matrix W and some bias or some shift b. And then to every element of the output of that linear map, you apply a nonlinearity, which is noted as sigma here. Oftentimes the most popular one is this thing called ReLU, which is just a fancy phrase for the max of x, 0. Although in this particular talk, that's actually not going to be sufficient because we're going to be solving differential equation problems, and ReLU poses a problem, which is that it's only one time differentiable. And so we're going to need to use smoother activation functions. But there's kind of a beautiful theory out there that says that so long as sigma is not a polynomial and kind of not stupid, it turns out that if I create functions of this form for W big enough and enough layers, I can approximate things in many different function spaces pretty well. So actually, from like sort of even a numerical analysis perspective, at least from approximation quality, somewhere out there is oftentimes a good set of Ws and Bs that solve my problem.
The only issue is that they're hiding inside of this nasty function sigma, which is often composed with itself many times, and so all of the beautiful linear algebra algorithms that we're comfortable with no longer apply. And so we kind of have to revisit the computational pipeline, which we've seen in many different talks and techniques in the last maybe 10 years. But there is some possible benefit to be gained.
One thing I really want to emphasize in this talk is that I think originally when we saw these things, especially I feel like in this kind of community, including my own, there was like sort of this eye roll that happens at this juncture where you say like, 'Ah, neural network, this is just trendy machine learning stuff, whatever.' But I think really what that community has done is made a really cool technological innovation that I think has kind of changed what programming looks like for a lot of numerical problems, completely divorced from data, which is that basically there's sort of this audacity of computing derivatives of functions that would be wildly complicated to do by hand. And I really mean that. Like in particular, when you hear about algorithms like backpropagation and so on, all that thing is doing is taking some function which is written as a piece of software, you know, like in this case composing this together in code that looks like MATLAB, and then giving you the derivative of some expression in terms of the Ws and the Bs, which is arbitrarily complicated, and just being able to evaluate it at different points with 100% accuracy up to numerical precision. And that's a really cool technology. So I think actually it's really changed the way, for example, folks in the applied side are solving modeling problems because you cooking up really crazy objective functions and classes and like experimenting with different classes of functions that you don't know which one is going to be relevant to your particular setting used to be really hard, right? I mean, when I was in grad school, you know, every time I want to switch from like a linear basis to a quadratic one, you know, I had to go back, pull out, you know, a notebook, do all of my integrals by hand, do all the finite element stuff, and so on. Now that's just not necessary, and so you can think a little more about modeling, a little bit less about all this stuff that comes downstream.
So I think even if you kind of put aside, you know, all the fancy language like backpropagation, multilayer perceptron, really I think you can think of modern machine learning software as just a giant derivative machine. And from that perspective, there's a lot you can do and a lot that that enables.
So for example, see, I tried to be relevant to spectral geometry. Let's say that I want to solve Laplace's equation. Here might be one kind of strategy for doing it in this new kind of modern crazy universe that we're in. So if I want to solve Laplace f equals g, as we all know, that's equivalent up to the inevitable sign mistake that I'm sure I make, to some optimization or variational problem like what I've written on the first line here.
Essentially, in the world of machine learning and stochastic gradient descent, I can take all of our integrals that we're used to writing in the first line and just use different notation. I can think of that as an expectation up to, of course, a constant of just some x that's drawn from the uniform distribution over my domain omega of the same quantity on the inside. And then what we're going to do is make a small approximation. That is, rather than working in the class of all functions f, I'm going to specifically choose f to be a neural network. Right? But here's the kind of interesting thing that happens in this domain, whereas from here, like what we normally would do is work out our mass matrices, stiffness matrices, and so on. We can't do that anymore because now these parameters theta are hiding in all kinds of weird nonlinearities.
But what we can do is compute the gradients of this expression with respect to x and with respect to theta very easily, even if this is very complicated in terms of the elements of theta. And when I say easily, I don't mean by hand. Like the expression is really ugly, but behind the scenes the software is very efficient for doing that. And so that leads to a different class of algorithms. So, for example, if I want to do stochastic gradient descent, that isn't just an algorithm for, you know, optimizing loss functions in machine learning, it can also be used to solve this kind of problem. Right? Because what I can do is a Monte Carlo estimate of this integral, this expectation here, by just randomly drawing x. And then now this thing becomes a finite sum. This is a thing that is easily differentiable in the thetas. I can take a gradient step and I can iterate. And all of the beautiful guarantees from stochastic gradient descent in machine learning also apply to this case. Like there's nothing different. And so essentially what I'm arguing is that this same technology that has been developed in this world of, you know, randomly subsampling a data set because data sets are really large, also applies to a lot of our problems just because they're giant integrals. And we can borrow a lot of the technology and optimization procedures that are popular in that community.
Incidentally, I should also note and mention that this is an unbiased gradient estimator. So you can actually, you know, if f is expressive enough, you actually kind of expect the gradients with respect to theta, if you sample enough x's, to truly kind of move you in the right direction, which is kind of cool. Up to like so many different disclaimers that I'm totally ignoring right now.
So, like why would we, why might you want to do that? I think if you have to compare these things, I think there's a lot of different ways you could put this into a framework. On the right-hand side, I'm showing you a very typical image. I think this is actually the only AI-generated image in this talk, because I just didn't feel like writing code last night. Which is as follows. Essentially, I could take a multi-layer perceptron, which again is just a function of that form. It's like five lines of code in MATLAB. It's really not that complicated. And I could randomly draw all those matrices W and B. And let's think of a multi-layer perceptron that takes as input a point in R2 and then outputs a number in R1. And the intermediate embeddings could be in all kinds of weird high-dimensional spaces, whatever, but at the end of the day, it's just a parameterization of a function on the plane.
And here's kind of a cool experiment. I randomly drew those, up to trusting AI-generated code, but like theoretically expect this to happen. I randomly draw those matrices W from just a uniform Gaussian distribution and do nothing to it. I don't optimize it whatsoever. And then I plot the function on the plane. I believe here the activation is ReLU. And I think the obvious thing to notice here is that this is a smooth function. Right? And that's not a mistake. In fact, people talk a lot about the fact that sort of the bias of these neural networks as architectures is towards smooth objects. And you can contrast that really strongly with finite element, where in that hat basis, if I adjust any one of those parameters, I'm making a very high-frequency change to my function. Right? And I think in some sense, that's a good intuition for why maybe gradient descent procedures work okay in this space, because there's a lot of redundancy and a lot of kind of bias towards smoothness, which tends to happen in our universe. Whereas, you know, if I did stochastic gradient descent on just like a vanilla formulation of a finite element problem, I think you'd probably end up with a very noisy result. I should have coded that, but I didn't.
So that's the sort of comparison that I have. I mean, certainly for like very smooth elliptic problems, I mean, you know, you might expect like, 'Oh, well, even a randomly generated neural network is already pretty close.' So it's not all that surprising that maybe these tools, at least empirically, are performing a little better than one might guess just like looking at the sort of AI hype that's out there.
Moreover, something that we were just talking about last night over dinner is that there's some really interesting theory that's emerging in the machine learning community that tries to explain that there's certain phenomena out there that arise from this wildly nonlinear universe and put you back in sort of linear-looking domain. So in particular, I just wanted to draw to everybody's attention in this community. I don't have actually a whole lot of results or comments about this space, but I think it's a really cool one and one that's relevant to the discussion here. This literature on this object called the neural tangent kernel, which arises from this interesting empirical phenomenon, which does the following. So here this animation is showing a bunch of randomly initialized neural networks. So each pink curve is a different random initialization. And then you do gradient descent, in this case on a very simple kind of regression-style problem. And here's the kind of interesting observation. In terms of those matrices W and those biases B, the process of gradient descent on this problem is wild. Like it goes all over the place. It's very stochastic. It's hard to compare these things to each other. In particular, there's a symmetry group that operates on neural network neurons, which is really nasty. Right? Like if I swap the indices of two neurons, that doesn't change the function, which means that there's like a whole permutation group hiding worth of redundancy in there.
But if I plot the function that is parameterized by these different initialized things as it undergoes this gradient descent procedure, the kind of interesting empirical takeaway is that that process of gradient descent in function space, not in parameter space, actually often looks very self-similar among different initial guesses. And so there's these beautiful statistical physics-style arguments out there that sort of say, 'Well, in some limit as I increase the number of neurons or layers in my network,' that community has kind of messed with different ways to take that limit. This starts to look more like a gradient flow in a very particularly carefully designed kernel space that comes from the architecture of your network rather than this really nasty nonlinear kind of combination of ReLUs and all this other stuff.
Which I think is a really interesting little toehold into like, 'Oh, wait a second, maybe there is linear algebra we can do.' You know, in particular, now we're back into the world of like Hilbert spaces, which is something that we know about and are comfortable with. Just kind of with a little bit of glitter on the outside coming from these crazy MLP kind of architectures.
Now, the open problem I would share with all of you guys, as I see there, I mean, there's all kinds of literature and excitement in this space specifically for kind of explaining training procedure for neural networks. I don't see a whole lot kind of links to problems whose objectives involve like the derivatives of an unknown function, where now you're moving into kind of like Sobolev space territory instead of L2. And I think there's some really interesting kind of intersections to be had there.
So anyway, there's all kinds of interesting open problems in this space. I guess I haven't actually told you any of our research. I'm just kind of like vibing at the moment, but I promise I'll give you something more concrete in a second. But I think there's so many fun problems for this community to study, and that's why I thought I would come and evangelize a little bit today and also kind of defend the kind of eye-roll reaction that you often get to this machine learning community. That essentially, I mean, there's so many problems from the basics of like, 'Why do these experiments work so well?' to, you know, 'How do you have a systematic method for incorporating the constraints?' So, for example, if I have one of these multi-layer perceptron architectures for implementing a function on R2, and now I want to have Neumann boundary condition on like a weirdly shaped domain. There's some ways you could cook up to do that that aren't so complicated, but you might ask like, 'Well, did I lose all the beautiful properties that MLPs have in terms of expressiveness or, for example, in terms of kind of ease of optimization, which I think is sort of well understood or sort of not at this particular domain.' And I also think that's sort of a rigorous and careful numerical analysis style comparison of like, 'Okay, if I can give you 200 neurons or a grid of 200 cells, which one should I use?' is still pretty much open to my knowledge. Like there's some initial results in this space, but not a whole lot.
So in any event, now that I've had the world's longest introduction where I spent roughly half of my time, I thought I would tell you about some of the interesting almost purely empirical success stories that we've had in sort of using these extremely unconventional and weird discretizations for problems that have this sort of variational nature to them that I think begins to give some intuition for now going back and propagating to theory where like we could come up with a more systematic way to determine like these are the problems we think are likely to be relevant, these ones maybe less so for these crazy discretizations.
So I'll tell you a bit about some work from my student David on computing minimal surfaces and solving problems in GMT. Some work from my student Anna on computing generalized barycentric coordinates and skinning weights. These are good examples of problems from sort of optimal control looking domain. And then finally, and for like about 30 seconds, I'll tell you about kind of a funny thread that we've had in the last year or two involving kind of an adjustment to neural network architectures that's borrowed from the large language modeling community, but is a nice connection between all the stuff I'm talking about today and just like SVDs and numerical linear algebra, which was heartening for me as a person that's done that for a long time.
So let me tell you a bit about minimal surfaces. And I think this is a good example of a problem that A, is really hard for me to understand, but B, I think it is really difficult to discretize for some ways very subtle and some ways very obvious reasons.
And so, in case you're unfamiliar, I'm guessing there are mathematicians in this audience who know far more about these problems than I do, but essentially the typical minimal surface problem is as follows. I give you some curve embedded in R3. It's closed. Maybe it looks like this trefoil knot, whatever. And my goal is to find a surface whose boundary is that curve and has minimal surface area. In this case, I'm going to let it self-intersect. I'm going to look for the global optimum. And my curve is going to be oriented, which is a little bit useful.
And so there's this beautiful theory out there from GMT, which I'm not going to pretend to understand entirely. I put this picture of Federer's textbook as if I've gotten past chapter two. This textbook is fascinating, right? Then there's a second textbook, which is a user guide to this textbook, which itself is hard to read. But in any event, my student David was brave enough to plow through all of this stuff. And I think the kind of interesting takeaway is specifically if you just care about that minimal surface problem and not all of the beautiful generalizations, convergence result, and other theory that of course is really critical in mathematics, there actually is a really beautiful numerical problem hiding there that isn't so hard to implement. And so the story looks something as follows. Essentially what happens there is that you take this original L2 problem and you map it onto something called a minimal mass problem, which kind of looks closer to something in measure theory, where it works with this slightly kind of generalized object called a current, which is kind of like a thing that integrates against a vector field or a form.
And in that language, the kind of beautiful takeaway from a thousand miles away, which I actually hadn't really seen articulated cleanly until there's some really beautiful work from Albert Chern at UCSD that wrote this down in a really nice way. It's really just kind of an L1 looking problem that you have this object T which is a current or like very roughly a vector field up to the fact that it turns very very singular. And I'm trying to minimize this thing called the mass norm which basically looks like L1. It's like the integral of this vector field not squared. But it has a very particular constraint which is that it's sort of curl-free piece. And like essentially what you want to do is that you have a piece which is curl-free which is really your variable plus a field which comes directly from the boundary curve which is actually the Biot-Savart field. It's the same one that shows up in like, you know, electromagnetics.
And so really it's this kind of interesting L1 looking optimization problem that happens in this particular domain. And then the kind of interesting thing that happens is this vector field T is sort of like the normal vector field to the surface. That's what comes out when you optimize. Or like I guess the better way to think about it in terms of currents is like measuring the flux through the minimal surface.
Now, in reality this is a problem we can implement numerically. I mean this is a thing we can actually play with. And that's what Albert's work did to give him credit. Like we're just building on top of it. So like my students and I often joke that we compete for like the world's most complicated way to draw a straight line. And I think David currently holds our record for that. Although there's some multiple transport papers that can compete. Which is as follows. So okay. Straight lines are minimal surfaces of the plane and their boundaries are two points. In this case I guess oriented like a plus one and a minus one on these two guys.
So here's what this computational pipeline would look like. I compute the Biot-Savart field which looks like this. It's a, you know, kind of unsurprising. Looks like something you might see in an electromagnetic textbook. And now I optimize for this field that kind of cancels it out. So here's F. So its gradient is the thing that's trying to reduce the norm in this particular mass norm sense.
And what I get if I take the norm of the sum of these two pieces is a function which is peaked right at the minimal surface. And indeed, as you sort of in this case, this is just happening on a grid, as I refine my grid more and more, what you end up observing is that this is thinly sliced right at the minimal plane.
Now, this is super cool. I love L1 problems, they're convex. I work a lot on convex optimization, so that's cool. But this also, I think from a numerical perspective, is an interesting weird space where like at the end of the day, all the cool data and like the thing that I actually care about is sitting on this tiny line, but in order to find that tiny line, I had to solve a problem on a giant grid. And so I think this is a good example of a problem where like adaptive resolution and all these things that we think about a lot are really critical because like in some sense, there's this chicken and egg style challenge, right? Like I want to have the highest resolution minimal surface I can get, but F is very smooth away from the minimal surface. Whereas here it has a little jump discontinuity. The only problem is I don't know where it is.
And so that led us to do this totally bizarre thing, which is to basically follow that pipeline I outlined for you at the beginning of the talk, to translate that problem into one that uses neural networks. And in fact, it actually translates really cleanly. So if you go back to this formulation here, actually on the upper left here, obviously this constraint is kind of unnecessary. I could take this form for T and just plug it into the norm on the top, because my unknown is just the function F.
So what do I do? I parameterize F as a neural network, like a, in this case, not a multi-layer perceptron. We have to use an activation that has at least two derivatives, rather. And now I think of that mass norm as just an expectation of this L1 thing inside of it. That's perfectly fine. It's just a change of notation. Now I optimize it with stochastic gradient descent. So in every step, I randomly draw a bunch of points x, I evaluate this function here, I take its derivative with respect to theta, I take a little step, and I optimize. This is like purely a weird side project for David. Like we read Albert's beautiful paper on top, and we thought like, 'Oh, this would be kind of a fun thing.' And here's the really surprising result is to my knowledge this is state of the art for computing minimal surfaces, which is not the kind of problem you would expect neural networks to be very good at.
So for instance here, it's actually really challenging to kind of make a fair comparison because obviously the degrees of freedom here are very different. I think here we've matched the number of degrees of freedom, which is not entirely fair because the problem on the right is much harder to solve numerically. It takes a long time. But when you run to convergence, what you get are these beautiful sharp minimal surfaces without that many more variables in your optimization problem.
And I think what's going on, in fact we played a little bit with like looking inside of the structure of these neural networks and where the values kind of are or the, you know, where the knee of these activation functions is living, is that this is sort of a nice way to do some version of adaptive style solution to this variational problem without actually having to implement all these crazy mesh refinement style strategies because what these MLP representations can do that a mesh can't is kind of decide where to place the resolution, right? And so basically by putting all of the high frequency information in these activation functions close to the minimal surface, which seems to happen empirically during the optimization procedure, you can resolve the interesting part of the problem without having to resort to some really tricky kind of ad hoc style numerical method.
Now, I should note in this paper sadly we tried to share it with the computer vision community and they did not care about minimal surfaces. They said, 'Well, you need to be able to represent stuff like data.' So here's data. Sadly, if we take, in this case, I mean the input would be a curve and the output would be a minimal surface. Sadly, this is a pretty poor approximation. We do, if you're curious, I don't think I'll go into here, but there is an extension of this where you can represent any surface, well, a set of surfaces with boundary. And the basic trick here is to just insert an unknown Riemannian metric into your minimal surface problem. And sort of it proves a small lemma which is, you know, any surface is a minimal surface, you know, if you believe in yourself and you choose the right Riemannian metric here.
So if you add B as a second variable, it's actually kind of an interesting weird expressive surface representation which surprisingly was convincing to the computer vision community. Although I would not regard this as a very practical technique for 3D surface reconstruction. So for instance, here is what B looks like if I want to draw a circle. And then I get this new function which is now the world's most complicated way to draw the bottom half of a circle. And we validated this on a big set of stuff.
More recently, we proposed a fun extension of this problem to fibers sitting above a curved surface. And I wanted to highlight this only because this problem we were unable to discretize using this neural technique. We had to go back to weird grids. And I think it's an interesting example of like kind of a weird frontier in this domain where it's not even obvious how to express problems like this in this new universe. So in particular, what we were after in this problem was design of frame fields. So this is a generalization of vector fields to like tuples of directions per point on a surface. And you typically want them to be smooth and have kind of other nice properties. And the thing that people really care about in this domain is the placement of singular points. And so if you want to optimize for these frame fields specifically when they have unit length, then things like Dirichlet energy don't make any sense because they blow up near these singular points which are exactly the things you care about.
And so David did a kind of crazy thing which is he views it as a minimal surface problem instead where you look at the circle bundle living above the surface. So at every point, you kind of like glue a little circle which is the angle of your frame field at that point. And now you kind of each of these frame fields downstairs as lifting to a minimal surface in that product space. And he shows that you can actually kind of take this minimal surface problem from the papers I was telling you about before and lift it to this domain, but it gets really yucky because essentially you now need to incorporate the connection of the surface among a lot of other pieces of the puzzle that you didn't have before.
I wanted to bring this to your attention only because the formulation itself actually looks very similar to what we saw before. I've put this exterior calculus here to intimidate you. Essentially, unfortunately for us on a curved surface we were unable to discretize this thing with a neural network because it's not obvious what it means to parameterize the function in this particular domain. I think there's some really interesting and challenging open problems in this space. Even just like never mind proving anything, just getting it to work at all was something we were unable to do.
So in my remaining 10 minutes I'll tell you about two other kind of threads in this domain, but similar kind of. The good news is they all have the same takeaway, which is you know you have this kind of weird variational problem, you plug in this crazy discretization, and some nice things come out. I'll tell you about some problems that are I think more conventional in the computer graphics world that have been regarded as sort of ossified and there's like very well regarded algorithms in this space that I think we were able to revisit and show that there's some benefit to in the last year or two. This is specifically in these two problems of computing barycentric coordinates and skinning weights. Which corresponds to very nice problems mathematically behind the hood. In fact, there's some really cool open optimal control problems hiding here that I would love to share, but that's a different time.
So in general, if you're not familiar with computer animation, there's sort of two very typical ways people deform 2D and 3D models. I mean among many, but here are two that I think are very common. On the left hand side, what people might do is embed a skeleton in the interior of a 3D model. So here's a dog and you can kind of see this little skeleton curve in the inside. And then you deform the skeleton, and then the problem becomes how does deformation of the skeleton affect deformation of the surface? And then there's a different one where you place a cage around the outside of the surface. You deform the cage and that propagates inside. These are sort of similar and sort of not. I mean, the curve network on the left-hand side is 1D, the cage is sort of 2D. And so they lead to slightly different mathematical formulations.
Behind the hood. Under the hood. The first one — for some reason I decided to present these in my reverse order. The first one is this problem of skinning weights, which mathematically I would call computing a partition of unity, where I have the skeleton which is embedded in the interior of a domain.
And now essentially what I want is for each bone to have some function which is sort of the influence of that bone on the motion of the surface. And if you think about it, you want that to be positive and to sum up to one, meaning that every point on the surface is affected in total by some set of bones.
And typically the way people do this, it really does look like a lot of problems in this community. In fact, the original one was this idea of harmonic weights, which really just to solve a Laplace equation whose boundary conditions, you know, place a little delta function at each of the bone locations.
This is simultaneously beautiful because of like maximum principle actually has a really nice interpretation for animation because what it means is like if I move the bone, there's not a chance that like part of your surface could go the other direction.
On the other hand, as we probably know in this community, like there's a lot of electricity that happens at the bone. So you get this thing which artists call a tent pole artifact, which we would call poorly defined boundary condition, where essentially like you shouldn't be having point constraints on, you know, for example, a volumetric Laplace style equation. So then there's a big literature for how to fix that. And in particular, the most popular model is this thing called bounded biharmonic weights, which does the following.
It optimizes the bi-Laplacian of the weight functions because they really want smoothness. However, when you do that, you no longer have a maximum principle. So we're just going to enforce it by fiat. And so, like if you think about it, maximum principle and or even just non-negativity and partition of unity is a convex constraint. So you can kind of write down this crazy variational-looking problem and just optimize for the W's. And this is state of the art. The only problem is it's very hard to optimize. You now move from like a linear problem to a very non-linear one.
By the way, if you're looking for really cool, interesting like variational calculus problems, like proving that these W's have a maximum principle, to my knowledge is an open problem. But I, to my knowledge, also people haven't observed anything otherwise. And these are used like if you download animation software, that's it's in there.
But I thought I'd actually start by telling you, because this is actually the order in which we worked on these problems, about the second one, which is on barycentric coordinates, which adds one more constraint to that partition of unity problem. Essentially, in the barycentric coordinate setting, what I want is for every point in the interior of the domain, the kind of the influence of all the points on the boundary on the motion of that point. But I also want one additional property, which we call the reproducing property, which says if I leave the boundary of the cage alone, that point doesn't move.
So in other words, if I take the weighted average of the points on the boundary where the weights come from that barycentric coordinate function, I really should reproduce the point that I have. And so there's one additional linear but kind of nasty constraint that shows up in this problem.
The simplest example of barycentric coordinates is on a triangle. In this case, it's just linear algebra. So if I want any point in the interior of a triangle, I can write it as a linear combination of its three vertices. You can convince yourself just by solving this matrix inverse problem that you really do get barycentric coordinate in that the positive sums up to one, since even though maybe from linear algebra you wouldn't necessarily expect that from day one here.
So here we're just trying to take that problem and generalize it. So now instead of a triangle, I have a whole polygon and I've got an X in the interior. Now it becomes a design problem. This is underdetermined, right? There are many barycentric coordinate functions that write X as a linear combination of points on the boundary. And so that allows me to do a bit of design. Like for instance, maybe I want as X moves around that these weights become smooth.
I want to highlight by the way, there's an interesting question that is kind of understudied in this world, which is what happens if I take the V's and I just refine. Like if I take that boundary mesh and I want to take a limit to a surface. Interestingly, I really couldn't find many methods out there that analyze what happens in that case. So for today, we're going to talk just about the sort of discrete version, which does correspond to what artists actually do. In general, these cages are quite coarse. But there is kind of an interesting differential geometry thing kind of hiding here.
So what do people typically do in that community? They work really hard to cook up like a very specific function maybe for these barycentric coordinates that satisfies the desiderata you want. But a different thing you could do is just write down a big nasty variational problem and see what happens, right? And so that's what we decided to do in this particular work is say, 'Okay, the constraints of this optimization problem here are all the things that we need in order to solve for barycentric coordinates. So for every X, I need them to be positive and sum to one. And then this last thing is this reproducing property. There's also one more subtle thing which is a third condition which has to do with what happens as I approach the boundary.'
And maybe I cook up just a generic functional in terms of these unknown alphas and their gradients that maybe express smoothness or whatever else. Like you could imagine maybe a total variation thing or like trying to localize the effect of certain motion. To my knowledge, there actually are not any papers that try to study this problem in general generality, and I think there's a good reason for that. Like as for example with classic finite elements, what would I have to do? I'd have to mesh the interior of the volume of that boundary cage. I'd have to have a different alpha for every vertex there, each of which is vector valued because I need one alpha value for every point on the boundary. So this would translate to a giant numerical problem in practice.
So typically what you see, there's a lot of kind of closed form formulas where people just guess things that happen to satisfy the constraints and look nice. And that's typically what's implemented in software. There are a few optimization techniques out there. There's one that optimizes a harmonic energy and then a second one that looks like total variation. And these are not really used in practice because they are quite slow and not very robust.
So here's what my student Anna did. She wanted to ask whether you could solve this problem using these neural network style approaches, which seem valuable for a few reasons. One is you could imagine compressing a bit, and the other is that like for the most part you just want these functions to be smooth. Like it's okay if you don't solve the variational problem to like many degrees to many digits of optimality, so long as you satisfy the constraints of this setup.
This is also kind of an interesting challenge because you really care about satisfying the constraints. So for example, what you don't want is you stick the cage around the surface, you glue everything onto the cage, and then the surface itself just moves before you even begin animating. And that's the kind of thing that can happen if you don't satisfy these constraints really closely. And so a generic neural network architecture doesn't work. This is kind of an interesting example of a problem in this domain where you really do need to satisfy the constraints, which means you need to build in additional features to this architecture. So in this case we kind of had one foot in mathematics and one foot in very empirical stuff to design our approach here.
And so here's the basic observation. Here's what our surface is, pretty simple. If I take any point in the domain, here's a really dumb way I could get barycentric coordinates is figure out one of the triangles of boundary points that contains that guy. Clearly there's at least one. And call that the, you know, call the barycentric coordinates inside of that triangle the barycentric coordinate on this whole boundary. That gives me some function. It's discontinuous probably, but it does satisfy the constraints.
And so the basic observation here is you can do that with every possible triplet of vertices on the surface here. And so basically you can enumerate all the triangles of vertices on the boundary. For each one of those I get a different barycentric coordinate function. And moreover, if I combine them I get another one. And that's her kind of clever observation here.
Is rather than having a neural network that models just inputs of position and outputs coordinates, she instead inputs position and outputs a distribution over triplets of boundary vertices. This is obviously expensive, but there are some clever empirical techniques we can use to make it better. And this thing automatically satisfies all the constraints of our problem without needing additional kind of clever construction here.
And now we can apply that sort of outline that I've shown you before. I have a neural network style architecture that satisfies all of my constraints. So now I have an unconstrained problem for alpha and I can solve it with stochastic gradient descent. Which leads us to do all kinds of fun modeling things. So now we can experiment with different objectives and see how they affect the actual coordinate functions that we get.
Which is interesting and really crazy. So in this paper we illustrate for example if you kind of change out different notions of smoothness you get different coordinate functions. I'm not going to even attempt to explain the difference because the screen is a little yucky. But as you can imagine total variation essentially leads to coordinate functions that are a little more localized in exchange for smoothness.
You can also use this to deform a horse. But kind of interestingly you can pick up all kinds of weird non-linear objective functions for understanding these coordinate functions. So for example here, what we did is as follows. At the end of the day, typically, barycentric coordinates aren't the end of the story. You solve like a little rigidity problem to actually do your deformation because you expect your surface to kind of bend like a piece of rubber.
And so one thing you might do is say, 'Can I find the barycentric coordinates where when I deform, for example, the tail of this cat, the motion I get best resembles something that optimizes a very non-linear constitutive model from mechanics.' And indeed, what you get are different barycentric coordinate functions. I wish this projector were a little better, and I wish this cat weren't moving so fast. I will share this animation with you guys later, and if you really bond with it, what you'll see is that the tail of the cat — this is by the way just linear in the motion of the vertices of the cage here — but for this particular animation on the right-hand side, using a very simple constitutive model, you end up with deformations where you kind of don't see the kinks in the cage as much because that would look bad from the perspective of rigidity.
And so there's actually some value in these very non-linear kind of setups for these problems, which I think there's no way you would have been able to do with the kind of typical discretization where you could have gotten stuck far earlier in the pipeline.
I'm low on time, so I'll probably skip over. I should mention we've kind of done similar things for this skinning problem where you're looking for a partition of unity. In that case, you don't have that reproducing constraint. It's a little easier to cook up architectures and functions that satisfy the constraints that you have.
This is this bounded biharmonic weights problem. By the way, one thing that I'm a little embarrassed by as an optimal transport person that I kind of realized a year ago is like, I mean, I tell my students like anytime you're encountered with a convex problem, the first thing you should do is compute its dual. And strangely, I've worked on this problem for years and never bothered. So if you're curious, it has a beautiful dual, and in this paper on the right-hand side, we worked it out. That's my bad. But I wanted to share the one on the left hand side, which is another kind of argument for using these crazy discretization techniques.
So essentially Anna, the same student that was on the previous paper, who's just like an amazing engineer in addition to the mathematician, she wanted to make skinning weights that work on these really yucky domains you encounter in graphics. So for example, this pig on the right hand side was drawn using a virtual reality 3D modeling tool where you kind of like pet the side of your 3D model and it like gets these little beaded traces. This thing, if you put it into a tetrahedral meshing tool, obviously would completely fail. It's like not watertight.
But one thing that she did just for fun is she took this giant benchmark of 3D models called Thingy 10K and put it into the standard tet meshing software, which is what you need as a precursor to solving for biharmonic weights, and found that it fails on some huge fraction of those models. And so I think it's a thing that our community sometimes thinks of as a solved problem, which is just you need to discretize your domain before you apply all these tools, actually is a really hard problem in practice in these really nasty geometries that we encounter in this space.
So on the right hand side is another one. This is actually a watertight model, but I believe all but one of the tet meshing softwares that we tried wasn't able to mesh this particular nasty model of two gears that barely touch. They don't touch. We tested to numerical precision, but you really have to get to that last couple digits.
So what Anna did is combine some of that intuition from that neural network world with in this case hardware ray tracing for visual — like essentially, if you're in the interior of this volume, you can do checks that really check like which part of which connected component am I in by using sort of techniques borrowed from rendering to cook up a function parameterization that really carefully respects the boundary conditions using winding numbers.
And then she's able to make really robust versions of these skinning weights that extend to all of these crazy domains. So I'll omit the details because I'm pretty much out of time, but I encourage you to just take a look because it's a really interesting combination of clever engineering. At the end of the day, actually the reviewers in the computer graphics community saw the word robust and then really panicked and had her test on like I think 10,000 models, which she did.
Which is actually — I think we tested part of this pipeline on a whole one. There's a precursor to this one which you need a skeleton and there's not that much data out there, so I think the full pipeline was tested on more like 20. But some of these really nasty cases are successful here because we just bypass I think what we often consider as sort of a precondition for algorithms in this domain, but actually is the source of a lot of non-robustness in reality.
In fact, it's kind of funny if you read, if you just dig through like online forums with artists that are working with these skinning weights tools, they get really frustrated and the error messages that they're seeing are totally impenetrable, but really just are saying, 'I'm sorry, I was unable to tet mesh this weird singular-looking thing.'
Okay, so in our final like 5 minutes here, I'll just tell you about one little extension of this. Well, the good news is this really is a short thread. Then we'll call it for the day, grab coffee, and I'd love to learn all the cool problems that you all are working on. And this is sort of borrowed from the sort of variational image processing world.
I think you can view all of the papers that I've shown you so far as examples of what people in computer vision would call instance-specific neural fields. Now you're in the know, you can drop that at bar trivia. But essentially all that is sort of saying is that I have a single problem I care about and I'm using a neural network to represent the unknown function. Right? And so this phrase neural field in general is what it kind of refers to, that object. It's used not only in the geometry domain, but also sometimes in image processing.
So here's actually Anna's dog. Her name is Bella and she is — this is currently a neural network that's overfit as a function from R2 to RGB. And then you can, you know, take your whole image processing pipeline and write it in this really crazy kind of very non-linear language. I think the advantages disadvantages of that are not super clear, but it's a thing you could do experimentally, for sure.
One reasonable question is this — so here my student On said, 'Let's say that I take one of these overfit objects, and now I modify it a little bit.' So like On actually, interestingly, this previews On's next research paper, which I'm not going to have time to talk about if you recognize this diagram. I've never seen a student add a teaser to one paper that... Anyway, right.
So here we overfit a neural network to this image of what do you call it? A support function. And then he adds these little stripes, and then he just fine-tunes this neural network, which is a very slow procedure because there's many weights.
And essentially, the question that one might ask is oftentimes in these geometry image processing problems, you're presented with a bunch of different instance-specific neural fields that are just small modifications of each other. This happens a lot, for example, in time-varying PDE, where maybe in each frame of the thing you're solving, you know, you don't expect there to be a whole lot of change.
One thing that I haven't told you about is that a lot of these neural network representations just are big. And like, in order to make a lot of the pipelines that I'm telling you about work, you really do need quite a bit of weights in these matrices, which I think is a really big issue for these kinds of problems. It's mitigated a bit by the fact that software like PyTorch and so on can deal with them blazingly quickly, but from an asymptotic perspective, it's not so great.
And indeed, like if I'm trying to fine-tune the image on the left to get the one on the right, I, you know, using a classical technique, here there's 340,000 parameters I have to optimize to get from one to the next.
So essentially, what On was asking is whether if I start with something pre-trained that has sort of the structure that I'm looking for and now I edit it in some fashion, could I get there a little bit faster? And you can imagine why this is relevant including to some of the problems I've shared already.
Now, as a reminder, here's our multi-layer perceptron. Here I've decided to omit the bias. By the way, bias again is just the affine part of the linear map. And essentially all the expense comes in storing and optimizing these WHs.
So here's kind of an interesting empirical observation from On, which is as follows. I take that image that I overfit to and now I optimize it somewhere to fit to this other image which is slightly different. And I look at the difference between those matrices W. And one thing you can observe is that distance, that difference actually ends up being pretty low rank. Which is surprising. I mean, there's no reason why like layer three of this five-layer thing should have that structure.
And some of the changes that he's making are like editing the boundary of this shape. So like in pixel space, it actually is changing many pixels on the image.
An interesting side note is in order to notice that he had to develop a special notion of SVD. That's because the matrices in neural networks are randomly initialized, so they actually have high rank. And so you have to be very careful how like what you mean by a low-rank change. So you have to incorporate actually the statistics of the data that are moving through this thing.
But in any event, this actually suggests kind of a sneaky trick one might do, which is rather than re-optimizing this W every time you want to make an adjustment, maybe you optimize instead for a low-rank change to this thing. This actually has a name in machine learning literature. This is called low-rank adaptation or it's actually become so trendy, they call it LoRA. I think it's really cute.
I borrowed this figure from one of their papers that like there's a lot of discomfort with equations and so they've actually managed to illustrate a low-rank update using like a bar chart, which I think is kind of fun. But in any event, from a mathematical perspective, it was very simple. Rather than updating a matrix, I instead update this product of two factors here. And like, embracing the fact that our problems are non-linear and nasty anyway, like, what the heck? We'll add like a little more non-linearity hiding there.
And indeed, essentially, when we saw this originally our research group, we said, 'Ah, like, well, we don't know anything about neural networks, but we're really good at numerical linear algebra. And like, here's this little tiny domain where that is actually very relevant.' So we had a whole series of research papers that study very classical linear algebra effects about these little low-rank matrices that are hiding inside of a lot of updated language models.
So for instance, you could like take a big collection of these adaptations for a language model, which exist now. You can just go online and download them. So like, maybe there's ChatGPT, and then there's like ChatGPT that, you know, talks like Shakespeare, and those are low-rank changes to one another. And so you can download for every base model, you can download a whole collection of low-rank updates to that thing.
And that begs the question of like, can you do PCA on that collection? What is its dimensionality, and so on. Which you can actually tackle with very classical linear algebra tools. And so we did that to great effect, and even showed you could like make a high-throughput server of personalized language models by compressing these things in this clever way, and then swapping in and out these low-rank updates. Which is this fun side project that was really just borrowing our knowledge of numerical linear algebra and putting it in this funny new universe.
But that aside, I think one of the takeaways from our work — this was my student Rickard mostly that was doing this stuff — is that there's a lot of opportunity for compression. So in particular, like Rickard took this big collection of these low-rank updates and did some version of SVD on that collection. You have to be a bit careful because they're products of two matrices, and so how you do that. And what he showed is you can get away with really high reconstruction error without affecting performance of these neural models.
Which is nice. Essentially, this is another sort of clue that there's a lot of room for like a lot of wiggle room in these representations to kind of play with how we represent our functions.
And indeed, essentially my student On took that idea, borrowed it, and used it for these image processing style problems, and showed that the same thing applies. So, for example, here he has dramatically fewer parameters, and he's able to do basically the same version of an image update.
And indeed, that allows us to solve even kind of variational problems in the nature of what I started with. So, for example, here's a very classic model of total variation denoising from Chambolle and Pock and company, where you kind of have a noisy image, and you're trying to optimize that with a TV regularizer.
So here maybe your noisy image is itself one of these neural network representations, then you can do a very good job on these TV style problems just optimizing a low rank change, which is much more efficient. So this becomes actually practical because the number of parameters has been reduced quite a bit.
So there's many examples in this paper. I think basically that's the high-level takeaway, and then from there he just has a bunch of threads showing where this kind of high-level idea is relevant.
At this point, I'm out of time. So essentially to conclude, if we're just like a hot second here, I think the basic takeaway is that we're in this exciting new very experimental era of totally bizarre discretization strategies that feel very uncomfortable for my undergrad finite element course, but have some possible promise for certain applications. So I don't think it's a thing we should dismiss.
In particular, like for problems where there's adaptive resolution, for problems where you need to build in weird constraints that would be hard to satisfy normally, or for problems where like there's many different variations of the problem and you're trying to incorporate data. In these kinds of settings, these representations actually really do make some sense, but on the other hand, they're basically wide open in terms of the mathematical understanding of when it should work, when it should and making a systematic and informed decision about that.
I mentioned some open problems at the beginning, so I'll leave that there. I also tried to articulate some interesting open problems in the spectral geometry domain. I'm going to defer those to our talk on April 21st, where suddenly you'll be stuck hearing from me some more. But in the meantime, it's my pleasure to answer any questions or we can just go grab coffee and keep chatting up there. So many thanks for your time and it's really a pleasure to be here.
