Prof. Justin Solomon | Machine Learning for Spectral Geometry and Vice Versa

Thank you. Thanks for the introduction. It's such a pleasure to speak to you guys twice in two weeks and to share somewhere between a tutorial and some of the work that our group is doing in this intersection between machine learning and spectral geometry. Incidentally, last night while I was frantically getting ready to speak today, I was thinking I did want to spend just two seconds thanking our organizers. Actually, I was digging through my email to find how I got in contact with everybody in this community originally, and by some funny coincidence, it's almost precisely a decade ago. And we sent this blind email to Yosif and Alex at the time asking for help with, like, there seems like there's this spectral operator, we don't know anything about it. You know, is this thing going to be useful or not? And he actually was kind enough to write us back, and that ended up starting this ten-year-long collaboration relationship with this community, which I think is really cool.

And in any event, I was thinking about that while I was looking at this current workshop on AI and spectral geometry, that somehow this is a very unique corner of mathematics that is pretty open-minded and collaborative, and I just thought that was cool. In any event, today I thought I would tell you about some work that I guess one thing that I did, which in 2026 maybe is controversial, is the title of my talk. It says machine learning for spectral geometry and not AI. I think if you ask me to explain the distinction, I would have a lot of trouble doing that. Although I guess these days maybe AI is sort of confined to language models. But in any event, I think what I'll talk about today is pretty squarely in this first category of techniques for sort of training different models and really on the learning side rather than maybe deploying models that are already learned or trained on some outside data.

And I think the sort of dirty secret of a lot of this work is that really this is just, you know, numerical methods in a trench coat, and that these days, you know, no matter how hard these communities try, we're all staying relevant and have a lot of interesting and useful skills even in the presence of really exciting and new tools that are quite different than maybe what we've been used to. And I think that's really the right way to think about work in this space. It's not intimidating. It's just a new set of problems, and in fact, I think the people in this room have a lot to contribute to this space, even if the community itself moves really, really quickly. So I should give a little bit of a disclaimer. I feel like I have to do that a lot this week. This talk is somewhere between a tutorial and a research talk, and in particular, I noticed when I was looking at my slides this morning that it's like a totally biased perspective where it looks like my research team introduced this entire field, which is completely false. It's only that we sampled the research papers that I'm familiar with, which unsurprisingly are the ones that we worked on. And I'll try to mention some spaces where there's obviously many people that are energetically contributing to this space at the same time.

So I kind of speculatively wrote an abstract for this talk, and then I prepared the talk afterward and realized that, you know, that's not necessarily a best practice. But essentially, I thought in these 45 minutes, or at least we'll see how far we get before the end of our time slot, I'd try to give you my answer to two related questions that are in this discipline that are sort of reverses of one another. One is how can, since this is a workshop on spectral geometry specifically, how can techniques in spectral geometry benefit applications of machine learning? So I think so far we've talked a lot about the other direction, like how could we use AI to assist with proofs and numerics and so on. But actually the reality is that there's a lot of spectral calculations that happen in certain machine learning architectures that are state of the art for a lot of tasks in geometry. And I thought I would call that out as a bit of an alternative route and a new set of interesting problems that I think actually are almost directly relevant to the skills in this space, even, you know, without having to adapt to this new very non-linear regime that we're in today.

And then if we have time, I'll also tell you a bit about my personal thoughts on using some machine learning tools to accelerate solution of spectral geometry problems. Kind of like, one fun thing is that I posed an open problem last week. I have a student at home that already implemented some of the proposed solutions to that. I'll show you some pretty wild examples where he's managed to make this kind of work and kind of not, and this is very much work in progress. So let's get started. I'll tell you a bit about how I think spectral geometry can benefit applications of machine learning, especially from the perspective of some of the work that our team has done in the sort of 3D machine learning space.

So for the uninitiated, one of the big applications of machine learning that actually I think kind of in a funny way was more popular maybe five years ago, and then now that we see the sort of dominance of these language models, quietly this funny thing has happened that these tasks don't work so well in a language model because they're not language. And because people are so excited about language models, they can't seem to fit the square peg into a round hole. We've actually seen energy in this little corner of research decrease a little bit. That's not to say that it's less relevant, and in fact there's still a lot of really fantastic open and challenging problems. So essentially one of the big problems that my team works on quite a bit is machine learning for 3D geometry. And there's a lot of tasks computationally that essentially ingest a 3D shape in some format and then want to output some inference about that shape. So two classic tasks that I put on the slide here is maybe I ingest a 3D model like this point cloud here. It's a very famous one. It's called the Stanford Bunny. It's famous because it was easy to 3D scan in the '90s. It's made of terracotta, which has a very predictable kind of response to light. But in any event, the task on the left-hand side would be called classification, right? You input a shape and you output a label. On the right-hand side of the task here is segmentation. We have a scene and you're trying to break it up into different pieces. These are very sort of abstract tasks. I think one can easily ask like, 'Okay, is the segmentation tool, for example, useful on its own?' Probably not. It has to be paired with an application. And segmentation for like autonomous driving is probably different than segmentation for a robot. But these are the kinds of things that often happen in the computer vision community where you take these very... These tasks are kind of repeated in different fields and try to abstract them away a bit and then propose machine learning architectures that, for example, take big data sets of 3D information that maybe somebody has hand-labeled or segmented and then try to infer from that moving forward.

Now, of course, these days these tasks have been moved to giant data sets and in all kinds of interesting settings. So, for example, here is I just randomly screen grabbed a figure from a recent paper from one of my postdocs, Xiang Yu, who's now on the faculty at Westlake, where the task he was studying is just identifying the cars on the road for an autonomous vehicle. And this is a fantastic example where, for example, a lot of the machine learning tools that we've talked about this week are totally irrelevant for like 12 different reasons. In particular, let me just to kind of get you thinking about this sort of problem. So the data that you're seeing here, this is a 3D scene that's collected from a lidar scanner. For some reason, it looks a bit like a potato because I think it's passing through Zoom on its way to your projector. But in any event, essentially each of these scenes is maybe thousands or millions of points scattered in 3D. You can see it's not a uniform distribution. It's sampled kind of based on the procedure, which is this lidar scanner. It's sitting on top of a car, which is spinning around. And the task here is to identify the cars and the pedestrians in the road for obvious reasons.

Now, just if you're curious about the kind of things we have to worry about beyond just algorithmic design and labeling, you know, for example, when you guys work with ChatGPT, one thing you might have noticed is you put in your text prompt and then it says thinking dot dot dot, and in fact then it starts talking to itself for a little while and like, you know, there's this whole monologue or like we saw these funny examples yesterday where you like rephrase your problem in different fashion. Think about what happens here. So your car is driving 60 mph on a highway. You put in a 3D point cloud of everything on the road and you say, 'Okay, find all the pedestrians.' And now it says thinking dot dot dot, and your car is still driving 60 mph. And then it says, 'Oh, there's a pedestrian to your left.' You know, by the time the inference comes in with a standard machine learning pipeline, unfortunately, you are now, you know, 100 feet forward than where you started, and it's pretty much irrelevant. So in fact, there's a lot of interesting considerations in this domain beyond just a really high-quality AI, like efficiency, or actually another one that I think is really cool and possibly an interesting inspiration for work in this space is a lack of data.

So one of the things that really drove forward language models and computer vision models specifically is the internet, right? There's so much information out there. You can download gigabytes upon gigabytes of text and use that to train these AI models. The reality is for these 3D tasks, that data just doesn't exist. Right? And so in fact, a lot of the machine learning work in this particular domain has to work with a paucity of information and come up with really clever algorithms that work around that fact. For example, inferring 3D structure from giant 2D data sets, or for example, actually this, I love this research paper from my postdoc. He had this really clever idea, which is identifying cars and pedestrians on the road is roughly identifying moving objects. So he created this totally bizarre machine learning task, which is find all the moving objects from a static scene, which of course makes no sense. But it turns out a good proxy for that is finding cars. I see two hands in the back.

Yeah, it's a good question, right? So the question was, can you use a slow learning technique to train a fast one? Sometimes people call this distillation, and absolutely, this is one technique. Then of course, for problems like this, the first question you should ask is what's the slow method, which is itself very challenging. So in any event, essentially... Oh, yes, Nolda. Just make sure I understand. So you've got maybe a sequence of frames, and each of them is a point cloud.

Yeah, yeah. Oh, sorry. For this specific problem, yeah, it's kind of interesting. So essentially, Shawn Rue noticed these big data sets for autonomous driving come in sequences, which makes sense. They just drive these cars around. In fact, actually kind of an interesting thing in the 3D space is it's not hard to collect unlabeled data. What's hard is to get labeled data. And so he collects these sequences. From a sequence, it's very easy to find the moving objects. So then he makes a new data set, which is all the moving objects labeled in static scenes, and then trains a static machine learning tool on that, which is really sneaky. Yeah, it was clever. Anyway, it has nothing to do with spectral geometry, but I thought it was just a cool trick.

In any event, I thought what I'd tell you about... Oh boy, I'm already way into this talk. In the first part of this talk is a bit on what it takes to design machine learning systems that ingest data like this and output a decision, which is very different than text. And in particular, I think along the way, since this is a group of mathematicians, I thought it might be useful to kind of share some machine learning buzzwords so that you can, you know, use them at bar trivia and impress your friends. And in particular, I think a thing to think about a lot when we talk about this sort of architectural design is what we call inductive bias, which is sort of the idea that there's some assumptions about the relationship between the input and the output of a machine learning model that are built into the model itself. So for example, for 3D learning, where typically maybe you take a 3D scene and you rotate it by some angle and you'd say, 'Well, I don't expect the labels of the object to change, and so I'm going to build an architecture that has that as an assumption built in as a foundation.' And so oftentimes what we'll do is we'll look into areas of mathematics like spectral geometry, or representation theory is another big one, and say, 'Well, these guys are all about coming up with parameterizations of functions that have certain invariances. Can we borrow that mathematics and use it to inform how we design models in this space?' So hopefully I'm starting to use language that maybe is appealing a bit and seeing that really these problems are not that far off.

So for example, when I talk about some of our own work in this space, here's how I usually kind of motivate it. I think when you looked at early attempts to do machine learning from 3D geometry, typically because there's so much excitement in this space, what you saw a lot of people doing is taking essentially software that already worked really well for like computer vision tasks, for example, on images, and then just trying to like squash it into doing 3D tasks. So for example, some of the earliest machine learning architectures for 3D data, which by the way perform shockingly well, there is like a baseline here you have to compare to, is to do stuff like take basically somehow a virtual camera and just render the 3D object from different scenes and then put it into a 2D, you know, object recognition system. Turns out that that's kind of hard to beat, which is a little disappointing. But I think it's really representative of the strategy you see, which is you take stuff that works in this other domain, you kind of try to make it squash for the area that we care about. And in particular, there's one object that you've probably heard of called the convolutional neural network, which is extremely popular in image processing and basically is all about making machine learning that ingests data that looks like this. Now, these days, of course, the CNN has been replaced by all kinds of cool architectures involving attention and all these other buzzwords, but basically they're all operating on some version of a pixel grid. Essentially, one question that we've asked for quite a few years now is what would it make sense to start from the geometric application you care about and engineer an architecture that processes this kind of data? Like, would you end up in the same space or potentially not?

Now, and I think in doing that, we have to have a bit of comfort with the fact that we're not going to be able to prove theorems about every single property of these learned systems. I think this is a thing we all worry about in this community, but I do think that there is something that's possible to do as an engineer. And the term that I've sort of converged on is to make architectures that seem sensible. You know, so for example, if you look at a convolutional neural network, it's built on the architecture of the language of convolution on an image, which I think is actually language that this audience is very familiar and comfortable with. And essentially, a CNN is nothing more than alternating between convolution and a simple non-linearity. And the magic of these learning systems is the fact that composing those together can lead to a lot of complexity. So from that perspective, I think one thing you could say is, 'Well, what is that building block? What is that convolution?' Where maybe we don't know how our system works end to end. There's a lot of big questions hiding there, but at the very least, the building blocks of our machine learning algorithm kind of feel like a thing we would do in the geometry space, right? And I think that's a completely reasonable question. And so let's dig into that for just a few minutes and kind of see where that at least led our team to think about some problems here.

So in particular, here's our famous 3D model again. This is the Stanford Bunny. And here's in particular two different meshes of the same object. If you're in the geometry processing community, you spend your whole career generating, you know, upsampling these things, downsampling these things, making nice triangles, worrying about their interior angles, all this kind of stuff. There's all kinds of beautiful algorithms that have been built over the last 30 years for all kinds of tasks involving kind of working with these shapes. But as discrete objects, they're kind of interesting for machine learning. In particular, there's all kinds of interesting invariances. If I take this object and I rotate it, I think it's probably still a bunny. That much is certainly true. But beyond that, I think there's some interesting things that are a little harder to capture. For example, here I'm showing you two different meshes of the same object that at least to your perception are probably very similar, but from a mathematical perspective are completely different. I'm going to have some coffee. So one obviously has many more vertices than the other. The topology connectivity is very different. They're resampled from one another. And so building up machine learning pipelines that don't think of these as completely different pieces of data is actually quite challenging.

So one thing you might initially do is say, 'Aha, well, there is one structure that is very popular not only in this little niche area of 3D learning, but actually very broadly in machine learning, which is that this is a graph.' Right? Clearly, we're looking at it. We've got vertices and edges. We also have triangles, but maybe we forget that for now. And one thing we could think about is whether we could take some of the state-of-the-art methodologies for learning from graphs and just adapt it to this field. And indeed, that was what many of the early attempts did in this particular domain. Now, why is graph-based machine learning popular? Sadly, most of that community is not concerned with bunnies, but they think a lot about molecules and social networks and so on. And I think one thing that we probably all have some vibes about is that the graphs you see in those domains are very different than the graphs we're looking at here. But nonetheless, you could basically borrow a lot of architectures which are called graph neural networks for learning from these objects. And actually, I do think they pass this sensibility test. And so if you're not familiar with how graphs are processed, actually a graph neural network is a pretty straightforward object. It's sort of sitting in this funny equation here which has a lot of abstraction, but at a high level is very... So you have a graph, so a bunch of vertices and edges. Typically, they're decorated with some information.

So for example, maybe I have a social network and I have, I don't know, the income of every person on my social network and I'm trying to figure out where to put my advertisements. Then maybe the vertices, you have something called a vertex feature, which is like your income. And in every layer of a graph neural network, you have some operation that looks at your neighbors, applies some non-linear function that kind of relates every vertex to its neighbors. This is called message passing. And then just takes that information and gathers it back to the vertex. So for example, you can imagine taking the average of my value and my neighbor's value and summing over my neighbors and then dividing by the number of neighbors, and that would of course lead us to the Laplacian operator, which is roughly how you might be able to map this notation onto the stuff that's familiar. And then what you do is you make each of these little units learnable. Like, so for example, the thing that is trying to figure out the relationship between me and the guy next door is like a little small learnable neural network with just those two features. And then you maybe use some simple way to kind of move information back to a central vertex with the property that it's permutation invariant, which is giving you the variance you want. Like, if you renumber your vertices in your graph, you wouldn't want the inference to change. So these models are very effective for a lot of tasks. So for example, here is a molecule and kind of what inference looks like in one of these graph neural networks. But I think from the perspective of the applications we care about in 3D data, it's actually probably not the right thing.

And in particular, in terms of our AI buzzwords, let's introduce another one, which is receptive field, which is a big issue in computer vision-related 3D learning. Where essentially, here's one thing to think about. In one layer of these graph neural networks, a vertex talks to its neighbors. So let's say for example, I have a machine learning task, which I probably have no business applying machine learning to, but I want to compute the diameter of a graph. And I want to figure out the farthest apart pair of vertices. Well, here's a challenge. I mean, I could apply machine learning. It would be goofy because there are algorithms for that. But, you know, you could, whatever. You could put in a bunch of graphs there with their diameter labeled and see if you can infer. Completely reasonable. But I guess the question is whether these architectures make sense for that task. And here's one way to think about it. So in every layer of this network, a vertex talks to its neighbors, right? Which means that if my neural network has one layer, then essentially the machine learning tasks that I can solve effectively are the ones that only involve vertices and their neighbors. Or if I have two layers, a vertex talks to their neighbors, those vertices talk to their neighbors, and now I can kind of communicate information in the two-ring of a vertex. So if you think about it, for computing diameter of a graph effectively for large graphs, you would need to have a neural network with many, many layers. And essentially then you could easily break that system by making a graph with just one more diameter than you had before. Right? And so you can see how the inductive bias of your learning architecture determines the kind of task that you can solve effectively. Right? In other words, we've put some assumptions kind of hiding in the way that we learn, right? Like that message passing is the right way to make inference, which is often right. Right? I mean, for example, on a graph with like this Kevin Bacon style phenomenon, like where information travels pretty quickly just in a diffusion process, this algorithm is very effective. But you can see that essentially there's some assumptions about the structure of your network that are hiding in the learning procedure itself.

And so, for example, in the machine learning literature, sometimes you see this term under-reaching, which is exactly this issue, that there's some tasks that you just can't solve with this particular network architecture. In fact, there are some other ones, too. So here are two more that I think to the spectral geometry crowd will actually sound very familiar. It's funny because I would argue a lot of ideas that we've known for a long time in our community are being kind of rediscovered, which is great. That's not a negative in this very different context. For example, one is over-smoothing. So remember that I mentioned what do these architectures do? Well, in every layer you take information at a vertex, you kind of communicate with your neighbors, and then you summarize it back to a new feature at your vertex. Well, if you think about it, that operation looks an awful lot like, for example, you know, the heat equation or like a time step of a diffusion process. And indeed, what ends up happening is that very deep neural networks in this domain, like with many, many layers that have this message passing, information tends to diffuse throughout the graph, and it can be very difficult to, for example, learn tasks that involve like one very small interesting piece of information hiding in a vertex because it got diffused out in some interesting fashion. Similarly, there's an issue of over-squashing. I love these terms. They're very colorful. So if you think about a graph with like a bottleneck, so like one edge and then two big busy things happening on either side, if for example I wanted to compute the diameter of that graph, if you think about what happens in message passing, there's one really critical message that happens moving along that one bottleneck edge. But that bottleneck edge has to communicate everything about those two sides of the graph to each other because it's the only place where that happens. And so these are the kinds of challenges that happen in this domain, and there's a big literature on how to figure out how to fix these kinds of issues. And in particular, I think they're particularly challenging in the 3D space where our graphs are structured in a really nasty way. We don't have this Kevin Bacon phenomenon that's like three degrees of separation on a graph. Vertices tend to be very far away from one another. And moreover, we want to be invariant to things like resampling the surface, which is not even hiding it in these particular issues here.

So just as a kind of quick summary of some spaces where spectral geometry shows up in this domain, it's actually been a tool that's been around for quite a long time in this corner of machine learning, and I thought I would mention two. One is featureization, which involves mapping a piece of geometry to a vector in Rn. So, for example, one way you might imagine learning from a shape is to take a shape and find some encoding of that shape just as a generic vector of useful characteristics, and from that point on, you could apply a generic machine learning methodology just fine. Or even if you don't want to do that globally, you could imagine putting interesting information at the vertices of one of these graph neural networks, you know, whether it's curvature or something else, and then using message passing to kind of do some little local information beyond that. Both of these are procedures that are totally reasonable. And then the other one is maybe replacing this message passing procedure, this little step where you're sharing information with your neighbors, with something that is maybe more invariant to the kind of things that we care about in the geometric domain.

So let's talk a bit about featureization, which again involves taking a 3D model and then mapping it to Rn. As we all know, this is probably impossible to do in a way that completely captures every piece of information you care about in a shape, but it is possible to do in a very descriptive fashion, right? Like we know from spectral geometry that we can capture a lot of geometric information by doing certain spectral calculations. And in fact, actually when I teach this stuff at MIT, here's the slide that I use to teach my students about this stuff, which is that essentially one of the very first procedures for learning from 3D geometry did exactly that. So you would take your 3D model, effectively, you know, virtually hit it with a hammer and observe the vibration frequencies, and use that list of frequencies as a reasonable featureization of a 3D model. This has been around since the '90s in different fashions, and the learning architectures have changed, but the technique has not. Indeed, this is a completely reasonable way to go about basically machine learning from this 3D shape in a way that quotients out a lot of the really hard stuff we don't know how to think about. Right, because for example, if we compute these vibration frequencies using finite element method, then maybe they're a little bit invariant to resampling of the 3D model. It's certainly invariant to the rotation and so on. And some of the things that would be really hard to deal with working with raw coordinates in this coordinate system are not so difficult.

On the other hand, in this audience, it goes without saying that using this encoding loses information. And in fact, you know, one could argue whether that's the right thing to do or not. So, for example, one of the kind of cool early observations in this domain, just empirically, but it makes total sense from a geometric perspective, is that these architectures were used for two different tasks involving 3D human data. One is determining a person's identity. So, like you have a 3D scan of a person's body and you want to know who they are. And a different one, which if you think about it is kind of slicing your data set a different way, is separating by pose. Like give me all the humans whose arms are up.

And I'll let you think about it as spectral geometry folks, if I have the same data set of humans and I want to slice it in these two different ways, it turns out one of those is much easier than the other using these kinds of spectral methods. Because for example, humans vary a lot based on like their belly size and these kinds of things which are very much changes to intrinsic geometry. Whereas like if you bend your arm, the amount of intrinsic structure that's changing in your surface is relatively small. And in fact, these spectral calculations are very insensitive to that. So again, this is a nice example of inductive bias informing how well your learning procedure does downstream. And a good example where these mathematician kind of ideas that we've known for a long time have really practical bearing on the computational pipelines that we're engineering.

But in any event, if you're curious about the history of this, these are the two most popular techniques that have been around this domain forever. They were called the heat kernel signature and the wave kernel signature. Roughly this is the trace of the heat kernel and then the trace of a modification of the heat kernel with like a complex number hiding in there somewhere. And both of these are just ways to feature as a 3D model based on spectral calculations. So for this community, these are nice 5-minute read and a nice way to kind of justify and think about some of the calculations we're doing in a very practical setting.

I did want to note by the way just to kind of put a cap on that that funny email story from before. The email to our colleague Youssef actually led to a paper that kind of asked, "Well, we could take all of the spectral calculations people do in this domain and maybe make them intrinsically sensitive by using this Dirichlet to Neumann operator we've heard about quite a bit." And indeed, this is one of the techniques that for example my student constructed this little pair of 3D models with a cube with a bump in and out. And he actually did it in a really careful fashion so like the edges of this triangle mesh are identical length. So like these spectral algorithms cannot distinguish between these two shapes. And shows of course for the operators that we've talked about here that that's no problem for learning.

So if you go into the history of the learning literature, essentially the very first wave of these algorithms just computed these features and then directly tried to match them. The next wave started introducing little learnable pieces in different parts of the pipeline. So for example, you might, you know, you could imagine observing like, "Oh, well, you know, the kernel of the wave equation, the kernel of the heat equation just differ in terms of functions of eigenvalues. And so maybe what I do is that's where I put my learned machine learning methodology." It's kind of a sneaky trick. It's to learn just a function that inputs an eigenvalue and outputs a scalar or vector. And then this gives you some procedure for composing together a feature for your surface that's intrinsic. And you know, maybe you'll use a neural network to parameterize that function F.

And indeed, this is one of the early architectures for learning from 3D data. In our research group, we also thought about, as sort of suggested from this work where we just threw in this Steklov operator, you could also kind of ask just like what are other replacements for the little Laplacian more broadly. And we thought about like what would it take to make for example a learnable differential operator that you can plug into one of these pipelines that preserves the properties you care about, quotients out things that you don't like, but still is very sensitive to maybe the motions that are particular to a particular geometry task, like determining pose again is very different than determining identity.

So that led our team to think about for example looking at the sort of building blocks of a Laplace operator. So here's a very famous figure in actually the computer graphics domain, this particular figure, which is one way to compose a Laplacian from the sort of Hodge sort of perspective where combinatorially you can view building Laplacians on triangulated surfaces as combining sort of two objects. One is basically diagonal matrices and then the other are matrices that are like subtracting things along edges of meshes or for example looking at cycles on triangles. This is the perspective of discrete exterior calculus that comes from our friends at Caltech and other places.

So this actually led us to think about kind of a fun question which is okay so kind of topologically one thing you might remember from this corner of mathematics is that the d operator sort of is not geometric, is just topological. So if we're trying to solve geometry tasks then maybe the learnable object should just be these Hodge stars. And so my student Dima proposed doing precisely this. So he said okay, we're going to compose a Laplacian using this nice formalism here. If we only operate on zero forms or functions then we can drop one of those two terms or more generically maybe we leave them. And the nice thing about Hodge star is they're just measuring very simple things like angles and areas locally and for that it makes good sense to use one of these graph neural networks that is kind of a local operator by construction.

And so that led him to this funny, you know, the paper titles in this domain are a little bit cringey, but this procedure called HodgeNet where he essentially is proposing this parameterized set of Hodge star operators as functions of local geometry. That's the object you learn and then the whole learning pipeline composes together some Laplace type operator out of that, computes its spectrum, and then uses the spectral features for that for machine learning. And the amazing thing is he was able to back propagate through this whole procedure, like differentiate through the whole thing and optimize.

Just recently my student Anna has kind of reconsidered that pipeline and generalized it a bit and then I'll highlight that in a second. And so for example, here there is this learnable zero form Laplacian where it's just very simple features in terms of edge lengths and vertex positions in a local neighborhood. It's very easy to cook up ways to quotient out rigid motion and so on in the kind of local regime. And actually what we showed is that it led to some state of the art learning at the time. Of course, now it's been replaced 50 times over for some of these tasks like surface segmentation.

So here's my colleague Daniel being cut into pieces in many different poses. And indeed in this case, you can actually look at the spectra of this operator, which is kind of fun, and say like, "Well, what is this particular weird thing sensitive to?" And sometimes if you get lucky, for example, you can kind of see the joints of the 3D model. This is admittedly not super replicable. Sometimes we look at the spectra and they just look confusing.

Oh, these are different scans. Yeah, this is Daniel doing some kind of parkour in front of a 3D scan. You can download this data set online. Yeah.

So in the last... Oh, no, oops, sorry. I'm in the wrong place. Incidentally, I should mention that all of that work was on characterization, right? Just taking a model and mapping it to a vector and then basically just applying generic machine learning to that vector of features. A different thing you might do is say I want to stay in the message passing kind of graph neural network regime, but use a procedure that's a little more sensible than just passing information on your local neighborhood, which as you can imagine is not great for these 3D models. Like if I take all the triangles and I refine them, like I apply subdivision, the message passing would now do a completely different thing, right? The receptive field would change.

And so I did since it's not my work, I won't put it... I didn't have slides that I could steal really quickly. But I did want to point you guys toward actually a very influential paper in this domain, which actually showed up at exactly the same time as our hash thing, which does exactly that. So it replaces the message passing part of a graph neural network with a diffusion operator, in their case coming from just the Laplacian. They can use point cloud or triangle mesh Laplacian operators. They left the rest of the learning pipeline basically the same and showed that this was a nice procedure for learning from 3D data. And then I mean the kind of funny thing in this community is like people are energetic and experimental.

Once you look at this, you say, "Aha, well, I have my favorite, you know, electric operator. I can plug that in as my message passing procedure and now of course any follow-up works have come to that."

One thing I wanted to note is that I've told you a lot about 3D learning. We've also thought about learning from very high-dimensional data. So for example, if you want to compare data sets to one another, you might ask whether you can use similar procedures, and the answer is yes. I don't have time today, but I'd be more than happy during coffee break to tell you about some of the experiments and applications we have in that domain. We had a really amazing post-doc for a while, Tal, whose expertise was in diffusion maps, which is I think an area that originally came out of applied mathematics involving Laplacian operators on point clouds applied to very high-dimensional data. And she was able to show that it's useful for tasks like choosing which large language model is likely to be effective on my new data set or something like that, which is a little more abstract than the 3D stuff I'm telling you about today.

So I've managed to give myself 10 minutes to tell you a little bit about some of our work on this second problem here, which sort of flips our task backwards. So, the first one was basically, how can spectral geometry help machine learning? And hopefully, I've convinced you guys that even just the spectral calculations that we all know and love are hardly irrelevant in this domain. They're actually key pieces of a lot of top-performing architectures specifically for these tasks where the data that's coming in is not just a piece of text or an image, but rather something like a 3D model where the kind of information you want to communicate to your learning procedure is quite different.

And then once you're in that space, now we can take all of this beautiful mathematics that's been developed in this community and drop it into this learning procedure in ways that I think are really sensible and actually are quite beneficial for tasks in that space.

A second question that you could ask kind of goes the other way around, and I think this is one that we see in a lot of the tasks in this particular workshop, so it's probably okay that I'm mostly out of time, which is to ask how machine learning tools can accelerate solution of spectral geometry problems. Now, I am not a theorist, so when I say spectral geometry problems, I mean numerical solution. I don't mean theoretical, which of course leads you to a very different path.

I did want to highlight, by the way, I obviously don't have space to tell you about it today, but there is a cool piece of research from my student Anna and our postdoc Ahmed that sort of generalizes that Hashnet procedure, and I think might be fun for people in this room to play with when it comes out. It was just accepted 2 weeks ago, so sadly, my screen grabs still have the anonymous markings, but I double-checked it's okay to share, which is essentially a nice piece of software for backpropagation through geometric calculations that show up in geometry processing. And this includes a lot of the spectral stuff that we do in this particular domain. So, behind the scenes, Anna basically has a really nice implementation of the adjoint method where... So, the kind of beautiful thing...

Oh boy, this figure is like completely impenetrable, but you see these two rectangles in the middle. Essentially, what you do is you just implement the sort of forward version of your like eigenvalue problem using whatever numerical method you prefer, and it can call all the standard, you know, solvers and so on. And then behind the scenes, what she did was replace the little piece of backpropagation in the PyTorch software. Right? Like basically, PyTorch is this giant derivative calculator. So, she replaces that little module with one that knows how to kind of compute the proper adjoint for these particular settings, and then is able to solve a bunch of inverse problems without a lot of coding.

Which is kind of fun. It will be an interesting thing that we're going to put out there, and hopefully you guys can play with. It's like somewhere in between the really cool fine-tooth comb numerical methods we've seen, especially for example in last week's workshop, and then some of the cool like AI-generated, you know, completely out of our control methods that we've seen. You know, this gives you some way to give at least implementation of the forward pass, and then differentiate through the backward one.

In any event, I thought I would tell you a bit about some of our attempts to solve eigenvalue problems using these style machine learning domains. And in particular, building on the talk that I gave last week, one kind of reasonable way that you might think about doing that is to use a neural network not as an object that's doing learning, but just as a parametrization of an unknown function. So, here your neural network maybe takes a point in R2, and it maps it hopefully to, you know, the value of for example, the first Dirichlet eigenfunction of the Laplace operator, and then your task, which I don't think is really learning, but it's still interesting, might be to try to actually find that object.

So, for example, one thing you could do would be to maybe optimize for a Rayleigh quotient of some sort. Here, I've kind of converted to machine learning style notation, right? Rather than integrating over our domain, the Dirichlet energy, we just write it as an expectation, which is of course exactly the same thing. We could try to optimize this object, which actually suggests some ways that you could do this kind of thing by sampling, right? So, the same as the algorithms that we have for stochastic gradient descent, right? Where you get randomized estimators for derivatives with respect to parameters. You could imagine randomly drawing a bunch of points X to get an estimator of your Rayleigh quotient, and then computing this thing empirically optimizing and so on with respect to your parameters.

This turns out to not work terribly well for a lot of different reasons. For one, notice that there's a ratio of two randomized quantities here, which means, you know, if you remember your Jensen inequality, that actually the thing that you get when you sample is not the gradient of your problem, but rather some biased object. Moreover, working with a constraint like that your neural network has to have L2 norm of one, for example, is not a particularly nice constraint to work with. So, just translating eigenvalue problems to this kind of space is kind of an interesting and challenging thing. I should note, by the way, there are many papers out there that already think about this stuff. We are not the only people to do it, and I'm being an irresponsible academic by not highlighting them here.

But instead, I thought I would tell you about our particular work in this space, mostly as a cry for help to this domain to share what I think is a fun theoretical result for my student, Hamid, working with our postdoc, Sina. Both of these guys are absolutely brilliant. I'm just communicating the cool stuff that they've been doing. And essentially we have a hammer in search of nails. You guys have all kinds of cool problems that I think may be relevant and what we're looking for are like, "Oh, you know what? I bet you could use this tool for space in my domain."

So, essentially, here's the kind of interesting theoretical result from Hamid. Now, I'm... 4 minutes, this is not a good slide to have. But essentially, it's a parameterization of the set of orthonormal operators or functions, depending on how you want to think about it. And the idea is as follows.

Essentially, suppose you have some orthonormal basis for a function space on a nice compact domain, you know, it's well-behaved, got a smooth boundary, whatever. So, Fourier comes to mind. Then one procedure you might use for computing a different orthonormal basis would be to start with your Fourier basis and then write down an ODE that evolves your Fourier basis into something else. Right? And then there's a reasonable question you can ask, which is can you get any other orthonormal basis by some ODE behind the scenes? And so, Hamid's answer is in the positive, that in fact you can. And in fact it's a very kind of straightforward one.

So, here's the ODE that he solves. So, here Q kind of abstractly for now, if you think of it as a matrix whose columns are the orthonormal basis functions, I think that's perfectly fine in this setting. You solve this kind of linear-looking ODE, but specifically where operator K has this anti-symmetric rank-one form. And he's able to prove in a constructive but kind of yucky way that you can get to any other orthonormal basis by some choice of these alpha and beta of t functions here.

And so, what that suggests is a really interesting machine learning style technique for learning an orthonormal basis, which is rather than learning the object Q, which has all of these crazy constraints on it, right? You need orthonormality, unit length, and so on. Instead, the objects that you're going to learn are these functions alpha and beta of x, t, where x is, you know, position maybe in the plane and t is time from zero to one. And that has no constraints on it, so you can parameterize it with a neural network.

And so, here's his sort of practical in quotation marks, and I'll come back to the quotation marks in a moment, procedure for obtaining an orthonormal basis. As you parameterize alpha and beta as neural networks, you numerically solve this ODE for one unit of time, and what comes out is actually an infinitely long sequence of orthonormal functions that no longer are the Fourier basis even if that's the one that you started with. At least on a square.

So this allows us to... By the way, this is drawing inspiration from this neural ODE research paper which does exactly this for learning diffeomorphisms of different spaces. And this gives us kind of a fun orthonormal basis parameterization where you can now optimize like you have an optimization problem where your unknown is an orthonormal basis for like L2. And secretly your unknown are these alpha and beta functions and you differentiate through this big messy composition of a lot of stuff including even a numerical integrator.

And we're just beginning to show some signs of success. So this is very preliminary work but I thought just for fun with this audience I would share a little bit of the positive and negative results and then ask all of you guys to help me during coffee break with like A, what are all the inevitable problems with this technique? And B, are there some kind of fun applications we could try?

So for example, I think one of the well-known things that all of us probably has to teach in our undergrad linear algebra class is this eigenface technique where you have a bunch of images of people's faces and you think of them as big vectors of pixel values and you do principal component analysis. Here you can do the same thing except now you think of your faces as functions on a unit square. And then you can ask like what is the operator or the sort of orthonormal sequence that diagonalizes the covariance operator of that object. That's a pretty easy optimization problem to write down. And in fact we can see that this parameterization kind of works and kind of doesn't. You have the eigenfaces toward the center. And then these are the Fourier modes as you go further out.

We parameterize. And it's sort of in the center is able to learn these eigenfaces. As we go into the higher and higher kind of frequency domain, you get back to sadly just the eigenfunctions which seems to indicate the numerical stiffness of the problem that we're solving. Which I think is a really big challenge to overcome.

We also have some preliminary results computing eigenfunctions of Laplacian, but this is kind of a messy thing that I don't have time to cover, so I'll tell you about it over coffee. Or to show you a really bizarre experiment he sent me 2 days ago. This actually suggests an interesting way to do image generation where you randomly draw coefficients in this basis and then you add a loss that says I would like that randomly drawn thing to look like a cat. That turns out to be a loss that you have in computer vision. And now you can optimize the basis so that that procedure is as successful as possible. And in fact, you can see here that Hamid is actually able to really reliably compute a pretty good eigenbasis for the set of cat images out of this very abstract task.

In any event, I'm all out of time. We have many questions here, you know, what questions should we be answering, what objective functions should we be using, and so on. So essentially, this is my plea for help and my question is, you know, what should we be doing with these tools and how do we fix it? So I'm all out of time. Thank you so much for hanging out today and it's such a pleasure to be here.

I think there's someone right after me, right? Thank you very much for the master.

That's right. And I'm around until Saturday. Please come along here. That's also an opportunity to ask questions.