Paul Christiano - Mechanistic Estimation - IPAM at UCLA

Thank you to the organizers. Really excited to be talking about this stuff with everyone. In this talk, I am mostly going to try and present a math problem. I'm going to say a little bit about why I think it is extremely important for aligning and controlling AI, which is what I've been working on for basically my whole career. I promise that this is a really important math problem if we could solve it. And for at least the first part of this talk, I hope you can just extend me some credit on that, and then we can talk about it towards the end and in questions.

I will be talking about the same topic Jacob just spoke on. He talked about a concrete example; I want to try and talk about what we mean by mechanistic estimation, like how we set up that problem, give some conjectures, and explain why I think this approach could be scalable. The outline for the talk is: I'm going to give some examples of mechanistic estimates. I think this is actually an idea which most practicing mathematicians are pretty familiar with in their ordinary day-to-day life. So I'll try and explain what I mean by reference to some examples, give some definitions, build a little bit of intuition, state some conjectures, and then I will talk about implications for machine learning—what the relevance is to alignment and what kinds of things we really want to mechanistically estimate in the long run. People should definitely feel free to jump in with questions. I'm excited to talk about this now and throughout the rest of the week, or at least the next few days while I'm here.

The main concept in this talk, as Jacob introduced, are mechanistic estimates. You might also call them deductive estimates, causal estimates, or analytic estimates. I'm going to talk about some examples. A very simple example is: suppose I pick a 128-bit integer, and ask is $n$ a twin prime? Then I can run it over a bunch of random samples, and I can tell you empirically that like 0.017% of the time you get a twin prime.

There's something completely different you can do where you say, "Well, we can prove that about 1.1% of these integers are prime. We can treat $n$ and $n+2$ as if their primality was independent and conclude that you'd have a 0.013% chance of them being prime." And then you can start to just notice mechanisms that would cause this answer to be different, or corrections that you need to add to the independence assumption, right? You can say, "Well, you have correlation of the divisibility by two and anti-correlation of higher primes, and you have some phenomena where smaller numbers are more likely to be prime, which induces a further correlation." I can keep going in this way, right? These are both methods where I can spend more and more time to get a better and better estimate, but they operate in a very different way.

On the left, we have something where we are, as Jacob said, doing inductive reasoning, right? We made an observation. We don't know exactly what's going on or why that happened, but we can infer something about what future inputs will look like—what is true over the whole interval from a few examples. On the right, we are more making a deductive argument, which is in some ways like a proof, though we'll talk about the differences. This is an example in math. There's a ton of examples in number theory especially, but also all throughout math where we can make this kind of estimate, and we'll talk about more of them later. I'm going to give a couple other examples.

Another case that's near to my heart is suppose I have some circuit and I want to ask: if I feed in a random input, what is the probability that the circuit outputs one? I could do the same thing empirically: I start with my circuit, feed in some random inputs, run them through the model, and see what happens. I run that 100 times, record that in 30 of those cases it output one, so I'm going to estimate 30%, again generalizing from the examples I saw to a bunch of unobserved examples.

We can also do something that is more structural, more deductive, where we say, "Hey, I know that each wire is on 50% of the time. Every time I encounter a gate, I'm just going to start by treating those inputs as independent because I don't know whether they're positively or negatively correlated. So an AND of two 50% inputs is on with 25% probability, and so on." And then I get to the bottom and I say, "Okay, I think the whole circuit is on with 35% probability." Obviously, this whole thing is really stupid for a circuit of this size; you might imagine instead of being whatever, 10 gates, these were a billion gates to make it more interesting.

As before, you could start to notice correlations that change this estimate, right? Maybe I say, "Oh, these two gates have a common ancestor, so they're going to be correlated with each other. So the OR of them is less likely to be on than I would have thought," and propagate that through. And now I'm down to 28%. I can keep going in this way where again I can spend more and more compute, not to run more samples through my model, but to try and find additional considerations, like churn through a more careful deductive argument to get a better and better approximation.

One more example: imagine that I ask the question, if I simulate billiard balls bouncing around on a table—I start off putting them on the left half of the table with, say, velocities in random directions, wait $t$ seconds—what fraction of them will still be on the left half of the table? This is still just a perfectly well-posed computational question. One way I can answer it empirically is I just run a bunch of simulations. It's not so hard to simulate what will happen in the scenario, and then I can say over my thousand simulations, what was the actual number of billiard balls that were on the left versus right half of the table? I get some graph like this with some error bars that would shrink as I run more samples.

But I can do something very different where I say what I want to do is instead represent the distribution of potential configurations on the table and then deductively chug through time and say how should that distribution evolve as time passes. So I'm not just drawing a bunch of samples and inferring from those samples what happens on average; I'm representing what is going on and propagating that forward in time. I can use something like the Boltzmann equation, where I basically treat individual balls as independent from one another, and I say what is the probability distribution over the position and velocity of a single pool ball, and then I track how that evolves over time. I solve some equations, move those, and take sort of the same time steps I'm using on the left-hand side, but instead of tracking an individual sample, I'm representing the probability distribution in an approximate way. Then I get out some other graph which matches experiment pretty well. It doesn't match perfectly, and again, I could spend more time to account for more complex interactions. This does account for balls bouncing into one another—otherwise this graph would look completely wrong—but it doesn't account for phenomena like pool balls being less likely to be near each other because of the possibility of collisions.

Those are some examples. If there's any really high-level questions about what I'm even talking about, now might be an okay time to ask them, and I will start to dig a little bit more into what we mean over the coming slides.

I'm talking about mechanistic estimation. I'm going to start by defining what I mean by an estimator and then talk about what it means for an estimator to be mechanistic. I'm going to talk about functions $g$ that take two inputs: one is a function $f$ whose expectation I want to estimate, and the second is a computational budget saying how long I want to spend to get that estimate, and then spit out an estimate for the expectation over a random input $x$ for the function running in time $n$.

We're going to talk about the performance of estimators over some distribution $D$ of functions. If I fix a particular distribution $D$ over $f$'s that I might care about, we can define the mean squared error of a particular estimator $g$ on the distribution $D$ with a budget $n$ by sampling a random function, then running our estimator to get our estimate, comparing it to the truth, and then averaging the squared error. That's a very natural notion of how well does this estimator work. We're going to talk about that throughout the whole talk. This is all I mean by an estimator. There's all kinds of ways you could try and estimate a quantity, right? Sampling is one estimator; the mechanistic estimates I'm talking about are a second, different estimator.

The more interesting part is what does it mean for such an estimator to be mechanistic. I'm going to start by saying a little bit about what it means to not be mechanistic. As Jacob alluded to, what we kind of want to say is "don't use sampling." I want to be a little bit more precise by what I mean by that. We're not in fact going to completely avoid running samples through our model; we're not going to avoid any randomness. What's important is a certain kind of inference that we don't want to make.

This is going to be the prototypical example of what is not mechanistic: I compute $f$ of a single sample $x_1$. Let's say $f$ is a plus/minus one function. I see the answer was one. I infer, "Well, I don't know what's going on with $f$, but it's reasonably likely it has either a bias towards being positive or a bias towards being negative. I saw the coin come up positive one time, so apparently the bias is more likely to be positive. If I'm a reasonable Bayesian, I should infer that for some other unrelated input $x_2$, it is more likely that the expectation will be positive."

If I were to draw this schematically, what's going on is there was some unknown structure in the function $f$. We made an observation down here of what happened on a single input. We then inferred something about the unknown structure and used that to say, "Well, now I can make a prediction about another value $f(x_2)$ or about the average value." My unhappiness with this is because what I really want is for estimation to go causally downstream from what was actually happening in the function to making a series of deductive steps about what we should expect to see for the behavior of the function on inputs, rather than this move where I say, "I don't know what's happening, but I'm going to make a Bayesian inductive inference about it from an observation."

This is one angle on what we mean by not being mechanistic. You could just think of a mechanistic estimator informally as something that never makes this kind of move—that never goes causally upstream based on inferring unknown structure or properties of unknown structure from having observed behavior on a specific example.

If I want to give a positive account, things get a little bit messier. There's a bunch of algorithms we could look at. We're kind of building up this story. I'll talk about one rough approach. This is actually very similar to a Terence Tao blog post from like 15 years ago on the ABC conjecture, which is also where we ended up basically independently.

You can say what I want to do is reason deductively in the same way that you would in a proof, where you have some premises, you have locally valid reasoning steps, and you have some conclusions. The difference is that we are going to add a bunch of default presumptions. If we don't know what's going on, instead of saying, "Well, it could be anything," we're just going to make a guess. What do we mean by those guesses? For example, if I don't know anything about the correlation between two variables $X$ and $Y$, I'm just going to guess that they are uncorrelated. I'm going to say my guess for the expectation of $XY$ is the expectation of $X$ times the expectation of $Y$. That's what we did sort of in all those concrete examples we just talked through—they made some assumptions of this form.

Slightly more subtly, if some $Z$ captures everything we know about the correlation, then I might say, "Well, at least I'm going to assume that $X$ and $Y$ are conditionally uncorrelated given $Z$." Or a different kind of statement we could make is: if I have a bunch of variables and maybe I know things about their pairwise interactions or about their three-way interactions, but I don't understand the joint distribution very well, I'm going to assume that higher cumulants are zero. It's not super important if you aren't familiar with the idea of a cumulant, but roughly speaking, there's kind of a default way to impute the $n$-way interaction from the lower-order interactions, and we just assume that the higher-order corrections are zero in the same way we assumed that the correlation was zero. It's just a generalization of this first line.

In addition to making those kinds of moves, we're also going to say that even a deterministic quantity we can still treat as random, where the randomness now is not just sampling error over a particular $X$ that we drew; it also includes our uncertainty about what is going on. We're going to treat these symbols $E$ as representing our betting odds or our best guesses about some proposition, even if it may be the case that it was perfectly well-defined. Normally these symbols $E$ in the kinds of examples I discussed are going to involve both kinds of uncertainty rolled together. We imagine that there was some unknown input, and we're both uncertain about the behavior of that input, but also we are uncertain about structural facts about the model.

One way we could think of it that's in that direction is it's kind of like a proof where you have some additional rules of inference. I think it's less like writing a program, right? Because we're sort of saying, "Well, I want to make some estimate about this quantity. I want to make some deduction about what is going on with this model. I'm not going to re-represent the behavior." All of these are very slippery, though; none of them are fully satisfying. It's very hard to capture this as a proof system.

It's not necessarily so. If we consider the circuit case or something, in some sense the description of the circuit was a perfect description of what happens, right? If you feed in some random inputs, you have a great, beautiful description. In some sense what we're going to be doing is writing more and more refined estimates for the output distribution. And it's a little bit hard to say—there's going to be a sequence of those estimates that get better and better as we spend more time.

Yeah, basically. That's right. For the output, you could think of it as a formula for the output distribution in terms of the input distribution, rather than a formula for a specific output in terms of a specific input.

They are allowed to be, or—

In general, they will be. If you had more time and you were able to think about it, you'd be like, "Oh, actually these things cannot all be true." Similarly, if I look over time, the later iterations will contradict the earlier iterations in general, or they'll be hopefully just more refined, small changes—but they could be very big changes, which we'll talk about a bit.

I think as a Bayesian, you should think about everything we're talking about kind of lives in the prior. If you imagine you're sort of running your graphical model forward in order to understand what is even the distribution of these nodes and saying, "I need to do a bunch of logical work to compute what is actually my prior distribution over the outputs of the circuit," yeah, maybe you'll get there. But how do we do it efficiently, I think, is a big challenge. In general, the whole game is going to be: how quickly do these estimates converge, and how well can you make them converge?

Yes.

Yes. A way you could think of this, as we were just mentioning, is it's kind of like a way of relating: if you know some stuff about the input distribution, you want to infer some stuff about the output distribution, and how do you transform whatever we know about the input distribution to the output distribution? For this talk or for the things I'm going to mention, it's not important that the input distribution is uniform, but it is very important that we have a formal representation of the input distribution for this whole thing, because we're starting from statements about the input distribution and correlations and cumulants and so on.

In an actual application, a thing I won't talk about is you would be doing machine learning with data from the real world, and there will kind of be both dynamics: where you are being a Bayesian and you're learning what is actually true about the real world from what you've seen—what do I infer about the real world—while combining that with this kind of mechanistic analysis and saying how does the model, the operation of the neural net in front of me, transform stuff from the real world (which has these distributional properties I'm learning about) into stuff about the output behavior.

In the talk, I'm going to just focus on the case where the input distribution is known to try and isolate this one part of the problem. We want to understand: how does a neural network transform stuff about the input distribution into stuff about the output distribution? But fundamentally, there is going to be this whole part which is less mathematical, and a lot of difficulty comes up even when you're in the purely mathematical setting. It's extremely hard; you start to see a lot of these dynamics. So I think it's extremely fruitful to study, and if we solved it, I am quite optimistic currently—though there's room for debate—that that really would play nicely with the rest of ML, with the rest of being a Bayesian.

I want to say a little bit about the difference between a mechanistic estimate and a proof, because I do think a proof is probably what you should have in mind as the example we have of people performing a series of locally valid deductive steps in order to make really complicated conclusions where a human might not understand the whole structure of the argument, but a human understands the mechanisms—the actual steps that are being taken.

The big difference is that we are interested in best-guess estimates that change over time, rather than saying we have settled this question once and for all. In particular, most of the time we hope those estimates converge quickly to the truth, but it is always possible that we missed something big. When we do a proof, you have ruled out the possibility of a big update; we have not ruled out the possibility of a big update. That could either be because there was a structural feature that we missed—like we just missed the correlation or divisibility by two or whatever, so our estimate is off by a factor of two—or it could also just be, as far as a proof is concerned, there might have been a coincidence. A proof has to rule out the possibility of some coincidence or unexplained conspiracy which causes the statement to be false.

An example that makes this really vivid is suppose that $f$ is just some very chaotic function which takes values plus/minus one roughly equally. It's very hard to say anything about that function. In general, it's going to be really hard to prove that that function is balanced over some large input space. A proof would need to rule out the possibility that, by coincidence, the function happens to be one a lot. Generally, to do that, you actually would have to find structure. If this object was extremely unstructured, it is very hard to do a proof, but it is very easy to make a good estimate. In some sense, the whole way your estimate could be wrong is if there's structure you haven't noticed. In the absence of structure, we expect our estimates to be quite good.

There's some very similar character between them, but there's also some very big difference which is extremely important to why I believe it is possible to make mechanistic estimates about complicated systems, despite the fact that I am extremely bearish on the possibility of proving things about machine learning systems. Basically, because of this kind of issue: they often will involve some chaos where actually you would have to find a bunch of new structure in order to rule out the possibility of coincidences.

As we mentioned, a really big question here is: how fast do these estimates actually converge? In particular, how quickly should we expect the mean squared error of an estimator $g$ on a distribution $D$ to decay as a function of the budget $n$? I'm going to set up some conventions for the rest of this talk. I'm always just going to normalize my functions so their variance is one over both the randomness of the function and the randomness of the input to that function. You could think of these as being plus/minus one functions if that makes it a little bit easier to think about. And I'm going to assume, at least for this slide, that you can compute the value of these functions in time $t$. I'm not going to specify exactly what the model computation is; it's not that important.

Then if we do this Monte Carlo estimate—the empirical estimate on the left side of the slides—that always decays like $1/n$. If you have $n \times t$ time, enough time to run $n$ samples through the model, you're going to get $1/n$ error. When we say that something has a closed form, what I would interpret that to mean in this context is usually that you have exponential decay, where I can spend time and just keep turning out more and more digits of the estimate. I think it's natural to ask: most things aren't going to have a closed form, right? Even if we understand a system well, there's no reason the answer should have a closed form. How well should we expect an estimator to work?

I'm going to talk through an example which is roughly the worst case for mechanistic estimation we believe: the behavior I mentioned of these chaotic functions. Suppose that this distribution is a random, unpredictable function from $k$ bits to plus/minus one. As you might imagine, I take SHA-2, choose a random salt, and then read off the first bit and interpret it as plus/minus one. The best mechanistic estimate for this function, if the hash function is working as intended and has no structure you can exploit, is you just evaluate it on $n$ points. It doesn't even matter what the $n$ points are, and then you say, "My estimate for the value is the sum of the points I evaluated divided by the size of the domain."

One question you might have is: that looks a lot like sampling. I kind of just ran some samples through the model. Why do I think this is a mechanistic estimate? Go back to this picture of a causal diagram. What's going on here is we are evaluating this function at a bunch of points, and we are calculating the expectation of the function as just the average of the points at which I evaluated. In this case, I think this is the right causal diagram: there's not hidden structure above these nodes which I'm making an inference about. Actually, all of these nodes are just independent from one another. The value of my function at each point is independent plus/minus one, and so this really is the right way to draw the causal diagram: you should think of those as independent nodes, and the expectation of the function is causally downstream from them.

I talk about this because people have mentioned you probably do need to run inputs through functions in some cases. The thing that distinguishes it from sampling is that we treat the other inputs as independent. We don't make an inference about the behavior of the function on other inputs because of the ones we observed, which is why this coefficient out front is the size of the domain instead of one over the number of samples we evaluated.

What do you mean, the average of the inputs?

Well, this will have lower error, right? Suppose I draw one input and I run a single input through. The point is I should basically output zero. I don't want to average the inputs, I don't want to average the outputs; I want to say zero. If I'm saying what is the average value of SHA on 100-bit strings, if I saw one string, the main thing I should say is it's balanced approximately.

That will have a higher error. As a mathematical fact, let's just draw graphs, and then we can talk about it in the context of results.

If I do Monte Carlo and I ask a function—this is on a domain of size eight; the graph's going to look exactly the same if the domain is larger, it'll just be scaled differently. On the x-axis is the computational budget, like how much time do I have to evaluate my hash function, and on the y-axis is the mean squared error. The performance of Monte Carlo looks exactly the same for every function of variance one: it always decays like $1/n$. In this case, what happens is if I have time to evaluate eight inputs, my error goes all the way to zero. If I have time to do zero inputs, then because I'm outputting zero all the time, or because I'm starting from something very close to zero, I get very low error. This is basically the unique algorithm that has this curve of loss.

If I evaluate one point, it actually really is right that I should say, "Oh, if I saw a single one, it's now a tiny bit higher estimate." And it is important that it was the value of the function; it only really becomes important once I evaluate a significant fraction of the domain, though. If there's a trillion possible inputs and I've only seen a hundred of them, I should pretty much ignore them, right? I'll be at this part of this graph where it's very close to just outputting zero still.

Then it will get destroyed by random sampling, right? If I think it's zero, but actually, as it turns out, 52% of the things are up, then this graph on the right is going to look horrible. It's going to stay at extremely high error, much larger than random sampling, for a very long time. What we need to happen is if there was a bias, that's kind of one of these considerations where, if you missed it, you can really be in trouble. In order for your estimates to be good, we have no guarantee that the estimates are good; we have to have actually found the important considerations, which I'll talk a lot more about over the coming slides.

In some sense, that's really what we're interested in for a neural network: saying, can we find enough structure in the neural network that actually our estimator is good rather than being worse?

That's right. Frequentism is very robust: you're like, "I don't know what's going on, but I know this is going to work all the time." Here, this is like an anti-advertisement, but we're maximally non-robust to black swans. We're trying to really understand what's going on. If we missed anything, our estimates are going to be extremely bad, but if we got everything, our estimates are going to be significantly better than doing random sampling. Part of why we care about it is we're going to treat this as a test for: do we understand how a neural network works? If we just trust our understanding and lean into that and try to make these really good estimates, if we really understand how the neural network works, these estimates should be great. But if we're missing anything, these estimates are going to be catastrophically bad. Fortunately, for the loss, we're going to be able to evaluate that. We're going to be able to say: do we do a really good job in fact?

This is the discussion we were having before. If there's a small domain, eventually you've just seen all the points. It was a little bit weird of a thing for me to say because you're imagining this giant domain and you've seen 20 points. There was this $2^{-k}$ in front where, if you evaluated 20 points out of a size of a trillion, you should be like, "Well, I take my actual average, I scale it down to one in a trillion," which becomes important over the long term once I've explored the whole domain, but not important until I've explored half the domain or something like that, right?

Okay, so I'm going to state a formal conjecture and an informal conjecture that captures some of what we've been talking about. Let's take this distribution $D$ to be uniform over some parameterized family of functions, so it's $f_\theta$ for a uniformly random $\theta$. As before, I'm going to assume each one of those $f_\theta(x)$ is computable in time $t$, and the variance over the choice of both $\theta$ and $x$ is one. That's just kind of this normalization question.

Then we have the conjecture that there exists an estimator such that the sum of its error over all the different budgets you could run it on converges to the cost of evaluating the function once. This is contrasted with, if we take the same quantity—we call this the error sum for sampling—it diverges logarithmically: $t \log n$. We're saying we can asymptotically improve over sampling by this factor of $\log n$.

Of course, the thing we really care about is not this. As a computer scientist, the claim that sampling is never optimal for any distribution over estimation problems is actually a very surprising statement a priori. But the real reason we care about it is because of this vaguer conjecture: we think there is always a mechanistic estimator that has this property. I can't state that conjecture formally because all I've been able to do is handwave about what it means to be mechanistic. I do like that we have a nice non-trivial conjecture that follows from these intuitions, but we really mostly care about getting a mechanistic estimate that's able to beat sampling. Our conjecture is that you can do that no matter what the distribution of functions is. I would just note that this error sum property implies that you're better than sampling at least up to a constant. My guess is that that constant is less than two.

It is often the case for many problems that there'll be some more sophisticated version of sampling which gets lower error than sampling. You can then apply the same conjecture not to the original program, but to your new program that is slightly more sophisticated and has lower variance, and you do then get some reduction. But the main relationship is that is not always possible, or there's not always an algorithm of that form that works. Maybe our formal conjecture could be true without actually giving us any insight into the vaguer conjecture. That would already be a very non-trivial claim if there's control variates for every distribution over estimation problems. We also often end up doing quasi-Monte Carlo methods, which are just normally going to be better than Monte Carlo, and so there are in many domains just better things to do than random sampling.