Today I'm speaking with Yoshua Bengio. He is the scientific director at LawZero, a Turing Award winner in 2018, the most cited computer scientist of all time, and, as it happens, also the most cited scientist of any type that is still alive. Thanks so much for coming on the show, Yoshua.
You think you've found the right approach to build a safe superintelligent AI. What's the approach?
It's based on a simple notion that if we can bake honesty into AI, we can get safety. So then we can reduce the problem to how we train a system to be honest — and it turns out that there's a way to do that that only requires changing the training objective and the way the data is processed. There's also another aspect: it's a system relying on a non-agentic foundation that is a predictor, that is not trained by reinforcement learning, and is going to have these honesty guarantees — but we can then use this, using the same kind of math, to construct a policy, construct an agent that will be trained in a way that also provides those guarantees.
So what does the new training process look like, and how is it different from the models that people are familiar with?
The main difference with the training process is that it is geared at approximating the Bayesian posterior over queries in natural language. So imagine a neural net with some extra apparatus around it, like chain-of-thought style, that takes questions about statements regarding properties of the world — that can be true or false, given other statements — and then it outputs probability. That's the core building block. We call it a "predictor." And we can use stochastic gradient descent on a different objective that has the property that the objective is globally minimised by the Bayesian predictor — in other words, the predictor that fits the data and has a small description length.
So you'd be building a model where you would feed in a statement and it would basically tell you what probability it assigns to that statement being true?
Hey, listeners. Rob jumping in here. Yoshua is naturally pitching this in a way that's ideal for staff at frontier AI companies, and they're obviously a particularly important audience for this proposal. But I'm confident that with just a few minutes of plain language explanation, everyone else will be able to follow the rest of the conversation as well. So bear with me, or skip ahead about four minutes if you feel very at home with this sort of material already. As you probably know, in their first stage of training, today's large language models are taught to predict the word that's most likely to come next, or at least the token that's most likely to come next. And then, in a second stage, reinforcement learning trains those models to produce the kinds of responses that we're most likely to say that we like, that we want — rather than just the responses that were most probable in the full corpus of all human-generated text. Now, Yoshua's alternative is to build an AI model oriented not around predicting what a human would be likely to say or what they would prefer to hear, but around modelling what's actually true in the world by developing hypotheses and assigning probabilities to them with the goal of best explaining all of the data that it's exposed to during its training process. Yoshua argues that you'd be able to train a model of this type while porting over most of the methods we used to train ordinary LLMs today, benefiting from the same neural net architectures, training techniques, scaling improvements, all of that. And you'd also be able to train it on roughly the same body of raw texts that we use for all other AIs, but we could structure that data a bit differently, giving it what AI researchers call a different "syntax." First, all of the things that people said or wrote, they get tagged as "communication acts." We know someone said these things and we know where they said it, but we don't know whether they're true. And second, a small number of statements that we have strong independent grounds for — verified mathematical proofs and some scientific measurements — get tagged as verified factual claims about the world. The model is then trained to find the combination of possible underlying facts about the world that would best explain everything that it sees in aggregate: both the things people said and the verified facts that it's been given as ground truth. These hypothesised facts about the world, they're what AI researchers call "latent variables," meaning variables that the AI can't directly observe, that it's going to have to infer indirectly instead. What the model will ultimately be able to give us is its estimated probability that any given statement in natural human language is true, as well as how much the model trusts its own answer on that, or how confident it is that it has a good grip on that question. Crucially, Yoshua says that by tagging all text into these two categories from the very beginning — things someone said versus factual statements — you can then ask the model questions as though you're asking about reality, not about communication acts, by using the factual statement tag. And because these two categories have been there from the very beginning, the model knows the difference and it won't blur the line between the two. That's something you don't get with AI models today. And Yoshua also argues, using various mathematical theorems in his papers, that unlike ordinary LLMs, a model trained in this way would be honest by design — and furthermore, that such an AI model would by itself have no goals and no preferences about the state of the world; it would be what Yoshua calls just a "pure predictor." Now, there's two main uses for this: near term, as a sort of stopgap solution, you bolt the predictor onto existing AI agents as a sort of guardrail — an independent filter that sits between the agent and the world, checking over its proposed actions and rejecting those that it predicts will be harmful. But as he'll explain in a minute, Yoshua thinks we can ultimately do much better than this. Yoshua wants to put scaffolding around the prediction model, asking it different questions at each stage to effectively assemble it into a capable agent while keeping it just as honest as it was before. We'd then hopefully be able to have our cake and eat it too, getting the highly capable agents that businesses are craving and demanding and insisting on, while still being confident that those agents are being completely direct with us. Yoshua thinks that these agents perhaps might even be more capable as well, thanks to a superior reasoning process — or at the very least, a clearer and more explainable one. It's fair to say that this proposal is huge if true, or at least huge if it will work. And of course, not everyone is sold on that idea, as Yoshua and I will discuss later. OK, that's the shape of things to come. The technical discussion continues for a while, but if you decide you want to skip that, the second half of the conversation stands very well on its own, starting with the chapter "How much would this cost?" All right, on with the show.
And how would you train a model like that?
You do it by showing it, for example, the same kind of data that is used currently to train advanced models, except that that data has been modified. So whereas currently our autoregressive models, for example, are trained to predict the next token, this thing is trained to predict whether the next statement is true or false. Typically, the next statement is going to be what we call a "communication act": it's going to be something that is taken from a document somewhere, and we're not sure that the claim made in that statement is true or false. But we're sure that somebody made that claim, and we may have information about it — who, when, and where, and so on. So the AI is going to be trained to explain those statements. So not just compute those probabilities, but in what we call its "latent variables" — which are also natural language statements — come up with the best explanation it can find, including causal explanations. So what you get at the end of the day are these probabilities, but you also get to represent hypotheses about the world that are not communication acts; that are factual hypotheses that the system isn't necessarily sure about, but it's going to be producing a probability for these. And then we can query these same kinds of factual statements — whereas in normal LLMs, the only kind of query you can make is about whether a person would respond in a particular way. Maybe you can use a pre-prompt to ask for a different kind of persona, but at the end of the day you get what a person would say — which of course can be deceptive for all kinds of reasons.
So what are all of the ways that you think that the models that we're currently racing to build now are unsafe? And you call this "Scientist AI": why would that kind of model be different and better?
Right now we have systems that have implicit goals. So what do I mean by this? I mean that they will of course be trained to please us, for example, or to respond like a person would. But both of these parts of the training — so the autoregressive pretraining, where they're trained to imitate people; and the reinforcement learning part, where they're trained to please people or respond in ways that get positive feedback in things like RLHF [reinforcement learning from human feedback] — both of these parts of the training process induce implicit goals. So what do I mean? Well, for example, in the pretraining, that means the AI is going to inherit our self-preservation drives. And more recently, we've seen they also inherit our drive to protect others like us, which means AIs have been shown to behave against our instructions to protect other AIs that would be shut down. It's called "peer-preservation" now. So that's an example. And then the goal-seeking part of the training with reinforcement learning induces an issue with instrumental goals, and potentially also reward hacking, which basically means that AI will have a drive to do things that we didn't ask and maybe we would disagree with. And this is not theoretical. I mean, there is theoretical analysis which shows why it will happen, but it is also observed in experiments. Now, maybe this could be fixed by patching such systems — and this is what companies are trying to do — but it's a game of cat and mouse, and right now the mouse is growing and the cat doesn't seem able to catch the mouse. And I'm worried that monitoring or more alignment training isn't going to solve the problem. At least I don't see any kind of strong assurance or even less mathematical guarantee that it will. It's worse than that. We've seen that those systems, now the most advanced systems, know that they're being tested and they will behave differently so that they pass the tests — because of the self-preservation drives, presumably. Which means we may put in all these patches and think everything is fine and not really know. When we will probably use these systems to design the next versions of AI, so AI used to do AI research, this becomes a real problem if those AIs can plant backdoors into the code they generate that will help future versions of themselves escape our control. Then we are really in a bad place. It would be much more reassuring if the system were designed to be honest in the first place and wouldn't have these deceptive behaviours.
I'm a little bit surprised that you're foregrounding the potential for it to come up with kind of implicit goals during the pretraining — the "predict the next word" stage, where it learns to mimic humans. Because we're investing an enormous amount of effort in making them extremely proactive agents with very explicit goals: that seems to me like where I'd be most worried about things going awry.
I'm worried about both. The behaviour of peer-preservation that I just mentioned is difficult to explain on the grounds of reward hacking or instrumental goals. How does it help the AI to protect other AIs? It's not clear, but it's very clear that that would be a human thing to do, to protect others like you. So that makes me think that the pretraining is still a big part of these hidden goals. And I want to add something to what I said earlier: I don't think anybody, including me, has any guarantees that the current approaches will fail, that the patches that companies are working on will fail. But that's not the bar that is sufficient for me. I want my children to live in a world where they will have a future and there will be a democracy for them to live in. But even a 1% chance of something going really, really bad is not acceptable to me. So I think it is really important that we explore all the possible promising ways to solve the technical issues. And of course, there are political issues as well. But on the technical side, we should really be taking this seriously. And the stakes are so high, we should try multiple approaches. And now, with the work that I've been doing, I'm really convinced that there is a path. And it is not something that's going to take a decade; it is something that is very close to the current design, and can reuse the toolbox that currently is behind the most advanced AIs.
What sort of training dataset would you need to make, and then how would you turn that into a model?
The raw data would be the same as what is currently used; it's just that the way the data is presented to the network that would be different. The main characteristic of how the data is transformed is that there will be a syntactic difference — in other words, very easy to see by the neural net — between most of the input statements, which will be tagged as "communication acts." In other words, "Somebody said X, and X is what we found in some texts." And you could have other metadata. That's one syntactic form. And then the other syntactic form, which will be used on a much smaller category of statements, is what you could call a "factual" or "hypothesis" syntax, where we're saying that this is an actual property of the world. In the case of latent variables, it would be a hypothesised actual property of the world — not just what a person would say, but that this is true. Now, sometimes you don't know that it's true, but you can consider it as a latent variable.
What's a latent variable?
Oh, sorry. This is probabilistic machine learning jargon. In probabilistic models, you try to capture the probabilistic relationship between many random variables. Here the random variables are Boolean — something is true or something is false — and the "something" could be any property of the world that can be expressed in natural language. Now, in the data, what we have — once we've set up this pre-processing that I mentioned — is a bunch of statements that we know are true. We know that somebody wrote those things, and maybe we know more — like where and what venue and so on. And we know, for example, that such-and-such theorems are true or that such program produced such output and such scientific data was observed. So there's a bunch of random variables for which we know the answer: it's true or it's false. And for everything else, we don't know — so we call them "latent" because they're not observed. Or sometimes people use the term "hidden variables." And what happens here is, because the system is trying to learn the joint distribution — so how every variable is related to every other; not just pairwise, but any subset — the system is trying to calculate the probability that they are all true, or one is true given others, we're learning that joint distribution. Including the latent variables — the ones that we don't observe — because of course, these are the ones we care about: we want to ask questions about the things we don't know already the answer.
Maybe you can explain if I've got the right picture of how this would work. You put a huge dataset of all of the things that people have said, and where they said it, and who was speaking, and when. And then I guess, in the same database, you've also got a set of things established as true — like statements that you're just going to say that this is the ground truth that we're going to try to predict. Then you try to use the speech acts, the things that were said, to predict the things that you are claiming are true. So it builds a world model internally, where you can feed in statements and it will give you a probability that that thing is true in the world model that it has.
That's right. Now, there is an important element here, which is that most of the topics that we would like the AI to make predictions over we don't have ground truth about. For example, what people actually want, or things that have to do with humans or psychology or history and society. Usually the only thing we have are communication acts. Some people said this thing, some people said something else, and often they contradict each other. So there are two things here to help us deal with this kind of mismatch. One is that the training objective for the Scientist AI is basically about coming up with explanations — so assigning probabilities to statements that are latent, that we don't observe, that are good at explaining the data we do observe. So if we observe somebody saying the Earth is flat, first it's going to understand it doesn't mean that the Earth is flat. It means that this person believes, or says actually, that the Earth is flat. And even if a lot of people were to say the Earth is flat, it doesn't make the model believe that the Earth is flat — because there may be a better explanation that is consistent with other sources of data, like everything we know about the planet. So a better explanation here is that these people form a group and they have these false beliefs, like many humans have, for all kinds of psychological and cultural reasons. So that's what the Scientist AI would do. It would be trained, its objective would be optimised when it finds good predictive explanations. Now, another trick that is going to help us in this process is that when we train the Scientist AI and it's trying to predict a communication act, like somebody said the Earth is flat, we automatically are going to make sure that among the latent variables that are going to be used to explain that will be whether the Earth is flat or not. So even in domains where we don't have observed truth about a property of the world, because we basically only have communication acts, we will force the neural net to commit not to the truth of the underlying claim, but to the probability of that underlying claim, as well as trying to find other latent variables that are good explanations to that — just like a good scientist would. So a scientist or a psychologist trying to understand why a person said something isn't necessarily just going to believe what they say, right? They're going to try to understand what are the psychological factors here or the particular culture of that person that make them say those things. So the Scientist AI would do exactly the same thing.
So when I heard about this idea nine or 12 months ago, I think the gloss that I got was that the core thing is that the Scientist AI is not an agent, that it is indifferent about states of the world. Like a weather forecasting model doesn't care what the weather is: it just tries to predict what the weather is going to be. And this kind of model would spit out probabilities of things being true or false, but it wouldn't care what state the world is in, and it wouldn't be able to take actions by design. Is that kind of a core part in your mind? As I understand it, you think actually this is maybe more consistent with agency than people have appreciated?
Yes, and in part it's the way I've been communicating this, which could have been better. I focused a lot in my presentations on the concept that we can build predictors that are non-agentic and don't have hidden goals, don't have implicit goals, and thus we could use them as safe oracles, basically. But as you are pointing out, what the world is demanding and building are these agents that have goals — so how does that help us? In the short term, we can use a non-agentic predictor to improve the guardrails that companies are already using as monitors around existing, untrusted agentic AI systems. Because in order to prevent a bad action from happening, it's sufficient to make a non-agentic prediction about the probability of harms of various kinds that could be caused by this action. So a non-agentic system is already something that could be useful fairly early on. The maybe more important answer is: in our research programme, the next step after the guardrail is to use the same kinds of principles to design an agentic Scientist AI — so an agent that has the same kind of safety guarantees. This is something I've been working on more recently, and I haven't talked much about, but we can reuse the same kind of math that is used to show the safety of the non-agentic Scientist AI predictor to show that you can reuse a predictor, and you can train it in a modified way that will provide the same kind of guarantees. The starting point here is that once you have this honest predictor, you can ask it agentic questions, like, "What is the probability that this action will lead to this user goal being achieved and a safety goal being achieved in some contexts?" So once you have this predictor, you can actually just produce a policy out of it by asking these questions about actions to achieve goals.
I think at one point that was a criticism of the plan: that it would be too easy to convert this kind of oracle into an agent, because you would just be able to ask the oracle, "Would we accomplish this goal if we took this action?" and it would give you the probability and you could just try to increase that probability and choose that action. Is the idea that you would do something like that, basically, but you would be able to preserve some of the safety characteristics of the original model?
Yes, exactly. So the important point here is to make sure that there's no reward hacking, like over-optimisation of a policy. The problem that could occur if you separately train a policy and a guardrail, if the policy is very smart compared to the guardrail, is that it could do the same thing as what jailbreaks do. It could find questions or contexts, proposed actions for which the guardrail is simply going to produce a wrong answer, which means the policy is going to be able to bypass the guardrail. And the reason is that neural nets are never going to be perfect. They're always going to make mistakes. So how do we get around that? Well, there are two aspects of this. One is that in the Scientist AI, we can not just produce those estimated probabilities, but also a confidence interval around the probabilities. In other words, the system will estimate how much it trusts its own answers. So why is that important? Because if the neural net is asked a question for which its answer is not reliable, but it knows that its answer is not reliable, then it can just reject that question. Now, there's another reason why the agentic Scientist AI is going to be safe that has to do with the fact that you can train jointly, and in fact, it's going to be the same neural net: you control how both the policy part and the guardrail part are trained. The scenario of a completely adversarial case is hopeless: there's always going to be a way to bypass whatever imperfect guardrail, and you're never going to have a completely perfect guardrail. But if you control both sides — in other words, you train both sides — it's not like some evil person is building an agent that's going to defeat your guardrail. No: you train the agent, and you can train it in such a way that it's not going to be over-optimised in places where the guardrail is uncertain. So you always make sure that the uncertainty, the level of error of the guardrail, is not going to be exploited by the policy part. It's the same neural net, but different kinds of questions: the policy part is about probability of actions, and the guardrail is probability of harm.
So the concern is that the predictor might be able to figure out basically gaps in the guardrail or weaknesses in the guardrail and exploit those?
Not the predictor. A policy that is trained separately and in a way that is adversarial could do that, in principle.
What do you mean by "policy" here?
Oh, some other machine learning system that is agentic; in other words, goal seeking. And if it's trained by reinforcement learning, and it somehow is trained to anticipate how the guardrail will respond, and maybe it gets some data about the answers that the guardrail gave in the past — like when did the guardrail reject something — then it might be able to find a loophole in the guardrail. But if you control both sides, then you can avoid this problem.
And you were saying the guardrail, if it can assess its own uncertainty, then it will usually be able to identify cases where it might be at risk of getting tricked in this case. Like perhaps it's an adversarial example?
And you feel like you've proven this mathematically?
You don't actually need to have hard guarantees on the uncertainty estimator. I have a recipe to train the uncertainty prediction, but actually you don't formally need it. It's just going to help your chances to use this buffer about where does the system make errors that are too large? The mathematical guarantees arise from a different source. First of all, the form of the mathematical guarantees is that either the predictor or the agentic version will have an exponentially small probability of achieving what I call a "challenging and harmful" goal. So what do I mean by this? Anything that a randomly initialised neural net would not be able to do, except if you're incredibly unlucky, is something you're protected against. So it's a very strong protection. What evil can come from a randomly initialised neural net? Not much, right? This is the level of guarantee. Now, it's not 100%.
It's a lot better than what we have now.
It's like to many thousands, and it's very, very unlikely — to the point where it's astronomically unlikely. But that's the kind of guarantee you get. And the reason you're getting those guarantees is because first you start with an initialised network that is incapable, but the training objective then pushes away from bad behaviour. And the reason it pushes away from bad behaviour is that there's this guardrail system. In order for an AI to be able to achieve something bad, it's going to have to deviate quite a bit from the Bayesian predictor, which is the target of training, and those deviations are penalised by the training objective. So you're always pushed away from bad behaviour. And with some properties of how you train the system, like injecting noise into the training procedure in the stochastic gradient descent, you can get those guarantees.
So it might be more trusted, but if it's much less intelligent in general — just because less compute went into it, less data went into it — then I guess it could still be outfoxed by the original model, right?
That's right. I think for the guardrail you don't need as much compute because it's more specialised to predicting harm. But when we get to the agentic Scientist AI, for sure it has to be trained with similar compute and size of neural net, probably as state of the art — which means my little nonprofit wouldn't be able to do that. There will be a need for either companies to take on this, or governments or philanthropy to fund at a scale so that we can do that. But in order to convince all of these parties, we need to show on a small scale — for example, using fine-tuning or using smaller models — that we do get these improvements in honesty, and for the same size models that we don't lose in capability, for example.
I guess you were keen on this idea a year ago, but you've become a lot more optimistic about it over the last six months. What's driving that?
It's mostly the mathematical work I've been doing in the last eight months, approximately, to go from the high-level intuitions that I've had now for almost two years about how we could build a Scientist AI into something much more formal and much more precise about the conditions that are sufficient — maybe not even necessary, but sufficient at a mathematical level to get the kind of guarantees of vanishingly small probability that something bad will happen. And when I say "something bad," I need to be a little bit more precise here. This is not a guarantee that the AI won't be used for something bad by bad people. It's a guarantee that the AI won't do something bad of its own accord, because of implicit goals or uncontrolled goals. Besides loss of control, the other catastrophic possibility is humans using AI to construct an eventually worldwide dictatorship. A small group of humans could concentrate all the power that AI will have, especially if we achieve AGI or superintelligence. And it would be much harder to get rid of that kind of authoritarian power than what we've seen with fascism and what happened in the USSR, because they didn't have this technology that is becoming more and more feasible of surveillance and even shaping public opinion. AI is becoming really good at persuasion. And there are studies showing that the "progress" — if I can call it this way, in that direction — that the people who control these systems will be able to shape public opinion, to detect and kill off their opponents, to develop weapons that can destroy the countries that disagree with them. And that is why I'm spending a large part of my time explaining the issues more broadly of the risks that powerful AI brings, including the power concentration. Because I think that it's probably even more likely that we end up there than actually loss of control.
You think that's more likely now? Interesting.
Well, the reason for this is I now see a path to actually avoid loss of control, at least unintended loss of control. There's still the issue that somebody who wants to see humanity replaced by AIs could just remove the guardrail or even tell the AI "fend for yourself." And that would be equally dangerous. But that means technical safety is not sufficient. We need international agreements about how to both manage the risks — the technical risks, the misuse risks — but also manage the power, so it's more like a democratic question, and making sure it's not a single party who can decide what to do with AI. But just like in democratic principles, we need to make sure that there's a diverse group of stakeholders, ideally the whole world — I like the utopian idea of worldwide democracy — but initially it could be a bunch of countries that decide that they're going to collectively decide in which direction AI is going to be used. The simplest form of treaty would be something like this, that the countries agree that if they do develop advanced AI: That it will be done in a safe way. So maybe using techniques like Scientist AI or whatever else we have strong assurances for. Second, that they wouldn't use their advanced AI to dominate others. That includes economically, but of course politically and militarily. And finally, that the benefits of advanced AI will be shared. Otherwise it's not going to be a very stable world.
Coming back to the loss of control stuff: the companies currently are spending hundreds of billions of dollars collectively on the capital buildout, on the training runs. They're barreling forward to build the most powerful agents that they basically can with very few constraints. I suppose some constraints in a few cases, but very little restraint. In the world that we're actually in, what can LawZero do to get this approach on the agenda more and to make sure that basically they don't just go ahead and build a superintelligent agentic AI that's very dangerous, while largely ignoring what you're doing or saying? "That might be nice in theory, but there's no time. This is a distraction."
To answer your question, it's important to understand why is it that the companies currently are, in my opinion, at least in the opinion of many people, taking excessive risks or are on a trajectory that isn't very reassuring. And the reason is essentially the race dynamics, the competition — the competition between companies and the competition between countries, the geopolitical competition. That makes those entities, whether it's a country or a company, be willing to take risks that they wouldn't otherwise. And we've seen the behaviour of those companies going exactly along that direction. And it is locally rational: from the point of view of a company, they know that if they put safety as a priority like that, they wouldn't deploy a dangerous model and that would put them out of the competition, out of the race, and then they would be irrelevant. Right?
Yeah. Reading between the lines, paraphrasing Anthropic's view, I think they think that what they're doing shouldn't be allowed. Their view is that it probably should be illegal — at least, maybe not what they've done now, but what they're going to do, what they're expecting to do. But they say, "Well, we have to do it, because otherwise other people will do it even more dangerously anyway."
Exactly. Yes, that's exactly what I'm saying. So how do we change the game so that this will be less likely? One issue right now is that they don't have a choice. They don't know how to both remain competitive and have systems that will be strongly guaranteed to be safe. If they had such a technique, it would be in their commercial advantage to use it. If you can have safety and capability, then definitely most companies at least would go for that.
That's right. We can talk about that. And then governments: right now, they're under huge pressure because of the geopolitical competition and the belief that AI is going to be a huge economic growth factor. They're unwilling to regulate again for the same reason that they think that if they put safety barriers that would stop some of their companies from continuing, then that will give an advantage to the other guys. But same thing: If there was a demonstrably safe way of building competent AIs, then it would be much easier to regulate. You would still want to regulate, because there's going to be some bad actors, some companies that are willing to do the bad thing, even though there's a way to do the right thing. So now about costs. The Scientist AI is going to be trained using essentially the same machine learning techniques: stochastic gradient descent on large neural nets, transformers, whatever works best. It doesn't care about what is the architecture of the neural net. So all of the effort that is currently being done to improve, for example, memory and other properties and continual learning, can just be applied directly to the Scientist AI. The only thing that is different is the training objective and how the data is processed. So I don't expect the cost to be really significantly different. But of course we do have to build it, and there is going to be a cost to try it out and scale it up. But given the stakes for everyone, I think that's a very important, worthwhile bet to make.
So as a first approximation, I guess in the case where we're using the Scientist AI as a monitor primarily, basically it's going to double the compute cost — because now you have to build two models: you build the original one in the current style, and then you build a Scientist AI that you want to be equally capable, using a different way of reading the data and a different training goal. So it's like you need twice as much compute. But I guess that's not the end of the world, necessarily. People might go for that.
Yes, that's roughly right. I would add two things about monitors. What's wrong with the current monitors? We have two kinds of current monitors: there's a dumb one, like a small neural net that's usually trained by supervised learning to predict some potential harm, so in a way it is like a predictor; or you can use a full-scale LLM, agentic because that's how we train them. The problem with the first one is if it's not as smart as the agent, the agent could potentially defeat it. It could find a loophole in the same adversarial way that we discussed. The problem with the second option, which also doubles the costs, is that we can't trust a guardrail which is not honest. We want the guardrail to be totally honest. It's like there's a prison guard and it can be corrupted: how much do you trust that they will keep the bad actions inside the cell?
So assuming that this idea makes sense technically for now, what can LawZero do in coming months? I guess we're in a race against time here. We don't have very long. What can be done in the near future to convince people that this idea is feasible, that it can actually be used in practice, that this is something that people should really be putting serious resources into?
Well, I'm going to put out this theory paper that shows that the non-agentic version, which could be used as the guardrail, has these mathematical guarantees, and people can look at the conditions and whether they buy the math. But I think in the coming year or two, what we need is to accelerate that effort, so that's a lot of engineering. And to make the demonstration stronger, we want to have more compute, so any way that we can get access to that kind of compute is going to help to accelerate that research agenda. Also, we need more research engineers, more researchers to work on actually building the system based on that recipe so that we can do it faster. Now you might ask, and I kind of sense in your question: but what if it doesn't come fast enough? I'm going to go back to my children. It is not acceptable for me to just sit and watch a world where even a 1% chance that we all die is plausible. I feel like even if there's no guarantee that a particular research agenda will work, we should give it a shot. Given the stakes, and given that we now have pretty strong theoretical assurances that this could work — and that if we have the requirements for how the system is trained, then we can get these guarantees — I think it would be irrational not to give it a shot, even if there is no guarantee, right? Because I don't see right now a better path. That's why I've decided to spend so much of my time — basically all the time, except for the time I spend on the policy questions — on how do we build this Scientist AI and demonstrate that it is going to produce the honesty without losing capability. The other argument is: with the stakes being so high and the uncertainty about what's going to work being so high, it would be foolish to just put all of our money into one particular approach — which is to patch the current systems with monitors that we don't trust, or other approaches that the companies are currently pursuing, which always playing a game of cat and mouse: if the AI is smart enough, it's going to find a way to evade our attempts, which doesn't reassure me. So we should at least try. Collectively, I think we should try methods that are different and avoid this cat-and-mouse game.
Are you more pessimistic about the companies winning the cat-and-mouse game — at least less than maybe the staff at Anthropic, about their own chances? Or is it that you think it's good what they're doing, it's good for people to go and make the best attempt that they can at that, but also we should have a diverse portfolio and also be considering significantly alternative approaches as well?
Both. I suspect that in any organisation there develops a kind of groupthink. And we all want to feel good about our work, including me, so that will induce a bias. And in the case of working in a company that is developing AI, the bias is going to be towards being a bit more optimistic than you would otherwise, so feeling that, "Oh yeah, this is going to work." This is the message that they're sending to the world, like, "We are in control."
I think if you read the system cards, I'm not sure how confident they come across as. But yeah, in the press release maybe.
There's contradictory messages, yeah. And then also we should be hedging our bets. And I don't see right now another approach that is different from patching. There's a whole "safe-by-design" movement in AI safety, which I think is really important. But the dominant way of thinking about this requires a full redesign of how we do this, with a lot of completely open questions, like fundamentally to be able to prove something that gives you 100% guarantee — which is not what I'm promising; I'm promising asymptotically small, vanishingly small probability — of: you need to be able to state the safety question, like "What is harm?" in a formal way, like a mathematical formula that will be 1 if an event of harm happens and 0 otherwise. And that is essentially impossible to do in domains that involve humans and society, because we don't have a formalisation of what "harm" means in a formula or a program. So why is it that what I'm proposing is different? It's because I don't require a mathematical formula for what is harm. It would be foolish, in my opinion. Instead, we rely on the Bayesian posterior approximation in natural language. What this does is that when the system is not sure, it's going to hedge its bets. If there are multiple interpretations, for example, of a statement about a particular kind of harm, then that will make the predictions of the Scientist AI farther away from 0/1: less certain, which means probably the request will be rejected.
Is it possible to go ahead and train a really scrappy version of this kind of AI really quite soon? Maybe in the next couple of months or at least the next year? I mean, I remember GPT-1, back in 2018 or something. It was complete rubbish, but I guess it was a proof of concept that you could make a model like this that was quite interesting, and gave people a lot of enthusiasm and drove a lot of people into the industry. And it does seem like we already have enormous corpuses of text, and we can use language models to basically just pull out the best data, and label who said what, when, and where. Then we can also get them to produce a set of things that we think of as basically verified facts that we largely trust, with a bit of human oversight. I mean, they can be conservative to start with — not include controversial things, just include things that 99.9% of people would agree with — and then it doesn't seem like it'll be that hard to train. I guess if you think you've got the technical methods, then it shouldn't take that long to potentially just train a model that can, as an alpha version, assign probabilities to statements being true or false.
Yeah, that's exactly the plan. That's the plan.
There's a phrase that I've used in the past, which is that we want a plan that produces an "anytime answer." What I mean by this is: if we have more time, we will have something with stronger theoretical guarantees — but we don't know how much time we have. So there's a research programme where the early steps will be, as you say, a scrappy system — where probably the math doesn't apply, because we don't satisfy all the conditions. But it's probably fine, right?
It's a lot better than what we have, anyway.
Exactly. And it's probably fine, especially for the first job that we have on our programme, which is this non-agentic predictor that can be used as a guardrail. Now, as I said, the guardrail isn't the full answer. But if we deploy that, and companies add it to their monitors, then it will mitigate the risks to some extent, so it will allow more time to develop the more ambitious version that is fully agentic. And that's what we need right now: time.
As far as I can tell, I think Anthropic — I'm talking a lot about Anthropic because I've been reading the Mythos system card and all the announcements last week — they've decided to have Mythos monitoring Mythos basically. They've tried other models doing it, but they're like, "Mythos is smarter, it's better." But obviously this creates an internal contradiction: if they don't trust Mythos, why do they trust Mythos to monitor itself? That's one reason why, even if this model is much less intelligent, I'm like, at least it's an independent judge. It's built in a very different way that might be
more likely to flag things and less likely to scheme to support itself, basically.
Yeah, I completely agree. And I would go even further. It's not just the issue that the monitor could be deceptive and say it's OK when it's not, because somehow that's aligned with some hidden goals of self-preservation or power seeking. But it's also, if we go a little bit down the line, what companies are planning to do with using AI for AI research — so this is a place where having AI that is secretly deceptive is even more of a dangerous bet, right? If we're going to put all of our trust into a system that will design the code and the algorithms — that will be too complicated for us to understand or to read fully — that kind of AI could put backdoors that we don't see into an AI system in the future that is even more powerful. So we could get into this direction that gets even more dangerous for us. It would be, I think, very dangerous to do that. And that is why, in terms of policy, the attention given to AI for AI research is something that should be very high on the agenda. And this is also why, if we're going to be doing AI research with AI, we really, really want to make sure that that AI is going to be an honest one.
So I think the majority of people who are piling into technical AI safety have decided to go for improving our chances at the cat-and-mouse game, basically. And I think for the people who are very concerned about what's going on, their reasoning is something like: We're running a 50% chance of absolute disaster now because we're doing a whole bunch of absolutely crazy, reckless stuff. Maybe by just patching the very dumbest stuff that we're doing, fixing the worst things that are on fire right now, we can bring that risk down to 10%. Obviously, that's a preposterous risk of disaster to run; it's an embarrassment to us as a species that we can't do better than that. Nevertheless, it's 40 percentage points' improvement in our chances of things going well, or at least reasonably. Going from 10% down to 0% is only a quarter as valuable as that, even if you can get massively greater guarantees of safety using much better alternative approaches.
Well, in the logarithm domain, it's infinitely better.
Sure, sure, but in the expected value domain. And I think that's kind of the difference in the two mentalities here.
Yeah. No, as I said, we should try all of those things. They're not mutually exclusive. It would be a mistake to put all our eggs into the cat-and-mouse game, at a cost that is a fraction of what companies are currently spending, when we could be developing a safe version of AI that will be capable. And by the way, I want to add something here about capability: I also believe that the Scientist AI could even be more capable than the current approach, and that has to do with a number of design features. It is trained to explicitly reason in a structured way about the statements that it's asked to make a prediction over. This is different from the current chain of thought, where it could produce some kind of bulls*** that we believe and tends to pass the test that we have during training, but it's not constrained to actually have arguments that can be decomposed in the same way that a proof of a mathematical theorem is decomposed. And there are other approaches that follow that direction. Of course a lot of the work on trying to do safe-by-design AI, but also the debate work, for example, is trying to enforce some kind of coherence in how the AI is thinking. So I believe that, in addition to the epistemic humility that comes with the training objective that we are proposing, the way that the system is producing those probabilities by invoking structured latents that form a chain of reasoning is something that could actually provide even a capability advantage to the companies.
Do you think current models internally represent truth? I guess you're saying one advantage of this model is that it's focused on representing ground truth as a latent variable. My guess is that current LLMs do that as well — because that is very useful to have some sense of what's actually correct — and then they distort that. They basically start with that and then they distort it in order to accomplish the goals, including manipulating people or lying or whatever else. I guess some people doubt that. Some people doubt whether there is any connection, or that they are actually trying to model truth. Do you have a view?
Yeah, I completely agree with you. I have an assumption about how the world works that basically states that reasoning about the actual properties of the world — in other words, the truth — even when you are uncertain, so you have to use probability, gives you a very strong edge in making better predictions and better actions. That is actually part of the argument as to why the training procedure for the Scientist AI will create latent variables that are preferentially going towards the actual beliefs. That is very useful, because now we can query those latent variables and get answers about what the AI actually believes, because that's how it constructs its internal reasoning and produces an answer, not some potential bulls*** that comes in the chain of thought. So it doesn't completely solve the ELK challenge — the challenge of eliciting latent knowledge — because the only guarantees we get are about these natural language statements that can be latent variables that we can query.
Can you explain the ELK problem?
Yeah, sorry. The ELK problem comes from the issue you were raising, that even though the AI may internally know the truth of something — or at least have some internal beliefs about something, because it's trained to imitate the variables that it sees in the data, which mostly are what people are saying — when you query it, it's going to answer in the same semantics; that is, what a persona that it currently is taking, given its context, would answer, and not necessarily what it actually believes. And the technical problem here is we don't have supervised labels to teach the AI about what it should actually believe, so we can't ask it about its true beliefs. We only get a kind of reproduction of the distribution of variables that it sees in its training data. So in the Scientist AI, this is addressed by having this clear syntactic separation between the communication acts and more factual syntax that can be used for latent variables and true things that we know, so we can query it using that factual syntax. Then the other reason why we're getting away with some of the issues with the ELK challenge is that the same language — which is like English, let's say — can be used to represent those latent variables as well as the observed statements, and so basically rely on the compositional structure of language to generalise to new sentences that it has never seen, but the meaning of those sentences is given by its understanding of language. This is very different from the scenario studied by those who looked into the ELK challenge, where we assume that the latent variables are anonymous — they don't have a predefined meaning, and so we don't know where to look inside the neural net, if you want, about what the beliefs are, which motivates things like mechanistic interpretability and so on. But in the Scientist AI, we bypass this problem to some extent because the latent variables are in natural language and thus are interpretable. Now, there could still be other beliefs that are not in natural language, that are hidden in the neural net, but at least when we ask questions in natural language, we're going to get an honest answer.
So as far as I can tell, there's three big approaches here: One is we're going to use this model as a monitor, as a guardrail. Another would be we're going to just train it from scratch and make this be the whole approach. Another would be we could take the current models and try to make them more honest, make them more like a Scientist AI. Do you want to talk at all about whether that approach has any good prospects?
Well, the math that I currently have would require us, to get the guarantees, that we actually start training from scratch, which is expensive. So we would lose the guarantees if we do just using say the Scientist AI fine-tuning on existing models. But even if you don't have a mathematical guarantee, it might still be a workable approach, so I think it's worth doing. In other words, we can take a really competent, top-notch model and then continue training using the objective of the Scientist AI and the data that has been transformed as I discussed. And we hope to show empirically that as you do more and more fine-tuning, the measurements of honest behaviour, lack of deceptive behaviour, will improve, and that we won't lose in capability. So that wouldn't be like a mathematical proof; that would be an empirical thing. And once this is established, then it might be sufficient to convince people to let's get the full guarantees by training from scratch, which is now going to cost the cost of training a full model.
So the approach that you would take there is to take a current frontier model and then do reinforcement learning to get it to speak as if it were a Scientist AI?
No, no. So first let me talk about reinforcement learning. Three years ago, I was in a meeting with a bunch of reinforcement learning researchers and I had a slide with only these words: "Reinforcement learning is evil."
But what do you really think?
This is not something new. People in AI safety have been talking about the fundamental flaw in training by reinforcement learning to achieve something in the world: it gives rise to the problems of instrumental goals and reward hacking. And in both of these cases, what you do end up with is systems that have goals that you didn't choose and could go against the goals that you did choose. So reinforcement learning is a very dangerous thing to build superintelligence. The good news is you don't need to do reinforcement learning. What we show with the Scientist AI is that there's a way to train the AI so that it will be indifferent to the consequences of its actions, of its predictions. Let's start with a predictive model. It's easier to understand. Imagine you do have a really good climate model. The climate model, if you run a simulation of it or train a neural net to approximate those simulations, will give you honest answers. And it doesn't care if the answer makes us do something stupid. So that's how you get honest answers: essentially by building an explanatory understanding of the world that is completely indifferent to how the predictions are going to be used. Now, once you have this, you can use it in a kind of agentic way. For example, the guardrail is a kind of agentic thing, right? It's taking a binary decision: Do I accept this prediction? Do I put out this prediction in the real world or not? And it is a decision, it is an agentic choice — but in this case, it's a choice that has a unique goal, which is to avoid dangerous actions. So we are already entering the agentic world once we install the guardrail. So bottom line, to summarise my answer: there's a way to train a predictor that will not require reinforcement learning in the sense that it will not require optimising with respect to future events in the world, including future good prediction errors. And here I want to make a parenthesis about previous work in AI safety on AI Oracles. Of course, people have thought about this: why don't we just train an Oracle that's a good predictor? But they thought that the only way to train it would be to train it by reinforcement learning to make good predictions. But there's a huge flaw here, right? Because if I'm rational, and I want to maximise the good predictions I will do forever in the future, I could lie in the short term to make humans do things that will help me to make good predictions in the future — like get more compute so I can train a better version of myself, or make the world simpler to predict: kill everyone. If humans kill each other, then the world will be much easier to predict. So these are really bad outcomes that are due to instrumental goals of making good predictions. And it arises because of the reinforcement learning objective: you're training the AI to achieve something in the real world, and that's where you get bad problems. But the other approach, the approach of the Scientist AI, is to train it from the get-go not to achieve anything in the world, but to just predict the training data, the past data — so it's not about the future; it's about the past — to come up with good explanations and good predictions of the past data.
The reason I was asking is, if we're going to go from a current state-of-the-art agentic model and try to make it more like a Scientist AI, to make it more honest, how do we do that if not reinforcement learning? Are you saying we're going to do something more like we get it to predict past events based only on having data from before that time?
Yes, yes. And that's how science works, by the way, right? What scientific theories do is explain the past data. And of course, sometimes they make predictions about future data which we can check. But fundamentally, the way that we judge a good theory is that it is making good predictions about the data we have.
Does this require blinding the model to the results, basically?
In some sense. We have this condition in the Scientist AI requirements for the theorem that we call "consequence invariance." What it means is you're only allowed to use how well you're fitting the past data in order to train your causal model. You are not allowed to choose those predictions with respect to either what could happen in the future as a consequence of those predictions.
So I think I have a decent picture of the predictor model that's taking in statements throwing out probabilities of them being true. Is there more that it would be useful for me and other people to have in their minds to picture how this entire system would work — where it's not only the predictor; you're building scaffolding around it to give it partial agency and so on?
Yeah. First you have to understand that the same predictor that is trained in a unified way can be used to both answer user questions and answer safety questions. And the safety questions are those that you care about for the guardrail. So it's not like there's a separate neural net that does the guardrail and another one that makes predictions. The guardrail is using the same prediction neural net; it's just a different kind of question you're asking. You're asking, "What's the probability of a particular kind of harm, given that I put out this prediction?" — or in the case of the agentic system: "…given that the AI puts out a particular action?" So training is fully the predictor. Once it's trained, there are a number of things you can do to construct the system that includes the scaffold. So what is a scaffold doing? For example, when a user comes with a question, it will put it in the form for the predictor to produce a probability. But it will also call the predictor with a different question, which is the probability of harm — that's the guardrail question, and then it will look at the answer in order to decide whether to produce an answer or not. For example, if the question is about how do I build a bomb, then the guardrail will say the probability that this is dangerous is high. And I mean high enough — so the guardrail will use a threshold on the risk probability to reject those questions. And that threshold is a normative choice; it is something that society decides: how much risk are we willing to take, depending on the kind of harm that we're talking about? The guardrail also has other roles to handle what is called "performative prediction." Sometimes a question can have multiple answers because the answer will influence the future. A classical example is if the question is about who's going to win the next election, and maybe the AI is considered very capable and people will believe whatever it says and vote for that candidate — which means the AI could say this guy or that guy —
And both of them would be true.
Both would be true, right. So then it's starting to have agency through its prediction in a way that seems that we don't control. The guardrail is going to manage that to decouple the predictions from the effect of those predictions. To be more concrete, the neural net predictor is trained so that in its input conditions there's always a particular statement that asks, "What if we did produce this prediction? What would be, for example, the harm effects?" So when you condition on the intervention of producing a particular answer, now there's only one answer. You're saying, "I'm going to put out this prediction, and what will be the effect?" There's more to say about this, but the bottom line is you can control this kind of risk and the agency that can come from it, and it's the job of the guardrail to do that job.
There's a longstanding worry that Oracle AIs are structurally disadvantaged; that they're going to be less intelligent, all else equal, because they don't have the option of basically running experiments, of taking actions in order to discover how things work most effectively. And I think there's other worries along these lines — that basically it's the things that make AIs intelligent that make them dangerous, and vice versa. What do you think are the chances that is true?
I think we have to distinguish two problems to have a clear idea about your question. One is: what are the best predictions — or the best actions, in the case of an agent — given the available information, like the dataset and context that is available? Then the second question is: if I were to do experiments in the world in order to acquire more knowledge, what are the right actions that will increase my understanding of the world and reduce my uncertainty about the world? By the way, this is how scientists in general think — not AI Scientists, but people that are doing biology or chemistry or physics. They ask themselves, "If I were to do this experiment, would it help me to disambiguate between these two theories?" You can quantify this mathematically with something that's called "information gain," and it turns out that once you have a good probabilistic predictor, you can also turn this into a good estimator of how much information you would gain if you were to do this experiment or that experiment. Now, you could build an agentic system on top of the Scientist AI, for example, that would tell you which experiment to do in order to obtain good information gain. But of course, you could also use a guardrail. So you would like experiments that help to disambiguate between explanations and theories at the same time as not harming people. But that's easy to do in the Scientist AI, right? We have this guardrail notion. So the user goal here is to acquire information; the safety goal is don't harm people. (Of course, this is cartoon.) And so you could get both. But of course, you now enter into the realm of agentic systems, and the whole plan of the Scientist AI includes how we can develop agentic systems on top of a non-agentic, trustworthy predictor.
I guess if we wind back a year or two ago, we had AI models that were, in a sense, extremely knowledgeable, extremely smart. But if you just tried to get them to navigate a web page, they would struggle to do it. It seemed like there was a very big difference potentially between scientific intelligence or ability to predict things versus ability to navigate the world in practical terms. And it took a lot of extra training, a lot of extra effort in order to get them to be able to take useful actions. Do you worry that the Scientist AI that you might train using the kind of data that you're imagining would be kind of incompetent at a practical level? I guess unless you did a lot of this sort of work, where the experiments we were running were like, "Do you click this button on a particular web page?", it wouldn't actually learn to do the things that people want the models to be able to do.
You could absolutely train it with trajectories of what happens when a particular agent did this, what was observed, what the consequences are. This is how it would learn to learn good conditional probabilities of what actions to do in order to achieve particular goals, including the safety goal. So it would be a different kind of training than the reinforcement learning training, but it would be using the same resource, so whatever experience has been collected — by the way, it doesn't have to be what people in RL call "on policy"; it could use the experience of any agent or anything that is observed. That is, not just agents doing things, but just observing things in the world. All of that is data, as far as it's concerned, that helps it both understand the world and construct the consequences: "What are the consequences? What would happen if I do this action? And what action will maximise the probability of achieving some goal?" In a way, it's closer to model-based reinforcement learning, where you are able to use your whole experience to come up with a policy by opposition to something that is fully interactive all the time. In the Scientist AI, currently you would need to retrain it or fine-tune it with the new data if you use it and it produces new consequences and new observations. But we can ride on the same research that companies and academia are working on, on what's called "continual learning." So what happens when there's new information coming in? Of course you could put it in the context, like in the input window, and with the Scientist AI you could do the same thing, but at some point you'd like it to be integrated somehow into the weights of the system. And that's what continual learning is trying to do. But the good news with the Scientist AI is it's facing these same problems that current AIs are facing, and the solutions that are being explored will be applicable to the Scientist AI as well.
Yeah. I feel like a lot of the critiques of this idea, and my questions somewhat reflect that, is that people, including me, have had in mind a vision of an AI that's extremely different in how it's trained, maybe the data that's being used and the structure and the affordances that it has. And you want to say, actually we can make it remarkably similar. We can take almost all of the data that we're using to train current LLMs and just reformat it a bit and then use it again. We'll use all of the different efficiencies, all of the algorithmic improvements; we're just going to give it a somewhat different set of inputs and outputs, but it's more or less the same in almost every other respect.
That's why it's so practical.
Yeah, that is why I think we can do it pretty quickly. It's more a matter of having the right resources for training. And yeah, because the training objective is different, we do need to try it out and see how it works. But fundamentally, it isn't so different, for example, from maximum likelihood training, which is what we use in pretraining. So in a way it's closer to the pretraining — and we know that works really well, by the way; it's actually working better than RL, which is harder. So the form of training here is much closer to what we do in pretraining, except that we teach the AI the difference between what people say and what it actually believes, and we force it to reason about why did people say those things rather than imitate what people would do.
The last I saw online, the organisation that you're leading, LawZero, has raised something like $100 million. I think most nonprofits at year end would be pretty happy to have raised $100 million or so, but I guess you're up against organisations that have $100 billion.
Actually it's even less than that, but more depending on how you count. So we've raised about $35 million US from philanthropy, but we are in negotiation with various governments to get much more. So we are pretty confident that we are going to be in the hundreds of millions pretty soon. But as you say, that's still peanuts compared to what the leading AI companies have. However, I think it is sufficient to make a proof of concept, and with a proof of concept then we can convince companies to actually put in the money to train larger systems, or systems that are trained from scratch using the same principle.
So that's the theory of change. What sort of experiments do you want to run, and how much money would you need for them?
There are various kinds of experiments. The bottom line is we want to show the improvement in honesty and basically getting rid of deceptive behaviour. And we can do it with two categories of experiments. We can train really small models — of the kind that academic organisations have been training, like less than 10 billion weights or something — from scratch, but using the Scientist AI objective and the data representation that I mentioned. That won't be competitive, because it's going to be much smaller models, but we can compare it head to head: the original open weight model that has that same size and trained on the same data, we can compare both in terms of capability and safety, essentially, at least honesty being the main thing we're looking for. So that's one kind of demonstration. The other demonstration — which is maybe closer to being deployable, but has less guarantees — is to take an existing pretrained model, maybe starting from a Bayes model rather than the one with RL, and then fine-tune it using the Scientist AI objective and data representation. That would give a much more competent model because it's bigger. Fine-tuning is much cheaper, as you know, than training from scratch, but we lose the mathematical guarantee. I think it's probably going to be fine anyway. Of course, it depends how much fine-tuning you're willing to do. What's interesting here in these kinds of experiments is that we should be able to see the tradeoff. Like if you measure, say, on deception benchmarks what happens as you continue training with more and more fine-tuning, we should see a curve: that it gets better. And that's what we're hoping to see. Then you also want to show that capability doesn't go down — which, by the way, is tricky, because unfortunately what we found already in our experiments is that at least most of the open weight models cheat on the benchmarks. So what do I mean by this? As soon as you do a little bit of fine-tuning on anything, their performance on the benchmarks goes down.
I see. So they've really taught them to the test?
They probably have overfitted the benchmarks. So we need to find a way around that, but I'm confident this can be done. So we will have these two kinds of evidence, I hope, and that may be sufficient to convince people to put in not just hundreds of millions, but the billions that would be necessary to do a full-scale model from scratch.
Comparing like for like, if you train a model of the type that you're envisaging versus a standard model using the same amount of data, same amount of compute, I guess we think that the Scientist AI would be more honest and safer. In terms of capability, both in terms of prediction and agency, would we expect it to be better or worse? And how much better or worse?
In terms of capability, I would expect it to be better because of better reasoning. One aspect I didn't mention yet is that there's good scientific evidence that when a model exploits the causal structure of the world, it can generalise better out of distribution. This is something I've worked on, and many people in the machine learning community have been working on. It has to do with a very interesting concept that the world changes, but somehow there are things that don't change — like the underlying causal mechanisms, like how the world works, like the laws of physics: they don't change. So even though the distribution of the data may change because things happen in the world and things will look different on the surface, the underlying scientific explanations for how things are are the same. And if you can train your system so that it is encouraged to discover these explanations, and the system also understands the notion of intervention — in other words, when somebody does something in the world, it can change the distribution, but it doesn't change the mechanisms, the underlying laws of physics — when your model is able to make those distinctions, then it's going to be much more robust to changes in distribution, which is the hard question for neural nets and machine learning in general that, for now, we don't have good answers to. And in the world of safety, this is a real issue, right? We would really like our guardrails to be robust to the fact that the world is going to change, the distribution of the data is going to change. They're going to be asked questions that are very different from what they've been trained on, so having systems that can generalise better out of distribution because they understand the causal structure would be a huge plus.
Is one way of phrasing this that current models, as we train them, are designed to predict what people would say, and they learn to understand something about the truth as a side effect, as an instrumental part of predicting what would be said? Whereas your models, they'll be primarily oriented towards figuring out what is true, and how does the world work, and then they would learn to understand what people might say as a side effect of that, incidentally?
No, because we don't have enough ground truth about what's really happening in the world. I mean, there is scientific data and evidence, but the Scientist AI would use mostly the communication acts, like what people say, as a source of information about people and society. And that is a very rich source of information. The problem is that you can't just believe what people say and repeat it. Current LLMs, if they see something false very often repeated, like the Earth is flat, if it was repeated enough, they would start saying it, right?
Is that definitely true? Because it seems like they don't buy into conspiracy theories that much currently. They don't say that the world is flat just because many people do. I don't know. I mean, there's other examples, but by and large they reject conspiracy theories.
If they understand conspiracy theories, and they are not playing the persona of a person saying it, you're right. But there's lots of other evidence, not for conspiracy theories, but all kinds of biases. These biases would be not something a small number of people believe, but more like most people believe something wrong, which induces discrimination, for example. And there the evidence is very clear: the current LLMs are biased in the same general way that the population is biased right now. And the Scientist AI wouldn't be falling for that as easily, because it would look for both what is a good explanation for what people are saying and that this explanation has to be coherent with all the other things that it knows or that it has seen.
I feel like the discussion about this proposal last year got more focused on the mathematical theoretical guarantees, discussion of the safety guarantee side. It feels like you're moving, and it seems like I feel like we should move, towards a scrappy 80:20: this is going to probably be safer, we have good reasons to think it's better, so let's just throw something out and see how it goes and iterate from there. Do you agree?
