So I will get started. I'm going to give a deliberately provocative talk called 'LLMs are not the way to alignment.' If I had more time to prepare the talk, there would be less text on my slide, so I apologize for that. I don't yet work on technical AI safety. I've been thinking about it, but it's not really my area. But I think it's one of the most important problems facing humanity. Probably you think the same if you're in this room. I'm going to give you some thoughts about the AI safety problem from the outside, from someone who has spent his career scoping out weaknesses in AI and pushing for better approaches, because I think we need some of that in AI safety. So I'm here to shake things up. I'm going to say some extreme things. I hope that they will make you think. You can ask me questions at the end, and that's fine.
And as I say, I'm an outsider, so there might be important papers I haven't read and things like that. But my main point will be, and we'll start from the fact that I'm pretty certain that LLMs on their own will never solve alignment. I'll just say that one more time: I don't think that LLMs on their own are ever going to solve alignment. If I'm right, that's really, really bad because almost all of that big pile of money that we saw in the morning being spent on alignment is being spent on LLMs. And so maybe that's the wrong way to go. Transformers are nearly a decade old. We've spent more money investigating large language models than I think anything else in history. Maybe somebody else has a candidate, but as far as I know, we've spent more money investigating LLMs. And it's all kind of an experiment, right? It's a scientific study: can we get to AGI by scaling LLMs? So we have spent more money investigating that hypothesis than any hypothesis, I think, in history — more money than trying to solve cancer, things like that.
And I would argue that in exploring LLMs, we've hardly made any significant progress on alignment. Yeah, there's a lot of studies out there. There are things you can point to that have been published. But if I said, you know, show me the five biggest results, like a mathematician would have a result in alignment and LLMs, I don't think there are any really big ones. You can argue with me in the discussion period, but I don't see them from the outside. I'll explain that thing in a second. Of course, there have been tons of LLM alignment fails. I don't think anybody really even noticed them like five years ago, although Victoria had been pointing out failures even before there were LLMs — all kinds of problems with reward hacking and so forth. So some people have been in this business for a long time. But nowadays, LLM alignment fails become front-page newspaper stories when we hear about, you know, deceit and blackmail and various kinds of jailbreaks. I'm not going to go into them today, except I'll give you one from earlier this week, which is ChatGPT advised the Florida State University shooter that a mass shooting would get more attention from the media if it involves several children, according to a lawsuit. We'll, I guess, we'll see the facts there. That to me is a pretty potent example of an alignment fail, right? You do not want a system to give advice that is going to cause mass harm to human beings. There are lots of these, of course. So AI systems should strive to make the world safer, not more dangerous. That's the kind of thing we should be worried about.
One hypothesis has been that making large language models bigger might solve alignment. I've seen that either explicit or implicit for the last, I don't know, half decade a lot, but I don't think that they really have. And I think if you look at the empirical literature on this, it's just not true that making them bigger necessarily solves these problems. So making LLMs bigger does do some things. For example, you get a better approximation to human text that can make them better for brainstorming. But I don't think it's convincingly — well, it certainly hasn't convincingly solved the alignment problem. There was a February Anthropic article, and they said the longer that models spend reasoning and taking actions, the more incoherent their errors become. Like, that's the opposite of the usual positive scaling trend where you get better performance. And they said there's an inconsistent relationship between model intelligence and error incoherence. You don't want a negative correlation. You don't want a zero correlation. You want a really strong positive correlation. If you don't get a really strong positive correlation, maybe you're not trying to solve the alignment problem the right way. They also mention in the same paper the reward hacking stuff, which I think Victoria was really a pioneer on. The honest truth is we don't know how to make LLMs safe, and we probably never will. I mean, that's the most provocative thing I have to say. I think it's probably true.
There's scaling is one thing we've tried. Another thing has been trying reinforcement learning, and that's where we get big time into the reward hacking kinds of problems. Yoshua Bengio, who used to disagree with me about some things, basically said that reinforcement learning is a dangerous path for building superintelligence. I agree with him. I'm not going to sort of spell out the argument in detail, but unless we have a solution to reward hacking, I don't see how RL is going to get us there. To my mind, these problems are principled. It's not an accident that LLMs are not solving the alignment problems. Before I get to why, I want to kind of warm you up with an analogy to another problem that has really stubbornly resisted scaling. So again, scaling has fixed a bunch of things, but there's some residue that scaling really doesn't seem to help that much. And the other problem I want to talk about is truth. So truth is a related problem in a way that I'll talk about in a minute. In fact, technically it is an alignment problem, and we really haven't made that much progress by doing things like scaling or things like RAG, if you know what that is — retrieval-augmented generation, etc.
So let's call the canonical form of alignment relating to things like blackmail and deceit as goodness. I'm thinking philosophy, truth, and goodness and so forth, right? I mean, these are just different values. I think Anthropic talks about being honest, harmless, and so forth. So honesty is a form of an alignment problem: don't tell me lies, be honest, etc. Some people have even tried to stick that into their system prompt. I think that that's part of Anthropic's constitution that I'll talk about in a second. Apple famously had in their system prompt when they did their BBC news summaries, 'tell the truth, don't make stuff up,' and the system made stuff up anyway. Like, putting this into a system prompt has not been that great. LLMs are in fact terrible at truth. I would say that they're indifferent to truth. What they're really doing is they're modeling how people use words and sentences, but that's not the same thing as truth. Sometimes it's correlated with truth. You know, people tell the truth more often than not. And so if you just mimic what people say, you're going to do okay, but you're not going to do great.
I think I was maybe the first person to point out that statistical approximation machines that lack certain kinds of symbolic representations would tend to hallucinate. And that was in 2001, right? I've been worrying about this problem for 25 years. And I keep having people say, 'Well, next year hallucinations are mostly going to go away.' Reid Hoffman two years ago or three years ago said, 'You know, I pretty much can guarantee,' or he said, 'I'll bet any amount of money that they'll be gone in a few months.' And I offered on Twitter to bet him $100,000 and never heard from him again. The hallucinations have not, in fact, gone away. This problem's not been solved. I don't think that the goodness problem is easier than the truth problem. I think it's harder than the truth problem. I gave one example of the hallucination problem in my 2023 TED talk. That one was an LLM that claimed that Elon Musk had died in a Tesla in a 2018 car accident. Now, what was really remarkable about this is it could have easily avoided this error for a bunch of reasons.
One is that you could just go to Wikipedia and you could see that he was still alive in 2023, or in fact you could go anywhere else. Almost every news provider on the planet had current information about Elon being alive because that man loves being in the news and the people who write the news love that man. And so if you thought in a sort of old-fashioned Bayesian terms, like weighing evidence — what is the evidence that Elon is still alive as opposed to the evidence that he's dead and died in the 2018 car crash? There was no evidence that he had died in a 2018 car crash, and there was enormous evidence that he was still alive. How did the model even do this in the first place? It's an interesting question, right? Why would a model do this? You know, some hallucinations are because some fool on Reddit said something stupid and the system is just repeating the stupid thing on Reddit, which was maybe a joke or an ignorant person said or whatever. But this is not coming from that. I think if you'd done a Google search, you wouldn't have found that. In any case, you should be able to go to the Wikipedia stuff.
The problem is a bunch of people died in Teslas in 2018 in car accidents, and Elon Musk owned a Tesla — in fact, owns Tesla. He owns the Tesla company. And so these little bits of information get recombined by these systems. What LLMs do, basically, to a first approximation — this is not perfect, but to a first approximation — what they do is they break information into little bits that are distributed representations, and then they reconstitute them. And often they reconstitute them correctly, but they don't always. They're probabilistic. And so sometimes they come up with these errors. So I have a new favorite example that's very similar, which is: I have a friend who's a movie star, a voice actor. You all know him from The Simpsons because he does Mr. Burns. My friend's name is Harry Shearer, and he was born and raised in Los Angeles. And one day someone sent him a Twitter — or, I mean, a ChatGPT biography of him.
I left out a little bit of context, which is somebody used to send — or people used to often send — ChatGPT biographies of me to me, and I wrote a bunch about how one of them described me as owning a pet chicken named Henrietta. So Harry Shearer thought this was hilarious. 'Of course I don't own a pet chicken, let alone one named Henrietta.' And so one day he sends me a biography of him. He says, 'No Henrietta, but...' And then it describes him as being a British voiceover actor and entertainer. It's the same kind of mistake as Elon Musk. Harry Shearer is born in Los Angeles. You can go to Wikipedia because he's a well-known actor. You can also go to IMDb and Rotten Tomatoes and a million other places. The correct answer is unambiguously represented in the training data, but the system still doesn't manage to get it because it doesn't really have representations of individual people and their properties and so forth the way a database would. It just reconstructs things statistically.
So the key thing that's missing in this particular case is really representations of individuals distinctly from kinds. If you go back to some of the old-fashioned semantic network stuff, that's a primitive in some of the old-fashioned semantic networks, and it's a primitive in every database — or not every, but in many databases it is a primitive. You're going to have an individual separately from a kind, but you don't have it there. So I pointed this out in 2001 in my book The Algebraic Mind, and the problem has not been solved. What I argued there is if you don't have symbolic representations of entities, structured representations, and so forth, you can't really do proper reasoning over them. And I think we're still having that with LLMs, which are much more sophisticated than the models that I was critiquing in 2001, but fundamentally share a lot of the same properties. Large language models lack the means for stably representing and reasoning over things like world models. I'll say more about that in a second. And truth suffers as a consequence, but goodness suffers as a consequence, too. That's where I'm going with all of this. You can't figure out goodness if you don't have stable representations of the world to reason over. If you just have text, all kinds of weird things can happen.
Okay. Even in the best research on ethical alignment, errors are still really pretty common. I took this graph from another study from Anthropic called 'Teaching Claude,' and there are all these beautiful charts — and I'm going to make fun of this one in a second and zoom in on it. There are all these charts that show how errors on tasks involving blackmail and financial crimes decrease as a function of how much reinforcement is given and so forth. Ostensibly, this is great; it means better aligned systems. Then you look at the graphs, and they're just total chart crimes, right? Like, there is no y-axis actually even labeled. So you don't know how many training steps this is or how much it costs. There's no x — or the x-axis is different in every graph for where it runs to and so forth. But — and there are all these distributions. They didn't even label what the lines are. So I don't even know what these are. Anyway, the scientist in me looks at this kind of stuff and gets a little bit nauseous.
But what you can tell in every case — and there were other graphs that I didn't show, but they're kind of similar — is that you asymptote typically at like 5%, sometimes at 40%. 5% error. That's really bad if you're talking about alignment, right? 5% error can be fine in a recommendation engine, right? You could be 20% wrong in a recommendation engine and be okay. But if you're trying to make a decision about whether to tell somebody what they should do in the context of a school shooting, you don't want to make any errors there at all. Right? So there's a very high bar for alignment. You know, 5% is just not good enough in the case of alignment. And as with truth, you get reasonable approximations by doing the statistical approximation thing, but you don't really get accuracy, right? You can get like 90% correct in citations from legal stuff, but you're still — a lot of errors still through the same things happening in alignment.
Okay. Despite my pessimism about a bunch of things, I think Anthropic's idea of a constitutional AI is a good thing. I think we should, in fact, think about what constitution we want our systems to follow. As The New Yorker just noted, there's an article about Anthropic's constitutional AI maybe yesterday in The New Yorker. We don't want Anthropic to decide this, right? This should be a mission for all of humanity. We don't want five big tech companies to decide, right? None of them are elected. You know, it's the least democratic means possible. And yeah, Anthropic goes out and asks some people for advice, but that's not really the right methodology for determining what the constitution should be. We need more civil society represented. But let's temporarily set aside that concern. They give some empirical data that their constitution is obeyed, but again, it's not really very strongly obeyed. So I think it's fine to think about constitutional AI, but we need to have systems that actually understand — to use a word from an earlier talk — that actually understand those constitutions well enough to obey them.
We shouldn't have to do the kind of stuff that Victoria talked about in her talk. We should be able to stipulate, 'don't do these kinds of things,' and the system shouldn't do the kinds of things that we have stipulated. It is the fact that the system prompts are not being obeyed that allows her to give that talk. And it's a very interesting talk, but I don't want her to have to keep giving it.
A second problem with their implementation — their implementation of constitutional AI, which is an LLM basically — is that LLMs are too easily influenced by what they read. They're just too sensitive to the data, right? You want your system to be able to learn from data, but you also want the principles to exist independently of the data. And what we have are these giant mimics. So you get crazy results like this one just came out in Nature like yesterday: state media control influences large language models. So it turns out that the more that a particular nation governs what gets printed, the more the models suck up to those who are controlling that data, right? That's not constitutional AI. That's some state having — not even an intentional, well, partly intentional — having a semi-intentional influence on how things turn out. So like, LLMs are just not the way here to make alignment work. We need for AI to be able to reason with respect to a set of values. That's probably the most important thing that I'm saying.
LLMs are not true reasoners. They do things that look like reasoning with chain of thought, but they're not actually representing things like sets of values or world models. And they always have trouble when you go out of distribution. I think I'm going to go a couple minutes over, but I'll try to wind this up. So 30 years ago, I started pointing out the problem of distribution shift. I showed that an LLM that was trained on the identity function f(x) = x on even numbers would fail to generalize to odd numbers. Everything I've written ever since, everything that I've done to piss off the AI community, all stems from that one paper in 1998. And we're seeing the same thing here, which is LLMs will kind of follow something sort of, but you get out of the distribution and you fail.
So, the Apple illusion paper — the 'Illusion of Thinking' paper — is another example of that. I won't go into detail. But what we need are world models. And by that, for me anyway, what I think we need is explicit representations of the entities of some domain. World models don't need to be complete. They don't need to represent everything in a domain, but they need to be complete with respect to something that is required. Any chess-playing program has a world model built in. GPS navigation systems have world models built in. They're not some exotic thing that we don't understand. The problem is we don't have to induce them from data. So the example I like to give is: how does my child or my children read Harry Potter and induce that there is a different world there with different people with different properties, some of whom have magic and some of whom don't, where there are brooms that you can fly on and so forth? They build a complete world model, and my kids know that that is not the same as the model of the rest of the world. They know that that's different somehow. In the course of reading a novel, they can do that, or every time you watch a movie, you do that. That is what we need our AI systems to do.
And this matters for moral reasoning because moral reasoning is always over world models. You have, you know, you're making an ethical decision: what should I do with respect to a set of entities, motivations, locations, etc.? You need world models for that. Moral reasoning revolves around situations and states of affairs in the world. You can't really do that without world models. And so you wind up with these bizarre errors. I'm not saying you'll get this one today, but here's an example. These are river-crossing problems. And the situation got so embarrassing for Anthropic that they put in their system prompt instructions about how to handle river-crossing problems, right? Like, I used to argue about innateness — they innately put in river-crossing problems. But you would still get these crazy examples. Maybe you don't get it today. Here's the riddle: a man and a woman are standing in a river. The man has a boat. How could they both be brought to the other side? You could just stick them in the boat, as my daughter pointed out when she was nine. But the LLM came up with these like ridiculous series of steps.
That is because what they are doing is they're analogizing to the stuff in the database without actually reasoning about things like optimal paths. You don't want that in your moral reasoning. So I'm going to recommend to you a paper that you can read offline, which is by me in 2020 called 'The Next Decade in AI.' And I argued that we need four steps to get to trustworthy AI. It was not a paper about ethics per se or alignment per se, but it said we need neurosymbolic AI, large-scale knowledge, reliable reasoning, and world models. When I wrote the paper, nobody paid attention. Partly because it was me and nobody likes me, and partly because it was the beginning of the pandemic and nobody paid attention to anything because it was published two days before everybody realized that we were all in trouble. But you will note that five or six years later, people are actually doing a bunch of this now. So Claude Code is actually neurosymbolic AI, and Yann LeCun, who ridiculed me endlessly back in those era, is now working on world models and even is advising a company doing neurosymbolic AI. So the world has actually shifted somewhat in this direction.
We need to bite all four of these bullets. They're all really hard if we're actually going to solve the alignment problem. I honestly don't think we have a snowball's chance in hell at solving alignment. Otherwise, even then it's going to be really hard. But it's time to take a new approach. LLMs are not going to get us there. Let me see if I can skip this. I have one last observation, and then I'll stop and then we'll have a panel. The AI safety community has to stop cheerleading everything that looks to them like, quote, 'terrifying progress.' Like every iteration of the METR graph — maybe some of you saw last week the time-horizon graph, which itself makes no sense, but that's another story for another day. One reason they need to stop doing this is because it gives enormous power to the frontier AI companies, and I don't think it's achieved anything else, all this expression of terror. It's not like it's led to some specific piece of legislation or something like that.
And second, the real question is not so much about capabilities — like, can the system find bugs? I mean, that's interesting in its own right — but it's really about, if you care about alignment, which is what everybody in the AI safety community should care about most, it's really about reliability. We need some kind of system that you can actually count on. And going on about how, you know, there's this or that thing in this new graph is not really getting at reliability. That METR graph is about 50% reliability. Who cares? That's not the thing we need to care about. Autonomy is really where the world is going, and it has to have the highest standard. You know, if there's a human in the loop like Claude Code, the human can fix those errors. But when we talk about AI systems operating on their own, they need to be reliable. If we're going to reach alignment, reliability — whether it's for truth or goodness or whatever — must be our north star. And I thank you very much.
