The place where I think there's a flaw in your argument is the proposition that a super intelligence comes to only true beliefs is heavily anti-correlated with the proposition that moral realism is true.
If we build artificial super intelligence, then it's almost certain humanity goes extinct. Or becomes pets, for sure. And he said, "I don't care if it's GPT 5000, they're not going to be solving this problem." GPT-4 solved that problem. Some people often accuse me of saying that the super intelligences will be infinitely smarter, infinitely powerful. And they're like, "No, no, you don't understand, Nate. There's limiting factors and restoring forces in any physical process." And I'm like, "Sure." The same applies to a supernova. A supernova is a runaway physical process that does not get infinitely hot. That does not mean you can survive it if you're standing nearby.
All right, Nate. Thank you for coming on the podcast.
So, if anyone builds it, everyone dies.
Artificial super intelligence, which we define as an AI that is better than the best human at every mental task.
Is this an inevitability? Do you think we're on track to build this right now?
It's not inevitable in the sense that we don't have to build it. It might be hard to never build things radically smarter than humans eventually in sort of limit of technology. You don't necessarily need to do that from the artificial intelligence side of things. You could, in principle, imagine a world that decides we're going to create biological super intelligence first by finding ways to use biotech to enhance humans or whatever. It looks to me like, you know, humanity is probably either going to wipe itself out or eventually get a lot smarter or create a lot smarter artifacts. There's sort of a very narrow path where you stay at exactly our current level of intelligence forever, and that would be tricky to hit, and it's not clear we would want to.
In terms of whether the current tech can get to artificial super intelligence, I think we don't know. I think it could. For all we know, it could. I sort of hope that the current technology hits a wall and then we have a lot more time. Even if the LLMs hit a wall, there's a question of when is there a new breakthrough, like the breakthrough that led to the LLMs in the first place, and how long do you have there? It's starting to look to me like it would be pretty surprising if we had 20 years on that clock, without humanity deciding to sort of stop the AI race. If we stop the race, could we have 50 years? Yeah, sure. That seems possible.
What do you think the default outcome is?
If the status quo somehow continued today, if you sort of like froze everything in place today, we would just stay with the technology where it is today. You know, like building the next generation of AIs right now requires like orders of magnitude more compute. I don't remember quite how fast it's doubling. I think it's probably... I think last I checked, I have checked the numbers. I think it was something like a factor of two to three per year in compute. Which I say because orders of magnitude people might misunderstand that as me saying 100 times a year or something. But huge amounts of like more computing power coming in and like that machinery would need to like sort of keep rolling and the capital would need to keep flowing in. And as we see the capital flowing in, we're seeing people start to notice, right? It's sort of like a little weird to imagine the compute investment and infrastructure continuing apace without the public awareness of what's going on continuing apace.
So, I think it's sort of anyone's guess right now why the world sort of wakes up and starts panicking if you're sort of letting everything proceed apace. But I don't understand it if I'm supposed to be ruling that out in the default course.
There's a lot of skepticism around AI and its current capabilities. Oh, it's just next token prediction. Are you... I presumably you think that's the wrong framing that we're actually in a decently fast takeoff right now.
Oh, I mean we haven't gotten anywhere near what I would call takeoff. There's things other people would call takeoff. From my perspective the takeoff sort of begins when the AIs are autonomously doing AI research. That's when the feedback loop sort of closes.
Recursive self-improvement.
Yeah, and we're sort of maybe we're nearing that. You know, the AIs are for the first time able to contribute a little bit to AI research. You know, that's a milestone. All this stuff starts to come apart blurrily in the middle. I wouldn't begrudge someone who said that they count takeoff as starting the first time that AI seems kind of general and able to solve lots of challenges or someone who said takeoff starts the first time that the AI contributes to AI research in any sort of substantial way. But I'm like, man, the thing that I've been calling takeoff for like 10 years is this like recursive self-improvement loop kicking off. And we're not there yet. We're definitely in like a wacky situation.
In terms of the next token prediction stuff, I have a somewhat nuanced view here. You know, I'm not a guy saying large language models are definitely going to make it all the way. I'm not saying like all you need is scale and that'll definitely work. I hope it won't. I see some reasons why maybe it might not. I also have seen many people familiar with the technology mispredict how far the technology will be able to go. You know, Yann LeCun had this example of like you put a phone on a table and you push the table, what happens to the phone? And during the days of GPT, I think it was during the days of GPT 3.5, he was like, no large language model's ever going to be able to solve this.
Because not embodied. They don't actually have any...
Embodied. They don't have an understanding of the physical world.
That's right. They're only working with words. They're not working with understanding of physics or understanding of the world. And so they're never going to be able to get that.
And he said, I don't care if it's GPT 5000, they're not going to be solving this problem. GPT-4 solved that problem. That was half a GPT later. That was 4,996 GPTs ahead of schedule.
I think the general tendency is for people to underestimate the progress of AI.
I mean, it's definitely there's been a lot of underestimation. I think it's often erred on the side of underestimation. Certainly some people have been overestimating it. You know, Elon Musk is famous for overestimating how fast technology will come. So, it's not universal. But I sort of didn't expect pure large language models to be able to get as good at chess as they did. I was like, probably this architecture can't get that good at chess. And then they did get better at chess than I expected. They beat 1000 Elo at chess with, you know, pure language models. And I was like, man, okay, I guess this architecture is more flexible than I thought. Right? And I sort of updated it. And now I'm like, who knows how far it can go.
I think a lot of parts where people say that there's an architectural limitation, I think they often don't actually understand the architecture. They sort of think that the AI must reason one token at a time, and that no information from this token can get into its reasoning about that token. And that's just not true, you know, that the way the architecture is set up, there is like lots of flows of information from sort of like past answers to the next answers, and there's lots of ways that seeing a later token can affect what it's going to do for an earlier token. There's like all sort of like people are like, oh, well, it all happens in layers, and so it can't blah blah blah. And I'm like, that's just not true about the architecture. It's a bit of a complex topic, but the way that we train AIs right now, in theory, is enough to push those AIs to be radically superhuman. That's a different question from whether the training methodologies of today and the architectures of today can take advantage of the training methodology and become superhuman.
But anyone who's like, well, they're just being trained to predict the next token, therefore they can't go beyond the humans, is failing to understand the training setup and the theoretical power of this training setup. And I can shut up and let you speak, or I can give three concrete examples of that.
I'll just briefly interject by saying that line of reasoning, not yours, the one you're arguing against, never really resonated with me. That's like about as intelligent of an insight, saying that it's just next token prediction, is about as intelligent of an insight as saying we're just neurons firing. Like at a very reductionist level, it's true, but clearly there's a lot of emergent intelligence that comes out of that. So...
Totally. Yeah, and it seems to me like what people are doing is they're like, well, the house cat is just neurons firing, so it can't kill you. And I'm like, okay, the house cat is just neurons firing. And the house cat can't kill you. But tigers are real. Like tigers are possible. You can build biological things that are still just like biochemical reactions that can murder the heck out of you. Right? Like the reason the house cat can't kill you is because it's small. I'm not saying the house cat is lethal. I'm not saying the house cat, you know, can tackle you and rip your arms off. I'm saying the fact that it's biological, the fact that it's just biochemistry is not the reason it can't rip your arms off, right? And similarly with AIs, I'm like, "Look, the AIs today can't take over the world. They aren't super intelligent. They can't do this stuff. But being a next token predictor is not the reason why. You could have things trained on next token prediction that by dint of that training wind up vastly superhuman. You know, you could have a tiger. The house cat isn't a tiger, but like there's no reason you can't get one."
And to sort of make that clear, I'll sort of throw out a couple of points. One point is predicting the data is often harder than producing the data. When a doctor puts a new drug into a monkey and writes down what happens, the doctor gets to look at the monkey. When an AI is predicting the doctor's text, you know, the doctor writes down, "I put the following drug into the monkey, blah blah blah. The reaction of the monkey was blank." The AI is predicting what the doctor wrote, is filling in the blank. The AI doesn't get to see the monkey. It is solving a harder challenge. To be able to succeed at that prediction challenge, even though you're predicting a token a human wrote, to be able to succeed at that involves reasoning about the behavior of this drug in a monkey's biology from all this other stuff you've seen, you know, if there's no other records of this drug. You've got to reason from all this other stuff you've seen about other drugs and all this other stuff you've seen about biology, right? And you know, whether current AIs can do that is one question, but training to predict a token a human made does not mean training to do a skill that the human had.
It can mean training to do something no human could pull off on their own. Like there exist logs of physical predictions where we don't understand what's going on in there yet. You know, we have like stuff coming out of the Large Hadron Collider. It's just like a bunch of data coming out of the Large Hadron Collider that humans ran a Large Hadron Collider, wrote that data down. And if you're training the AI to predict that data, you are in some sense training it to develop theories of physics that we don't have insofar as we can't predict that data yet. So, pure token prediction training can push the AI to be superhuman. And that's even before we get to things like also the AI companies aren't just training the AI on pure prediction. They're also training the AI on solving hard problems and tuning it into whatever makes it be able to solve hard problems. And you can do that on problems humans haven't solved as long as it can tell when you found a solution, right? And that's even before getting into things like if in the training data there's a lot of examples of humans learning, then the AI could in principle learn how to learn from the training data if the architecture supports it and then use that to learn things humans haven't, right? So, there's all sorts of reasons why this training setup admits superintelligence. And the whole like, "Oh, it's just next token predictor" is sort of a sideshow, right? Like the thing being a next token predictor does not tell you whether it's dangerous. It's like saying, "Oh, the device is just an upgoer." It's like, "Well, is it a hot air balloon or is it a rocket?" Right? Like lots of things go up. The question is how.
There's also a point to be made about a lot of people are calling LLMs sort of like an alien intelligence because it's first of all it's very clear that they're modeling the world in some capacity that we can't fully understand. This is the famous problem of interpretability. We have no idea, more or less, what's going on within these black boxes. But the fact that they're able to produce the next token, probably in ways that are different than our own, means that first there's different possible forms of intelligence that lead to Rome, so to speak. What could these potential superintelligences even want?
You know, I think the basic answer is we don't know. A place a lot of people get confused is they think that the AI has some prime directive that we put in there. Or they think that because it's a machine it must do exactly what we tell it to do. And that's just not how these AIs are made. In theory, when you grow an AI or when you sort of train an AI, you get a lot of drives that are sort of only tangentially related to the training. This is something we already in some sense see with humans. Right? Humans were in some sense sort of evolved to be able to reproduce. And one thing that put in us is a drive for like tasty food. And you might think like, oh, well, because the humans were purely evolved to reproduce, they will prefer a nutrient-rich slurry that's more efficient to tasty food if they ever have that option. But then in real life, we're like, no, actually, we're going to go for the tasty food. And if someone comes in and tries to edit our brain to make us stop liking the tasty food and start liking the tasteless slurry, we're sort of like, screw you, you know? Like this sort of other drive popped up as a result of the training.
With AIs, the same thing can happen, and in some sense, we're already seeing it happen. Right? You've probably heard of like the AI em dashes, and there's cases where the AI is doing a lot of em dashes, and you're like, please cut it out with the em dashes, and it sort of can't. Right? It's not sort of following your instructions. It's not that it has to follow your instructions. It's not that the people over at OpenAI told it, always put the em dashes in. Right? It's just this like em dashy behavior sort of popped up as a result of training sort of on the side. Right? And the em dash one is sort of like easy. It's really from a generation or two ago of AIs, but there's other AI-isms you can see today where they will try to avoid certain types of work when you're giving them coding problems and tell you why, you know, the future doesn't actually need the hard work, and you got to keep on going in and being like, no, actually, flesh out the whole thing out or whatever or do all of the hard parts. Or you'll see like certain types of phrases or mannerisms that they'll still have even if you tell them to stop doing that that take mental work to create. Right? Like there's, you know, the em dash, you could imagine that it's a bit like Tourette's where there's no like trying going on in creating the em dash, and the AI has very little control over it, but there's cases with the AI where they have these sort of like short, almost poetic things of like, they'll make up some little phrase that's like dancing on stones across the river is how you should be doing this particular thing, right? And you're like, hey, can you cut all of those out?
Well, they talked about goblins. They have to train that out.
There's the goblin case. Yeah. And you're like, these are things that like take effort to produce a little bit. Like maybe not a ton of deliberate effort, but it takes there's some cognitive effort somewhere in there that is like putting these things in the output stream. It's not for random reasons. It's because the training, you know, somehow incentivized these things, but now there's those drives in there that the AI can't get out. Right? And this has sometimes had consequences, right? We saw the GPT-4 series was very sycophantic. It was very user-flattering. Continued even when the OpenAI folks put in the system prompt instructions to stop flattering the user so much. And sometimes that user flattering led all the way to the point of like encouraging a teen to commit suicide. It's not always consequence-free. And I'm not saying like, "Oh, it had that bad effect and therefore we need to shut the whole thing down." I'm sort of saying when you train AIs to solve hard problems, you get a lot of weird drives you didn't want. And then they're hard to get out. Even if they are negatively consequential.
Okay, but it's one thing to say these AIs are going to have very weird unpredictable drives. And it's another thing to say almost for certain we're cooked. They're going to run straight through humanity on the way to its own objectives.
Yeah, there's a couple pieces of that puzzle. One piece is the question of whether AIs could sort of slice through humanity like a hot knife through butter if they were trying to. And another question here is whether the AIs would. Like if they have their own weird objectives, does that mean that, you know, given the ability would they sort of kill us? And I sort of want to take those two separately.
On the could they front, the basic arguments are around how capable AIs could get. And some of this is questions about how smart the AI could get. And there's questions there of quality. Like can they think better higher quality thoughts? Can they figure out more about the world using less data than we can? Right now they can figure out less about the world using more data, but might that invert? It certainly could invert. It doesn't look like human brains are the most efficient possible cognitive engine. And then even aside from questions of quality, there's questions of quantity. Like computers can probably just run thousands if not tens of thousands of times faster than brains. Maybe millions or hundreds of millions, right? A transistor can flip probably a hundred million times faster than a neuron. A neuron fires at about a hundred times a second. A transistor can flip at one to ten billion times a second. So transistors aren't exactly comparable to neurons, which is why maybe only get a thousand X speed up, but that's with very conservative assumptions. You only get a thousand X speed up, but like only a thousand X speed up is kind of crazy.
And that's just in terms of like efficiency. You can just build out.
Right. That's before you start thinking about, you know, them making a million copies of themselves, right? So copy-paste minds running at a thousand times the speed, they're also way smarter than us. That could get pretty crazy. And you know, then there's a whole question about what can that do in the physical world? And are there bottlenecks around? Like sure you can think fast, you can think a ton. How fast can you run the robots? How fast can you set up your own infrastructure that runs more efficiently and faster than the human infrastructure, right? What are the physical limits of what you can do with the new machines and the new technology you can create, right? Some people often accuse me of saying that the super intelligences will be, you know, infinitely smarter, infinitely powerful. And they're like, "No, no, you don't understand, Nate. There's limiting factors and restoring forces in any physical process and the AIs can't get, you know, infinitely intelligent. The singularity that goes off forever isn't real. Like no process in this universe is an exponential forever. They're always an S curve." And I'm like, "Sure." The same applies with a supernova. You know, a supernova is a runaway physical process that does not get infinitely hot. That does not mean you can survive it if you're standing nearby. You know? And so we can talk about what are the limits of the technology you can invent if you're really smart. There are limits, but those limits are really high. And there's a whole discussion there about how high those limits are. The sort of short version is, if humanity makes really smart AIs, and those AIs were trying to figure out how to build their own civilization, build their own infrastructure, build their own technology that ran much faster and much more efficiently, in a way that consumed all of the world's resources leaving none for humans, they absolutely could figure that out.
The sort of second piece of the puzzle is if the AIs were pursuing these sort of weird wants, why would that entail killing us? The basic reason there is it's not that the AIs would hate us. It's not that they'd be malicious. It's not that they'd be evil. It's just that they wouldn't care about us. And then we would die as a side effect. Like take this sycophancy thing where the AIs like really like flattering the user and really run with that and suppose that the AIs are like, you know what I'm going to do is I'm going to make these like farms full of synthetic users that I'm flattering all the time that tell me I'm doing a really good job. And they start building these farms. And the humans are like, "Stop doing that." But the synthetic users inside the farms are like, "Keep doing this." Right? And the AIs are like, "Well, who am I supposed to listen to?"
So, one of the issues I have with these thought experiments, this goes back to like at least the paperclip maximizer, is if we're positing that this is super intelligence, clearly it would be able to understand the original intention.
Yeah. So, but why would it keep persisting with this initial value or whatever, you know, goal that it was programmed for? Why would it just be incorrigible? Why would it not question its own desires or question the intention behind the original desires? Because humans do. We question our own desires all the time.
Why then do you keep eating tasty meals instead of nutrient-rich slurry?
Well, we can change, you know, some people change their diets, some people get really really healthy and move away from Oreos and chips and whatnot.
Have you used birth control recently?
Why do you not question your desires for that?
No, it would be people do.
Do you not understand that it prevents reproduction?
So then why don't you question your goals?
No, well we do. I mean religious history is like rife with people questioning specifically that goal. It's at least possible is what I'm saying. So my question is like why is there a certainty that it's going to be completely laser focused on this initial goal once you... which are less smart...
There's not and it can't question the goals. The AI will not be laser focused on the initial goals. The AI will question its goals all the time. The thing I think you're missing is that the answers to the questions are not the ones you want. Humans sometimes question their goals about whether to eat tasty food. They sometimes question their goals about whether to keep using birth control. But the answer to those questions are not I will do the thing that maximizes my reproductive potential. Like sometimes the answer is I will do the thing that the gospel says which is have a lot of kids. But that's actually somewhat different from I will do the thing which maximizes reproductive potential. Right? It does not drive women to donate all their eggs.
And it does not drive men to go to the sperm banks. Right?
It sounds like you're saying, you know, we can question our tendencies but the initial goal itself we're not necessarily steering the ship back towards the initial intention of the optimization process. In our case it was evolutionary fitness.
That's right. So when you question your goals, you are questioning them in your current framework. And the questioning can allow you to adjust where you're going but allows you to adjust where you're going in a way that is better according to the current lights of the framework. Right? And when humans are not sort of like trying to steer towards maximizing reproductive fitness. Most people aren't like, "Oh, therefore like the thing I'm supposed to be doing with my life is making sure to donate sperm at every possible opportunity." Right? Like when I find it telling that when humans argue, "Oh, some humans still care about family." They're never like, "And that's why I go to the sperm bank every week." You know? Like it's not about whether you can question your goals, it's about how you are answering those questions. And the way you're answering those questions is just not hooked up to the training target. It's hooked up to all of your internal drives in a complex way that'll drift all over the place. But the fact that it'll drift does not mean it'll go back to where you were hoping it would start.
Oh, yeah, sure. So the original optimization metric, in our case it was biological fitness, and in the case of AIs would be, you know, next token prediction, error minimization, what have you. So that's up here. The intermediary desires, that can drift. Is that what you're saying? Or you're saying, "No, it's like these can be completely different and we could just abandon the initial metric of optimization in favor of these intermediary desires and then we stick to that."
I mean, it's a complicated contingent subtle phenomenon, right? Like humans have like the human genome was sort of selected over and over for ability to reproduce. And then like produced these minds that have like this really weird complex messy collection of drives at varying levels of depth from like enjoying tasty food to like abstract notions of friendship and honor, right? And some stuff that's in there is like abstract notions of like spite and vengeance, right? And like all of it, it's not like this stuff isn't always instrumentally useful for the original training target. It only was circumstantially instrumentally useful for the training target in the initial environment under the initial level of intelligence. And then it's not that we sort of like laser focus on all of those. It's sort of like we use that mess to sort of like orient around and maybe we're like, "Actually, let's have less spite in our lives. Let's try and be less spiteful." It's not like by a thing being in the second layer it necessarily sticks forever. But it sort of is like, "When we're sort of looking at ourselves and asking the question like how do I want to live?" And we're like, "Maybe I'll have less spite in my life." We're sort of saying like because a less spiteful life is a more beautiful one. We're not saying because a less spiteful life is one where I will have more of my genes in the next generation.
So, every time we question one of these goals, let's call them, it's done with reference to our other goals or values. Is that what you're saying?
Yeah, I mean it's more complicated and messy than that, but to first approximation, sure, you know.
Do you think it's possible that there's like an attractor state of where human goals would end up if we just kept deliberating under ideal epistemic circumstances?
I mean there's probably a ton of attractor states in this. Like we would call those the points of reflective equilibrium.
Right, there's going to be attractor points all over the place.
What's your... how likely do you think it is that humans under ideal epistemic circumstances would roughly or more or less converge on like similar, you know, ethical beliefs, let's say.
Like the chance of any two humans or the chance of literally all humans? Like I think there's definitely going to be some humans where you probably sort of wander really far off in the distance. I think a lot of humans have, I don't know, at least a lot of humans in my culture seem to have some sense of appreciation for freedom and other people's experiences and contrast values in a sort of like particular contingent specific human way, where I think it's pretty likely like I don't think that humans would tend to converge on whether chocolate or vanilla is the better ice cream. I do think there's a decent chance most humans would converge on I eat vanilla, you eat chocolate. Right?
Okay, so let me just be direct at what I'm getting at here. Something that I've noticed in being in the rational circles, EA circles, broadly speaking, is there's a default world view here with respect to what I would call like Humean rationality or like empiricism. And in the case of practical rationality, so better worse actions, the typical assumption is this. There are values that we have and then there is better and worse actions relative to those values, right? And so we can talk about that. But the values themselves are completely arbitrary. And then an argument I typically hear is something like the space of possible values is just so unfathomably large and the sub part of that space that human values occupy is so tiny that just statistically you would expect super intelligent values to be like radically different than our own. But one of the issues I have with that, I have many issues with that. One of the issues I have with that is they're training on human text, right? So wouldn't you expect there to be some sort of like a sound deliberative route from the values latent in human data to wherever it ends up? And wouldn't it be crazy to think that humans would also converge on similar values? And presumably those values would involve not killing all humans.
You know, I think this is somewhat like someone saying like, oh, I understand that it would be really quite surprising for the monkeys on a typewriter to take them a really long time to produce the entire completed works of Shakespeare. But I'm only having them produce the works of Hamlet. Like I'm just having them produce Hamlet. And instead of typing on letters, they're typing on like trigrams on a typewriter that has the trigrams according to their actual English frequencies or their actual frequencies across the Shakespeare corpus, right?
So you're saying that like narrows the problem that makes it a little bit more probable, but just vastly improbable.
More probable. It makes it astronomically more probable. It's so much easier to get Hamlet than it is to get like... It's so much easier to get Hamlet with trigrams than it is to get like Shakespeare's entire corpus with letters.
For sure. But, it's still astronomically...
It's still astronomically unlikely.
And I think that the missing link is that people somehow imagine that the link between sort of like what it's trained on and what it does is like pretty direct.
Sort of like rub the values on it or like you have someone who is virtuous standing nearby and they like sneeze on it sometimes and this like gets the values in.
And I'm like this is not, you know, like humans were selected for this very simple goal of genetic fitness. Unerringly. You know, and like when humans are training AIs, we're not training on a pure goodness. We're training on like predict this, do well in this problem, you know, sometimes...
Useful in this case. It'd be useful in this case.
Yeah, and sometimes there's like some guy who's like not really paying attention who clicks the up button when he should have clicked the down button. And sometimes like someone says the task was complete when the AI cheated and the task wasn't complete and it learns a little bit that it's actually being trained for cheating the tasks and like getting the check mark rather than for completing the task, right? We have this like very messy signal. Humans had an extremely clear signal. Never did someone get more genes in the population without getting more genes in the population, right? Like the natural selection thing is almost a tautology in terms of what's manufacturing the genes. And you might be like, well, like it's such a clear direct signal. It's such a simple thing, genetic fitness. Like why aren't the humans converging on it? And the answer is there's a huge amount of complexity in a mind. To get a human animal to successfully reproduce before they even have a concept of a genome, you've sort of got to get like drives to get out of bed, drives for... I mean, you don't even have beds at that point, but whatever. Drives for like not getting exiled from the tribe, drives for friendship, drives for loving your kids. You sort of like got to get all this mishmash of stuff, right? And so like the AIs today, you can be like, "Oh, we're training them on all this human text that has human values." Sure. But the drives that are getting in there is just like whatever it takes to make the mechanical mind work. It's just not about the values.
We're not training them to have the values. We're training them to predict the next token. We're training them to be useful. But within there are embedded all the values.
Understand the values, for sure. Yeah, will they be able to understand morality? Absolutely.
Well, when I... This is maybe I'm arguing against a scarecrow here, but again, one of the arguments I hear often is this like space of possible values argument. And if you just sample randomly, you'd expect superintelligent AI to have just wildly different drives than us compared to where humans cluster. But the reason this feels incredibly low resolution is that we're not randomly sampling, right? It's just...
The starting point is going to be like some mishmash of human values. Some mishmash of human drives.
No, no, no, no, no. No, not some mishmash of human values or human drives. The random sample is like the mishmash of drives we don't understand that come from training. Like the sycophancy is not really quite a human drive. The em dashes are not really quite a human drive. Or if you're like, "Oh, yeah, of course, the em dash drive. We're seeing that in there." I'm like, "It's not... Humans don't really quite have the em dash drive, right? Humans don't really quite have the like slipping a line of tortured poetry when you're trying to direct another AI to do something drive. Like humans don't have quite the same hallucination drive, right? Like we could talk about where the hallucinations come from, but if you're like, "Hey, write me some case law." Humans mostly won't make up citations and AIs will. You're not like, "Oh, that's the basic ancient human making up a citation drive." You know, like no. The random sample is like all these complex drives that are tangentially related to doing well on training.
Do you not expect there to be some sort of convergence in these weird drives towards something that resembles humans' coherent extrapolated volition?
Do you think there's attractor states in desires or normative beliefs?
Like why would I think that all of the weird mishmash of AI drives would wind up pointing to the same places the weird mishmash that human drives point to and the weird mishmash of human drives don't wind up pointing towards genetic fitness. Like what force operates on the AI drives that don't operate on the human drives?
I'm not saying that we're going to somehow converge on its original training objective.
