There's probably a few things to keep in mind here. One is the kind of Tiananmen Square factual knowledge, like how does that get embedded into the models. Two is the Gemini, what you call the black Nazi incident, which is when Gemini as a system had this extra thing put into it that dramatically changed the behavior. And then three is what most people would call general alignment, RLHF, post-training. Each of these have very different scopes and how they are applied. In order to, if you just look at the model weights, in order to audit specific facts is extremely hard because you have to comb through the pre-training data and look at all of this, and then that's terabytes of files and look for very specific words or hints of the words. So I guess one way to say it is that you can insert censorship or alignment at various stages in the pipeline. And what you refer to now is at the very beginning of the data selection. So if you want to get rid of facts in a model, you have to do it at every stage. You have to do it at the pre-training. So most people think that pre-training is where most of the knowledge is put into the model, and then you can elicit and move that different ways, whether through post-training or whether through systems afterwards. This is where the whole like hacking models comes from, right? Like GPT will not tell you how to make Anthrax, but if you try really, really hard, you can eventually get it to tell you about Anthrax because they didn't filter it from the pre-training data set, right? But by the way, removing facts has such an ominous, dark feel to it. I almost think it's practically impossible because you effectively have to remove them from the internet. You're taking on a... did they remove the Tiananmen thing from the subreddits? It gets filtered out, right? So you have quality filters, which are small language models that look at a document and tell you like how good is this text, is it close to a Wikipedia article, which is a good thing that we want language models to be able to imitate. So couldn't you do a small language model that filters out mentions of Tiananmen Square in the data? Yes, but is it going to catch wordplay or encoded language? People have been memeing on like games and other stuff how to say things that don't say Tiananmen Square. But or like, yeah, so there's always like different ways to do it. There's, hey, the internet as a whole does tend to just have a slight left bias, right? Because it's always been richer, more affluent, younger people on the internet relative to the rest of the population. So there is already inherently a slight left bias, right, on the internet. And so how do you filter things that are this complicated, right? Is it like, and some of these can be like, you know, factual, non-factual, but like Tiananmen Square is obviously the example of a factual, but it gets a lot harder when you're talking about aligning to an ideal, right? Which is, yeah. And so Grok, for example, right, Elon's tried really hard to make the model not be super PC and woke, but the best way to do pre-training is to throw the whole freaking internet at it, right? And then later figure out, but then at the end of the day, the model at its core now still has some of these ideals, right? You still ingested Reddit r/politics, which is probably the largest political discussion board in the world that's freely available to scrape, and that's left-leaning, right? And so, you know, there are some aspects like that that you just can't censor unless you try really, really, really, really, really hard. So the base model will always have some TDS, Trump Derangement Syndrome, because it's trained so much. It'll have the ability to express it. But what if, what if you... there's a wide representation in the data. This is what happens. It's like a lot of what it's called post-training, it's a series of techniques to get the model on rails of a really specific behavior. And I mean, it's like you can also have the ingested data of like Twitter or like Reddit r/The_Donald, which is like also super pro-Trump, right? And then you have like fascist subreddits or like you have communist subreddits. So the model in pre-training ingests everything. It has no worldview. Now it does have like some skew because more of the text is skewed a certain way, which is general like slight left, but also like, you know, somewhat intellectual, somewhat, you know, it's just like the general internet is a certain way. And then as Nathan's about to describe eloquently, right, like you can elicit certain things out. And there's a lot of history here, so we can go through multiple examples. In what happened, Llama 2 was a launch that the phrase like too much RLHF or like too much safety was a lot. It's just that was the whole narrative after Llama 2's chat models released. And the examples are sorts of things like you would ask Llama 2 chat, how do you kill a Python process? And it would say, I can't talk about killing because that's a bad thing. And anyone that is trying to design an AI model will probably agree that that's just like, hey model, you messed up a bit on the training there. I don't think they meant to do this, but this was in the model weights. So this is not, you didn't necessarily... there's things called system prompts, which are when you're querying a model, it's a piece of text that is shown to the model but not to the user. So a fun example is your system prompt could be talk like a pirate. So no matter what the user says to the model, it'll respond like a pirate. In practice, what they are is you are a helpful assistant, you should break down problems, if you don't know about something, don't tell them, your date cutoff is this, today's date is this. It's a lot of really useful context for how can you answer a question well. And Anthropic publishes their system prompt, which I think is great. And there's a lot of research that goes into this. And one of your previous guests, Amanda Askell, is like probably the most knowledgeable person, at least in the combination of execution and sharing. She's the person that should talk about system prompts and character of models. Yeah, and people should read these system prompts because you're like trying to nudge, sometimes through extreme politeness, the model to be a certain way. And you could use this for bad things. We've done tests, which is what if I tell the model to be a dumb model, like which evaluation scores go down? And it's like we'll have this behavior where it could sometimes like say, oh, I'm supposed to be dumb, and sometimes it's like it doesn't affect like math abilities as much, but something like, if you're trying, it's just the quality of a human judgment would draw through the floors. Let's go back to post-training specifically, RLHF. Around Llama 2, it was too much, too much safety prioritization was baked into the model weights. This makes you refuse things in a really annoying way for users. It's not great. It caused a lot of awareness to be attached to RLHF that it makes the models dumb, and it stigmatized the word in AI culture. And as the techniques have evolved, that's no longer the case. Where all of these labs have very fine-grained control over what they get out of the models through techniques like RLHF. Although different labs are definitely different levels. Like on one end of the spectrum is Google, and then like maybe OpenAI does less, and Anthropic does less, and then like on the other end of the spectrum is like xAI. But they all have different forms of RLHF trying to make them a certain way. And the important thing to say is that no matter how you want the model to behave, these RLHF preference tuning techniques also improve performance. So on things like math evals and code evals, there is something innate to these what is called contrastive loss functions. We could start to get into RL here, we don't really need to, but RL also boosts performance on anything from a chat task to a math problem to a code problem. So it is becoming a much more useful tool to these labs. So this kind of takes us through the arc of we've talked about pre-training, hard to get rid of things. We've talked about post-training and how post-training, if you can mess it up, it's a complex multifaceted optimization with 10 to 100 person teams converging at one artifact. It's really easy to not do it perfectly. And then there's the third case, which is what we talked about, Gemini. The thing that was about Gemini is this was a served product where Gemini, Google has their internal model weights, they have done all these processes that we talked about, and in the served product, what came out after this was that they had a prompt that they were rewriting user queries to boost diversity or something, and this just made it, the outputs were just blatantly wrong. It was some sort of organizational failure that had this prompt in that position. And I think Google executives probably have owned this, I don't pay that attention to detail, but it was just a mess up in execution that led to this ridiculous thing. But at the system level, the model weights might have been fine. So at the very end of the pipeline, there was a rewriting to something like a system prompt. It was like the system prompt, or what is called in industry is like you rewrite prompts. So especially for image models, if you're using DALL-E or TBT, generate you an image, you'll say draw me a beautiful car. With these leading image models, they benefit from highly descriptive prompts. So what would happen is if you do that on ChatGPT, a language model behind the scenes will rewrite the prompt, say make this more descriptive, and then that is passed to the image model. So prompt rewriting is something that is used at multiple levels of industry, and it's used effectively for image models. And the Gemini example is just a failed execution.
