Generative AI 101: Tokens, Pre-training, Fine-tuning, Reasoning — With SemiAnalysis CEO Dylan Patel

Thank you for having me.

Tokens are in fact like chunks of words, right? In the human way, you can think of like syllables, right? Syllables are often viewed as chunks of words. They have some meaning. It's the base level of speaking, right? Is syllables, right? Now, for models, tokens are the base level of output. They're all about sort of compressing, you know, sort of this is the most efficient representation of language.

Yeah. And each individual token is actually not just one number, right? It's multiple vectors. You could think of like, well, the tokenizer needs to learn king and queen are actually extremely similar in terms of the English language. Extremely similar, right? Except there is one vector in which they're super different, right? Because a king is a male and a queen is a female, right? And then from there, in language oftentimes kings are considered conquerors and all these other things and these are just historical things, right? So a lot of the text around them, while they're both royal, regal, monarchy, etc., there are many vectors in which they differ. So it's not just converting a word into one number, right? It's converting it into multiple vectors and each of these vectors the model learns what it means, right? You don't initialize the model with like, hey, king means male monarch and it's associated with war and conquering because that's what all the writing about kings is in history and all that, right? People don't talk about the daily lives of kings that much, they mostly talk about their wars and conquests and stuff. And so there will be each of these numbers in this embedding space will be assigned over time as the model reads the internet's text and trains on it, it'll start to realize oh king and queen are exactly similar on these vectors but very different on these vectors and these vectors aren't, you don't explicitly tell the model hey this is what this vector is for but it could be as much as one vector could be like is it a building or not, right? And it doesn't actually know that, you don't know that ahead of time, it just happens in the latent space and then all these vectors sort of relate to each other. But yeah, these numbers are efficient representation of words because you can do math on them, right? You can multiply them, you can divide them, you can run them through an entire model. Whereas your brain does something similar, right? When it hears something, it converts that into a frequency in your ears and then that gets converted to frequencies that should go through your brain, right? This is the same thing as a tokenizer, right? Although it's obviously a very different medium of compute, right? Ones and zeros for computers versus binary and multiplication being more efficient whereas human brains are more like analog in nature and think more in waves and patterns in different ways. While they are very different, it is a tokenizer, right? Language is not actually how our brain thinks, it's just a representation for which it to reason over.

Yeah, I think that's the objective function which is just to reduce loss, i.e. how often is the token predicted incorrectly versus correctly.

So, first of all, the model doesn't just output one token, right? It outputs a distribution. It turns out the way most people take it is they take the top K, i.e. the most high probability. So, yes, 'blue' is obviously the right answer if you give it to anyone on this planet. But there are situations and contexts where 'the sky is red' is the appropriate sentence. But that's not just in isolation, right? It's like if the prior passage is all about Mars and all this and then all of a sudden it's a quote from a Martian settler and it's like 'the sky is' and then the correct token is actually 'red', right? The correct word. And so it has to know this through the attention mechanism, right? If it was just 'the sky is blue', always you're going to output 'blue' because 'blue' is let's say 80%, 90%, 99% likely to be the right option. But as you start to add context about Mars or any other planet, right, other planets have different colored atmospheres, I presume, you end up with this distribution starts to shift, right? If I add 'we're on Mars, the sky is', then all of a sudden 'blue' goes from 99% in the prior context window, the text that you sent to the model, the attention of it, all of a sudden it realizes 'the sky is blue' preceded by the stuff about Mars. Now, 'blue' rockets down to like, let's call it 20% probability and 'red' rockets up to 80% probability, right? Now the model outputs that and then most people just end up taking the top probability and outputting it to the user. And that's sort of how does the model learn that is the attention mechanism, right? And this is sort of the beauty. The attention mechanism is the beauty of modern sort of large language models. It takes the relational value in this vector space between every single token, right? So 'the sky is blue', right? Like when I think about it, yes 'blue' is the next token after 'the sky is' but in a lot of older style models you would just predict the exact next word so after 'sky' obviously it could be many things, it could be 'blue' but it could also be 'scraper', right? You know, 'skyscraper'. But what attention does is it is taking all of these various values, the query, the key, the value which represents what you're looking for, where you're looking and what that value is across the attention. And you're calculating mathematically what the relationship is between all of these tokens. And so going back to the king queen representation, right? The way these two words interact is now calculated, right? And the way that every word in the entire passage you sent is calculated is tied together. Which is why models have challenges with like how many documents can you send them, right? Because if you're sending them just the question like 'what color is the sky?', okay, it only has to calculate the attention between those words, right? But if you're sending it like 30 books with insurance claims and all these other things and you're like 'okay, figure out what's going on here. Is this a claim or not?' in the insurance context, all of a sudden it's like 'okay, I've got to calculate the attention of not just the last five words to each other but I have to calculate every 50,000 words to each other', right? Which then ends up being a ton of math. Back in the day actually the best language models were a different architecture entirely, right? But then at some point, transformers, large language models which are based on transformers primarily, rocketed past in capabilities because they were able to scale and because the hardware got there and then we were able to scale them so much that we not just put some text in them and not just a lot of text or a lot of books but the entire internet which one could view the internet oftentimes as a microcosm of all human culture and learnings and knowledge to many extents because most books are on the internet, most papers are on the internet. Obviously there's a lot of things missing on the internet but this is the sort of modern magic of what it was sort of like three different things coming all together at once, right? An efficient way for models to relate to each other, the compute necessary to scale, the data large enough and then someone actually pulling the trigger to do that at the scale that got to the point where it was useful, right? Which was sort of like GPT-3.5 level or 4 level where it became extremely useful for normal humans to use chat models.

So pre-training is called that because it is what happens before the actual training of the model, right? The objective function in pre-training is to just predict the next token. But predicting the next token is not what humans want to use AIs for, right? I want it to ask a question and answer it. But in most cases, asking a question does not necessarily mean that the next most likely token is the answer, right? Often times it is another question, right? For example, if I ingested the entire SAT and I asked a question, the next five tokens would be like 'A is this, B is this, C is this, D is this'. Like, no, I just want the answer, right? And so pre-training is called pre-training because you're ingesting humongous volumes of text no matter the use case, right? And you're learning the general patterns across all of language, right? I don't actually know that king and queen relate to each other in this way. And I don't know that king and queen are opposites in these ways, right? And so this is why it's called pre-training is because you must get a broad general understanding of the entire sort of world of text before you're able to then do post-training or fine-tuning, which is let me train it on more specific data that is specifically useful for what I want it to do. Whether it's in chat style applications, when I ask a question give me the answer or in other applications like 'teach me how to build a bomb'. Well obviously no I'm not going to help you build a bomb because that's what I don't want the model to teach me how to build a bomb. So, you know, it's sort of got to do this and it's not like you're teaching it, when you're doing this pre-training you're filtering out all this data because in fact there's a lot of good useful data on how to build bombs because there's a lot of useful information on like C4 chemistry and people want to use it for chemistry, right? So you don't want to just filter out everything so that the model doesn't know anything about it but at the same time you don't want it to output how to build a bomb. So there's a fine balance here and that's why pre-training is defined as pre because you're still letting it do things and teaching it things and inputting things into the model that are theoretically quite bad, right? For example, books about killing or war tactics or what have you, right? Like things that plausibly you could see like, 'oh well maybe that's not okay'. Or wild descriptions of really grotesque things all over the internet, but you want the model to learn these things, right? Because first you build the general understanding before you say 'okay now that you've got a general framework of the world let's align you so that you with this general understanding of the world can figure out what is useful for people, what is not useful for people, what should I respond on, what should I not respond on'.

Right. I mean basically you have loss is how often you're wrong versus right in the most simple terms, right? You'll run through passages through the model and you'll see how often did the model get it right. When it got it right, great, reinforce that. When it got it wrong, let's figure out which neurons in the model you can tweak to then fix the answer so that when you go through it again it actually outputs the correct answer and then you move the model slightly in that direction. Now obviously the challenge with this is if I first, I can come up with a simplistic way where all the neurons will just output 'the sky is blue' every single time it says 'the sky is' but then when it goes to 'hey the color blue is commonly used on walls because it's soothing' and it's like 'oh what's the next word is soothing', right? And so that is a completely different representation and to understand that 'blue' is soothing and that the sky is blue and those things aren't actually related but they are related to 'blue' is like very important. And so over often times you'll run through the training data set multiple times, right? Because the first time you see it, oh great maybe you memorized that the sky is blue and you memorize the wall is blue and when people describe art and often times use the color blue it can be representations of art or the wall. And so over time as you go through all this text in pre-training, yes, you're minimizing loss initially by just memorizing, but over time because you're constantly overwriting the model, it starts to learn the generalization, right? I.e. 'blue' is a soothing color, also represents the sky, also used in art for either of those two motifs, right? And so that's sort of the goal of pre-training is you don't want to memorize, right? Because that's, in school you memorize all the time. And that's not useful because you forget everything you memorize. But if you get tested on it then and then you get tested on it 6 months later and then again 6 months later after that or however you do it ends up being you don't actually memorize that anymore. You just know it innately and you've generalized on it and that's the real goal that you want out of the model. But that's not necessarily something you can just measure and therefore loss is something you can measure, i.e. for this group of text, right? Because you train the model in steps. Every step you're inputting a bunch of text. You're trying to see what's predict the right token where you didn't predict the right token. Let's adjust the neurons. Okay, on to the next batch of text. And you'll do these batches over and over and over again across trillions of words of text, right? And as you step through and then you're like, 'Oh, well, I'm done.' But I bet if I go back to the first group of text, which was all about the sky being blue, it's going to get the answer wrong because maybe later on in the training, it discovered it saw some passages about sci-fi and how the Martian sky is red. So it'll overwrite but then over time as you go through the data multiple times as you see it on the internet multiple times as you see it in different books multiple times whether it be scientific, sci-fi, whatever it is you start to realize and it starts to learn that representation of like oh when it's on Mars it's red because the sky on Mars is red because the atmospheric makeup is this way whereas the atmospheric makeup on Earth is a different way. And so that's sort of the whole point of pre-training is to minimize loss, but the nice side effect is that the model initially memorizes, but then it stops memorizing and it generalizes. And that's the useful pattern that we want.