All right. So, does anybody want to hear about a little short story about AI?
Let's do it. Like, nobody wants to hear.
No, I was yesterday because we were talking about this and I have this, you know, somewhat funny idea about it. So, I'm an electrical engineer and mostly a computer architect. And there's been times when I wrote a lot of software, but to be honest, I don't write AI software. I use AI and then we make AI software go faster. And I run a — I'm CEO of a company called Tenstorrent where we have a thousand people, probably four or 500 working on an AI software stack. And some of them really get it mathematically, you know, and some of them just get how to build compilers and make levels of software. But when you run AI software, what you find out is...
That's actually — that's all of AI software, you know. Now, of course, you repeat this, you know, there's a repeat statement, repeat 10 to the 18th times that calculation. So at the fundamental level, AI software is pretty simple today, right? And then there's arguments about does this have anything to do with how a brain works or not, right? And I think maybe it does, or maybe the actual calculation doesn't matter very much, but this is the core component of a matrix multiply. And generally speaking in matrix multiply...
So you draw a picture like this, you say I want to multiply matrix A — well, in this case, you know, maybe it's B, B and C — but to get up a new matrix. And basically a matrix multiply says, so say this is a, you know, I'm going to take a row and a column of the matrix multiply and I take that guy times that, plus that times that. So that's the times. So this is indexed by going across there by I, say, and this is indexed by J going across there, and this sum — this is just accumulating the sums of this and this gives you that piece of data. So this is just matrix multiply. You go look, there's all these pictures: take a row times a column and get a piece of data. And then that piece of data is this row times that column, and that piece of data is this column times that row. And you slowly do it and then you find out that the computation is something like what we call n cubed operations on n squared data.
Right? So we do a lot. So you say I got this data and this data and then I have this ratio of computation. So matrix multiplication is computationally intensive. Turns out...
Turns out, which is cool and in an interesting way, it's the relationship between everything here and here, right? And then when people talk about AI today, they will mark one of these called weights and one of these called activations.
Okay, and then that gives you a result which is sort of the relationship of this to this. Matrix multiply is some level of universal function — you can build a MUX out of it, an adder. This data here could have just ones on the diagonal. It could have a checkerboard pattern. It can be a lot of things. A lot of times when people look at big matrix multiplies, they'll say there's a region with data, but it's all zeros, right? So, you know, and you could say this is a learned relationship about, say, cats to fur, right? And this is a learned relationship about pointy eyes or something. But there's not really any relationship between pointy eyes and fur.
So there are zeros. So who knows, right? So again, as a computer scientist, you know, I think this is interesting. And then technically there's a pipeline because then you multiply this times what's called — like there's a whole bunch of words for it, like there's sigmoid functions, there's, you know, take the strength of this stuff, but you kind of get the activations for something else and then...
Amazing. And then modern — and then modern LLM, there could be a hundred layers of this, so it doesn't just feed back, it goes... Right? And then there's a couple other features like embedding. How do you take a string of words and make them into a matrix so you can multiply it? Because we don't typically think that...
I see a dog and the dog is brown. So in AI you can add those two sentences, right? I see a brown dog. Does that make sense?
It's amazing, right? So it turns out in English language with normal spellings, you can't add those two sentences together. But with proper embeddings in a trained model, you can. You can also query it. You can say, what color was the dog you saw? It's a brown dog, right? So AI is a wild thing. So the basics are unbelievably simple. And then the next thing I'm going to tell you is going to really screw up your brain.
If you don't know how this works, but questions.
Oh, no. No. I was just going to say, is there some notion that has been discussed of maybe like concept space or something? I know Stephen Wolfram may have mentioned that at one point. I don't think it's the same, but what I see there, I'm like, all right, what's the concept? Something — attributes of a dog, seeing a dog, to try to find where — I don't know, maybe it's just next to there.
Yeah, I'll try to give you a representation of this. But so all right. So AI uses matrix multiplies and then some activation functions which tells you like what's the strength of these relationships or something. And this is also trained. This is trained and it's all good clean fun. It has this computational intensity...
Computational intensity which is high, n cubed on the data, which is amazing. That's why AI computers work. And then you can do amazing things like take things we think of as a sentence and put it in a space where you can manipulate it. Right? Does that make sense?
All right. So then I'll say something about — so your brain is made out of cortical columns which are little circles and inside the cortical columns are, you know, they call macro columns and minor columns, right? And then these are in layers. I think a human brain averages six. So human brain has 20 billion neurons — I'm going to just use powers of 10 because you guys are physicists and you get this stuff — so, you know, 10 to the 10th neurons, and there's something like 10 to the 6th cortical columns. So each one of these has 10 to the 4th neurons, right? And then wildly, this is all the neurons in each layer talk to each other. It looks like an all-to-all calculation. Super fun.
And then if you look at the brain, it's this uniform mat of cortical columns. It's shockingly uniform. So your cerebral cortex — the lower brains are less uniform. But your cerebral cortex, where we think, we do most of our thinking, and I would posit it evolved as a planning accelerator because simple animals have two neurons: one for sensing the world and one for motor — like, you know, somatic and motor nervous system. And then as animals evolved, they started having, I want to get there and I want to get away from that, and then you have emotions around that. But at some point, the animal that could say, if I do this and this and this, what happens? Or if I do this and this and this, what happens? And the animal who could plan the furthest in the future accurately won. Currently that's us, for five minutes. Okay.
And then there's some really cool features of — so your average neuron has about 10 to the 5th connections. So you've everybody seen that picture of a neuron where you've got these really long things and there's all these little connections and dendrites and axons. Okay.
And if you just take the math of neurons, so the math of computation for a brain is something like 10 — so a common number is 10 to the 18th operations per second, which is a relatively big number, and that's somewhat equal to 10 to the 10th times 100 hertz — that's how fast it fires — times 10 to the 5th. So that's 10 to the 17th times some mysterious number of operations that happens right there, 10. So every input that comes in, there's some number of local operations that does it, and then the neuron itself is summing up all the inputs...
Which is cool. So 10 to the 18th operations per second, there's a brain. And then the really wild thing is the cortical column is about 0.1 teraflop. Right? Teraflop. It just falls out of this math. And most of the computation — oh, most of the connections are in the cortical column, but they have lots of connections adjacent and then less further and then less further.
And then your neurons have a really wide speed range. So in the layers here, the neuron propagation is a fraction — you know, it's like a millisecond or so — but the further you go, the speed stays pretty good, right? So the signal to your hand is really fast. So we have a half a dozen methods to modulate nerve speed. So long nerves are really fast and short nerves are really slow. And that one thing that as a computer scientist I keep forgetting: multiple firing. So when this guy is thinking about [__] and it says go kick this, it wants to kick it together. It doesn't want to kick half of it with half the information, the other half, and then the whole thing and waste energy. And also the slow neurons are really low energy.
So as your brain gets bigger, your brain is technically 1.3 meters of cortical column, six neurons thick, right? And we grew it just by having genes that say make more cortical columns. So our brain architecture hasn't evolved for millions of years. Once mammals got cerebral cortexes, we grow just by — can the animal support that much computation. Does that make sense?
So all right. So then I'm building a computer which is made out of arrays of tensor processors. So they say tensor, they could have said matrix, who cares, right? And our tensor processors are about five teraflops. And you could argue that's too big. But we also have to manage a lot of software to do it. So TPUs, this number is really big, like, you know, half a petaflop. So they made very large tensor processors. Some people have made small tensor processors. There's a tradeoff because when I take a big matrix and I put it across this chip — so this chip has 140 tensor processors on it. So we lay the matrix out across things. And then...
All-to-all calculation means every piece of data has to get everywhere in the row or the column, right? There's a power cost of moving the data. So if the tensor processor is too big, you put too much power in the tensor processor. If it's too small, you waste more power spreading it around. So neuromorphic architectures are super efficient about how they signal each other. So they've been able to tolerate smaller neuromorphic processors with more communication. I think our processor is pretty efficient at doing this. Does that make sense? So...
Again, like I said, I don't know what this exact thing is. To me, it looks like a matrix multiply. It looks like low precision signals. Today we do — most of the math here is on four or eight bit floating point, which seems like a ridiculous thing. People have done one bit, you know, data types with calculations. And some people think the signal between neurons is a single bit. Some people think because of timing, because of how it transitions, it's multiple bits. But there's been a convergence. The kind of math you do in a tensor processor and the kind of math that you do in a stack of neurons is fairly similar. It's between one and a few bits. And the number of operations is also similar. It's within an order of magnitude of teraflops. There's trade-offs, right? And then neurons do an association matrix multiply and then an activation, which is what do we fire next, which is exactly what do...
Okay.
Question: is the brain digital or analog?
It's mostly digital. It's mostly digital.
It looks mostly digital because things fire, and firing is a discrete event.
And it's semi-synchronous. You can see waves of computation happen. And the reason you want synchronous is you don't want to do half a calculation and then something else shows up and you do it and then it inverts the calculation you just did. Right? So, but it's not exactly synchronous. But there's a whole bunch of methods in the brain that make it look like it's firing in patterns. So it's mostly digital and mostly synchronous.
And as I understand, most of the signals in the brain with neurons have to do with frequency. They're like little tiny songs being sung to communicate between each other as opposed to simply on or off or amplitude modulation.
Yeah, there's a lot of theories and papers about this and I'm not an expert on that. I'm taking a relatively simple explanation of how firing leads to accumulation of results, but people do see lots of really cool behaviors in that. But also when neural networks are running, we see lots of really cool behaviors in the neural networks. So, and again, I think there's a lot of work to do. All right. So I'm now building a computer. So the human brain...
Human brain — so people think it's on the order of 10 to the 18th operations per second. Now there's people who think it's quantum or this or that. I don't know. We're getting really smart results with way less computation than that. And they think it's on the order of one trillion — no, 100 trillion parameters.
Yeah. So parameters is the number of weights in the brain. So the brain generically has...
10 to the 11th parameters, 10 to the 18th operations per second.
Yeah. And then I'm building a server which has about 10 to the...
10 to the 7th difference.
Yeah. So I'm building this fun computer which is 10 to the 16th operations and it's 10 to the 10th data. So if you take this times 100...
I'm building a computer. I'm building a one human operations per second computer. A one hop.
I'll let you know. It's powering on.
Human operations per second. One hop. I think that'll just make that capital. It's a one hop.
So this is, you know, just, you know, we're building this training computer and it just happens to have one hop's worth of computation right now. Right.
Yeah. Yeah. How long until it's...
Yeah, a couple months. We're...
Well, this — by the way — is not that big. There's way bigger computers. They're just not coordinated, and it depends how you do it. And then there's also a funny thing of human beings — like, so the modern training runs, they try to run them in one to three months for the really big models, and human beings mostly take 20 years to train.
So, you know, it's a different — so the training computers for modern AI are way bigger than human size just for that reason.
Because they're — my respect for someone who's enough of a mad scientist to actually keep a human brain in a vat in their mad scientist lair.
Yeah. Yeah. And by the way, this right now costs about 100 — costs about $10 million, which, you know, we're going to drive it down 100x in the next couple years. So $100,000 to build a human brain's worth of computing that takes a couple years to train. It's, you know, but there's a couple wild side effects. All right. So I want to point out — walk through two things that people assert. So LLMs are limited and we don't have enough data.
All right. Right. So if you go on the internet news, you'll see this. People, you know, people love the argument: we can't do something, right? Wow, it's amazing. All right. I made a strong argument we can't do something, but nobody checks like five minutes later when it's done. You know, do you come out and apologize? No.
There's no apology channel on this stuff.
Yeah. Yeah. It's a funny thing.
All right. So these are very common statements: we can't get artificial intelligence with LLMs because they're too simple. And what I tried to walk through for part A was the core part of this is simple, right? The core part of how neurons fire and talk to each other is really simple and curiously related in terms of operations per second, number of exchanges of information, data type.
Like it's pretty interesting right now. We don't train — like we don't train LLMs today like brains are trained. As a matter of fact, it's not exactly clear how a brain is trained, right? Which is a curious phenomenon given that what we call the forward pass of computation looks similar, but the backward pass doesn't. And I'll just give you a picture. So we can think of it as a bunch of — let's see — people, I'll just draw boxes. Let's imagine each one of these is a pile of this, and then we put some input in. Right?
And you can think of it as inside the box there are the weights and then activation. So each block has weights, or what we call parameters, right? And a human has 100 trillion of these, which is a fairly big number but not radically big. It's a couple of disk drives these days, right? And then the way these are trained today is called guess the blank. So imagine you put the sentence, I see a dog. I see a dog. And then what comes out of here is...
Right, noise. So we start the training with all the weights set to noise, right? And then we do one more clever trick, which is what they call guess the blank. We don't put that word in, and we are asking this to produce the word 'see'. And then we subtract what we got from what we're looking for, right? And that's the difference.
And then we simply push the difference back through the training. And this is where Hinton and a couple of geniuses figured out something called gradient descent. So these could be very large matrices, say 2,000 by 2,000. This might be 100 deep, right? That's a lot of [__], right? And so what you don't want to do is just take the whole difference, which you can just calculate — just ask, right — and just drop it there. We want to do some calculations on why was this wrong compared to the difference and why was this wrong. So gradient descent is a mathematical formula, and I'm sure if you go into Wolfram Alpha...
Different weights waiting for — how do they know what they're waiting for?
Generically, you don't care.
Oh, you don't care. I got, I got, no, so...
When they first started doing what's called convolutional models — so what's called CNN, convolutional neural network, versus LLM, large language models — at the bottom, one uses convolution as a primitive, the other uses matrix multiply as a primitive. In the layers, like say you put in — you're looking for cats in an image — some of the layers would look like they had trained for pointy ears or round eyes, like you can see. And the image you put in is segmented, and there's some clever tricks to get this to train fast, but an LLM and your brain look unbelievably generic, right? And mathematically setting this up to kind of allocate stuff different ways can speed up training. And there's a current idea which is called mixture of experts. So there's a core model and then there's a bunch of different models around it, and somewhere in the input stage they said this is about politics or this is about math or this is about horses, and then it's sort of generically training these things, but they're only training like this is the horse expert and the politics and the math and, you know, music, right? So it's a little direction so that...
There's, you know, people are going through this: is this really better or just easier to train? Or, you know, because sometimes you want to know about the relationship between horses and music. You know, it's kind of complicated. All right.
And how do you actually break...
Yeah. And then you can do the same thing with a picture. So you have a picture and there's a boy and there's a dog, a cow, right? And then you put the picture through the pipeline and you put a blank in and you guess the blank, right?
Is that how we get a minator?
Is that how we get a minator?
Right. And then you just do the same thing.
Okay. Oh, so for a page, it's basically just filling in what should be given the parameters the most.
All right. So what the hell is happening here? All right. So then it turns out for these large models, the more data you give it — so there's two things. If you have way more data than parameters, the parameters get what's called overfitted. They can only see what the data set says, right? And there's a couple really good curves which says, you know, there's this line and then people can't get past the line. And the line is the product of computation times data times parameters. So as models have gotten smarter, right, we get more data, we get more parameters, right? So if you have a gigabyte of data, a terabyte of parameters is too many, but a megabyte of parameters is too few. It'll be overfitted, right?
Okay, this is starting to explain why, because allegedly...
And I'm about to say this is [__] in a second. So just don't get too attached to it.
It's a really — allegedly people were paying like Wolfram Research to just buy just a whole bunch of just random math. Like we actually do the computation here, but they were buying it to help train the things, which I think even Wolfram's like...
They want to give us money for just a whole bunch of gobbledygook, fine.
Yes. Well, but by the way, it's, you know, they say the unreasonable effectiveness of math — the unreasonable effectiveness of this equation is driving the modern AI. So the data they're buying isn't gobbledygook.
Yeah, I was going to say, I use a lot.
And you can also do stuff like you can take a whole bunch of, let's say, input questions like can you do this math? Can you recognize a dog? Right. And then you look at the error rates of different things, right? And then imagine there's a mathematical space where you could study that and say I need to shore up my mathematical space with the following things that Wolfram Alpha can easily produce. So you send this packet of requests to Wolfram Alpha and he goes, what the [__]? This is [__]. It's actually exactly what the model needs to reduce the error rate on that stuff.
Okay. Right. The same with images — like the internet is full of boobs. Right.
So the models are overtrained with boobs and image generators like to put boobs on everything, right? So but in the mathematical space, boobs have a certain, you know, attractive force and you can compensate that. So...
Yeah, it's not my fault. I'm literally just an observer here, right?
Okay. And then here's a theory, and I talked to a few computer scientists. Imagine there's a weight in here which seems to really be effective at connecting a bunch of things, right? So then think about human language, right? We don't have words for everything. We have words for the things that are pertinent to have words for. And an efficient language has words for a whole bunch of things that we use all the time. But if I want to describe the corner of your backyard where the grass has a clump of dried grass on it, there's no word for that. I just describe it. Yeah.
So then if you go through a modern language model trained on language, you can actually dump out the words, right? So you can train a language model with images and then dump out the names of all the things in the images, and then you can build a translator from the language model's definition of those words...
To the — no, to the English language words.
So you can naturally train this to give you a dictionary and then with, you know, with some kind of function make a translator to English, right? You ask, is there representational space? Yeah, because the simple models only ever get to describing cat and dog and chair and run and walk. But as you add more layers, you start to see association, affiliation, friendliness, training, you know, pedigree, right?
And as you do more of this, so imagine there's a computational space which as you go up, you know, you got objects and actions and associations and who the hell knows what we call — you know, what we said yesterday, like, you know, like the kind of research you're doing is a high-level, you know, metaphorical space. I'm taking the associations of math, physics, measurements, computation and putting them together, which is not — you know, like we have some simple equations like E=MC² but they don't mean anything to most people. For that to mean something, you have to have a higher associational space. And as you go in a higher associational space, there's actually less words. Like...
Yeah. So down here, all the objects in our life have names.
All the actions have names.
Right. But in the higher — people write whole books describing the political transformation of, you know, postwar Germany in relationship to the economics of the Turkish immigration wave. Like there's no word for that.
Actually, in German there might be a word.
Have you guys been using any category theory to try to...
I don't do this. I'm just reporting.
