And again, maybe you would go back and say, I wish that had never happened. That's fair enough. But it has happened. What I think is better and healthier is to appreciate that all the things that bother you are ultimately human issues that have been exacerbated or accelerated by technology. By anthropomorphizing technology as good or bad and saying we need to make it good or bad, you're assuming way too much power that you don't have and you're actually skirting closer to even more disastrous outcomes than what may or may not have happened.
Yes. And OpenAI should incorporate ads into the ChatGPT product at some point in 2026.
Yeah, it's a big problem. Yes, I look forward to chronicling that particular adventure as the year unfolds. For now, we will move to Concrete News. Exiting the dorm room here from Bloomberg on Christmas Eve, Nvidia agreed to a licensing deal with artificial intelligence startup Grok, furthering its investments in companies connected to the AI boom and gaining the right to add a new type of technology to its products. The world's largest publicly traded company has paid for the right to use Grok's technology and will integrate its chip design into future products. Some of the startup's executives are leaving to join Nvidia to help with that effort. The company said Grok will continue as an independent company with a new chief executive. It said Wednesday in a post on its website.
So, Ben, I just want to say dropping this news at 4:00 Eastern on Christmas Eve is one of the greatest news dumps of my entire life. I still can't believe it really happened.
It's really a... What is the meme about my t-shirt? It's raising questions that are...
No kidding. And the timing of me putting on that t-shirt. TikTok did that a couple years ago where it was like two days before Christmas and they announced, 'Oh yeah, our internal investigation. We did find out that we were actually spying on people.' But Nvidia beat TikTok here. They went on the afternoon of Christmas Eve. Lot to talk about here. We don't have that much time, but can you walk me through what Nvidia is acquiring with Grok and why Grok? We could cut the whole dorm room segment. I was just going to be curiously backpedaling for the rest of this episode.
No, it was a wonderful. I enjoyed the dorm room segment. We'll be back to the dorm room at some point in the next couple weeks. But Grok and Nvidia, why does this make sense?
We're slowly developing a nomenclature. We have the mullet section. I guess we have a dorm room section. Occasional visits to the dorm room. And of course, we have TikTok, our parenting segment that we revisit too infrequently. But talk to me about Nvidia here.
Okay. What is Grok? Grok has a very different approach. Grok was actually started before the LLM AI explosion. It was focused more on machine learning tasks generally, which a key thing is they're not as memory intensive as LLMs are. And the idea with Grok is it's sort of like software-defined inference in a way where the first thing the Grok team built was not a chip, they built a compiler.
And a compiler is where you take code and you change it into the ones and zeros. It's actually machine code, but basically the ones and zeros that actually run on the chip. And the idea was that if you have a super deterministic algorithm that's very simple, which these sorts of algorithms are, that's why they work better on GPUs than on CPUs. We've talked about this. GPUs are simpler calculation machines. They just do a bunch of calculations at the same time as opposed to CPUs that are much more complex. They have to handle if-then statements, branching, where stuff is. It's a great irony that deterministic computing rests on probabilistic chips, which I would put CPUs in, particularly with things like branch prediction and all these bits and pieces, whereas probabilistic AI rests on very deterministic algorithms. It's a weird interesting paradox. Grok takes that to the extreme where the idea is if you know exactly what the calculation that you're running is, there's no ifs, ands, or buts. You're just running a very straightforward calculation. The way to make that run the absolute fastest is to define where every single thing is. So, if you're pulling into a gas station, this game dithering, right? You have the little arrow that points right or left on your gas tank. Apparently the guy who invented that just died. So John...
It's a wonderful invention and discovering it like 10 years ago was such a big personal win for me. So thank you to that man. Rest in peace.
We should look what it is. Yeah, he just died. There's articles about him. But like, so you pull up to a gas station, you have to figure out, okay, where's the pump? Which side am I on? What's available? Sort of XYZ. It's going to take you a little bit of time to pull up to the pump and then you have to get the gas thing and you have to slide it over and put it in, blah blah blah. Contrast that to a Formula 1 car pulling into a pit stop. It's super defined. There's lines on the ground. There's people in place. You know exactly where you're going. And that's how you can get in and change tires and be out in two seconds. They used to add gas, but even then it would still be in the seconds because they pull right up. Gas right there. Go in. Boom. Out.
Mhm. If you can predefine everything, it's going to be way faster than dealing with uncertainty in the process of trying to execute something. Okay. So even GPUs, which are very focused, are still somewhat probabilistic in nature and indeterminant in their calculations. So the big thing is memory. They run on DRAM. Now there's a variation of DRAM called high bandwidth memory, HBM. But yeah, it's still the general concept. This is memory that has to be powered. You're having to constantly supply a current to the gates to keep them where they are. And it's a little indeterminant exactly the state of what it's in at any one time. So you go to get information, you go through these abstractions of where it is, the exact location. You have to go and get it. You might have to wait a second for it to be refreshed, then you grab it, then you bring it back. It's still incredibly fast. You can't really grok, no pun intended, how fast it is, but it's not as fast as it could be if you're operating in a Formula 1 context where you know exactly where the bit of data is. You go there, it's guaranteed to be there. It doesn't need to be refreshed because it's like a physical gate that is set. It doesn't need electrical current running through it. Okay. Grok uses what's called SRAM, which is RAM that's actually on the die itself. So you have the chip with all the logic gates on there. You're not going off to a piece of memory. You're actually on the same chip. It's defined. It's either on or off. It doesn't need a continual current to be refreshing it. It's kind of degrading, then it goes back up, blah blah blah. No, you know exactly where every bit of data is. You go there and get it. And what this means, if you're running a super well-defined algorithm that is not branching, there's no if-then, you're just calculating. If you can go and grab stuff and know exactly where it is, you can calculate much faster than if you're inserting all these tiny bits of uncertainty throughout the process.
So, what this means is Grok is super fast. So like when it came out, it was mind-blowing. Two years ago they did a live segment on CNN that I embedded in the article when I first wrote about it, of someone talking to Grok as a real conversation. Now again, it speaks to technological progress that we kind of get that now with the regular ones. But all that speaks to is that the potential is even faster, even more. So yeah, there are applications. Real-time speech, for example. Even in that demo you could see it was a little slow, but you can imagine the speed that's happened with regular AI models being applied to that model and getting faster. So you and I having a conversation here, right? There's a very fast back and forth. When we're in person, that back and forth is actually even faster. We have some latency here, right? So we sometimes speak over each other, go back and forth, which we don't get when we're in person because in person you're talking and I'm waving my arm. It's just very apparent. I want to talk right now to give you space to talk instead of me talking to have a meltdown if I don't give him some space. Well, no. I mean, that's the perfect example. If you're dealing with a chatbot customer service agent, you want the fastest answers you can possibly get.
Well, here's another example. What is one of the AI things I'm super optimistic about? Because I'm a weirdo who thinks ads are good. Personalized ads, right? Like you have stuff tuned to what you want. So the whole ad ecosystem is actually pretty incredible, right? When you submit a web page, it goes out. It says here's who I think this person is or this sort of broad bucket of identifiers that I have. It runs an auction, then goes and gets an ad and puts it in and then serves you in a matter of 3 seconds. To your point on us not properly appreciating how mind-blowing the progress of the internet is independent of AI, that's a great example of what we're dealing with here in 2025. Imagine you want that ad to be perfectly personalized to you. Now again, I get how this sounds sort of terrifying and bad. So let's just set that aside for a moment. But if you want real-time generative AI so that it's not just that generative AI lets you A/B test and find a thousand interesting pieces of art instead of just the 10 that you generate on your own, what if it's all generated in real time immediately for you specifically? Every time you... Once again, stipulate that Ben is a weirdo here, but this might be possible in the years to come.
Like basically massive speed in inference. There's going to be a market for that. That's a market that Grok was going for, and it's a market that Nvidia with its architecture was always going to be slower at. Now there's a huge tradeoff. You can't fit very much memory on a die. Grok chips had like 256 megabytes, I megabytes not gigabytes. I think you had to link a gazillion of them together to even run a basic model, much less a high-end model. Things like context window take up a huge amount of memory. The way these things work is they generate a token, and then they run the exact same process to generate the next token. It's hilariously inefficient. It's like running these calculations again and again just token by token by token. So you can see why something like Grok, even those little bits of being faster, manifest as being hugely faster. But the point is to generate a cohesive answer, you need to be storing every token that's already been generated, and to do that takes a huge amount of memory. So that's why you still have Nvidia. If you want these thinking models that actually go through and evaluate different things and all these bits and pieces, all that's doing is blowing up the context window and these memory requirements. So actually one of Nvidia's announcements with Vera Rubin this week is this new part of a full-scale Vera Rubin server. Within that whole multi-rack unit are SSDs, actual solid-state hard drives that are dedicated to storing this KV cache. So you have these very large context windows. And you think about it from an agent perspective, if you have an AI that goes off and does work, it's actually okay that the fetching is a little slow because you're not part of the loop. The AI is just off doing work, right? And so having larger context windows, being able to maintain more context and stay grounded, is more important than super speed. The point being, and also the higher value the answer, the more willing you are to tolerate some latency in terms of what's being served, right? But at the end of the day, if I'm calling a customer service agent, if I have to wait 15 seconds for an answer, I'm making up, right? So the point is the market is going to fragment. It's not going to be just one way, one architecture, whatever it might be. So what Nvidia just acquired or made a deal for, again I keep saying acquired. They didn't acquire anything.
Get it right. TechMeme had to delete a tweet on Christmas Eve because they characterized it as an acquisition. This is a licensing deal. What did they think? 90%? I think like this was an early article. I think it was actually like 90% of the employees, including the actual architects of this, like Jonathan Ross in particular who invented the first TPU, was the CEO of Grok. And they now have something in their portfolio to do this. They licensed a product that will serve one of the fragments in the market of the future here. Now, Nvidia can make this way more valuable because this is going to be a tricky thing: what problems go where? What should be served by a Grok-type model? What should be served by the SSD hard drive, very complicated east-west maximizing traffic models? Wouldn't it be nice to abstract that? Wouldn't it be nice to have a software layer where you can easily say what you're trying to do and it figures out the right way?
A one-size-fits-all solution.
