Leaving Google, Founding Groq & Building the Future of AI | Jonathan Ross

So the most interesting part about it is the call where the idea was first floated was about 3 weeks before money was in the bank.

Of course, that's how you stay ahead.

So, we had been working on integrating GPU and LPUs together. The best way to describe why this helps is if you were building out a logistics network for the United States and I told you you could have either 18-wheelers or vans for last mile delivery, which one would you pick? And the answer is both, right? And so GPUs and LPUs combined ended up giving better performance across the performance curves. We had implemented it and we had gone to Jensen asking could we buy about a 100,000 GPUs because we were going to deploy them ourselves and Jensen saw what we had done and thought maybe it would be better to make this available to all of their customers.

When you're processing an LLM token, what's happening is you're doing all these different matrix multiplies. And some of them are more compute constrained and some of them are more memory throughput constrained. And the ones that are more compute constrained, we put on the GPU and the ones that are more memory throughput constrained, we put on the LPU. And the bottlenecks are all over the place. There's all sorts of different bottlenecks. There is no one strategy. There is no one perfect architecture. So the realization was you put these two things together and you defeat the bottlenecks across all of the different maps.

Yeah. I mean a human can wait a second or two to get a response when they type a command into a computer. AI is just sitting there waiting because it produces these tokens so much faster. Now you bring in LPUs and the speed is so much faster that it just becomes all about how do you move as fast as you can. AI is really good at using AI. That's really what Agentic is, right? So humans benefit from using AI, so does AI. Just like you would do research about me before I show up on your show, AI is going to kick off a job doing research on different tools it's going to use while it's using this other tool. And so kicks it off to another AI. And so you get this exponential growth.

Yeah, I think it's still early and one of the limiters is payments aren't really built for this yet, but if you can make micro payments, the number of payments is going to skyrocket. Like I did a little hobby project and for the hobby project I needed a couple of different phone numbers so I could have a bunch of different agents on with me on Signal and WhatsApp and I had to go to Twilio and I had to prove I was a human in order to get the number and all this stuff and it was just this big pain to get it done. If on the other hand I could have just done if I had allocated a budget to the AI and it could have just used that budget they would have spent it and I would never have known and it would have just been within the budget.

I like to do cutting edge stuff on my personal computer. So I don't have access to the work codebase like things where there's risk. I'll spin up a server in GCP or AWS and I'll just start building stuff. I built everything from apps that tell you what airplanes to take if you're traveling and what routes and which routes have the best seats and all this stuff all the way to apps that do certain mathematical things like a daily brief and all of this stuff. Very simple stuff. At work, I use it very extensively, but I always start with a hobby project before I bring it to work. What's the daily brief? Every morning, I get an email to me that says what's going on in the world based on what I'm interested in and how I interact with it and just a whole bunch of research. It's very much like the presidential daily brief except personalized for me.

Yes, you read it. It's text with links that I can click through to learn more. The big shift I did on the daily brief though was I started off by getting a whole bunch of text and I would read it and then I realized, oh, it's AI. It's done a whole bunch of research. It has the context. Why don't I have it just summarize everything? Just give me a bunch of headlines and I can ask follow-up questions.

Yeah. I mean, for me, the interactivity of it is important because it's sort of a you ever play the game 20 questions as a kid? Yeah. So, you can figure out within 20 questions what's in someone's mind. Questions allow you to sort of distill down into what you actually care about. If you have a podcast, it's a static form. You hear it, but if you can interact with it, then you can just get to the information you most want. So, as I'm learning through AI, I'm not reading a static piece of content. I'm interacting with it.

Yeah, and this also goes towards a shift of people moving from being ICs or individual contributors to all being leaders but leaders of AI. And what really good leaders do is they don't do the work themselves. They don't have the answer themselves. They're just asking the question. They're just considering everything they're hearing and then they ask the question that no one else asked or everyone is thinking but is afraid to ask. So with AI because it can go off and solve all of these problems for you. It can do the research report. The question that you ask determines what you get and that determines the output. From information age like we were all trained to just answer questions. Like that's what school's about. Remember this, remember that. With AI, you just ask AI. It knows. You just have to think of the right question. It's a fundamental shift.

So, I got this from a John Levy book, but it didn't detail it much. And the first principle of leadership is you have followers. Duh, right? It's that simple. You're not a leader unless you have followers. But when you think about leadership as having followers, it's also like thinking of investing as making money. There's a lot of ways to be an investor. You can be a venture investor. You can give debt where you can take equity. You can be seed stage. You can be series A. You can be growth. You can be crossover. You can do convertible notes. There's private equity. There's so many different ways to be an investor. Public market, right? Same with leadership. And so with leadership, the mistake I often see with new founders is they're like, 'How do I be a leader?' And the problem is they don't realize that there's an infinite number of ways to be a leader. And so they go off and they listen to someone and they get all this advice and they try and execute on it. It's not true to them. So one of the things that was very different for me as a leader, most of the people that I hear on your podcast, they are control freaks. They want things done their way.

On Well, both actually.

And so, had I tried to be an absolute control freak, that wouldn't have worked for me. I'm one of these weird people where I can go to a restaurant and I can tell the waiter, 'Just bring me whatever you think is best.' Or I don't even have a driver's license. I haven't had a driver's license since I was 18. And I've...

Because I don't feel the need to drive. I'd rather think. I don't need to control driving. I want to control thinking. I want to focus. I want to be on my phone. I want to be doing something useful. And that's been the case since I was 19 years old. And so, I'm happy to delegate things in a way that others are not. So, one of the things that's different about my leadership is when I hire people, I hire very autonomous people who often go off and execute on their own. And they would be terrible in most corporate environments. But also I can't hire the same kinds of people. And that was true to me. Had I gone the other way, I wouldn't have been very successful. If other people go this way, they're not going to be successful. Then again, there are a lot of things that are very similar, which is most of the best Silicon Valley leaders, they lead from a place of inspiring their people as opposed to trying to get people to be afraid. But that is a form of leadership. There are plenty of leaders out there who are successful because they cause people to become afraid. And so you just have to pick the form of leadership that works for you. But also, when you're picking where you're going to work early in your career, you should probably work somewhere where you're going to learn the lessons that are good for you. If you're more of a kind of person who can show appreciation and gratitude, you probably should not work for someone who inspires fear because you're not going to learn any lessons that you can use yourself.

It would probably be AI. So I think this is going to be a shortcut for founders in the future and we can get into this but the first thing that you have to do as a founder is you have to go from the technical thing that you know how to do and that you can add value with to learning how to manage people and for me that probably cost Grok 3 to four years.

I was a terrible leader. I was one of the world's worst leader when I started. I had a lot of I gave people a little too much latitude because I'm more of a delegator, but I entrusted people who probably shouldn't have been entrusted with that level of autonomy. I didn't hire people who could operate autonomously, but I was naturally someone who would delegate and give autonomy. So, what ended up happening was things would just grind to a halt because they wouldn't know what to do and I wasn't telling them what to do and they were used to being told what to do. And then finally I would get so frustrated I would go in there and tell them what to do but it was so unnatural to me that they didn't accept it.

I work through questions. I like to set high-level direction. So for me my leadership style was come up with a goal that was so simple that I could put it on a challenge coin and give it to everyone. So, everyone at Grok had a challenge coin that said 25 million tokens per second and had a little graph of it going up and everyone knew that was the thing to do. And it's sort of like when you ask an AI agent to do something. The fewer constraints you give it, the more freedom it has to solve your problem. And so I liked to work with incredibly creative people who would come back with surprises. Say that part again. This is important. The fewer constraints that you give someone the more freedom they have to solve the problem and the more freedom they have to surprise you with the solution. So if you want to run a highly creative and innovative organization then what you really want to do is minimize the number of constraints but you also have to give them the things that matter. If you aren't able to very crisply distill what you're trying to accomplish, then either you're going to over constrain or under constrain the people. When you're trying to do something as a founder, you are inherently trying to disrupt an old industry. There's a moat. You're trying to do something differently. If you're not doing something differently, what's the point? They're already well-established, well-funded companies. If you can bring a team together that is also disrupting and innovative and it's not just you, then it goes from being Superman to being the Avengers. And so that was just my natural leadership instinct. Had I been more of a command and control person, I should have doubled down on command and control. You have to just do what's natural.

Not too much.

If you don't give people a clear enough but under constrained enough objective that they can't surprise you in a good way with the result then you are not giving them the ability to innovate. I want to double down on that for a second. The only way for your team to innovate, right, without you being the innovator is they must be able to surprise you in a good way, which means you must not over constrain the goal.

That in basic management dealing with people like little things like one of the great things that I've learned at NVIDIA. Actually, let's take a lesson from Nvidia because now I'm there. Jensen is world class. And one of the things that I observed because I've worked at other tech companies. There's no politics. It's the least political large organization you will ever see. And I learned this lesson at Grok, but I didn't take it to the extreme. So seeing it in the extreme shows me just how valuable it is. There is no circumstance at NVIDIA where Jensen has one-on-ones with people and tells them one thing. I learned this at Grok because what would happen is I would have a conversation with one person and then I would have a conversation with another person and then what would happen is both of them heard very different things and would talk to each other and come to very different conclusions. But when I had a room full of people and I said something to them, it's amazing how they all heard the same thing. And so when you are leading groups of people if you want to reduce the amount of politics and people going off and forming side cliques and all this, stop having one-on-ones. Have big meetings with everyone who you want to tell something and tell them all at once. And don't allow anyone to send you an email. Copy everyone on the email. Just like the moment someone sends you something. If someone says, 'Hey, this person's screwing up on this thing.' Copy that person on the email. Let them jump in. Otherwise, you were allowing politics to happen.

I got way too cute on trying to play 3D chess. Whereas Jensen is very much just like what does the customer need? I want to develop trust with the customer. I want to always tell the customer things that are true and I believe and I can support. And if I have a thing that isn't what the customer wants, I'm not going to sell it to them. I'm going to sell things to customers that they actually need and that I believe they need. I'm not going to think like, how do I build moats? How do I do any of this? It's just like, what does the customer need? Just build that for them and everything else follows.

Yeah. And but I think this is universal. I think any good founder is able to do this. And again, going back to those early stages of going from a non-founder to a founder, you're going to end up hiring people and they're going to be like, 'No, no, I'm the expert in this area.' Like, just trust me. You're a kid, just trust me. And you have to learn confidence. So, one of the things that happened for me that was very helpful early on in being a founder was I actually got to back when we were maybe 35 people. I got to shadow someone who was running an organization of 2,000 people. And it was funny because like there's no NDA in place or anything, but every meeting we went into, he's like, 'Oh, this guy's got an NDA. Don't worry, you can say anything in front of me.' And I was just like, 'Okay.' But we would go into every meeting and I was sitting there silent and I would think, what would I do? And at the end of it, each time when he would make his decision, it's exactly what I would have done. What I hadn't realized before that moment was I didn't have the confidence. Part of the problem with being a leader was I needed to have the confidence in a decision so that other people would have the confidence to execute. Many people have too much confidence. Some people have too little confidence. And the question is, if you have too little confidence, you're probably the kind of person who thinks through things a lot more, but you need to still get to a point where you act with confidence. And when I realized I was making the same decisions as this very experienced founder or CEO, I started acting with confidence and people started following my direction much more. I didn't change my decisions, but it changed my leadership.

About 450, but it was more difficult to manage than a typical group of 450. So, in the military, depending on your rank, you are allowed to have a certain number of people reporting into you. And the higher your rank, the more you can have as an officer. But when you have scientists reporting to you, the number is actually much smaller, like dramatically smaller. Because I had such a creative organization, it was much harder to manage them. The problems manifested. And so it probably was more like managing a group of 5,000 people than it was 450 in many ways. In other ways, it was like managing an even smaller group because innovations would just happen on their own. But the better the people, the harder they are to manage.

So early on at Grok and this is a lesson for founders because if you're doing a capital intensive business you're going to need to raise money and one of the things that we went through was we had raised money from some VCs who fell out of favor with other VCs and others didn't want to co-invest and so every time we would try to raise we just struggled and there's also a little bit of volatility in how...

Well, the way I like to put it is typical typical not all, but typical West Coast VCs are more like lemmings. And typical East Coast VCs all think that they're smarter than each other. So, when you try and raise from the West Coast, if one VC puts money in, all the others want to put money in. In New York, one VC investing means nothing. They're going to run their own analysis. They really do not care what other VCs are doing. The flip of that is if you're on the west coast and one VC passes, they're going to go tell every other VC and like lemmings, they're all going to pass as well. So, we had this problem where all the VCs on the West Coast didn't want to invest in us. We had very few of the typical VCs invested in us at the end. We had like a bunch of crossover funds from the East Coast invested.

No, I don't think so.

Yeah. So going back to what you asked about almost running out of money. We were about 3 weeks from running out of money at one point.

That was many years ago. And...

No, no, no, no, no. We were fine towards that.

It was really only a little over 2x. So that wasn't a huge jump. And we also had the ability to raise at the amount that we did the licensing for.

Well, it was that 3-week period from the time when we presented and asked to buy GPUs until not only the deal was done, but money had been wired. But how many months before were you working on this? It was probably three or four, maybe a little bit longer. But what had happened was I didn't initially think that it was going to be that big of a deal. So I didn't suggest doing it. In fact, it was Sunny, my COO. This goes back to the whole autonomy thing.

Yeah. So he had the idea of trying to put our chips together and he didn't have like a...

So the LPU and GPU as mentioned they're better at different parts of what's called the decoder layer of an LLM. The GPU is better at the attention portion and the LPU is better at sort of applying the weights which is the thing that gets trained as opposed to the memory. And what we realized was and this is what most people get wrong when they're trying to do this themselves. They'll take the reading of tokens or what's called prefill and they'll do that on one piece of hardware and then they'll put the generation of tokens on another piece of hardware. But the generation of tokens is the hard part. That's the thinking. Reading is easier than writing, right? And that's true for AI as well. And so what we figured out was if and again this was all a bunch of people. It wasn't any one person and it was a group that all innovated. Once he brought the idea of why don't we put them together because they're different bottleneck then the team figured out oh this part goes here this part goes there. We implemented it and it worked. And so we also weren't afraid to show it to Nvidia because we wanted to become a customer of Nvidia. We wanted to buy GPUs. And so what ended up happening was we went we presented it made a lot of sense and the deal happened.

Yeah.

This is the nature of a successful entrepreneur. You move quickly. You don't wait. There's opportunity cost to waiting. Technology is not a business where you can wait a year.

Right now, when you go to use AI, it's a little it may feel somewhat fast, but that's because you're not used to using it much faster. Sort of like when you first use the internet, it felt fast compared to mailing things around. But when you got broadband, you realized, oh my gosh, like this is so much better. I'm never going to go back. The difference is broadband actually needed people to make their websites faster to make it usable. If the servers are slow, you don't get a benefit. So, it took a while to roll it out and make it good and get video that could stream and all that. The difference is you put these LPUs into a system and all of a sudden the generation of tokens gets faster. It's like getting broadband instantly on these existing models. And so now rather than having to wait a minute to get an answer you can get an answer in 10 seconds and that really starts to compound. So let me walk through an example of why it's not just speed but it's also quality.

The other thing that I did was I created Google TPU and at Google there was this time where I'd already moved over to Google X. I was no longer working on the TPU at that point. had already done it and someone else from the TPU team was at Google X came by, showed me this email from Deep Mind saying, 'Hey, we've got this competition. We think we're going to lose. There's a prize purse. Is your chip as fast as we've heard?' And we're like, 'Yes.' Like, it's like Ghostbusters, when someone asks you, 'Are you a god?' You say, 'Yes.' Someone asks you, 'Is your chip as fast as I've heard?' You're like, 'Yes.' So we reply back yes and they're like great competition's in 30 days. We're going to play the world champion and go and we played our test games and we lost. We need to win. So they had no choice but to port over to that TPU chip. So we did it and a bunch of interesting things happened. I think Do you know what an ELO score is? Yeah. Okay. So for those who don't know what an ELO score is, it's sort of your ranking in chess or go and something like a 200 point advantage is insurmountable. Like the probability of you winning is basically zero. Alpha Go running on GPUs had an ELO score of about 3,200. Lee Sedol was about 3,550. And I might be getting the first digit wrong. It might be 2,000 instead of 3,000, but it was more than 200. And then when put on LPUs, it actually went to like 3,900 or something ridiculous or 2,900, whatever the first digit was. It jumped dramatically. And so I think he didn't expect he was going to lose, but he lost badly, but it was the exact same model. So what changed was the ability to compute more made the results smarter. The way that these models work, thinking fast, thinking slow from Daniel Kahneman. Yeah, I read that book. What AI does is if I have 270 possible moves, which is what you have on a go board, the AI is going to rank those moves and say this is the best move, this is the next best, and so on. What happens is you virtually play that best move, and then you virtually play the counter move, and then you virtually play the next one. and then you see how the game unfolds. What you can also do is try that second best move. And occasionally that second best move when you play it out actually turns out to be the right move in this context. It's not the one that you would normally do, but in the context it's better. And you can see that as you play it out. In the second game, there was this famous move called move 37, which was creative. It was original. It actually wasn't completely original. It was a 1 in 10,000 game move. It had been in the cannon of games that we're trained on, but when we went back and played it on GPUs, it never found that move because it was too deep in the chain. So over time, these GPUs have gotten as good and better than TPUs. As the creator of the TPU, I have to admit GPUs are now better. This is the benefit of having an entire industry behind you and ecosystem and everything. But at the time TPUs had some novel innovations. Right? Now bring in the LPU and you can go deeper faster. You can search faster. And so you can actually make a model smarter by making it faster. And so the realization is and now that you've got this ability to reflect, think deeply and change the outcome based on your thinking, being able to think faster makes you think smarter. And so that's the advantage of pairing the LPU and the GPU.

So, one of the things that we hired for at Grok was reality quotient, which is different from intelligence quotient. And the way to think of it is there are plenty of really smart people who wouldn't recognize reality if it tapped them on the shoulder. [laughter] You got to say more about that. Yeah. So, like you just know these people who will construct these very elaborate stories in their mind that are completely disconnected from reality, right? And then there are some people who are just incredibly street smart but couldn't do basic arithmetic or anything like that. Reality...