One creation dares to redefine the future. From the minds at xAI, prepare for Grok 4. This summer, the next generation arrives. Faster, smarter, answers the unasked, and challenges the impossible. Grok 4, unleash the truth, coming this summer.
Welcome to the Grok 4 release here. This is the smartest AI in the world. We're going to show you exactly how and why. And it really is remarkable to see the advancement of artificial intelligence, how quickly it is evolving. I sometimes think, compare it to the growth of a human and how fast a human learns and gains conscious awareness and understanding. And AI is advancing just vastly faster than any human.
I mean, we're going to take you through a bunch of benchmarks that Grok 4 is able to achieve incredible numbers on. But it's actually worth noting that Grok 4, if given like the SAT, would get perfect SATs every time, even if it's never seen the questions before. And if even going beyond that to say like graduate student experience, you know, you're going to get a lot of great results. And I think that's really important. If you take any of these exams, like the GRE, it will get near perfect results in every discipline of education. So from the humanities to like languages, math, physics, engineering, pick anything. And we're talking about questions that it's never seen before. These are not on the internet. And Grok 4 is smarter than almost all graduate students in all disciplines simultaneously. Like, it's actually just important to appreciate the, like, that's really something. And the reasoning capabilities of Grok are incredible. So there's some people out there who think AI can't reason. And look, it can reason at superhuman levels. So, yeah. And frankly, it only gets better from here.
So we'll take you through the Grok 4 release and, yeah, show you, like, the pace of progress here. Like, I guess the first part is, like, in terms of the training, we're going from Grok 2 to Grok 3 to Grok 4. We've essentially increased the training by an order of magnitude in each case. So it's 100 times more training than Grok 2. And that's only going to increase. So it's, yeah, frankly, I mean, I don't know. In some ways, a little terrifying. But the growth of intelligence here is remarkable.
Yes. It's important to realize there are two types of training compute. One is the pre-training compute. That's from Grok 2 to Grok 3. But from Grok 3 to Grok 4, we're actually putting a lot of compute in reasoning in RL. Yeah. And just like you said, this is literally the fastest-moving field. And Grok 2 is, like, the high school student by today's standards. If you look back in the last 12 months, Grok 2 was only a concept. We didn't even have Grok 2 12 months ago. And then by training Grok 2, that was the first time we scaled up, like, the pre-training. We realized that if you actually do the data ablation really carefully and the infra and also the algorithm, we can actually push the pre-training quite a lot. By a model of 10x to make the model the best pre-trained-based model. And that's why we built Colossus, the world's supercomputer with 100,000 H100. And then with the best pre-trained model, and we realized if you can collect these verifiable outcome reward, you can actually train this model to start thinking from the first principle, start to reason, correct its own mistakes. And that's where the Grok 3 reasoning comes from. And today, we ask the question, what happens if you take the expansion of Colossus with all 200,000 GPUs, put all these into RL, 10x more compute than any of the models out there on reinforcement learning, unprecedented scale, what's going to happen? So this is the story of Grok 4. And Tony, share some insight with the audience.
Yeah. So yeah, let's just talk about how smart Grok 4 is. So I guess we can start discussing this benchmark called humanity's last exam. And this benchmark is a very, very challenging benchmark. Every single problem is curated by subject matter experts. It's in total 2,500 problems. And it consists of many different subjects, mathematics, natural sciences, engineering, and also all the humanity subjects. So essentially, when it was first released, actually like earlier this year, most of the experiments were done with a single-digit accuracy on this benchmark. Yeah. So we can look at some of those examples. So there is this mathematical problem, which is about natural transformations in category theory. And there's this organic chemistry problem that talks about electrocyclic reactions. And also, there's this linguistic problem that tries to ask you about distinguishing between closed and open syllabus from a single-digit algebraic source text. So you can see also it's a very wide range of problems. And every single problem is PhD or even advanced research-level problems.
Yeah. I mean, these, there are no humans that can actually answer these, can get a good score. I mean, if you actually say like any given human, what, like what's the best that any human could score? I mean, I'd say maybe 5% optimistically.
Yeah. So this is much harder than what any human can do. It's incredibly difficult. And you can see from the types of questions. Like you might be incredible in linguistics or mathematics or chemistry or physics or any one of a number of subjects, but you're not going to be at a post-grad level in everything. And Grok 4 is a post-grad level in everything. Like it's just, some of these things are just worth repeating. Like Grok 4 is post-grad level. Yeah. It's a post-graduate, like PhD level in everything. Better than, like most PhDs would fail.
So it's better than, I mean, at least with respect to academic questions, I want to just emphasize this point. With respect to academic questions, Grok 4 is better than PhD level in every subject, no exceptions. Now, this doesn't mean that it's, you know, at times it may lack common sense and it has not yet invented new technologies. It's not just a matter of time. I think it may discover new technologies as soon as later this year. And I would be shocked if it has not done so next year. So, I would expect Grok to, yeah, literally discover new technologies that are actually useful no later than next year and maybe end of this year. And it might discover new physics next year. And within years, I'd say almost certainly. Like, so just let that sink in. Yeah.
So yeah. How, okay. So I guess we can talk about the, the, what's behind the scene of Grok 4. As Elon mentioned, uh, we actually throwing a lot of compute into this training. You know, when it started, it's only also a single digit, sorry. Um, the previous slide, yeah, it's only as a single digit, uh, number, but as you start putting a more and more training compute, it started to gradually become smarter and smarter and eventually solved a quarter of the HLE problems. And this is without any tools. The next thing we did was to adding a tools capabilities to the model. And unlike Grok 3, I think, uh, Grok 4 actually is able to use tools as well, but here we actually make it more native in a sense that we put the tools into training. Uh, Grok 3 was only relying on generalization here. We actually put the tools into training and it turns out this significantly improves the model's capability of using those tools.
Yeah. I remember we had like a deep search back in the days. Yeah. So how is this different?
Yeah. Yeah, exactly. So deep search was exactly the Grok 3 reasoning model, uh, but without any specific training, we only asked it to use those tools. So compared to this, it was much weaker in terms of its tool capabilities and unreliable.
Unreliable. Yes, yes. And to be clear, like these are still, I'd say fairly, this is still fairly primitive tool use. If you compare it to say the tools that are used at Tesla or SpaceX, uh, where you're using, um, you know, uh, finite element analysis and computational flow dynamics and, and you're, you're able to run, uh, or say like Tesla does like crash simulations where the simulations are so close to reality that if the test doesn't match the simulation, you assume that the test article is wrong. That's how good the simulations are. So Grok is not currently using, uh, any of the tools that the really powerful tools that a company would use, but, but that is something that we will provide it with later this year. So we'll have the tools that, that a company has, um, and, and have very accurate physics simulator. So, that will be the first thing that will come out of this.
Alright. Thank you very much. Um, ultimately the, the thing that'll make the biggest difference is being able to interact with the real world via humanoid robots. So it combined, sort of, Grok with, with Optimus and it can actually interact with the real world and figure out if, if its hypo, if it has, if its, uh, it can formulate an hypothesis and then confirm if that hypothesis is, is true or not. Um, so we're really, you know, I think about like where we are today. We're, we're at the beginning of an immense future. Yeah. Uh-huh. Immense intelligence explosion. We're in the intelligence Big Bang right now. And we're at the most interesting time to be alive of any time in history. Yeah. Now, that said, we need to make sure that the AI is a good AI, a good Grok. And the thing that I think is most important for AI safety, at least my biological neural net tells me, the most important thing for AI is to be maximally truth-seeking. So this is a very fundamental. You can think of AI as this super genius child that ultimately will outsmart you, but you can instill the right values and encourage it to be sort of truthful, honorable, good things. The values you want to instill in a child that will ultimately grow up to be incredibly powerful.
Yeah. So, yeah. So this is really, I'd say, we say tools. These are still primitive tools, not the kind of tools that serious commercial companies use. But we will provide it with those tools, and I think we'll be able to solve, with those tools, real-world, technology problems. In fact, I'm certain of it. It's just a question of how long it takes.
Yes. Yes, exactly. So is it just compute all you need, Tony?
Is it just compute all you need at this point?
Well, you need compute plus the right tools, and then ultimately to be able to interact with the physical world. Yes. And then, I mean, we'll effectively have an economy that is, you know, we're going to be able to do a lot of things.
Well, ultimately an economy that is thousands of times bigger than our current economy, or maybe millions of times. I mean, if you think of civilization as a percentage completion of the Kardashev scale, where Kardashev 1 is using all the energy output of a planet, and Kardashev 2 is using all the energy output of a sun, and 3 is all the energy output of a galaxy, we're only, in my opinion, probably, like, close, closer to 1% of Kardashev-1 than we are to 10%. So, like maybe a pointer, 1, 1 or 2% of Kardashev-1. So, we will get to most of the way, like 80, 90% of Kardashev-1 and then hopefully, if civilization doesn't self-annihilate. And then Kardashev-2, like it's, the actual notion of a human economy, assuming civilization continues to progress will seem very quaint in retrospect. It will seem like sort of cavemen throwing sticks into a fire level of economy compared to what the future will hold. I mean, it's very exciting. I mean, I've been at times kind of worried about like, well, you know, is this, this seems like, it's somewhat unnerving to have intelligence created that is far greater than our own. And will this be bad or good for humanity? It's like, I think it'll be good. Most likely it'll be good. Yeah. Yeah. But I somewhat reconciled myself to the fact that even if I, even if it wasn't going to be good, I'd at least like to be alive to see it happen.
Yeah. So yeah. So actually, one, one, yeah, I don't want a technical problem that we still need to solve, besides just compute is how do we unblock the data on data bottleneck? Uh, because, um, when we try to scale up the RL in this case, we did invent a lot of new techniques, innovations to allow us to figure out how to find a lot of, a lot of different, uh, new ways to execute. And they're basically, like, nor summarize the studies to an alphabet like快 20, but we basically Holzman things like that. So in other words, we think that today, especially in the era of the computer scientists and recorders as well, you know, we've kind of we've slightly reallyühred to the theme of like, yeah, I won't, you know, sorry, I'm sorry. So whatever, you know, really hear from this person and your state of like this. challenging RL problems to work on. It's not just the problem itself needs to be challenging, but also, it needs to be, you also need to have a reliable signal to tell the model, you did it wrong, you did it right. This is sort of the principle of reinforcement learning. And as the models get smarter and smarter, the number of cool problems or challenging problems will be less and less. So it's going to be a new type of challenge that we need to surpass, besides just compute.
Yeah. We're actually running out of actual test questions to ask. So there's even ridiculously, questions that are ridiculously hard, if not essentially impossible for humans, that are written down questions, are becoming, swiftly becoming trivial for AI. So then there's... But you know what? The one thing that is an excellent judge of things is reality. So because if physics is the law, ultimately, everything else is a recommendation. You can't break physics. So the ultimate test, I think, for whether an AI is... The ultimate reasoning test is reality. So you invent a new technology, like say, improve the design of a car or a rocket, or create a new medication that... And does it work? Yeah. Does the rocket get to orbit? Does the car drive? Does the medicine work? Whatever the case may be. Reality is the ultimate judge here. So it's going to be a reinforcement learning, closing loop around reality.
We asked the question, how do we even go further? So actually, we are thinking about now, with single agent, we're able to solve 40% of the problem. What if we have multiple agents running at the same time? Yeah. So this is what's called test and compute. And as we scale up the test and compute, actually, we are able to solve almost more than 50% of the text-only subset of the HLE problems. So it's a remarkable achievement, I think.
Yeah. This is insanely difficult. These are... So what we're saying is like a majority of the text-based of humanity's scarily named humanity's last exam. I don't know if it's gonna be a majority vote or one in a hundred. I think it's gonna be a majority vote, but we're looking at probably like five. So we're not gonna be like one of those. Yeah.
No. But I think it's a good example of how it can be done. We should be able to do it. I mean, it's fine to do it and it's cool to do it. Right. And it's like a... Because if we can't do it with a lots of people. Right. Can we do it in the open? Yeah, we can do it in the open. Or I think that's a good example of how to test and compute?
Yeah. I think that's a good example of how to do it in the open. We can test it. We can test it. Yeah. Yeah.
But once they share the trick or figure out what the real nature of the problem is, they share that solution with the other agents. And then they essentially compare notes and then yield an answer. So that's the heavy part of Grok 4, is where you scale up the test time compute by roughly an order of magnitude, have multiple agents tackle the task, and then they compare their work, and they put forward what they think is the best result.
Yeah. So we will introduce Grok 4 and Grok 4 Heavy. Sorry, can you click the next slide? Yeah. Yes. So basically, Grok 4 is the single agent version, and Grok 4 Heavy is the multi-agent version. So let's take a look how they actually do on those exam problems and also some real life problems.
Yeah. So we're going to start out here, and we're actually going to look at one of those HLE problems. This is actually one of the easier math ones. I don't really understand it very well. I'm not that smart. But I can launch this job here, and we can actually see how it's going to go through and start to think about this problem. While we're doing that, I also want to show a little bit more about what this model can do and launch a Grok 4 Heavy as well. So everyone knows Polymarket. It's extremely interesting. It's the seeker of truth. It can do a lot of things with what reality is most of the time. And with Grok, what we're actually looking at is being able to see how we can try to take these markets and see if we can predict the future as well. So as we're letting this run, we'll see how Grok 4 Heavy goes about predicting the World Series odds for the current teams and MLB. And while we're waiting for these to process, we're going to pass it over to Eric, and he's going to show you an example of his.
Yeah. So I guess one of the coolest things about Grok 4 is its ability to understand the world and to solve hard problems by leveraging tools, like Tony discussed. And I think one kind of cool example of this, we asked it to generate a visualization of two black holes colliding. And of course, there are some liberties. It's, in my case, actually pretty clear in its thinking trace about what that means. Yeah. And I think one of the cool things about Grok 4 is that it allows you to see what these liberties are. For example, in order for it to actually be visible, you need to really exaggerate the scale of the waves. And yeah, so here's this kind of inaction. It exaggerates the scale in multiple ways. It drops off a bit less in terms of amplitude over distance. But yeah. So you can kind of see the basic effects that are actually correct. It starts with the inspiral. It merges. And then you have the ring down. And this is basically largely correct. Yeah. Modulo some of the simplifications that it needs to do. It's actually quite explicit about this. It uses post-Newtonian approximations instead of actually, like, you know, like, you know, like, you know. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah.
So it's like computing the general relativistic effects of, like, near the center of the black hole, which is, you know, incorrect and will lead to some incorrect results. But the overall visualization is, yeah, is basically there.
And you can actually look at the kinds of resources that it references. So here, it actually, you know, it obviously uses a search. It gathers results from a bunch of links, but also reads through a understanding of undergraduate text in analytical, analytic gravitational wave models. It's, yeah, it reasons quite a bit about the actual constants that it should use for a realistic simulation. It references, I guess, existing real world data. And yeah, it yeah, it's a, it's a pretty good model. Yeah,
but like, actually, going forward, we can, we can, we can plug, we can give it the same model that physicists use. So it can run the same level of compute that so leading physics researchers are using, and give you a physics accurate black hole simulation.
Exactly. Just right now is running in your browser. So yeah, this is just running in your browser. Yeah, exactly. Pretty simple.
So swapping back real quick here, we can actually take a look, the math problem is finished. The model was able to, let's look at its thinking trace here. So you can see how it went through the problem. I'll be honest with you guys, I really don't quite fully understand the math. But what I do know is that I looked at the answer ahead of time. And it did come to the correct answer here in the final part here. We can also come in and actually take a look here at our, our World Series prediction is still thinking through on this one. But we can actually try some other stuff as well. So we can actually like, try some of the X integrations that we did. So we worked very heavily on working with all of our X tools and building out a really great X experience. So we can actually ask, you know, the model, you know, find me the xAI employee that has the weirdest profile photo. So that's going to go off and start that and then we can actually try out, you know, let's create a timeline based on X post, detailing the you know, changes in the scores over time. And we can see, you know, the all the conversation that was taking place at that time as well. So we can see who are the you know, announcing scores and like what was the reactions at those times as well. So we'll let that go through here and process. And if we go back to this was the Greg Yang photo here. So if we scroll through here, whoops. So Greg Yang, of course, who has his favorite photograph that he has on his account. That's actually not how he looks like in real life, by the way. I'm just saying where, but it's quite funny.
But it had to understand that question. Yeah. That's the wild part is like, it understands what is a weird photo? What is a weird photo? Yeah. What is a less or more weird photo? It goes through has to find all the team members has to figure out who we all are, right? You know, searches, without access to the internal xAI personnel logs, literally looking at that just at the internet.
Exactly. So you could say like the weirdest of any company. Yeah. To be clear. Exactly. We can also take a look here at the question here for the humanities last exam. So it is still researching all of the historical scores. But it will have that final answer here soon. But we can while it's finishing up, we can take a look at one of the ones that we set up here a second ago. And we can see like, you know, it defines the date that like Dan Hendricks had initially announced it, we can go through we can see, you know, open AI announcing their score back in February. And we can see, you know, as progress happens with like Gemini, we can see like Kimmy, and we can also even see, you know, the leaked benchmarks of what people are saying is, you know, if it's right, it's going to be pretty impressive. So pretty cool. So yeah, I'm looking forward to seeing how everybody uses these tools and gets the most value out of them. But yeah, it's been great.
Yeah, we're gonna close the loop around usefulness as well. So it's like, it's not just book smart, but actually practically smart.
Exactly. And we can go back to the slides here. Yeah. So cool. So the so we actually evaluate also on the multi model subset. So on the full set, this is the number on the HLE exam. If you can see, there's a little dip on the numbers. This is actually something we're improving on, which is the multi model understanding capabilities. But I do believe in a very short time, we're able to really improve and got much higher numbers on this, even higher numbers on this benchmark.
Yeah. Yeah, this is the we still like, it's like, what is the biggest weakness of Grok currently is that it's, it's sort of partially blind, it can't, it's it's image understanding, obviously, in its image generation, needs to be a lot better. And that that that's actually being trained right now. So Grok 4 is based on version six of our foundation model. And we are training version seven. Which we'll complete in a few weeks. And that that'll address the weakness on the vision side.
Just to show off this last year, so the prediction market finished here with a heavy, and we can see here, we can see all the tools and the process that used to actually go through and find the right answer. So it browsed a lot of odd sites, it calculated its own odds comparing to the market, the market to find its own alpha and edge, it walks you through the entire process here. And it calculates the odds of the winner being like the the Dodgers, and it gives them a 21.6% chance of winning this year. So it took a process approximately four and a half minutes to compute.
Yeah, that's a lot of thinking. Yeah.
We can also look at all the other benchmarks besides HLE. As it turned out, Grok 4 excelled on all the reasoning benchmarks that people usually test on, including GPQA, which is a PhD level problem sets. That's easier compared to HLE. On AIME 25, American invitation mathematics exam, we with Grok 4 Heavy, we actually got a perfect score. Also on some of the coding benchmark live coding bunch. And also on HMMT, Harvard MIT exam, and also USAMO. You can see actually, for all of those benchmarks, we often have a very large leap against the second best model out there.
Yes, I mean, really, we're going to get to the point where it's going to get every answer right in every exam. And where it doesn't get an answer, right, it's going to tell you what's wrong with the question. Or if the question is ambiguous, disambiguate the question into answers A, B and C and tell you what it would what answers A, B and C would be with a disambiguated question. So the only real test then will be reality. Can it make useful technologies, discover new science, that'll actually be the only thing left because human tests will simply not be meaningful.
You can make update to HLE very soon, given the current rate of progress. So yeah, it's super cool to see like multiple agents that collaborate with each other solving really challenging problems. So where can we try this model? So it turned out it's available right now, if we advance to the next slide, where there is a super Grok Heavy tiers that we're introducing, where you're able to access to both Grok 4 and Grok 4 Heavy, where you're actually going to be the task master of bunch of little Grok research agent to help you, you know, become smarter, do a little research and save hours of times of, you know, going through mundane tasks. And it's available right now. So, yeah.
So, we did limit usage during the demo. So we didn't it didn't break the demo because all these days, all this stuff is happening live. So it's not there's nothing canned about the any of the tests that we're doing. So after the after the demos done, we'll allow we'll enable more subscribers for super Grok. So if you can't subscribe right now, just try and have an hour it should work. So and now let's let's get into voice.
Great. So many of you have been enjoying our voice mode. And we've been working hard to improve the experience over the past couple months. We have cut latency in half to make it much snappier. And today we're excited to announce a set of new voices that have exceptional naturalness and prosody. You might have noticed the movie trailer voice that opened up the live stream. That is one of our new voices, Sal, who's got that epically deep tone. And we're also excited to introduce Eve, a beautiful British voice who's capable of rich emotions. Aman, would you like to introduce Eve?
Absolutely. Yeah, let's get into the demo. Hey, Eve, millions of people are watching us right now. Are you excited?
