Joining us on stage is the co-founder and chief science officer at Hugging Face, Thomas Wolf.
Hello, everyone. Hello, Olive. Nice to have you on stage.
Hi, nice to meet you. Thanks for having me, yeah.
So, I think you're in for a treat today because you just saw GLM, which is currently number two on the artificial intelligence leaderboard. I take Fable out because nobody can use it. And now we have number four. So basically, you will have all the top models, at least the top open-source models in a row. And we're very lucky to have Olive, who has a pretty amazing path in life. So she came to the US, Pennsylvania. She was studying, doing a PhD at NYU in the lab of Yann LeCun, working on JEPA, but we decided we won't talk about JEPA today, right? Something for another day. And then instead of joining Hugging Face, which was in New York also at that time, she decided to go join MiniMax.
So for those who maybe don't know all the neo-labs around the world, and you're forgiven because I think there's like 64 neo-labs right now, MiniMax is one of the top of what we call the AI dragons in China. So these are the new—there's DeepSeek, which is very well known now, Moonshot who does Kimi, and GLM that you just saw, and now we have MiniMax. They're all extremely good, an extreme A-team fighting for the first spot. So the latest release of MiniMax was M3 just earlier in June, which was the top model at the time, top open-source model. Very impressive. There's a lot of very interesting things about this model, so we'll quickly dive into them and then talk a little bit about what's specific about MiniMax, what's great there. So maybe, Olive, to start a little bit, can you give us a little bit of your view of M3, what you like about this model, how was the release?
Yeah, we released M3 earlier this month, and it is a smaller model with around 400 billion total parameters and 20 billion activated. But it is very capable in terms of both coding performances and also it understands vision. So that's what open-source models don't usually have. It's that the model can only deal with coding, but it can also understand videos, images, and it has a super long context of 1 million, with our new architecture called MSA, MiniMax Sparse Attention. So we really put these three things together because we know that they will be very important in future AI applications: coding capabilities, agentic capabilities, longer context, and multimodal understanding. Yeah, I think that would be very interesting about the model.
Yeah, so there's a lot to unpack in this model, and it's still, I think, the only top five open-source model that is actually multimodal, so we need to talk about that. But maybe first about the long context, because that was also the first one that really had this real 1 million token long context that's actually functional. And you guys had also the MiniMax Sparse Attention, which is this one technique to make that efficient that you also published and shared extensively. So can you talk a little bit about this? Maybe how the project went from the attention, how to make this long context?
Yeah, I would say the story about long context went back to even MiniMax M1 and MiniMax 01, where the model was actually able to perform tasks of 10 million token context.
10 million, yes. But then it was not an agentic model, right? It was just, for example, you dump in a book, it would be able to give reviews on it and stuff like that. So what we realized was that longer context actually unlocks a lot of capabilities, especially when interacting with users. And now when the agent is interacting with the whole environment and getting all the tool responses, getting multi-rounds, the shorter context wouldn't be enough to perform the complex task. So for this version, we said, "Oh, we have to have our longer context back." So what we pursued was with our MiniMax Sparse Attention, which was the architecture that was scalable and had a simple design.
So I would say from a higher level, right, it has an index branch that selects on a higher level what matters more in the context, and then we have a sparse attention branch that performs the calculation on the selected blocks to actually perform the task. And so, yeah, like that, we really designed an elegant architecture so that we can scale the length and then scale the model size in the future with that.
That's beautiful. I like how, for those who've been in the field for quite some time, we had a lot of work on attention, right? This N-squared, and there was a lot of linear attention.
And then somehow all of these disappeared at some point when FlashAttention came around. We discovered we just needed more efficient compute. Now, I like how we come back to thinking first principles: what is attention? How can we make that more efficient? So 1 million token is crazy, right? GPT-2 was 1,024, and everyone was like, "Oh, that's really big. We never need more." Where do you see this going in the future? Jeff Dean was pitching me the other day a trillion-token attention. Do you think we should go to a trillion-token attention?
So that's definitely something we can explore towards, right? Ultra-lengths of the context. Definitely, it's something that's very exciting to explore with, and something that architecture design along with hardware would require a lot of research onto that, yeah.
You think there's still a lot of low-hanging fruits? Typically today we saw OpenAI reducing—we don't know how yet, but reducing their inference bill by half, by probably having some more efficient processing around attention of any type. Do you think there's still a lot of low-hanging fruit that can be gained in how we can process that? One thing we're still very interested about M3 is how cheap it is, in particular because of this sparse attention and partly because of its small size, but it's also very efficient.
Do you think we can go even way further? And maybe how did you guys invent MiniMax Sparse Attention? Was it an agent coming up with the idea? Was it a human still coming up with the idea? Tell us a little bit about that.
Yeah. So we do think there's still a lot of work that can get into architecture and inference optimization so that the model can be more efficient, especially if there are tasks that are very latency-sensitive but require very strong capabilities, right? And for those kind of tasks, we really want the model to be efficient. And who came up with this part? Actually, I think an intern from our team worked on that. Yeah, an intern. That doesn't usually happen in a lot of labs because I think in some labs interns don't have access to the data, the work, and stuff. But yeah, we are open to anyone who would like to contribute to our models. So the architecture was actually designed by an intern.
It's very good. Still some work for interns here. Good news. That's also a good segue to how MiniMax is working internally. We were discussing before coming on stage, I was saying everyone can propose a project. Can you tell us a little bit about how you are organized, how you do your research?
I think that is very different from even in school or even in the earlier tech companies. It's pretty different. What we make sure is that we have good foundation and good infrastructure so that anyone can play with the model and can think of what they can improve with the model. And then after model releases, when they are free, right, they can play with the model, they can think of their own evaluations, they can find their own weaknesses, and propose a thing that they want to improve on the model. And then other people who are interested in that would propose to join the project, and they will work on it for a couple of weeks or even a couple of months. And when they work out, the final thing is shipped to our model. We use that in our final training, and it's shipped out to the audience.
Interesting. So you can have people working for a really long time on a project. When you say a couple of months, it can be like really deep exploration of—
Yes, yes. I would say, for example, architecture might require longer time of investigation, research, experiments, even redoing the evaluations for pre-training. Yes, so it might require longer time.
Very nice, yeah. And I know you're also very big on evaluation. I agree we could talk about that. One thing probably related to that is this unique specificity that M3 and your team has around multimodality. So not just text, but the model can also understand image and video. And as I understand—but please explain better—when you read the model card on Hugging Face, it says the model was trained from the first step as multimodal, not just having a user one as an afterthought, right? Can you tell us a little bit more about that and why you think it's important, and why starting from the first step on multimodal training and not just training this later?
So we call it native multimodality. It is somehow typical for model labs to train the multimodal, let's say vision understanding capabilities, after the text pre-training is done. They put adapters and then train that part. But what we found out was that that would actually harm the text performance, and the vision understanding performance wouldn't converge that well because the model kind of converges towards the text understanding. And it's just not the most optimal, and also not the most scalable if you think about it. We want to scale the data, right?
Also, some labs train this capability from halfway through the pre-training, for example, continued pre-training. But what we found is that this would be very recipe-sensitive. The recipe would be different for different architectures, different data mixtures, different learning rates. It's hard to control, hard to scale. You can't really scale your experiment results and conclusions to a larger model.
And so what we thought was, why not just train it from the very first step? That comes the most natural. We know that a lot of labs run into problems doing that. The model would collapse after a couple of steps of training both text and vision understanding, but we managed to solve that problem. We did a lot of work on ViT, and we did a lot of work on the data that we actually train on. For example, we do interleaved data, what we call interleaved data. It's actually natural data, but we keep the images and videos in instead of masking it out, and we do some pretty good cleaning and masking on the data, and we do very good reward modeling so that we train it from the first step and it scales up a lot. Yeah, it does not collapse.
That's really impressive. Should we expect much larger models in the future? So this one is still fairly small, right?
