I would guess that the world is going to get a lot more like a lot funnier and like weirder. If you think that something someone is doing is bad and they think it's really valuable, most of the time in my experience they're right and you're wrong. I am worried that we're just removing all the friction between getting totally reward hacked by our technology. We're trying to build a coding agent that advances Llama research. I would guess that sometime in the next 12 to 18 months we'll reach the point where most of the code that's going towards these efforts is written by AI. I tend to think that for at least the foreseeable future this is going to lead towards more demand for people doing work not less. If you've gotten the cost of providing that service down to one-tenth of what it would have otherwise been, maybe now that actually makes sense to go do.
All right Mark, thanks for coming on the podcast again.
Yeah, happy to do it. Good to see you.
You too. Last time you were here, you had launched Llama 3. Yeah. Now you've launched Llama 4.
That's right. What's new? What's exciting? What's changed?
Oh, well, I mean, the whole field's so dynamic. I feel like a ton has changed since the last time we talked. Meta AI has almost a billion people using it now monthly, so that's pretty wild. I think this is going to be a really big year on all of this, especially once you start getting the personalization loop going, which we're just starting to build in now really, from both the context that all the algorithms have about what you're interested in, feed and all your profile information, all the social graph information, but also just what you're interacting with the AI about. I think that's going to be the next thing that's super exciting. So really big on that. The modeling stuff continues to make really impressive advances too, as you know. The Llama 4 stuff, I'm pretty happy with the first set of releases. We announced four models and we released the first two, the Scout and Maverick ones, which are kind of the midsize models, midsize to small. The most popular Llama 3 model was the 8 billion parameter model, so we've got one of those coming in the Llama 4 series too. Our internal code name for it is Little Llama, but that's coming probably over the coming months. But the Scout and Maverick ones are good. They're some of the highest intelligence per cost that you can get of any model out there. Natively multimodal, very efficient, run on one host, designed to be very efficient and low latency for a lot of the use cases we're building for internally. That's our whole thing: we basically build what we want and then we open source it so other people can use it too. So I'm excited about that. I'm also excited about the Behemoth model which is coming up. That's going to be our first model that is sort of at the frontier. It's more than two trillion parameters, so it's quite big. We're kind of trying to figure out how we make that useful for people. It's so big that we've had to build a bunch of infrastructure just to be able to post-train it ourselves. We're trying to wrap our head around how the average developer out there is going to be able to use something like this and how do we make it so it can be useful for distilling into models that are of reasonable size to run, because you're obviously not going to want to run something like that in a consumer model. But there's a lot to go. As you saw with the Llama 3 stuff last year, the initial Llama 3 launch was exciting and then we just built on that over the year. 3.1 was when we released the 405 billion model. 3.2 was when we got all the multimodal stuff in. So we basically have a road map like that for this year too. So a lot going on.
I'm interested to hear more about it. There's this impression that the gap between the best closed source and the best open source models has increased over the last year. I know the full family of Llama 4 models isn't out yet, but Llama 4 Maverick is 35 on Chatbot Arena, and then on a bunch of major benchmarks it seems like 04 mini or GPT Gemini 2.5 Flash are beating Maverick, which is in the same class. What do you make of that impression?
Yeah, well okay, there's a few things. I actually think that this has been a very good year for open source overall. If you go back to where we were last year, what we were doing with Llama was like the only real super innovative open-source model. Now you have a bunch of them in the field. And I think in general the prediction that this would be the year where open-source generally overtakes closed source, the most used models out there, I think is generally on track to be true. I think the thing that's been an interesting surprise, I think positive in some ways, negative in others, but overall good, is that it's not just Llama. There are a lot of good ones out there. So I think that's quite good. Then there's the reasoning phenomenon which you are alluding to with talking about 03 and 04 and some of the other models. And I do think that there's this specialization that's happening where if you want a model that is sort of the best at math problems or coding or different things like that, I do think that these reasoning models with a lot of ability to just consume more test time or inference time compute in order to provide more intelligence is a really compelling paradigm. But for a lot of the applications that we care about, latency and good intelligence per cost are actually much more important product attributes. If you're primarily designing for a consumer product, people don't necessarily want to wait half a minute for it to think through the answer. If you can provide an answer that's generally quite good in half a second, then that's great and that's a good trade-off. So I think both of these are going to end up being important directions. I am optimistic about integrating the reasoning models with the core language models over time. I think that's sort of the direction that Google has gone in with some of the more recent Gemini models, and I think that's really promising, but I think there's just going to be a bunch of different stuff that goes on.
You also mentioned the whole chatbot arena thing, which I think is interesting and it goes to this challenge around how do you do the benchmarking right and basically how do you know what models are good for which things. One of the things we've generally tried to do over the last year is anchor more of our models in our Meta product northstar use cases because the issue with both open-source benchmarks and any given thing like the LM Arena stuff is that they're often skewed for a very specific set of use cases which are often not actually what any normal person does in your product. They are often weighted, the portfolio of things they're trying to measure is different from what people care about in any given product. Because of that, we've found that trying to optimize too much for that stuff has often led us astray and actually not led towards the highest quality products and the most usage and best feedback within Meta as people use our stuff. So we're trying to anchor our northstar in basically the product value that people report to us, what they say they want, what their revealed preferences are, and using the experiences that we have. So sometimes these things just don't quite line up, and I think a lot of them are quite easily gameable. On the Arena you'll see stuff like Sonnet 3.7, it's a great model, and it's not near the top. It was relatively easy for our team to tune a version of Llama 4 Maverick that was way at the top, whereas the one we released that's the pure model actually has no tuning for that at all, so it's further down. So you just need to be careful with some of the benchmarks, and we're going to index primarily on the products.
Do you feel like there is some benchmark which captures what you see as a northstar of value to the user, which can be sort of objectively measured between the different models, and you're like I need Llama 4 to come out on top on this?
Well, I mean our benchmark is basically user value in Meta AI, right? So it's...
But you can't compare other models.
Well, we might be able to because we might be able to run other models in that and be able to tell. And I think that's one of the advantages of open source: you have a good community of folks who can poke holes at where your model is not good and where it is good. I think the reality at this point is that all these models are optimized for slightly different mixes of things. I mean everyone is trying to go towards the same thing, I think all the leading labs are trying to create general intelligence and super intelligence, basically AI that can lead towards a world of abundance where everyone has these superhuman tools to create whatever they want and that leads to dramatically empowering people and creating all these economic benefits. I think that's sort of what a lot of the labs are going for. But there's no doubt that different folks have optimized towards different things. I think the Anthropic folks have really focused on coding and agents around that. The OpenAI folks have gone a little more towards reasoning recently. And I think there is a space which, if I had to guess, will end up probably being the most used one: quick, very natural to interact with, very natively multimodal, that fits into throughout your day the ways you want to interact with it. You got a chance to play around with the new Meta AI app that we're releasing. One of the fun things we put in there is the demo for the full duplex voice. And it's early, there's a reason why we haven't made that the default voice model in the app. But there's something about how naturally conversational it is that I think is just really fun and compelling. Being able to mix that in with the right personalization is going to lead towards a product experience where I would guess that a few years from now, we're just going to be talking to AI throughout the day about different things we're wondering. You'll have your phone, you'll talk to it, you'll talk to it while browsing your feed apps. It'll give you context about different stuff. You'll be able to answer questions. It'll help you as you're interacting with people in messaging apps. Eventually, I think we'll walk through our daily lives and we'll either have glasses or other kinds of AI devices and just be able to seamlessly interact with it all day long. That's kind of the northstar. Whatever the benchmarks are that lead towards people feeling that the quality is what they want to interact with, I think that's ultimately going to matter the most to us.
I got a chance to play around with both, and also the Meta app and the voice mode was super smooth, it was quite impressive. On the point of what the different labs are optimizing for, to steelman their view, I think a lot of them think that once you fully automate software engineering and AI research, then you can kick off an intelligence explosion where you have millions of copies of these software engineers replicating the research that happened between Llama 1 and Llama 4, that scale of improvement again in the matter of weeks or months rather than years. So it really matters to just close the loop on the software engineer and then you can be the first to ASI. What do you make of that?
Well, I personally think that's pretty compelling. That's why we have a big coding effort too. We're working on a number of coding agents inside Meta. We're not really an enterprise software company, we're primarily building it for ourselves. So we go for the specific goal. We're not trying to build a general developer tool. We're trying to build a coding agent and an AI research agent that basically advances Llama research specifically and is fully plugged into our tool chain. I think that's important and will be an important part of how this gets done. I would guess that sometime in the next 12 to 18 months we'll reach the point where most of the code going towards these efforts is written by AI. I don't mean autocomplete, I mean you give it a goal, it can run tests, it can improve things, find issues, writes higher quality code than the average very good person on the team already. I think that's going to be a really important part of this for sure. But I don't know if that's the whole game. I think that's going to be a big industry, an important part of how AI gets developed. But there are still guys, I think one way to think about this is this is a massive space. I don't think there's just going to be one company with one optimization function that serves everyone as best as possible. I think there are a bunch of different labs doing leading work towards different domains. Some are going to be more enterprise focused or coding focused. Some are going to be more productivity focused. Some are going to be more social or entertainment focused. Within the assistant space, there are going to be some that are much more informational or productivity, some more companion focused. There's going to be a lot of stuff that's fun and entertaining and shows up in your feed. So there's just a huge amount of space. Part of what's fun about this is going towards this AGI future. There are a bunch of common threads for what needs to get invented, but there are a lot of things need to get created. I think you'll start to see a little more specialization between the groups if I had to guess.
It's really interesting to me that you basically agree with the premise that there will be an intelligence explosion and something like super intelligence on the other end. But if that's the case, why even bother with personal assistance and whatever? Why not just get to superhuman intelligence first and then deal with everything else?
Well, I think that's just one aspect of the flywheel. Part of what I generally disagree with on the fast takeoff thing is it takes time to build out physical infrastructure. If you want to build a gigawatt cluster of compute, that just is going to take some time. It takes Nvidia a bunch of time to stabilize their new generation of systems, then you need to figure out the networking, build the building, get permitting, get energy, whether it's gas turbines or green energy, there's a whole supply chain. Some of these are just physical world human time things. As you start getting more intelligence in one part of the stack, you'll run into a different set of bottlenecks. That's sort of the way engineering always works: you solve one bottleneck, you get another bottleneck. Another bottleneck in the system or another ingredient that's going to make this work well is basically people getting used to and learning and having a feedback loop with using the system. These systems don't tend to be the type of thing where something just shows up fully formed and people magically fully know how to use it. There is a co-evolution where people are learning how to best use these AI assistants, and on the same side, the AI assistants are learning what those people care about, and the developers are able to make the AI assistants better. Then you're also building up this base of context. So now you wake up a year or two into it and the AI assistant can reference things you talked about two years ago. That's pretty cool, but you couldn't do that if you just launched the perfect thing on day one. There's no way it could reference what you talked about two years ago if it didn't exist two years ago. So my view is there's this huge intelligence growth.
There's a very rapid curve on the uptake of people interacting with the AI assistants and the learning feedback and data flywheel around that. And then there is also the buildout of the supply chains and infrastructure and regulatory frameworks to enable the scaling of a lot of the physical infrastructure. I think at some level all of those are going to be necessary, not just the coding piece. Even if you go back a few years ago, we had a project on our ads team to automate ranking experiments. That's a pretty constrained environment. It's not write open-ended code, it's basically look at the whole history of the company, every experiment any engineer has ever done in the ad system, look at what worked, what didn't, the results, and basically formulate new hypotheses for different tests that could improve the performance of the ad system. What we found was we were bottlenecked on compute to run tests based on the number of hypotheses. Even with just the humans we have on the ads team, we already have more good ideas to test than we have compute or cohorts of people to test them with. Even if you have three and a half billion people using your products, each test needs to be statistically significant, so it needs hundreds of thousands or millions of people. There's only so much throughput you can get on testing. So we're already at the point where we can't really test everything we want. So just being able to test more things is not necessarily additive. We need to get to the point where the average quality of the hypotheses the AI is generating is better than what the things above the line that we're actually able to test, like the best humans on the team have been able to do, before it'll even be marginally useful. We'll get there pretty quickly, but it's not like the thing can write code and all of a sudden everything is improving massively. There are real world constraints that first need to do a reasonable job, then you need the compute and the people to test, and then over time as the quality creeps up, maybe in five or ten years no set of people can generate a hypothesis as good as the AI system. Then in that world, that's how all the value is created. But that's not the first step.
Publicly available data is running out. So major AI labs like Meta, Google DeepMind, and OpenAI all partner with Scale to push the boundaries of what's possible. Through Scale's data foundry, major labs get access to high-quality data to fuel post-training, including advanced reasoning capabilities. Scale's research team, SEAL, is creating the foundations for integrating advanced AI into society through practical AI safety frameworks and public leaderboards around safety and alignment. Their latest leaderboards include Humanity's Last Exam, Enigma Eval, MultiChallenge, and Vista, which test a range of capabilities from expert-level reasoning to multimodal puzzle solving to performance on multi-turn conversations. Scale also just released Scale Evaluation, which helps diagnose model limitations. Leading frontier model developers rely on Scale Evaluation to improve the reasoning capabilities of their best models. If you're an AI researcher or engineer and you want to learn more about how Scale's data foundry and research lab can help you go beyond the current frontier of capabilities, go to scale.com/tharkcash.
So if you buy this view that this is where intelligence is headed, the reason to be bullish on Meta is obviously that you have all this distribution, which you can also use to learn more things that can be useful for training. You mentioned the Meta AI is now has a billion active users.
Not the app. The app is a standalone thing that we're just launching now. It'll be fun for people who want to use it. We're kind of experimenting with some new ideas in there that are novel and worth talking through. But I'm talking mostly about our apps. Meta AI is actually most used in WhatsApp. Got it. WhatsApp is mostly used outside of the US. We just passed 100 million people in the US, but it's not the primary messaging system in the US. So people in the US probably underestimate Meta AI use somewhat. That's also why the standalone app is going to be so important. The US is one of the most important countries, and the fact that WhatsApp is the main way people are using Meta AI and that's not the main messaging system in the US means we need another way to build a first-class experience in front of people.
To finish the question, the bearish case would be that if the future of AI is less about just answering your questions and more so being a virtual coworker, it's not clear how Meta AI inside of WhatsApp gives you the relevant training data to make a fully autonomous programmer remote worker. In that case, does it not matter that much who has more distribution right now with LLMs?
Again, I just think there are going to be different things. It's like if you were sitting at the beginning of the development of the internet and it's like, what's going to be the main internet thing? Is it going to be knowledge work or is it going to be massive consumer apps? I don't know, you get both, right? You don't have to choose one. The world is big and complicated. Does one company build all that stuff? I think normally the answer is no. People do not code in WhatsApp for the most part, and I don't foresee that becoming a major use case, although I do think people are going to ask AI to do a lot of things that result in the AI coding without them necessarily knowing it. But we do have a lot of people writing code at Meta and they use Meta AI. We have this internal thing we call Metamate, and a number of different coding and AI research agents we're building around that. That has its own feedback loop and can get good for accelerating those efforts. Again, there are going to be a bunch of things. AI is almost certainly going to unlock a massive revolution in knowledge work and code. I also think it's going to be the next generation of search and how people get information and do more complex information tasks. I also think it's going to be fun. People are going to use it to be entertained. A lot of the internet is memes and humor. We have this amazing technology at our fingertips. It is sort of amazing and funny when you think about it how much human energy goes towards entertaining ourselves, design, pushing culture forward, finding humorous ways to explain cultural phenomena. That's almost certainly going to be the case in the future. If you look at the evolution of things like Instagram and Facebook, if you go back 10, 15, 20 years ago, it was text. Then we all got phones with cameras, most of the content became photos. Then the mobile networks got good enough that video became common. Over the last 10 years, most of the content has moved towards video. Most of the time spent in Facebook and Instagram is video. But do you think in 5 years we're just going to be sitting in our feed consuming media that's video? No, it's going to be interactive. You'll be scrolling through your feed and there will be content that looks like a reel to start, but then you talk to it or interact with it and it talks back or changes what it's doing, or you can jump into it like a game. That's all going to be AI. So there's just all these different things, and we're ambitious so we're working on a bunch of them. But I don't think any one company is going to do all of it.
Okay. So on this point of AI generated content or AI interactions, already people have meaningful relationships with AI therapists, AI friends, maybe more. And this is just going to get more intense as these AIs become more unique, more personable, more intelligent, more spontaneous and funny. How do we make sure people are going to have relationships with AI? How do we make sure these are healthy relationships?
Well, I think there are a lot of questions that you only really can answer as you start seeing the behaviors. So probably the most important upfront thing is just ask that question and care about it at each step along the way. But I think being too prescriptive upfront and saying we think these things are not good often cuts off value. I don't know, people use stuff that's valuable for them. One of my core guiding principles in designing products is that people are smart, they know what is valuable in their lives. Every once in a while something bad can happen in a product, and you want to design your products well to minimize that. But if you think that something someone is doing is bad and they think it's really valuable, most of the time in my experience they're right and you're wrong. You just haven't come up with the framework yet for understanding why the thing that you're doing is valuable and helpful in their life. That's kind of the main way I think about it. I do think people are going to use AI for a lot of these social tasks. One of the main things we see people using Meta AI for is talking through difficult conversations they need to have with people in their life. Like, I'm having this issue with my girlfriend, help me have this conversation, or I need to have a hard conversation with my boss, how do I do that? That's pretty helpful. Then as the personalization loop kicks in and the AI gets to know you better and better, I think that will be really compelling. One thing from working on social media for a long time: there's a stat that the average American has fewer than three friends, three people they'd consider friends. The average person has demand for meaningfully more, like 15 friends. So the average person wants more connection than they have. There are a lot of questions about whether this will replace in-person connections or real life connections. My default is probably no. There are all these things that are better about physical connections when you can have them, but the reality is people don't have the connection and they feel more alone a lot of the time than they would like. So I think a lot of these things that today might have a bit of stigma, I would guess over time we will find the vocabulary as a society to be able to articulate why that is valuable and why the people who are doing these things are rational for doing it and how it is adding value for their lives. But the field is very early. There are a handful of companies doing virtual therapists and virtual girlfriend stuff, but it's very early. The embodiment in the things is pretty weak. You open it up and it's just an image of the therapist or the person you're talking to, sometimes with rough animation, but it's not like an embodiment. You've seen the stuff we're working on in Reality Labs with the Codec Avatars, where it feels like a real person. I think that's where it's going. You'll be able to have an always-on video chat, and the gestures are important too. More than half of communication when you're having a conversation is non-verbal.
I did get a chance to check out Orion the other day and I thought it was super impressive. I'm mostly optimistic about the technology because generally, as you mentioned, I'm libertarian about if people are doing something, probably they think it's good for them. Although I actually don't know if it's the case that if somebody is using TikTok they would say they're happy with how much time they're spending on TikTok. So I'm mostly optimistic about it also in the sense that if we're going to be living in this future world of AGI, we need to keep up with it, humans need to upgrade their capabilities as well with tools like this. And generally there can be more beauty in the world if you can see Studio Ghibli everywhere or something. I was worried that one of the flagship use cases your team showed me was sitting at the breakfast table and on the periphery of my vision is just a bunch of reels that are scrolling by. Maybe in the future my AI girlfriend is on the other side of the screen. So I am worried that we're just removing all the friction between getting totally reward hacked by our technology. How do we make sure this is not what ends up happening in 5 years?
I think people have a good sense of what they want. That experience you saw was more a demo just to show multitasking and holograms. I don't think the future is you have stuff trying to compete for your attention in the corner of your vision all the time. I don't think people would like that too much. It's one of the things as we're designing these glasses that we're really mindful of: probably the number one thing glasses need to do is get out of the way and be good glasses. As an aside, I think that's part of the reason the Ray-Ban Meta product has done so well. It's great for listening to music, taking phone calls, taking photos and videos, and the AI is there when you want it, but when you don't, it's a great-looking pair of glasses that people like and it gets out of the way. I would guess that's going to be a very important design principle for the augmented reality future. The main thing I see here is it's kind of crazy that for how important the digital world is in all our lives, the only way we can access it is through these physical digital screens. It just seems like we're at the point with technology where the physical and digital world should really be fully blended, and that's what the holographic overlays allow you to do. But I agree a big part of the design principles around that are going to be that you'll be interacting with people and you'll be able to bring digital artifacts into those interactions and do cool things seamlessly. If I want to show you something here, here's a screen, you can interact with it, it can be 3D, we can play with it. You want to play a card game? Here's a deck of cards, we can play. Two of us are here physically, you have a third friend hologramming in. But in that world, people are not going to want their physical space to be cluttered. It wears on you psychologically. I don't think people are going to want the digital physical space to feel that way either. That's more of an aesthetic and one of these norms we'll have to work out, but I think we'll figure it out.
Going back to the AI conversation, you mentioned how big of a bottleneck the physical infrastructure can be. Related to other open-source models like DeepSeek, DeepSeek right now has less compute than Meta, and you could argue it's competitive with the Llama models. If China is better at physical infrastructure, industrial scale-ups, getting more power and more data centers online, how worried are you that they might beat us here?
