This is going to be a thing that billions of people want to use.
What does building things feel like to you? Is live forever a possibility? Okay, so like if you take me into your head and we flash forward to like 2030, what does it look like?
This is Mark Zuckerberg. 22 years ago, he built a social network that connected billions of people and changed the world forever. But now he's decided to build something even bigger. To do it, he's making massive bets on the metaverse, smart glasses, and AI. For years, skeptics saw these as three separate impossible gambles. But they missed the bigger picture because right now those bets are converging into creating an entirely new kind of super intelligence. So today we're going to show you that and I'm going to ask Mark questions that he's never been asked before and hear his vision for the future so you can get ahead of it and build the next big thing.
Thanks so much for coming on the show.
Yeah, to have you.
Yeah. No, I'm excited to do this.
To prepare for this one, I watched every interview that you've ever done.
Okay, that's more than I've ever done.
It was a lot of fun. It was great. Um, and like two major things stood out to me. Number one, your like love of building. Feel like you're just like one of the best builders. Um, and then number two, your ability to take like massive bets. And I think like this week is kind of the week that all of them are coming together. So, let's start there.
Yeah. Um, yeah. I guess we've been working on a lot of this stuff for a long time. I mean, it's uh the I mean the AI work I mean we've kind of been doing as a company I mean almost since the beginning. I mean there's like the first version of newsfeed from the beginning is sort of this machine learning product and so I mean yeah there's there's that version and then the the AI research lab we we started gosh it's like not quite 15 years ago but uh but then you know we're in a new version now because about a little more than a year ago we uh kicked off Meta Super Intelligence Labs.
Yep. And that it was a kind of wild reboot of the research to bring in a lot of great people from across the industry. But it's been exciting because now, you know, we're seeing the models get better and better. We have um the next generation launching soon, not a connect. Um but and then we have um you know, the Muse personal agent that is, you know, it's uh the reception so far has been great. You know, when
Yeah. I mean, when you're building this stuff, you don't quite know.
Um, you know, even from just earlier in the year, you know, just kind of duct taping together like my own version of Open Claw at home and getting a sense of like what how does this work and like um what would make this a magical experience that you could give to anyone. Yeah. like we like basically as soon as I I saw that you know we we basically you know Nat um and Alex and I sat down and we were like okay this is an amazing and magical experience and if we can deliver a version of this that just works to people who are not necessarily technical enough to want to go install their own Mac Mini or get in a terminal and set it up or debug stuff when when it's not working. um then I think that this is going to be a thing that billions of people want to use. So we've basically been working on that over time since then. You know, getting the models to be tuned specifically for it. Um, building not just the agent and the scaffolding, but effectively the technology to give every agent its own computer, right? We built this whole Muse secure VM and um yeah, so we thought this was pretty special and internally if people have really liked it, but you never really know when um when you ship something how people are going to respond and you know every once in a while it's a it's a home run and people love it out of the gate, but most of the time you know you get some feedback that's positive um you have to iterate on a few things before they they kind of really click. Um this one just really clicked right out of the gate. So, it's been it's been very exciting to see and and the fact that we're like what like 2 weeks in and there's millions of people using it is um
I don't know this is pretty rare. I mean, every few years we get something like this, but it's um this is definitely you know the fun part of building a company.
Yeah. I feel like your ability to like stay in the game and keep trying things is admirable and like also so many hits like it's it's great and there was an interview that you did I think it was like three years ago with Joe Rogan where at the end you were talking about one day you'll like throw on the glasses and then like an AI agent will come in as well. So, I feel like it's so smart that Muse is already embodied because that feels like that was going to happen.
Yeah. I think it also just makes it friendly and cute.
So, part of this is like I think so many people talk about AI as if it's this like super scary thing. Yeah. And I don't know. I mean, I think it it should just be useful and fun and um so I I I thought that that embodiment um one of the designers early on in the project put together that character and um and for whatever reason they kept on trying to iterate on it, but the first one was just the best. So there were like like some at some point someone was like, 'Oh, it's got to be blue because it's meta.' And it's like, 'No, like I think this guy is really like I think you nailed it on the first try.' So yeah. No, it's just fun. It was a good touch.
How does it change how you like train the model if you're trying to make it like really good at personal stuff versus like a general intelligence model?
Well, I think one of the things right now is basically there's generally making the models good as agents.
Um, so I think for the last year the biggest thing has basically been these coding agents and there are two ideas that are sort of embedded in that. There's the expertise at coding and there's the generally being a good agent. and our strategy. Coding is important because even if a person isn't thinking that they're writing code, your muse is writing code for you to do all this stuff in the background all the time. So, so being able to code is important. But we basically took the strategy that we thought that the agent should be primarily a great agent and um and so we focused on that. There's also a bunch of things that um that basically I think are just more important when you're building a personal agent than if you're building like an enterprise software uh product, right? So for example, if you're building cloud code, your your model it doesn't need to have any sense of like discretion like what
Like what to divulge and not divulge.
How do you think about that?
Well, I think that that's a very important thing for something like Muse, right?
Yeah. So, I mean, you have to train that into the model. I mean, just like any other capability, it needs to be good at knowing like you tell it a lot of things and then you want it to go achieve your goals, but you know, let's say you want it to like go get you a reservation at a restaurant and you know, you like you're trying to find a good restaurant, but maybe you have some allergy, but you may not want to like disclose the allergy or if you're pregnant or something like that. But, you want it like the Muse will know if you tell it. um and you're gonna want to be able to do the job but while disclosing as little information as possible. So that's a specific skill and it's something that's like kind of basic social skills or common sense that people have. But if you're just building like a coding agent, most of these companies haven't trained that in. So that's like the type of thing that we can do because we're doing the full stack, right? Because we're training the model. We're not just taking someone else's model off the shelf and trying to build a scaffold and an agent around it. We trained the whole model specifically to be good at that. It obviously needs to be good at a lot of things in order to be generally intelligent, but it's good at that. And um and then you build the kind agent and all the all of the details around it from the memory, the the kind of harness, the way that it like the heartbeat that it like wakes up and checks. Okay, here are the goals that I know about you. Is there anything I can be doing to to help advance any of those? Now, um, and we had a whole team that was just focused on making good kind of real time animations of the avatars because it's not just like the default avatar. You can make whatever avatar you want and then it just like naturally animates and it's awesome and you know, we're rolling out this voice mode and you can like have a live voice chat and it's just like and your guy is there like whatever it is, you know. So, um, all these details I think you basically just get from kind of like being full stack and doing all the models and the product together.
100%. And I feel like the other big thing is the virtual machine. Um, can you explain like why that matters and what is unlocked?
Yeah. So, I know there's a few things. Um, basically an agent in order to be able to do stuff for you needs a place to store your information. And we don't think it should just be like in a pool with everyone else's information, right? So you want like your own thing. I I guess one way to think about this is your agent is going to know a lot of sensitive things about you. A lot of when people started getting into agents like OpenCloud, they basically got their own like Mac Mini thing at home. And so we basically had this idea which is okay like a lot of people they're not going to want to buy a Mac Mini or set it up. What's the best experience that we can create that effectively approximates that? you can just get an app and sign up and you get your own computer provisioned for your agent to work with to store all your stuff on to be secure and to design the security model around that. So that way it kind of approximates having your own computer under your desk that you know even like a company like us isn't going to be able to if you're using the Muse confidential VM which is another thing that we're working on now like we won't even be able to see what is on your um on on your in your virtual machine. I mean, there's all these things that we built around that, the secure credential store, so that way when you're doing passwords and stuff like that, your agent doesn't need to be able to see that. It just needs to be able to kind of like plug it in when when it um when you ask it to log into something, but only when you ask it to do that. It should be kind of designed in a way where that information isn't just accessible because I don't know, accidents happen, right? Someone might try to hack into the thing or um or there could be an issue. So, so basically you want to set it up so the agent just doesn't have access to that and that meta just doesn't have access to that. Um, so giving every agent its own computer and making it as secure as possible is just like a very fundamental piece of technology that had to get built in order to give um Muse both the power that it needs to be able to like go help people with your goals and the security and privacy to be like a kind of worldclass industryleading product in that area.
So does that mean like you use Muse and it has a virtual machine that is connected to but all the data is encrypted in the same way that it's encrypted on WhatsApp like how can people think about what the data actually looks like when it's living in the virtual machine.
Yeah. So we basically built two versions u Muse secure VM okay has all these different privacy features including you know we built this whole Sentinel agent architecture. So you have your normal Muse agent that's doing stuff for you and then you basically have this security agent that we call Sentinel that is just monitoring data going in and out to your Muse. So if um something comes in and someone's trying to like like compromise the security, it'll just cut it off and make it so it doesn't happen. If it thinks that your muse is going to take an action y that you should be in the loop on, it will basically override your muse and and trigger that back to the person to say for permissions. It's like, do you want your muse to be able to do this? this whole kind of constellation of systems and that plus the secure credential storage plus um just like several layers of security through through um through depth there we um make up the the Muse Secure VM. We're also working on this other thing that Nat and I specifically recruited Moxy Marlin Spike, the guy who we worked with to do the WhatsApp encryption to design the Muse confidential VM. And the idea behind that is that it's an on top of everything else, it is a virtual machine where you get an encryption key. So that way meta can't even access the contents of that. Um there's it's it's much harder to implement because obviously if meta can't access what's within it then it's a lot harder to debug so and get the system working. So like so it's it's taking us a little bit of time to get that shipped but um but that's coming soon and it's basically going to be the standard of security that people are used to for WhatsApp and and some of um you know our most secure products like that.
And is the advantage of that just like psychologically people will think it's more secure or are there actual like benefits?
Well, no. I mean, I think that that Well, I think that that security is a big deal. I mean, it's if if we're we're trying to again approximate having you have an agent with a local machine under your desk. So, what do you get when you have the local machine under your desk? That means that no company has access to it. So, okay, let's say Meta wants to offer you the service. How do we make it so you have that same level of privacy and security guarantee that no company, in this case, Meta or you know, anyone who tries to hack into us or if you're in some country where you don't trust the government. if the government might try to get us to do something, it's like we can't we can't get into it because we don't have access to it. I think that that really matters and that's a lot of why people feel comfortable with WhatsApp.
Um, so that I think is a real that's a real thing for for privacy and security and trust. And if you're going to have an agent that knows a lot about you, which I would guess pretty much we all are going to have that. I mean, I think you fast forward 5 years, like everyone is going to have an agent that like really intimately understands your goals and everything around you and um and can help you get stuff done. And in order to do that, I think that that level and being industryleading in privacy and security is very important. We wanted to do that from the beginning.
That's cool. Yeah. Something also very interesting about you is that you studied psychology in college. Yeah. Um when I was there for a very short period of time, took two years, but I feel like it's kind of informed a lot of what you built. Um, and so I'm curious like when we look at benchmarks a lot of times we'll be like, um, the model is super smart. Um, but it also feels like when we pick our friends, we definitely pick our friends because they're smart, but we also definitely pick them because we like their energy or being around them. Um, how do you think about shaping the model and like the personality?
Yeah. I mean, I think ideally it will be adaptable enough to be able to fit different people's energy, right? I think this is actually something that I think a lot of people get wrong in the industry. um you know a lot of the other labs I think have have focused on like how do we get the personality right and I just never thought that it's one thing and this is part of the reason why I believe so strongly in open source and people being able to customize stuff and also why we designed Muse to be this thing that is very personal um you can personalize it and customize it you know you can you can kind of set up the not just the avatar and the voice the way that you want but like right when you sign up for the first time one of the first things that it asks you to is like what do you want my basic personality to be and you can go and edit it at any time. Um so we try to design the models to be very steerable right so that way you can kind of define how you want to interact with it and I think that that's a really important thing. I mean you were asking about like how do we make it so that this is good as a personal agent. I think that like having that adaptability around the personality is um is a big part of it.
What's yours like if you were to describe its traits?
I think it's I I think I just told it to be like pretty direct and like worklike. Um so I mean I don't know. I think I think it's kind of fun. An earlier version I had was pretty sarcastic and humorous, but um I don't know. This incarnation of it I think is just a little more straightforward.
Yeah, it's um yeah, my my guy has he's like the normal default muse um character, but I give him a toga and he has and he just has this like kind of like um extremely deep to the point of being humorous voice and it's just like a fun thing to engage with.
Yeah. I feel like to be funny you have to be really smart. Like I think a lot of people don't get like comedians are some of the smartest people in society. You gotta be fast. Yeah. You gotta be like quickwitted. Um one of the traits you definitely have. Um I'm curious like
I mean you totally do. I like watching all the interviews. You're always great. Uh you and the very nice. I mean it.
Um but okay on that note in the Theo interview you guys talk a lot about like this next frontier of like the technology and what it's all building towards. Um and I think about a lot like with AI there were these like winter seasons where people be like this is never going to happen. Like there's no breakthroughs are coming. Um it feels like the metaverse has had several of those where people kind of think it's a bet that then doesn't come to fruition. I tried out yesterday the new hologram feature.
I know. It's pretty cool.
Yeah. In the Lex interview you guys did, I feel like it was like an 11 hour setup to get your face in there and now it's 3 minutes.
What's coming with that and like what were the breakthroughs that led us to this moment?
Yeah. So, I mean, overall the development of the metaverse, um, it's interesting because when we started Reality Labs, we always thought, you know, you're going to get like normal looking pairs of glasses and over time they're going to be able to do both um, deliver this immersive sense of presence as well as being a great AI device because glasses are the only form factor where you can let the device see what you see and hear what you hear and um, talk to you in your ear throughout the day and eventually display um, images. Y but you know 10 or 15 years ago I just assumed we'd get holograms before we got very advanced AI. But I think like it's this funny thing about the way the tech tree has worked that we actually are just getting AI and like personal super intelligence before we actually get the technology to make holograms ubiquitous and affordable enough for everyone to use. Um, so I wouldn't have predicted that, but I'm glad we're working on both. And the fact that we are doing all this work on glasses has definitely just made it so that we are sort of very well positioned now that um all these AI agents are ready. So a lot of what we're basically announcing at connect is bringing Muse and a lot of AI functionality to the glasses um in a way that I think people are really going to love. So that's that's a big deal. Um, but yeah, some of the stuff around presence, we're still working on it. A little bit slower burn in the background because we've definitely shifted most of our focus to to building all the Muse and AI features for the glasses. But one of the projects that we've had for a while has been um has been the real-time kind of like photorealistic avatars.
And yeah, I mean, like you said, it used to be uh maybe like three or four years ago, you needed like a whole room in order to scan someone from all the angles and then it you could like experience it, but you needed like uh like your own GPU like a like kind of like enterprisegrade setup to be able to do it. So yeah, so we set that up as a demo for the Lex podcast that I did. What was that? Maybe like a couple of years ago. And um and yeah, I mean I think the way that these things work is like first you try to just make it good and then you try to make it more efficient. So now we basically have it working in this pair of VR glasses. Um which is I mean this is the first pair of like the first like amazing VR experience that is in a pair of glasses, not a big headset because I think a lot of people don't want to have a whole headset. And um but yeah, you basically just set up your avatar with like a few photos at this point. It's like but it's pretty amazing how far it's come.
Yeah. And it's also based on the voice. Like it was so interesting in the demo. Um, basically it had me like laugh and interact and then understand how my face would move with uh the audio track. Um, it was interesting in that podcast you talked about how people that maybe are less expressive will actually want to be more expressive in like the virtual world.
How do you think about people like delineating between like their virtual self and then like their in real life self?
Um, I think we're still pretty early in understanding the sociology and psychology around that. I mean, I think people are going to I think that often how people perceive themselves and what they want to project is somewhat different from what they actually are, right? I mean, people I mean, from the beginning of social networks, people have curated profile photos. Um, I think we're seeing a version of this with the avatars with muse.
