Giving Humans Superpowers with AI and AR | Meta CTO Andrew “Boz” Bosworth

Every person superhuman vision, superhuman hearing, superhuman memory, superhuman cognition. That's what I'm talking about. A tremendously equalizing technology. That is the vision that we have for these wearables.

Yeah, people are often surprised by the fact that I grew up on a farm, and my family's farming from way back and still is farming to this day. But actually, if you know about farmers, three things important to know about farmers. Number one: they're governed by time, daylight, and seasons. They have X number of daylight hours to get the work done, and they've got to get it done because the seasons are moving forward. That forces two other things. The first one is every one of them is an engineer. You've got to fix that tractor now because you've got to get that crop done. You've got to mend that fence today. You don't have time to be dealing with getting this livestock back in the pen. So they're all engineers—not scientists, I mean real, like, get-it-done-with-what-you-have-on-hand kind of engineers. And the second one is they're entrepreneurs. My family runs a horse ranch. Well, a horse ranch means you've got a lot of manure. Well, you've got two ways to handle that. One is you can pay someone to pick that manure up and haul it off, and that's going to cost you money. The second one is you can market that manure as a fertilizer and you can make a little money. That's a two-for-one swing. So my cousins who are still farming and my uncles who are still farming, everything's either a cost or an opportunity. The more you can turn things into opportunities—and the margins are slim—so you've got to make those opportunities work. So in my experience, it's not as big a stretch as people thought. But I do want to shout out the 4-H National organization and California State 4-H. I learned how to program in 4-H. It's not just cows and cooking, as we like to say. The first person who taught me to program was a fellow 4-Her, and got me into computers that way. So it's a great program, and I still am involved with it to some degree at the national level. My cousins, my nephews are all still in 4-H AG. So it was a great program for me, not just from an entrepreneurial and engineering standpoint, but also directly programming computers.

Well, it's interesting. I think this point that humans are busy, they've got a lot of things on their plate, and the tool either has to work for them or it's just not worth it. I really think that when we think about how we build these tools, a lot of times we're beset by the tremendous value that we see, especially in our industry. We see how great it could be. And this isn't new. This goes back to Douglas Engelbart, right? Who invented the mouse. To some degree, his vision at SRI failed because he had such a complex idea. He wanted humans to do a ton of work to get skilled enough to unlock the full power of the machine. And my understanding is even in his retirement, late in the 90s, he lamented the fact that we took the easy exit. We took the computer mouse and we ran with it and just did this point-and-click stuff, whereas he wanted to replace the keyboard with the chord set. He had all these ideas. And I think that's a lesson we keep relearning: it has to be so easy. The big pot of value at the end is great, but it still has to be so, so easy to get at it that you can lead people down that path. Because if they have to take a course to learn how to do it, they're not going to do it. It doesn't matter how valuable it is. And I think that's true for—that's nothing to do with anything but just how humans are. But that's true in a wood shop, that's true in an auto shop. The tool that you can pick up and you're like, 'Yep, I see it, I get it, I use it, it works,' that's the tool people reach for time and time again, not the super elaborate complex one. So I think about that all the time, because that was the truth growing up on a farm: you just had to get it done. You didn't have time to be trying to learn a new thing. You had to get it done.

Yeah, this is the construct of the metaverse, which I think has been pretty broadly misunderstood, or at least is understood differently by different people. It's this idea of a blending of the digital and the physical together. Actually, let's keep with the farming thing because it's great. Farmers are pioneering some amazing work. Autonomous driving is not as far along as autonomous farming is. The tractors that are able to plow the fields automatically, drones that are out there doing reconnaissance on what fields need what kind of treatment—that is actually pretty advanced technology, and it really is very much about blending the physical and the digital capabilities together. We had this amazing explosion with the internet and software, and I think we took software as far as it could go within the construct of a phone, a laptop. It's like we just took it to the absolute limit. And what's exciting me now is these really physical manifestations where, through advanced sensors—whether it be video, audio, mechatronics, whether it be drones or robotics, automation, and then ultimately, I think, through wearable devices—we're getting to a new plateau of hardware that can allow software to breathe and expand. AI is such a fun example because it's so vogue right now. If people have had a chance to really use—ignore all the hype and really try to use these tools in their daily lives—there are some areas where it's mind-blowingly useful. I'm doing a little home automation project and I'm debugging, and there's these obscure Internet of Things devices that have unlisted APIs, and it would take me a long time to build a fuzzer to discover those. Man, you can do it in minutes with these tremendously useful AIs. Of course, there's a ton of things that they're not good at, but I still find the interface to them very awkward. I'm going to my phone, or I'm going—whether it's voice or text—it's this very transactional thing. And what's so funny, I find myself, I'm like the cut-and-paste machine suddenly. I'm doing a coding project here, and I'm like, 'Okay, I've got to cut this result, the debugger output, into here, and then it gives me the answer, and I'm cutting and pasting it back into the thing.' And I'm like, 'Why am I—what system of events happened where I'm now the cut-and-paste machine?' This should all be integrated. And we have demos internally on our wearables. With the Ray-Ban Metas that are in market today, through our Early Access program, you can use this tool called Live AI. 30-minute session until the battery runs out because it was kind of bolted on after the fact. But for 30 minutes, it can see what you're seeing, it hears what you're hearing. And the difference in how useful it is to me as I go about my day in the world—I was doing a film development project, and film developing is a real fussy business with how many what kind of chemical at what temperature for how long. So usually I'm doing it with a laptop next to me, and I'm trying to type into the laptop, 'Okay, what's the thing?' Doing it with the Live AI session was incredible. It just sees what you're doing. It's like, 'Okay, you've got 20 more minutes of doing that, and then you're going to do this other thing, and why don't you go ahead and prepare this now?' It's stunning. So for me, it is about blending the physical and digital together in both directions. You have to give the AI access to the physical context in which you operate, and the digital context in which you operate on your desktop or on your phone. And conversely, you want to be able to bring those digital constructs into reality through robotics, through automation. So I see a very, very exciting kind of decade ahead of us in that synthesis.

Yeah, so there's three layers to this in my mind. The first one is we're building these very exciting models, and I have a profound belief—I find myself actually sometimes at war with both sides of the AI discussion—I have a profound belief that these are hugely important, meaningful things that will meaningfully advance human capability. I kind of liken it to a word calculator, right? In the year 1960, a calculator was a person. In the year 1970, it was not a person anymore; it was a different thing. And I think all of us—at first they banned them from school and tried to get rid of them. I grew up with 'You'll never have a calculator with you at all times.' I have three calculators with me at all times. My high school teachers were wrong about that. I think of the AIs that we are building as word calculators. I really mean that. Word, image, visual—they're really complex calculators that have moved beyond the simple symbolic space of mathematics into this higher-order space. I also don't think it is even the kind of thing that is human intelligence, as we understand intelligence and agency and consciousness and thought. And so I find myself having to fight both sides. AI is both a huge deal and also not that kind of a huge deal. So that's my first kind of belief about it, which gives me tremendous confidence in how I use the tool. The second one is we're running into the information-theoretic limits of it. We've talked about throwing compute at it and scaling laws. But if you go all the way back to Norbert Wiener in cybernetics, the first constructs of information theory, this idea of how many bits can we pull out of something that are generalizable, sufficiently generalizable bits. And we're finding out that for all the corpus of human media ever produced, it's not enough. It's not enough. We're finding that in robotics. Robotics is an effort that we picked up recently at Meta, in partnership, kind of as an adjunct to our Llama program. And no matter how many videos you have of somebody grabbing a coffee cup, you're actually not getting the data you need because you don't know the proprioception of how much force is applied and how we detected, 'Okay, this is a plastic cup, it's going to deflect to a certain point, and there's condensation on it, so I need to apply a little bit more force to counter the loss of friction that I'm experiencing.' We do that autonomically. There's not a single conscious thought in our head when we're doing those things. When you're taking your phone out of your pocket, you don't know what the angle of your second digit is or how much force you're applying with your thumb to avoid getting the keys. To some degree, the things that we think of as intelligence—the higher-order functions of the human brain—that's arguably the less impressive part of intelligence. The deep brain, the mammalian, the amygdala, that lizard brain intelligence, that is wildly hard for us to capture in the modern era. So as much as I'm excited about the word calculator, I really do believe in Yann LeCun's vision that you have to do this pioneering work to break through to a world model that has common sense, that understands causality in a more substantial way, not in a statistical soup way, but in a model-based way. He always uses this example, but it's true. My four-year-old daughter—she's now seven—when she was four, could drive a golf cart. And if there was like a turkey in the road, which would happen in Carmel sometimes, she could drive around it. Whereas like, you know, Waymo would just stop. Like, 'I don't know, I don't have a turkey filter. I don't have the data on the turkey. I've got to stop. I give up.' And then my third layer of this stuff is I want it to be embodied. I really think it almost goes back to J.C.R. Licklider, who was one of the first computer scientists to sit down at a terminal and do live programming. He believed in that vision, and he really was the one who then funded what would become SRI, found Doug Engelbart, funded a bunch of DARPA. I feel like we're in that era. We're in the terminal era of AI, and it doesn't want to be like that. It wants to be everywhere. It wants to be ubiquitous. It wants to be in full context of what your life is, and with history of who you are and what your life is. I think about Douglas Hofstadter's book, I Am a Strange Loop, and how we all have little mini versions of each other's consciousness simulations running in our brains that allow us to collaborate effectively. And my AI obviously doesn't have that. It has no idea what I want, what I'm about. It can't infer anything from context. So for me, I love where we are. I'm a huge believer. I also want us to invest in these world models, and I want to free it from the terminal.

Yeah, I think the world model is a new, different thing. And as a consequence, I think it's an unknowable timeline. I'm reminded a little bit, when I think about the world model, of where we were with machine learning. When I was an undergraduate, I graduated Harvard in 2004, and I taught a class, Introduction to Artificial Intelligence. It's actually where I met Mark; he was a student of mine in that class. We taught that neural networks were a once-promising, now-known-to-be-dead technology. And everyone had to build the handwriting recognition neural network, and it worked, and they're like, 'Yep, that's all you can use it for. That's it. Congratulations, here's your degree.' Okay, and now neural networks run the world. And Yann was right, and Geoffrey Hinton and Bengio, they were right, and they all won the Turing Award, God bless them. And it took a series of other unlocks—GPUs in particular—to get to the point where that technology could flourish. So I wouldn't bet against Yann LeCun over a long enough timeline, but I don't know what the unlock is yet on that piece. I think the embodied part can do both. The embodied part will benefit a lot and maybe help a lot with world modeling. Once you have these sensors out there and you have better, richer data, when you have robotics data which will give you proprioception, which will give you friction—I think that is going to be a big unlock. So I think the embodied one is probably—I sequenced it wrong—the embodied one is probably the in-between. It benefits the current models a huge amount to get that data and to be in that context, and also it probably is some of the data that you need to start to understand what it takes to build a world model, which we appear to be born with.

Yeah, so we've got the Orion glasses on right now. I've also got this wristband, this neural interface wristband. It's got these little metal bumps on the back of it all down the band, and those are EMG sensors—electromyography. They're measuring electrical impulses going down my hand. And what I'm able to do—and you won't be able to see anything hopefully if I'm doing this right—is if you're looking straight at me, I've got a screen in front of me. I have an internal part so I can do my email, I can do Instagram—I'm not doing that right now, I promise. We have little games that you can play. I have been caught playing the games in meetings before. They weren't my meetings to be fair, I was just listening in, but I have been caught. And I can do it. I could use my hands—it's got cameras, so I can use my hands forward-facing, that's a little awkward. So I can do with my wrist at rest by using a small number of gestures. This is the home gesture—oh, I've brought myself home. And this is the select gesture. I'm using eye tracking to try to direct things. So what we have to do is a lot of tough problem solving on photonics and optics. Some of the stuff we understand—we understand how to build apps, we understand how to do these things. And there's some novel interaction design here with the wristband and eye tracking, but I think we've made that pretty straightforward. Doing the neural interfaces was super hard. You're building an AI model of what the hand is doing based on these electrical impulses that you're able to observe from the surface. So you just need a lot of people to build a generalized model that works so that anybody can put this on. And I think we've had success with it. I think we're well into the 95th percentile of people who put this on and can right away—we know what shape the hand is in, and that allows us to do these gesture-based controls even with your hands in your pocket, behind your back. Subtle wrist swipe gesture—swiping up and down, left and right, tapping. This works. You can do a double tap to get your AI. So that was a lot of work. The hardest one was the optics and photonics. Inside of this, there's a tiny projector. It's using micro LEDs. We build these micro LEDs ourselves, we manufacture them ourselves. Three of them would fit inside of a red blood cell, and they're individually brighter than the sun. And they have to be incredibly efficient because you do not have a lot of battery on the headset. And actually, worse than battery, you don't have a lot of thermal space. You just can't draw a lot of heat because we can't burn your face. Burning faces is terribly bad, I'm told. So we have a tight thermal window. It had to be super, super efficient in how they generate things. Some of these things are really hard. Red wavelengths are really hard. Red is a really long wavelength of light; it's hard to generate in a very, very small space. You have to use a lot of mirrors and reflections and complex geometry. You're making these at absolutely tiny scale for consumer electronics. So you're generating these photons, then you have to bend the photons into the light. We use something called a waveguide, and waveguides use the construct of total internal reflection of light, the same way fiber optics work, to try to bend light in a certain place. But of course, there's a couple other problems when you have glasses that aren't fiber optics which are encased in a cover, which is that other photons enter from the world and get caught in those same light pipes. And that can create rainbows and haze and these kind of artifacts that are distracting to you. And also, some of those photons escape, so people outside can sometimes see them. In this case, we have a kind of an exit grating. So if I put my face at just the right angle, you can see a little blue light escaping from the gratings. So you want to minimize the artifacts because these need to be real glasses first and foremost. When they're powered off, I have to be able to see your eyes, to see my eyes. That human connection that we have is important; otherwise, I wouldn't use the glasses. You have to do all of this in a comfortable, all-day wearable form factor. There are a lot of challenges. This was 10 years in the making, and we thought when we started this program that we had less than 10% chance of being able to build it. So the fact that it exists is a true testament, I think, to a vision that Mark Zuckerberg had, that the research team—Michael Abrash, our chief scientist, and his team—had for a long, long period of time. We're doing everything from material sciences, developing new chemicals, new forms of glass. The photonics work was cutting edge and it was super hard. Now, we always thought, to your point about the metaverse, that the first thing would happen would be these holograms in the world. And at first, there'd be just a reference to you, which is your personal interface. And then over time, they would be attached in the world. And then eventually, you would have an AI that was doing it. What's been surprising to us is the AI came first. The AI moved up in the—we always had this vision, it was a vision that Mark and I had from day one, that AI and AR were the things, which is why we had this big investment in Facebook AI research (FAIR) and this big investment in Reality Labs. We had the sequencing wrong. The AI showed up earlier than expected. So what's been so exciting is, just like I mentioned earlier, on the Ray-Ban Metas, there are Live AI sessions. There are a lot of products now that are totally valid products that are easy to use. Between full AR glasses, which are spectacular but will be expensive, and the Ray-Ban Metas, which are super affordable but a little bit more limited in their functionality, that entire spectrum is now open to us. And it's really, really exciting.

Yes. These are binocular, so obviously you'll experience the same limited depth perception that you would experience normally. Although depth perception isn't primarily based on binocularity; it turns out it's based on movement of things through space and size. But anyways, I think so. You'll still be able to use these. Over the future of time, I do expect one of the real option spaces here is what if you go to monocular? What if you just had the display in one eye? There's some challenges there; it creates some binocular rivalry. So for people who have full vision in both eyes, they might struggle sometimes to know that their eyes are seeing different things looking at the same space. You would actually be better off in those displays. Can't wait. It would be a true benefit, a superpower. You would have a little advantage. The monocular displays would probably be great for you—cheaper, lighter, and probably just as good. So I think you've got a little inside opportunity on the rest of us.

Yeah, so there's two parts to this answer. The first one is, with News Feed and products like News Feed, as much as people were surprised by it and outraged by it, the usage was immediate. People often ask us, 'How did you guys decide to stick with the News Feed given the backlash?' It wasn't actually hard because internally we saw all of our users numbers doubled overnight, and they never went back down. Likewise, News Feed was a product built—there's a famous idea in design, which is, I'll use the example that is most often given: in a park, a natural place in a city, people will tell you where the paths should be by cutting paths in the grass. So wherever you see the grass is trampled down, you put a path there. That was the kind of innovation that News Feed was. We watched the behavior of people on the site and just built a path that made what they were doing easier. Things like AI are different kinds of things. These are different types of innovations. These are truly new tools. This is a new space. This is a building that's been constructed, and you can't just figure out where the paths should go later on. So these are different types of innovations. They do come from the same place, which is my alpha and omega every day. My team will tell you: 'What human on this Earth is better off because they have this?' And that involves thinking about what are they doing instead of it. What is their current solution to the problem they have? What is the problem? Is it a big problem? By the way, there's two types of big problems: there's big problems like rare but super big, and there's also super common but small. Those are both collectively big problems that we can go solve. But I'm like, 'Find me the human whose life is just, oh my God, way better.' AI has been uneven at this. The answer for AI so far has been homework. Kids doing homework and getting help with homework has been by far the number one use case for AI. Number two has been coders who used to go to Stack Overflow. Those are pretty close to the park path story because these are people who were previously using Google or online resources and are now using a consolidated form of that. But the smartphone was a kind of a disruptive thing where it was like, 'Hey, you have a phone and you have an iPod and you have the web. We're going to put all those in one place.' That was the pitch. This will be similar to that. This will be like, 'Hey, look, you already have a phone that you like doing this, you already like Instagram. This is a better way of doing a thing that you already do.' And once you've established that beachhead through comfortable paths, then the huge opportunity presents itself. You talked about having superpowers. That is the vision that we have for these wearables. Every person: superhuman vision, superhuman hearing, superhuman memory, superhuman cognition. That's what I'm talking about. That's what I really believe is going to happen here. A tremendously equalizing technology. Today, we talk often in our society about the vagaries of the birth lottery, and we're rightly doing so. We usually talk about the birth lottery in the context of race and gender or physical geography, and those are hugely important factors in how our lives play out. But we don't talk about the other parts of the birth lottery: who has incredible memory, incredible vision, incredible hearing. Some people are born that just have those talents, incredible ability to think creatively, to pattern match. There's no reason we couldn't all be Garry Kasparov at chess if we have the wearables. It probably takes the fun out of chess, so I'm not recommending anti-competitive performance. But my point is, the vagaries of the bell curve and the ranges of human capabilities are beyond just the demographic ones. They exist in terms of fundamental capabilities. How interesting will we be as a society if everyone has the full access to those facilities? That is the future I see. And I think that's a pretty compelling pitch to people to get them to try a new thing. But it still has to do Instagram exactly.

Yeah, I think what's interesting, the hard part of this question is the timeline of it. I think you and I probably would do better in one year and 10 years. Three years is the most awkward time because I know one year, I have a good sense of. 10 years, three years is tough. I don't think we'll be at the proactive place in three years quite yet. I think early adopters will probably have always-on systems that are capable of it. The degree to which I think it's going to be reliable enough for the average consumer, I think we're probably a little further out from that from a world modeling and cognition standpoint. But for a decent portion of people—tens of millions—we're going to have people who are in regular conversation with AIs in casual, comfortable conversation. Everything from, 'I cook all the time, I'm the cook in my family. Hey, how many quarts is it in a gallon? What's the conversion, tablespoons to ounces of water?' To, 'Hey, what did my wife say I need to get at the grocery store? Remind me, she told me yesterday.' 'Hey, when I opened the refrigerator, did we have that cheese that I like?' And there's stuff that I really want to push for, but we have to advocate for regulatorily. I think one of the classic ones is the cocktail party problem. You see somebody coming up to you, you recognize them, you know them, you don't remember why you know them or how you know them. And right now, we're just like, 'Hey, guy, good to see you, friend.' And you're just looking for clues to try to remember. I'd love to be able to have your AI whisper to you, 'Hey, you know this is who this is. The last time you saw them was here.' And you're like, 'Oh, right, right, right.' But that one we need help with. There are regulations in Illinois and Texas, BPPA and QB, that make that kind of thing tenuous at best if it's doable. So there's a bunch of very human problems where we could be a little bit more proactive, and I think comfortably so, in a totally privacy-secure way. We've got to do work. So three years from now, I do think you'll have people at the leading edge—not the bleeding edge, just the leading edge, the early adopters—who will be having tremendous usefulness in cognition and memory help from their assistants.

No, I think they want to work together. We probably are more than 10 years away from having the efficiency of compute from a thermal perspective to have the glasses stand alone. And if they're going to have another thing with them, today Orion comes with this kind of—we call it a stage or a puck—a little battery that comes with it. You toss it in your bag, you toss it in your pocket, you don't take it out. And that's totally fine, actually. It works pretty well for that. And it's nice enough to take it out. One of the problems that is this universal problem that doesn't sound like a problem, but actually I think weirdly is, is taking your phone out of your pocket. It seems like it's literally no effort at all, and yet what I found when I'm wearing the Orion is I will do things that I would not have taken my phone in my pocket to do. But because it's right there, I can do it. It's interesting how much that friction actually creates once you get into it. It sounds totally insane—we are the most entitled species, I accept it—but it is the reality that there are things that you'd use. So I think for the foreseeable future, you do want this to be a constellation of devices. And there's a lot of value in being on the wrist. The neural interfaces we've developed here show that there's a tremendous amount of additional incremental signal and control we can give consumers without having to hijack their eyes or make them reach up and tap something on their temple arm. I think there's also a huge opportunity for the range of glasses. So if you have glasses that have no display, you want to pair them with a watch that has a display or with a phone that has a display. If your glasses do have a display, okay, now maybe you just have a simpler band on your wrist and you're wearing a conventional wristwatch on your other wrist. We're in this space where something that we actually haven't had to grapple with in industry for a while is the luxury presentation, the identity that people want to bring to the world about themselves. They want to look a certain way. So if I'm somebody who wants to look a certain way, I have to have the options to maintain that vision of myself while also being a part of the modern era. It's funny, there's been a lot of discussion about the app stores on Google and Apple devices. I really don't have a problem with those. That's not the issue I see. What worries me more is the degree to which these devices, which are the natural center of compute for a constellation of wearables, are locking down the access to third-party wearables manufacturers to things like critical Bluetooth channels. The famous example that I'll use is the AirPods. You can tell by my headphones, I'm a bit of an audio guy. I care about the quality of my audio. There are better headphones that you can get than the ones manufactured by Apple. AirPods are great, it's a great device. They have 70% of the market share, and they don't have that because they have the best product, certainly not the best value product. They have it because they have a proprietary Bluetooth channel that makes it super easy to pair. And also because I think they're easy to lose, you have to buy a lot more. But it really bothers me. That shouldn't be that way. I'm a 90s guy, so I'm an old school computer construct guy. I'm not maybe full all the way to Tim Sweeney at Epic, but I'm somewhere in between the two. So that's the stuff that worries me: that your phone should be more useful to you than it is, and there's no reason they couldn't do that except that they want to keep you in the ecosystem.

Fair enough. Every now and then, I've caught myself looking at my phone when I have the Orion on, and there's literally a permanent clock in view of my face. And I'll still do it. So it's not just you. Okay, good. Thank you, I appreciate that.

Yeah, there's quite a few, but I'll tell you one that surprised me recently which makes total sense in retrospect. One of the most popular demographics purchasing our Ray-Ban Meta glasses are blind people. Now, blind people buying glasses is a surprise. But if you watch them in the user research sessions, it makes total sense. We have members of the team who are blind. We have a partnership with Be My Eyes, which is a great service. But in my head as a seeing person, I'm like, 'Oh, I wonder if they have problems navigating to the restaurant or getting across the street.' They don't. They have solutions for that. They've got Google Maps reading directions into their ears. They've got a stick or a dog. There's a bunch of systems they have. They can get to the restaurant. You know what they can't do? They can't find the door. Where's the door? So what they do is they ask the glasses, 'Hey Meta, look and tell me where the door is.' And it's like, 'Oh, okay, it's to your left.' And if they can't get that kind of thing done, they can call into Be My Eyes, and now they've got a live video stream going to an AI agent at first, and if it fails over to a human who helps them out. So there are these moments where you know it's going to happen because you yourself are a human and you're like, 'Yes, I as a human would want this, so other humans would want this.' But there are also these really fascinating times where you build the capability and this stuff comes out of the woodwork that you never saw coming.