One of the reasons it is such a chaotic time in the industry not just at Meta but across every company and there's a tremendous amount of unease is like this is the least clear vision we've had of even the relatively near future in a long time. I think two years ago we started to get a sense of three years ago we were excited, two years ago we started to get a sense like wow this is bigger than we thought. I think last year it got to a point of seriousness that it became an almost existential conversation for a lot of us.
Hello, I'm Nicholas Thompson and this is the most interesting thing in AI, a podcast produced by Rethink, the Atlantic's creative marketing studio. If you're above a certain age, you might remember a very different version of Facebook. It was sort of a directory of your friends. You couldn't do much. You would just comment and like and there's a thing called poke. But in 2006, Facebook did something controversial. They added newsfeed. Suddenly, everyone's Facebook interactions were blasted out to the whole network. Users said it violated their privacy. May have been the company's first privacy scandal. Facebook briefly apologized and then just pressed forward. You know what? Became one of the most important successful products that Facebook ever made. The man behind it was a guy named Andrew Bosworth known as Bos. He's been a very early Facebook employee and he has been there ever since. Often he's in the middle of the storm pushing things forward that users are upset about, but that end up being really core to the company's business. Part of the decision to rename Facebook Meta was a charge of their reality labs, their VR glasses, their AR glasses, their current glasses that some people love, some people say are introducing mass spyware. He's partly in charge of their AI operations, which of course is a massive, massive bet for the company. Also a little controversy there when Facebook said it was going to record the keystrokes of their employees in order to make their AI better. They then rolled that back amid employee outrage, but who knows what's going to come next. So, here's my conversation with Andrew Bosworth. Andrew Bosworth, welcome to the most interesting thing in AI.
You're here for the glasses. Explain where we are in the development of the glasses because in order to have them become a mass product and to displace the phone, you need a big developer ecosystem. But to get a big developer ecosystem, you need mass adoption of the glasses. To get mass adoption of glasses, you need a big developer ecosystem. Where are you in this flywheel?
Yeah, we're making great progress. I tell you that the glasses don't need a developer ecosystem to be great. We knew that was going to be the case when we launched them is that these have to be great on their own terms for people to want to use them. And actually, what's funny is even without the AI features, which I think are such an exciting part of them, they're great glasses. People are using them all the time. They're using them for the audio. They're using them for the video and the hands-free recording. Very popular with parents with young children so they can capture that moment, share it, but not be separate from it from their phone. Very popular with athletes who are going on long runs or going on rides and want to be able to do hands-free capture or have audio going at the same time. So the glasses are selling very well on the merits. We didn't even conceive of AI as it exists today when we built the glasses that we have. Now that we have AI, they're even better. So they're even more capable of helping people. And then with the display glasses, we just launched a vibe coding platform for the displays. And we're seeing tremendous enthusiasm from developers and because it's not about taking every app in the world and bringing it to the glasses. It's about what is the thing that you need for your glasses specifically and you can go get that. I do think where we're headed over time is away from this concept of apps or even static development. I think we're headed towards very dynamic AI generated applications where you just ask your AI on your glasses or wherever for what you need and it generates it on the fly and delivers it to you. And all the service providers get paid for the value they're creating. It's all negotiated on your behalf behind the scenes, but you get what you need when you need it.
My favorite use case for the glasses is I put them on and I just talk to them like talk to AI like I'm making breakfast and I want to learn about something. Today I was asking this morning while I was making my kids breakfast about the components in Muse Spark just so I would be a little more up to speed on that. The killer app for me and I think for the world would be if you could adjust vision with them, right? And if you could enter your prescription or if you could turn a knob and when will that be possible?
Yeah. So those are called liquid lenses. They exist in a lot of different formats. So right now I say we want to clear the glasses can correct your vision. Right now
You just get prescription lenses.
Right now they have prescription glasses.
Take these, right? And I'll like, take these lenses out, put them in here, and then I can see you perfectly. Right. But
So they certainly do prescription lenses. That's right.
Yeah. So I want to be clear that the glasses do handle prescription lenses. You're asking like when can you have autotuned liquid lenses
Or software that adjusts it like whatever the solution is because I feel like this is the killer app for these things.
Software adjustment is really really hard. There's not a technology path for that. But the technology path that probably does exist is you can have lenses that are two membranes that deform under pressure, under tension, that have some kind of a liquid inside and that's quite controllable on either one or two axes. That is a possibility. I think that technology exists. The main reason we don't do it is that those are a little more fragile and the robustness of getting prescription lenses and getting those distributed to people has gotten quite a lot better over time.
The second app that is most controversial and harder is face recognition, right? It'll obviously be supremely useful, but obviously it weirds people out.
Controversial. Where are you on that?
We just donated 180,000 pairs of Ray-Ban Metas to every blind veteran in America. Why'd we do that? The glasses are incredibly popular with the blind community.
You had a blind designer on them early, right? Wasn't one of the product leads was blind here.
Yeah. And actually I asked her, you know, why is it important to blind people to have these glasses? Why are they helping blind people so much? And she's like, you know what? People get blindness wrong. I can get to the restaurant. I've got Google Maps. I've got hearables. I've got a cane. I've got a dog. I can get to the restaurant. She's like, you know what I can't do? Can't find the door. Google Maps told me I'm at the restaurant. Cool. Where's the door? I can't read the menu. So they lose independence. So they call services like Be My Eyes. Mike Buckley, CEO, former Meta employee. And they talk to somebody who's looking at the camera and tells them, 'Okay, the door is to your left. Okay, there it is. You got it.' Now they can still do that with a partnership that we have with glasses, but they can also just ask the AI, and the AI can answer it for them. So they're gaining independence. If you ask the blind community what the number one request that we get from them is, it's 'Who's in the room with me? Tell me who's here.' They're not asking for a global central database of faces. They're like, 'People I know, like what people that I know are here?' It's funny. I was at Oscar Mike last week. They came to campus. This is a veterans group that deals with people who had traumatic brain injuries. So they often struggle with memory. They struggle with cognition. And this came up and this is again a top ask for them. Hey, like I just want to know who I'm talking to and like what how do I know them and what's my history with this person? They're not asking for a central database of people. They're asking for like this is someone I know.
Wait, so the idea the program that has been written about is called name tag. The idea of that is not 'here's a person you haven't seen before. Here's who they are. Here's their LinkedIn.' It's that this is a person you've met or who's in your Facebook graph
Encrypted locally to your device. Somebody that you met in person with your glasses on who introduced themselves or you said, 'Okay, this is David. Remember this person.' Only available to you when you're wearing your glasses. This is a person you've met before. Here's their name. They're right in front of you. That's what we call a name tags feature.
And it's so I think people get confused because they hear face recognition and they think, 'Oh, there's a central face database that everyone's being scanned constantly into.' It's not that. But you're right. It's still controversial. I think it's one of those things that you can't even do in some states based on the state laws. We've got BIPA in Illinois and QB in Texas and these things. So it's a thing that I think it would be a great feature. By the way, I've told the story through the blind low vision community or told the story through Oscar Mike. You'd use it. I'd use it. I go to parties where I definitely have met somebody before. You're a journalist. You must meet people constantly and constantly in your head be like, I know this person. Name, face, what was the last time we talked? What's the context here? It's what we call the cocktail party problem. It's a very universally human problem. Nearly every human I talk to no matter what their status of their vision or their memory is, understands that this is a problem that would help them feel more comfortable in social situations. If they had it, but it has to be done in a way that people feel comfortable.
Right? And it's definitely the thing that creeps, no question, everybody can see the utility. Like if you've been to a high school reunion, my lord, you could use the utility, but also quite clearly creeps people out. The advantage that you have as Meta is that everybody's using it for consumer use cases. You're able to feed that back in. Explain how you're training on your own employees. This has obviously been controversial recently with the model capability initiative. What are you trying to learn from your employees? As has been reported in the press, you're tracking what they type in, learning from their intelligence. I believe it was Zuckerberg said, 'Our employees are more intelligent than the average person. We want to see how they use machines so that we can make our AI better.' Explain that.
Yeah. Although that program is now paused. So
The whole program is paused just for now because of the data issues.
Well, we're trying to make sure we have a full accounting for the program. Let me explain the theory of it. The theory is actually very straightforward. You've got a bunch of employees that are already doing this work every day. And for no incremental effort on their part, you can potentially improve your models.
Yep. And that benefits not only your employees when they go back and try to do their jobs, but also consumers, people who want to use AI.
Now, the scope of this research, and it really was research, it's worth noting this is research that we didn't expect to yield for like 18 months, at the earliest would be the earliest that you would actually have a look at what this would look like inside of a model. It needs very long-running data. It was literally things as simple as how to use a computer, a thing that AIs are surprisingly bad at for being themselves computers. Like literally how do you navigate UIs? Like what do you click next? And a lot of our workflows are in these digital silos. They're going to be there for a long time. A year, a year and a half from now, it's still going to be the case that the flows that we're going through every day as we go about our work. So the most basic premise was like, hey, can we teach AI to use computers? It's a data set that doesn't really exist at scale. And you can do acute versions of this. You can certainly hire people through Scale AI or other places to give me a quick walkthrough, but we wanted to see over the course of months how do people make progress on work literally using a computer. And so that was the premise of this research.
What were you recording and tracking them?
Yeah, it was literally like your mouse movements, what are you clicking, where are you, what window are you activating, and try to create enough of a data industry here.
And was it all employees? Like were you tracking yourself?
Uh in the US though there was a few employees excluded. So a good example is if you look at someone like me where in a position that I'm in, we're under any number of legal holds.
Where it's like hey any information that's created on my behalf is automatically stored and preserved by law.
That's not that makes me like a relatively poor target for this work. So there's a bunch of jobs people who regularly worked on sensitive data that was like hey you're excluded. So we excluded a bunch of teams, we excluded individuals, there's a whole process by which people can opt out of this stuff.
No it would make for an amazing data set. How do executives at companies dealing with like legal holds handle computers?
That's a proprietary data set that you can imagine having value.
Yeah. Taking a step back, I think we're at a very interesting place where if you saw the trial, the opening AI trial with Greg Brockman's writing a note to himself as a diary and that's being exposed. And they've now I think there is at least one legal precedent that shows, hey, like asking questions to an LLM is not you can't have that be protected even if you're asking a legal question, right? I do think there's an interesting question that we need to raise at some point relatively soon on privilege and like what should be privileged and not privileged in a world where you can credibly get a consumer somebody who couldn't even afford a lawyer necessarily could get credible legal advice. Set aside, nobody cares about corporations, they got enough money to hire all the lawyers. But somebody who can't afford a lawyer but could actually get expert legal advice from an AI given that the premise of our system is that should have legal representation. What does it look like? So I think there's an interesting question but I'll leave that for the courts.
We'll leave that for the court legislators. All right, let's go back. So you and other people who work on sensitive information are excluded from this program but the majority of Meta employees are?
Right. You like Europe would never let you do this, right?
Didn't even ask. Right. Okay. So majority of them are included. You're kind of keeping track of mouse movements. What did you learn? How valuable was it and why did you stop?
Yeah, so we took a look at the data a couple of weeks in and there was a lot more duplication than we expected. It's like one of the things interesting about this is when you think about high volumes of data you're usually talking about pre-training and it's like when you're talking about exquisite data where you've got a really detailed expert explanation of something that's a lot of times that's mid-training or post-training. So these data were meant to be long-running but you need variance. Variance is far more important than volume. Like a high volume of the same thing that gets collapsed into one example basically. And so one of the first things we saw was actually there was a lot less variance than we expected. So that was why, I think a couple weeks after we initially launched it, we added okay um expanded opt-outs for people who didn't want to do it. A pause like infinite pause, like whenever you don't want to have it just press pause. You can do it as many times as you want. So we added a bunch of things. We're like, 'Okay, there's a lot of duplication here.' So we probably overshot needing as much as we got. And then we went through that process of like, okay, having looked at that and reduced the variance, now we've added these tools. And then this last week, we had an issue where the data that we were storing was quite secure, relatively small number of people had access. One of the researchers who was working downstream with that data. And nothing, there was no breach here, but like had put it in a place it wasn't supposed to go. So we're kind of locking the whole thing down until we can like
The employee, there's like
These employees used these keystrokes and moved their hands in these ways.
Not the raw data. It was a transform on top of it, but it still landed someplace it shouldn't have landed internally. Internal only. No reason to believe foul play. We're looking at every single access kind of things. And so we've just paused the entire program until we can get to the bottom of this. While we're doing that, we're actually going to run an early simulation of like, hey, what would it look like to do a train just with this small data set that we have to understand the long-term value of the data? Because this was a research project. This wasn't like, hey, we're guaranteed this is going to be the game changer.
Why did you do it on your own employees as opposed to hiring 10,000 task rabbits?
Yeah. The main thing is what we wanted were these long traces. You want like months-long every day work. And I think if you really, it's very important that the data actually look like what real work looks like over a long period of time in a professional kind of environment. And so I think it's a pretty appealing data set, a unique data set. I think one of the long-term concepts that you have to have with AI is that data is the font of all value. All the tremendous wizardry involved in transforming it into these models is almost magic, right? But if you don't have the data set up front, the chance that it's going to be in the distribution of the model in the back end is low. And so I think one thing I expect to happen in AI as we continue to fight along these curves, these scaling laws, is an increased demand for these long-tail data sets. You know, we think of all the corpus of human media on the internet as being surely this is all of human knowledge, but it's not even close. I don't think it even scratches the surface.
I think all the time like
You've got somebody who's working on Sarbanes-Oxley compliance. There's a tremendous amount of just judgment that goes into each of those things. You can read the statutes and get it wrong. Because you're not plugged into how people in the industry are accounting for things and how the regulators are thinking and what the real concerns are. And so the long tail is very long.
So what's the dream data set that you would like to have access to that you don't have access to?
Well, I think all data sets that represent real human value creation are interesting. And these are some that are probably impossible to get. What's the creative process somebody goes through when they're creating a story that's an amazing story? There's just not that many amazing stories out there, let alone the people to monitor while they're doing it. And if you ask those people what their process was, there's a good chance they don't even know what their process was, you know, like I was in the shower, I had this idea, I followed it. So AI is a data hungry vehicle and the more unique data you can feed it, the better it's able to help you on the other side, and the more useful of a tool it is.
One of the reasons why your program was uncomfortable to employees was they were like, 'Hold on a second. Is this being used to replace me? Is this going to make it easier to wipe me out?' And like, where do you think the line should be drawn around the data that we create that really is just ours and shouldn't go into AI?
Yeah, I'm personally somebody who thinks this is a tool that's mostly about maximizing each individual person. Like when I think about what AI does for me today, it's like doing tasks that I otherwise would have done and capable of doing, but I just don't think are the best use of my time, right? I think every time we find one of these productivity improvements going back quite a long ways in human history, we worry it's like, oh no, the thing that it means to be human is going away. And like every time, no, it's not. It turns out that wasn't what it meant to be human. It wasn't the case that we needed to stitch the fabric manually for it to be a valuable procession to design garments for humans. We keep finding meaning in that work and that's good for us, but that is not the exclusive domain of human meaning. It turns out we apply meaning to whatever we put our energy into. And so I think of AI as individuals unbounded.
