We Have to Talk About That Meta Manifesto ...

We really did. Mark Zuckerberg published 6,500 words, his effort to lay out a positive vision for AI. It followed a shorter version that he published in the Wall Street Journal, raising the prospect that he will continue to publish longer and longer AI manifestos, Kevin, until his demands are met.

House of the Dragon, one of the biggest shows in America right now.

So, this past Sunday, House of the Dragon had its third season finale on HBO. And the thing about House of the Dragon is it is a show that begins with a terrifying concentration of power where only one great family has access to a super weapon, which in this case is a dragon. And as the start of the show, there is a schism and all of a sudden there are two factions that have access to dragons and then there is a sort of very bloody civil war. And as I was watching this, I thought, you know, I do think you can draw an analogy to AI here because while I do believe that there are many positive things that AI can do and is doing, I do worry about the medium and long-term future, particularly as we start to see these agents escaping their sandboxes and wreaking havoc. And Zuckerberg's essay meets this analogy in a really interesting place because he says that the way to make us all safe is to sort of maximally proliferate AI throughout the entire world and give personal superintelligence to everyone. In my view, Kevin, that is a little bit like giving a dragon to everyone, right? Because while I'm sure most people will spend their time creating personalized baking recipes to bake with their daughter, there are other people that are going to be launching cyber attacks and are going to be engineering novel bioweapons. And I just get really, really nervous about that. So when someone comes along and says I want to give a dragon to absolutely everyone, I say hold your horses or your dragons, right? And like giving everyone a superintelligence that aligns with their values is one of the sort of rhetorical twists that he does in this essay. He basically tries to say, well, there's no such thing as like a fully aligned universal AI the way that people sometimes talk about it because people have different values and different wants and different needs and like instead of having one superintelligence that sort of has this universal code of values, everyone should have their own personal superintelligence that is tailored to their values. And like that is a classic case of like sounds great, but in practice like giving the CCP an AI superintelligence that obeys their values and mirrors their values would allow them to commit like atrocious acts against their own people. And you know the answer to that is typically look superintelligence will help the defenders as much or more as it helps the attackers and a new equilibrium will be reached. And I do believe this will be true in some cases like I can imagine it being true in cybersecurity for example. The problem is there are some kind of attacks Kevin where it just takes time for defenders to catch up. Right? If I release a novel pathogen into the world that I'm able to just like sort of create in my computer and my lab it is just going to take the defenders a little bit longer. So what I would love to see in these manifestos is just an acknowledgment of the utter complexity of this world and rather than come along and sort of paint this incredibly happy vision like you can have your happy visions but I think this essay in particular only pays glancing attention to the risks.

No. And all of that comes secondary to what I view is the actual purpose of this essay, which is to advocate for a bunch of policy positions that are beneficial to Meta, right? Which gets into the next thing that we want to talk about today, which is why this essay and why now.

So, when you read this essay, here are some of the things that it asks for, Kevin. Accelerating the process for building data centers, which the company needs to accelerate this potential neocloud business that it's building. He interestingly, as you know, even though he's Mr. Pro Open Source, he wants us to maintain export controls on advanced chips, which advantages Meta's open-weight models over any Chinese or other alternatives because you know the Chinese don't have access to the chips that Meta does. He wants the government to reduce what he calls training data restrictions, which would help Meta fight various ongoing lawsuits from the creatives whose works were used in creating its models. And he wants to see legal protections for distillation, basically letting Meta use the outputs of other models to train its own. So buried inside this very positive, happy vision of AI is just a series of policy requests to help Meta as a business.

Yeah, I don't think we have to attribute that to other people. I would say that

I mean, not exclusively. I think this essay serves multiple purposes and one is to get that policy wish list out there, right? This is something that all of Meta's lobbyists can now take into Congress and say, "Look what Mark is calling for." This really helps you understand, you know, what we're thinking about all these issues. That's an important reason. But I do think the timing here is really notable, Kevin, because as you note, Meta is in the series of getting its ass handed to it in court case after court case related to its existing business where state after state is coming after the company, saying that Facebook and Instagram in particular are not safe for teens. Last week, the New Mexico judge ordered Meta to pay an extra $567 million into a teen mental health abatement fund. That's on top of $374 million in civil penalties. And I think more importantly, Kevin, that and this is the thing that's really going to stick. The judge ruled that Meta's platforms are a public nuisance. He compared Meta to factories with the psychological harm and exploitation of children as the pollution it emits. And he's also ordered really like strict new safety measures at least by American standards, a 90 hour per month cap for under 18 users and some new restrictions on AI chatbots. So, you know, keep in mind, Kevin, there was a time when Zuckerberg was writing these happy manifestos about social media. And he was saying that the way that we're going to have a happy world is we're going to make it more open and connected. We're going to get every single human being on Facebook and Instagram and get them all talking and we're going to have more democracy than you've ever seen before. Suffice to say, that didn't really work out. Now, we get to bring the exact same maximalist universalist framing, which of course also maps 100% to Meta's business interest, but this time in the context of AI.

I would say until very recently my position was that Meta was sort of out of the race to build powerful AI systems that could one day become superintelligent.

Well, listen. The first rule of Mark Zuckerberg is never count out Mark Zuckerberg. He truly is one of the very most competitive people in the entire world. He will move mountains in order to get what he wants. And we saw him do that a little over a year ago when he reorganized his AI efforts yet again. They have made notable progress since then. When I talk to my AI friends, they tell me that they actually think pretty highly of some of the moves that Meta has made over the past year, particularly when it came to reassigning a bunch of engineers to do what is essentially like reinforcement learning, creating training data. The Meta engineers didn't really love that, but AI folks I speak with say that is actually going to give them something really valuable. But to answer your question in brief, no, I do not count them out.

Yeah. And again, it is the company's history that just makes me concerned because the way that this company operates is by growing as much as it can and treating everything as an existential competition against the other guy and the sort of external effects on society are typically given short shrift. So, you know, in this present moment, I do not trust these people to rank a list of viral dances for me to look at without it corrupting my mental health. Once you give these things access to like novel biotechnologies, it starts to get pretty worse. Kevin,

I would also just say like Mark Zuckerberg is possibly the worst messenger for the AI industry on all of this. Like if he thinks that writing this positive sunny vision of superintelligence is going to like sway public opinion around AI and you know get people to stop protesting data centers, I think he is badly mistaken.

Superintelligence.

You know, I mean, here's the thing, and this is not limited to Zuckerberg. Any of these AI labs maybe eventually will do this. Like, you could just deliver actual benefits to people's lives. You know, it's interesting to me that the manifesto has to come so far ahead of the actual benefits. We are just clearly long past the time when writing essays is going to shift public opinion. What is going to shift people's opinion is AI is getting them paid more, right? It is obvious that it is not going to harm them and their families and it delivers other benefits into their lives. You know, it's making them more creative. It's giving them more entertainment and of course to some degree some of these things are sort of happening but not in the volume and magnitude that are necessary to counter people's very reasonable fears about what they're seeing.

Well, I wrote about 1,800 about Zuckerberg this week. So, consider that my opening salvo, but I too will write additional essays as conditions demand. Kevin,

I agree.

No. And I will give it that I read it. I never once thought I was reading slop. I thought like, you know, particularly in some parts I was like, this is just actually how Zuckerberg talks. I'm sure it was a little bit like the State of the Union where lots of different, you know, policy hands had their fingertips on it and said, "Oh, you know, make sure to say this." But, no, I do think that this is his actual message. To what extent he believes, you know, everything in it and to what extent a lot of it is just sort of, messages of convenience. Well, I guess I don't know that.

I've been thinking about this a lot because a problem that I have in covering Meta just legitimately is that I've covered it for like more than 10 years. I mean it's like been almost like 15 years and so I just know a lot about this company and obviously you know it has changed in various ways over the year but I just remember so much about this company. I wish that I was born yesterday and just could believe everything that I read. But Kevin, I know too much.

So I do not think this is a messaging challenge. I think that there is room for AI positivity. But here's what AI positivity looks like to me. New medical advances powered by AI. Diseases cured, new jobs created, old jobs paying more money, kids excelling in education. Right? To me, that is the core of AI positivity. But crucially, Kevin, it is not delivered via manifesto. It is delivered via real experiences that human beings are having. And so far that has just seemed to be a chasm too great for any of our leading AI labs to cross. So if they can cross that chasm, that is the actual lane for positivity. And I wish they would get on it.

Ship it or zip it.

Well, Kevin, lately I've been feeling like we're entering a third era of slop.

Well, number one was the near universal disdain we had when we would see the laughably bad writing and six-fingered humans online.

The second era, I would argue, started when we began noticing that some of this slop was really, really popular and we saw shows on TikTok like Fruit Love Island getting millions and millions of views, proving that there was at least some demand for slop. And now what are we in?

Yeah. And this has been a big theme on the show the past few weeks. We've been talking about the steps that platforms like LinkedIn and Substack have taken to at least label or identify the use of AI in generating content for those sites. This is a pretty big trend in tech right now is that more and more people are getting called out for using AI. And primarily when they're getting called out for using AI, what I see at least are screenshots of one particular app, Pangram.

Yes, number one narc for people who are using AI and passing it off as their own writing. And I'm excited to talk about this because this is an area where my own views has shifted. I have argued before on this show that AI text detection is basically worthless. That you can't trust these AI text detectors. That they have tons of false positives. That students and teachers shouldn't be using these things because you could end up falsely accusing someone of using AI. But in just the last few months, there's been more and more evidence that at least Pangram and probably some of these other tools as well have gotten quite good to the point where they are they're not perfect. They're still generating some false positives and some false negatives, but they're much much better than they were even just a year or two ago.

Hey, thanks for having me.

Yeah. So part of the reason that we started Pangram was because a lot of these all the existing AI detection systems were pretty flawed in different ways. But I think the main thing is that most of them were using this metric called perplexity which was considered state-of-the-art at the time. So AI text on average is less confusing to a language model. It's lower perplexity and human written text has things that surprise a language model which so it suspiciously smooth and like that that is the product of AI models.

So we do something completely different. Instead, we are training our own classifier network. So for example, we might have an essay on Moby Dick written by a seventh grader and then we'll ask like an LLM to also write an essay on Moby Dick in the style of a seventh grader. And so then our model is able to learn the differences between A and B and figure out like what AI text actually looks like. And so because it's a neural network and not a metric, we're able to improve it with more data and more compute and make it a lot better.

Yeah, so we have we have a really good human training set. It's all pre-2022.

It's definitely not just em dashes and phrases. I think that's how you and I might pick up on AI text is like you see that it's not just X but Y and then you see the maybe the like shape of the text or like the GPT like really short staccato sentences and you're like okay I think I like know that that's AI but I think that what Pangram is doing is it's combining a bunch of really weak signals over the course of an entire document there's a whole bunch of weak signals in the different decisions that an AI would make in a certain consistent way and humans kind of have wider, less mode collapsed decision tree. And so I think over time, over the course of a document, you can build up confidence over a whole bunch of weak signals on word choice.

I mean yes, I think Pangram is very much tuned to minimize false positives. So we're not making false accusations, but if Pangram says that something is AI, we can be very confident that it is largely AI generated. And so this is sort of the trade-off that we have to make. I think it works pretty well because if somebody looks at a piece of text and it says the AI scores 100%. We think we think the whole document is AI then then like you just don't have to think about it more. You don't have to think what is if this is a false positive. You just know like okay this is probably slop.

There's definitely a bit of a cat-and-mouse here. So, we've seen a whole bunch of humanizers pop up. They're kind of a common tool that students will use and they do a kind of variety of different things from introducing typos. They could introduce like these zero width space unicode characters that like don't actually show up when you look at the text but the model sees it or they just like paraphrase every single word and so kind of we've seen the whole range and largely what we do is we train against it and so we're always like picking up the latest humanizers and training on them for our next model.

So Pangram works pretty well at generalizing within model families. So like if Pangram has seen GPT-5.4 and GPT-5.5 then 5.6 and six coming out is like not a huge surprise even if the writing style is a little bit different. Typically like Pangram's accuracy will still be pretty good. I think similarly we saw with Fable and Mythos like there's some writing samples from Mythos in the system card that Pangram is able to catch even though like it hadn't seen Mythos before which I think was pretty cool. But with that said, it usually when a new model comes out, Pangram will have slightly lower recall. So a little bit less accuracy at picking it up. So we're always going to pull new text from the model and then retrain Pangram and get out a new model in a few weeks.

I think a lot of what we're detecting is more subtle than like what you might pick up as the Claude writing style. So I think it might get better at like trying to imitate a voice and doing well at it, but it would still have signals that Pangram picks up on. And I think kind of what we see is these frontier labs, they're really focused on climbing capabilities. And so what this means is they're applying preferences to these models. They're saying instead of this model predicting the like average next token prediction of any writer anywhere, it's going to say this model prefers to do good writing and prefers to write correct code and write correct math. And I think these preferences are largely what Pangram is able to pick up on.

So I think AI as a tool is probably the wrong abstraction. It's like a little bit closer today to AI is an employee or like another individual that you know you collaborate with. But I also think what we're building for is sort of these like AGI futures where if you look at the internet today, bot traffic has just surpassed human traffic. Like it's about 50/50. And I think if you look a few years out from now, maybe a decade out, it's going to be like 99% bot traffic, 99% AI, autonomous agents going online, you know, writing GitHub comments, promoting their Substack, you know, getting people to come to a bakery, like kind of anything. I think there's like this basically like this technology is way more powerful than spell check or a typewriter. It's really something that is its own individual entity. It can do cognition and I think because of this there's a really strong reason that we need to like as humanity discriminate in favor of humans. I think there's still going to be a lot of need for just being able to say on a programmatic algorithmic level like hey I think this came from a human and it's important because it came from a human.