REPLAY: Signal's Meredith Whittaker on Backdoors and AI

Such a treat to be here, Joseph. I am such a huge fan of your work and 404 Media. So, I'm just going to put a plug in right at the jump that folks really should support. You all are doing so much heavy lifting in an ecosystem that has sadly been hollowed out a bit too much in the last couple of years. So, thanks for that work, and again, really happy to be here.

That number is growing, and I will just off the bat be annoying and coy here and say we don't give exact numbers of our user base. One reason for that is that it's pretty volatile. We will see massive growth in a region in response to a political issue or what have you, and then perhaps some drop-off, and it really expands and contracts a bit. But it is overall growing. We are seeing increased sensitivity to privacy. That's evidenced everywhere from our user growth charts to the fact that Apple and Meta are spending billions of dollars on advertising, really indexing on that one value, to simply the tenor of the public conversation as things like the AT&T data breach, which you all reported on, continues to make people aware of the very real material dangers of the mass surveillance that most of our core digital infrastructures are conducting on us every day. So, you know, Signal, if you want a little shortcut here, Signal has been downloaded 200 million times, about—I don't have the exact number in front of me—just from the Play Store alone. And of course, that's one of three clients. And we are, you know, we're looking at trends that really point up and to the right and have, you know, some pretty big plans in the future in terms of how we can ensure that Signal as core infrastructure, both the messenger and the protocol and the work that we're doing across the stack to enable privacy in an ecosystem that defaults to surveillance, is able to grow and spread and becoming increasingly foundational to a digital ecosystem, the internet, whatever metaphor we want to use for the fact that, you know, this now is the nervous system of our social and political and economic life.

Well, we are introducing features that we hear from people across the world, and these are different features depending on the region, the context, even the age group are core ways that people communicate. So stickers are not hugely popular, for instance, in many contexts in the US, but they're wildly popular. They are core communications tools in much of East Asia. And so a messenger app without stickers is not very useful to people who are, you know, who have facility with that paradigm, who, you know, think in stickers, who send stickers as part of a conversational rhythm that is, you know, very familiar. So part of what we're doing there is trying to put the core function of communication at the center of what we do. People don't pick up Signal to flex the fact that they care about privacy. They pick up Signal to talk to someone they love or like or want to speak to in some other way. It is fundamentally about human communication, which itself is fundamental to being human. So we want to respect the fact that that takes many forms and many args, and stickers in one case is very important. So it's, you know, it's to enable growth. I think, you know, often growth comes when we're talking about communication networks, the network effect really matters. So, you know, communication networks, to be, you know, 101 about it, they increase in value with the quantity and quality of the nodes added to the network. So the more, the more valuable.

So, you know, put another way, no one buys the first telephone, right? Put another way, if you switch to Signal but your friends don't, you haven't really switched to Signal. You're not really able to use it, right? So we see growth perpetuated by sort of collective events or collectives. You'll see, you know, a very easy one is political volatility when, you know, the distance between physical safety and digital privacy collapses. Ukraine, one of our hugest markets, and, you know, we saw that rise from a very small market to a very large market, you know, obviously in relation to the Russian invasion. We see growth in groups that happens in relation to just big tech messing up, the incumbents making a policy change. In early 2021, WhatsApp announced a change of terms of service where they announced to their users that they were going to be, you know, giving data to Facebook. And they had promised for a long time not to do this. This was a broken promise, and people noticed. People do actually care about privacy, and they switched in droves to Signal. We, you know, basically our servers—we had to recover from that—did a very good job. You know, bless the team who stayed up nights doing that hard work. And then for over three months, we were number one in the app and play stores across 70 countries. So, you know, you see what we want here is to be ready for those precipitating events, to have the features that make Signal seem easy and intuitive, you know, across the globe. And those aren't going to be the same features in every jurisdiction. You know, stories, for instance, which are not hugely popular in much of the West, hugely popular in South Asia, hugely popular in Brazil and other parts of South America. So again, we're, you know, we're trying to cover a lot of ground, but ultimately it's about making Signal as easy to use to communicate in the way that you're familiar with and that your friends are familiar with as possible, so that when you need it, it's truly there for you.

You are missing out. You are really missing out.

Oh my gosh. Well, I'm going to—I'm just going to say I really love stories. I think, you know, it's for me, I actually love, you know, like I love seeing what my friends are doing. I love cute people. I love a cat, right? You know, no shame in that. And I feel really uncomfortable with Instagram. I feel really uncomfortable with sort of, you know, that stuff being scraped for AI or, you know, have ads run against it or sort of show up in some stranger's feed. The whole thing, you know, went from a photo hosting app that was slightly better than Flickr to, you know, a biometric surveillance hellscape. So having a place where I can share like cat photos or, you know, cool photos that I took that I know they're not going anywhere, like, you know, they are literally deleted off of, you know, my device and everyone else's. They are encrypted. No AI is run against those. I'm not exposing a stranger's face to facial recognition that could imperil them because they overstayed their visa and that's run against some Clearview system and the Meta backend that goes to law enforcement. You know, I don't think that is happening now, but it's incredibly technically possible and it's the exact kind of thing that could happen. So anyway, I love stories. I think folks should enjoy stories. And it doesn't come with all of the stress that Instagram and the other surveillance apps come with, but we also give you the option to turn them off permanently and never think about them. So we're different than big tech in that way as well.

I mean, that's a good question. I think the paradigm is the more complex the code, the more room there are for bugs to hide. We, you know, we are audited regularly. We engage in, you know, security best practices. We have code review practices, and I think it's, you know, we're also developing, unlike almost any other alternative large-scale messenger, in the open so people can watch. Even before the feature is pushed, we are developing in the open on our repos. There are people who will go in and, you know, bits of code behind a feature flag that's sort of, you know, being iterated on and begin to hypothesize in our forums or on Reddit or wherever they do it about, you know, what we might be building. So there's a lot of scrutiny and a lot of eyes on our code, you know, which is honestly, it's a gift, right? We have people looking at it, finding issues, raising concerns. You know, sometimes it's bad faith, sometimes it's annoying, whatever. But nonetheless, that's a powerful immune system. So that's not a one weird trick. There is no one weird trick. The trick is you never stop asking that question. You never let down the vigilance. You never think you're safe and out of the woods, right? But I do think we have a lot of practices and a development paradigm in place that puts us in a much better spot than any other competitor. And I mean, let's be real, like, if you look at the competitors' apps, there are every single quarter some product development team has got to ship some crappy feature because let's say someone named Mark is obsessed with the newest tech hype, right? And so, you know, in terms of bloated features, I think Signal is actually incredibly lean and elegant. And it really is that balance between how do we not fall behind the norms that mean that we're genuinely not useful for certain forms of digital communication that most messengers would be useful for, you know, while maintaining our integrity as best-in-class for security and privacy.

And we go out of our way to get that as close to nothing as possible. So yeah, you can look at signal.org/bigbrother and you can look through other subpoenas that we have received and unsealed and posted there. We fight every subpoena that we get. If we are forced to comply, if we lose that fight, we then have to comply. That's how it works. And that's why to be a truly private communications provider, you cannot have the data because ultimately you can be forced to comply or shut down or, you know, what have you. So we do comply, but what we comply with is vanishingly small. And if you go to that site and it hasn't been updated recently—I believe it is updated. I'm not actually looking at it right now—but we're in the process of posting some newly unsealed subpoenas there now. So you'll get a number of examples of how those look and just how little data is available there.

I don't think in stack ranks...

Yeah. I mean, we're thinking, like, we're talking about a kind of nexus of threats, and a primary one among them is, you know, centralized power tends to constitute itself via information asymmetry. And so since, you know, I don't know, you can look back in 1976 with the US government trying to prevent the publication of Diffie and Hellman's paper on public key cryptography because they didn't even want the paper out there, right? There has been anxiety post-World War II, you know, around the idea that any network for communications that people can access could be off-limits to government scrutiny, and that is not going away. I think we need to be on the lookout for threats to the viability of encryption itself. Threats to implement a backdoor or a front door or bolt on a surveillance service as a mandatory component of any end-to-end encrypted communication service, thus completely nullifying the entire point and tiring us out with having to argue against rhetorical tricks over and over again, as the European Commission has just done and then withdrawn. So those are legislative threats that we've seen many of in the UK, in Europe. You know, we saw Australia propose some of these, but seems to be walking it back with some fairly sane language in some of their latest regulatory and policy proposals. So I think we'll see more of those. But I do get a sense that the arguments are getting through to some extent, that, you know,

to the policymakers and the people who may not have understood the type of threat that their well-meaning legislation was posing. Because I think in some sense we're dealing with a scenario where the pretext is incredibly inflammatory, very emotionally charged. Child abuse, child sexual abuse, we need to prevent this. No one disagrees with that. People, yes, we need to prevent it. We'll do whatever we can. Well, here's what to do. And a lot of people, I think, just, you know, took the instructions and said, 'Okay, we're going to diligently do this,' not looking at the fine print or not understanding that there were some Trojan horses in there in terms of fundamental rights and liberties.

Yeah. I mean, there's the long version, which I don't think is terribly interesting for the lay listener. But in effect, the conceit was that there are issues with child sexual abuse material online, and that law enforcement, governments, well-meaning NGOs, in order to deal with this, they need to be able to effectively break end-to-end encryption and scan private communications to find those that may contain such imagery, and we assume, you know, prosecute both, I guess, the culprits and... So that's the high-level rationale. And so we've gone back and forth for, I think, a little over a year. You know, there's been a proposal to do certain kinds of scanning, and then the scanning that would break into encryption constituting a backdoor. And then the technical community and ourselves and others would sort of come out and say, 'No, that's actually a terrifying cybersecurity risk, and there's no evidence that it actually helps to protect children.' And then a new proposal would come and say, like, 'Oh, well, we're not going to create a backdoor. We're going to put the scanning in front of the encryption so it doesn't actually hurt encryption.' But of course, if you bolt surveillance as a mandatory component of an encrypted system onto, you know, whatever the communications platform is, you have undermined encryption. You've moved the target for hackers from unbreakable math to some crappy vendor-provided software that the government is mandating, which is going to be easy to break, right? So we've had to kind of go through a cycle of what I would call, like, rebranding the same old thing in an attempt to get it through and get it past the technical community's vocal opposition, the long-standing consensus that, you know, this is math, not magic. You have to recognize the limitations of your desires here. And so that's where we stand now. The latest round was maybe a month ago. There was a drawdown, effectively, and now the European Commission is reconstituting itself after elections. We have Hungary coming into the presidency, who is rumored to be largely supportive of this. We don't know if it'll be picked up or not, but I, you know, again, coming back to the first point, power constitutes itself by information asymmetry. And this has been a long-standing wish of law enforcement to undermine strong encryption. I don't have much confidence this will ever be put to bed entirely. This is not a disagreement. This is not a misunderstanding that we can educate away. This is a battle for power that we're going to have to contend with on those terms.

And before that, there's, you know, in the '80s, Reagan was also discussing some of this. So it's... there's never been the prospect of networks that law enforcement can break that hasn't freaked out government, in my historical reading. I think there are other threats we could discuss that aren't legislative that might be interesting as well. And one of those that has occupied my time is the move of AI systems into the operating system, and things like Microsoft's Recall in particular. You know, I would say pretty alarming shift in a long-standing paradigm in which the operating system provides a, you know, a trusted basis on which various applications, not from a single vendor, right, are able to, let's say, form contracts with a given user. You know, Signal's contract is we collect none of your data. If you're using Signal on a device that isn't compromised, it is secure. It is not going to be leaked. And thus, when we fulfill a subpoena, for instance, you know, it only has a vanishingly few bits of information in contrast to any other application. And that's, you know, that's a kind of an incredibly important paradigm. And what we're seeing with things like Microsoft Recall, which was meant to ship with Windows 11, the new Windows operating system, is a violation of that paradigm where in the name of, you know, feeding an AI system that will do some convenient or inconvenient or completely useless thing for you—in Recall's case, it's constitute a kind of eidetic memory of everything you were doing online for the last three months. Like, anyone wants to know that you were doom-scrolling at 3:00 a.m., right? Like, you know...

No, you don't. I mean, just a, you know, plain-text honeypot on your OS that includes screenshots of your Signal Desktop messages. If you're using Signal Desktop, that fundamentally violates that contract between Signal and the person using it, which is then being subverted by the operating system manufacturer, the OEM. So I think, you know, I think for me, we need to be a lot cannier about simply accepting that if AI systems are running on the device and not phoning home, they somehow constitute a private system. Because what we're actually seeing is the need for data that these systems have and the functions that these systems are being put to, whether it's Recall and this sort of silly eidetic memory that's like a desperate search for a market that's not materializing, if you want my read on that, or Google's Gemini, which is, you know, going to scan your phone calls for scams, but could very easily be purposed to scan your phone calls for discussion of drugs or scan your phone calls for people seeking abortion care or anything else that we really need to pause before we allow these companies to make those kind of determinations, to violate our privacy, to make those decisions about us. I think we need to re-evaluate that paradigm, and some of my thinking and my work right now on maybe the scholarly side or the, you know, kind of analytic side is really focused on that.