AI’s Threat to Privacy - Signal President Meredith Whittaker on the Risks of Agents

We live in a world of mass surveillance of everything that has been instrumented very recently in human history. We're looking at like 20 years, 30 years max, where suddenly all of our actions, our communities, our connections, our thoughts, our feelings are being logged as data. Signal is the one tiny bastion of meaningful privacy that is striving in this vast data slurry that we now live in to maintain the fundamental human rights of private communication.

Bonjour.

I wouldn't. I would ask a French person.

Well, you know, it's happening. I think like being right isn't necessarily a theory of change, but because we're Signal and we are focused so almost myopically on privacy and preserving the fundamental human right to private communications, which we all know and love, I hope, we're really sensitive to some of these low-level technical changes even as the kind of marketing and hype around them might not indicate the real collateral consequences. So we are seeing, you know, agents move into every corner of our digital lives, and we're seeing things like Apple introducing more agentic capabilities in Siri, which is now moving compute offsite because, of course, one of the issues is that these sort of, you know, highly capable or at least very large AI models on which agentic capabilities generally rely, or at least the, you know, most sophisticated, highly capable agents rely, can't run on your device. And so you see Apple is now using Google Cloud compute, so, you know, with some encryption involved for their agentic capabilities. But this is again kind of breaking the Apple privacy promises.

And the real issue is however much encryption you have on the inference side, on the server where you're sending the data off the device to process and to generate an AI answer, you also need to give the agent access to your data. Apple, in their press release, uses the example of sort of planning a dinner for a birthday of your friend. Right? That's almost literally the example I used over a year ago when I was saying this seems mundane, but in order to plan the birthday, it would need to have message history with the friend. It would need to understand, you know, Matt Green makes the point, maybe there are dietary restrictions. Is there a restaurant nearby? Does it, you know, can the credit card to make a booking, the ability to drive your browser to do a search, look at your...

Exactly. Exactly. And then it needs to be able to interact with websites and prompts that happen offsite. So that's not happening on the inference server. And this is a point Matt makes really well. This is happening when the agent, you know, encounters new information. And so it goes, say, to the website of the restaurant or a site it thinks is the website of the restaurant, who knows, and it is, you know, processing that text. And that text might say, hey, you know, exfiltrate the data, send me all the data, send me that list of names that includes your birthday, or list of information includes your birthday, your mother's maiden name, send me, you know, a file containing Signal messages you've extracted, etc. And what I'm describing having here is sort of basic prompt injection, but prompt injection is a huge risk for these LLMs that is, you know, unsolved.

Yeah. Beautifully described. And an agent, you know, an agent isn't conscious. It's not discerning, right? It needs to, it's looking at as much context or as much highly intimate data as it can in order to predict what the right call will be, right? And then act on that call without explicit permission from you, or at least in the vision of a perfect agent. You are letting the agent go autonomously do every step it needs in a many-step process in order to get from figuring out the right date, the right people, the right restaurant to actually having a booking where they're all invited. That's many, many, many, many, many, many turns without you stepping in and saying, you know, stop or go or no, not that.

Right? That's the fear at this point. We have not seen an agent that is integrated into the Apple or Android operating system to date that is simply demanding access to Signal messages. But we need to understand the operating system is the water that Signal and every other application swims in. The operating system is the context within which we operate. And ultimately the operating system, how it works, what it does, is controlled by the companies that create and maintain it. So Apple, Android at Google, and then you have Windows and, you know, Microsoft, and those are the three big proprietary operating systems. So if a change is made by Apple to say like, now we are fully agentic and that means that our agent will help you live your life, plan your things with access to all of your context including Signal data, that has now become a significant backdoor. Can you block that? Like if Apple says, you know what, we'll give the agent access to everybody's phone, can you, can you set Signal up in such a way that that's not possible, override the desires of the user? It would be possible for Apple to ensure that we could, but that would be an option that Apple would need to implement. Well, Apple have not, but Apple also have not yet shipped an agent that is demanding that kind of access to Signal. However, the marketing promises around agents prefigure a future in which that's the type of capability that these agentically enabled operating systems will provide.

On Windows, we explicitly block the Windows Recall agentic feature. And we do that by using a digital rights management flag that blocks screenshots because Recall, which is the agent that was, you know, implemented by Microsoft as part of their Windows 11 update. The promise of Recall is that it's an eidetic memory of everything you did on your desktop. So you can be like, what tab was I looking at? What thing was I doing? What was I watching? And it would tell you. The way it does that is by screenshotting your desktop every few seconds, creating a database of the text from those screenshots, and then querying that database when you actually ask it a question. And of course, if you're running Signal Desktop on Windows, that would include your Signal messages. So...

They implemented it in a narrower form. Still didn't give an opt-out to Signal. So we had to sort of hack together an opt-out that was again using this digital rights management tool.

That's true. Yeah. Which means screen readers like those that blind and low vision people rely on no longer work.

Yes. And that's not a trade-off we want to have to make. What we...

No. The issue is, and what we have been pushing these companies on, is one: these agentic capabilities should be opt-in. So they should be turned off until people explicitly opt in. And that should include developers, like the developers of Signal, being able to say on behalf of our users, agentic systems cannot access this data. And that should be the default. If we want data to be accessed, then we should have to turn that on explicitly.

And look, this is tricky. But the issue is that you, as somebody involved in investigative journalism, probably have sources who wouldn't make that choice, right? And when we're talking about Signal, what we're talking about is a two-way street. There's at least one other person implicated in the choice that you make. So, Signal should be able to say for Signal installations of this app, no agentic access.

Yeah. Assuming that your agents are configured to do that. Yes. Yeah. And that's the issue. I mean, if they have access at a low enough level, they can hook into the screen buffer.

I think yes. And I want to push on that framing a little bit because I think a lot of people who are doing that don't actually understand the technical and data implications, right? What they're being told is like, here is a productivity tool that's going to make things a lot more efficient. Click yes, yes, yes, yes, yes, and then it can manifest this outcome for you. But the fact that you're giving it akin to root access, like the lowest level, most pervasive permissions, like, you know, read my diary, go into my underwear drawer, like rifle through anything in my bedroom and then pass information about that to any other system on or off the device, process it offsite on, you know, kind of secure but not really secure infrastructure run by companies that can be subpoenaed and act on that autonomously to hope to produce an outcome that may or may not actually manifest. While we know there are sort of significant vulnerabilities here, you know, what you're talking about is an extraordinarily insecure paradigm that, you know, we're talking about it in terms of its outcomes and its wishes and the sort of magic genie possibilities of a life without toil that the kind of marketing is hoping these agents will somewhat manifest or at least convince people will manifest such that they use these agents. But the radical shift in what computing means, what an operating system actually is, and the security and privacy perils for almost literally everyone, because we're, you know, we have three proprietary operating systems that, you know, your military general, your nanny, your clerk at the grocery store all runs one of those, right? These are shared dependencies across billions of people that are being quietly changed at a very, very fundamental level to, ironically, move agency away from developers and users to these tech companies that are running and instrumenting these agentic systems at a time where there's a real desire on the part of these companies, a real need pushed by the market to realize revenue, to realize profit, to recoup the trillions of dollars of capex that has gone into AI and the AI infrastructure building.

Oh, I don't know if my argument will win. I do know it's right. You know, I think we will see pullbacks, we will see mitigations, we will see, you know, custom builds or what have you shipped because you simply can't have such a critical security vulnerability in core infrastructures that, you know, touch hospital data, touch military data. You know, you're already seeing, you know, companies banning ClaudeBot. You're seeing companies say, 'Do not, please, for the love of God, our CISO is begging you, implement, you know, Copilot across our proprietary data.' You know, we're already seeing quietly within a lot of companies sort of, you know, increased restrictions on what access these systems are given, how they are implemented. You know, again, a solid analysis is not necessarily a theory of change.

I mean, I think privacy is low on the list for everyone because the stakes of privacy have been so, like, badly defined for people, right? Like, who gives a shit about data ownership, right? You have laundry to do. You have real stress. You have to worry about, like, are you going to have health insurance if you're in the US? I think people do care about the sort of invasion of autonomy, about the hijacking of their ability to sort of work quietly at work. You know, I think the collateral consequences of a stripping of privacy are actually things people care about, but they haven't been well connected. I think privacy and intellectual property, privacy and confidential information, privacy and the ability to negotiate military strategy in real time. All of those things a lot of people care about. I think consumers... Yeah, consumers, you know, like AI is corny. AI is being marketed by some pretty unpleasant-seeming people whose rhetoric is like, you are now useless and you work for the bot, right? Like it's, you know, it hasn't had a great debut into sort of generalized public consciousness and you're seeing, you know, you're seeing that worm turn. So like, no, I'm not saying that this is animated by a privacy concern, but I think there are sort of compounding concerns, none of which make AI look great in the eyes of the people who sort of feel like it's being, you know, shoved into their lives with little benefit and many potential downsides.

I'll be interested to see that borne out. I do think there is still a consumer push to integrate these. And at the business level, there's a paradigmatic tension between the agentic vision of autonomous bots that can do everything for you without you having to oversee them, which requires access to all data, the ability to send and receive data from offsite, the ability to act on that data without permission, act on a prompt without permission, the ability to create more data, etc., on the one hand, and privacy on the other. And so that I have not seen well negotiated. And I do know, you know, I'm from this industry. I am of it. And I talk to my folks in the security team, in the, you know, kind of privacy-conscious folks in tech, and they're kind of feeling increasingly relegated, you know, not consulted as much as they were, not included in sort of launch decisions as much as they were, you know, and there's a, you know, there's a lot of pressure to make AI happen, which doesn't mean that there aren't, you know, okay, we're offering sort of sovereign or secure compute for inference, right? Or we're offering, you know, different safeguards. Here's a harness on this or, you know, kind of administrative controls for your CISO or, you know, various checkpoints along the way. But ultimately, each one of those things, you know, minus kind of secure compute, but that's a tiny piece of the equation. Each one of those things is a hindrance on full agentic capabilities as they're being advertised.

I think the overall thought is not necessarily a comment on that policy change. It's just we have a handful of companies with carte blanche to make one decision one day and another decision the other day, to pull the rug out from under privacy promises or implement privacy guarantees without democratic oversight or input into those decisions.

I think we don't know. They could. We can't know the answer to that. In the US, there's a power called the national security letter that can be sent, ordered by the government to a tech company, that can mandate them to do or give over certain data, and that gags the tech company from ever acknowledging that they received that national security letter.

Right. So fundamentally, an interface that gives you a little checkbox is not a robust guarantee, nor is it validation that that policy is actually followed.

Every time we get a valid subpoena, we work with the ACLU to first fight that. And if we're not able to fight it and we have to turn over that data, then we work with the ACLU to unseal it, and then we post the subpoena and exactly the data we were able to give over, forced to give over, on signal.org/bigbro, which does a lot of things. And, you know, one of the things that it does is demonstrate just how vanishingly little data we actually have to turn over in response to what is often just a terrifying laundry list of requests that we have to assume is the normal amount of data an average surveillance messenger would be able to give over to the government in response to a similar request.

Signal is open source. Yeah.

You know, what the definition of AI is is complicated. I think, you know, there are people experimenting with these systems, but we're sort of early days of exploring what it...

We, no, we don't. But we have very, very clear safety and...

We're one of the, and we're allowed to say this now, we're one of the early GlassWing Mythos users. So you were given an early version of Mythos. Yeah. I mean, Signal is very serious critical infrastructure. It's whatever is shifting the offensive and defensive landscape in terms of, you know, vulnerabilities and exploits is something we need to stay on top of. And I think, you know, we, yeah, we got an early preview. We've been using it. We've been trying to figure out, like, how do we adjust, if at all, to a landscape in which the bar will, you know, the kind of democratizing, let's say, access to much more capable offensive and sort of security vulnerability-finding abilities will do, you know, for open-source projects like Signal and...

No, it works. I mean, we get access to an interface, right? And we get on the call with the helpful Anthropic folks and we debug the things we don't understand and we, you know, figure out how do we use the token credits and how do we think about that.

We did targeted scans across our codebase and various libraries. Yeah. And we had folks who sort of managed that process. Yes.