The Individual in the Machine: Meredith Whittaker on reclaiming Privacy in the Age of AI

Thank you. Thank you all. I want to keep these remarks brief and there's two things I maybe want to talk about. You know, what does it mean to be Signal in an ecosystem that is fundamentally hostile to the thing you're all about? And then I want to talk a little bit about the threat that agentic AI, which is the sort of futuristic term of the moment, poses to the project that Signal is dedicated to and to, you know, many of the things I think that are animating the folks in this room, if I'm guessing right. So, you know, I'll start. Everyone here uses Signal. I assume there are people in this room who knew parts of Signal's history better than I do because you were in that hacker meeting where it was discussed, right? But like, I was there too, so I know some of this. So I don't think I have to go into sort of the spiel I would give to a less familiar audience, but I do think it's important to position Signal within the political economy of this very specific, you know, tech industry that we have, of the, you know, the surveillance business model driven monopoly platform tech ecosystem that has now, you know, taken control of, is maybe a strong way to say it, but, you know, been seated control of the nervous system of our social, economic, and political institutions globally. Yeah. And we can look at how this ecosystem formed. You can, you know, do some of the historical surveying that I've done in some of my more academic work and just recognize at every step how contingent it was and how not inevitable the form we are now faced with is.

You can look at the 1990s when the, you know, Clinton administration put a lot of their hopes in the prospect of a commercial internet to solve the economic woes that the US was facing as manufacturing was offshored and as the reality of a neoliberal economy in which sort of, you know, national sovereignty meant less and less as capital could travel across borders but people couldn't, etc., etc., became sort of evident in the US. And the idea that you could commercialize the internet and that would be, you know, what they were calling it back then, the New Deal without the socialism. It would be economic revival without redistributing wealth, you know, simply allowing the private sector to take over the infrastructures that had been built for government, military, and research purposes in the past. This was, you know, this was sort of promoted with a lot of fanfare. It was enabled by the fact that databases had become a bit more powerful, that the personal computer was now something that could actually ignite an imagination where everyone had a hub at their house and time sharing wasn't just big computers in giant closets in universities, but was something we could all kind of participate in through networked infrastructure. And the idea that you could somehow, you know, have another media ecology. So you had TV, you had print, and now you had internet. Through this decade, you know, the sort of debate around privacy, the crypto wars, a lot of things, some of you, I believe, you know, certainly know more about some of these histories than I do, kind of raged. But, you know, ultimately the bones of what we, you know, the body that was created, the infrastructure we have now were set in place largely through policy decisions and other decisions that were made, you know, during and around this decade. And there is sort of two key decisions that I'll index on here. They're not the only ones, but I've called them sort of the sins of the 90s because I like a dramatic flourish, I study literature. And these were, you know, two key choices that were made in 1996. You know, it was the, I forget, it was something like cyberspace was in the title of this bill, but basically this was the initial policy foundation, the rules of the road for the commercial internet as set down by the Clinton administration. And two key decisions that I think helped shape where we are today were made then. The first was there were no restrictions on surveillance for the private sector. No privacy guardrails. And this is not because we didn't know, because there was no warning, because the architects could never have foreseen the results of their creation. No, there was a lot of warnings from within the Clinton administration, from, you know, across the board. There were organizations advocating about the, you know, constitutional issues with allowing mass surveillance on platforms, but nonetheless, no privacy restrictions. And they did that cute little policy thing that they often do where they say like, 'And if this is a problem in the future, we'll revisit it.' Well, it's 2025. We have no federal privacy law in the US. And, you know, the entire world economy now runs on surveillance conducted by five companies based in the US. So there we are. Didn't revisit.

The second was that as the business model of this commercial internet, from among a number of choices that were clearly on the table, including a more public broadcasting effort, including a, you know, kind of a social tax on this, including a number of different options, the option that was selected was advertising. So you not only have permission to surveil and no privacy safeguards, you have an incentive to do it. You have an incentive to know your customer and to compete with other platforms and others, you know, on the basis of, you know, we actually have the data about the types of people. The lady with curly hair who lived in Brooklyn who likes yoga, we know how to target her because we have information about her. And so this combined with the constitutively centralizing monopolistic formation of the communications network that we've known since the postal service, since Western Union, since, you know, the state-managed monopoly of the telecom companies, and, you know, that was not regulated here, combined to create key incentives that led us to a world where the economic engine of tech is surveillance. You monetize data. You feed it into your advertising models to know types of people that you sell access to advertisers to, or you now feed it into your AI model, which produces more data, which you feed into your AI model until the centipede collapses. But ultimately that is the business model of tech, or, you know, you're Apple and you own fab to chip to device to the entire thing and you collect the data and you kind of have a slightly privacy-preserving, but, you know, ultimately that's the business model. And so the shape of what we call tech, the shape of what we call innovation is ultimately a sedimentary history of choices and imperatives that were made around these key objective functions. You have to collect data because you can't make data money without collecting data, and you do that for sort of advertising, and then you grow at all costs, you scale, you, you know, the classic sort of Netscape, Peter Thiel, you know, platform model. And so, you know, I think what is special about Signal, many things are special about Signal, and many things are very mundane about Signal. You know, you pick up Signal, it works like a messenger, it's mundane, but the fact that it exists as much as possible, you know, to continue outside of that paradigm, that the objective function, the entire, the north star, the goal of Signal is do tech, but do it privately, do it while resisting this set of economic imperatives, is really, you know, it is a remarkable feat. And it takes us a huge amount of effort to create something that works in that environment when none of the nutrients needed to sort of feed the plant that we are, to use a little bit of a natural metaphor, exist in the soil we grow in, where we have to rewrite core libraries. Because if we were to use one off the shelf, we would be importing with it the assumption that we are collecting data, sending data, doing the normal thing, which is wanting all of the data to feed this economic engine. So that's sort of the first part. And, you know, then I want to talk about the fact that this doesn't mean Signal can live outside of this ecosystem, this ecology, right? We are dependent on many things. We are dependent on the hyperscaler platforms because you can't make a call from Bangkok to Berlin without edge server capacity, and we don't have hundreds and hundreds of millions of dollars a year to run edge servers in every single geography. The norms of tech are set by the environment we're in. We don't get to create sort of a solipsistic, individualistic world that only we occupy. There are two app stores, and yeah, you can sideload. Okay, but I'm sorry, your crush isn't going to sideload. So who are you going to talk to? And there are a handful of operating systems. We have three client teams. We have Android, we have iOS, and we have, you know, the desktop client that works on Mac OS and Windows. And right now we are seeing AI agents, and this is where you see I brought it back, integrated, you know, at the application and most concerningly for us at the operating system level. And now what is an AI agent? An AI agent is an invasive species in some sense. It is a name for a set of capabilities, software capabilities that are wrapping around a handful of large AI models. This is a marketing term, so it's not always specific. And, you know, that present a set of functions that allow these AI models to act without explicit permission. This is another name for autonomously to complete complex tasks on your behalf. So I can say like, I want you to call my insurance company and get a quote on this and then send that to me and then send it to my lawyer. Right? That's the fantasy of what these agents will do. It's not something they're actually capable of. That's a different story. I want you to book a restaurant for my friends and then, you know, I want you to find a time that works for my friends and then book a restaurant for us and then tell my friends that the reservation has been made. To do any of this, you need extraordinary permissions at the operating system level. This is root level permissions. You know, and you need access to data across your system. So I've used the metaphor of breaking the blood-brain barrier between the operating system and the application layer. You can think of sandboxing. That's not technically specific, but it gives you a sense of what we're seeing. The paradigm for a number of decades has been you can more or less trust that your operating system is going to work, you know, the way a developer wants it to work or the way that the person using a system works. It is now being sort of instrumented by the companies that build these operating systems in order to perform things autonomously that require, say, access to your calendar data to check what times would work for a restaurant reservation, access to your credit card to make a reservation at the restaurant, access to your contact list to find your friends' information, and, you know, maybe access to your Signal account to text with your friends, message them and tell them, 'Hey, the reservation has been made at this time,' and then maybe put it on your calendar. All of those are attack vectors, vulnerabilities into those systems. You're talking about a giant, you know, set of data that you have to have access to. And you are talking about something that for Signal constitutes a backdoor. The vector to attack Signal is no longer, you know, you have to figure out how to perturb the one-way function, you have to undermine our robust cryptography. No, it's you have to figure out how to access the same pathway that the agent accesses to get into Signal. And so, you know, I'm sort of frontloading what could seem like a fairly arcane concern here because I think given the activity I saw in Mastodon and others and the folks I was so excited to see at this event, you all are exactly the folks to, you know, think about this deeply and recognize just what a profound threat it poses to the possibility of privacy at the application layer in the future, privacy at the software layer in the future, and the centralizing power of ceding that type of autonomy or, you know, permissionless control to a handful of companies that ultimately control the operating systems that we rely on on our devices across the board. So that's my opening spiel. I will now turn it over to more conversation. We're going to have like a conversation here that will be streamed, and then we're going to turn the stream off and have a real talk Q&A that is not streamed. So now, thank you very much.

I'm sorry, I can't say AI without agentic in front of it because I'm a VC and we... Okay. What's the AI in agentic AI? No. I mean, it's, you know, I don't know, a surveillance empire wrapped in a trench coat with some fairy dust on it. Like... That's a pretty good definition. Yeah. I mean, it's as good as any. Do you guys want to fund me now? Like, I have kind of a potted history that I often pull out. You know, the first neural networks were like '53. That was over a decade before the term AI was invented in '57 or '457. And actually the term artificial intelligence was invented in contrast to the neural network approach. There was like the symbolic approach, which was McCarthy and the guys who invented the term AI, which is not what we're using now. Although we're kind of backing into it as we realize once again neural networks are inaccurate, but whatever. Neural networks. There was another school, there was sort of the connectionist school, the neural networks, and then there was the symbolic school, and they were like opposing. And one of the reasons you invented the term AI was because you wanted to exclude certain people from your little party because you weren't into the neural approach, or like Norbert Wiener, and you wanted to get grant money, and AI, artificial intelligence, was a good way to get funding, and it was a good way to found a field you were the father of instead of being a disciple of someone else's field. And anyone in academia knows exactly what I'm talking about in terms of incentives. So like that's not what AI is, but that does give like a very ground-level material history. Like this is a word. It's been used as an umbrella term to basically describe a heterogeneous collection of approaches that has changed pretty dramatically over the near 80 years, over 80 years I guess, since it's existed. And it's not a technical term of art. It's a marketing term. And marketing doesn't mean it's fake. Just means it's doing a certain type of work in the world. And it had, you know, actually the neural approach had gone out of favor because it, you know, you kind of have these waves of like hyper, hyper-exacerbated promises and then those are not realized and then the funding dries up and those are called winters and then you have another sort of, you know, whatever. You know, what happened here was basically the surveillance business model took hold, the platform business model in the 90s, growing through the 2000s and the 2000s, the business model of collect all the data and then process it to sell ads against search, ads against blogs, ads against email, whatever we can do, ads against social media. And that was the business model. And to do that you had to, you know, serve a bunch of content globally and you had to collect a bunch of data and you had to have infrastructure to process all that data. So you're kind of putting in place the ingredients for AI at that time. And then in 2012, there was a really important paper that was published and it kind of brought these ingredients together, the massive amounts of data, the infrastructure and algorithms, neural network algorithms that were authored initially in like 1989, right? But were then reanimated when people recognized like, oh my God, now we have a huge amount of data and now we can use the parallel processing power and gaming chips or just a ton of CPUs that exist to make these old algorithms, these old approaches to AI do new things in the context of the business model that has emerged that has sort of shaped this superstructure of, you know, data and infrastructure in the hands of a few companies. So this was 2012. This was the AlexNet paper that I'm sure many of you are familiar with. And right after that, and I was inside Google at this time, suddenly there was a recognition that like, oh, what is AI good at? It's good. It's an optimizing function, right? And what is an optimum? Well, ad engagement. So the three authors of that AlexNet paper immediately hired to Meta, hired to Google, starting little sort of machine learning groups inside, you know, trying to optimize ads. You can see the growth of AI alongside the sort of changes to the YouTube algorithm, which are like championed as sort of feats of machine learning on one side and then like bemoaned as like radicalizing problems on the other side. But nonetheless, it was a recognition by industry that with that AI or these old approaches could do new things that were productive in the context of this ad-supported surveillance business model because they suddenly had all of this data because they had infrastructure already calibrated to process this data. So AI is kind of a narrative that is wrapped around the reality of the, you know, surveillance business model and the resources that is accrued to a handful of companies. It's not a singular innovation. And actually many of the approaches we're using in AI, you know, it's still the neural network paradigm. It's still deep learning. It is reliant on, you know, kind of this business model. If you were to cut off the data, if you were to shift the infrastructural requirements, say they have to meet a climate threshold, etc., you would not have this technology available. It is a derivative of the surveillance business model, not a significant new set of, you know, paradigm-shifting scientific innovations, but it's very convenient for those who benefit from it to narrate it that way. So, that was a long answer.

Well, yeah, I think that's a really good question. And in some sense, I want to back away from sort of the individual responsibility model of us needing to build up an immune system to this because, like, again, we just pointed at three companies that control the three major operating systems. Like that is not a product of our having been hoodwinked by a fantasy. That's like control and power that's wrapped in a narrative. Like they put Apple Intelligence into my iOS. It's very unlikely I'm going to stop using an iPhone because I can't participate in any part of my life without it. So there you go. They might have a story about that, but nonetheless, I think what is happening with the sort of turn to agentic AI is that, like, AI doesn't do what they say it does in a lot of cases, right? It's useful. It's, you know, like optimizing. It's great at finding patterns in data, but if the data is, I don't know what the patterns are, right? And this idea that it's going to turn into a helpful, solicitous robot butler is not like a fantasy that is augured in reality, right? So, but trillions, I don't know what's the other number bigger than trillion. I grew up poor. I don't know that math. But, you know, it's like so much capital is now invested in this fantasy, and like the S&P 500 is AI companies wrapped in a trench coat, right? Like we're looking at a global economy where investment capital, you know, the tech industry, nation states are leveraged against AI. And so there always needs to be a new future that is guaranteeing the present trajectory. So it, you know, the last thing, what were they promising? It was like LLMs was what we were talking about, and LLMs were cool, and like everyone needed an LLM, and they were sort of, ChatGPT was a chatbot, and then it like, agentic AI was like a year and a half ago when like the Davos conversation like switched to agentic AI, and it was everyone's talking about an agent. No one knows what they're talking about, right? But the idea, I think, you know, I think part of what is happening is you constantly have to have a renewed fantasy because the promises of, like, the promises of the future are what is guaranteeing, like guaranteeing our ability to continue on the pathway we're on. It's what's promised at the quarterly board meetings. It's what's in the investor prospectus. And so a lot of things that used to be called assistants, Siri or whatever, are now being called agents. And, you know, part of what is happening is there's a desperate need for return on this investment. Like revenue is coming in, but it's not enough to make a profit. And that consumer market fit hasn't been found yet. And so this sort of burrowing deeper into our infrastructure, like casting off the shackles of consent and privacy to enable sort of rampant AI systems, which is, you know, software, hardware, and a handful of models on some cloud server somewhere, to sort of do what they, you know, as much as they want, whatever they want, is, I think, you know, almost a desperate, not totally desperate, I don't predict like the bubble popping exactly in the near future. But I do think there is a, there's pressure right now to figure out profit and returns, and there is tech doing what tech has done for decades, which is sort of, you know, the future promise of almost working, allowing the present failures to persist as sort of, you know, one phase in inevitable progress toward a goal that is sort of, you know, always shifting, always almost happening, but never quite there.

Yeah. I mean, like, there's three people. Satya Nadella, Tim Cook, Sundar Pichai. Like, and I name that not to be like cute, right? But I'm like, we're looking at in some sense like the concentration of power means also focus. Like those are, you know, like in terms of the threat on the operating system here. Now there are other, you know, there's an ecosystem behind that that is sort of betting on this working. There's sort of a direction of travel. There's a herd mentality. There's a, you know, whatever. But, you know, ultimately when we're talking about the most existential threats to Signal, this is coming from the sort of vision of an operating system that no longer works for the developers of the apps and no longer works for the people who are kind of owning the system on which the operating system is running, but is now controlled by the mothership or, you know, the companies that are implementing the agents that are retraining the model to do something else, that suddenly decided that, like, you know, it's not going to respond to context this way, it's going to do it this way, that are sort of tinkering in the background in a way that is, you know, I think the danger is not just a Signal, right? This is, you know, pulling down all the...

power from the application layer. You know, this is a danger to Spotify. Like, what can Apple do with front door access to Spotify data on behalf of an agent that's going to make a playlist for your crush or whatever, in terms of, you know, Apple Music and competition, right? Like, we're looking at something pretty profound there. And so I think it's, you know, there is an odd coalition of shared interests here across the industry and, you know, in defense, you know, if you're using, you know, this is not just a privacy issue. This is also just a security disaster. Like, 10 years ago at Google, like, I would be laughed out of the room if I proposed anything close to this. Like, the security team would have been like, 'Are you crazy? Are you kidding? What's going on?' Now I think a lot of that is just being pushed aside in part because we're just in a, I don't know, intellectually, I don't know what's happened, but it's like it's gotten a lot less rigorous, let me tell you.

And I think this sort of, you know, phantasmagoria of like the AI godhead and all of this, and then the market pressure on it means that, you know, you're just seeing circumventing security and privacy protocols in ways that are like reported out to me from people in these companies that are like, 'Yeah, I didn't know about that.' And I'm like, 'Wait, you head security for that?' Like, and that was just sort of rushed out. So there's an AI exceptionalism that is pretty dangerous here, to be quick about that. Thank you.

I think on the individual responsibility, like, I gave a talk at CCC last year that I'm like continuing to develop, but I'm going to bring up something from that talk here, which is this concept that it's growing like bigger and bigger in my sort of theoretical toolbox, and it's like intellectual shame, and this idea of sort of intellectual shame around particularly knowledge about tech and how I see that operating here.

And I first started toying with this kind of concept because I was like watching it in my own life, right? I sort of was shunted into Google from a like literature and rhetoric, which is critical theory, what they call it at Berkeley, like background. And I was shunted into Google because they paid the most and hired me first. Like, I wasn't like, I didn't get an Atari when I was six and then everything was computers. That's not like, I just needed a job. And I remember like I had like kind of been good at being smart in class and that was like a, you know, it's an instrumental skill. It's not like a value, right? Like I was just good at tests and I could figure out that hustle. And so suddenly I was in Google and I was like, whoa, there's a whole other intellectual hierarchy that is working here that's really strange. It's like if you know this very narrow set of disciplines, you're like, you know, assumed to have knowledge about every other part of the world and that's actually like a value judgment on you as a capable human. And like that's a really weird experience that like sensitized me to this. And so I had my own version of this like intellectual shame where I was like going home at night to like read the man pages and like get, you know, Google the weird standards bodies and like learn how to program and taking these god-awful like JavaScript webinars on like, you know, like just because I was like, there, I don't want to be left out of what the boys know because then I don't get to like have an opinion. Then I don't like get to engage in a way that is fulfilling to me. And so I was like, I learned on the back of that. And sorry, I take a long time with these answers, but these are questions worth thinking about. And then I started seeing it sort of operate beyond Google as tech became hegemonic, right? And it was this idea that if you didn't know about tech, that was somehow like a deficit in your character and who you were. And I was like, what's going on here with this particular topic area? Because like a lot of people don't know about critical theory, but they don't walk around feeling like they don't have an opinion about things, right? Or are not allowed to have an opinion, or are somehow like, you know, lesser for it. And then I started seeing it like particularly among like people who had power and standing as I moved into those rooms, right? And it would be this weird thing where somebody like a world leader or someone like a captain of industry would suddenly start like fumbling and be like, I'm a lite, I'm an idiot, like I, you know, like I can't navigate a poorly designed UX. I must be stupid. And it was this strange like acceptance of tech as somehow like, you know, the arbiter of intelligence and as like an idea that it was very humiliating not to know about tech. And I think, you know, part of that is because tech is the, you know, episteme, to use that word from critical theory. The like, you know, it is the sort of technology of control in our world right now. So not knowing about that is sort of to admit that you are not one of the controllers, you are maybe one of the subjects, not, you know, the people being controlled. I think that gets, you know, we can get into patriarchy, we can get a lot of things there, right? There's, I'm not pulling on that string because I'm having a lot of restraint right now. But, and I think it's, you know, tech is also sort of ephemeral when we're talking about computational tech, right? It's code that runs behind the scenes. You know, you can read the code, but that doesn't quite tell you the like the weird bugs that it's going to sort of like manifest when it runs on like hardware that like, you know, if you don't know BASIC, you can't really get into that, right? There's a sort of, you know, complex systems are hard to scrutinize and yet they have so many effects on, you know, the rest of the world and then they're hidden behind, you know, corporate secrecy so we can't even see the code bases. And so there's a sort of like ephemerality to it that I think means it's easier not to know about. It's hard to know about tech. I've spent 20 years trying to know this stuff and like, so me knowing about AI, like every day that was like a struggle of having to wake up and like read that stuff in a sense because I was driven by some of this. And I think, you know, I think we're now in a world where suddenly the nervous system of everything is tech, it's controlled by a handful of companies and it's not, you know, like, and people are really embarrassed about suddenly about not knowing that, about not being part of that, you know, that culture. And you see this in venture capital, you see this around like, you know, investors investing in something that everyone in the room is sort of assuring everyone else they know about but not actually sort of understanding. You know, and I think it's, you know, so in a sense we're going to have to work against, you know, something that I think is sort of deeply rooted in the culture of tech and domination, the idea that like, you know, like ignorance is somehow shameful, ignorance of tech is particularly shameful, and, you know, that we should be asking basic questions that make us look stupid, like what is AI, in a room full of people who'd be like, come on bro, what the, if we ask that question and then we would be left out and then we would feel bad and then, you know, like we would go home and we would be like, oh, maybe I'm not going to be let in on that fund, maybe I'm not going to be asked to found that company or whatever, you know, there are real stakes for that. So that was long, but like there's something there that I think we need to look at more carefully around the sort of psychological and sort of broader social dynamics of like what tech means and the role it's taken on in terms of identity and knowledge production.