Hi and welcome to Futurology. I'm your host Don Nakagawa. Today we have a conversation between Niels Gilman and Meredith Whitaker, the president of Signal. Meredith is a different kind of Silicon Valley leader. She's not a tech utopianist. She actually became the head of Signal, which is the premier privacy platform offering end-to-end encryption for all of its users. And she came into that position and developed, I think, this company to a large extent out of concern about digital surveillance. We live in a world where privacy is increasingly a luxury. Our every step, our every breath, our every query is captured as data by large tech monopolies in a data center or in the cloud somewhere. In this episode, Meredith speaks to Niels Gilman about the threat to democracy posed by tech monopolies. So with that, let's go to the episode.
Meredith Whitaker, welcome to the Futurology podcast.
Delighted to be here, Niels. How are you doing?
Good. I've been really looking forward to this conversation for a long time. You've been a prominent voice and critic of the tech industry for the last 10 years plus. And I think part of your credibility in that is that you actually started out inside the industry. Tell us a little bit about that story and initially what you were doing when you were working at Google, and then how that got you to the interest in power politics and what the role of policy should be in the tech space.
Yeah. I mean, life is not a story, to paraphrase Simon Critchley. But I, you know, maybe I'll schematize it. I came out of arts. I came out of literature. I have a degree in literature and rhetoric from Berkeley. And I had no plans to join the tech industry. No one gave me an Atari when I was six. I wasn't interested in computers. I was really interested in art and literature and, you know, remained so. But out of college I needed to make money and Google was the first place that hired me. So that's my origin story. And I ended up getting hired at Google in 2006. I entered into an entry-level temp role in customer support.
And I was like, what the heck is this place? It was so strange culturally and otherwise. It was extraordinarily rich, but also kind of shabby in a way that was fascinating to me. Like, it was almost a flex to be a little schlubby, but at the same time, everyone had more money and more free time than I'd ever seen in my life. And I became really fascinated both with the culture and with the sort of certainty and grandiosity of a lot of the claims that were underwriting it, right? So, you know, like organizing the world's information, like that seems really cool, but what does it actually mean? How does it work? How do I get purchase in the rooms where people are saying that so confidently? Right? And so I think in some part I started by being such an extraordinary outsider that I didn't know if I was asking dumb questions. I didn't know that the check I was trying to cash wasn't actually made out to me. I didn't realize that, you know, certain parts, you know, like I was constantly engaging in cultural taboos and sort of crossing red lines. But it ended up working for me because I ended up, well, working for me insofar as like I'm here. And I guess that worked. I don't know. Hi. But you know, it's one story. And you know, this was at a time where, you know, Google was smaller than Yahoo as measured by monthly active users. This is a time where it was, you know, still there was still some purchase on the claim that it was a sort of aspirational startup even though it was making, you know, huge amounts of money, you know, as the pioneer in the surveillance advertising business. And you know, the sort of virtuousness of that narrative was just Teflon at that time. So I was really, you know, like I was asking a lot of dumb questions. I figured out how to get from customer support into engineering. I kind of hustled my way into there. I ended up starting, you know, co-founding a project that basically built the benchmark by which you could measure net neutrality by measuring internet performance across a number of different, you know, perspectives globally. I didn't realize that was such a hard project when I co-founded it. We ended up succeeding. That bought me a lot of credit within the company and otherwise that I used to start my own research group. We did a bunch of work on privacy. We did a bunch of work on security. You know, how do we fund core infrastructures on which sort of web computation relies? I got really interested in AI fairly early when it was first becoming a thing or like when deep learning emerged, re-emerged as a paradigm in the early 2010s. And I was in part very sensitized to that because I had co-founded and run this large-scale measurement project. And so I was really sensitive to the methodological hurdles of trying to create data that you can claim is a reliable proxy for reality and just how ultimately political, subjective, partial, fallible data is, how it certainly isn't a synonym for fact. Even if you're measuring at, you know, a very low level of the stack in a very sort of, you know, technical way, you're still encountering, you know, choices and contingencies and limitations on what that data can do, what it can mean, and how your infrastructure and systems ultimately affect the shape of it. So then along comes AI and, well, we called it machine learning at that point. And I was like, wait, so you're taking like weird social data measured ultimately or, you know, created via measurement to satisfy the questions asked by advertisers for whom you're using it to shape models of like types of people that you charge them to reach, right? And you're saying that's a synonym for intelligence. You're saying that you can sort of build models of the world based on that. And that was, you know, one of the reasons I was kind of early to a type of critical concern which I would say is, you know, not critical like, you know, dismissive, but critical like we need to ask fundamental questions about the materiality and shape of these tools so that we can govern them and understand them in a way that is reasonable and can be, you know, create a future we want instead of a future we don't.
So, you know, I want to go back to the fact that you majored or you studied rhetoric at Berkeley because I think a lot of people will hear the word rhetoric and they think that this means you studied how to give speeches, which is part of the rhetoric major. My brother was a rhetoric major at Berkeley.
I mean, I think the real...
You're a good talker, lady.
You are a good talker. Well, you know, but actually the rhetoric major at Berkeley, what they actually are teaching you how to do to a large extent is really do close analysis of how, among other things, power operates through text and how text gets used in all sorts of ways that are not overtly what they're about. You're taught to interrogate the use of text. Obviously what, you know, one is doing ultimately with computers is one is interrogating text. And so I'm wondering whether the fact that you majored in rhetoric ultimately, while it maybe made you an outsider in some ways, actually maybe gave you an edge internally in terms of the way you thought about the problems that you were encountering at Google.
I mean, I think it could have. I don't... it's hard, you know, it's hard to account for the water you swim in. It's hard to account for consciousness. I think, you know, when I was at Berkeley, and again I'm an undergrad there, right? So, yeah, you know, it was post-structuralism was the meme, right? It was the thing. You know, I'm reading a lot of Derrida. I was like, you know, psychoanalysis was sort of, you know, as a sort of critical discipline. I was reading a lot of Freud, a lot of Lacan, was, you know, reading Foucault. And I don't know that I understood most of them, right? Like it was almost like an affective attraction I had. Like I loved close reading. I loved banging my head up against the text. It was, you know, like almost a sort of, you know, it was difficult, but it was, you know, there was something there. And I could sort of see, you know, not just through the text, but through the like social and intellectual environment it was cultivating. Like these people were weird. They were smart. There was like something that attracted me to it. But I don't know that it was until I'd been at Google for a while that I started to sort of back-propagate a lot of that understanding and be like, 'Oh, power-knowledge. I see like there's, you know, like I'm, you know, my hands are around it. There's a, you know, I'm seeing it happen. And I'm seeing sort of like epistemic authority translated into a kind of common sense around like what is and is not sort of acceptable, verifiable understanding of the world.'
Immaterialized and naturalized through the systems that we're interacting with.
Exactly. Which are then invisibilized. Right. And so I almost think like those two pieces fit together for me. And I really needed the embodied almost like, you know, like labor experience to then sort of really appreciate what I had studied. You know, like I hadn't read a bunch of Hegel. So there's a lot of that that you're just not going to get until you sort of dig into the primary sources that are underlying a lot of that sort of critical theoretical literature. But, you know, I do think still close reading is the Earth skill.
Yeah. And like I will say, you lose that by AI summarizing at your peril because if you can't like sit down and really sit with a text, which is really how you can learn anything and do anything, then you're at the mercy of like whatever says it can do it for you. And I think that's, you know, like if I'm giving one piece of advice, I'm just like you learn to closely read because from there you can do almost anything.
I totally agree with that. In fact, I just published a piece in Noema, which is the Berggruen Institute magazine.
Thank you. Yeah. Like a month ago about how there's a paradox about the rise of LLMs and the AI, which is that all these computer programs have built these machines that are supposedly going to render all these particular skills that people have in various jobs, particularly their skills, more and more redundant. And I actually think that the skill that will remain is exactly the one that you just described, the ability to closely read and the ability to interrogate texts. And that of course is like a core skill of the liberal arts. So one of the paradoxes or surprises I think of the age we're entering into may be that actually the classic liberal arts, by which I don't just mean humanities, but the ability to cross-fertilize across different disciplines, understand the relation between art and science, understand the difference, you know, the relationship between quantitative and qualitative methods and all of those things, and that ability to see across all those things may in fact become relatively more valuable as simply producing text becomes commodified.
Yeah, I mean 100%. Because like what are we doing it for, right? Are we producing text so we can produce verbose text? No, ultimately there is a thing we want to do in the world or we assume, right? And you know, in some case I think that has been almost abstracted away because so many people have like waged work jobs where like yeah, you have to turn in the report because you have to turn in the report. No one's going to read it. No one's going to, no one cares. And so like there is, you know, like a... we can hold space for the idea that AI can replace that, you know, like a McKinsey report or something, right? But you know, ultimately like does the thing do the thing we want it to do requires contextual analysis in a dynamic reality that can't be just, you know, predicted from a bunch of text that was fed into a machine. You can sort of gesture at the, you know, potential of a like prediction, but it, you know, you will need somebody who can say like that is commensurate with this but actually doesn't work here and you better know that this guy is sensitive to that or you're going to hit this tripwire and then that's not going to move forward because the guy's never going to like, you know, like whatever, whatever, and you're going to have to know, you know, like there is a set of contextual cues that will ultimately be really important in, you know, written text and just being able to read the text of the world around you and the way social and political and economic relationships sort of, you know, contribute to X, Y and Z, which, you know, I guess like the soft skills maybe, but they're, you know, they're the skills that make things either happen or stall.
Totally. I mean, so understanding those kinds of human relationships inside organizations or across organizations and how you can get people to work together effectively, I think that's going to continue to be huge. This is maybe a segue to talk about, you know, how you became more overtly involved in sort of political activism at Google specifically, because that in some ways, I mean, you were, I think, a well-known person inside Google and the tech world, but this is really where a moment where you sort of became much more prominent in the public sphere.
Yeah. I mean this was a moment where we sort of decided we needed a media strategy, right? And before then I was sort of, you know, prominent within the company and then had my sort of niche outside within the discipline of people sort of thinking about and theorizing AI. But at that time, we're talking like 2016, 2017, that was still pretty niche, right? It wasn't, you know, like I don't know, it wasn't AI everywhere the same way it is post-ChatGPT. And it was, you know, I don't know, we've seen sort of the showdown with Anthropic and the DoD, you basically put into very stark relief a lot of the issues and a lot of the questions that we as a group of people, many of us coming from the AI research space at Google, were asking and getting unsatisfactory answers to back then. And this is, you know, I don't know if the listeners would be familiar, but there's sort of a history to the moment we're in right now and particularly the move of the large tech monopolies into defense contracting and becoming kind of, you know, core geopolitical assets for defense and, you know, I would say geopolitical power for the US state in general. And there's, you know, there's something called the Third Offset Strategy that was around like 2014, 2015. And that was basically like, you know, hey, you know, let's recoup some of the language of the Cold War. And the sort of offset we have is just technological integration into our military capabilities and technology being sort of the computational network computation and the products and affordances coming out of Silicon Valley with the idea that we need to move those into the military to kind of beef up our ability to compete with Russia and China.
So just explain to our listeners what the first and second offset were relative to the third offset. There was nuclear was the first, I believe, and it was like we need nuclear capabilities.
It was an offset in the sense that it would give disproportionate power to the United States.
An advantage, right, that would then...
And that could not be easily matched is the idea. So like, and then would forestall the enemy and sort of, you know, dispatching their weapons because we had this advantage. They would know we had this advantage. So we're sort of staving off the possibility of like nuclear annihilation by having this offset. And this, I don't actually know what the tech, you know, in the third offset in some sense we're dealing with like borrowed terminology that's a little sloppily applied, but the third offset being like, you know, China and Russia are rising threats and the US will sort of stave off their threat, you know, they're making good on these threats by integrating this capability that we have sort of an advantage in, which is the sort of, you know, tech giants, their infrastructure, their products, the, you know, internet business model into our military capabilities to XYZ. And that sort of has moved into kind of AI as, you know, the center of tech, the center of everything. But this is around 2015, 2014 when I started hearing this. Then there was the Defense Innovation Unit, which was kind of a very self-aware strategy to move Silicon Valley companies, you know, Google in particular, closer to being military contractors with Project Maven, which is still ongoing, being this sort of first contract that was signed between Google and the Department of Defense to build drone targeting and surveillance artificial intelligence systems for the US military. And so, you know, the questions we were asking then is like, you know, okay, what does it mean for a massive multinational corporation that has gained access to create and control so much, an unspeakable quantity of intimate information about a global population, not just US citizens but including US citizens, to be yoked to one nation's military capabilities? You know, what, you know, when a surveillance giant is suddenly a, you know, a tacit arm of the US military. How do we think about that? How do we think about democratic control?
How do you think we should think about that and think about democratic control?
I think it's pretty terrifying. I mean, I think like, let's be real. One of the reasons that Project Maven doesn't have intimate population-level data on any user of Gmail, Search, Google Pay, etc., Maps is because Google ultimately cancelled that contract, right, because of massive, massive employee pressure. And that's, we can talk about, you know, Anthropic saying no mass surveillance of US citizens, but we're not just talking about US citizens. And we're talking about a set of companies that as shareholder-driven corporations will sort of follow the political winds because their objective function is growth and revenue increasing over time. So this, you know, this was what I saw for years. I started there in 2006, right? So I like went through the Obama election. I went through the Trump election and every presidential election, you see kind of the same thing. They clear the policy shop, right? Everyone who was sort of close to the old administration is, you know, pushed out. You know, they go work for K Street and then they bring in, you know, like who's the person who's sleeping with the chief of staff? Who is the person who goes to the baby shower of the press secretary, right? Because that proximity is how you gain purchase and how you figure out what type of policy positions and other, you know, kind of positions to take to get as much advantage as possible. And so we have a set of actors who are sort of, you know, incentivized by the surveillance business model and who are also incentivized to, you know, bend and shape themselves to whatever the political winds are with the type of data that is unprecedented in human history and now the technological capabilities to synthesize and weaponize that data at a scale we've never seen. So I don't know, what do you think about it? Because like, it's not a heterodox position, I would say, once the facts are laid out.
No, I don't think so either. I mean, I guess one thing I would add though is that I think the tech companies face what's known in international relations theory as a security dilemma, which is that if I don't do it, somebody else is going to do it and they're going to steal the march on me. And in this case, they're worried perhaps fictitiously about foreign companies doing it instead of them. But in the case of the DoD contracts, they figure if they don't do this, some other startup company will do it and steal that opportunity from them.
Yes. And I think we need to like do a cross-section of the political economy here and what it means to be a, you know, big tech monopoly versus a startup that's sort of bolted onto the side of one of these monopolies. And this was, I was at CFR the other day sort of having this conversation and this is one of the points that I think, well, I felt it was important to make in that room and I will make it here. I think, you know, when you look at the Anthropic DoD dispute, you know, the reason Anthropic was so important to the, sorry, DoD, whatever, was not because of the model necessarily, right? Like Claude is a capable model as measured by one or another axis, sure, but Claude is not actually, you know, deeply integrated into the Maven system that is now sort of run by Palantir, right? Claude is a, you know, an LLM that's used for synthesis and search, but the Maven system is sort of, you know, integrated into, and the Maven system runs on AWS. AWS is the system that runs most of the classified workloads and it's part of the Joint Warfighting Cloud contract, this big contract that sort of, you know, licenses cloud compute from primarily Amazon, but also Google and Microsoft, with Claude being the one large model that runs on AWS. Right? So change it. What we were actually talking about was dependency on these cloud giants, one of which happened to have one of the models from the sort of AI startups integrated into it, and pulling that out and replacing it would be a giant pain in the ass. And so when we look at how many cloud giants are there, how many startups could replace those? There are three, right? And there are three that cross-subsidize these massive computational footholds which control sort of standards for, you know, AI training frameworks, whether that's like, what is it, Trillium, I got that slightly wrong, but the Amazon sort of framework or, you know, PyTorch or TensorFlow, you know, whatever it is, that have, you know, kind of set the labor practices for what it means to deploy across sort of multi-region clouds that then cross-subsidize those by, you know, kind of platform economies, whether that's social media with, you know, supported by surveillance advertising or the Amazon, you know, marketplace, and then are using that infrastructural and data advantage, data from the platforms, infrastructure because they're able to sort of, you know, subsidize that through their, you know, platform business models, those advantages then allow them to sort of train and deploy AI. So we're not talking about a political economic structure where you can have like a new entrant who can just do it, right? We're talking about an extraordinarily concentrated monopolistic business model where there's only a handful of actors ultimately that can sort of nose-to-tail create and then deploy these systems, right? So I don't know if that, I mean, that comparison holds somewhat, right? Like, but Anthropic, you know, when they're getting this new investment from Amazon and from Google, that investment is coming in the form of compute, right? It's basically we will trade you access to Anthropic shares if you give us more compute to train and deploy, right? Anthropic is still a barnacle attached to the hull of these big tech infrastructure giants, not an actor who's actively competing, you know, on the same terms as an Amazon or a Google.
Right. I think the other thing that's really worth emphasizing about your description of this, which is really important, is that it's all super tightly coupled.
Yep. Right. And so tearing out any one piece of it, I mean, I think that's why people at DoD were so pissed about Anthropic is they're like, 'My god, we're going to have to like, there's going to be all this spaghetti splatter if we try to tear this thing out. Forget it. Why are you being such a pain in the neck? We can just replace you. It's just going to be a huge amount of work.' And so, you know, now the other question for me, you know, you also raised this issue about like, okay, it's one thing to say we're going to have some restrictions about like not allowing these companies that are deploying for defense and intelligence purposes using data from Americans. But, you know, that's only 5% of the world population or 6% of the world population is American. And they're, you know, now, of course, many of these companies are excluded from China and excluded from some other markets, but still like there's not even a debate in this country right now about whether there's some ethical questions about when they're surveilling, I don't know, Frenchmen or, you know, Mexicans or what have you. People don't care about that at all in the political debate in this country.
Yeah. And I mean, let's be real, like I don't see the FISA court jumping up to restrain the purchase of ad broker data by the US state, right? At a time when like one of the places where these LLMs are highly capable is the synthesis and search of massive amounts of data which has newly revitalized the utility of a lot of surveillance data that otherwise was probably like sitting in databases ultimately not that usable because you would have to get human analysts in there to like munge around, right? So you're, you know, you're seeing the capacity for the use and abuse of, you know, musty old data supercharged while the appetite to do anything about that kind of weaponization has, you know, all, you know, it's like Wyden occasionally standing up and saying something, but like the idea that we're going to pass any legislation that means anything is fairly laughable to anyone who's paying attention to the actual process right now. Let's talk a little bit about your definition of privacy. Because the way you just described this, Meredith, you know, one of the concerns that I have when I look at where things are going in terms of privacy is it used to be that one had all sorts of privacies that were based on the fact that there were all these disaggregated databases in different places. And so, you know, you had de facto security through obscurity because the...
Left hand didn't know what the right hand was doing. And all these databases were scattered all over the place and your credit card information and your telephone information and your, you know, phone tracking you as you walk around New York City. There's no real way to integrate those things. Now, that can be done almost trivially if one has access to the databases, which increasingly one does. So tell me a little bit how you think the ability to do these, you know, large joins at scale is changing the entire security and privacy environment that we're living in.
I mean, I almost feel like you answered that question, right? It's, you know, increasingly trivially possible to anyone with access to this data to, you know, whether that data is correct or not, which is actually the key point that I'll get to in sort of this definition of privacy that I anguish over a bit. It's really like, you know, it's incredibly intimate information about you or at least an intimate profile about you available not only to advertisers and corporations but to the state and foreign states. We know that China buys data broker data. We know that anyone who wants it buys it because it's on the open market. And so, you know, the ability to, you know, I mean, let's be real, the ability for those who have power over you, if they so choose, to access intimate information about you or purportedly about you and weaponize that to do you harm, has never been greater in human history. And you know, there's a little thought exercise that I sometimes do with audiences who are sort of prone to the like 'I don't have anything to hide' type of reflexive response, which I think is a bit of a like self-soothing delusion. But, you know, nonetheless,
Imagine every text you've ever written in your life in a database somewhere and someone walks you into a room and they say like, 'Yo, here's a link. You click this link. You get access to this database. It has every text you've ever written, right, to everyone available to anyone who clicks on that link. And it's instrumented with an LLM, so you can search that. People can name search, they can theme search, they can find, you know, like when did you talk about X, Y, and Z? Everything, every lie you told, every shitty little thing you said, every offing you were doing, every, you know, I hate my boss text to your colleague, whatever it was, right? All available there. What would you not be willing to do to prevent that being sent to everyone you know?'
Yeah.
QED, like there is not a person in the world. This is not sort of a like you're a bad person. It's that we have a variegated life. We have different relationships. Our minds change. Sometimes we need to just talk a little and get it out of our system. Our relationship with our best friend is not the relationship we have with our therapist or our oncologist, right? And those things are, you know, like they're different and they're mediated differently. And that is part of having a dignified and full life. Part of being able to explore ideas and backtrack and figure out what we actually think and community with others safely from the gaze of our boss or our teacher or whoever.
Like that's part of doing life and it's basically going away. Well, it's under significant threat. And I think that, you know, that's and it's under significant threat in a context where what was sort of, you know, the normal affordances and rights of, you know, regular life are being curtailed or narrowed, right? Yeah, there's a, you know, proposed bill that, you know, wasn't brought forward, but that sending a link that includes instructions for abortion care would be made a felony, right? So, okay, you know, maybe 20 years ago, you would be more comfortable to send that anyway because who would know, right? And you could be anonymous. You know, no one knows you're a dog on the internet. Well, those days are gone, right? We're age-gating everything. You know, we have these capabilities to track everything. You know, criminality is just whatever, you know, whoever in power defines as criminal, right? And we're watching those definitions shift. So, I think we also need to be very conscious of that. There's, you know, there's a man in Tennessee who just got out of jail for 39 days because he posted things that were critical of Charlie Kirk in the aftermath of Charlie Kirk's killing, right? Right. You know, we're dealing with some thresholds that are being crossed in the context of that type of information being more and more and more widely available and weaponizable.
Yeah, I mean, that case was particularly egregious because he was expressing his opinion, but let me pose a slightly more complicated one to you. So the big fires that happened in Los Angeles back in January, two Januaries ago, a year and a half ago, there's a man who was arrested in Florida for allegedly starting these fires. And as you probably know, one of the pieces of evidence that's been used in his indictment has been some of the conversations he's had, I believe it was with ChatGPT, with some LLM, that were used to establish his supposedly pyromaniacal state of mind, his mens rea. And they specifically brought up two pieces of evidence in the indictment. One, so these fires happen in early January. One piece of evidence from the chat logs was a prompt he gave back in July, so two years ago now, where he asked for the LLM to create an image of Los Angeles burning and poor people running past a gated community with the rich people laughing at them. Okay. And this was supposedly used to establish that he's got a pyromaniacal state of mind. And then the second one is the evening that the fire started. He's geolocated with his phone as being very close to where the fire started. And then he asks ChatGPT, 'How do I, can I be held liable if I start a fire?' And that too was used. Now I think the question that I would ask is, do you think people should have an expectation or a right to privacy? To the government, you know, you have no right to privacy at all with your LLM conversations today. And a lot of people today are using LLMs as therapists, for medical advice, for legal advice. And even though the LLM will always tell you, 'I'm not a therapist, I'm not a doctor, I'm not a lawyer, I'm not your spouse,' categories that have legal privilege around them, people are having intimate kinds of conversations and revealing intimate information about themselves to those platforms. Do you think that people should have a right to privacy with respect to that?
Absolutely. I don't, you know, I don't think we can have a democracy or beyond that have an expectation of a good and full life that includes creativity and intimacy and the ability to sort of transform and iterate on our ideas about the world together without privacy.
You know, I agree with you totally on this, but I hear lots of people who say, 'No, you should just, if you want privacy, go to your therapist, go to your doctor.' And I say, why should people have to pay $200 an hour to get privacy to talk about these things?
I mean, that, okay, so privacy is just a clinical right between me and my therapist who's then writing down our notes on an iPad that goes up to the cloud anyway. Like, that's not what privacy is. Privacy is the ability to have intimate or, you know, exclusive communication with one person or a group of people that is not subject to the weaponization and abuse of those who have power over us.
Right. I agree with you about this too. Now, here's where I think we may get to a disagreement. Meredith,
Let's do it. Let's do it.
