Hi, I'm Taylor Owen from the Globe and Mail. This is Machines Like Us.
So, I've just spent the last two weeks in conversations about the shooting in Tumblr Ridge. As I'm sure most of you already know, on February 10th, an 18-year-old opened fire on a local high school, killing eight people and leaving more than two dozen injured.
Now, I'm not an expert on this kind of violence, but the reason reporters and governments wanted to talk to me was because of what the shooter had been doing before the attack. According to several people at OpenAI, the perpetrator's ChatGPT account had been flagged and eventually banned for using their models, quote, 'in furtherance of violent activities.' But OpenAI chose not to inform law enforcement. And now in the aftermath of one of the deadliest shootings in our country's history, Canadians are asking why not?
Look, leaving these decisions in the hands of a company like OpenAI is incredibly misguided and frankly dangerous. But the narrative that's emerged over the last month, this idea that AI companies should be reporting all violent conversations to police, is much more complicated than it sounds, and it has very real implications for our collective privacy.
So, I wanted to talk to Meredith Whitaker. Whitaker is the president of Signal, the encrypted messaging app used by journalists, dissidents, and governments all over the world. Signal doesn't collect your data or serve you ads or track who you're talking to. When authorities ask Signal to hand over their user data, their response is simple: we don't have any.
That means Whitaker oversees the most private messaging platform on the planet. But it also means that there's almost certainly illegal activity happening on Signal that no one, including Signal, has any knowledge of.
So, the conversation you're about to hear isn't just about Tumblr Ridge. The idea that we should be willing to trade privacy for safety has permeated the entire technology sector. Amazon recently rolled out a feature that lets you scan in entire neighborhoods worth of Ring cameras. And countries like Australia and Denmark have introduced laws requiring teenagers to verify their ages before they can use social media.
I want to be clear: all these technologies have real value. But we need to be equally clear-eyed about what they ask you to give up in return. And I wanted to ask Whitaker why she thinks that trade, your privacy for your safety, might not be worth it.
Meredith Whitaker, welcome to the show.
Happy to be here. Thank you.
I want to start with an incident that a lot of Canadians, frankly, are thinking about right now. A few weeks ago, there was a school shooting in a small town called Tumblr Ridge, BC. And it turned out, we learned last week, that the shooter had been talking to ChatGPT about the shooting and that that conversation was flagged somehow to people within the company, 12 of whom discussed it and urged the company to report that conversation to law enforcement, but then they ultimately decided not to. This happened eight months ago. So, seven months before the incident occurred.
I want to start here because the conversation in Canada has gone immediately to, well, they should have, and that there should be mandatory flagging of these kinds of conversations. Do you think that's where this conversation should be going?
Well, that is quite a nexus to unpack. I think, you know, school shootings and the rampant occurrence of school shootings in the US, and, you know, now expanding to Canada and other places, is horrifying. And I think there is a, you know, reflex to do something after these school shootings, and I think we often, you know, get fixated on sort of technical solutions, hoping that that will cure really gnarly social problems.
And, you know, I think mandatory flagging of conversations — I don't have information about how many other conversations are flagged. How long does it take law enforcement to act on those? What types of terms would be flagged in such conversations? How would those impact potentially marginalized people or people who are engaging in activities that under one administration may have been legal and under another are criminalized? Who gets to decide what is done with those flags? And really importantly here, what other sources of information are available? What other levers are available to law enforcement?
And I think in particular about, you know, some of these debates around you that Signal gets implicated in, where if you zoom out you recognize that, oh, you know, a lack of data isn't often the problem. It's a lack of resources to pursue these leads. It's a lack of social services that would, for example, here hypothetically, have flagged this troubled person. What is a holistic approach to what is a horrifying social problem? You know, gun control comes to mind in the US. So, I am very, very cautious around things like mandatory flagging.
I think what this does tell us is that everything you send to a ChatGPT server is in the hands of, you know, OpenAI, potentially Microsoft, and that that is being reviewed by people, and that that, you know, they could turn that over to the Canadian government, and then, you know, perhaps that would — and they are — to the US government, the US government, immigration control, you know, another regime that pushes on them for that. And that does raise awareness around some of the potential consequences of these systems and the type of centralized control and insight that they give those who run and operate them.
What may some of the risks of this mandatory flagging be from your perspective? You know, I want to bring it to something concrete for Signal, and that's, you know, in the EU there is this legislative proposal that is in the name of protecting children, which we all fervently agree with, would mandate scanning private messages and undermining end-to-end encryption, which is the technology we have in a world where surveillance is ubiquitous to protect the fundamental human rights to private communication.
And, you know, the danger there is simply that the definition of what is and is not permissible shifts. We have seen abuse of surveillance powers throughout history, but we have never seen surveillance powers that are as pervasive as what the large tech companies, with their huge amounts of data, their ability to instrument our social, economic, professional lives, now have in hand.
And we need to recognize that, you know, scanning private messages for criminal behavior could begin to extend into scanning private messages for dissent. We know that in the US, administrative subpoenas have been used to ask the tech companies, you know, very recently, for information and identification of people who have exercised their First Amendment rights to criticize the government and criticize its immigration policies.
We know that there's a woman who lived in jail and went to prison, was convicted because Meta turned over — we can say scanned if we want, but turned over her private messages with her daughter that showed her working to help her daughter access reproductive care and deal with the aftermath in the state of Nebraska after the Dobbs decision, which had enabled Nebraska to criminalize what was a formerly legal procedure.
And so I do think we need to understand that a world in which we can't communicate freely is a world in which we can't dissent, is a world in which we can't live beautiful, intimate, open lives in which we can't have new ideas. And that, you know, there is a — the Venn diagram is a bit of a circle when it comes to that and authoritarianism.
I want to come back to the types of conversations that happen with Signal and some of the regulatory environment around that, but I'm wondering if —
We don't know what types of conversations happen, but they also — we don't have access to the data.
But we do know, and that's important, right? Because Signal is encrypted and —
OpenAI chats, for example, are not. OpenAI chats, you are talking to a server controlled by OpenAI that has that data. They can do what they want with it.
And that feels like a material difference to me, because there's a number of things there that are going on. One is with Signal, there's an assumption that it's a conversation between two people with an assumption of privacy. With a chatbot, we're talking to a product that's creating content that might be encouraging us to engage with it like a person, which might be creating an expectation of privacy.
So, we open up and speak to it in a way as if we would with a person, but it's not a person. It's a product. So how should we think about privacy in that context? Should we expect privacy, or should we even think about it as a private space in any way?
I don't think it's reasonable to expect privacy. They're not built for privacy. Indeed, OpenAI would not be able to roll out advertisements if these were truly private. And I think, you know, it is concerning. You hit the nail on the head, in my view, that the simulation of an intimate conversation, a, you know, a sentient and patient interlocutor, lulls us into treating this as an intimate conversation, into, you know, perhaps disclosing or, you know, saying things we maybe don't even believe, but we want to, you know, engage in a way where we feel heard and testing out ideas.
And that then on the other end of that is not a loving and, you know, sentient interlocutor who is interested in our every word and encouraging our every dream. It's a large company that's outlaid a huge amount of capex that needs to capitalize on that, that can do what they want with that data, that is participating in, you know, the core business model of the tech industry, which is collecting as much data as possible and monetizing that.
But it's, you know, I think there is a — there's almost a psychological bait and switch, because if you look back at the ELIZA effect, which is, you know, it's well documented, the tendency for people to, you know, I'll use a psychoanalytic term like cathexis, to these systems, to project onto them.
And ELIZA was one of the first chatbots, a crude version of a chatbot. Not even really AI, right? Just a very crude system.
It was a command line interface that simulated a therapeutic session. So it was a decision tree, you know, nothing we would call AI. Now it was, you know, Joseph Weizenbaum at MIT created it, you know, a long time ago. And he wrote about his experience, you know, being troubled by the experience after creating it, on seeing a person he worked with ask him to shut the door because she was talking to ELIZA and wanted privacy for that interaction.
And, you know, even in this rudimentary system, which now we would scoff at, you know, we wouldn't have any kind of, you know, the sort of feelings for it, there was this, you know, we humans will go 90% of the way to fill in the gaps. And I think, you know, of course these chatbots are virtuosic. It really does feel like you're talking to someone, and it's very difficult to rein in, you know, the feelings and the sort of sense of ease and security that you have, because our experience with that is, you know, 99.99% talking to another human.
Right. And so we're bringing a norm from a previous world into a new technology which is ill-suited for it, I think, or mismatched with, or allows us to be taken advantage of in a way too.
But I wonder if one of the things then we might need to sacrifice, if we accept that these are de facto not private, is some of the utility that people get from them. Because people are doing that in many ways, are using it because they find some usefulness or value in using a chatbot in that way, but maybe that's just not aligned with the way the technology is designed, and we just might have to sacrifice that use case as a society.
Sure. I mean, I don't know what the lever of sacrifice is, how we do that. You know, I think social media was very good at capitalizing on the insight that people's favorite subject is themselves. And so this is, you know, this is kind of catnip, particularly in a world where, you know, we have stats up and down, people are more alienated, they work longer hours, the commutes are worse, finding friends is hard.
And so we're, you know, is that a willing sacrifice that we're putting onto individual consumers to say now, you know, grit your teeth and buckle up? Yes, we understand that, you know, the best sort of gambling psychologists in the world have worked to make this extremely irresistible, but you are a bad person if you don't resist it. Or, you know, like, how are we thinking about that mechanism actually working, you know, particularly when we think about, you know, more vulnerable, you know, I don't know, children. I think about my parents and they're being, you know, kind of old and, you know, very, very smart, but guileless when it comes to these things.
Well, now we're doing the same in a much more intimate way, where we're offloading that intimacy and that personal relationship rather than just the broad information flow that we were in and relationship. Can you think about how that can be weaponized? You know, particularly in the context of selling advertisements, which is, you know, it's a tiny hair between selling a product and then selling an ideology, selling a political perspective, selling a vision of the world that may or may not map to reality, but is convenient for one or another actor.
Do you think there's a version of this technology that does protect privacy that's possible, or is it just an inherently privacy-violating technology?
I would say yes and yes in some sense, right? I think it is absolutely possible to build AI systems that are much more privacy-preserving than what we are seeing today. Moxie Marlinspike, who is the founder of Signal, has just launched something called Confer, which is a, you know, it's a chatbot system, but the server provides privacy and uses encryption. So yes, you are talking to, you know, a server on the back end, but it is not going to AI, it is encrypted, it's not being disclosed. So there, you know, that's a way that you're certainly raising the privacy bar.
Do you think that's a good thing to exist?
I think it's more private, you know, than the alternative. But, you know, we also have to recognize that AI, this current deep learning paradigm of AI, which sort of became, you know, it's still what we're working with with LLMs and these large models, and sort of kicked into gear in the early 2010s, is built on a business model of surveillance.
And this is not, you know, this is not hyperbole. This is really just looking at the history of the tech industry and how it formed. You have, you know, you had large platforms that emerged out of the commercialization of the internet in the 1990s. They scaled. They, you know, built platforms, whether those social media platforms or marketplaces. They invested in large-scale computational infrastructure to gather and index data to serve people on those platforms. Those platforms gathered more data. There's a feedback loop there. Economies of scale were established.
And then in the early 2010s, it was, you know, they recognized that old deep learning approaches from the late 1980s, early 1990s were actually really useful when they were paired with huge amounts of data. And that really explains why the winners of the first wave of tech are also the winners of AI, because it requires these resources. Yes, you have OpenAI, but OpenAI is bolted onto Microsoft because Microsoft has Azure. You have Anthropic, but Anthropic has deals with Google, with Amazon, because that infrastructure is so important.
So yes, it, you know, relies on surveillance, this data. It has also produced a hunger for more data. If you see the sort of privacy laws being pushed back, if you see, you know, Amazon, you know, a few months ago announced that, you know, the Amazon Alexa is just going to send all your queries for AI training. We're not even giving a pretext to privacy, whereas in the past that would have been a much more controversial decision.
So, you know, all of that is just a material cross-section, like this is the technical reality of what this large-scale, deep learning-based AI does and how we need to think about it. You know, it is not saying that it is only a surveillance technology or that, you know, it is somehow not useful for other things, but our analysis needs to include this fundamental material understanding.
The core genesis of this current technology, as you said, is this vast amount of data and also this incredible compute needed to process and make sense of it. Both of those, and certainly the latter, requires an enormous capital investment. I wonder if that just creates a financial incentive that makes these inherently path-dependent towards a surveillance economy. I mean, I'm not sure how you get out of that, right? Like, how do you commoditize to the degree they need to without breaking privacy?
Yeah. Well, I think, you know, OpenAI announcing it will introduce ads is exactly that.
It's that pressure. It's that annual revenue requirement that increases every year.
Yeah.
You know, ads are not sexy. No one wants to see an ad. You know, they were roundly dragged, and Anthropic had some pretty elegant Super Bowl ads, you know, in response.
Sam Altman said himself he never wanted to do it because that would be gross, right? And —
Sam's good at the one-two, yes. No.
I wonder, is there a public option to this here? Like, is this just the problem of leaving all of this to four or five large American companies? Like, is that the root cause of this problem, and we should be thinking much more creatively about financing AI?
Well, I think we should be thinking much more creatively about financing, but I think we, you know, like what I don't want to do is get stuck into a technologically deterministic trap. Like, you know, we want exactly these systems, like a chatbot-based AI economy, which, you know, chatbots to me are, you know, they're virtuosic, but they're kind of boring, right? And everyone is just obsessed with chatbots. And there's so much more you can do with AI. There's so much more you can do with tech that gets kind of left in the dark because everyone is just indexing on this one form.
And, you know, so, you know, one, the question is, what do we want this technology to do to create the world we want? That's a political question, that is not a technical question. And the question of what world we want is the, you know, core question of, you know, politics and democracy and other forms of citizenship.
I think, you know, in some sense it's like, let's take back the agency to make that determination, right? What do we want our schools to look like? Do we want, you know, like, are healthy lunches for all children better for the health of our youth and society than, say, you know, a license with a Meta educational chatbot, right? Like, how are we weighing these things?
But the goal there is healthy youth and society, right? A pedagogy that enriches, that, you know, allows people to live full, joyful lives, contributing and, you know, being included, all of that, right? What meets that goal? And then what technological investments would help get there? What technology aids us on that path? You know, how do we govern it? Because I don't think we have spent much time thinking about how we govern this type of technology.
Okay, we've built tech that is meant to ensure, you know, this healthy youth and society goal for our education system. How are we determining that the code is actually doing that? All of those to me are almost more interesting questions than, like, who should pay for it. It's like the it, and what is the it doing, and how do we make sure what it's doing is beneficial and legit.
Yeah. And what these are doing is evolving really quickly. We've gone from pretty basic chatbots to agents that are out doing things in the world and online. Does anything concern you about that new capability?
Yeah. I mean, I think the coding agents in particular are doing some pretty — it's pretty impressive. You know, it's like a competent junior programmer that sometimes does weird stuff, but hey, who doesn't?
So, you know, I almost caveat that because, like, yeah, that's not the problem. The issue is that this sort of — again, agency is a bit of a funny word, or AI agent is a bit of a funny word, because it's not a technical term of art. It kind of indicates capacities, and ultimately, like, the two key features of what we're calling AI agents is that they, you know, have access to context, we call it, but, you know, context is your data, you know, as much information about your environment or your, you know, the, you know, whatever it is, apps you use, the folders on your computer, your login on your accounts online, your files, sheets, your performance reviews, whatever it is.
So access, pervasive access to data as much as possible. And then it has to have the capacity to act independently, right? An agent that checks in every three seconds like a cookie banner saying like, 'Is this okay? Is this okay? Is this okay?' is not actually an agent. So you can build agents that have less access to data. You could build agents that check in every step, but at some point they kind of lose the characteristic of agents, or they are simply not that useful.
And, you know, we've done a bit of an example, right? You have an agent that is running on your device, and it's an agent that's supposed to, you know, help you with administrative tasks. And I'm giving kind of a hypothetical marketing pitch, but this is the type of thing you hear from these companies. And, you know, you say, 'Okay, agent, I would like you to plan a birthday party for me and invite my four closest friends.' And that's cool, right? The agent can go off and do that. And maybe you get to, you know, go to Pilates class or, you know, I don't know, meditate, whatever you do.
But in order to do that, of course, the agent would need access to your browser. So driving your browser as if it were you, kind of faking it's your persona. It would need access to your credit card to book the restaurant. It would need access to your calendar. It would need access to your contact list. And in this case, if you're a dedicated Signal user, as I hope everyone listening is, it would need access to your Signal messages to text with your friends on your behalf, to see when they're available, to let them know about the reservation, and to schedule the time. And it's that access to Signal messages that in our case is deeply concerning.
