Why nobody is stopping Grok | Decoder

Thanks for having me on.

I have been in various tech policy-related roles at Stanford for a decade at this point, and I started off focusing on encryption policy. And now, way back in the mists of time during the Obama administration, it was usually terrorism that got invoked as the reason why we shouldn't be allowed to have strong encryption for our devices and for our communications. But then that rationale shifted to child safety, which turned out to be really effective in terms of policy and regulatory efforts.

And that meant that in addition to working on encryption policy, I had to start getting up to speed on child safety issues as well. And moving through different roles at Stanford brought me to working on a more trust and safety-focused role, which is what I called the last time I talked to you, Nilay. And now working full-time on AI policy because inevitably all tech policy in recent years has converged upon all AI all the time.

But as we're going to talk about, clearly a lot of the underlying trust and safety studies that I had been doing remain highly relevant now today. And so even in terms of where my research has focused, in the last couple of years, a lot of my research in the last couple of years has focused on child safety, especially in online safety issues, and more broadly in the AI context.

I'll give you the lawyer's answer, which is it depends. The difficulty is that some of these probably do cross a line, and others of them probably do not. So we have for three decades now had a federal law on the books criminalizing using a computer, whether that was Photoshop or now with AI, to modify a real child's image into being a sexually explicit image, into child sex abuse material, or CSAM as we'd call it for short, what used to be called child pornography.

That law has survived various legal challenges at the courts of appeals level. There is Supreme Court precedent saying that fully virtual CSAM is protected speech unless it also is obscene, because there's no real child being harmed. The Supreme Court didn't deal with the part of the definition of child pornography, again to use the term on the books, that dealt with what are called morphed images. But several courts of appeals have, and they've said, well, the harms that come from morphing a child's image into a sexually explicit image are more akin to the kinds of harms that are experienced in real hands-on abuse. And so, no, there's no First Amendment protection for morphed image CSAM either.

So, some of the imagery that's coming out of Grok maybe qualifies as that. It's an open question whether bikini pictures themselves cross that line. And then more recently, we have the Take It Down Act, which got passed last year by Congress and signed into law by President Trump, that criminalizes non-consensual intimate imagery, whether it's real or deepfake, whether it is of a minor or of an adult. So, that is now a crime to publish or to threaten to publish.

There are takedown provisions, hence the backronym for the name, that don't go into effect until May. And this is certainly pointing up that a lot of people might be having to wait a long time before they can try and invoke that to require X to remove material that it has not necessarily seen fit to remove so far, even though X nominally participates in a voluntary initiative called StopNCII for removing NCII.

But again, there too, not all of these images will necessarily qualify under the federal definition of what's called an intimate visual depiction, which itself by and large requires nudity. And to the extent that we're talking about bikini images or underwear images, those again might not necessarily cross that line into illegality. Whether state laws make the definition differently, I'm not sure. I'm not a big expert on NCII because the First Amendment questions that you adverted to are so much trickier than when we're talking about CSAM, which has no protection whatsoever.

Different places around the world, though, have laws that kick in differently and have decided in some instances actually yes, this is something that we can take action upon to block Grok entirely. We've seen the UK threatening to block access to X perhaps entirely, but under US federal law it's going to be very fact-intensive whether any of these images crosses the legal line.

The fact that I've seen reporting that X has removed at least some images that Grok had generated suggests that some of those maybe did cross a line in the opinion of whoever is left minding the store at xAI on content moderation, and that those were taken seriously enough to remove potentially as child pornography. I'm just speculating here because I don't know the people who would though.

Honestly, I think there should be congressional hearings to call up people from xAI, from the Department of Justice, from the Department of Defense, which has a huge contract to integrate Grok into now like sensitive Pentagon systems, which seems abominable. And I think that they should call the National Center for Missing and Exploited Children, which operates what's called the CyberTipline, by which online platforms report CSAM on their services, and say, "How many reports have you gotten from xAI since Christmas-ish of 2025 when this new feature got rolled out?" That might give us some better insight into at least what percentage of the torrent of imagery that has happened online does even xAI think crosses the line into where even they are not willing to let that remain up.

I think that is what makes this different from previous rounds of the kinds of image-based abuse that you mentioned. Where, you know, I published research last year looking into AI-generated CSAM, and what emerged was this picture that it really required leveraging a bunch of different platforms at once.

So you have these undress apps that some of them slip past app store review. Some of them slip past violating ad policies for display ads, where, you know, the app stores and major like, you know, CDNs for ad distribution nominally ban advertising for apps for notification purposes, but at scale, as you know, some of those are going to get through. And so somebody might encounter an ad for a notification app, let's say on Instagram, and be able to take somebody's picture off of their Instagram or off of some other platform and download this app through one of the app stores and undress that image and then circulate it in a group on Snapchat or a channel on Telegram.

And so you have all of these different platforms that were being brought to bear to make that happen. And therefore you have all of these different potential pressure points maybe to try and cut those things off. Even if you could reduce, you know, the volume of non-compliant ads, or if you could improve the level of app store review to keep those from happening in the first place. If you could ban or filter out some amount of the sites that are accessible even without an app for doing notification, that might be some amount of defense in depth that you could do to try and take a bite out of this problem.

And then instead we now have a one-stop shop in the form of a major social media platform that integrates an image generator that to all appearances does not have safeguards in place to prevent this kind of imagery from being generated, and like you said, is now fully integrated to do this at scale pretty much instantly. And that's where I agree with you that it doesn't seem like our regulatory frameworks are really equipped for that.

We have seen from the UK, for example, that they're saying we're doing an investigation under the Online Safety Act because they have already put in place obligations essentially to force certain online platforms, like covered by their various categories, to step up and keep certain kinds of content from happening on their platforms in the first place. And so they have a different set of tools that they are looking into using.

Again, going back to the question of like how much of this actually violates applicable laws, there's a whole round when they were negotiating the Online Safety Act in the UK about, you know, one version would have required platforms to moderate what is known as lawful but awful content, which to an American viewpoint is a terrible idea. What do you mean even if it's legal for me to say something or paint something or photograph something that you're going to require online platforms to stop me from being able to say that online, distribute that online?

And you know, we'll see whether this causes the UK to go back and say, "Let's revisit that whole lawful but awful thing when it comes to bikini pictures that maybe don't actually qualify as CSAM, you know, under our laws." Because I agree with you of the robust free speech protections that we have in the United States, and because of the various online legal structures that we've got, it may be that other countries will be able to respond more quickly under the Digital Services Act or under the Online Safety Act.

I was talking to somebody earlier who said, "It seems like there just isn't the muscle memory in the United States to know how to respond here because we haven't really had both the kind of integration like you mentioned between a social media platform and a notification app like this into a rapid-fire one-stop shop for this kind of victimization of people."

We've historically had, you might have a hidden site for serving up CSAM that does not care what the law is. They exist for one purpose. You have undress apps that are increasingly we're starting to see regulations for those apps in particular, but again, they're targeted at something that is specifically intended only for non-consensual deepfake pornography, not a general-purpose image generator.

And you also have regulators who are used to dealing with by and large big mainstream, often publicly traded companies that operate these social media services and that are pretty used to behaving in a pretty legal-risk-averse, optics-sensitive, answerable-to-shareholders-or-to-paying-customers-and-the-public kind of approach to what they allow on their services.

To the degree that even if they could theoretically allow everything that is legal under, you know, the law in any given jurisdiction, they don't do that because you rapidly turn into a cesspool if you do. And so I agree with you that it seems like we're in just kind of uncharted territory where that part, this particular sort of perfect storm, not only of poorly secured image generator, social media platform that already had become known as the Nazi bar, and somebody who is the richest man in the world owning both of those things, seeming to thumb his nose at regulators, and who we're going to find out like is he so powerful and rich as to be untouchable even in this instance, even when we're talking about child abuse imagery and non-consensual deepfake pornography, it feels like a combination that we just were not equipped to handle.

I think this goes back to the difficulty of are we trying to target the platform that is enabling the creation and distribution of this material as opposed to the end users who are engaging in this? Because, you know, I would disagree with you that we have no language and, you know, just nothing to say about this form of what is often called image-based sex abuse by people who work in and study this space.

We have seen the advent year over year of what used to be called revenge porn laws and now more broadly non-consensual intimate imagery or NCII laws that got passed state by state by state by state before we had the Take It Down Act last year. Now, like I said, I don't study NCII for myself really, but it has been a many-years-long campaign by people who are in that sphere, who are academics, scholars, lawyers, etc., to figure out how do you get accountability for the people who are victimizing you in this way.

And while we now have Take It Down as like a federal statute that defines a federal crime, we have long had other torts, things like intentional infliction of emotional distress, false light, invasion of privacy amongst the traditional privacy torts. The kinds of things that may seem like less perhaps of a powerful tool compared to a federal criminal law that Congress saw fit to pay attention to and pass.

So if you were to talk to people who, unlike me, have been up to their elbows in the NCII problem for a number of years, they could probably tell you, yeah, here's the approaches that we have taken over time. And they might also tell you like, yes, people are being victimized in this way. And even if you can see who's doing it, you can see that they have their handle attached to their username and you have the image right there and you can document all of that and, you know, get the timestamps and everything.

How easy is it going to be to actually hold that particular end user accountable in court, and what remedy do you get out of that? And so I think that's probably why a lot of people are focusing more on like we have to go one or two levels up, and instead of saying okay, let's go onesie-twosies after the people generating that image, what are we going to have in terms of potential liability or accountability for AI image generation services or for platforms that enable the distribution of it.

I agree with you that I am not a fan of the Paxton decision and that underlying rationale of like, well, set aside the security and privacy issues. You just got to start like making sure that you can collect everybody's like biometrics or their ID or whatever in the name of keeping children from accessing pornography.

It's like, okay, well, we've seen everybody's easily able to circumvent those, whether it's with a VPN or with using a realistic-looking like game character to fool like an age verification tool. And so you're not keeping people from accessing the content you're trying to keep them away from. And on the flip side, like the moment we started seeing these laws actually come into effect, there was like a massive data breach at Discord of like the third-party vendor that they were using for doing age verification.

So great, you don't get the upside that we were promised with these age verification laws, and you also get the downside that everybody warned you was going to happen. It happened immediately. We didn't even have to worry about being called slippery-slope, you know, hysterical zealots. It happened immediately, and everybody took the saddest "I told you so" victory lap because of course nobody wants to be in a situation where the laws are only impeding, you know, British people from accessing Wikipedia or whatever and not actually doing the thing that they were promised to do.

And if anything, I think age verification has gotten such a degree of cultural influence of like now this is the thing we're going to use to fix all of these problems that I see it being deployed in the context of non-consensual deepfake pornography. I was giving a lecture a few weeks ago about the research that I've done on this area, and one of the questions that came up during Q&A was, well, if you want to deal with like undress apps and making inappropriate imagery of children, isn't age verification the answer? And no, the problem isn't whether it's a 14-year-old or a 24-year-old who's making non-consensual deepfake imagery of somebody, it's that any deepfake imagery without their consent is being made of somebody, that's a problem.

And so it was this weird sort of window into when you have an age verification hammer, now everything starts to look like a nail. And that is just not the case. Not having adequate like, you know, controls over who can view and make the non-consensual like bikini imagery is not the place to be focusing on. And so it remains to be seen whether this is the thing that breaks everybody out of that mode of thinking that like if we just pass age verification in all 50 states, you know, the internet will somehow be a better place to be.

And instead we're just seeing like now not only are we seeing the failure of age verification for the benefits without all the drawbacks that it was purported to have, but also we're seeing what used to be with regard to undress apps kind of confined to this like seed underbelly of the internet as like escape containment, and now is integrated, you know, vertical integration directly into a major online social media platform.

Yeah. I mean, if anything, my hope is that the Grok incident is revealing that age verification is not this one-size-fits-all solution to like the ills of the internet. And in terms of like what to do about it, like I have some fear that this will be interpreted by some regulators around the world as, see,

This is why we need to expand bans on being on social media entirely to people below the age of 16, like they have done in Australia, which is being challenged legally by, you know, a group of teenagers themselves. I'm afraid that we'll see the total ban approach of just like don't even let teenagers get on the internet at all, or at least on some defined subset — always a weirdly defined, both under- and over-inclusive subset of online services — until they hit some magic number of an age. That might further that kind of request. Not only are they not happy with age verification in some regulatory quarters of the world, but also that they will want to push to exclude young people from even larger swaths of the internet, irrespective of the upside of what you might be able to find on certain services. I'm not going to sit here and say that there's a lot of good stuff to be found on X anymore these days, but surely, you know, we have so many stories from kids of how important it is for them, how marginalized or isolated they may be, that they can find community in the spaces online that regulators are trying to ban them from. You know, we've seen some countries have at least temporarily decided to block Grok entirely under their own nation's laws, but I am afraid that this is going to feed into even more thirst for similar under-16 social media type bans like the one that we have seen in Australia. You will not find a lot of big fans of Section 230 among the people fighting against NCII. I personally am a big fan of Section 230, but I understand why the anti-NCII folks who have been trying to fight this thing that really harms real people in their lives and can follow them around for years, why they are not big fans.

So a few things. One is Section 230 has never barred federal criminal enforcement of the law, and both with respect to whatever content counts as CSAM under federal criminal laws and whatever violates the new law. Take It Down now defines a federal crime of publishing non-consensual deepfake pornography of adults and minors alike. So for those particular whatever images qualify as one of those two categories, nothing stops the DOJ from taking action against xAI or X in that regard for something that violates federal criminal law. That's just the DOJ. It doesn't get at civil liability or like state AGs, like the one that California has now opened an investigation. What you're mentioning is something that actually there's also this other portion of Section 230 that says like it has to be information provided by somebody else, right? Some other third party. Usually that's a user, and it doesn't bar liability, including civil liability, where it is the platform itself that's contributing to what it is that makes the content illegal, or where the platform is generating the content itself. We have seen statements from Ron Wyden, the senator who has been around in Congress long enough that he helped write Section 230, saying that AI output is not meant to be covered by that. It is not supposed to be immune by Section 230. My understanding is that it's not yet a settled question in the courts, and my prediction is that this is the situation that's going to give rise to some amount of litigation where that question will be squarely presented: is the output of an AI, a generative AI tool, covered by Section 230 immunity or not? I think we're going to find that out in court opinions probably sometime this year. And in fact, we're recording this on January 15th. I am surprised that I haven't seen a class action complaint filed against X and xAI in federal court yet. I would have thought that some enterprising plaintiff law firm would have filed that immediately already with the ripped-from-the-headlines quotations. Hasn't happened yet. Maybe it's because things are moving as fast as they are and people are trying to just make their best case. But we've seen, you know, even what's called a cottage industry of law firms that have focused in recent years on bringing litigation against platforms over things like social media addiction allegations for teenagers, and over we've seen some lawsuits about the outputs of chatbots allegedly culminating in suicide. Unfortunately, in one of those chatbot cases, you know, the defendants didn't raise a Section 230 defense. It's always incumbent upon the defendant to raise that. I think they raised something with regard to the First Amendment. That case just settled pretty recently. So we don't really have an answer yet in that regard. What does Section 230 do with regard to civil liability for the output of generative AI tools? But based on both the general weakening of Section 230 over the last decade or so, where courts have seemed to be much more willing to bend over backwards to find a way to say no, it doesn't apply, and the success that plaintiffs have had here and there in terms of pleading around Section 230 to say this isn't about user content, this is about defective design features, and having some success — not 100% of the time, but some success there in not getting Section 230 to apply — combined with, you know, I think there's Ninth Circuit oral arguments, trial, and so on in some of these social media addiction cases. We are in a place where 10 years ago Section 230 might have been expected to form a much more robust defense, as opposed to here where we are now in a place where both the kinds of arguments that plaintiffs have realized that they can make and sometimes make successfully, and the willingness of courts to not be as eager to apply Section 230, might mean that there may be more likelihood for plaintiffs to have some success if they want to take on this Grok scandal in court. Again, though, we also should be continuing to pay attention to like where is the DOJ on this. We should not be letting them off the hook either. It can feel, I think, kind of hopeless to say, wow, the world's richest man is in cahoots with, you know, the federal government who allegedly they care about child safety and like protecting women online, as evidenced from signing the Take It Down Act. But where is the Department of Justice? Are they investigating xAI and X? If not, why not? We also are in a weird situation where going back to Take It Down, eventually platforms including X are going to have to remove this imagery when victims file a complaint and want to report. It has to come down within 48 hours. The enforcement of that is going to be the Federal Trade Commission. There's two guys left on the Federal Trade Commission because Trump fired the Democratic ones, and the ones that are left are like far-right anti-porn zealots. You know, like literally one of them was a fellow with the Heritage Foundation whose Project 2025 says pornography should all be illegal. It should be wiped off of the internet. The people who produce it should be put in jail, and platforms that distribute it should be nixed out of corporate existence. And oh, by the way, pornography includes trans and queer people existing at all. So what happens once those takedown provisions go into effect, and it is for the FTC to oversee whether platforms including X, including Grok, are taking this material down when victims depicted in it ask for it to be taken down? Or are we going to see Elon Musk at the zenith of his power being able to influence what either DOJ does or what the FTC does? Or are we going to see the weird far-right zealots who are now the only people left minding the store at the Federal Trade Commission saying actually this helps us with our overall anti-legal porn agenda?