Hello and welcome to the Vergecast, the flagship podcast of music that sounds eerily but not exactly like other music. I'm your friend David Pierce, and today on the show we're talking about training data. It's the raw materials of everything that is AI, and we probably don't understand it or talk about it enough. I'm talking to Alex Reisner, who's a staff writer at The Atlantic. And over the last couple of years, he has spent a lot of time investigating training data and how all of these books and all of these articles and all of these YouTube videos and all of these songs are compiled into these gigantic data sets that AI companies then use to form the basis of their models. I think the way that these models are created and the sources of this data has a lot to do with the way that we feel about AI, particularly generative AI as a creative expression. And understanding how this data works, where it comes from, and how it gets used, I think, is really important. So we're going to have Alex on. We're going to dig into it. I'm very excited about it. But first, here's everything else happening on The Verge today. This is 90 seconds on the Verge for Thursday, June 25th, 2026.
Apple just raised the prices of a huge number of its most important products. iPads and MacBooks in particular are now up anywhere from $100 to $500. And even like the Apple TV is way up. It's now $200, roughly the price of 900 Amazon Fire Sticks. We knew this was coming, but this is still a big moment. Apple is as well-managed a supply chain company as anybody and exists at really high margins. If it can't keep prices down in this era of AI-driven shortages of memory and storage, nobody can. And I suspect this is going to get even worse. My advice from yesterday still very much holds, which is Prime Day is not over. Go get your deals while you still can. And here's a way to help you pay for all of those more expensive gadgets. Go get some money from Disney. If you have YouTube TV or DirecTV Stream, you might be eligible for a cash payout from a new $50 million settlement. The case started in 2022, and the argument was essentially that because Disney owns ESPN and Hulu, which are both powerful streaming services in their own right, Disney was able to drive up the prices of all of its rivals to unnecessary heights. Live TV is lucrative and competitive and is going to keep being messy in this way. This will not end things, but hey, go get some of that $50 million. And finally, a quick PSA to advertisers everywhere. Yes, I get it. The whole idea of using AI creative and AI targeting to put your AI ads in front of AI people is exciting, I guess, but maybe check your ads to make sure that you're not, I don't know, advertising a bike with two sets of handlebars like REI did this week. That may not go over so well with people. Just a little free advice from me to you. You can read more about all of this at theverge.com. That's 90 seconds on the Verge for Thursday, June 25th. All right, let's talk training data. Alex Reisner from The Atlantic is here. Hi, Alex.
Hey, David. Thank you for having me.
Very excited to talk to you. I think we have talked a lot on this show about why people feel the way that they feel about AI and the sort of instinctive reactions to the whole idea of AI. And a theory that I have is that a lot of it is about training data. And I just want to talk about that. You've done a lot of work on this stuff, and you've been investigating how AI models get trained for a long time now, but I want to just lay a little bit of groundwork here. And just to ask the very obvious question up front: why does it matter what data is used to train these products? What difference does it make what's inside of these models?
I think it is potentially the most important aspect of a model is what it's trained on. I mean, if you take a model and you train it on, let's say it generates music, and you train it on 1950s jazz, that model will be very good at generating music that sounds a lot like 1950s jazz. If you train it on recent hip-hop, it's going to generate music that sounds like recent hip-hop. You know, these models have names like ChatGPT and Claude, but I think you could make an argument that the right name for a model is actually the description of the data it was trained on, because that is the description of its capabilities. That's what it can output. And so I think that the training data is really fundamental to the model, maybe more than the architecture to some degree.
Interesting. And I think, I mean, that kind of answers to some extent my next question, which is why is training data such a closely guarded secret from these companies? And it seems to me that there's both a straightforward business answer and maybe a slightly more nefarious answer as to why these companies would be so secretive. But what is your read on why this is such a closely guarded secret? What data is being used to train these models?
Yeah, I mean, the companies have argued that they need to keep it a secret because the data that they have selected to train on is their competitive advantage, right? Like Anthropic has done a better job at selecting data than Google and OpenAI, and if they were to let that come out in a court case or be public in some way, they would lose their competitive advantage. You know, there's another pretty obvious reason, which is that they have gone about acquiring a lot of this data in ways that the people who created the data, the authors of the books and the creators of the videos and the music, would not be happy about. And in a lot of cases, they just don't know that their work is being used. And when they find out, they're not happy about it. And I think it's a conversation that the AI companies have tried to just avoid having.
So a big part of what you've been up to recently has been sort of reverse engineering this process to like peel the model apart to figure out what data is inside of it. And it seems to me that you've had to do sort of the reverse of what all of these companies have done, which is go figure out where these mass sources of books or web articles or videos or songs are. So tell me a little bit about your process. How do you go about finding these things that are kind of otherwise closely held secrets?
Yeah, my process is actually maybe not that different from the company's process. I am a programmer by trade. I worked in tech for 20 years. I built websites and apps and did some statistics work, and so I've been aware of these models for a long time. And at a certain point, I started hanging out in the forums where AI developers were hanging out and talking about the work. And you know, they're talking about what data they're using to train these things, and so that was a useful source. And I've also been reading a lot of their research papers. You know, selecting the data is really challenging. There's a lot of effort that they put into it. They have to come up with some notion of what is high-quality data, like what do and don't we want in the model, and they write papers about that. You know, I think they want to be involved in this conversation that's happening about training data within the industry. One thing that's been really helpful to me is that there's an open-source AI development community, and they believe that the work should be done more out in the open. And I think a lot of them are doing really good and important work, and they have really interesting things to say about AI, just philosophically and socially. And so they're pretty transparent, places like Allen AI and LAION, about what they're using to train. And so that's been helpful. But then even at the companies, you know, the AI world is a little bit like academia in that it is good for your career to publish papers, and the companies don't like that, but they also acknowledge that they have to let the employees publish something. And so the lawyers will go over it and tell them what they can and can't say. And over time, they've clamped down more. And so the companies are revealing less through the research papers. But yeah, a lot of my research is just from re-reading a Google paper, you know, for example, for this last article where they said we trained on, you know, tens of millions of songs.
Yeah, I remember not that long ago Apple going through a big cultural issue with this with its AI team because Apple is so secretive and so reluctant to let any of its work be public that all of its researchers were like, well, if you're not going to let us publish, we don't want to work here. And that push and pull seems like it has kind of morphed in a bunch of different directions over time. But interesting to hear that it is definitely, everybody is retrenching a little bit as this space gets hotter and hotter.
Yeah, it's not 2021 anymore. The research papers read very differently now than they did a few years ago.
That's really interesting. But tell me a little bit about these databases. One thing I think I had not given enough thought to until I started really reading your work is that there's a real business in making and maintaining these databases. Like one company you wrote about, Common Crawl, I think is like a thoroughly fascinating player in this. And they essentially, as far as I can tell, just crawl the internet and make it available to whoever wants it, which is weird and complicated. And we should probably come back and talk about Common Crawl sometime. But it seems like if you look around a little, these gigantic databases of books and articles and videos exist. Like, are people making these and selling them? Why do these databases exist? Mainly, well, I don't, I mean, for AI training, why else would they exist?
That's a huge, you know, it's an extremely labor-intensive process. To find, you know, in one sense, you just go and download all of Library Genesis or all of Anna's Archive. That's one sort of naive thing you can do. But the companies realized pretty early on that they need to filter the stuff pretty carefully. So the organization you just mentioned, Common Crawl, yeah, they've been crawling the web since the late 2000s, maybe 2009 or something like that. And they just make the whole thing available every month. There's a new, they've scraped a few more hundred million web pages, and it's available to anyone who wants to do any kind of research with it. In fact, it's mostly AI researchers who are using it. But all the early large language models were trained on Common Crawl. If you go back and read those early OpenAI papers, everyone was training on Common Crawl at first, and the models were terrible because if you train a model on the whole internet, it just, you know, you get all the junk that people say on the internet along with the intelligent things.
Yeah. Mostly junk. Statistically mostly junk.
Statistically, yeah, mostly junk. And I think the large early models were proof of that. But yeah, I think Common Crawl is a nonprofit, so you know, they would argue it's not a big business. They do get a lot of money from AI companies and AI investors. But yeah, I think the topic of training data selection, the challenge of selecting the right data for a model, is still really hard. The AI companies, I would say, still have a very primitive understanding of what data will make their model better. It's an area of research that I think even they at this stage are not very good at. They do it mainly by trial and error, as far as I can tell. So again, the reason these, you're asking why do these data sets exist? I think it's people trying to share what they've learned from curating data sets in different ways and training models with them.
Yeah. Yeah, I mean, part of the reason I ask is I think one of the things I have come to believe about the AI industry is that this shift that went from AI research being fundamentally a research thing, like if you go way back, OpenAI was basically a research organization, right? And you talk about Common Crawl, and I think its early users were largely researchers, and these things were academic things for academic purposes. And from what I understand, the kind of rules of the road are different, right? That like if you want to make copies of a bunch of things for academic purposes, these things are generally considered less problematic, right? But if you then do it and become a trillion-dollar company on the back of it, people are going to rightly feel differently about the way that you went about getting that information. And just the speed with which AI commercialized, all of these companies just moved so fast from we are essentially an academic thing to oh my god, we're making so much money, everyone's filthy rich, that it feels like they just hoped everybody would ignore the ways in which they got this information. And so that's why I'm particularly fascinated by where these data sets even come from, because it does seem like in many cases, like you're saying, it's not that there is some gigantic business in being the one to sell the songs to somebody. It's that this stuff is being compiled for other purposes. It's just that now the main purpose, because of the sheer volume of work, is AI research. All this stuff has been sort of co-opted from every other purpose to AI, and it just all feels so concentrated now. Does that feel right to you? Does that make sense?
Yeah, I think I agree with most of that. I do think that I'm not sure how much these data sets were really collected for other purposes. Common Crawl likes to talk about, I think they're one case, they probably have the strongest argument that their data could be used for other purposes. But when you go back, you know, they've been cited by over 10,000 papers. I didn't read all 10,000, but I read a lot of them, and they are mostly AI, right? And it's early, a lot of it is stuff that people wouldn't mind as much as with generative AI, right? Like Common Crawl, I think without Common Crawl, you know, AI translation tools might not be as good as they are. I think it was really a huge help because they scraped web pages, the same page in multiple languages, and people were able to train translation models based on that. So that was helpful. But the thing that, you know, there is still what I would call a data laundering network, where the AI companies are still relying on, they'll do a collaboration with a university and they'll have the university download, you know, millions of images to train a model or download millions of articles to train a model, and the AI company can say like, well, we didn't do it, this was like an academic thing. You know, the same goes, Common Crawl is not the only nonprofit that's like doing a lot of this scraping for the AI industry. One of the data sets I reported on in the music, the article about music training data, is this organization based in Europe called LAION. They have a data set of 12 million songs from YouTube. So anyway, this is like, is it academic? Like, not really. Like this is, you know, it technically, yeah, there's universities and nonprofits, but they're all receiving money from the AI industry.
