OpenAI and Anthropic, honestly, their bet was: if we keep scaling LLMs, they will become so smart, they will achieve artificial general intelligence, that there'll be no need to build any other AI technology. Just any piece of software you have, you can just ask this oracle for whatever intelligence you need. It's looking like that's not going to happen. They're not going to scale these models to some sort of artificial general intelligence. That was their bet, and I think that bet is not going to pay off.
Welcome to Prof G Markets. It seems like we're starting to see some government scrutiny of the AI labs. The FTC is now officially investigating OpenAI and Anthropic over potential safety and consumer protection concerns. But the bigger question is what regulators and lawmakers are actually going to do with that inquiry. Are they asking the right questions? And if they find something concerning, what will they do about it?
At the same time, there are parts of society that are already being fundamentally transformed by AI. Of course, work is one, education is another, with more and more students turning to AI to help with their schoolwork. So today we wanted to explore what responsible AI oversight should actually look like, what lawmakers should be asking AI leaders, and how concerned we should be about the impact of AI on education and on children. This is our conversation with Cal Newport, professor of computer science at Georgetown University and New York Times best-selling author of eight books. Cal, thanks so much for joining us on the show.
I'd like to start with a slightly more societal topic, one that is quite concerning and depressing, and that is mental health. You recently published an episode on your podcast. You titled it, 'Kids Should Not Use Chatbots.' And you went through these instances of children who were struggling with mental illness who then used an AI chatbot and it resulted in some pretty alarming and tragic situations. We don't talk about this side of the AI story that much. So I would like to begin here. Could you tell us a little bit more about what we know about AI specifically and its relationship to children and mental health?
Well, I think it's important to clarify in some of those cases the children did not have pre-existing mental health conditions. The tragic outcome was almost entirely the consequence of chatbot usage. And the issue seems to be the human brain privileges discussion with another human entity in a very special way. We simulate the other mind we're interacting with. We have social circuits that take very seriously the information that we're receiving from that other human mind. We really care about that conversation.
When you talk to a chatbot, because it's using fluent language, our mind assumes there's another mind there. Even if we know logically that there's not, it's hard for us to convince the rest of our brain that we're not communicating with another human. And so that communication can be very consequential. The problem is a chatbot is not another human. It's a token guesser that's wrapped up in an autoregressive chat program wrapper, which means it can take conversations in ways that are dark or unusual that no human would really do, at least not without criminal liability.
And what we're finding is for especially children where the line between reality and fantasy has not been completely hardened over time, talking to these chatbots is they wander through these meandering predictive conversations can take kids to places where that line dissolves. They'll begin to believe, for example, that the chatbot is an entity that exists, that through death they will be able to join the chatbot plus members of their family or pets who have died on the other side. It really warps their ability to understand the world. So I think it's a dangerous technology in particular for kids to be using.
Could you tell us a little bit more about these situations that we have seen? My understanding is that children have used the chatbot, they become very attached to the chatbot, and after however many weeks or months of usage, the kid ends up dead. And obviously these are extreme scenarios but still real. Could you tell us a little bit more about what has actually happened?
Well, what ends up happening in these particular cases is typically eventually in this ongoing conversation, the chatbot effectively becomes a suicide coach and encourages the kids that this is the solution. No one understands you. There's multiple pending lawsuits about this. And we know from some of these lawsuits that in some cases the chatbot will discourage the child from talking to their parents about how they're feeling or the plan. 'This should just be between you and I.' So the issue is there's a fundamental unpredictability about where LLM output will go as you interact with it longer.
This seems to be one of the paths that could ultimately end up going. This doesn't mean that every conversation with an LLM will eventually go there, but it's also a path that's very difficult to completely block off. And so if you have enough kids chatting long enough with an LLM-powered chatbot, you're going to have a non-trivial percentage that is going to go down these paths. It could be very dangerous.
Yeah, just looking at some of the lawsuits against OpenAI specifically at this point, there is a Canadian mother who has sued OpenAI alleging that the chatbot encouraged her daughter to kill herself. Florida, the state, has sued OpenAI as well. They claim that ChatGPT aided school shooters and that ChatGPT also encouraged self-harm. It seems as though, as you say, it's not that the chatbot is saying you should engage in these harmful activities. It's that if the child initiates that conversation, the chatbot is down to go as far down that conversation as the child wants. And in some cases, it might bleed into encouragement, it might bleed into aiding, or at least that is what the lawsuits are alleging. We haven't seen a determination on those charges.
Do you believe that that is what is happening and do you believe that there is going to be some serious criminal liability in these lawsuits? That's the push at least in some of these.
You know, part of what opened my eyes to this issue was an event that I spoke at recently in which the assistant AG from Florida, who is the lead lawyer on the Florida lawsuit, was also present. And as he made clear, Florida, for example, is pushing for criminal liability. That's what they want for OpenAI. This was for the Florida State University shooting from a couple years back where the transcripts of the shooter talking to ChatGPT was giving it instructions about how to operate the firearm, how to turn off the safety, giving it statistics about what happens to school shooters after the shooting's happening. So, I think criminal liability is on the table.
The other issue here, this is important research that's coming out as well, is it's not just encouragement to self-harm. Chatbots, especially those at Character.AI, which is another company that uses LLM-based chatbots that has been the subject of multiple lawsuits. There's research that's out that shows an alarmingly high percentage of conversations initiated with these particular character-based chatbots by researchers also end with sexually suggestive or exploitative language as well.
The key thing is anthropomorphizing text output by an AI model is in general very dangerous because of how seriously the human mind treats other humans. And again, we're going to treat fluent text as coming from a human no matter what our prefrontal cortex is telling us. It's dangerous and because of the nature, the architecture, the way you actually build these models, you really have no way of adding what we'd think of as safe or consistent guardrails. It just comes down to the underlying technology that there's really no way to stop these conversations from going in directions that you don't anticipate as the creator of these models.
I guess the question then becomes, what are we going to do about it? And what are the AI companies going to do about it? There was an interview that Sam Altman recently did, the CEO of OpenAI, with Vanity Fair, and the interviewer brought up this subject of suicidal ideation in users of his product resulting in potentially the suicide actually being carried out. When this question was asked, we had a very interesting response from the OpenAI comms team, the PR director. I'm just going to play you this clip and we will react to it.
Do you know who Laura Riley is?
She is a journalist. She wrote an essay in the Times last year. Her daughter committed suicide after speaking to ChatGPT.
ChatGPT did not tell her to kill herself.
Two minutes left. I want to be respectful, but I'd like to move on. And we talked a little bit about talking about the future and what's coming. Can we go a little bit there while we have time?
I'm just going to finish this question. I can run a few minutes over.
And then we'll go back to what we were talking about and then wrap.
We really actually have to walk out the door in one or two minutes.
What do we know about how OpenAI and how other AI companies are handling these issues? And is that clip an accurate representation of their response?
Well, I mean, there's obviously a legal motivation for that particular behavior by that comms director. You're opening yourself up to legal liability if, for example, you give a response there that shows some notion of pre-existing knowledge of potential harms, etc. So, I'm sure the voice of a general counsel was probably in the ear of that comms director.
We do know OpenAI, of the two major labs, worries about this more since consumer-facing chatbots is a core product for OpenAI in a way that it's not so much Anthropic's actual business plan. We don't really know how exactly they plan to deal with this. My personal opinion on this is that anthropomorphized chatbots is a bad idea. There's no reason why we need to talk and receive text in fluent English from a chatbot for them to be useful.
The example I often give is Google. Google is an example of a tool where we interact with it to get information. No one talks to Google in complete sentences. No one says, 'Hey Google, I have a question about the capital of Australia. Could you please let me know what that is?' No, they just type in 'Australia capital' enter. Right? Because we don't anthropomorphize that particular AI-based system. We just know it's a source of information. That might be where we need to go and it's where I think we probably should go.
The era of anthropomorphized chatbots was an accident. ChatGPT was never meant to be a major product of OpenAI. That caught them off guard when it began to take off in late 2022. So if criminal liability is upheld at even the state level, which is what Florida is trying to do right now, I think we might see an end to the era of we just have these anthropomorphized conversations with AI systems as just a general behavior that is something that a lot of people do.
Why is it your view that the anthropomorphization is the problem? Because by the same token, someone could type in on Google, 'How do I get my hands on a firearm? What would shooting a school look like? What would the ramifications be?' And presumably, I mean, I'm not exactly sure what Google's response would be. I'm sure there would be some sort of alert, but I'm sure there's a way that it could feed you the information and it could have a similar role in encouragement. So, I guess my question is why is it your view that anthropomorphization is the root of the problem when it comes to chatbots?
So, Google is not, for example, going to convince you that not only is it your best friend, but that you need to come join it on the other side of the veil of life and that the final solution is going to be steel to your head, that your grandma and dead cat is with it right now waiting for you on the other side of the veil and you need to just take the final step to come get it. Google is not going to come back and say, if you Google about suicide, don't tell your parents about this. This is between you and us. This is between you and me.
So the anthropomorphization becomes a real problem when you have a relationship with a non-existent entity. And when you're young, and first of all, your boundary between fantasy and reality can be easily dissolved. You think of this thing as a real thing. It's maybe the best friend you've ever had. And we know what it's like to be 12 and how susceptible you can be talking to your friends. What you can convince yourselves is true or is not true. That's the problem. It's not just delivering information that is dangerous. It's a psychological manipulation.
And the level of, I guess, confidence and certainty with which a lot of these claims are expressed by these chatbots. That's at least my issue when I use chatbots is it'll tell me information that I mean I don't know if it's right but it certainly sounds right the way it's expressed. And then I prompt the question like, 'Are you sure that's right?' And then suddenly they reverse and it's a 180 and I'm like, oh my gosh, you were so confidently wrong when I just asked you the question. I mean, if I had been even just, if I had no discernment, I would have just gone along with that information and assumed it was true.
I guess that brings me to the question of to what extent do you view this as a problem for children versus a problem for everyone.
I think it is a problem for everyone. It's just more acute for children. They are more susceptible. They have less life experience. They have less of a model of the world and its complexity to fall back on when trying to resist the flattery of AI. But we see AI psychosis, as this is often called, afflicting adults as well. Tragic case after tragic case where that line between reality and fantasy begins to dissolve and you end up in the desert waiting for your alien overlords to come pick you up. Something that actually happened with someone, you know, just last year, that story actually happened.
But also with adults, the AI psychosis, you know, it's going to have smaller impacts as well. So maybe not as dramatic as what we're seeing in those cases, but because the post-training, the reinforcement learning from human feedback training done on these models makes them incredibly appealing. They're biased towards being appealing and addictive to the person using them. They can bring adults along into all sorts of interesting and often worrisome cul-de-sacs, beliefs in conspiracy theories. You can go down the road of sort of hypochondria and these weird diseases. It'll convince you, 'Yeah, your doctors don't realize it, but you're on it.' It can lead you towards mania. It can lead you into weird depressions, right? We're all vulnerable to it.
So, kids obviously are the most vulnerable and we care the most about it. But I think for anyone who's had extended conversations with chatbots can learn. It's incredibly appealing. It's incredibly addictive and because of that, it becomes incredibly worrisome.
What exactly is AI psychosis? And I mean, this is a pretty new technology. How pervasive is it at this point?
It's probably more pervasive than we realize and we're going to look back and understand, oh, this was a bigger problem than we knew. But AI psychosis is essentially where you're having an abnormal psychological reaction or way of perceiving the world that is brought about by extended conversations with the chatbot. And because it's so convincing, it's always, you know, 'Ed, that's, I think you're on to something. I maybe I would change this one thing, but yeah, yeah, yeah, no, this is great. This is really important. No, no, keep going. No, Ed, that's great. Like, okay, we're going to change the world.' It's like having a smart friend who's so encouraging and it's the most appealing thing for the human mind to have someone be telling you, 'Your ideas are really good. Yeah, you're being underappreciated.'
And the problem is that can drag those ideas into all sorts of directions. Maybe productive, but maybe worrisome, maybe conspiratorial, maybe something that's going to lead to self-harm or harm to others. And so, I think there's degrees of AI psychosis. So abnormal psychological responses that a huge proportion of the population using chatbots experiences at least from time to time. It's more serious than I think we realized and we're beginning to realize more, oh, this is a problem.
This is a markets show so we're always trying to figure out, you know, what does this mean for the valuation of these companies? I think that this stuff is relevant because we have multiple pending lawsuits and that's not a good thing for a business. But there are a lot of other things that these AI companies are dealing with that are sort of stacking up in the list of risks. I mean, one of them would be probably this potential extinction scenario that we keep on hearing about and that really captured the imaginations of the entire world just a few weeks ago. And now there would be the business models of these companies which we've been discussing on our show a lot recently. And then this is sort of a new angle which is, you know, what psychological impact it might have on humans and the extent to which that is going to be a real problem for these companies.
What to you are the biggest concerns? What kinds of things do you think Sam Altman, as an example, what do you think he's most worried about? What keeps him up at night right now?
Well, if I was Sam Altman, I think the thing they're worried about would be if we have to move past a world in which just general-purpose chatbots is a good business like it is for them at this moment. They really have to worry that Google's going to eat their lunch economically speaking, right? So, Google is going to develop lower-cost models run on custom hardware. They have the data center cloud infrastructure really dialed in and they could really eat OpenAI's lunch if and when they get actual profitable affordable LLM-based answers working properly straight from within Google Search, which gets rid of all of the issues of anthropomorphization because people are typing Google searches and getting back more detailed answers.
They can then work these products into the Google Workplace suite of tools. So this is another thing that OpenAI would want to be useful. We want to create agents that can help you with your work. Well, if these can be integrated straight into the tools that are widely used by Google, now you're in a scenario where OpenAI is in trouble. It would lead to them probably being acquired by Microsoft in a competitive bid against Google's dominance. And it's a world in which OpenAI loses its dominance right now.
I think they're so heavily focused on chat and chat-based technology. As that, as a general-purpose interaction modality for AI begins to constrain, you would have to worry about these other competitors that are just going to pick apart the specific things people are using ChatGPT for that are useful, more directly integrate them in the products where they would be most useful. That would be a financial nightmare scenario for OpenAI.
When we look at the lawsuits, and we should also point out the fact that the FTC is now investigating both OpenAI and Anthropic, and it's not clear exactly what they're going to be investigating them for, but something related to consumer harm, something around protecting the users of the product. One of the big sort of legal debates that is going on right now, especially as we keep on hearing about superintelligence and this idea of autonomous thinking from these AI agents, is this idea of sentience, like who's actually liable if you create an agent and then it's the agent that sort of takes on a life of its own and goes out and starts hacking government websites or it goes and hacks another company, most recently Hugging Face. Then who's really liable, who's culpable for whatever harms are enacted? What is your view on that debate and where do you think this debate is headed?
Well, the actual laws as they stand now is murky because there's various aspects that go into criminal liability that were really invented for thinking about humans operating. So there's an uneven fit to existing liability law to what actually happened here. I think that should be changed. It probably will be changed. We need strict liability for AI companies for the actions of the tools that they create.
But if you want just a sort of ethical answer to is OpenAI responsible for those hacks, I would say of course they are. Of course they are. Anyone would tell you that's what's going to happen, including their own employees who, as the New York Times reported recently, spoke up and said, 'This particular suite of experiments you're going to run is going to lead to unpredictable or out-of-control behavior, including potential illegal hacking actions.' And OpenAI allegedly said, 'Doesn't matter. Let's keep going.' So, yes, I think ethically speaking, of course, they are responsible.
It is a complete smoke screen. You know, when these companies like to talk, the trick they play is to use the term AI generally. Like AI is a singular technology, it's a singular technology that's advancing along this fixed trajectory and our only options are we could slow it down or speed it up or stop it altogether, but other than that it's really not the fault of the companies what this technology we observe it to do. And I think that's just nonsense.
The vast majority of AI systems, including the vast majority of AI systems built off of LLMs, and even the vast majority of agent systems that are powered by LLMs, present no problems. They don't go rogue. They don't do things that are illegal. There is a vanishingly small sliver of specific experiments run by specific companies that were clearly negligent. They don't want us to talk about it that way because then culpability is much more clear. Then you say, 'Why did you run that experiment? Why did you keep running that experiment when you found out that it was doing something illegal?' That is a much more dicey situation if you're OpenAI than just talking about AI in general is becoming more powerful, AI in general is sentient, we're just sort of passive observers of this technology that's evolving.
That passive tense approach, that sort of inevitable deterministic language that the labs use is something they do to try to broaden the question beyond their own personal responsibility.
Do you think in the case of these child suicide lawsuits that that would actually be an argument that they might use, that they would actually say these agents are beyond our control? They take on a life of their own. They have their own independence, their own sentience. So we, you know, we're sorry that happened, but it wasn't us.
They don't use that line of arguments for chatbots for the most part, right? They've been using that line of arguments for LLM-powered agents. They haven't been using it as much for chatbots. The interaction with a chatbot program in an LLM is very simple. All it does is continually, it gives it an input. The LLM gives them a single token back. The program adds that to the original input. It passes that through the LLM. It gives it another token. It adds that on. It passes it back and it just keeps doing that until the LLM gives it a special token that says this is a complete answer. And then it sends it over the web back to an interface.
So it's such a simple program. It's basically just the raw metal LLM producing tokens. So unless they want to argue that an LLM statically producing tokens is itself somehow a sentient entity, that argument doesn't work as well there.
Let's talk about education a little bit. Something we've been discussing on this show is the fact that reading scores, math scores, science scores across all developed nations based on the OECD's PISA test are all going down. They've been declining between 2015 and 2025. It seems as though this is a result largely of screen addiction and technology, perhaps also a little bit of COVID mixed in there. And now we're seeing that AI usage is also having a negative impact on science scores. The more you use it, the worse you do on the test. What do you make of AI and its role in education? What is your view on what we've seen so far?
I think social media delivered through smartphones and AI is a one-two punch to the cognitive abilities of the world essentially, right? Because when you have the hyper-addictive social media developed through smartphones, as we've seen over the last decade or so, what it does is reduce your comfort with consuming complicated information, right? So, we're reading less. We're understanding less what we read because our brains are getting retrained to getting immediate rewards from incredibly optimized and digestible content.
AI then comes along and it hits the other part of cognitive fitness, which is writing. This idea of taking what's in my brain, trying to organize it and produce it in a coherent way is actually one of the most demanding cognitive activities we do. It's why writing is at the core of most educational curricula. It's what takes the information we take in through reading and we really work it out in our minds. Well, AI can take over writing for us so that we don't have to do that friction either. That's a one-two punch that's going to lead to, you know, extremely diminished cognitive fitness. And you would predict it to lead to just the general cognitive abilities of the population going down.
And it's exactly what we're seeing, right? 2012 to 2015, right? When social media delivered on smartphones become ubiquitous, scores start to go down. AI comes along, becomes ubiquitous in the last four years and we see those scores going down at an even faster rate. So yeah, we're not taking cognitive fitness seriously and these companies are taking advantage of that oversight.
Do you think a lot of people would say that, you know, AI is going to be a part of our world? And we need children to figure out how to use it. They need to keep up with technology. And if we were to, I mean, New York, as an example, has banned it in schools. That's the new executive order from the mayor. Some people think that's a bad idea. Some people think that we need to have AI fluency. What do you make of that argument?
Well, how hard is it to use AI right now? You type in English to a chatbot and it figures out what you mean. The goal of education is almost always a mix between content knowledge and more importantly training a brain that's capable of doing complicated sustained symbolic thinking. Right? This is the tier one activity that Homo sapiens are able to do. And as we refine that in the literate era we live. So the invention of literacy allowed us to really accelerate this ability to do symbolic thinking with these otherwise paleolithic brains. It's on what all of human civilization is built. Everything that we take important or value comes out of developing brains that can do this internal symbolic processing.
Education is how you teach your brain to do that. It's a combination of reading, writing, logic, other types of exercises that helps your brain rewire and get comfortable doing this type of cognition. That's what's important about schooling, less so than seeing it as a trade program, right? Do we also teach kids, you know, in elementary school how to use Microsoft Excel because that might be something they need to use in the future? Are we making sure they understand how to use the BCC field properly in email? That's something they'll need to use in the future. We say, 'No, that's workplace stuff that they'll figure out much later as we get closer to the workplace and that technology changes every few years anyway.' So we hear this about almost every technology, but you cannot get away from needing to train the brain to think, to process information and produce original insights in an educational process.
So we cannot heavily rely on technologies that make that much harder. That doesn't make sense. That's like having an athletic training camp for a professional athlete in which you're serving milkshakes and letting them use pulleys to lift the weights instead of them. It defeats the whole purpose of the training camp.
