An AI Expert Challenges an AI Skeptic, with Ethan Mollick

Thanks for having me. I'm glad to be here.

I actually play them as well, so it's not just the show, although that would be probably the nerdiest possible thing to try and impress people with.

There's so much to unpack. First of all, part of this is just the fault of how these things are built, just even how their use cases are. So you may run into a classical case of hallucination, right? The AI says something and then justifies it with more lies, which sort of is an outcropping of how these AI systems work, which is they can only go forward. If it says the word no, he's not in the movie, then it can't go back and delete that. So then it tends to create an elaborate justification for its false answer. However, I would ask, were you using a paid or free version of ChatGPT for this?

Okay, so I can tell you that one of the things, there's a lot of things that change as AI models have gotten more advanced and larger and bigger scale, which has all sorts of consequences. And one of those is hallucination rates have dropped. And we have some information on that. There's been some good studies in medical journals and other stuff showing hallucination rates drop as model size gets larger, often below human level. However, if you're talking to ChatGPT and you're just using their auto feature, and I don't even know what feature Siri is using, it's probably one step down from there. There are basically two versions of ChatGPT 5 or 5.1. One is the chatty version that's designed to be your friend and the other is the serious version that answers questions and looks things up on the web. And the auto router decides whether that's important or not. So it probably decided this is a trivia question, send it to the dumb model. The dumb model makes lots of mistakes, but it's supposed to be very fast and chatty. You don't know that. They don't tell you that. It's another reason why it's very confusing to use these systems. I'm very convinced that if you went, and we could even try it later, if you went to GPT 5.1 Thinking or one of the more advanced models or Gemini that has web access and you ask that question, you'd probably get the right answer. But there's a reason to be suspicious because why would you know all of these things, right? And will you take my word for it next time? So there's a lot of things happening at once in AI that are both good and bad. There are things that are kind of miraculous and things that kind of suck and the user interfaces are complicated in ways that are hard to explain to people in a first pass. So I would be willing to bet that you'd get the right answer out of any modern model asking the question, but you're going through Siri to ChatGPT, which is probably routing to a worse model, and things change very quickly. So I get your suspicion. There are tons of positive use cases too and we actually know about error rates overall and have information about that. So it's a complicated problem.

Sure. There's tons of them. I mean, look, at the time of recording this, it is 3 years and a day since ChatGPT came out, right? So by all stretches, we're in early stages of technology. And I find it very, by the way, the fastest adopted economically consequential technology in history, as far as we know, but still early days. And so we have a lot of good and bad stuff happening all at the same time with AI. I could tell you for example on the use case side, there's a nice controlled experiment out of the World Bank that shows that kids in Nigeria using AI as a tutor with teacher help had huge improvements in performance. We've seen similar things that test at Stanford and Harvard and other places. But if you just use it on your own for education, it tends to just give you the answer. People tend to cheat and they learn less. But using it in the right kind of setting, you do more. We have, there's some nice controlled experiments showing that the most recent AI models actually often outperform physicians in giving diagnoses, but again they don't do everything a doctor would do. So you might want to use it as a second opinion, but I wouldn't use it as a first opinion. We know that we're now starting to see the first original science being done by these systems. So they're capable of doing math that we weren't able to do before. As somebody who uses these systems a lot, they're extremely helpful for doing real science. I don't do all the work autonomously with AI, but certainly for coding, you get huge improvements in performance and outcomes. I mean, across a wide swath of economically valuable areas, AI is a good expert advisor with caveats, right? So I mean, it's weird to me to have the question, is there value in this when there's a billion people using these systems on a weekly basis? We have survey results from companies showing that they're getting positive returns on investment. You know, I guess the anecdotal failures are absolutely true, but that doesn't preclude the fact that there's obviously lots of value for lots of people in these systems.

Well, okay. So, you know, not to criticize the study too much, but it was interviews of 52 people at a conference by one person who coded it themselves, right? And as somebody who cares a lot about data, I worry that people who want AI to not succeed, and there's lots of reasons why you'd be worried about AI, corporate control or changes to our future or the way they're trained, there's lots of legitimate problems, but I find they get grouped into a mega thing which is like AI sucks and let me pick out cases where AI sucks. I think if you looked at that study, you'd be like, what's going on here? And there's a dozen other studies since then that have found different results. I mean, I've done two controlled experiments at large companies where we found large performance gains from AI use. When I talked to corporate people in corporations, they're finding use cases. My colleagues at Wharton just completed a large tracking survey, 75% of companies say they're getting positive ROI from AI, only five negative. I just feel like we're looking for these cases like this is definitely going to fail. This sucks. Look, I found a sucky case. And it's really funny that you kind of illustrate with a case of failure, which is legitimate and real. You just said you had a lot of useful conversations, too. It's just interesting that we fixate on one versus the other. It's both good and bad. There's good stuff and bad stuff. There's failure cases and useful cases. And it doesn't help that AI makes it hard for us to tell which is which.

I guess my objection is we can evaluate this, right? Like there's a nice study that just came out of the University of Chicago showing that when agentic coding systems were added to Cursor, which is one of the main coding tools people use, there was a 39% increase in the amount of code being merged into systems with no increase in error rates, and actually quality seemed to go up, right? We have numbers on this. So I worry that we're in this world of anecdotes and where we have evidence, we have information about this, we've done the randomized control trials, we have data on this. And part of this is also the complication of using these systems. Again, if you were just talking to the free version of ChatGPT or the auto version of it, that is a system that is very bad at giving you consistent answers and is very sycophantic, will give you the answers you want. If you use the more advanced ones, the sycophancy goes down and the results are higher quality. It cracks business school cases, it does pretty well on finance problems. There's a whole bunch of studies in everything from retail to finance to healthcare. So it's kind of hard to have this conversation because on one hand I can say well we've got all the data that this actually is helpful and there's certainly edge cases of failure because the system isn't as good at some stuff and bad at some stuff, but it feels like I can give you data but there's always an anecdote of like yes but this person says it's bad. There is bad stuff, right? What can I tell you? Both things can be true at the same time. We're three years into a new technology that is evolving very rapidly. And I feel like in most cases the people who are very critical of AI would be listening to data more if it wasn't complementary of their view that this is garbage. And it's very easy to make the system produce garbage. And so to me, as somebody who's for example been trying to think about how you democratize education at scale for a long time, the fact that every kid in Mozambique has actually the exact same tool you do and that we can actually make it a good tutor and find tutoring results seems like one of the most exciting things in the world to me, right? And then we have to do the work to make sure it does a good job and not a bad job. But to start with the default that oh it's garbage and always produces the worst output when we know it can produce good output, it just feels like a strange position to be in when this could be a liberating technology as well as a negative one.

So I mean I do a lot of fairly complicated quantitative work. I'm an economic sociologist, right? So at a business school. So I study things like crowdfunding. I've studied things like adoption rates with new technology, gender differences, all kinds of stuff. And I mean to be honest, writing code takes a very long time. Checking data takes a very long time. The AI does a great first pass. It does a good second pass. I mean one of the most crazy things is I recently took an academic paper that's highly cited. I wrote it. It was published. It had people who were some of the top people in the world read through it. It was peer reviewed at a top journal and I was able to throw it in GPT-5 Pro and it spotted an error that no one's ever spotted before. It was a minor error that didn't change the direction of the data and then it rechecked it, redid the math, gave me the right results from it. So in the hands of an expert, I can easily save 40 or 50 hours on working on a paper using AI help. Right? My expertise matters and it doesn't work for everything but it works really well for a lot of those cases. Or writing. I do a lot of writing. All my writing online is my own. People probably think, it's my fault I tweet too much or BlueSky too much. I do both. And I write blog posts and I do all my own writing but I absolutely have the AI read through everything I do afterwards and it usually finds errors or mistakes that I make. It helps me solve problems that I didn't have. I use it for all sorts of things, both professionally and personally. And using it lets you figure out what it does. And on the anecdote side, for every anecdote about a bad thing, I can tell you some, you know, I have people who personally have found solutions to diseases that they had that doctors had been stumped on and the AI was able to solve the problem for them and put together a picture. I'm sure there's plenty of cases of misdiagnosis as well. I mean, so we kind of have this dueling anecdote problem which is like it helped me here, it cheated here. That's why I have difficulty with this because it's a global phenomenon. So AI is both absolutely causing mental health crises and from controlled experiments seems to be helping some people with mental health crises. Like how do we deal with a situation where a billion people are using this? I could find you terrible anecdotes or positive anecdotes. It's why I find the anecdotal discussion to be kind of less useful than talking about what we know about this technology.

And it's a deeply uncomfortable technology. I mean I start my book with this idea that I think when you use AI you have an existential crisis. Like I certainly did. I call it three sleepless nights. I stayed up being worried all night like what does this mean, why does it seem like it's thinking? What does it mean to be a professor that it's doing a good job doing an initial teaching run? This is something I devote my life to. I mean I think that if you're not uncomfortable with this, that's weird too. Like this is a deeply uncomfortable technology in many ways, right? Whether you view it from what does it mean for humanity or corporations, whether you view this as an existential threat. But even if you don't view it those ways and I'm generally a pretty technologically optimistic person, it is an uncomfortable technology and there's lots of reasons why people legitimately feel uncomfortable with it. And that intimacy that you mentioned is one of the major reasons. Like this is a weird thing, people are willing, like the first anecdotes people were telling me initially was the things they were having AI write for them were bedtime stories for their kids, eulogies for their parents, wedding vows, like that was the first thing they were doing, the most intimate of all things, and that was what they're giving to AI, not like work stuff, not like Godfather trivia, but help me with this difficult time. That's a very strange technology. And it is a very strange technology. It's very uncomfortable that it pretends to be a person. Like that's upsetting, I think.

So a lot of what we do is working with data, right? Survey data or census data or economic data or data that we gather ourselves. Just to tell you about one of my AI experiments for example, with my colleagues at Harvard, MIT, and University of Warwick. We went to a consulting group and we did a randomized trial where some people got access and some did not and we measured performance on different tasks, found very large increases when people used AI systems versus when they didn't. And what I found was I could just point the AI at the PDF document and at the reproducible data set they made available and the instructions how to do it and was able to reproduce the numbers, tell me where they differed and whether they differed systematically, was able to do the research work. Again, the more sophisticated models, right? If you use a GPT-5 Pro or 5.1 Thinking, they can do this kind of work where they actually write the code, do the math, download the files, look up the references, execute the code that they wrote. I got all the graphs. I could check all the data to make sure the code was right, a good source of truth, and it was. I didn't find any errors in doing that work. Part of the interesting issue is that checking the work takes time, but less time than doing it. So you might save 10 or 20 hours but it might take you an hour to go through the data. Still a good trade-off. That to me is really impressive, and the models are getting better at these kind of things. We just keep finding these objective things that the systems are really good at. We're in early days still of trying to figure out how to use these things in the right reproducible way.

I mean first of all that's a completely legitimate fear and it's absolutely happening, right? AI didn't create the cheating problem, but it creates a universal cheating tool. So we actually knew there's a cheating crisis for a long time, right? There's actually these nice tracking studies that show the percentage of people who benefit from doing homework has been dropping over the last decade because they're no longer doing the homework. They're just copying it. So they're not learning, and the problem with education is it's just like any other form of exercise. It has to kind of hurt a little. Like if you're not being pushed, you don't learn. And unfortunately, nobody likes to be pushed, right? And so the AI undermines a whole bunch of really valuable educational approaches. Like nobody asked for the essay to be dead as a concept, right? The take-home essay. We do know how to solve this stuff. We can do more at-school blue book kind of work. We can do active learning in class. So there's positives and negatives, but there's absolutely, and my colleagues at Wharton have a controlled experiment that shows if you just let students use AI, they get answers to questions. They think they learned something, but they're just regurgitating AI information and they learn nothing, right? On the other hand, if you ask the AI to be like a tutor, because part of the problem is the AI is helpful, so it just gives you answers to problems, you don't want that. A good tutor actually challenges you. It makes you answer the question, it guides you. And when you make the AI work more like a tutor with either prompting, better prompts to do that, act like a tutor, or now all the AI companies have these learning modes, they have been independently evaluated, but they do some of that same kind of thing, then you get positive results. So there's this nice World Bank study in Nigeria that did after-school exercises using AI and with teachers in the classroom. The teacher part is really important because they're doing guidance after-school activities and they found like a .31 standard deviation increase, which is actually a really large one from six weeks of after-school things. That's equivalent to like an extra year of extra classroom learning to get that kind of impact. Now, it feels like a bigger number than I would have expected but that's what they found. And we're finding similar results. There's a nice set of studies out of Harvard looking at computer science learning, a physics study out of Stanford, if I remember correctly. Well-built AI tutors have big impacts on education. What you'd expect them to do, right? If they're not factually wrong, having a universal on-demand tutor who can give you feedback, explain a concept that doesn't work, give you new examples to exercise from, help you diagnose your learning problems, like that's pretty great and the tools can do that. But again you kind of need to use the most recent model and you need to want them to act like a tutor as opposed to just give you answers.

But there's no doubt there's lots of things broken in education. Part of this is having the willpower to do something, right? Or the direction to do it. So, one way to do it is you have to use an assigned AI that's set up as a tutor. Another is you don't ask it for results or you use the study mode. All of the AIs have them. That makes it less likely for them to answer your question and better at asking you a question. If you just ask 'help me understand this concept, don't give me the answer,' you will have a good experience with a large-scale AI system to help you learn concepts.

We don't know. I mean we have survey results that teachers appear, actually when they use AI, at least, there's a Walton Family Gallup survey suggesting that teachers generally are feeling positive about it after they start using it, or at least for helping them do better teaching. They report six hours saved of classroom time a week in doing prep and help. It's a complicated picture, right? And so we're looking across hundreds of millions of users and a primary use case of AI is cheating for students at least, but all they need to do to not cheat is to not let the AI cheat, right? And basically say, help me understand this. We can motivate people to do that just like we do in other cases.

People in class tests and the people who learn will do better than the people who don't. We can give you a prompt — like if you paste in a prompt, the AI saying be a good tutor, don't give me the answers. That's all you need to do, right? It's not like you need an entirely separate AI system. You could do that in ChatGPT and it will be a good tutor, but you have to say act like a tutor. Don't give me answers. Help me understand concepts. So I mean, part of this is we have a tool that can do many different things. If you want to use it to cheat, you absolutely can use it to cheat. And that is probably the default for many people for the reasons that you gave. And that is really destructive to education. Just like it's destructive that, you know, we don't know how to teach about using AI properly. We're again three years in, right? I'm an educator myself. It's chaos out there right now. We don't know what to do. Everybody's teaching different things. Everyone has different information. We're all figuring it out as we go along. So I think it's fair to say the impact initially has been really negative, but there's a lot of broken stuff already. And it's sort of weird because on one hand you're saying the system's deeply broken, and on the other you're like, AI made it worse. There's also some possibilities that AI is making it better. We just don't know yet.

I mean, I think that there's a lot of reasons people feel negative, and I think a lot of them are very legitimate. I mean, they are based on — you know, everyone's IP goes into these systems one way or another, right? And we could have a discussion about that problem here in Hollywood, yeah.

So, like, we always talk as economists about, you know, in the long term the industrial revolution is really good and people get new jobs, everyone does better. But if you actually lived through the industrial revolution, it kind of sucked, right? There was a lot of chaos happening all at once. So I mean, I think there is negative sentiment in the US, less negative sentiment in other parts of the world by far in surveys. China is much more optimistic. But I mean, I think there is reason to feel negative, and I think that the job threat feels real. It is disturbing to see AI make art that seems good, and that feels uncomfortable. There's lots of things that are uncomfortable about AI — how it's controlled and how it's done — that I think are legitimately things to worry about. My concern is that I think we need to be thinking about how to mitigate the negative and embrace the positive. And I find a lot of absolutist conversation, right? AI is really bad for education, but honestly, you just told me how broken education is. And if I can increase outcomes in Nigeria with an off-the-shelf AI system for pennies per intervention, that feels like one of the most exciting things on the planet to do. And so we can embrace it both ways — we have to mitigate the negative impact while thinking about the positive impact. And I think that's going to be the broad case. It's a general-purpose technology. It's going to affect almost every part of society in ways that are good and bad.

Well, I mean, the worst thing to me, the worst argument is 'this is useless and it's going away.' And that's sort of the position we started from — like, oh, look at all the failure rates. That's just not what's happening in companies using these systems. That's not what's happening when scientists are using these systems. They're not useless, right? They are actually quite valuable. They do lots of actual valuable work. They probably could be useful to you when used properly, and they're getting more useful over time. So my fear is that — everything you're talking about — I want the people who are very negative on AI to be part of the conversation, about not assuming it's going to go away, but thinking about what do we do to mitigate these negative impacts, right? I agree, AI should not be subbed in to solve all the problems in education, but it might be part of a solution. So how do we redesign or think about the education system getting through this crisis of AI and the pre-existing crisis to create something better, rather than just seeing what happens and hoping that this all goes away? So my concern is that the people I want part of this conversation really are just hoping they can ignore this whole thing. And that's just not the case, right? This is an epochal technology. According to every survey that we have of forecasters, this is a very big deal. And pretending it's not a big deal is, I think, a really big problem, because the critics' voices don't get heard — they could just ignore it. And because the critics are in a sense sometimes removing themselves from the conversation by refusing to have it. And, you know, muting anyone who has this conversation — I'm sure there's a lot of YouTube comments here on, you know, AI is bad, and it is okay to think AI is bad. I'm not going to try and convince you that there's no bad impacts, because there are bad impacts. I just worry that if the answer is 'AI is bad, it's going away, it's all a scam,' that's the most dangerous way to think about this stuff, I think.

And nobody really knows what's going on, by the way. I talked to all the AI labs. I talked to nonprofits. I talked to government officials. Like, you should be clear there's not somebody steering the ship. It just turns out the AI models do this stuff. We didn't know they would do this stuff. When they released ChatGPT, it never occurred to them that they were destroying a lot of college education because people would just be cheating on essays. They never thought of its use in medicine. It turned out it was pretty good at diagnosis. Nobody expected it to do that. Nobody expected the impacts that these things have had. And the truth is that the AI labs are going to keep releasing what they do. If they didn't do that, there's Chinese labs that are releasing open models that are available to everybody on the planet. I mean, the chance that this is just disappearing doesn't seem a realistic one to me. We could argue about how far the technology gets and does it hit a wall or what. Right now, it hasn't, right? We're seeing pretty steady exponential growth on almost every measure we have that's long-term measures of AI ability — long-term being three years at this point, or five years. But we didn't ask for this, but that doesn't mean that we can bury our heads in the sand. There are reasons to be hopeful as well as reasons to be negative. So there's reasons to not take the maximally 'it's all going away, it's all a scam' position, right?

So, I think that — first of all, things take longer than people think, right? Actual work is complicated. There's all kinds of — processes have to change, approaches have to change, adoption takes time. Nothing happens as instantly as people think they're going to. That doesn't mean — there's something called Amara's law when you predict the future, which is people overestimate short-term change and underestimate long-term change. I think we're going to see that same thing happen here. But I mean, a nice way to think about AI and jobs is think about jobs as being exposed to AI or not. Like, that doesn't mean replace, it means overlapping. And I think about this a lot because in the initial work on what jobs overlap most with AI, business school professor was 22 on the list of 2016 most disrupted jobs. And I think about this a lot because — but think about my job. Okay, I'm a business school professor. I teach, and what I do is many things — I go on podcasts like this one, I have to create assignments for my students and grade them and teach them and provide emotional support to them, help write letters of recommendation, be an administrator, do academic research, write books. My job is a bundle of many tasks. And if the AI is better at some of those tasks than me — say it's better at grading, which turns out it is actually better at grading, though I do my own grading for social reasons, right? My students would be angry if I didn't do that. Even if the AI would give better grades, I still grade myself. But let's say grading gets taken away or part of my writing does. Then there are other parts of the task bundle I will do more of and expand it and change what I do. So I think transformation is a more likely outcome for most jobs than destruction, right? What we do changes. If the AI helps you do podcasts quicker, maybe you launch a second or third podcast series, or maybe the research side you can now expand to do a wider swath of things or translate to other languages. So I think that some of that will be negative, right? And there's obviously corporations have an interest in removing jobs. But I think because the AI is jagged — it's good at some stuff and bad at some other stuff and it's hard to know a priority what those things are going to be — I think we'll see a lot more change than we will see replacement in the short term. Now, it is worth noting, as you did, that all the AI companies really do think they can replace all human labor in like the next 10 or 15 years. I'm more doubtful about that. There's reasons to be anxious about that because they think they could do it. I'm more doubtful as somebody who studies technology and change in work, but I think it is worth noting that there are going to be jobs that change and transform and some jobs that disappear, and we just don't have a clear picture of that future. Like, I'm talking to Nobel Prize winners about this stuff — we don't know, right? We can only make projections about this. I think change is very likely, wholesale replacement of jobs is less clear at this point.

Yeah. I mean, I think there are cases where — AI writes a pretty good press release at this point. I mean, it's getting better at creative writing, but it's nowhere close to the top 1% of creative writers. I mean, one of the most interesting things we found in repeated studies of AI use is that low performers get the biggest boost from AI use, and it's actually a task-based problem. What happens is you have to do many things as your job — as a podcast host, you're really good at a couple of those and you're probably pretty bad at a bunch of other ones. So if the AI is at the eighth percentile on the stuff that you would have been bad at, that's actually a huge net gain for you, because those were probably the parts of your job that stressed you out the most, you got the least joy from, and were in the most danger of being in trouble. I'm a former entrepreneur. I teach entrepreneurship and innovation. And entrepreneurship is all about being really good at the 99th percentile in something and then being terrible at everything else and hoping the things you're terrible at do not destroy your thing you're good at. So the fact that the AI helps you support the things you're less good at and brings those tasks up a level doesn't necessarily mean destruction, right? And this is another thing I wish people would get more of — this idea that, sure, if it helps you with some of this, it doesn't have to be 100% to give you a lot of support in areas that you have less expertise in to help you do the things you do well better.