Scaling Laws: Ethan Mollick: Navigating the Uncertainty of AI Development

Thanks for having me.

So I feel pedantic on a scaling law podcast saying this, but I think that the product releases are probably less interesting than what they're telling us about the scaling laws. And especially the reason why we're kind of in the place we are is there's now scaling laws and not just laws. So these laws, just to make it clear for everyone, not empirical proofs, they're observations. But the first scaling law is that the bigger your model is, the smarter it is. Right? Until very recently, hyperscaling, this idea of building larger and larger data centers, training your AI on larger and larger amounts of data, that has been the driving force for AI development and why they've been getting better. But there's sort of a cyclical nature to that because you need to build a new data center and get a lot of data together. And we're seeing the new releases of models are much larger than the last ones. But then OpenAI showed us there was a second scaling law, and there's probably more as well, which is the longer your model thinks about a problem, the smarter it is. So there's now two ways to make your model smarter.

You can either spend a lot more computing power training it or spend a lot more computing power where it thinks through the answer to each problem. So to kind of go back to the fundamental kind of question, is this still rowing? Well, the first scaling law, the scaling laws also say, by the way, you need like 10 times as much compute to get 10 or 20% improvement. So they're always sort of decaying, right? You need more and more effort to move the needle by the same amount. And so I think there was a lot of worry that maybe the first scaling law would run out. Now that we have other ways of making models better, it feels like there's a lot of room left for improvement. So I have a very different feeling, I think, than Silicon Valley does today versus how they would have felt, you know, if you asked them six months ago before we knew there were other ways to scale models. So this is a very long professory way of saying that I don't see a wall right now. It's more that the labs don't see a wall. I'm just a conduit for what I'm seeing in the world there, and I don't see a reason we should suspect that AI development is going to cease sometime soon.

Whenever we're layering our own uncertainty on top of uncertainty, right? Because what's AGI? What's the end goal here? Is there some sort of reinforcing mechanism or feedback loop? There's a lot of questions that start to come up pretty quickly in this kind of situation. And so I think you're right. I think that there's arguments that the third scaling law is about parallelism, there's probably a fifth and a seventh. These are just multipliers at some point. It stops being scaling laws. It's like there's a bunch of levers we can move to make AI better. And we're not even talking about algorithmic improvement, right? There's a whole bunch of papers lying around saying that well rather than just using LLM, you can do this or you can combine LLMs with this clever mathematical technique and you end up with something better, or tool use is another angle that people are pursuing. If you have AI use tools, does that change things? Even just adding more memory, does that do things? There's a lot we don't know. There's a lot of unexplored paths.

And part of what gives me some confidence that the AI labs believe what they're saying is they don't seem to be exploring those. There's a whole bunch of things they could do to add efficiency to make these models move faster. They're not doing it. I think they feel they have a straight shot. Are they right or wrong? I don't know. I think it's unlikely that we've hit a dead end, a wall into all of AI in the near future based on the number of options available. And as you said, they're all confident. They're confident privately, they're confident publicly. I think one of the most dangerous things you could see is dismissing this stuff as marketing. They might be wrong, but I don't think that they're just making this up to be impressive that they think they're going to get to AGI. They really believe it, for better or for worse.

This is why I try to not talk to long-distance runners. This is, you know, why you're going to get invested in ski-joring. This is the winter, Allen, that I'm counting on you becoming the ski-joring champion of Minnesota.

I mean, the most academic answer of all is it depends. Which is what I kind of feel like. I think the issue here is you're exactly right. One thing that gets obscured, Tyler Cowen said o3 is AGI. I wrote a whole piece on this, like for some jagged frontier version of AGI where it's clearly superhuman across many characteristics and not others. Definitely, right? We already know that if you're doing differential diagnosis, as a doctor you 100% want to be using AI to do this. It's already better than humans across most of that field but not across other parts of medicine. So part of this is figuring how it works, building the systems around it.

And I really do believe that the systems now are so much more disruptive than people believe. If you talk to a product manager, they believe product management has to change. You talk to a marketer and you can see already marketing is going to have to change. That's year-long, years-long projects to figure out how that's all going to work. Law is clearly going to change. Talking to lawyers, I am not one, but who've used deep research before it's even connected to good law libraries, and it's like this is producing 40 hours of associate work pretty quickly and takes an hour to check. What do we do with a billing model on that front? So disruption is already baked into the system right now.

I would just say we could focus on that alone because what ends up happening is a conversation about what the future holds, the hype question. But the truth is that the labs keep showing amazing things and I'm surprised by some of these releases and I think they're surprised by what they could do because they don't even know, right? They're doing limited testing of all of these things too. So I think that we do have to pay attention to where things are going. The question I never weigh in on because I have no insight into it at all is: is a superintelligence possible? A machine that's generally better than humans at everything and kind of wakes up one day and decides what it's going to do with us. That is the big fear everybody has. That's the big policy question, the nuclear weapons style question to deal with. And I just have no insight onto that at all. Other than there's enough smart people worried about this that we should probably be worried about it. But what ASI means, what AGI means, what that means for employment, we're figuring this out as we go. And I think in some ways it's already baked in.

I mean, yeah. The productization of this stuff is terrible. It's a chatbot. A chatbot should not be the way you do real work. There's a reason we don't use Discord as our universal format for getting all work done. It's an insane tool, right? It's built for individuals to use. It's supposed to be a friendly helpful assistant and yet we're having it write essays for us and people are just bolting on things into the system. We at Wharton, at Journal, at Penn, I run, we've been building tools on the API that let it listen in conversations and give you feedback. The capability set is very, very large. The UX is holding us back, right? The experience is holding us back, which is why again I'm so confident that these systems could do a lot. I think they could do email. I think if they were actually integrated into spreadsheets in a useful way, and we know how to do that but no one's bothered. Even just the reward functions of these systems are set up really around math and coding and not around writing or engagement or the many other things that you can start to see them do. So I think we're held back by this.

I think a large part though, the underlying belief of the labs is intelligence solves all these problems. If we make the AI smart enough, who cares, right? Why would we waste any time building an interface when we could spend all of that compute time building a better model? And then the model will build the interface. You're just going to ask Claude code to build it. It doesn't matter anymore. It just will know what you want, right? It's heard this podcast, we get off of it, and it's rearranged my emails and dealt with the whole situation. The implementation is, you could do science, why do engineering, is sort of where a lot of them are. But there is this genuine belief that smarter models solve everything.

And there is a little bit of truth to that, or at least some truth we already know, which is the companies like Microsoft that early on kind of committed to the ability level of models. So they built Copilot, which I'm sure a lot of your listeners have used in one way or another, was built around the idea that GPT-4 is a limited system and its capabilities are limited, so they've isolated the AI and it only does very narrow things. And now Microsoft's kind of stuck in this Copilot model when the AI is sort of floating everywhere now and can do a lot more work than it did. And I see this in companies all the time. They committed to building an AI system around the limits of a Llama 3 model or a Llama 2 model back in the day and they built all of this cool infrastructure to support the fact that it wasn't very good at this stuff, and now you could just ask Claude or ChatGPT and it just does the whole thing now.

So there is this kind of wait equation problem too that I'm sympathetic for the labs about. And what I worry about, of course, is that you decide to create a product today as a startup and then OpenAI or Gemini crushes you later. The number of times my students, I teach entrepreneurship, the number of times my students have launched ideas that help you do virtual clothes try-ons and those have received funding is quite large, not just my students but lots of people. And then Google just sort of announced, oh by the way we have virtual clothes try-on, it's just a thing that the system does now. Congratulations. And the scream of dying startups echoes through the void. So I do think that you're absolutely right, the interfaces hold us back. I don't know if they're not making a good bet with like, let's wait till the smartness plateaus before we start building the interfaces.

Yeah. Part of the problem is they're not really forecasts, right? Because we can't forecast anything. So they're science fiction stories that might be right. They're provocations to get you to think about these issues. And it's just interesting to hear 2027 be referred to as a report. Like literally our vision ends right now, right? The labs have another six months that we don't have, six to eight months. But they don't have six to eight months on what the implications are. They just have six to eight months of model stuff. They don't know what these things are good or bad for either.

We don't know. It's a hard thing to deal with because you started with saying what do you know about prompt engineering and what are the results of this? How do we start thinking about what people should learn or do? And that requires taking a bet on where things are going and we instantly run to a wall there, which is what, let's say we have super-intelligent machines. What does the world look like? I have trouble even imagining the world where we have o3, Gemini 2.5, Claude 4 level models that we actually exploit the way we're supposed to. Where you have whatever the magical Jony Ive device is, because it's possible to imagine a bunch of devices that just, at this point the AI could listen to everything you're doing, process it 40 different ways, take action in the world to help you. There's a failure of imagination about what these systems could do.

So we're trying to do this ambidextrous thing of thinking about the far future which might be four years away and also trying to adapt to today and we have systems that need to adapt. Like people need training today, what should I learn to be good in a world of AI? And the answer tends to be adaptability, basic management skills, more than a technical set of skills. So we're dealing with second-order change, right? Not just things are changing, but the rate of change is changing. I don't know. It's a hard question to answer.

Well, there's two things here. One is if we do get to the level where we have the machine overlords, who cares about our skill set, right? Because it's like, we'll do whatever. Humans will adjust. We'll find meaning and whatever. But I think that the more profound immediate question is what skills we're telling people to get, assuming a world that stays somewhat normal, but AI keeps getting better. And is it great at everything? And I do think there are answers. And I think that your list of things mixes together answers I would give and answers I wouldn't give, right? So I don't think emotional intelligence and critical thinking. I think those are attempts to draw bright line skills where like, oh, humans can do this and AI can't. Or to deal with a world of AI BS, we need critical thinking, or only humans are emotionally intelligent. Neither of these things are true, right? The AI is showing emotional intelligence. It's showing creativity. Our attempts to do bright lines of machines can't do this because they're human skills is a really bad way to approach things.

On the other hand, I think that there is value to be said in prompt crafting is less important than management. Turns out AIs respond really well to management ability. If you can articulate a problem, be able to give feedback when that problem works or doesn't work well, be able to envision what steps you want to see, that stuff is all actually very useful. And it turns out experience is also very useful. We have a paper showing that junior consultants did much worse using AI for giving recommendations than senior consultants because they get a document back and it looks good and they don't have the deep knowledge to understand what's good or bad about it. So I think it's important to both say there is a little palliative kind of like, oh humans are still valuable, we still need your insights, that I don't think we have as much evidence for. But I think at least in terms of using AI today there are things that make you better at using AI, we haven't articulated all of them yet, and that we don't want to dismiss those by just saying a lot of things people say about what you need to do to survive in a world of AI are kind of fake.

So there's a lot there, and so let me try and answer in a couple segments. First off, at the current state of AI, going back to the interfaces question, it's not built well for education. And we know that also because it doesn't match what we know about pedagogy. So if you ask AI to help you with something, it often gives you the answer. That's not tutoring. In tutoring, as we know, you want to force people to try and answer the question themselves, guide them to an answer, adapt to what they're doing. So if you just give AI to students, they ask for answers, they don't go through any intellectual effort, they think they learn something and they do worse on tests. On the other hand, if you configure AI to work like a tutor and give it people classroom support in the right kind of way, you get fairly big boost to learning outcomes.

So there is a little bit of sophistication here beyond just everyone should use AI to boost their learning because it doesn't do that by default, just like it doesn't do a lot of other things by default that it could do. System prompts can change that. A good prompt makes a difference. Adding other tools makes a difference. And in general, there are topics that we don't want to teach with AI. When the calculator came out in the 1970s it caused a crisis in math education and as a result now math education is probably one of the best thought-out areas in education because we had to reconstruct it from the ground up. So there's things you don't do with calculators, things you do do with calculators. There's a careful plan of how to think about pedagogy on that front that we haven't had to do in other fields. We haven't had to do in law, which has done things the same way for a very long time.

Maybe that's the right way to teach. Maybe it isn't. But there's not really a theory of learning behind it. It's just that this is the approach we've always used to teach law, or business school cases are literally taken in the 1920s to make the business school look more like a law school. That was a professionalization move, not a pedagogical move. So our pedagogical world is a mix of things that we know work and things that have seemed to work for the last 2,000 years, like essays, even though we don't know why they work, and now they broke. So there's a lot of crises happening all at once in education.

We also know that there is a big role for kind of student-to-student contact and classroom stuff. Do we want everyone to be as advanced as possible or is there reasons we want to have people in classrooms of the same age range even if they're much smarter? We've all seen people accelerated through five classes and it doesn't necessarily do well for them either. So there's a lot of complex questions there. So I'm going to put the education policy piece aside to say there's obviously huge benefit to AI. Educators are pushing back against that in ways that I think may not be productive, especially when you start looking at education in places where people don't have access to good classroom education. Big gains. Okay, so that aside.

AI in my classroom. When AI first came out, when ChatGPT first came out, 3.5, I kind of went viral first for my syllabus which said you can use AI but you're responsible for the errors. That was great for the three months that ChatGPT was using 3.5 and then as soon as it switched to 4 it became as good as my very, very smart students when they're not putting full effort into something. So I had to go back, revise the syllabus, and it just led to a complete riot.

Yes, completely changed. And now my classes are 100% AI based. There's AI simulations. You teach the AI things. There's a co-create a case with the AI. You get debriefed by AI. So I teach an entrepreneurship class, though. I'm lucky I can do something like that. And instead of launching a product at the end of a six-week class, you do it in the first day. I can do those kind of things because of the kind of classes I teach. If I was trying to teach English writing, I would approach things very differently. I think AI is less useful for that than making people write and maybe getting feedback. So there is differences and there's opportunities available, but we have to think hard about what we're trying to accomplish educationally.

And I think it's a mix of banning AI and fully embracing it. And again, the interfaces are fighting us, right? A chatbot is a very bad way to do teaching at scale. So I think the transformation is there, but you're seeing the same problem you're seeing in every other case, which is rushing to yes, we should throw out all the lawyers or throw out all the teachers. It's too early for that kind of work and probably always going to be too, always a terrible idea. And we've been through this before. Technology alone doesn't solve these problems, but for the first time I'm hopeful that it can solve a lot of these issues because what the models can do is very impressive off the bat. So we're in an interesting set of changes and I think thinking about how we embrace this in universities while also not saying we need to throw everything out is also important.

So I think there's two kinds of red lines to draw. The one I drew is a great example of a system thing. There's an expectation for my students that I grade their papers, even though I can tell them that AI is better at grading than I am and will give them more insightful comments. This is what they expect, there's a social requirement for this. So the system that's holding us back is a social requirement. And that's completely fine, right? They're getting worse results from me, but maybe I offer them both options.

And then let's take one step further in this. Let's think about another thing that has to be human, which is letters of recommendation. I write letters of recommendation all the time. The goal of a letter of recommendation is to set my time on fire as an indicator that I care about somebody. In 45 minutes, I write a pretty okay-ish letter. The students will give me some guidance of what they want to write. And I get an infinitely better letter if I take from a student their resume, the job they're applying for, and I say, 'I'm Ethan Mollick, write a good letter of recommendation.' But I'm no longer indicating anything of value to the outside world.

When I ask my students what they'd rather do, all their hands go up that they'd rather have the one-minute letter than the 45-minute letter because it's more likely to get them a job. And that's because the signaling still suggests Ethan Mollick cared enough to write a letter for me. How long that lasts becomes an open question. I also had a student for the first time last year and they allowed me to share this, who sent me a prompt that I should use to write their letter of recommendation. So literally they just emailed me, please use this prompt. Feel free to adjust.

But you could see both sides of the social coin, right? My students' expectation in one area that's supposed to be human and in another area that's supposed to be human. And this is what we're really navigating at a very micro level.

And then there's some really interesting cases where maybe those kind of constraints are actually helpful. So there's a big deal where Hollywood actors negotiated a bunch of AI rules and my sister's a producer in Hollywood, and it's been really interesting because those rules actually allow for creativity with kind of constraints. So, for example, before she does a photo shoot with actors, they'll use AI to mock up potential photo shoots, but they can never use the AI to actually produce a final photo shoot. She's just done a movie with Michelle Pfeiffer. They can use Michelle Pfeiffer's voice and automate it for internal testing that they're doing to figure out what ADR lines might work somewhere, but they can never show an external audience the voice of a Michelle Pfeiffer ADR by AI.

So there's a really interesting place also when you think about lines, there's those three versions of it. There's no one should use AI to do this. There's we're embracing AI and it's completely destroying the underlying meaning of the system and we're going to have to grapple with that. And then there's this mid-level thing of we're going to create these particular bright lines that we know will actually protect people in some useful way while allowing for AI to cut down on the grunt work that would have been miserable that nobody wanted to do. And I think that's going to be a compelling problem in law and everywhere else.

So I think there's two pieces to what you're doing. One is assessment. It is a completely reasonable approach to assessment in law. And by the way, if you graded active learning as opposed to an exam, that would also be reasonable. So both in-person exams and active learning are really good ways to grade. They've always been good ways to grade. Tests have always been useful, even though we hate them. And it turns out testing also makes people learn stuff. If I was going to give you one piece of advice, by the way, it would be more small tests along the way is the way you would actually want to run this so you get feedback as you go.

Okay, so let's leave that aside right now, which is the complaints about we don't get pedagogy, we just do the thing we always did, the systems are in the way. I would at the very minimum, for your large school class, I would be building tutors. I wouldn't trust people to just ask ChatGPT. I have a tutor for different subjects where I have the AI do stuff. I would have people do a simulated case discussion with the AI because not everyone gets to be part of a case discussion. So I would have a case discussion available in each case. And by the way, I'd be increasing the realism by having some of those be where the AI acts as the attorney on the other side. The AI acts as the judge and grills you on stuff. I would have people apply them to a fictional setting. So like you show up and the AI, you don't know what case it's going to play out, but it's about a case on Mars and now you have to make a decision about that. So we're changing structure and approach like simulation. I'd have the AI act as a befuddled law student because teaching others is very helpful. So have you teach the AI how to do something, right? I'd have it, you know, prepare me for the test kinds of models that are also pulling back maybe in RAG style from previous tests. And these are relatively easy to build. And by the way, if you go to the Generative AI Lab at Wharton, we have a whole bunch of Creative Commons prompts that we've released that are designed for teaching and you can modify for yourself. So I think you are leaving learning on the ground. Even just keeping your classes structured the exact same, the idea that you're not offering a bunch of tools that could help people learn is interesting. Then make them assessments. If the TA is the problem, then why not have AI, like you could use AI grading for that. I'd say, look, you'll be using this for feedback along the way. And by the way, you can actually throw in every part of what the AI has given the students feedback on and you throw that into the system and you say, what themes am I missing? Where am I failing as a teacher because people are not getting this? What should I do in my next class to close the gap between where I think they are and where they should be? And also on the teaching side, there's some early evidence that teachers who use AI to help them prepare for teaching do better as well, especially if you integrate your interaction with the AI. So I think you're leaving a lot of value on the table. To be very honest with you, I don't think the core problem of assessment is as big an issue as there are ways of having people catch up on learning and it's relatively easy to do. And if you don't care about how they're interacting with AI outside class, then these are easy things you could take advantage of.