AI has a jagged frontier, and this jagged frontier means AI can't do everything, but it transforms parts of work. So bottlenecks change. So the value of your job kind of depends on how much of a bottleneck you are.
The firehose of AI news can make it difficult to separate hype and hope from reality. So we thought it would be helpful to talk to one of the world's leading experts in what's actually happening. This expert, helpfully, is not selling anything. Ethan Mollick is a professor of management at Wharton. He's also the author of the very popular Substack, One Useful Thing, which is dedicated to tracking the development of AI. Professor Mollick says AI tools are advancing even faster than he expected when he started following the industry closely a few years ago. We discuss how they have already radically transformed his classroom and what his students can do. His students are not just encouraged, but required to use AI. Here's our conversation.
Professor Mollick, great to see you. Just speaking as a consumer of your Substack, One Useful Thing, thank you for writing that. It's so great to have somebody sort of sitting between the Silicon Valley hype machine and the audience who does not have time to look at every new thing, knowing that you will look at it and tell us what it means. So, thank you.
Well, that's really nice of you, and I'm glad to be here.
And folks who don't know you, they may know you through your otter test, which has also been very fun to follow over the years. Tell us what that is, because it's gotten quite impressive recently.
Yeah. So, you know, there are all kinds of benchmarks for AI, right? And all of them are flawed. So you might as well pick one you like to some degree. So my daughter is a huge fan of otters on airplanes. At one point, I was sending her AI-generated images of otters sitting on airplanes using Wi-Fi cuz the Wi-Fi was going in and out. I posted them, and the threads went mega viral. And ever since then, I've been testing AI's image generation and video generation ability by asking it to create pictures of otters on airplanes. And it turns out that now we've gotten into this phase where not only are they really good, but you could generate an entire movie about it from a prompt, which is, I think, pretty impressive.
Yes. It's extraordinary. They get ever more lifelike and funny and so forth. So it's amazing. All right. So you recently stepped back in your Substack and basically said, "Okay, the future is now emerging. It was visible a couple of years ago, but now it's actually happening." So where are we?
You kind of think of AI as having three phases, right? There is the pre-large language model AI, which was big data, and that was a huge business, right? A lot of the huge investments through the 2010s and early 2020s were based on if we got our data organized enough, we had the right data scientists, we could do really great predictions of the future numerically. So we could choose how to price things to people, where demand might go, all kinds of things like that. Then with ChatGPT, we sort of entered what I'll call co-intelligence, for my own books, but you could call it other things as well. And that was a phase where you use chatbots. So you'd work back and forth with a chatbot to make things happen, right? So I'd say write me this piece of code, or write me this email that wasn't quite good enough, can you make it better? And that was the way we sort of worked with AI.
What's happened the last few months is we've gotten into the stage of having practical AI agents, which has been a marketing term for a long time, but now is a reality. And an agent is an AI tool that can be given a goal and can accomplish that goal autonomously using its own tools, often many hours' worth of human work it can do at one time. And that is a real transformation because agents start to mean I can delegate tasks to AI and have it do it. I can expand what I can do because the AI just does the work. Rather than me being in the loop at every moment, the AI goes out and does stuff and then reports back to me afterwards. And that has created this really big step change. We saw it first in coding. Coders are getting very nervous or excited, depending on who they are. But we're going to see that in other fields as well.
So give us some examples of that. A lot of it seems very complicated. There are a lot of deep tech folks who are doing it, but everybody else is sort of watching from the sidelines. So when you talk about that actually happening, what are you seeing?
I think among software developers, it spread very quickly from a deep tech weirdo thing to fairly mainstream, right? We have evidence from the previous generation of the early agents that came out a year and a half ago. They were increasing coding ability in randomized trials by something like 38%. So that was already a huge leap. What's happened now with coding agents is you basically can code in plain English. You could tell them, "I want to build a piece of software that does this." And if you're expert enough about coding that you can look at the results and give feedback, a lot of the best programmers are barely touching code anymore. And that's not just a weird high-tech sort of San Francisco thing. I see that kind of happening everywhere. We can start to see signs of other forms of AI, you've heard of OpenClaw and all these other approaches. Those are much more nascent and weird, but the coding stuff is pretty normal at this point.
And you talked about one company that has created something I think they call the software factory, where basically humans cannot either create or evaluate the code.
Right. So if you think about software as the sort of early signal, right, the harbinger of what's going to come next, you're also going to start to see a few interesting issues. One of them is software development has basically operated in very similar kinds of patterns. There's sort of two patterns: there's waterfall development, where you have a big plan and do step-by-step work, and then there's been agile development, which is what most companies do, where you have short sprints of creating software one time after another. And it's all built around these kind of human meetings, right? You have design meetings, you have standup meetings. Suddenly, when people can create a hundred times more code, those old methods of working that worked really well for humans don't work anymore. So we're starting to see radical experimentation with new approaches to developing software.
One of these is a company, StrongDM, that developed what they call the software dark factory. And much like a dark factory is an entirely robot-run factory so you don't need lights on in it, this is a software dark factory. So you just put into the software factory roadmaps of what you want to accomplish over time, and the AIs actually generate the code. Then they have fake customers, software that's built to test the code. And the customer agents have built their own fake versions of Slack and Gmail so they can actually test it in a real environment. And then in the end, what you get out of this whole thing is just software. And no human ever sees the software. No human ever tests the software. Ideas go in one end, software comes out the other.
And do you need to be, at this day and age, a great engineer to do that, to oversee it, to be able to specify the product clearly enough for the AI system to do that?
So I think those are three different questions, actually, and the difference between them is important. Because with software development, probably your ability to write just good boilerplate code, which is an important skill, has devalued. Your ability to be a good manager of people who write software is increasing because the AI kind of works like an employee doing work for you. So there is a shift in what is important to be a good software developer that is sort of unexpected and fast. I think we'll see that across many fields as what makes work hard changes from, "Can you write enough lines of code each day?" to, "Are you good at envisioning what a product requirement document (PRD) looks like? Are you good at giving feedback and building test steps?" Those are very different questions.
We'll talk more about the impact on jobs and what we're all supposed to do if our skill, our primary skill, is now done better by AI and much faster. But just in that specific example, I know from having run a team of engineers, the jump from being an actual line coder and developer to being a manager, it's tough. It's tough in everything. It's tough in journalism going from a writer to an editor, and at universities, a professor to a dean, and so forth. Is that something, do suddenly everybody has to, instead of learn to code, actually learn to just manage big software projects by having others do the work?
A few things. One is that's probably also a temporary state, right? I think people are overindexing on where AI is right now compared to where it was a little while ago. And the truth is that we're still in early days for a lot of this kind of stuff. Right now, that's the answer. But of course, part of what makes becoming a dean hard or becoming an editor hard is you're not just managing output, you're managing people. But it starts to be easier when you're delegating tasks but don't have to deal with all the other things around delegation that make delegation hard, right? Can you give good feedback to a human, and can you make them happy, and can you get a team of people going? Can you run a good meeting? So it's weirdly parts of those managerial tasks that go to your level and parts don't. And again, it's a moving target in a lot of ways.
And another recent example of the incredible productivity improvements that you've talked about is teaching your class on entrepreneurship, where you basically, I gather, ask your students to, okay, create a startup and use the tools to do that. And it sounds like it's changed radically in recent iterations versus a few years ago.
Yeah. As academics, we think of semesters, right? So before AI came out, it was a semester's worth of work to produce... You'd come with a startup idea that was like a month in, and you'd pitch a PowerPoint presentation, you've spoken to a couple of customers. That was the expectation four years ago at the end of a semester's worth of work with very smart students. And then AI came out, and I introduced my class immediately and it became, "Oh, you can get some working software out of this, you can do..." But it was a co-intelligence piece. So I expected more work, but better quality.
My most recent version of this, vibe founding, people had three days to launch a startup company. It was really two days, like a day was teaching tools, and they had to have a working product, and they had to have a PowerPoint, and they had to have simulated customers. And I've been doing this for a long time, and I saw incredible outputs. People did have working products or at least demo working products at the end of a day and a half of work. That changes how we start to think about how even entrepreneurship operates. Should you try 50 companies at a time and see what works or not? What does it mean that what was holding you back was some weak spot, like you weren't good at finance, but the AI is good enough at finance for you to advance what you do?
And what's the reaction of the students when you have them do that and you say you have to use these tools because it's got to be much faster?
Generally, I'm lucky enough to teach an entrepreneurship course, right? So because I do that, it's pretty straightforward because people want to launch a company. So anything I give them doesn't feel deskilling, it feels like an advantage, right? Because what they want to do is launch their organization. And entrepreneurship anyway—and you know this having been through it—is really about being world class at one thing and hoping the 99% of other stuff you're not good at doesn't kill you, right? So it's perfect for AI stuff because in their one specialty, they're definitely better than the AI, with their background experience. In fact, I made them all do what I called the Voight-Kampff test, after the test in Blade Runner, where I had them write down what their unique skills were and what their unique experience was, and their startups had to include that. So I felt like the reaction was very positive. When I've talked to other groups, it can be very negative or very disturbed, and I think there's reason to feel disturbed and confused by where things are right now.
Yeah. I think you've said that students should learn to use AI and use it in school.
So with a caveat: it turns out if students just use AI and just let the AI do the work for them, then they don't learn anything, right? And they could be fooled by that. They could say, "Help me solve this problem," and the AI explains how to solve the problem. They're like, "I learned how to solve the problem." But if there's no friction, if there's no difficulty, you don't learn. So my classes are 100% AI-based in that AI tools they use, and they interact with AI systems. That's great. AI mentors, AI simulations, they build a case with the AI, and all kinds of cool stuff you can do. But it does have to be deliberate, right? It can't just be "use AI and everyone learns more," because they don't.
And how do we deal with that exactly? I used to be a research analyst a long time ago. You'd study a new industry. You'd talk to everybody in the industry. You'd develop a thesis. You'd come back. You'd write your report. It would take months. I had ChatGPT do a market analysis for me. It comes out in 6 minutes. Is it the best thing ever? No. Is it way better than I probably would have produced in my early years? And did it take six minutes versus months? Absolutely. But when I look back on that, I think, yeah, but those two months, that's when I learned and that's when I came up. So it's straight to your point. How do we, given that the technology is so much better and faster than we are at a lot of these tasks, how do we also manage to learn?
That is a big question. I'm not worried about classrooms. Things are screwed up right now, but we'll figure it out. It's already pretty obvious we'll use AI tutors outside of class; inside of class will be activities and engaged interactions. We'll be fine, right? We'll figure this out. We still want to test people, just like we figured out with calculators. I am deeply worried about the next thing you talked about. I teach people to be generalists at Wharton, and I send them out into the world and they become specialists the way they've always been specialists. Like in your case, they work for a bank and they write analyst reports and they get yelled at by their boss if they're not good. And they're learning a lot about the world beyond just the report they're writing, because they're talking to people, they're at conferences, they see how people react, and they learn to be experts the same way we've taught experts for 4,000 years, which is apprenticeship.
And that model is the most threatened because if you're an intern or you're a junior employee, your goal is to impress your boss, right? Even if you want to do the work yourself, you'd be dumb to not have AI do the work for you because it's better than you as an intern, and all your bosses would rather not assign work to you. Why have you do the research report when I can have the AI do 20 of them? I can check them and give feedback to the AI. Once it's running, that machine is much better than working with junior people who have emotions and feelings, and may or may not be good at a task, and someone gets sick. So it breaks the entire apprenticeship pipeline. So what are we going to do to reconstruct that? I think we have to be deliberate about how we learn at work and outside of it. Maybe it's longer university courses where you learn to be a specialist at school. Maybe it's that you have to start building official mentorship into what you do and having people not use AI to solve problems. Maybe we have to test people. We don't have answers, but I'm really worried about that pipeline.
And so you're a professor of innovation and entrepreneurship. Have you created a Professor Mollick bot yet that your students can talk to 24 hours a day?
I think that's the easy solution, right? I don't think you want a fake Professor Mollick bot. What I have built for them in various classes is... You want to think about what they want to learn. Talking to Professor Mollick bot is not a learning thing, right? They're just getting fake information from a fake version of me. Instead, I want systems built for tutoring very hard concepts. So we've been building tools and releasing them open source that do things like, for example, build simulations. You talk to the AI and you give it your syllabus, and it creates a simulation for you, or that act as a devil's advocate that you have to argue with, or that help you with a team setting. So I think we have to be more imaginative than "Professor Mollick bot teaches a class." I think it has to be beyond that kind of piece.
People tell me, one of the weird things about having a very narrow sort of fame is I occasionally run into people who are like, "Oh, I built an Ethan bot and I talk to it all the time about advice." I'm like, "All right, I'm not quite sure what to do with that information." But I don't think we have to replicate a person to do this well. I think we have to think about what makes learning work.
Right. And in that, there has been a dream of folks who do hope to bring technology to education that we can have truly customized education for each one of us. There is that image—I'm sure you saw the original new Star Trek where on Vulcan, all the young Vulcans are educated in their own little bowls that have their AI teachers or what have you, and it's all individualized and so forth. Is that something that is promising, where each student is going at their own pace?
They have a randomized controlled trial they did in Taiwan where they had their students use a personalized AI tutor based on the now-obsolete GPT-4o, so not even a great system, and they found the equivalent of an extra few months of teaching for people who used a customized tutor outside class. There's another experiment that was done with the World Bank in Africa. There's been a few of these. It's very clear AI as a personalized tutor is great, right? And all the AI companies have a sort of half-baked learn mode, which I would recommend using if you're a parent if your kids are going to use this, which asks more questions rather than being a helper that gives you answers. To me, personalized education, like the one-on-one tutoring, I think we're going to nail that. People already should be using it to help them with that, as long as they're careful. This is great for parents to weigh in and watch what's going on. We'll figure that out.
I think that people, though, underestimate the value of teachers and instructors in classroom settings, which is the uneven piece. You don't want to be in a Vulcan-like bowl, because a lot of the stuff we teach isn't just about whether you learn the rote material, right? It's about application, it's about dealing with the classroom setting. So teachers will have to keep transforming their role to more exercises and experiences and activities. Grading people becomes something that becomes very important to do in a classroom sort of setting. So we'll see that transformation happen. I just think that, again, the technology solution of, "If everyone has a universal tutor, it's great"—it reminds me of when I was at the Media Lab in the early 2000s doing work with the Media Lab when I was at MIT, and we all thought that the future was going to be one laptop per child, everyone will just educate themselves with technology. It didn't work out that way, right? Wikipedia did not transform education the way we thought it would in the early days of the internet revolution. So I think we do need to be a little deliberate about thinking through what we actually want to teach and how.
So how have you changed your classroom? How are your classes different?
They're 100% different, right? Literally, there is not a single piece of the classroom experience that doesn't have AI included in it. For final projects, we start with pedagogical principles. It turns out reflection is really important in education. If you don't stop and reflect, then you actually don't learn very much. So the final project actually is an AI that interviews you about your experiences and tries to pull out reflections. And that's great because you can't really fake it with another AI, right? It's based on your class experience in that kind of way. AI simulations are a big part of this. They have to actually engage in negotiation with the AI and give me how the simulation worked out, right? They're asked to build AI tools, and of course, they use AI to create and build things themselves that are powerful. So I'm using every dial, but I'm using it based on pedagogy, right? Having to explain something to someone else is really important, so let's build that in. Having to deal with a skeptical audience is important, let's build that in. Having a simulation where you have to put things into practice is important, let's build that in. Reflection is important. So building on the building blocks of good education, you can build some really cool things.
