Why every leader needs this AI strategy | Ethan Mollick | Strange Loop

You can either fire most of your staff and make more money per barrel of ale, or you can be Guinness and hire 100,000 people and expand worldwide. And I really worry about too many people taking the small path and not the big one.

So this is a little bit of stolen technical glory because I was not the coder with Marvin. I was the person from the MBA program who was trying to help the AI people explain what AI was to everybody else. So I worked with Marvin and a few other people at the Media Lab quite a bit on this. And what was really interesting was, this was sort of during one of the various AI winters, right? So no one was paying much attention to AI, and it was all about elaborate schemes for how we can create intelligence. There were projects to observe everything a baby did and maybe that would somehow let us make AI. There was Minsky's Society of Mind with all these complex interlocking pieces. And I think about how ironic it was that the actual solution turned out to be just shove a lot of language into a learning system and you end up with LLMs. It's interesting because a lot of the technical ideas turned out to be incorrect, but there was a lot of the core philosophies there that I think are back in fashion now.

Well, I mean, I think that's what we're all kind of struggling with right now. Now that we have these things in sight, you know, we're back to what is sentient and what are the — I mean, I think it was two weeks ago a new paper came out showing that the actual original Turing test gets passed — the three-party Turing test — that GPT-4.5 is capable of passing it. In fact, 70% of the time people will pick the AI as the human in the room, which I don't know what that means, but it's better than chance. But that's interesting. And so I think we're faced with all these exact issues that a few thinkers were worrying about for a long time. So does this replace humans and what do we use that for? And for augmentation, what does augmentation look like becomes the big question. That debate, I think, never got as far as it could be, partially because this was still kind of fictional. So what do we do with these very intelligent, also very limited machines, and then where do humans fit into the equation? And I don't think that was ever answered, and now it's suddenly very, very important. And the Turing test was a beautiful idea back then, but if we were to design a new test now — a Mollick test — what would be your Mollick test for AGI?

So I struggle with AGI as a concept all the time, right? Which is, it's badly defined. The reason why the Turing test is interesting, just like all the other tests, is they were great when we didn't have anything to test. The Turing test was great when computers obviously failed it. And similarly, we have the issues where the AI is acing all the creativity tests we have, but those were designed and they were always mediocre for humans, and now we're expecting AI to do them. The way we figure out whether someone has empathy in social science — the best test is something called the Reading the Mind in the Eyes test. We show a bunch of eyes and ask people what emotion they have. None of these things were designed for AI. So I think about this a lot, and I tend to be very practically oriented on this. First of all, everyone kind of has their own AGI-ish test. You know, I'm a business school professor. Some of the easiest are: can this agent go out in the world and make money and do things? That's a useful test. Can we discover new knowledge and actually test and come with results? But I mean, I think what we're starting to realize is AGI is going to be this sort of phase we're in rather than a moment in time. There's not going to be fireworks going off. Tyler Cowen just said O3 is AGI, and when asked why, he says it's like pornography — I know when I see it. And so we don't know what the answer to these questions are. And I think it's kind of realizing the meaninglessness of it, because it turns out also, as you guys have learned, if you connect AI to systems in the right way and you connect with company processes, suddenly you have something that's much better than the sum of its parts versus something you're prompting. If you're just doing conversation, that feels very different than can we do strategic decision-making, for example.

I think that is one of the most critical problems we're facing right now, because all of the people in the labs are math and science people, and they view the only good thing you could do with your life as coding. And then add to that the fact that they want to use AI to make better AI, and coding and math becomes the important things, followed by biology because they all want to live forever. So that becomes the angle this goes down, and there are very few benchmarks for other things. We know the AI companies build towards benchmarks — they build sketchy ways of optimizing for benchmarks, but also in more broad ways they use this for testing. So the lack of good business benchmarks is a real problem. I actually, one thing I've been pushing, is companies should be doing this themselves to some extent. Some of this can be direct number-based — like how often does it mess up in being asked to do an accounting process — but some of this is vibes-based, as they say. You actually could have outside experts — and we've done this for some of our experiments — judge the quality of answers: is this as good as a human or not? Have your own Turing tests for various important parts of your job. Is the analysis report good enough? What's the error rate on it? If we use this to give us strategy advice, how good is it? How good is it at a selection decision? And those are questions that are not that hard to measure. They're not that technical, but they do require a little bit of effort.

So the first thing you say is redesign to be AI-native is hard, because it wasn't AI-native. We are in this really interesting spot where we've had basically hundreds of years of organizational development that has paralleled the Industrial Revolution, the Communications Revolution. The first org chart came out in 1855 for the New York and Erie Railroad, and it solved a problem that never existed before — how do we coordinate vast amounts of traffic on train lines in real time using a telegraph? And McKenzie, the guy who came up with this, came up with the org chart as a solution, and we still use them today. 1910s — huge breakthroughs in organizing work: Henry Ford's production lines, time clocks — still use those today. Early 2000s — agile development. All of these things broke because they all depended on there being only one form of intelligence available, which is human. Humans come in humansized packages, can only be deployed with a span of control of five or seven people — the two-pizza problem. And now we're in a world where that isn't the case, so things have to be rebuilt from the ground up. And I worry a little bit that modern Western companies have given up on organizational innovation as something that they do. It used to be that the way Dow Chemical would win or the way IBM would win was they came up with new approaches to sales or new approaches to working with organizations. And now we've outsourced that — so enterprise software companies will tell you how to build your company because Salesforce sells you a product that tells you how to use sales, or a large-scale consulting company will come in and tell you how your organization should run. And now is a time where leaders actually need to innovate. So to return to your core question, it has to be building from the idea that we're heading in a trend line where humans are less necessary in the product, and then you have to pick whether you do augmentation or replacement. And then you have to start building the systems from that.

So I think those are key choices. One of the things I really worry about is when I look at early implementations, what people view this as is an efficiency technology. And I bear a little blame for that — our earliest work focused on productivity gains from AI, and I still focus on that because it matters. But I worry a lot that at the edge of an industrial revolution, or some sort of new revolution, what we're seeing happen right now is that companies are viewing this like a normal technology. So they get a 25% cost savings or efficiency gain in customer service — let's cut 25% of people. I hear that all the time. And there's a whole bunch of dangers with that. One of them is nobody knows how to deploy AI in your organization other than you. You can build tools and techniques that are really useful, but ultimately it has to be people in the company that figure out is this good or bad. They're the ones with the experience, the evidence to do it. If they're terrified of doing that because they'll get fired or punished for using AI or they'll be replaced if there's an efficiency gain, they'll never show you an efficiency gain. And then the second set of problems around that is: if we're really in a world where we're about to see an explosion of performance and productivity, the idea that you should be as small and lean as possible going into that — it's like if you imagine the early Industrial Revolution and you're a local brewer in the early 1800s. You got steam power. You can either fire most of your staff and make more money per barrel of ale, or you can be Guinness and hire 100,000 people and expand worldwide. And I really worry about too many people taking the small path and not the big one.

I mean, it is fascinating how much — like, everyone, the image of AI would be that if you tried to explain the concept of love, it would explode: does not compute. Instead we have these weird systems that are super emotional and have to be convinced to do things, right? Like, we've actually found in prompt engineering sometimes what you have to do is actually just justify to the AI why it should do a step rather than tell it to do something. It's like, no, this is why it's important and you should do it, which is super weird. And so the thing to think about with augmentation, though, is that our jobs are bundles of many different tasks. Nobody would have designed any job the way we have it. As a professor, what am I supposed to do? I'm supposed to be a good teacher and come up with good ideas and be able to have a conversation with you and do research and run an academic department — like, no one would want all of these jobs. And a lot of them are sort of hot AI jobs. I don't mind giving away grading to an AI if that helps. I wouldn't mind providing more counseling support through AI if that helps, even though these are very human kinds of things. So I don't think augmentation necessarily means — just because it does creative, engaging, sort of human knowledge work tasks, at least at the current levels we're at, it's definitely below the expert level in these kind of cases. Whatever you're best at, you're probably better than AI. So the questions become: augmentation level one is just hand off stuff you're less good at as part of your job bundle. And the second level is how do you use it to boost what you're doing right now, and we're starting to get some good evidence for that too.

Yeah, I mean, there's a couple things you said there that are really important. One of them is actually the more minor point, which is the idea that it gives me 10 papers to pick from. This idea of, you know, it's a hot word now, but abundance — we're not used to a situation where you can just get a lot of something and curate. So one of the things that actually matters a lot is taste and curation: that I want to be able to pick out of a subset of options and that still matters a lot. That kind of taste piece, or what to pursue — and it starts to look like management, which is not the end of the world. Management is what most of us aspire to anyway, or at least a lot of people aspire to. And it starts to be giving your direction and taste where it goes. But I think at the end of this, we don't know how good these systems get, and that ultimately every question becomes downstream of how good you think AI gets. Right? If it's good enough that it does all of our work at the high level of what the work you do and your organization does, and the work that I do as a professor, then you know, we're sort of in uncharted territory overall, and I don't know what the answer is. I think that real organizations work in much more complex ways than we think about. They're not always aimed for efficiency, and AI remains very jagged in its frontier of capability. So it can't quite do the whole paper because parts of it'll fail. But if I have experience, I'll know where those fail and can intervene and shape in those places, just like I would with a PhD student. So I think we're going to be in a longer world of limited autonomy than people think, where direction and guidance is still going to be important.

I mean, I think we're already in a place where narrow agents are very good. The best example of those are the deep research agents that now have been rolled out by Google, OpenAI, and xAI, right? And Perplexity as well. They're all very good and they do the narrow task of finding information, giving you answers, very well. And that is a highly remunerated task. They're not quite there yet because they don't have access to the kind of private data that people need to be able to use these systems fully, but they're starting to get very good at legal research, accounting, and market research, and finance research. So I think there'll be that kind of delegation to a fairly complex task to narrow agents that feels very doable. I think there are clever ways to do generalized agents with other agents watching them that no one's really pushing yet. We're so new into this that you kind of have to make two bets. One is the whole view of the frontier — when I came up with the idea of jagged frontier, it's like the frontier is constantly pushing out, so it's jagged. Some of that jaggedness will stick around for a while, some it doesn't matter if it's still bad at it because as the AIs get better overall, it still beats humans. So I think part of this is the question: do you wait for the frontier to move out and then solve the problems, or do you build around them today? And I think part of the key is doing both. But if you invest too much on trying to solve the jaggedness today, as long as models keep getting better, you end up stuck with a legacy system built around a jagged frontier that no longer exists.

So I tend to say you need three things to make AI work in an organization: you need leadership, lab, and crowd. We can talk more about leadership later, but that's the idea that the organization needs to start grappling with the questions at the CEO level, C-suite level, of the kinds of things you've been talking about here. What does our organization do? How do we want it to look? What experiments do we want to do in organizational form? Those are fundamental questions. And by the way, if those aren't answered, then the incentives aren't set correctly for people in the organization. Everyone in the company wants to know what the vision — you can't say people work side-by-side with agents without giving people an articulation of what that actually looks like in their day-to-day job. So that has to come from the leadership level. And one of the bottlenecks, by the way, has been that C-level people have not used these systems enough. And you can see where they do, because transformation happens much more quickly. You know, Mary Erdoes at JP Morgan, for example, has been very public about using AI and that's trickled down, and part of why JP Morgan does quite well on AI stuff. So there's this leadership piece, and then there's the crowd that you're talking about — everybody gets access to these tools in some way or another. And then how do you create the incentive so they share what they're doing? Because there's at least like seven or eight reasons why people use AI and don't tell you. Everyone thinks they're a genius. They don't want to seem like a genius right now. They know that efficiency gains get translated into people being fired, they don't want that to happen. They're working a lot less and why would they ever return the extra value to the company itself? They have come up with brilliant ideas that they don't want to share without taking a risk for it. There's lots of reasons people don't share this stuff. So you have to align that organization to do it. And then the issue is this is done through individual prompting, so to turn those into products, to turn those into agents, to test whether they work or not, you need to then extract some of those and start doing some actual real R&D work — which doesn't mean necessarily coding. Tool bases, like the kinds you build, are really important. But it's also just how do we start experimenting? How do we take what was a basic prompt and turn it into an agentic system? How do we benchmark that system? So you need all three of those pieces at the same time.

So I think it's really clear at this point. I mean, there's stuff that I think CSRs still struggle with, and I think that those are in some ways riskier things — external-facing, human replacement. The augmentation angle, the results are really clear. Individuals working with AI, and especially if you have ways of people sharing that information, ideation — it's absolutely useful to have you generate better ideas working with AI. There's some methods that work better than others, but that kind of approach for supplementing work of all kinds — translation, not just translation up and down levels of abstraction, not just translation, directly summarization. But where you start to see the really interesting stuff is trying to accelerate cycles. So I'm seeing a lot more of rapid prototyping and development — so going from like, let's take an idea, then let's have the AI generate 25 ideas, let's have it create a rubric and test those ideas, then let's put simulated people through those ideas and get their reactions to it, refine the ideas further, then let's go and create a working prototype and interview people about how to make it better, and then build a vibe-coded first version that is literally 25 minutes of work at this point, with just a command line in O3. So we're in a very weird spot where it's like — but then the organization ends up tripping that up, right? Because what do you do with the fact that now we have 45 great prototypes? Where's the manufacturing capability to build it? So that augmentation piece is pretty good at the beginning. And then research agents are looking really interesting. And then knowledge management agents also seem to have a lot of value — actually this is something you forgot or thought about. Where I'm starting to see really interesting stuff happen is advisory — the idea that we're going to give you advice that's timely or unfiltered is also really interesting.

Both, right? Systems take a lot longer to change. I mean, we've been talking to some of the DeepMind people and they are saying there's getting real drug development results in a year that look really good. So there'll be pressure to adapt to those kind of things. And I think part of the question — part of the issue with the uncertainty in the regulatory environment, whether for different reasons in Europe versus the US, for example, is that it makes it hard to figure out where to invest to make these kind of changes happen. Because we are going — there's going to be societal bottlenecks all over the place. And there's also, the AI only has limited ability to act in the physical world at this point. Robotics lags this. Organizational structure lags this. So how do we start thinking about that becomes a really big deal. I think part of why people find agents so appealing is in part the idea that they solve some of this problem by just doing stuff, so I don't have to worry about it. But at some point they're going to hit the real world, right? And at those friction points, that is where things slow down. On the other hand, if you can get up to that friction point and deliver, here's seven really good-looking compounds that might make a difference, that is a huge gain anyway. So I think that the gain will be more spread out, but we just don't know. I mean, part of this also is how autonomous these systems get.

Oh, that's a tough one, and based a lot on organizational choice. But I think management roles — roles that are sort of thinking about systems — tend to be very valuable because systems are problematic. I think experts anywhere become valuable. It turns out expertise actually is really good. None of these systems are as good as an actual expert at the top of their fields. We tend to measure against the average in a field and the AI does really well, but if you're in the top 2% of something, you're going to be beating the AI in that field. And so expertise actually matters a lot in this space. So either deep subject matter expertise, broad expertise across many areas as a system leader, or really good taste tend to be the three things that help you.

So there's actually a few effects happening at once, and I think it's worth unpacking them. Our Boston Consulting Group study was the first one to document in the real world the idea that there was this performance gain — the lower performers got the highest gain. But people don't talk as much about the why we found out that happened, which is we measured something called retainment: how much of the AI's answers the consultants only ultimately turned as their own. And for sort of 80% of consulting tasks, the only way to screw up was to add your own thoughts or ideas into the AI's answer. As long as you were just turning in the AI's answer, you did great. As soon as you're adding your own thoughts or ideas — so it's basically work at the eighth percentile. So when you say you're hiring a junior developer and the AI makes them better, I think it's worth specifying: is it just that the human is substituting for the things we can't do agentically yet — which is like, I'll paste in the requirement and I'll attend the meeting and the AI is actually doing the work — or is it actually bringing people up to that level? And at the same time, at this sort of really good person level, we're seeing effects where if you're very good and you use AI the right way, you can get 10 or 100 times performance improvement. So I think you need to think about both things. There is this sort of substitution effect, and my view has been that a lot of the benefit comes from having expertise and then using AI to supplement the areas that you're not — you're bad at. Like, I think about founders all the time. I was an entrepreneur, I teach entrepreneurship. Entrepreneurship is all about you being very bad at many things but really, really, really good at one thing. And your whole task as an entrepreneur — and the reason why I teach entrepreneurship — is to have the 95% of stuff you're bad at not trip you up. Like the fact that you didn't know you needed a business plan or that you didn't know how to do a pitch, because your idea is brilliant and you know how to execute it in this market. And so the fact that AI could bring you to 80% in all of that is a really good thing. And that is replacing your work, but in the area where you're at the 99.9th percentile, you get a 100 times multiplier. And I think that's the same kind of angle. And I think the danger is that if you're hiring junior people and expect them to use AI the whole time, how will they ever become senior becomes a real challenge.

No, I'm really worried about that. Because like any other university — I'm at Wharton — I teach really smart people, but I teach them to be generalists. I don't teach them to be — I teach them about how to do analysis, I don't teach them how to be a Goldman Sachs analyst. But then they go to Goldman Sachs or they go to a law firm or whatever it is, and they learn the same way we've been teaching any white-collar knowledge work for 4,000 years, which is apprenticeship. They're asked to do repetitive work over and over again. The repetitive work, doing it over and over again — that's how you learn expertise. You get yelled at by your senior manager — or else treated nicely — but you're basically given correction over and over again till you write a deal memo. But it's not just that you're learning to write a deal memo, it's that you're also learning why this approach didn't work. You're absorbing a whole bunch of stuff from your mentor about what the goal of this is. So apprenticeship, if you have a good mentor, is a thing that happens. We don't spend a lot of time training people for it. We just sort of — it's magic, and some people pick it up and then other people get fired. And they might get fired because they're bad, but they might get fired because they got unlucky and got a bad mentor or didn't learn the right things. That mentorship — that chain — just snapped this summer that's kept going for a few thousand years. Because what happens now is if you're a junior person, you go to a company, you don't want to show people you don't know something — it's because you want a senior job. So you're going to use AI to do everything. So you've turned off your brain because the AI is better than you. And every middle manager has realized that rather than going to an intern who sometimes messes up or cries, you could just have the AI do the work because it's better than an intern. And I really worry about that pipeline being snapped. And the problem is that we've viewed this as an implicit thing — like, there's very little work in law firms to teach you how to be good at teaching someone to be a good lawyer. Instead, you hope that you had a good mentor yourself and you replicate what they did. It's why bankers will often — 120-hour weeks are part of your job. Why? Because that's always been part of your job, and somehow that teaches you something. And so I think we have to move much more formally to: how do we want to teach people expertise and work on it. Ironically, the one place we do this really well is actually in sports, because that's an area where we've learned how to build expertise — practice with a coach. And yeah, we're going to have to do the same kind of thing in other forms of learning as well.

So there's a few things happening, right? One is what should we teach and the other is how should we teach it. I'm more concerned about two than one. I think there's a big thing of like, we need to teach people AI skills. And I think as somebody who's worked with these systems a a lot, you know, there's not that — the skills are, first of all, there's like five classes worth of skills to learn, right? Unless you want to build an LLM, which you shouldn't do. It's really like five or six skills classes, and then there's a lot of experience. And so I think it's less about teaching people to use AI. And in fact, I think a lot of the discipline stuff that we teach is really important. We want people to still learn to be good writers. We want that broad knowledge, as well as deep knowledge. I think universities are well-suited to that. Where we break down is how we teach. Right? And so everybody's cheating, right? And AI detectors don't work. And they were already cheating, by the way, but now everyone's really cheating. There's a great study that shows that from the beginning of the internet era and social media really kicked in, in like 2007 or 2006, students at Rutgers who did their homework almost all of them did better on tests. And by the time you reach 2020, almost none of them — like 20% — were getting better on tests, because everyone else was just cheating. So you have to do the kind of hard work. So AI doesn't let us skip the hard work, but it will let us — with AI tutors on a one-to-one basis, you can actually teach people at their level. We can help accelerate the learning process in real ways. And so I'm much more interested in how you teach. And I already did this with my classes — how do you transform how we teach with AI becomes a really interesting question. I don't know if the subject matter changes. And I think we can increase scale, also teach more people, but I think some of the core subjects stay the same.

Everything — my classes are 100% AI-based. I mean, so I teach entrepreneurship, so the easiest version is: it used to be at the end of a class — you know, people have raised hundreds of millions of dollars from my class and the ones taught by my colleagues, the same class number — but you would basically have a business plan and a PowerPoint. Now at the end of a week, I have people have working products. Like literally, when I first introduced ChatGPT to my entrepreneurship class, the Tuesday after it came out, one student was really distracted, came to me afterwards and said, I just built our entire product while we were talking. And that seemed entirely novel at the time — that it would write code was shocking. And now we're in a very different world for where that is. But I have — my students now have AI simulations they play, they have to teach the AI something, we have a purposefully naive AI student, there's AI mentors for all the class material, they have to build cases with AI, there's AI watching what they do in team settings and giving feedback or acting as devil's advocate. So there's lots of cool stuff you can do to supplement it. But that's all in service of having a classroom experience that's active and engaged. And so I think that classrooms don't go away, but what we do in them kind of transforms.