Can AI Translate to Real Productivity Gains — With Ethan Mollick

Thanks for having me.

I mean, I think anyone who's in an organization probably has the right answer to that problem, which is you don't get organizational productivity gains instantaneously from individual productivity gains. You can produce a hundred times more code than you did before, and that's great if you're a solo entrepreneur producing a lot of code. If you're in an organization that's doing two-week-long sprints and has to coordinate with marketing, and has stand-up meetings every day, which is a rational way to organize work, that hundred times productivity gain can actually be a barrier, right? No one has time to deal with all of that code. And we see that, too, in shipping stats. Twenty times more code being produced, thirty percent more products being shipped, right? And so, it's not an uncommon problem. We haven't adapted organizations to AI yet.

Well, actually, those are two different things. So, one thing is the flood of stuff, right? And we're seeing that everywhere. So, there's a lot more novels, a lot more games, a lot more apps. But I'm actually talking directly, there's a nice set of studies looking at product shipped, right? And we're seeing huge increase in the amount of coding, smaller increase in the number of product shipped. There's still an increase, but it's less, right? By an order of magnitude. And the reason for that is, again, we don't have multiplayer AI. AI is a very single-player experience right now, where you get individual performance, but organizations are about many people performing at once.

Right. I mean, in the book I talk about leadership, lab, and crowd being three things you need aligned. You need a leadership team that actually buys into this and starts making organizational change. You need a crowd. You need people who are using and inventing ideas and that's often the line employees are talking about. And then you need a lab. You need to actually be doing research on how to take this stuff and spread it out among everyone. I think the only thing I'd push back on a little bit there is we still have the multiplayer problem. Like, if you're a hundred times more productive coder, we still need to figure out how do we give you the roadmap to follow in. We still need the marketers to work with you to figure out how you get the material shipped out. We still need to figure out how to maintain the software. So, it's a little tough to just say if, you know, here's productivity gains alone from individual players even if they're given the freedom to explore, we still need to turn that into process.

Well, I mean we're talking about top-level productivity gains in the US economy. I talk to lots of CEOs of very large companies. There is almost nobody at this point, and the data from my colleagues back this up, who isn't seeing some ROI from AI. Like, this is not an issue anymore that people aren't seeing ROI. What they are trying to do is figure out how to move from individual productivity gains and, you know, $20 here, $30 here, which sounds like a lot to us but it's not a lot to a large company, to how do we transform our organization around this, right? So I don't think it's a question anymore of are you getting gains? I don't hear that from C-level people. I did a year and a half ago or a year ago. That's no longer the question. The question is what do we do now, right? So we have, you know, we have 100,000 employees. Like, I think organizational change doesn't happen that quickly. I mean ChatGPT came out in 2022, right? GPT-3.5, and like companies just started adopting this in the last couple years. There's no playbook, right? Nobody knows what to do in this case. All the traditional methods of like the consultancies that would help you, the enterprise software companies that would build this in. I mean it is a revolutionary change. So I think that organizations are absolutely seeing the ROI. I haven't heard of a single organization saying that AI isn't paying off at a large scale right now, but they're not getting the transformation yet, but that requires management to make decisions about what's going on and they've barely begun to do that. Many of them aren't even still using advanced systems.

I'm going to disagree with you again on this. Like, there is... So, first of all, we keep whipsawing between different views, right? Like, token maxing was big this spring, and then it was like, 'Oh, no, everyone's spending too much on tokens.' It is becoming a fast-growing expense. Companies are worried about this. When I talk to a lot of Fortune 50 companies, though, they are viewing this as something that is an R&D expense to try and get under control, to figure out what's going on. A lot of them don't know in the same organizational change... Like, I was talking to a very large company the other day, and they're saying we have some people who are 100 times more productive, and some people aren't really getting any benefit out of AI. We don't know yet how to spread the 100 times more productivity to more people. So, I think that there's news articles about people pulling out. Now, most of the people who are talking very publicly about how they're cutting AI spend tend to be people with competing AI products. So, like, that Microsoft, Meta, and Palantir are telling you, 'We're spending too much money on Claude.' I don't know if that tells you that much about the world, right? Revenues are still growing for these companies. Absolutely, nobody has cost under control, use under control, but I don't see any sign of people saying they're pulling back because AI turned out to be useless.

Companies are getting large returns that we're talking to, but they're also in experimentation and evolution phase, right? You can't get returns without making changes. Most organizations I know know that. Making changes takes time, right? And they need to figure out how to swallow this new technology. So, I think there's too much of a sort of back and forth like, 'AI's changing everything, AI's changing nothing.' When I look inside organizations, right? C-level people are deeply, you know, spending a lot of money and a lot of time trying to figure out how to make things operate well. I'm seeing transformation in lots of organizations, including really old-school biopharma companies and financial service companies, very large ones, who don't want to talk about exactly what they're doing yet cuz it's a source of competitive advantage. And I think people are worried about token costs, but that's often a management problem. Like if you don't make any decisions about how tokens should be used, right? Who should be using them? What process should be using them? Then you would incentivize people to use as many tokens as possible, they'll use as many tokens as possible. I feel like we're kind of in this chicken and egg where we're like, you know, where is the return? I keep telling you it's there because companies keep telling me it's there and surveys show 85% of companies have positive ROI from AI as of last year. So, I think though the major transformation hasn't happened yet, except it's starting to in some organizations. But again, you have 100,000 employees, how are you supposed to even think about mapping out all the processes in your company? How do you start deciding about who gets to use tokens under what circumstances? How do we start to adopt the gen-tech models when they're basically less than a year old? There's a lot of stuff happening all at once.

So, in terms of productivity and cost savings are already there. I've actually been worried that companies are pushing too much on productivity as the main goal. So, my colleagues at Workday have been surveying 1,000 large companies and like I said, last November, a year ago, 85% of large companies with over a billion dollars in revenue reported positive ROI from their AI use, right? We don't have detailed breakdowns on every piece of whether it's cost savings or spend, right?

So a lot of them I can't even talk about necessarily. I mean, so I talked to a, you know, there's another Fortune 100 company that gave its senior leadership team access to basically OpenClaude with access to the full company data and completely changed how management works. Like C-level teams are using this as a super analyst to find problems inside the company they weren't able to identify. I have spoken to a large financial services company that has launched a whole bunch of new product lines that they never had before as a result of AI. Talked to a couple of very large hedge fund and PE funds that are seeing huge returns to management technology right now and all the companies are growing faster inside the portfolio. Like lots and lots of cases like this, right? Often times they don't want to talk about it in huge detail and they're all still in the experimental phase. I mean a lot of these are outside of pilots now, they are rolling out at the organizational level. But it's tough because it's a complicated set of choices at the company level rather than a product you adopt and get an instant return from.

So two things, that is a famously bad study.

So that's the MIT Nanda study from last year and at the time it was somebody went out and just interviewed people at a conference and coded that himself.

I would hope that that's the case that most AI pilots fail. Again, the one thing I'll keep emphasizing again is I, you know, I don't take money from the AI labs. I don't really care whether they succeed or fail individually. The transformation that's happening in companies is pretty inevitable when I talk to organizations at this point and I think they have to be doing experiments that fail because if they don't do that how are they going to learn this? There's no playbook. So I'm hoping for high failure rates otherwise you're not engaging in experimentation.

So, there's no one move, right? It's about thinking about structural change, which takes years for a lot of organizations to do. And so, what I'm seeing, examples of things that I'm seeing on a sort of structural level, very common situation is pulling apart organization teams that existed. So, called disbursement, you pull out your best senior engineer, put them with subject matter expert and a marketing and strategy person, and then have them do like impossible projects. Like I just spoke to a company, they went from a $200 million data migration plan for their old mainframe data to they renegotiated to a $20 million plan just a few months ago, and the CTO is thinking like, well, maybe I can do this for $2 million in a couple months. Like real noticeable change, but it involves literally pulling apart teams and thinking about what do we want to do and resetting projects from the beginning.

So, because individuals are now starting... One of the other things we're seeing in companies is a dissolving of company boundaries, right? So, coders are doing design work, designers are doing coding work, product managers are doing everything. So, part of what's going on is how do you reassign roles that way? Some companies are assigning everyone the role of builder, right? And that's what they're doing with AI. But other organizations I see are trying to think about how do we take teams of highly talented people, these people already getting returns from AI, put them together, give them really hard challenges to do, high token budgets, and seeing how far they get. And if they fail, which happens some of the time, then how do we learn something from this formally rather than just burning tokens and setting them off into the air?

Yes. I mean, most organizations are not doing far exploration work. And especially in organizational design and form. So, 40% of the advantage of American companies in the 20th century came from better management according to a paper by Professor Sudhir and other people from Harvard. And we used to do a lot of exploration of organizational form, right? Companies were experimenting all the time with how to organize themselves, how to get work done. I mean, a large part of the story of the 20th century was the story of American manufacturing and firm organization transforming. That is not happening as much the last 30 years. People use outside vendors like, you know, that SaaS vendors will tell you how to run part of your business, use consultants, and your R&D is focused on your customers or outcome. So, a lot of organizations have very little muscle memory left of organizational change. So, it is actually pretty rare to see organizations play with fundamentally re-dividing how work operates, how roles operate, what mandate people have to get work done. I mean, that doesn't feel like normal operation of business to me.

No, it's fine. There's nothing wrong. I am... No, I'm excited. Like, I mean, and I should point out we don't have all the answers, right? I mean, but also 'we don't have all the answers' sounds like an academic result. 'Nobody knows what's going on' is a better way to phrase this.

I think the thing that I'm reacting to a little bit is this sort of... I don't think skepticism about AI's ability to create transformation is warranted at this point. I think that's a dead end. I think anxiety about how we shape this in the right direction is warranted and I think how do companies start to think about this in a way that helps large companies start to transform is a very reasonable thing to worry about and why we haven't seen those results. I spent a bunch of time in the book talking about why haven't we seen transformation happen. So, I'm very much on your page and I'm not going to defend that trend like, 'Oh, no, AI has changed everything today.' Part of the interesting puzzle of the book and why we're in the phase of coexistence, right, and stuff like that is we are trying to negotiate that right now. What does the world look like so we get productivity gains and good results for society.

I mean, services are the heartbeat of the economy right now. But if you want to talk manufacturing, I mean, I literally talked to some of the companies in that exact range that you're talking to, right? And again, I can't name names not cuz I'm trying to be cagey, but because I don't know if we should talk about it or not. But I, you know, I mean, there's an endorsement in the back of my book from the just stepped down CEO of Walmart to give you a sense of the kinds of companies I talk to, which I think most people would consider real economy stuff.

And first of all, there is a huge amount of change in these organizations. Some of this, by the way, is R&D work, right? Like, literally I know the equivalent of the better kind of Frito-Lay chip, but that is an expensive process inside companies that make new products. And speeding up new product development is actually a big deal to them in a big way, right? And finding new trends. I've been talking to large-scale manufacturing and agribusinesses that are thinking about things like how do we use the fact that AI can now look at cameras to figure out new things about which fields are in trouble and which ones aren't. We manage construction projects the same way we have since the 1940s. What does it mean when you have an AI that can give people advice about what to do differently and can actually give information about what's happening on the site or not? You have really interesting sets of changes happening everywhere, right? On the hotel side, like or even in the Walmart store, they've been pretty open about this set of stuff. How do you give advice to everybody, to a senior store manager who needs help figuring out what to do at the store when there's trouble? Like management is a real thing here and there's pretty transformational things. That's aside from AI for planning and AI for central management and AI for HR.

So, I mean, I actually think, you know, both services are the real economy, right? And it's a big deal of coding, you know, I think one out of every 16 jobs is management in the US, you know, and coding follows soon afterwards. Like these are big jobs that represent a large part of the economy, but also I think the whole point of this is general purpose technology with widespread effects. I mean, take even the most narrow thing, right? I was already talking about migrating databases over at large financial services companies and mid-size banks that have COBOL still running internally. These have literally been the projects that have been the backbone of a large part of, you know, IT for a long time, those are being transformed overnight. So, I don't think it's like... I think you see this everywhere. This is not just like coders and scientists and, you know, financial professionals.

Yeah, I mean, and that's sort of almost... yep, exactly. So, that's one of many outcomes Walmart has had from being pretty enthusiastic about AI adoption. And they've only been doing it for a couple years, right? And it's, you know, a few million employees. Like, it's not an easy thing to transform an organization that quickly. So, my flip side to you is, on one hand, is it slow? Yes. Is this faster than digitalization, cloud, e-commerce, any of the other things that have transformed American business the last 20, 30 years? Certainly.

I kind of feel like you're alternating back and forth between like, let's focus on the narrow use cases like what happens at Frito-Lay. And I'm like, let's not talk about biopharma. And then let's like, okay, let's talk about biopharma. Why does Frito-Lay matter? The whole point of this is widespread effects, right? So, first of all, I don't think human flourishing is a given outcome from this and not just because of existential risk, but also because there is automation pressure in all these organizations. Like, we haven't talked about like the easy way to use AI is automate your call center, automate your management, automate... like that's the danger is that we just automate everything rather than augmenting human work. So, I actually think that for me, the concern partially is that there's danger that we don't pursue human flourishing as an outcome without even any of the big AI pictures. But also, there is, you know, the way you get human flourishing, right? Is we free people from drudgery, we increase the rate of scientific advancement, we increase the rate at which everyday people get access to better education, better health care, better financial advice. The pieces are there to make that happen. We haven't necessarily seen the steps needed to make it all occur for everybody yet.

Yeah, I mean, I call this the overhang, right? And there's a huge gap between current capabilities and future capabilities. We don't need better models for complete transformation of a lot of how we work in the next 10 years, right? We need that adoption curve we started talking about at the beginning of how we get up that curve. And so in my mind, right? Like that means that we are lacking the imagination for the transformation piece, which is the point I keep making to you is like it's not... there's no playbook yet. It's not an automated process. We need to figure this stuff out. But the piece is already there, right? And it's sort of inevitable at this point the transformation will happen. So it's about guiding in a good direction.

So, I think there's three different things going on. The first is that we are in what we call in the study of innovation an era of ferment. Nobody knows the right form for these things to be in. So, you're going to see all sorts of ideas churning up and by the way being copied, right? So, OpenClaw came out in January, introduced the idea of personal agents, and what's happened is suddenly that has taken root as a form. So, now we have Muse and Dot and other things that are basically claw-likes, right? So, we're seeing an era of ferment. That's the first thing. The second thing is I think some of this is category differences, right? There's a difference in category between Vibe Coding and Agents. Vibe Coding is a human activity. Agents are a capability of AI systems. And they're not like competing product lines, right? There's a lot of debate over how people use AI. I don't think Vibe Coding in the companies I've talked to was ever viewed as the way they're going to be using AI systems. It was, you know, it's a... And by the way, Vibe Coding has taken off, right? I vibe code a lot of things that would have required me to have a coder involved in this before. People are building games, but people are also building very serious projects all the time. So, I think it hasn't gone away. It's just there's many things happening at once. And then the last thing I would say, and the most important, is part of what you're tracing is also the capability curve, right? My last book Co-Intelligence was really about the chatbot era because that's as good as models were. This book is about the agentic era in a lot of ways because that's where models are today. Practical agents were not possible until we reached Opus 4.5 level about November, December of last year. Like that was when agents became practical. And so, what you're partially seeing is views as we move up this exponential ability curve, it changes the kinds of products that are possible. So, all that stuff is happening at once. We're trying to figure out which products exist in a world where nobody knows what's happening. You are seeing multiple competing different groups all trying to claim the idea of AI products, and you're actually seeing real capability gain.

If progress stops today and we could continue... I mean, I think we're back to that exact same beginning part, which is the capability overhang is quite large. So, what's going to happen over the next five or ten years, we will fill in the capability overhang. Organizations will transform. The remaining technologies that need to make this matter will make a difference. We will figure out how to use this in ways that the jaggedness of AI and the jaggedness of people coexist together. And there'll be large-scale socioeconomic effects from all of that. For productivity gains, I don't think we're going to see some of the insane numbers predicted of 10% growth, but, you know, who knows. That seems outside of any expectation I would have. But we will see transformation happen one way or another. We don't need new models, but we're getting new models regardless of that. That's coming.

So, in the next couple of years, I think change will be... It's kind of an interesting thing to have these conversations because any coders who are listening to this, their job is radically different than it was two years ago. Right? Like it's just not the same job at all if they have any access to these tools. The best coders I know are not writing code anymore. Right? Like, and so you're saying like, oh, nothing's changed. It's like everything about their job has changed. I mean, but they're still going to walk down the same streets. They're still going into office buildings.

In what time period? I mean, I think that in...

I mean, so part of what's interesting here, right? Is like, if the technology doesn't develop any further, I am kind of at a point of like I don't know how to prognosticate about this. Like I think people will still be people and we'll be living in cities and walking down streets. Like, you know, you're starting to be like we're now in hypothetical land that I can't really answer, which is like do practical robots exist in this world? Well, there's some good signs or bad signs depending on how you feel about robots that general purpose models are actually quite good at robots. So maybe in the next five years we finally see robots emerge. Suddenly that reshapes the physical world. Otherwise, you know, are you more likely to have cyber attacks in your bank? Yes. Will that change how life on the streets is? I mean, people are people, right? And I think it's easy to overstate whipsaw technical change affecting people's lives. I mean, we went from, you know, if you look back over the last 60 years or so the built environment hasn't changed that much. Right. So I think it's reasonable to expect that unless we get robots doing work for us, you'll still walk down a city street. But I think that's not really the concern. The concern is like what do people do for a living? How, you know, who is making returns from this kind of technology? What's happening inside the office building is not the office buildings themselves.

So my answer to that is that technological revolutions result in disruptive change. And you already see that happening. Coding is not the same job. We are in debate right now in economics about whether or not there'll be more or less coders. We just don't know, right? A lot of these things depend on government policy that happens, how companies adopt this technology over time, and we just don't have the answers to these questions. I can promise disruptive change, right? Like that's the only thing I can do. I cannot predict because we just don't have the data yet about whose job is replaced, whose job is augmented, where the value goes in the value chain here. I mean, I wish I had a crystal ball, first of all, I'd be making investments based on my crystal ball. And second of all, this is not what you do in a period of disruption, right? Like you can't be like... this is literally one of the most impossible things to predict is exactly who wins and who loses under what circumstances. I can make stuff up, but it'd be made up stories.

Yeah, I mean, all right, let's take an example. Here's a good example for you. With Anthropic. So, Dario was at the DealBook conference a couple years ago and he was talking about like if he buys, you know, if he overextends on his compute purchases, then his company could go bankrupt if the demand is not there. It turns out that Anthropic is compute constrained now. He should have bought more. So, that's what I'm talking about. Like the capabilities... There were even... So, another... Here's another example. Ilya Sutskever, former chief scientist of OpenAI, said very clearly a few years ago that the technology was about to hit a wall, data wall. And what happened? Like the capabilities have just exploded ever since. So, there has been a degree of surprise there.

That's correct.

I think reasoning is something that they weren't expecting to work as well as it has. I mean, you take a look at and reasoning has been core to these math breakthroughs, right? So you take a look at the predictions of when people expected AI to be as good at math as it is, 5, 10 years from now. But it's already solving millennium problems.

Yeah, so why do they get it wrong?

Yeah. Where could the next wall be? So, when I talk to people at the labs, they don't think there is a wall. I mean, that's what they think. All technology, all exponentials become S curves eventually. You know, and there's... we hit some sort of limit at some point. I don't have a lot of future insight. You know, like most of the things we thought would happen, model collapse because of lack of training data and like that turned out to be technically solvable, right? Reasoners turned out to be a really good solution to problems. I thought agentic... One of my biggest mistakes, I think, was I thought agents would be harder to implement than they were because I thought you'd have to build complex organizational systems for agents to operate in. It turns out self-organizing swarms work really well. So, like the barriers I've thought I would see, I haven't seen yet. So, I've gotten... you know, I've run out of being able to predict good barriers. I am sure that this S curve has a limit, but like I don't have a technical point anymore where I can point to this and say, 'This is the big problem that's going to hold back AI development.' What about power? So, I mean, there are limits to how much compute we can deploy, right? And limits to the number of chips out there. So, you have to make choices about whether or not you're using the remaining power, like the physical power and amount of inference we have available is limited, right? And that will be a trade-off between R&D efforts that the labs spend on training new models and how much inference they want to sell. But yes, there's obviously going to be a bottleneck in the physical world in inference and compute. Doesn't mean that like I'm not predicting we go to super intelligence overnight, but I'm saying I don't see any stoppage to steady gains in AI ability. Does not mean the same thing as AI becomes super intelligent and wakes up and murders us all, right? There's enough people worried about that that it's worth a little of our time to worry about, but it's important to realize like we can get continual gains. It will be constrained by the amount of computing power we have.

So large language models are next token prediction machines. They predict one token or word or part of a word after another. And as a result of that, they can't really think. They think by writing. So the breakthrough of the reasoners was taking a technique many of us used in prompting anyway called chain of thought where you'd say, 'Hey, I think step-by-step out loud before answering,' which gave it a chance to basically write out an answer that could be incorrect before deciding to produce the actual result. So all that reasoners essentially did, there's some other tweaks on this as well, but reasoners essentially did was let the AI model generate a bunch of internal thinking like a scratch pad before it gave an answer. And that was one of the big breakthroughs that kept AI advancing, right? There have been others behind the scenes. Like I said, there's been a lot of issues with data quality, with post-training, with reinforcement learning that have been similarly big breakthroughs, but reasoners were the most obvious one. And basically it lets AI think out loud before it has to answer questions, and that creates much more high-quality answers, especially in areas like math or science where you have to explore errors before you get to the right answer.