What if the employees secretly saving 70% of their time with AI don't want anyone to know about it and they have good reason to hide? Welcome to the Super Data Science podcast. I'm your host, John Cone. It is not hyperbole to say that today's guest, Ethan Mollick, is one of the most sought-after AI experts in the world. Ethan is an associate professor at the University of Pennsylvania's prestigious Wharton Business School, where he co-directs the Generative AI lab. His book Co-Intelligence is amongst the most popular AI books of all time and nearly 400,000 people subscribe to his One Useful Thing newsletter. Today's episode on enterprise AI is short, yes, but be prepared. It's so information dense you very well may need to slow down your player's playback speed to enjoy it. All right, let's go.
Ethan, Co-Intelligence, your best-selling book from the spring of last year, focuses on collaborating with AI. Has the rapid evolution of agentic frameworks since the book's publication led to any major changes in your thinking? So, for example, if machines are so autonomous, do they eventually get to a point where we're not really collaborating with them at all, or is that autonomy a natural complement?
Well, so there's now a bunch of modes of coordinating with machines that are different. So with Co-Intelligence, when I wrote the book initially, AI made a lot of mistakes. It wasn't autonomous in any way and you had to kind of do all the work yourself. That still is actually a very valuable way of working with machines. But now even that's become a little complicated because there's two ways of doing this. One is interactively — I talk to a chatbot, push back and forth. But with thinking models and agentic models taking more and more time to do work, the models increasingly — I assign a task to the AI. I use my expertise to decide what that task is. I evaluate the results, correct my approach, and if that still doesn't work, I do it on my own. And that is a really good form of co-intelligence now. And then as you're pointing out, we are starting to see some real autonomy from these machines for some tasks. So I think people will increasingly be picking among those sets of modes or just doing things alone. So things I do alone, things I do co-intelligently, things I assign to AI and assign the work, and things that happen autonomously.
That's a great list. Very concisely done. You clearly spent a lot of time thinking about this. You mentioned in your talk at Scale Up AI today your recent Cybernetic Teammate paper involving a field experiment with 776 professionals at Procter & Gamble, and it demonstrates that AI often functions like a teammate as opposed to being a tool. In light of this, how should executives, how should our listeners rethink team structures and management?
We have no answers to that question right now. I think that this is — so part of what worries me a little bit about the state of AI is the capabilities of the systems are quite high and people are not really using them and everyone wants someone else to answer the question for them. So if you look at why American firms in particular did so well throughout the 20th century, 20 to 40% of the extra competitive value of American firms has come from better management and there was a willingness to experiment with management techniques and approaches that generally led to the dominance of American firms in the 20th century. I feel like that's something a lot of companies have given up on. They either go to outside consultants to do the work or trust SaaS vendors to tell them how to run their company. And I think that companies that start to experiment are going to be in the best possible shape. So we're already seeing some versions of breaking down traditional barriers, like let's pull senior IT people out of the IT department and have them sit next to a subject matter expert and vibe code applications together starts to really change how things operate.
Yeah, this ties actually into something — it ties into some of your contrarian stances. You touched on it a little bit there in your last response. For example, you've argued against external consultant AI implementations as well as other popular enterprise AI approaches like retrieval augmented generation, RAG. Given your contrarian stance, you might have, you know, unique ideas about where the best opportunities lie for AI in enterprises today.
So first, my qualifier — I actually think there are rooms for consultants and room for RAG. Consultants especially can help organizations make the change they want to make, map processes, things like that. But I think that the change does have to come from within to some extent. And the reason why I've been sort of pushing against RAG solutions — not RAG as a concept — but because the first and easiest thing to build is a 'talk to our documents' chatbot, right? And I don't think — I think there's space for this, but the thing is that if you built a 'talk to our documents' chatbot when I was warning you not to do that a year and a half ago, you now have a mediocre 'talk to our documents' chatbot that cost you a lot of money to make and is now easily beaten by an off-the-shelf model, right? Similarly, if you had a large consultant-built application, things have shifted very quickly. So I think that the goal is not so much that there's one bad application, but it's that you need to be at least doing some really ambitious work and it's important to do that work ambitiously internally and not just look at vendors. Vendors play a role, but you need to have some internal effort going on as well.
It sounds like based on your answer to the most recent question as well as the one before it that a key idea here is that enterprises, organizations need to be looking at the opportunities internally, figuring out how their IT specialists or AI specialists can be embedded with the people that will be impacted by the work. That sounds like a key part.
Sort of. I think you're putting too much emphasis on it. So I say you need three things in your organization: leadership, lab, and crowd. So leadership are the C-level people in the organization who are actually deciding what incentives will encourage people to use AI, giving a vision of what AI is going to do, deciding what processes need to be altered or changed as a result. They hire consultants to help them — more power to them — but they need to be making those decisions. The crowd is everyone in your organization using these tools. You want access to advanced chatbots because it's easy for people to figure out use cases in their own area of expertise. It's very expensive as an outside person to figure that out for you. So you need them experimenting. And then what do they do when they come with a result that works? That's when they go to the lab. So the lab is your group of people inside your organization who are not all from IT. In fact, most of them are — there'll be a couple percent of people in your company who are just really good at AI. Bring those into the lab. They're going to be doing three things at the lab. They'll be turning around those ideas right away. Someone comes with a prompt that works well — let's test it, refine it, and ship it out to the world. They will be thinking about benchmarking, and they'll be thinking about whether you buy or build advanced products.
Yeah. This buy versus build scenario — you've talked about how if you have, you know, these big expensive external consultant engagements, then you're very likely a year later to have spent that money poorly because someone else comes along, a frontier lab comes up with a model that can do a lot of what that external consultancy had built anyway. So how do you advise on how we should be making our buy versus build considerations?
So I think it's not just buy and build. There's also rent as an option, which is buy temporarily. I think you need to be thinking about how core something is to your business, how much extra expertise you don't have internally. So you need a really good coach or mentor — you don't have a mentoring program that's very good, that might be a good thing to use externally. Then there's core functions of your business — CRM, other sets of stuff — where you might want to use AI capabilities from vendors to do that. But I would ask my vendors to be much more transparent than they've been in the past, because your vendors generally don't have their own AI models. They are absolutely using someone else's models. What models are they using? What prompts are they using? Prompts, if you don't have an ability to audit them, can be easily manipulated in a way that makes results seem good that aren't. So you need a lot more transparency in the buy decision than you used to have.
