Hi everybody. Nicola Tangan from the Norwegian Sovereign Wealth Fund, and today I am here with Ethan Mollick, one of my favorite professors, a professor at Wharton, who was out not long ago with a book called Co-Intelligence: Living and Working with AI. And actually you can see it behind Ethan there, down to the right. If you haven't got it, run and buy it. Ethan, if you were a chief AI officer in a company for the next three months, what kind of top actions would you take straight away?
So I think that the most important thing is to get people actually aware of where the state of the art on AI is. I talk to companies all the time, and I think that a lot of executive-level people may have tried AI a while ago or didn't use it personally and don't realize how potentially transformative it is. And I think that I have a sort of general idea that you need to involve your team leadership, you need to set up a lab that's doing research, and you need to think about how to roll this out to the crowd, to everybody in the organization.
So you've got to kind of bring the whole company with you, which is not always an easy thing to do. And how do you maximize the organization?
So I think that's a really good question, and I think we're still figuring out the answers. But it's kind of like any other thing you want to do. You have to think about incentives, and you have to think about leadership, right? So why are people incentivized to use this? Now, the thing that makes AI interesting is everybody's already using it. So there's a new study that just came out that showed in a representative sample of American workers, the usage went from 30% in February to 40% of people are using AI at work, a little over 40% as of April. So it's used everywhere. The thing is, people just aren't showing you they're using it because they're hiding it, because they're not incentivized to show you. They're worried they'll be fired if they use it, either because they'll be told that they're using it improperly, or because people realize they're not necessary, or there's less workers needed. And so there's incentive problems, and then there's also for everyone else who isn't using it, there's role modeling problems. What do I do to get started? How do I use this? Why is it important? So you've got to both deal with the skeptics and the people already adopting it who may not be showing you they're using it yet.
Why would people not want to show that they use it? I mean, here if people don't use it, I'm like, you know, that's bad, right?
You know, so if you think about the incentives of the average worker who's using AI, first of all, they look like geniuses when they're using AI. And one problem that we see is people don't want to show you they're using it because that makes them look like they're not geniuses. Like the people give the AI credit. The second reason is in many companies it's viewed as a thing that is a cost-cutting measure. So if I could show you that the AI does some of my work, do I get a reward for that, or do you fire employees, right? Do I feel safe to reveal it? And a lot of people are just using it to work less. So you're working 90% less. Am I going to show you I'm using AI so I have to do more work as a result? Am I getting credit for doing this? So there's a lot of incentives to not share, probably a lot more than to share.
Wow. How do you measure AI usage in a company?
So the metric thing is really interesting, right? So what I see a lot of people drive towards, we're in a world of sort of new innovation, right? So it could be a little challenging to have a direct, easy answer to that question of like, what does AI do? How should we be using it immediately? So chief AI officer, you're doing experimentation. That said, I think raw usage actually matters. I mean, the typical KPIs you see are how many people are using our AI app internally. The downside of that is in most companies I talk to, that maxes out at 20 or 30% of the population. And that's because an additional third is probably secretly using AI or doesn't want to use your apps. Your apps aren't good enough. And another third is kind of waiting for instructions. Like they're happy to use it, but they're the opposite of the people who were early adopters. They want clear rules about how I should use it and why. So I think you can measure adoption by use of your app. I think you can measure adoption by other internal measures of how much you're using AI at work. But I think you have to be aware that there's still this secret cyborg problem of people not revealing their AI use.
So how are the best companies going about this?
So I'm seeing some really interesting examples of how this works. I can tell a few of the stories. I can't tell all of them. One example is, you know, radically changing incentives. So I've seen companies that offer $10,000 bonuses at the end of every week to every employee who best uses AI to automate their job, right? And they think they're saving money versus other approaches. I've seen people build this into their hiring process. So before you hire somebody, your team has to try and use AI to do their job, and then you adjust your request for hiring based on that experience. Or before you request money, you need to show how you're using AI to do it. Madna has this really great example that they put together. They use AI for everything, and what they did was build around the process of annual reviews. So basically they built a whole series of GPTs that helped people uncover their own performance improvement needs and what they had done over the year, and talk to the right people about their jobs and things so they could write a really good yearly update about themselves. And they said, 'Well, if you don't use these GPTs, you're probably not going to do as well on your performance reviews, and that will hurt your annual salary.' And everybody ended up using the series of things, which introduced them to AI. So putting these bottlenecks in place where people have to use it, thinking about building into internal processes in a way that encourages positive use rather than negative use, those tend to be really effective methods.
So you need a combination of the stick and the carrot.
I think you do, and I think you also need role modeling, right? A leader who uses AI will make sure AI seems critical. Someone who doesn't use AI and says use it is kind of a problem. I mean, you mentioned the chief AI officer at the beginning. One of the things that worries me a little bit about that is there is no ability to have a chief AI officer really at this point. I mean, generative AI has been around three years. Everybody was on their very first project. So if you hire a chief AI officer, that is often somebody who was actually doing machine learning beforehand, which is great, like AI at large, but not generative AI. But they're not going to have any idea of how to make an organization transform with AI. It has to be at the leadership level. I think it has to be at the C-level in the company.
But does it make sense to have a central unit which kind of disperses the best use cases and make sure that everybody is at it?
So that's the concept of the lab that I was talking about earlier. So you do want a centralized unit for experimentation. But that lab has to be staffed partially by the crowd, by the people who are the best users inside your company. And they're going to be sort of ambidextrous. They're going to be shipping out like, here's a great use case, and also building a full agentic system that doesn't quite work yet that automates the entire job, right? But I think the traditional view of having it produce things, I think that actually is on its way out. Some of the most interesting experiments I'm seeing, companies are dispersing engineers from the IT department out into work with subject matter experts because vibe coding has strong limitations, but it's its absolute best when you have a senior coder working with a subject matter expert. You can do incredible things in a couple of days that used to be whole processes. So I think there's going to be centralization of some AI functions, but it has to disperse other parts of the organization in a more decentralized way.
Yeah, yeah, yeah. We do that. But what about ambassadors and people across the firm being trained regularly?
Yeah, and I think one of the things we've been struggling with a lot as we think about AI use and observing companies has been what training means. So I think one of the things, I know this is an approach that you take that I think is smartest about training, is using an opportunity for contact hours with AI. I think one of the smartest things I said in the book that didn't always seem the smartest thing at the time but has turned out to be quite useful has been that you just need to use AI to get it right. You need to bring it to every work task and you'll figure out what it's good or bad at in 10 plus hours. And people are very resistant, even very smart, very well-meaning, very self-motivated people are often very resistant to using this for reasons they can't fully explain, right? It's a weird technology. It's not that easy to use. It's uncomfortable in some ways to sort of confront. I talk about the need for an existential crisis. So I think that it's hard to see adoption is strange that way, right? So training can kind of get you into adoption, but we don't actually have that much we can teach people about AI. Like a lot of the prompting techniques you learn don't really work that well or aren't that important as the model gets better. So it's out of that experience, and I think you will develop this idea of these champions. Some people will just get it, and it's really important, like you said, to get those people out there. They're the cutting edge. They're the representatives of the lab in the world.
Well, will the people who are top performers without AI be the top performers with AI?
So this is one of the biggest questions we're facing. If you think about it, there's four possibilities for what happens in an AI world on skills. So the first effect that we saw, and we saw this in our study, the Boston Consulting Group study I did with my friends at Harvard and MIT and University of Warwick, where we found big performance gains, and this came out like a year and a half ago, kind of made a big stir, 40% improvement in quality for people who use GPT-4 versus not. Big speed improvements, and a lot of other studies like that have shown a leveling effect. So bottom performers get the biggest boost. When you really look at what's happening, it's actually the AI doing the work of the bottom performers, right? So the AI is pretty good. So it moves everybody up to the 80th percentile. So one option is it boosts the bottom performers. A second option that can exist simultaneously is the idea that top performers get some sort of massive returns. We have a couple of studies show that, but not that many. It's hard to study. There's actually one of the best pieces of evidence for that was actually turned out to be a fraudulent paper from MIT that didn't exist, right? But I think there's a lot of suspicion that top performers using AI can get a huge boost, just harder to measure. So there's a possibility that maybe there's a hundred times return if you're already a good coder. And we'll know more about that in the near future. There's also a possibility that AI lifts everybody up. So everybody's performance goes up by a similar amount. And then there is this sort of other possibility that there's AI whisperers who are just good at AI and they're the ones who get all the returns. So we don't know whether it's concentrating the lower end on the top end, whether it lifts everybody up, or whether there's just sort of magical AI whisperers who are just built to do this. And then agents are coming to replace everybody if you listen to the AI labs.
And what do you think? Do you think that will happen? Well, first of all, just tell us, what exactly does an agent do?
Yeah. So nobody has a good definition of anything in AI. So if you are a leader in a company, you will have vendors selling you things with every possible label, and they'll say it's agentic and everything else because there's no clear definition. But the simplest version, and that sort of overlaps with what the labs think, is imagine any AI system that could be given a goal and can autonomously go and try and accomplish that goal without further human intervention, using its own judgment and tools. That's what an agent does. So an agent, I would say, prepare me for this podcast, and it would do all the research necessary. We can even demo a little agent right now if we wanted to do that, a little bit of agentic work. But it's a tool that goes out and kind of does things in the world. And the idea with an agent is if an agent could go out and do work for us, then I get to skip the whole problem of trying to figure out how to integrate AI in with my workers because the AI will basically be a worker. I'll say, write this code for me, and then deploy it, and then test it, and come back. I think we're further from that than the labs think. But I do think narrow agents are already very possible.
Have you got a good one available?
I mean, I think, yeah, let's pick an agent for fun here. I will actually share my screen, and I'm going to log into an agent while we talk. Okay. So let's look at an early version of an AI agent called Manus. Manus is a small Chinese company. Actually, I don't know how big they are. But they use Claude as their underlying dataset, Anthropic's data. And I just want to show you what an agent would do, right? So in this case, I asked it to come up with 20 ideas for marketing slogans for a new mail or cheese shop, select the best one, build a financial marketing plan, build a website carrying cheeses. And so what you'll see is what the model actually does here is it comes up with a to-do list, and then it starts coming up with a plan. So here it comes with a bunch of slogans and ranks those slogans and then decides on the best one. Then it goes out and actually does market research on the cheese industry. Then it goes ahead and puts together a whole financial plan for us that's many pages long. Then it goes ahead and figures out here's a to-do list that goes through to do this. It does market research. It comes up with a color scheme for the site. And then ultimately what it does is without any further intervention from me, launches a website. And this is 100% created by the AI, right? I can go shop cheeses. There's a little built-in shopping cart functionality. I've got a subscription model I can do that has forms built in. And this is without any further intervention from me at all.
Right. So how big was your prompt? How big was your prompt here on this one?
You saw it. The prompt was literally this one. Right. That's not that complicated. It figured out all my intent and everything else from this. So this is an example of an agent at work, right?
And when will these be widely available?
I mean, this is available right now. And I would also argue that o3, which is the model that everyone could use right now, is basically an agentic model already. So if we just go to ChatGPT o3, and I can just ask it a question saying, you know, let's ask it to do something. Let's say something like, give me 10 ideas for a new on-trend shoe design based on market research, and develop a pricing strategy and pro forma financials for the best one, and show me a photo shoot of it. And I'm not so interested in the prompt being amazing as to show you what I mean by this. So this is o3 is an example of both a reasoning model that goes through this process of thinking. We can talk more about why that's important or interesting if you want to. But what I want to show you about this is it's come up with a plan, right? I'm going to look up latest trends in footwear, and then I'll come with 10 designs. Then I'll pick the best one. Then I'll create a pricing strategy. So it's come with a plan, and what you'll see is that it's going to go through this stepwise. So just on that command, right, you can see it's doing web searches already about to do market research. Now it's choosing a design based on the research that it did. So it's the stepwise agentic approach, right, that the AI takes even for just a simple command. This is kind of how an agent works. So you can see it's doing multiple web searches. It's thinking about what this all means for the financials. It's figuring out, you'll notice also that it made a mistake in doing a search. So it's looking for different things that it did before. That's an agent at work. So this agent could have done these whole posts, right, and pretended to be you and me. So almost right. There's a weird disconnect. Video and audio is a little bit different than other approaches to AI, like large language models themselves, but very increasingly so. Yes, you can now bring a live agent onto a call, and it has a pretty good conversation with you. And live video is getting much better. So if not now, within the next months, that's a pretty plausible thing to be able to do.
Do you think he would do better than you and me?
No. I mean, you know, not just because I want to keep our jobs, right? And I'm not just trying to flatter you because we still have interview left and I want you to make me look good. But aside from those issues, right? I think the thing about AI is that mostly it's not superhuman. So you're a very good podcaster, and I would say that anyway. But if you're at the top of your field, whatever you're top at, you're definitely probably better than AI. And by the way, we got our shoe here. And so I think that it's not as good as us yet. That's the real question is does it get to be and does it get to be as good as us across every field, which we'd call artificial general intelligence, AGI, a machine smarter than a human expert across every field. And that's the biggest question in AI right now.
Absolutely. Absolutely. When will we be there?
So that is a thing I don't know the answer to. If you ask the AI labs, they think two years. You ask AI skeptics, they think 10 years, which is a weird place to be for skeptics to be like, yeah, this is definitely happening. It's just not happening right now.
Yeah. So what aspects of human work do you think AI will complement and what will it replace?
So I think that is a really interesting question. The whole premise of the book is about co-intelligence, the idea that the machine works best with a human. That's the same way, right? And so right now because the AI has what we call a jagged frontier, it's really good at some stuff you'd expect and not good at other stuff. And it's also missing functions, right? Like it's hard to have it kind of connect the world together. If I ask you to prepare for the podcast, I'm going to get only so far with it. It will probably give me great preparatory advice, but then I still need to go on the podcast and have this conversation with you. And it doesn't use my email well yet, so I can't let it interact completely with your team. So there's all these jagged edges that make it hard to use as a universal person replacement. Again, the goal of the AI labs is agents will solve this problem by doing the work for us. I'm a little reluctant to believe that we're going to be there as fast as they think. But I think complement is pretty wide. I think the question, it'll complement us across a wide range of tasks. I think where you're weakest is where it will substitute, but almost at the individual level. So if you're not good at idea generation, the AI is probably better at idea generation than you. If you're good at idea generation, you will definitely get value out of using AI, but you should probably be using your own idea generation as well. If you are terrible at email communication, the AI is probably better at that than you. But that doesn't mean that you don't have a role to play in making sure the email is shaped properly.
Yeah. No, I use it for a lot of my emails just to help because my English isn't so good. So it just really improves my emails a lot. Now, to which extent is it now being used for CFOs trying to cut costs, and to which extent is it just amplifying power and helping us to do things better?
So this is where companies get to make choices. And one of the things I worry about with AI is if the leadership isn't well informed in companies about how they work, they view this as another normal technology in the sense of like this is a cost-cutting measure. So I can increase productivity by 20%, so I can fire 20% of my staff. I think there's two things that worry me about that approach outside of any sort of moral or other kinds of concerns you might have, which is that first of all, no one knows how to use this right. There is no off-the-shelf product that just does things for you with AI yet. They'll come, but you have to figure out how to use it inside your own company. And doing that requires you to actually have experts figure out how it's used, and the experts of your own organization, your HR departments, your R&D. So if you start firing people because AI makes it more efficient, everyone will just stop showing they're using AI, and you're going to be in trouble. So I think there's some danger in making a cost-cutting move right away. That doesn't mean people aren't doing it. The second big danger for making cost-cutting is if you believe we're on the edge of a real revolution in how work gets done, which I do, then the idea that you're going to slim yourself down, if I get 20% performance improvement, I'll cut 20% of people, feels like a really bad solution in a world where everybody else is going to have 20% performance gain overnight. And so I think that organizations that are in growth mode will tend to outperform those who are using this as a cost-cutting technology. But we don't have all the models yet. People are still figuring this out.
