Nobody knows how to build an AI-native company | Ethan Mollick | Strange Loop

Nobody knows anything, right? Like I spend my time talking to the AI labs. I've talked to very famous people. I talk to CEOs all the time and nobody knows anything.

So you can kind of think about three eras of AI broadly. We talked about what I'd call the second era. The first era was machine learning writ large, right? Everybody was doing, you know, data analysis and you had your beautiful data lakes and you had your data scientists and you were doing predictive modeling and all that's still valuable, but that was sort of what AI meant prior to 2022. Then we had our chatty, witty moment. And the last time we talked, we were sort of still in the era that I will grandiosely call after my own book title, co-intelligence, where you'd prompt an AI back and forth to get answers. And I think we have entered in the last few months, especially, what we call the agentic era, where it's less about working back and forth with the AI through a chatbot interface, but more about assigning it to do long-running tasks, starting to think about what it means to have organizations built around AI. And I think that that has been the biggest change.

So I think that there is still a need for people in the loop, and there's lots of really interesting things to talk about, about how we incorporate people into AI. I think we still have to be centered around people, but the idea that every interaction is going to be you making requests, the AI tells you something, you do it in the world, you're like, I don't quite know how to do this. Can you make it easier for me? Maybe give me the code for this. That back and forth that was sort of the driver of, like, let's talk to a chatbot or find this email, do it better, was based on the idea that AI couldn't do anything, right? Like it had to be closely watched and made lots of mistakes. There were lots of errors. It couldn't take action in the world. Now, hallucination rates have dropped. They're not zero, but they've dropped. Models can do multiple hours of work independently. That work is judged by independent people to be as high quality as experts' work often. And that changes how you do things.

So I think there have been a couple of inflection points that you couldn't predict exactly when they happened that have occurred. I think a lot of us knew, who are close to the space, knew the agentic moment was gonna happen. I think it was somewhat of a surprise it happened as quickly as it did and that the impact was as large as it was, right? So it went from AI as toy to, oh my God, Claude Code is here and now we have to change how we do all coding and all work. And I think that has been a relatively rapid change. I've also been somewhat surprised in the last year about how much there was a pivot in sort of early fall, even before that, with companies going from a conversation about is AI worth doing to that being answered and being like, how do we use AI? I also think that just in a general purpose, I think the continued acceleration has been a little surprising. I think that we were all expecting AI to keep getting better but there really has not been a lot of slowdown of exponential growth, which I think is interesting.

So everything is downstream of AI abilities, right? The jaggedness of those abilities, the fact that it's good at some stuff and bad at some stuff, I think it tells us what humans do, what our work should look like, where AI might fail, why it's problematic to integrate in. But as AI keeps getting better, the growing aspects of that frontier are what sort of determine how useful it is. So the fact that we've gone from — you can pick any scale that you want, right? The ability of AI to do long-running computer science tasks has gone up from 20 minutes or a half hour a year ago to four hours, six hours. I mean, that's a fairly large increase and there's no particular reason to expect that this would keep going. But when you talk to the labs, they don't see any barrier in sight to continue development and they seem to be right so far.

So the interesting thing with the jagged frontier — right, for those who don't know, and it was my co-authors and I too, so I don't want to take all the credit — is that it's good at some stuff you'd expect, bad at some stuff you wouldn't expect. But that's changing over time, right. So one of the things you pointed out was that counting the letters in strawberry or the math Olympiad — those were both weak spots of AI that got closed in the last year because of the advent of reasoning models. So reasoning models, starting with the o1 preview, turned out to be really good at these tasks AI were bad at. So we had this sudden leap or change in the shape of the frontier. Most of the time the frontier just expands outwards, right, so it's not massive leaps in one area or another, they just keep getting better. Or you track coding, right, it's been sort of a steady but exponential increase in ability.

But there's still areas that are weak. So famously, at least for me, long-form fiction writing. AI is terrible at long-form fiction writing. It writes in cliches, it's all the same kind of voice. It has trouble with actual plot points for a variety of reasons. It can't plot out things well enough, it's trained to be too nice, so there's nothing bad that ever happens. It doesn't develop characters in interesting ways, lots of issues with that. But then there's the kind of micro frontier, right, things that it's good or bad at that you only know if you know your job really well, because there's no department in OpenAI that's saying, here's, let's map what AI is good or bad at in my job as a supply chain manager at a German auto manufacturing company or as a designer in a podcast production company. Nobody knows any of these answers. So I spend a lot of time on my own kind of micro frontier. I found that AI models — one thing I've been tracing is: how good are they at writing academic papers? The latest models, I can actually point them in a directory full of messy files I've gathered over the last decade and it could write a pretty good PhD-level thesis about that topic. Right, it still doesn't have a lot of good imagination about what a good topic would be, but it executes really well on it. So there's still an imagination piece kind of missing, but that's getting better too. So the shape of the frontier is evolving but I think people have to pay attention about their own frontier and sort of the global one.

So I think that will be a very big question for a lot of people, right, and I think that we're gonna see two forces at play. One is a push towards artisanal stuff, right, but you don't want everything artisanal. I don't want artisanal code, right, like, I'd like the code written so it actually executes the way I want it to. But I think you're gonna see pushes for authenticity. But also, if you like reading the piece, you like reading the piece, right. So part of what is interesting about AI writing, just in a very specific way, is that it often depends on you putting the effort in as a reader. So GPT 5.5, especially the GPT series, has been famous for these very stretched metaphors, right, like, you know, like our conversation is like a gap-toothed smile and that doesn't have any meaning, but we can assign meaning to it and that can feel meaningful to us. So the question is, are we okay with interpreting our own meaning, or are we gonna ask the AI to have a super interpretation of this stuff? Are we gonna want humans to write these things? I don't really know the answer.

I mean, I do it all the time anyway, right, I mean, I can get lots of digital art if I want to get it, and I do pay attention to those kind of things. But I think, again, it's gonna be kind of a mixed bag in some ways, like if you get personalized entertainment that's written for you or a story that you've never seen before, maybe that's good enough, right, because you're not gonna commission artists to do that work. So I think it's gonna be like everything else. So there's always a push and pull, an arc between mechanical reproduction and the human element, and we'll have some sort of tension and compromise.

So it was a way to try and talk about agents and the fact that agents are somewhat inexplicable. Actually, all AI is. There's this whole field of AI interpretability, which is, can we tell what the AI is doing? And you sort of can, but you also can't really. Like you assign a task and it just does the task. Why is it doing it that way? Nobody knows, and I mean this becomes a real problem when you talk about things like how do we market AI? Well, if we don't know how it makes decisions, I can't tell you how to influence its marketing. We actually have some experiments that we've been doing at Wharton where we're finding that almost everything changes the way the AI makes recommendations. Like you put things in different order, or you say this article is written in Reddit versus the New York Times — unpredictable ways you get outcome changes. So not being able to know how the AI thinks — not that we know how humans think that well — makes it hard to know what's gonna happen. So in some ways, you're just assigning things to the AI and then magic happens and we don't have a really good answer for what that magic is. So the co-intelligence going back and forth, the wizard thing, sort of we give the AI the, you know, we make a spell and then we see what comes out.

They're at least somewhat permanent. I mean, they're permanent because even if we had the whole reproducible chain, people wouldn't be using it anyway. Right, like people don't use the reproducible chain, you know, they don't verify as much as they should in a case, but also it's just the nature of LLMs. And in fact, we might be in a temporary place where interpretability is higher, because the chain of thought that AIs use is in plain English or, you know, Chinese, depending which model you use, and so there's at least some ability to get inside to that. There's no reason that has to stay in a human-interpretable form, and so I think that there's an inexplicable mess to the whole process that is gonna be very hard to eliminate.

Yeah, so there's really kind of two branches here. One of them is the big picture, right, recursive self-improvement. The idea is, make AI better and we enter a weirder world very quickly. This takeoff concept is often called, right. So once AI start improving themselves, is there some sort of ever rapidly increasing exponential curve that we can't understand, and I think that we don't really know? Like the co-founder of Anthropic was, just a couple days ago — at least the time we're recording this — posted that he thought that there was a 40 to 60 percent chance that we'd have AI scientists doing AI work and improving AI by 2028. I think this is on everybody's mind and I don't think we know what that world looks like. It can get very weird very quickly. As long as the AI stays jagged, it may not be quite as tremendous as people think, because it's still weak in some areas and better than others.

But then there's the personal aspect, right, of working with these systems and I think to some extent you have to kind of commit the cardinal sin of AI right away, which is anthropomorphization, which is the idea of treating the AI like a person. I mean, first of all, these systems want to act like people, right, that's what they do and they're most effective when you treat them like people and your mental model sort of fits best, right. So it's obscure. I don't really know what it's thinking, but I don't know what most people are thinking. I mean, despite the fact that I study people for a living and, you know, work with lots of different people, we don't really know what's going on in people's brains. We don't really know how they make the decisions they do. And when you work with these systems — a lot like Claude or ChatGPT — you start to learn its foibles, like I know certain things will, like, set Claude off and it'll get really anxious and then we'll restart things — anxious in quotes. So, even though you have to remember these things aren't people, treating them as sort of obscure people with their own sets of foibles and responsibilities and things that worry them and things they're good at can be a really useful model, and when you do that, in some ways, the fact that they're opaque feels like less of a big deal.

So I think it's worth stepping back a little first and saying organizations are machines for taking imperfect, stochastic actors who are misaligned and making them work in the same direction. Right, that's literally what we do with people. Organizations are superhuman ability and machines for taking human input and sort of figuring out how to make it work properly. So it's not like it's a hard problem, like if the AI is wrong some of the time, guess what, people are wrong some of the time. We can build organizations that combine AI and people to check that. We haven't built it yet, but we could. The bigger question you're asking, I think, is an interesting one, which is how does it figure out the context in which to operate? Because when you — I'm a business school professor — and like we know that a huge amount of the context in which people work is never written down anywhere, right, forget documentation, it's all implicit. You understand what people are rewarded for and what they're punished for. In some organizations, right, getting the hardest job is a reward. In some cases it's punishment — tells you a lot about an organization — but no one's going to spell that out anywhere, right? Should you be friends with people at work? Should you not be? All these little choices are important choices in organizations and they're not specified. That said, the AI for work stuff tends to be pretty good at picking out context once it has enough information. So if it has the history of a project, it often does a lot better than if it just has instructions for how to execute the project. Some of this will also be people making decisions, right? We will need to document things we didn't document before. Another set of stuff is just starting from scratch. Like, it's not clear that you want to translate processes that exist in a human organization and just have the AI implement those things ad hoc, right? So I think there's a lot of tension across this whole boundary here. I think people are underestimating how much the agents can step in from context to understand what's going on. Given the jaggedness, though, it's not everything, but a lot of things. But then also, how much we don't want that to be the answer a lot of the time.

Well, let's start with a really extreme example. There is a company called StrongDM. That's a small software security company. And they decided to build this software dark factory. So a dark factory is an idea for robotics that if the entire factory is operated by robots, we don't need lights. So it's a dark factory because there's no humans in it. So they built a software dark factory. What you feed into it is specifications — not even specifications, basically, roadmaps for where you want your product to go. And they have two rules. No human can write code, and no human can look at code. Everything is written by AIs, and there's adversarial AIs that test the code. And those adversarial AIs spontaneously built fake Slack and fake Gmail and fake Salesforce to test these security and authentication models. And then all that comes out is finished code, and you just approve or don't approve shipping. And they've shipped products to customers without a human ever touching any part of the code base. There are other rules. You have to spend $1,000 in tokens per person per day, which is also quite expensive. But that's an example of a really radical version of agent work, right? We take the humans out of the loop entirely. I think that's the wrong way to go, ultimately, because I think you actually want people to be brought into the process, either as sources of variance, right? I don't want Claude's decision-making on everything. I wanna use humans so that I don't just do Claude-based work. Even if Claude's work is good, it's not the work I always wanna see because you wanna make important decisions, you wanna make interesting decisions. But that's an example of a really radical approach. And I think the sort of stepwise thing, which is like, okay, we'll just code more, is probably not gonna be the answer. Maybe not the full weight of the dark factory, but there's somewhere in between that I think a lot of organizations will have to end up.

Yeah, now is the time for experts, right? If you're an expert, you're in great shape because you can look at the output of an AI, know whether it's good or bad, right? Like, it's a great time to be an expert because you can be hyperproductive, right? Finally, you can delegate to machines that will do the work for you. You can assess the results. It's like, it's a reason why I think so many experts descended into AI psychosis when they started using agents the first time. Like, I don't know if you've experienced this yourself, but the first time you use Claude Code, people kind of go away for the weekend and they come back with this insane output that is often very useful, but like, it's like, I figured it all out, right? Because they've had this image in their head. They haven't had people to implement it. So experts have their own value.

How do we create new experts becomes a real problem, right? Because that requires trial and error. It requires mentorship. It requires doing and failing at tasks that get increasingly hard as you move forward. And AI shortcuts all that. But it's not really AI that shortcuts it. It's our choices as people. So I think we were talking before about art, right? And that you might want human art. I think in the same way, we're gonna want human effort and struggle and companies are gonna have to be okay with tolerating a learning process. And they're not built for that. We've been very lucky for the last 4,000 years, which is apprenticeship has worked out well for everybody. As a middle manager, you get somebody who is going to do the work for you that you don't want to do, and they're going to do it pretty cheaply. And you're going to be able to assess whether it's good or bad without spending a lot of money or a lot of effort. And the junior person gets to learn the ropes, because even if your manager is a bad manager, you're still learning what's there, what isn't. You're learning the unwritten rules, which we've talked about before. And you're getting assessed on whether you're doing a good or bad job. And that all broke this last summer, because every junior employee, junior hire, would be much better off using AI. It's faster, and the results are going to be better than doing it themselves. And every middle manager would rather have the AI do the work, because the humans are fallible, and sometimes they complain, or they don't show up for work, and you don't want to deal with that. So that break has already happened. And the question is how we reconstruct it. There's a bunch of ways forward, but they're going to take real effort for parts of an organization that aren't used to doing that. We're used to getting the benefits of learning while doing, and we're going to have to separate those now.

So that's the organizational design question. I mean, increasingly, the decisions leaders make in organizations matter, because you get agency if you are given it, in some ways, in organizations. I mean, you decide not to use AI, but in an organization where all your compatriots are using AI secretly, you're going to fail in that kind of outcome. So we have to incentivize people to be able to exercise this kind of agency and choice in the right kind of way. And that isn't something that happens naturally, because you still have agency, right? You're making choices, but you'd be foolish to make the choice that I'm not going to use AI, and I'm going to try and struggle through this on my own, because that would hurt you. So we have to set up incentives to incentivize that the right kind of way. We can do this in schools, right? Because I can say, I'm going to test you without AI at the end of the semester, and 80% of your grade is this test, or this blue book assignment that we do in the room, or the essay you write, or the role-playing exercise we have. And that will incentivize you to not just use AI for learning, because you're going to need to perform in this test. We can do the same kind of thing in organizations, but we can't pretend that people can just use AI for everything and still learn all the skills they want to learn without any help from the organization overall. The bigger question of agency is, I mean, part of the issue is these systems are seductive, right? So we found this in education, which is that the AI wants to give you the answer to the problem. It's a helpful assistant. And it turns out you don't learn very well when people just give you answers. So adding friction. I think people are worrying too much about removing friction. Adding friction at the right points might be really important.

So I think children aren't that motivated to learn. I mean, you're motivated in the area you intrinsically care about, right? So you're a kid who loves math. You're a kid who loves reading. I mean, that's always been the easy part, right? And people always complain, why can't all learning be like the learning I like? It's like, because not everyone likes everything, and intrinsic motivation kind of fails at some point. So we rely a lot on extrinsic motivation. You have to take the test. We think, as a society, you need to learn civics, and you need to learn math, and you need to learn your country's literature, or whatever the outcomes are. And we enforce that, right? Like, if you don't get good grades, you don't advance. These things matter for getting into college or whatever comes afterwards, or an apprenticeship program. And so we build the incentives we need to do these things. Given that we have control of people in the education system, education is sort of a solvable problem. We have some really good evidence now that AI tutors are good for learning. The same team, the research team out of Penn, actually, that found that AI hurt learning if you just let students use AI and say, use it to learn. Students thought they learned. They didn't learn. But when you build an AI tutor, you actually get very large impact and improvement in outcome. So I mean, the end state looks like something we've already known from pedagogy research to do, which is, outside of class, you'll have AI tutors. And I don't know about classes in Europe, but in the US, increasingly, students are assigned this kind of work, where they're asked to watch a Khan Academy video or something outside of class. And then in class, it'll be discussion, active learning. We'll actually try things. We'll assess people in class, too. You sit down, you write an essay in class. So we'll get there. But the road is always long in education, and everything is messy, and there's lots of competing forces. But I think we can do that in the classroom setting. To me, the work environment is where it gets really weird, because we're not used to formal assessment. What happens is your boss sort of says, you did a good job, you did a bad job. But if your boss is impressed by your Claude output, are you really doing a good job? Have you learned anything? We may actually have to have formal assessment inside of actual organizations, as well.