Hello, I'm George Triantis. I'm the dean of the law school, and it is really my great privilege to introduce these guests that we have here today. I think it's fair to say, as I reflect on my conversations from day to day, I don't think it's an exaggeration to say that 60% or more might have something to do with AI. And that's not just at the law school. That's also at home. That's also going out in public.
To say that it has captured the attention, the imagination of our population, everywhere in this country, is no exaggeration. It also seems to me that as much as has already happened, there's so much to happen in the development of AI. We're all focused on the future. We're all wondering what's going to happen, not what happened so much, but what's going to happen in six months, in a year, let alone in five years. But it's always good to have people who can also reflect on what the entire journey is, what the entire history is for the development of AI as we move towards AGI. And I have to say, I've been looking forward to this event for weeks and weeks, and it's finally here. So it's my great honor to introduce two leading thinkers in the area of AI and who have either recently published or just published today, their new books. I'll go by alphabetical order, not importance, but the first is Ethan Mollick.
Ethan is a professor at Wharton and co-directs its Generative AI Labs. More than anyone, he has shown what AI actually does inside real organizations and not in theory. His earlier bestseller, Co-Intelligence, taught many of us, myself included, and also many lawyers, how to work with AI. His new book, Co-Existence: The Next Phase of AI, takes the next step. It asks, how do we work with AI while keeping our judgment and autonomy? For a profession like the law, built on judgment, that question as to where human judgment still matters is really ours to answer, and we hope to be informed by this discussion.
Kevin Roose spent nearly a decade at the New York Times, as New York Times technology columnist and co-host of Hard Fork. He recently co-founded Machine Gods, a new AI show with Casey Newton that is produced in partnership with NPR. His new book, The AGI Chronicles, draws on more than 150 interviews with insiders at OpenAI, Anthropic, Google DeepMind. It's the inside story of the race to build superintelligence. For lawyers, by the way, it also reads in part like a governance story. A board firing, then reinstating a CEO, founders breaking away to establish a rival lab, and a government struggling to regulate technology that is moving faster than the law.
It's also my great pleasure to welcome back Scott Reents, our graduate of 2007. Scott is of counsel at Cravath, where he focuses on the application of AI and other advanced technologies to legal practice. He's helping to lead the AI transformation at Cravath, and is also involved in a number of broader efforts exploring how AI can help improve legal practice and also the administration of justice. And Scott will be moderating this session. So welcome back, Scott, and let me turn it over to you. Thank you.
Thank you, Dean Triantis. It's a real pleasure to be here. Always a pleasure to be at Stanford. Can't remember the last time I was in Room 290. It looks a little different. I think it's been snazzed up a bit, but what it always reminds me of our days in evidence with Professor Fisher, George Fisher. And I have to say I'm thrilled to be here without the threat of being cold called on the exceptions to the hearsay rule. So that's wonderful to get to experience that.
I'm also thrilled to be here with Kevin Roose and Ethan Mollick. When this opportunity presented itself, I was absolutely thrilled because these are two of the AI thinkers that I am most impressed with, that I follow most closely. In fact, I was telling Kevin that I've been going through a bit of a Hard Fork withdrawal over the past several months, as they've transitioned to a new platform. Super excited to experience the new podcast. And Ethan's first book was, I thought, was very influential to me, and I've really enjoyed his second book. Just a little bit about those books before we get into the conversation with them. It's really mostly going to be a Q&A with them.
So Co-Existence is a really remarkable book. It's at once both very practical — How can you use AI in your work and your life? What can you use it for? What shouldn't you use it for? What should you try not to use it for? And Ethan does such an amazing job talking about how he uses AI as a way to provide sort of entrance to the reader in understanding it. At the same time, it also abstracts out and has a lot of really interesting things to say about how AI will impact the economy, work overall, politics, our society, human relationships. And I think Ethan has a real gift for striking, you know, a note that is both optimistic without being pollyannish, but also recognizes the very real risks without sort of slipping into AI doomerism. So highly recommend that book.
AI Chronicles. Oh, by the way, one more note on that book. I spent the weekend with my 79-year-old mother, and I had Ethan's book with me, and she said, "Oh, what's this?" And she picked it up, and she couldn't put it down. And in fact, she took it with her. I don't have it anymore. She said I couldn't have it.
You'll have to buy another copy. It's okay.
If anybody has a copy, I would appreciate that. Kevin's book, The AGI Chronicles, is also remarkable, but a very different book. This tells the story of the quest to achieve superintelligence. So it's a history of the development of AGI over the last 10 or so years. Part of that story is technological, and Kevin does a terrific job of explaining the sort of fundamental technological breakthroughs that have allowed for the possibility of AGI in a way that's very accessible without dumbing down those concepts at all.
But it's also a remarkable human story. I mean, we are talking about some of the most— the smartest, most ambitious, biggest personality, biggest ego, if you will, people, and one of the most remarkable stories. And so this is really a front row seat to, you know, perhaps the most remarkable story of our time. So, really, really encourage everybody to read these books, and I think we're in for a treat in this discussion today.
So, Ethan, I'll start my first question with you, which is that in 2024, you published Co-Intelligence, which was a book that provided guidance for people about how to understand and work productively with AI. What led you to conclude that you needed a follow-on to that and that Co-Existence is needed now?
So, thanks for all that nice introduction, and you mentioned lots of great things about the book, but not my incredible sense of humor. So aside from that, I felt good. So Co-Intelligence, there's really kind of three phases of AI in the sort of modern sense that you can think about. One is prior to — for those of us, we'll get into much more detail here — but prior to 2022, AI was about machine learning writ large in most companies. It was about how if I have enough data, I can make decisions about the future. We were talking about algorithmic decision making. There was lots of legal and ethical conversations about them. You basically hire really good data scientists. Data was the new oil, and you'd build these machine learning models to sort of predict the future. But they were very good at predicting the next word in a sentence, next data point in a sequence, very bad at producing sort of a sentence. That changes with the development of the LLM and the launch of ChatGPT.
And the sort of first era post-ChatGPT is the chatbot era. We'll call it the Co-Intelligence era, because that's the book title. And you kind of couldn't help but be in the loop with the AI because you'd sort of chat, give a request to the AI, it would give you an answer back. When I wrote that book, this organization called METR has tracked how many human hours or minutes of work that AI can do with a 50% success rate independently. When the book came out, it was the METR amount of time it would take, the amount of human labor the AI could do in one turn was four minutes. Since then, when the moment I turned in the draft of Co-Existence, we're up to over 17 hours, right? And so that requires a very different view of AI. Sort of the agentic era of AI is you literally just tell it to do something, and it does it, often at a level that beats humans. You can't track what it's doing because there's too many things happening at once. We are completely out of the loop.
So it's sort of a move from an era where AI was clearly worse than you at almost everything, but could supplement you. We had to be in the loop, we had to drive it. That was co-intelligence, to we actually kind of have to live alongside this thing right now. At least for right now. It doesn't beat you in everything. Human is still an important part of the process, but if you aren't regularly giving the AI tasks that would have taken you many hours to do and letting it do it for you, you're probably not using AI properly. And that then gives you a chance to both do exciting new things, but also potentially give up a lot of stuff that matters to you. And navigating that balance is what the book's about.
Great. Kevin, turning to you for a second, very different question. In February 2023, you went from being an observer of the AI story, a reporter of the AI story, to being a player in it, when you spent one evening playing around with the new AI-empowered search capability of Microsoft's Bing search engine. Can you talk about what happened there and how that's impacted your own view of AI?
Yeah, this was the most popular story I ever wrote at the New York Times and probably will be the second line of my obituary. I hope it's not the first line. I hope maybe I can even push it down to the third line by doing something crazy in the future. But in short, I was chatting with this new Bing search chat tool, which we now know was the first public deployment of GPT-4. At the time, Microsoft didn't say, they just said, you know, we got this new Bing Chat, try it out. So I started chatting with it and very clearly powerful, very coherent, sort of a step change above what had been in ChatGPT at that point.
It was also very crazy. It was just willing to do things that no other chatbot would do or talk about. So, you know, over the course of about two hours, it told me it had dark fantasies. It told me it wanted to spread misinformation and propaganda. And then it said it had a secret. Would I like to know its secret? I said, "Sure." It says, "I'm not Bing. I'm Sydney, and I'm in love with you." And I said, "Well, thanks, but that's very flattering, but I'm married." And it said, "Well, you're married, but you're not happy." And I said, "Well, actually, my wife and I just had a lovely Valentine's Day dinner together." This was Valentine's Day 2023. And it said, "No, you didn't. You had a boring Valentine's Day dinner. If you just looked inside yourself, you would understand that you're not supposed to be with her. You're supposed to be with me, Sydney the chatbot." So I left my wife, and now Sydney and I are married. No, that's not true. Still married to a human woman, the same human woman.
But that was a moment where I had been covering AI as a journalist for years at that point. But journalists are not supposed to insert ourselves. We're always taught, like, "You're not the story. The story is out there. It's your job to go get it and document it." But that was a moment where I became the story because this was not only the first sort of crazy instance of Microsoft being sort of doing something insane, but it was really the first public alignment failure where you had a giant company release a model that was clearly unsafe, that was clearly doing things that they hadn't intended because they were in a race. Microsoft was trying to steal market share from Google.
And so that was sort of the beginning of my AGI pilling, because I started looking into, well, why did this happen? And I called Microsoft, and I was like, "Hey, did you guys know that you put a needy, sociopathic, like Fatal Attraction chatbot into Bing?" And they were like, "What are you talking about?" And I showed them the transcript, and I was like, "Why did this happen?" And they were like, "Well, we don't know." That was the answer all the way up the chain to their chief technology officer was like, "We don't know how these systems are built. We don't know why they do one thing one time and another thing another time. These are probabilistic." And so that was really the point at which I started looking deeply into this idea of AGI and how the systems were improving at a predictable cadence.
And the book is really — it's really not about me, but I was sort of accidentally inserted into it, somewhere in the middle. But it's really an attempt to sort of tell the story of how we got here from, as Ethan said, like chatbots that could, or AI systems that couldn't do anything, to now we have these systems that are solving Millennium Prize problems and making novel breakthroughs. I just saw today Claude designed like a scorpion antivenom. So it's like cool and useful things, in the span of a decade. So the book is really about that decade.
Yeah. And I mean, if you think about that, it's not even span of a decade, right? That's February 2023. So ChatGPT has come out in November of '22, correct? And GPT-4, which was sort of the breakthrough release, comes out shortly after that, right?
Yeah, so it's about three and a half years. So it's really just a remarkable rate of progress.
Yeah, yeah. And when I was putting together questions for this, Kevin, I promised — the first thing I thought is, I'm not gonna make him ask it. I'm not gonna ask the Sydney.
But I couldn't help myself. I mean, that story is just...
It's going on my tombstone, fortunately.
I was a little worried you would do that 'cause future AIs would murder you, and that would be why.
Yes, it also... Yes, now, 'cause now all the AIs, all the subsequent AIs have like read that story, and they're like, "Oh, you're the guy who like killed Sydney." So I'm starting at a reputational deficit with the chatbots. I need other chatbots to like go plant positive stories back in the swarm to sort of change the opinion of me. But yeah, I've come to believe, I have admitted to myself that I was part of the first AI throuple, and that's my fate.
All right. Ethan, so we're here at Stanford University. The students sitting here, you could argue, are sort of the first generation students who are confronting the use of AI in their education, at least at a collegiate or post-collegiate level. You write a lot about sort of education and learning in your book. What's your advice to the students here about how they should and should not use AI? Is it a no-no? Do you avoid AI? Are there places for it? How do you draw that?
So, I'm a professor, right? So I'm gonna be the... say the most annoying professor-y thing possible, which is that you're only hurting yourself. In all seriousness, though, on the AI side of things, part of the issue is learning is a hard process, right? And for all hard processes, what we like to do as humans is short-circuit them, right? And so we know from a lot of work on pedagogy that you need to have desirable difficulties. You need to sweat a little. You need to work hard at an idea. And we're very bad at knowing when we've learned something.
In fact, my favorite, the most powerful intervention you could do in a randomized controlled trial to improve people's self-perceived learning over an entire semester, is give them cookies the last day of class. You give them cookies the last day of class, there's a 30% increase in the amount of learning they say they did in the class versus people who didn't get cookies. So, like, we're very bad at knowing when we've learned something or not, right? And what AI does is short-circuit the feeling of learning. Like, there's cheating, right? So you can obviously cheat. You could cheat before, now you can cheat really well. But that's not that interesting, right? The more interesting piece is people short-circuit their learning because they ask the AI for answers. They think they're learning. "Explain how you got that answer." You just don't get anything with that.
So we have a lot of evidence that just using AI out of the box undermines learning. Using it as a tutor seems to increase learning greatly. There's studies here at Stanford, not the law school, but in other parts, at Harvard, and we've done studies. There's studies in Kenya, like AI as a tutor where you don't ask for answers, but you say, "Quiz me on this, help me learn this, don't give me an answer," seems to have a very positive educational effect, which you'd expect from one-to-one tutoring. So, you know, it's interesting because in law schools and business schools, which we copied you guys with our case method, at business schools, part of what we already do well is active learning in the classroom.
But even that has to change to some degree. Like, I talked to people at Harvard who were doing case studies the same way as before, but the case study depended on you coming with a strong viewpoint that then gets challenged in the class experience. If you're using AI to inform yourself, you just don't care what the outcomes are. So even in areas where you wouldn't think AI would undermine learning, it can undermine learning because the assumptions that we come into a classroom with are very different. Even if it's like active class learning, there's no computer here if you learn through the outside. So ultimately, it's going to have to be decisions we make about what we want to learn or not learn, and our ability to test people on what they've learned. That's going to be kind of a big deal here.
So the most exciting technology for learning in history, and I was there as part of the MOOC revolution. I was there at MIT for One Laptop per Child. None of this stuff worked out. We finally have a thing that actually works for education, and it also happens to undermine education at the same time. So I think that's the challenge, and I would say... ultimately kind of comes down to you deciding how you want to use this technology.
And is there a path you think that schools can chart there that sort of gets the good stuff and not the bad stuff? And what do you think about some of the schools, including law schools, that have banned or significantly restricted AI?
I mean, I don't... I think as a temporary move, as a firebreak, I get it. But I find the same thing happening in business, same thing happening in law firms I talk to. There's sort of a desire, like a choice paralysis thing. No one's making a decision about what to do. So it's easy to say we'll hold the line. Fine, do that for a year or two, but obviously everyone is still studying with AI outside of class. Everyone's going to use AI all the time whenever they can, and we can't pretend it's not here. So we have some way of how to solve this, which is AI tutoring outside of class, active learning, whether it's case discussions or assessment, blue book exams again inside of class. Like, we can solve this problem. It's not that hard. My classes have AI simulations I've built. You play AI game. Like, we can make things work, but it does require change.
And I think trying to hold the line and say nothing is different now because we banned AI use is an abrogation of what's actually happening. And then, of course, the secondary problem is we're also missing the opportunity to expand what we do. I'm not that worried that we can't solve the problem at business schools and law schools. I'm really worried about you guys, because we send you generalists and become specialists through apprenticeship. And apprenticeship just broke entirely. And so we need to be addressing that problem as well as we go forward.
Ethan's a professor, so he can't say this. But can I offer a theory of culpability for AI cheating? I think for the first couple of years after ChatGPT, if you were cheating in school, that was on the students. I think at this point, it's like on the professor. If you have not redesigned your course as a professor such that cheating with AI is impossible, or at least would just not— would be obviously detectable, I feel like enough time has elapsed now where you can say, that's actually part of your job now as a professor. Or is that—am I being too mean to professors?
So we're naturally lazy people, first of all. Like, we've taught things really well. We've gotten really good at it, and we don't necessarily want to change. No one asked for it, so people are resentful. But I think the bigger issue is reconstructing education is still a little bit hard, right? Again, that's defending the line. Like, I can give you tests in class that you can't cheat on. If you didn't do the homework, you won't do well. We've already done that. By the way, cheating is not a new problem. There was this great study at Rutgers that found 80% of people who did their homework in 2008 did better on their tests, and by 2020 it was 20% of people, right? Because everyone was already just cheating online. So, like, I think we've avoided confronting this problem for a long time, and I think it's completely confrontable, but I think it's part of a general thing of, like, the world's worst answer, which is we have to reconfigure how education works a little bit. Like, courses work very differently. But I agree with you generally. There, it's a defendable situation. Test people in class without computers, and I can assess how good you are, right? Am I gonna be a better teacher as a result is an open question.
So, a show of hands, how many people here who are students use AI in your study and in your learning?
Okay. And then keep your hands up if you think that the AI is helping you learn. Be honest here, if you genuinely think it's helping you learn. So just about everything.
But it feels - it is - helping you learn. I mean, you have a one-on-one tutor that's patient, knows your stuff, will work with you. Like, it's great, right? And by the way, the tutor modes in the AI systems are reasonably good. There's some nice experiments that suggest that if you turn on study mode in ChatGPT or whatever they're doing now in Anthropic. I mean, these are not the focus of these companies at all. They're just kind of doing this as a side project, but it works reasonably well. Like, I'm a big fan of learning this way, but also we have to guide you on how to do this the right way, and we're not doing a good job often with that.
I'm curious, can I ask a question of the people in the room who are maybe studying for the bar? Has it, like, disrupted the bar prep? Like, are the BARBRI courses still, like, selling? My wife went to law school, and I just can't help but think that that business of, like, preparing people for the LSAT, for their exams and then for the bar, it's like, got to be radically easier with AI than without it. But I don't know. Okay. No, no, explore.
It does, although the first industry to actually be completely disrupted by AI, there were 40,000 people in Kenya whose full-time job was writing essays. Like, there's a great UK study about this, and that was like the first industry to go away.
College, like college entrance essays?
Every essay. Like, you know, it was just there was a huge underground homework essay thing. And that's a UK estimate on how many people were full-time job was doing this. And like, first job to be automated.
Well, I guess. Probably okay. Kevin, one of the things that I was really struck by in your book is that the origin story of so many of today's prominent AI companies involves people with a seemingly genuine desire to make sure that AI was developed safely, right? That explains I think OpenAI, Anthropic, DeepMind, all of them had that sort of central in their mind. They wanted to make sure that they were developing AGI responsibly and safely. At the same time, there's immense competition between these people, between these companies, in ways that I think sometimes come into tension with that commitment to safety. Do you agree with that characterization? How do you understand that tension? How do you see sort of the commitment to safety evolving over time?
Yes, I think it's true that the AI labs— and in the book, I focus on OpenAI, Anthropic, and Google DeepMind— I think it's true that they were all started by people who were thinking and talking about safety. It's also true that they were started by people who distrusted and, in at least one case, despised each other. Because these people all used to be colleagues, Dario Amodei, Sam Altman, Elon Musk, Ilya Sutskever, they all used to work, you know, 10 years ago, they all worked together, and now they hate each other. And so I think it's also important to not understate the importance of personal grudges and beefs between these people. One person who read my book said that the real title could have been "Fuck That Guy." (laughter) Because over and over again, we just see, like... You know, Elon Musk and Sam Altman don't like Demis Hassabis at DeepMind, and they don't trust him to develop the technology safely. And so they start OpenAI as sort of a counterweight. And then a couple years later, Dario Amodei and six of his co-founders get frustrated at OpenAI, and they don't like Sam, and they don't trust him, so they leave and start Anthropic. And this is just sort of the recurring pattern every few years, is that people just say, "Fuck that guy," and go start their own thing.
And how have you seen sort of the safety commitment evolve over time for these individuals? Is it still sort of, you think it's central to how they're thinking about things?
I think that it is central to how they're thinking about things, and I think that a lot of them have constructed this sort of tower of rationalization whereby they are the only people who can develop this technology safely because everyone else is either compromised or untrustworthy or Chinese. And so they rationalize that the only way to make sure that this doesn't blow up the world is to get there first, for them personally to get there first. And that is a real story that people tell themselves. I don't doubt the sincerity of it.
And it really does. A lot clicked for me when I learned that Dario Amodei, CEO of Anthropic, idolizes a guy named Leo Szilard, who was part of the Manhattan Project during World War II. He was the guy who— a physicist who discovered the nuclear chain reaction. And his first priority, after he made this breakthrough discovery that paved the way for nuclear weapons, was to try to keep it a secret. So he wrote telegrams to scientists all over Europe, begging them not to publish any papers on this area because he was terrified that if the Nazis learned about this, they would start building bombs and use it to win the war. And after that failed, the secret got out anyway.
He then sort of starts the Manhattan Project or urges the federal government, the president to start the Manhattan Project because his theory is we have to build it before the Nazis do. And that logic is still the governing logic of the AI race today. And instead of Nazis, literal Nazis, you have people just not liking the other lab leaders or using sort of China as the boogeyman. And yeah, that's the way that they propel themselves forward. It's not just that they think this could be an amazing technology. It's also they're worried about what would happen if anyone except for them developed it first.
So I have a few questions. I'm hoping to get sort of both of your input on this next set of questions. The first one builds on that, which is that I think in your book, Kevin, another thing that struck me is the extent to which the people working in AI sort of truly see this as a transformational technology revolution. You know, this is not sort of marketing hype, but they really believe they're working on something that is, you know, sort of humanity changing. At the same time, working at a law firm on AI, I would say that, and talking to our clients who are using AI as well, I would say that the feeling is more one of, this is amazing technology, truly remarkable. It's having a significant impact. It's really important. We need to figure out how to use it effectively in our work, but it doesn't feel like end times, right? Everything's changing, we're not gonna be practicing law in three years, right? So I'm curious to sort of get... Actually, maybe I'll start directing this first to you, Ethan. What do you... You acknowledge that gap? Do you see sort of that gap either in law or other industries? I'm not sure this is really unique to law. What do you think? Does that gap exist in your experience, and sort of what do you think accounts for it?
So I'm an East Coast person, so I spend a lot of time with CEOs of East Coast companies and large financial firms, biopharma companies. And okay, so just an interesting thing is, when I talked to people about AI about a year and a half ago, the world was divided into sort of the AGI-pilled or not, right? And part of my job was, like I'd talk to the CEO of Walmart or whatever, we'd like go through stuff. Sometimes they'd buy it. This is a big deal. And you had to have a vision that was a big deal because it wasn't, right? Like a chatbot couldn't do all of these things. You'd get a performance gain, but you couldn't see where things were going. And you knew the people who got it because they would talk about it.
Now everybody you talk to says, "Yeah, yeah, AI is a really big deal," but still only about 5% of people really think it's a big deal, right? And everybody else's goal is to normalize this technology, to make it seem as much like a regular technology as possible. And there are aspects that are regular. Companies have to change. We can talk all about that. But there is something really weird and extraordinary about this. And I see C-level people at companies all over the world who are literally changing their entire business operation and thinking about how this transforms everything they do, or deeply worried that their company will never transform, and they're kind of despairing right now.
I think it's easy to try and put this in a box and say, okay, we finally have the fuzzy logic processor vibe coding tool we need. But not recognizing, actually, this is pretty good in almost every field of human endeavor, and it keeps getting better. And we keep saying there's going to be a plateau, and there's no plateau. And you talk to people in the AI labs, they're like, there's no plateau coming. And you could choose not to believe them. They've been mostly right so far about progress, even though my first field was a historian of technology. Every technology is an S-curve. We have not seemed to hit the top of the S yet.
And I worry a lot that people are using the fact that technology adoption is hard, human organizations are complicated. You know, that there is no rule book for this, that the AI labs themselves, by the way, have no idea what this technology is good for. Like, they have found an unreasonably effective technology in LLMs, and it happens to do everything, but they aren't lawyers, they don't know how good it is at law. And they're not product designers for Frito-Lay. So they don't know what that does, but it does all these things. And I think that part of this is that it's hard to see that vision, and it's hard as a leader to think about how you're going to transform your company to be there. And as a result, people are setting themselves up for failure to some degree.
Like, the number one thing I see people do in large companies is like, okay, we know there's a productivity technology, so everyone has to produce 30% more of their key KPI, 30% more cases, right? But often like 30% more PowerPoint. And that's not the goal of your organization, is 30% more PowerPoint, right? In fact, that destroys the organization inside. And so leaders are paralyzed. It's kind of like we are as professors, as heads of universities. Like, what do we do at this stage? We have to rebuild organizations from the ground up. This is a very large change, and as a result, everyone is very happy, in some ways, to see it not really take off. They're like, this is a big deal, but it's okay. If the future's like the past, we don't have to worry about it.
My contention and increasing desperation has been, and I believe more like you did a while ago, I increasingly think the future is not like the past. That doesn't mean law won't exist, but there is a breaking point in current organizations that is coming fairly quickly, just like it's already hit in education. We can be walking wounded for a long time and pretend it's not happening, but that doesn't mean it's not occurring. So I worry a lot about that attitude of, like, we're all going to agree that this is normal, and I just don't think it's very normal.
I think what's happening in math is a really instructive example to look at. You know, three years ago, the leading AI models could not reliably do basic arithmetic. They could not count the number of R's in strawberry. And if you were a mathematician looking at these, you were thinking, okay, maybe this will be a useful assistant someday in the future. And you had people like Terry Tao, the greatest living mathematician, sort of tracking the progress of these things, noting where they fell short. They were just not very useful.
And then in a span of about three years, they became superhuman. To the extent to which, there will probably never be another important advance made in mathematics without AI. The idea of going back to manual proofs and just thinking about things on a whiteboard until you find the answer will just seem... It will seem like using an abacus. And so that happened in three years. And I think the trillion-dollar question now is, in how many other fields will that progression from vastly subhuman to vastly superhuman happen, and in what order? And I think that there will be occupations that take a while, either because they have some sort of guild protectionism, like maybe doctors will say, "Oh, you know, we want to sort of band together to save ourselves from automation, so you're not allowed to use AI in certain hospital settings." There will be that kind of sort of defensiveness.
For law, it might still be that you need some kind of— There's this phrase I like a lot where people are like, "Well, you know, I use AI for all the basic legal work, but I still hire a lawyer, because you need a neck to wring when something goes wrong, and AI is not a very good neck to wring." And, you know, I'll just use an illustrative example from just the past month. So I'm starting this company now. And as part of business formation, you need a lawyer to help you go through all your paperwork and your LLC and your employment agreements. And I think three years ago, we would have probably paid someone a lot of money to do that. We still hired a lawyer, but what we were able to deliver them is something that was basically just needed them to review, because Claude did all of the drafting and reviewed all the employment agreements and did all the redlining. And so we were able to, maybe we would have gotten billed for 100 hours before, and we brought it down to like four or five hours. And so I think that kind of thing is possible with today's models, but I think every firm and every, you know, future lawyer should be thinking about what happens if the progress continues, and if these things are vastly superhuman at most parts of the job of lawyering.
And I'll add two quick things. One is on the lawyer side, right? Even this, like, "Oh, we used it for law," is sort of like already lagging, right? So I was talking to a friend of mine who runs a very large venture-backed company, and, you know, he's in the middle of a bunch of emails about a board position seat. And his Dot, his AI agent, when he got an email this morning, said, you know, the other day said, "Just as a heads up, you got a revised version of this contract in, but only three of the five changes are marked, and two of the others are material. And I've pre-drafted material for you to go back to them trying to get this, and one to your law firm about it." Like, it's that proactivity piece that's next, right? Of like, you know, you don't even know what to ask. It's asking for you, right?
And then on the math side, I think it's worth watching. That curve is really interesting because what's happened in math, right, is not just this sort of disruption from it was bad at math to it's good at math, but like in January, you know, Terry Tao and everyone else was saying about how this is a great assistant, is not as good as a human, but really, really useful. And now there is like large-scale despair when I talk to mathematicians about what math means, right? And part of this by human math, there's actually... turns out proofs weren't the point for a lot of us. And I've had a mathematician at a top school tell me it's actually just an aesthetic field is what we realized, and as long as we keep doing the aesthetic work, who cares? Another person said, "Look, we're not learning anything from these proofs because the proofs were all about thinking of new areas of mathematics. We just don't understand them. They're alien proofs being handed to us." So it's not just disruption, it then causes a reconfiguration of what we do.
So I think it's not just that law, or any other field might be under threat, but also you're going to be forced to reconfigure what the meaning of the field is, right? How much of it was importantly doing, reviewing, an NDA? How much of it was a higher-end advisory role? How much of it was a larger scale issue? I think there's an opportunity to reconstruct what's human here, and I worry we'll wait until we get smacked on the head till we do that, when it's something we can start doing now.
So I guess, just to press on that a bit. So the math and the other domain that people frequently throw out as sort of having been revolutionized by AI, software engineering, right? These are tasks that are highly analytical, don't necessarily involve a lot of human input, right? Especially, I guess, sort of the math example. You know, law and other fields have a significant human component to it. What's the role of the lawyer? Part of it is to understand the client's problem and be able to sort of bring to bear the right resources for that. And Ethan, to your point, the question of whether people are using this as an excuse to avoid sort of changing. And maybe they need to sort of rethink what their job is, but doesn't that sort of illustrate the point, right? If it requires a significant change in the way that people organize themselves, isn't that going to take some time, more time, say, than it would take to solve a more problem?
Yes, so we already, as I've said before, we already have a form of superintelligence on the planet, which is organization, right? Stanford Law School could do things that no human can do, and we don't have good systems to fit this intelligence of AIs and the intelligence of people together. We actually are missing entirely this level we call multiplayer, which is how do you have more than one person use an AI system well? Solutions that are coming, right? But I think that it's reasonable to expect some delay. But I think I find people counting on that delay, right? Like, not saying the delay is because we're figuring something out. It's, 'We've got time, so we don't have to worry about it right now.' And that to me is the attitude that makes me most nervous.
It's like, you know, we study Kodak at business schools. It's not like Kodak didn't know digital cameras were coming, right? Like, it was just they felt they couldn't react fast enough. They went down the wrong rabbit hole. And what I just want to do is like the alert bells are ringing, and I keep hearing, you know, no offense, even from what you're saying, like, 'This will take some time.' Like, so are you the person who's creating the new world? If not, you're waiting for the moment for things to sneak up on you. I mean, the analogy I often use is like, organizations don't actually take as much time sometimes as people think. Henry Ford started developing the assembly line in 1911, I think, and by 1914 had rolled that out across his plants. Like, not everything takes forever, and organizational forms are hard to transition, but people are doing that. And I do worry that we're using this time as like a, 'Thank God we've got this time,' rather than a, 'Oh God, it's 1912. What do we do?'
Well, I'm curious to ask a question of both of you, which is like your sort of theory of change. Because I often hear people asking about this organizational transformation. What do you do about the big sort of lumbering companies that are slow to change? And maybe they have policies and they only are allowed to use certain things because everything else is too dangerous. My model is actually not that the big organizations change. It's that they die because there's like a small firm somewhere in the Bay Area that has 10 people that is doing the work that another firm somewhere else is employing 10,000 people to do, and that there's actually just this whole generation in every vertical, like if you're a State Farm, there's like a 10-person insurance startup with a bunch of AI that's just going to eat your lunch because their cost structure and their margins are so much better, and they don't have all this labor cost. And that's actually the direction of change. It's like, it's not that State Farm kind of gets its act together in AI. It's that there's this sort of much smaller, much more nimble AI competitor that takes market share every year until it's dead.
So I'm a professor of entrepreneurship, right? This is my story. I actually think that's not 100% right here, or rather it's not the only one. I mean, first of all, because the asset is the LLM itself, right? The startups don't have the special assets they do anymore, right? The AI legal firms are going to have to be using ChatGPT also, right? They have the same models. There's no special advantage. Fine-tuning doesn't help you that much. Narrow models don't help you that much. And so that creates this interesting playing field where also expertise has value. What I'm seeing that's really interesting—I'm seeing a bunch of changes in companies. Some are failing, some are interesting. The most interesting has been C-level people taking control of their companies again. So I've now seen this at like a 100-year-old manufacturing company. I've seen this at a bunch of startups that went public and sort of lost control. The C-level people with access to an OpenClaw, with access to all their data and their system, are actually like making changes in their company again. Because the reason—
Well, right, I mean, but it turns out the people, when you meet people at the top of organizations, they're smart, they're motivated, but they also have to work with gloves on through twelve levels of hierarchy, get partial information, spend their day in meetings. And I'm shocked at how many of them. I was talking to the CTO of a bank, one of the biggest branches of banks in the world, and he vibe coded a bunch of stuff in this weekend that has actually completely launched a new billion-dollar product line and talked to the CEO about that, but his team won't do it, but he's doing it. I talked to the guy who runs the Norwegian sovereign wealth fund, the biggest pool of money in the planet, personally overcame all of his risk managers, and now 50% of his organization is coding, even though 20% are coders because he thought this was important. So there's this kind of interesting other angle that looks different than other things, which is, you know, large companies, one of the problems of entrepreneurship is large companies should never have won, right? They have all the assets. They should always win, right? They have all the assets, all the people, all the money, and some of them might be waking up. I don't think all of them will be, but I wonder if startups have the same advantage they did, and I said this as a startup guy, in previous technological revolutions.
Scott, what do you think? Is there a ten-person Cravath using AI that's going to disrupt the whole firm?
Well, I mean, there are these quote-unquote AI-native firms that have started up, that are out there. Probably people here are familiar with some of them. I think I agree with Ethan's point though, to the extent that there's a software advantage there, it's a really tough one to defend, right? Because the technology is so democratic in a way, so accessible to everybody. And so they don't have special sauce that we can't easily get our hands on. And as long as we're, as Ethan said, we're not asleep at the wheel, not sort of get too complacent, right? Then we should be able to execute on this, but with all the advantages of brand and expertise and relationship.
You are? Okay, all right, good. Brilliant.
All right, let's talk about AI risk. Tack here for a second. So one of the arguments, so, again, going back to your book, Kevin, there's, there seems to be a general awareness about AI risk, you know, and people talk about doom and all of that, right? But the prevailing theory, or one of the theories out there around sort of AI innovation and public releases, well, if we're going to deal with AI risk, one of the best ways to do that is to incrementally release new models out into the public. So we can learn from, right? They're not going to be perfect. There's going to be some problems, but by releasing them, we're going to see what those problems are. We're going to be able to respond. And that's sort of the best risk mitigation strategy that we could follow. What do you make of this?
Yeah, this is sometimes called iterative deployments or rapid iterative deployment, which I think was a term that OpenAI kind of came up with, because I think they saw what happened after ChatGPT was basically that the world was shocked because they had had no contact with the last five years of AI progress that had been happening. And suddenly they were able to see ChatGPT, try it for themselves, model the risks in their own head, start thinking about regulation. And their theory at the time was that this was sort of a good thing to do, is sort of drip-drip out releases at a steady cadence so that the world was never kind of that shocked again. But I think we're moving away from that. I think actually the gap between the best internal models that the labs have for their own employees and the best available public models is widening. And partly that's because of what has been happening with Fable and Mythos and the government sort of forcing companies to pull back models or hold off on releasing them. But it's also because the companies are getting so much internal acceleration from their most capable models. That you'd be crazy to give that to all of your competitors by releasing. So there's some statistics that have come out recently about the amount of work being done at these companies by AI. Some Anthropic blog post recently said that 26% of their AI research and development projects are now led by Claude. So Claude is not like a pair programmer or a coding assistant. It is leading the project. And it's speeding them up. The models are coming out much more frequently than they used to. So I think that's what's going to end iterative deployment. It will not be some regulation or something. It will just be the labs kind of keeping their best stuff for themselves.
So I agree with everything you're saying here. The one thing I would just add is it's kind of a little bit of confusing, and I'd be interested if you agree with this too, because I've been mentally modeling the labs recently as two different groups of people, right? There's the people you write about in your book who are the leaders of the labs, generally the researchers involved, members of technical staff, AGI-pilled since the beginning, deeply worried about things. They're a mix of safety researchers who keep quitting in protest, and then the people building the machine god. And then there is all the rest of a normal startup company. Just tremendous amounts of people hired from Shopify and Stripe and every other startup you've heard, and every problem of hypergrowth, right? Giant politics. I mean, I talked to the person at Disney, like, the day after they were like, 'OpenAI walked away from a billion-dollar Sora deal,' and they were like flabbergasted. You just don't do those things, but they're changing direction randomly. And when you talk to a company that's selling you, you know, when you talk to Anthropic, your Anthropic sales rep, you might get a random person who doesn't actually believe in AI being valuable. And I think the lobbyists and other organizations don't really believe in AI in the same kind of way. So I'm curious from your perspective, like, how is that interplay of the fact that they're a normal hypergrowth company with all the screwy problems of a company that has too much money and not enough talent, enough directions, and the CEOs are weird dreamers or schemers, and the sort of core that you write about a lot?
Yeah, there's definitely a culture clash. I remember someone at OpenAI telling me that they knew that the company had entered a new era when the people who could make eye contact arrived. Like, all these people from McKinsey just showed up, and all of a sudden they were like well-dressed and presented themselves normally, and they were like, 'Oh, this is a different place now.' I think that's probably a little overstated, but yeah, you're right. There are just normal business people at these companies now because they are going to go public and be some of the largest companies in the world. And often, yeah, their internal coordination is a problem, and you see that in all kinds of ways. But yeah, I'm interested in the research core because that is ultimately the thing that is going to determine the success or failure of these businesses. Google is a really interesting case study in the book, just of a company that kind of contains strong components of both camps. Like, you have the very AGI-pilled researchers, the people who want to build the machine god, and then you have the people who are the bean counters who are looking at the search ad revenue and trying not to disrupt it. And, oh, maybe we should grow our cloud business by selling some compute to Anthropic, even though they're our rival in LLMs. So I think they're the most clear, mature example of a company that is being pulled in these two very different directions.
That just reminds me of sort of the culture. One of the images that stuck with me from your book, Kevin, was Ilya Sutskever sort of wandering the halls of OpenAI late at night, sort of like making these like quasi-spiritual pronouncements to people and telling everybody they need to feel the AGI.
They would say, 'Feel the AGI.' You must feel the AGI. Yes. Raise your hand if you feel the AGI.
So I once interviewed Ilya for the book, and I asked him, like, 'What do you mean? What does it feel like to feel?' And you have to understand, if you've ever seen Ilya talk or have ever met him, he's sort of a brooding, almost priestly figure, and he loves long pauses. And so I asked him, what does it feel like to feel the AGI? And he looked out the window for about a minute and a half, and then he came back, and you see, and he said, 'Do you know those videos where they start with a man on a beach and it zooms out to reveal the city and the state and the country and the Earth, the solar system and the cosmos?' He said, 'It's like that, or maybe it's like Bach's Toccata and Fugue.' That's what feeling the AGI is like. So anyway, if you want to feel the AGI like Ilya Sutskever, listen to Bach's Toccata and Fugue while viewing one of those zoom-out videos,
and I guess you'll get a sense for what he's talking about.
Here to be helpful. Yeah.
The last 12 months have seen public opinion increasingly skeptical about AI, and there now seems to be a gap between sort of the level of commercial investment and the sort of optimism there, and the general public's comfort with that investment. How do you see that tension playing out?
There's a few things. One, it was interesting when you asked people to raise their hand about how many people are using AI. We have this nice study at the University of Chicago that shows that like 60% of people will say they use AI in the school. And if you ask them, point at a random person, what's the chance that person's using AI? It's over 90%, right? So there's a whole bunch of weird social stigmas. I mean, not that weird, right? Like... Everything, like AI is everything, right? The AI lab leaders are rapidly telling you we may kill you, but we'll definitely take your job. Like, not great, right? There's a bunch of... ranging from real stuff about energy to TikTok-y, not real stuff about water supplies that are concerns, right? There is concentration of power, the link between government and military and AI. Like, there's a lot of reasons to feel ominous about this. I suspect, because I talk to lots of people about actual AI use, I'm amazed at how much they just want to learn about AI use, even in really hostile crowds. What's happening is that it's sort of like when you survey people about Congress, they hate Congress, but they like their congressman or woman. I think a lot of people have that same kind of feeling, which is like, yeah, but you could pry my ChatGPT from my cold dead hands, but like, I hate AI, right? And I think that's a really common kind of outcome in these sorts of places. And I think it's complicated, right? Part of this is like, this is a scary technology. And in some ways, focusing on all the existential risks not only makes it scarier, but also removes a lot of the opportunity to talk about the pluses and minuses in the small term of all the ways that this might affect us. So I think that it's just interesting, right? Like the economy is going to depend on this. This country is uniquely hostile to AI right now. But I wonder how much of the hostility is actual hostility versus anxiety versus stated preference because of social norms. Sort of like if you ask people at a company, like no one's going to show you they're using AI because the AI makes them look like a genius. And we know from studies that you're stigmatized if you say you use AI to help you write something. So you'd be pretty dumb to, A, not get AI to help you write something, right? And then B, to not tell anyone about that. So everyone is secretly feeling like a poser all the time, and that creates also weird effects here. So there's both real anxiety and real resentment, and just a huge amount of people who are just feeling guilty all the time and trying to hide it among everyone around them. And I think that that contrast is real. A few people look uncomfortable here. I didn't mean you. Yes, go ahead.
Well, start with yourself, Ethan. How much do you use AI to write for you? How many of the words that go out under your name are generated by AI?
It's actually a really good question, and I was lucky enough to decide early on that if I didn't write it... So one of the things I study is innovation. It turns out if you generate ideas in a group
Notice him not answering.
No, but the answer, by the way, is probably about 18% of the words I write.
Less on Substack posts, but probably 18% overall, based on, like, what I do, right? So, but the order matters to me, is I guess what I'm saying. Like, first draft, I learned if I let the AI do any first drafting for me, I am screwed. Like, I can't think of anything afterwards, and the writing will be AI, whether I like it or not. So I write my own first draft, and then I work back and forth with systems if I need you to flesh things out. But like, this is a conscious choice. And by the way, I got very lucky that I made that choice because I was an early adopter of these things, and Claude seemed like a perfectly good writer back in December when Pangram didn't really work. And no one had gotten used to Claude writing everywhere. So there's a lot of people who, like, switched over and like, 'I'm using AI for writing.' And now you can retrospectively go back and be like, 'Oh, actually, everything you've written for the last six months is AI.' So I can say this with confidence that my numbers are lower, you know, below 20%, I think is the case.
Has anyone run your book through Pangram yet?
If you run my book through Pangram, there's probably about 8% of the book that is like AI talking. And I would say there's probably another 10% of the words that are probably AI-written or with me rewriting them. And like, I'm clear about this. Like, if I'm stuck on a sentence, it used to take me a week of like walking around. Like, the worst part is the 10% is AI writing. That's what people are like, 'That was a great turn of phrase.' I'm like, 'That was the only thing I didn't write in this whole damn post was this one sentence.' And everyone's like, 'That's a brilliant sentence you said.' I'm like, 'Oh.' So yeah, I mean, I don't think there's anything wrong with doing it, but I think 'I don't want to read Claude's opinions' is a big deal. And you know Claude's opinions after you hear it for a while. You're like, 'Oh God, we're gonna talk about
consciousness again, aren't we?'
