Anthropic CEO Dario Amodei： AI's Potential, OpenAI Rivalry, GenAI Business, Doomerism

From the internet into the model. Now we have a second stage that's reinforcement learning or test time compute or reasoning or whatever you want to call it. I think of it as a second stage that involves reinforcement learning. Now both of those things are scaling up together, as we've seen with our models and with models from other companies, and I don't see anything blocking the further scaling of that. There's some stuff about how do we broaden the tasks on the RL side of it. We've seen more progress on math and code, where the models are getting pretty close to a high professional level, and less on more subjective tasks, but I think that is very much a temporary obstacle. So when I look at it, I see this exponential and I say, look, people aren't very good at making sense of exponentials. If something is doubling every six months, then two years before it happens it looks like it's only 1/16th of the way there. So my view of this has always been that talent is the most important thing. If you go back three years ago, we were in a position where we had raised mere hundreds of millions. OpenAI had already raised $13 billion from Microsoft, and the large hyper-cap tech companies were sitting on $100 billion, $200 billion. The pitch we made then is we know how to make these models better than others do. There may be a curve of scaling laws, but if we are in a position where we can do for $100 million what others can do for a billion, and we could do for $10 billion what they can do for $100 billion, then it's 10 times more capital efficient to invest in Anthropic than it is to invest in these other companies. Would you rather be in a position where you can do anything for 10 times cheaper or where you start with a large pile of money? If you can do things 10 times cheaper, the money is a temporary defect. Providing a given level of intelligence will go down. I expect the price of providing the frontier of intelligence, which will provide increasing economic value, might go up or down. My guess is it probably stays about where it is, but the value created goes way up. Two years from now, my guess is that we'll have models that cost the same order of magnitude as they cost today, except they'll be much more capable of doing work, much more autonomously, much more broadly than they are capable of today.

Larger models cost more to run than smaller models. The technique you're referring to is maybe mixture of experts or something like that. Whether your models are mixture of experts or not, mixture of experts is a way to run models more cheaply that have a given number of parameters. It's a way to train models. But if you're not using that technique, then larger models that don't use that technique cost more to run than smaller models that don't use that technique. And if you're using that technique, larger models that use that technique cost more to run than smaller models that are using that technique. So I think that's sort of a distortion of the situation.

In terms of the cost of the models, one thing you'd be surprised by: people impute this thing like, 'Oh man, it's going to be really hard to get the margins from X% to Y%.' We make improvements all the time that make the models 50% more efficient than they were before. We are just the beginning of optimizing inference. Inference has improved a huge amount from where it was a couple years ago to where it is now. That's why the prices are coming down.