A very small pool of these large tech players are becoming a single point of failure for the entire democratic block. That is not a resilient system. That is a single point of failure, and the first thing you learn as a computer scientist or engineer is don't build a single point of failure.
I mean, in many respects, yes. In reality, I think it's actually quite expensive to adopt open-source models.
I'm Eric Newcomer, author of the Newcomer Substack. Let's get into it. You're co-author on Attention Is All You Need. Take stock of where we are in this moment with foundation models, what we're capable of, and then I'll sort of follow up with where you think we're going. But are you surprised from the moment of writing that paper to where we are today that we've gone as far as we have in such a short time?
I don't think anyone on the Transformer paper had any idea what was coming. We built it for translate, like Google Translate. So it's a very scoped small problem to be solving. And it's a huge shock. I think an even bigger shock for me, given that it's been nearly 10 years now since that paper, is the fact that we're still using the Transformer. You know, like as a scientist, you hope you build something, but then you're not married to it. You want someone to come and build something on top of that that is much better, blows it out of the water, and that is progress. There has been a ton of progress, but it's hilarious how the Transformer has been such a great artist in its ability to steal the good ideas from every other architecture. Like when SSMs took off, this was like an architecture that would let you have a massively larger context window. And everyone was saying the Transformer is dead, we're going to shift over to SSMs, and then the Transformer just copied those ideas, integrated it into the architecture, and we still call it the Transformer. So, I don't know if we'll ever move on.
And yeah, so how much more room does it have to run? How much does the next great leap depend on a new idea? I mean, reasoning models were a big sort of leap outside of that mode, or how do you think about what it's going to take for models to continue to make giant leaps over the next couple years?
Yeah, it's exactly that. Reasoning is still running inside the platform of a Transformer. And so, I think people move up the stack, and that's where they invest their energy in innovating. In the same way that the fundamental training algorithms that train models haven't changed in much longer than 10 years, they're still the same thing. It might be the case that the Transformer is kind of just the platform. And we're going to continue to build on top of that for a very long time, and it would take something pretty monumental to shift us past it. But I still hope there's room for progress.
I mean, in many respects, yes, right? I think definitely. It's a general artificial intelligence, and it seems like whatever problem we point it towards, we can basically exceed human capabilities in that problem.
So, token maxing has become the buzzword of the moment, but it reflects a real issue, which is companies wanted to incentivize their employees to use AI and get the most out of it. So, they said, 'Run wild. We'll have leaderboards. We'll track how much you're spending.' Now, we've seen people like Uber and some other companies say, 'Oh, maybe we should track what's going on because we're going to blow through our budgets pretty quickly.' You work with a bunch of enterprises. What's your read on — are the inside tech companies ahead of this trend and there are lots of old guard companies that still need to have the token maxing failure before they pull back, or where are we in the arc of sort of spending for spending's sake on using AI?
Well, I think a lot of that exuberance came out of coding and those models. You see this typically in technology cycles. There's an exuberant adoption phase where the CEO tells the CFO no spend caps, like just adopt, adopt, adopt, and it goes into excess and then there's a correction period and that comes from the enterprise itself constraining spend, but then also from innovation and compression and efficiency within the modeling companies. So, I think we just did this big ramp up to a much larger tier of model. You'll now see one of these cycles of compressing back down. Costs will fall dramatically.
Is that going to hurt your revenue? I mean, if their customers pull back, there's enough growth elsewhere?
No, yeah. Like the usage growth as the costs come down, it will just unlock way more applications. We're still so early in enterprise adoption of AI. Yes, coding has taken off. And that might be hitting some sort of saturation point, but everything else within the enterprise is from my perspective still essentially untouched. Like the stuff that we're doing is so simple. There's like an iceberg underneath the surface to go do. And so there's probably another 10 or 100X in terms of token consumption just on the enterprise side.
After coding, what's two and three, or what are the next big applications?
It really depends on the industry. Like that's specific to the industry. So, in finance you'll see much more automated decision-making, communication, coordination between the different business units, automation of like regulatory and compliance. There's so much work that gets done inside these organizations beyond just writing software. Actually the overwhelming bulk of what's done is not writing software inside these legacy organizations. And all of that is going to be done by machines.
I want to talk about the response or one answer to token maxing, which is open source, or the idea that people move away from state-of-the-art models to find — I think there's a lot of conversation about routing to various models in a cost-effective way. What is your view, putting aside the geopolitical part of it, which we'll get into, what's your view on sort of the appeal of open source models from a cost structure point of view?
I mean, in some ways they're cheaper if you're willing to build up all of the infrastructure around them, like all of the software stack, the harness, maintain this thing, believe that you have the best serving efficiency. In reality, I think it's actually quite expensive to adopt open source models, and it looks cheap if you're willing to use like DeepSeek's API or Alibaba's API. You see the token prices and you're like, 'Oh, look, open source is cheap. These companies release open source models, and their prices are way lower than the American equivalent.'
What's wrong with that thinking?
What's wrong with that thinking? Well, one, you have to be comfortable sending your data to those parties. If you try to self-host, I think it's a very different equation. And then there's also the maintenance burden of that, right? Like these harnesses, these pieces of software that the models power, you're going to have to keep that updated over time as capabilities evolve, new models come out. And so that's another huge expense. I think similar to databases, in the same way, yes, you could take an open source database, self-host that thing, try to maintain and manage it. Nobody does that. Like very, very few people do that. Instead, it's all managed services. There are enormous database companies because it's a hard, painful, and expensive thing to operate.
So part of the pitch of coming to Cohere is that you don't want to manage your own service, models are going to keep changing, we'll be here with you, make sure it's safe. How do you think about why are your models shifting to being more open source in the first place? And then why did Cohere build models versus sort of being a facilitator of your own models and other models?
So on the first question, I think the piece around which model you use matters a lot. And for Cohere, it's very strategic for us to build our own models because we offer more leverage to the customer. Increasingly, what we're seeing is folks care a lot about cost, they care a lot about sovereignty, and off-the-shelf models aren't good enough, or they're too expensive for what they need to do. So the ability to customize becomes an essential part of the product and offering to our customers. The other thing is that we're not trying to compete at the very frontier. We're not trying to get into a like who can spend the most money competition. It's a losing race. And increasingly, what we see, the model is not the bottleneck. It's actually the market that's the bottleneck. So frontier models are so far out in front of what the market is doing that even if you have a model from a year ago, 18 months ago, 2 years ago, the companies are still trying to catch up to that. So the model itself is largely not the bottleneck. And when it is, they can't afford these massive models. And so, they're looking for a customized bespoke model.
