Wow, so I just wanted to start off. You were co-author on 'Attention Is All You Need.' Take stock of where we are in this moment with foundation models, what we're capable of, and then I'll sort of follow up with where you think we're going. But are you surprised from the moment of writing that paper to where we are today that we've gone as far as we have in such a short time?
Yeah, it's insane. I don't think any of the Transformer paper co-authors really saw what was coming.
Okay, so I don't think anyone on the Transformer paper had any idea what was coming. We built it for Translate, like Google Translate. So it's a very scoped, small problem to be solving. And it's a huge shock. I think an even bigger shock for me, given that it's been nearly 10 years now since that paper, is the fact that we're still using the Transformer. You know, as a scientist, you hope you build something, but then you're not married to it, you want someone to come and build something on top of that that is much better, blows it out of the water, and that is progress. There has been a ton of progress, but it's hilarious how the Transformer has been such a great artist in its ability to steal the good ideas from every other architecture. Like when SSMs took off, this was an architecture that would let you have a massively larger context window. And everyone was saying the Transformer is dead, we're going to shift over to SSMs, and then the Transformer just copied those ideas, integrated it into the architecture, and we still call it the Transformer. So I don't know if we'll ever move on.
And so how much more room does it have to run? How much does the next great leap depend on a new idea? I mean, reasoning models were a big sort of leap outside of that mode, or how do you think about what it's going to take for models to continue to make giant leaps over the next couple years?
Yeah, it's exactly that. Reasoning is still running inside the platform of a Transformer. And so I think people move up the stack, and that's where they invest their energy in innovating. In the same way that the fundamental training algorithms that train models haven't changed in much longer than 10 years, they're still the same thing. It might be the case that the Transformer is kind of just the platform, and we're going to continue to build on top of that for a very long time, and it would take something pretty monumental to shift us past it. But I still hope there's room for progress.
I mean, in many respects, yes. Like I think definitely. It's a general artificial intelligence, and it seems like whatever problem we point it towards, we can basically exceed human capabilities in that problem.
So, token maxing has become the buzzword of the moment, but it reflects a real issue, which is companies wanted to incentivize their employees to use AI and get the most out of it, so they said, 'Run wild. We'll have leaderboards. We'll track how much you're spending.' Now, we've seen people like Uber and some other companies say, 'Oh, maybe we should track what's going on, because we're going to blow through our budgets pretty quickly.' You work with a bunch of enterprises. What's your read on are the inside tech companies ahead of this and there are lots of old guard companies that still need to have the token maxing failure before they pull back, or where are we in the arc of spending for spending's sake on using AI?
Well, I think a lot of that exuberance came out of coding and those models. You see this typically in technology cycles. There's an exuberant adoption phase where the CFO or the CEO tells the CFO, 'No spend caps. Like just adopt, adopt, adopt.' And it goes into excess and then there's a correction period and that comes from the enterprise itself constraining spend, but then also from innovation and compression and efficiency within the modeling companies. So I think you'll — we just did this big ramp up to a much larger tier of model. You'll now see one of these cycles of compressing back down. So costs will fall dramatically.
Is that going to hurt your revenue? I mean, if their customers pull back, there's enough growth elsewhere?
No, yeah. The usage growth as the costs come down, it will just unlock way more applications. We're still so early in enterprise adoption of AI. Yes, coding has taken off. And that might be hitting some sort of saturation point. But everything else within the enterprise is from my perspective still essentially untouched. Like the stuff that we're doing is so simple. There's like an iceberg underneath the surface to go do. And so there's probably another 10 or 100x in terms of token consumption just on the enterprise side.
After coding, what's two and three? Or what are the next big applications?
It really depends on the industry. It's specific to the industry. So in finance you'll see much more automated decision-making, communication, coordination between the different business units, automation of regular regulatory and compliance. There's so much work that gets done inside these organizations beyond just writing software. Actually the overwhelming bulk of what's done is not writing software inside these legacy organizations. And all of that is going to be done by machines.
I want to talk about the response or one answer to token maxing, which is open source. Or the idea that people move away from state-of-the-art models to find, you know, routing to various models in a cost-effective way. What is your view? I guess putting aside the geopolitical part of it, which we'll get into. What's your view on the appeal of open source models from a cost structure point of view?
I mean, in some ways they're cheaper if you're willing to build up all of the infrastructure around them, like all of the software stack to harness, maintain this thing, believe that you have the best serving efficiency. In reality, I think it's actually quite expensive to adopt open source models. And it looks cheap if you're willing to use like DeepSeek's API or Alibaba's API. You see the token prices and you're like, 'Oh, look, open source is cheap. These companies release open source models and their prices are way lower than the American equivalent.'
What's wrong with that thinking?
What's wrong with that thinking? Well, one, you have to be comfortable sending your data to those parties. If you try to self-host, I think it's a very different equation. And then there's also the maintenance burden of that, right? Like these harnesses, these pieces of software that the models power, you're going to have to keep that updated over time as capabilities evolve, new models come out. And so that's another huge expense. I think similar to databases, in the same way, yes, you could take an open source database, self-host that thing, try to maintain and manage it. Nobody does that. Very, very few people do that. Instead, it's all managed services. There are enormous database companies because it's a hard, painful, and expensive thing to operate.
So part of the pitch of coming to Cohere is that you don't want to manage your own service. Models are going to keep changing. We'll be here with you, make sure it's safe. How do you think about why are your models shifting to being more open source in the first place? And then why does Cohere build models versus sort of being a facilitator of your own models and other models?
So on the first question, I think the piece around which model you use matters a lot. And for Cohere, it's very strategic for us to build our own models because we offer more leverage to the customer. Increasingly, what we're seeing is folks care a lot about cost. They care a lot about sovereignty. And off-the-shelf models aren't good enough, or they're too expensive for what they need to do. So the ability to customize becomes an essential part of the product and offering to our customers. The other thing is that we're not trying to compete at the very frontier. We're not trying to get into a who can spend the most money competition. That's just a losing race. And increasingly, what we see, the model is not the bottleneck. It's actually the market that's the bottleneck. So frontier models are so far out in front of what the market is doing that even if you have a model from a year ago, 18 months ago, 2 years ago, the companies are still trying to catch up to that. So the model itself is largely not the bottleneck. And when it is, they can afford these massive models. And so they're looking for a customized bespoke model.
I interviewed Ali Ghodsi at Databricks a week or so ago for the newsletter. He told a story about how he got in the model business. They said they spent 10 million on a training run. Really, the whole cluster cost them 20 million. And then 3 months later the whole thing they'd built wasn't worth anything. And his view was basically that unlike software which slowly loses its value — even if you have out-of-date software, your customers take a while to figure that out — models, they switch to the next thing. It sounds like you're offering a view that older models still have a lot of value like years out. Or you disagree with that point of view. What keeps customers using a year-old model?
Well, when was DBRX trained? It was like 2 years ago.
Yeah, this was 2024. So he left this a long time ago. This was sort of a reflection.
If you're on this super steep part of the capability curve, and when you're in a period where models aren't very good at anything, yeah, you'll feel that, because it's increasing super fast. But the reality is models have sort of gotten good enough at like 90% of what an enterprise wants to do. That last 10% is still super valuable. I'm not trying to discount that. And there's like a decade plus of technical work to go do to solve that 10%. But the idea that models change every 3 months and it's like some huge groundbreaking thing, it's just not true anymore for the vast majority of what the market wants to do. It is true at the very fringe, the frontier. Like in the pure sciences, yes, you can have a material breakthrough where the model from 6 months ago couldn't do it and the model now can. Like solving Erdős problems or protein folding, whatever.
