Cohere Co-Founder Nick Frosst on Enterprise AI and Sovereignty

Yeah, happy to. So I'm Nick Frosst, one of the co-founders here. I guess I got into AI back in 2012, 2013. I remember I was first introduced to the topic of neural nets. I was at the University of Toronto as an undergrad and I took Jeff Hinton's course and I remember learning about them in 2013 and thinking to myself, "Oh man, I really missed the boat on this whole if only I had been here a few years earlier, I really would have been in on the ground floor." I think about that a lot. I think about how I really was convinced that I had missed the boat and that, you know, this excitement was kind of just peaked. And yeah, I think that's it's really like humbling to think back to that and, you know, that amongst many other ideas I have had that have been wrong. But since learning about them in 2013, I've just been excited about neural networks and what they can do and how they can change the way we use computers completely. So after undergrad, I worked at Google for a while. I was a researcher there and I worked with Jeff Hinton for several years on foundational neural net research.

No, no, no. I took his undergrad course. But he did not remember me when we met.

Yeah. So then it was at Google that I became a collaborator of his and got the privilege of learning from him and working with him. And that's also where I met Aidan Gomez, who is the CEO and co-founder of Cohere. He was one of the authors of the Transformer paper. And he had met Ivan Zhang, who was a friend of mine from undergrad. So, the three of us got together in 2019, 2020 to create a company that sole objective was to make neural networks and transformers useful for businesses. So, we've been making foundational models since then, focused on the enterprise, focused on deploying them securely, customized into an enterprise environment so they can get real work done and we've been doing that now for six, seven years.

Yeah, I think it has been directional. So just to set the stage a little bit, right, like there's maybe I don't know maybe there's like 11 companies in the world that can make foundational models, maybe 10. Every few months, every like a few times a year there's one new one and there's one that stops doing it. But it's been pretty consistent for a few years now in that around that 10 ballpark. We are unique amongst that 10 in a few ways. We're unique in our singular focus on the enterprise. We're unique in that we're not American and not Chinese and that's pretty much all the others. So that puts us in a separate place. And we're unique in our approach to building a technology that empowers and uplifts and provides sovereignty for everybody who touches it. Because we don't have a consumer application, we're not fighting for people's attention, you know, we don't have ads in our models. We're not optimizing the models for stickiness to like keep you engaged in the conversation. We're just optimizing them to make somebody better at their job. That's all we train them to do. And when we deploy them into an environment, we deploy them securely on prem sometimes even. So we like drop our models into a customer's environment and then we don't see what goes in and out of that model. So that allows them to connect to, you know, real regulated data like healthcare data or financial industries or things within the government, real data that people want to get real work done on. And that is a pretty challenging way to deploy models. It's a challenging way to build them. It means you need to be thinking about efficiency. The number of GPUs that our models take is quite small for their performance. We think about that a lot. And we got there by being first and foremost interested in how to make this stuff useful now. How to make it useful today. Like we don't have a business model that requires AGI to be financially viable. We have a business model that is financially viable now. And that's just a commitment to pragmatism and utility that we have as people, like that's just part of who we are and that's part of our focus. So we've always had that objective.

Yeah, people are seeing great results with the technology and they're using it for a really wide variety of things. This is a complicated conversation in AI. People are always saying hey like what's the best use case? What's the best thing? It's kind of like asking what is the best use case for a word processor, what is the best use case for a spreadsheet. This is a fundamental efficiency optimization workplace tool and it is best used for whatever it is you are doing work on. So we build models, we build a search stack called Compass, we build an agentic framework called North to use them so that, you know, it's like a chat application but then it's also got automation so you can systematize, regulate, and automate consistent processes. But it's all stuff at this stage that people who are paying attention to AI are probably familiar with. It's just that we can deploy it entirely securely and customize and drop it into an enterprise company. What they then use that for is automating and augmenting whatever it is their knowledge workers are doing. And that's pretty varied. But just to give a few specific examples, right, we have people within financial markets inside banks using North to keep track of the quarterly earnings of way more companies than they used to be able to. So that's a really tangible like financial market specialists used to be able to keep track of some number of companies and create reports on them. Now they can keep track of two times that number because they use the model to help summarize and help write their reports, validate their thesis, use it to bring in information from their internal sources, cross-check it, and then they can be more efficient. We have people within customer support using it to resolve tickets. That's a great example. They used to resolve tickets, I think Kore was a customer of ours, they used to resolve tickets in like four days and then they got that down to two. And they got that by there's still a person in there validating making sure that hey this is the right thing we want to say but they've been able to decrease the average time considerably by automating and augmenting a large portion of it. Yeah, it's really all over the place. Like any work that you're doing behind a computer that is kind of boring and you know how to do it just is going to take you time, you can use North to do it.

Models have certainly gotten better over the few years, but it's pretty logarithmic, right? And mostly the real step change that has happened in utility of models over the past year has not been the underlying model capability, but has been the harness around the model. And that was super interesting. That's like the whole coding thing right now like using LLM for coding. We've been using LLMs for coding for, you know, years, but it wasn't until the past six months that the harnesses were made so much better that it became really viable. And that now there has to be an increase in the underlying model capability to make that legitimate. Like if you tried to use a model from two years ago and used a modern harness, it probably wouldn't work very well. But the difference between the model today and the model back then, it is pretty minimal. And you might be able to, you know, take one of our modern harnesses, rework it, and get one of the older ones working better. I think there is a lot of hype going on. I think there's a lot of disingenuous hype going on. I don't think, yeah, I think a lot of the rhetoric around this is like pretty clearly not accurate and mostly a marketing play but the utility of models has increased a lot and that's in part because of improvements to the model and mostly because of improvements to harnesses and to the way it's plugged in and connected and to what data sources you give it access to and stuff like that. Now we're always building better models and we released new models pretty consistently but we've been focusing a lot on efficiency. We're not trying to make the biggest best model out there in part because we do all these on-prem deployments and you would be surprised at how compute constrained even pretty established players are. So we're not making a five trillion parameter model because a lot of our customers do not have anywhere near the compute to serve that at scale. That's one of the reasons why you've seen those MIT reports that came out that said so many people built demos but didn't get into production with AI, right? Cost was prohibitive. Security was an issue. They couldn't deploy on prem because they didn't have the right compute or because it wasn't being offered by the provider they were going with. We have much closer the inverse of that. Like everybody we work with pretty much is in production. That's because we target that right from the beginning. But that does mean that we don't have the biggest best model. We have models that are optimized and efficient and like useful in a real world production setting.

I would also say that a lot of that cost is an issue, but the utility you get from these things is pretty huge when it's deployed correctly and when you're not just making a cool demo. When you're instead like empowering your employees to do stuff better, then the return you can get can be quite good, but it does need to be measured. It needs to be balanced. It's not a great strategy to put, you know, carte blanche onto token spends and just say like let it rip, you know.

Yeah. You know, I said that like a while ago. I was like, we're building ROI, not AGI. You know, it's quite pithy and so it has stuck around. But I think yeah, when I say not AGI, you know, I don't think autoregressive sequence models are like people. I think they're very not like people. I think you are very much not an autoregressive sequence model. And I think no amount of improvement of an autoregressive sequence model is going to make it like you. And so it's going to have very different behavior. And I think at this stage, like a few years ago I would ask people like hey do you think improving language models is going to get us to human level intelligence? And a lot of people would say yes. These days pretty much everybody kind of knows, hey, do you think just scaling LLM is going to get us to AGI? The vast majority whether or not I ask that question at universities or in public halls or in AI companies or in banks like anywhere I'm going I'm asking that question. Huge consensus is that no, autoregressive transformer-based sequence models are not enough to get us to humanlike intelligence. They are enough to get us an intelligence that is incredible and in many ways surpasses humans. Right? In the same way previous models of computing surpassed human intelligence in a bunch of areas, just this time the areas are bigger and the technology is much more powerful. But I don't think we're going to get to an intelligence that you treat or expect it to behave as a person. And subsequently, you know, we're not building a business model that requires that. We're building a business model that is instead as LLMs get better, we can provide more utility and I expect LLM to continue to improve, but we're not waiting around to say, hey, how can we reach profitability? Well, we'll build the digital god and then we'll ask it. You know, like that's not something. Instead, we're making something useful. We're going to ship it into our customers' environments. We're going to make sure that they're getting ROI on it, that they're getting something useful. We're going to keep improving our offer.

I mean it's an interesting one. I find like that really is the consensus. But if you spend any time on Twitter or listening to, you know, a handful of podcasts, you would get the impression that that's a fringe theory, right? And that's like, I've asked that question at master's classes at MIT. I've asked that at like all the ML researchers and in conferences like I've asked that of random people on the street. It is the broad consensus across every discipline. And the reason for that is that if you sit anybody down in front of an LLM and ask them to use it, they're going to be blown away. Like it's still the most incredible technology I have ever interacted with. It's why I've been excited about neural nets for more than a decade now and why I've been excited about LLM since I first found out about them. It's an incredible piece of technology. But you use it for a little bit and you realize, yeah, this is not a person. This is doing something very different than a person and in many ways better, but also in many ways worse. And if you try to treat it like a person, you're not going to get good use out of it. Instead, you have to treat it like a tool, like a targeted tool that you can set at specific tasks guided by your own strategy. One of the best uses of computers I think we've ever created. But it's got to be guided. It's got to be directed.