Why Cohere Is Betting on Enterprise AI, Not AGI

I first got introduced to neural networks when I was at the University of Toronto in undergrad.

In Toronto. Yeah.

Okay. Yeah. Happy to. So, I'm Nick Frosst. I'm a co-founder of Cohere. Prior to this, I was a researcher. I had the pleasure of working with Jeff Hinton at the Toronto Google Brain Group for a few years working on foundational research. I first got introduced to neural networks when I was at the University of Toronto in undergrad. I was pretty excited about the idea of AI at the time and learned about neural networks in the cognitive science department actually, and thought they made a whole lot of sense and were really exciting. And I remember learning about them in I think 2012 or something which was shortly after AlexNet, which was the paper that really showed that neural networks were the best form of machine learning I think at that time. And I remember learning about that in 2012 and thinking wow if only I had been here a little bit sooner, kind of missed it at the time. That turned out to of course not be true and neural networks have remained exciting since then and I've been working in and around them since learning about them back then.

Yeah. So I knew Ivan, the other co-founder, from University of Toronto. We were both there at the same time. Aidan and I were also there at the same time but we didn't really meet. So I properly met him when he was working in Google Brain as well and he came up to the Toronto office for a summer and then later came back and told me about transformers after having helped write the paper that introduced the transformer architecture. And we were all very excited about that and realized that there was an opportunity and a need for a company to make transformers useful for the enterprise. And we've been doing that since. Yeah.

Hm. So we started quite a while after OpenAI started. We were founded in like beginning of 2020, end of 2019. So were we starting at the same starting line? No, I think we were a little while later. But they had been working on scaling up transformers. We founded Cohere after the first GPT paper but before the second one.

Yeah. I mean all the architectures are the same architectures. Hilariously, the models that we're using now are still transformers, which is the paper that was introduced in 2017. So, we're almost a decade into pretty much the same architecture. There's been changes in scale and changes in how you train them. And now we're seeing changes in combining like having multiple models together and then using some switching mechanism to choose which one to use based on a query. But all of the models themselves are still transformer models. They're still the exact same architecture that was introduced by those authors back in 2017.

Yeah, they're really based on transformers. I mean they have these different tricks and things but ultimately it's still transformer-based.

I think the transformer, it happens to be very stable and easy to scale and scale is what has been making these language models suddenly useful. And I myself worked on a variety of other architectures. When I was working with Jeff we were working on a few different forms of capsule networks.

Which are very cool and had really interesting properties but did not scale well at all and so are not relevant from a practical aspect. They're not used in industry at all today mostly because they don't scale well.

Yeah. And I think transformers have some of the nice properties of capsule networks but are way simpler, way easier to scale, line up way better with the hardware that we have and so can be much, much more useful.

Yeah, let me handle those two questions separately. I don't think they have a ton to do with each other. So to address the goal of Cohere and as it relates to AGI or not AGI. So when people talk about AGI, I mean, we've been talking about it for many years now, but there's still ambiguity about what they actually mean. When someone says AGI, the only way that I interpret it is a computer that you treat like a person. That for all intents and purposes, you expect it to behave as a person. You ask it the things you would expect a person capable of doing and it does those. When you're not asking it to do something, it continues to behave as a person. Like that's kind of what I mean. And then when people say ASI, like artificial super intelligence, they mean that but instead of a person, a god. Like that's kind of what I mean. So I don't think we've built AGI and I don't think we're going to anytime soon. I don't think the transformer behaves like a person and I don't think making better transformers makes them more like people.

The analogy I use which I think was first introduced by Feynman is the flight, artificial flight analogy. And that's to say, you know, flight is a property. There are biological systems that can fly like birds. There are artificial systems that can fly like planes and they fly in completely different ways, right? Like a bird flies by generating lift by flapping its wings down. A plane flies by generating forward propulsion and then having a wing shape that results in a difference of pressure between the top and the bottom of the wing and that creates upwards lift. They share some things. The wing designs are kind of similar and that a plane can glide in the same way that a bird can glide, but they don't generate upwards lift in the same way at all. And that means that while planes are super useful and have completely changed the economy and even our understanding of the world and social dynamics and all kinds of things, there's lots of things that they can't do that birds can, right? Like we've made nothing as efficient as an albatross. We've made nothing as reactive as a bat. We've made nothing that can hover like, you know, can turn on a dime like a hummingbird. We've made nothing that can do that. The flip side is, you know, planes can carry insane amounts of weight. They're way larger. They can go way faster than birds. Like we've done all kinds of things that an artificial bird machine like an ornithopter wouldn't be able to do.

So I think we're at a very similar place with intelligence, right? Like we've definitely made artificial intelligence. It's just not the way humans do intelligence and subsequently it can do lots of things that people can't do. Like a transformer at this stage can summarize documents better than me, can find answers to questions in mountains of unstructured data better than me. It can figure out tool chains better than me. It can do mathematical reasoning better than me in many cases. But it can't understand nuance. It can't understand cultural dynamics. It can't work on its own in a goal-directed way in a way that a person can. And I think anybody who uses an LLM is aware of those limitations and they figure out how to make LLMs useful and they are really useful but understanding the pretty huge difference between working with a person and working with an LLM. Certainly all of that was a preamble to say that I don't think we're going to make AGI. And when I say we, I mean humanity. So I don't think anybody's gonna make AGI. And I think that used to be a controversial opinion. Like two years ago, I would say that and people would be like, 'No, look at where we're getting at this stage.' That's the widely held opinion outside of the very small Silicon Valley rooms. That's the widely held belief in all universities. That's the widely held belief for anybody who's kind of paying attention who isn't doing that within a small room in Silicon Valley. So that's not our goal because I don't think it's possible right now and I don't think we need to do that in order to make something super useful, right? Like I think LLMs are massively transformative, massively useful and they can still be adding so much more value to the world than what they're adding right now. So our goal is to make these useful for the enterprise, useful for people working with them. And we've made a lot of progress in that and there's more progress to go, like we have more work to do. And so that's what we as a company have been focusing on since we started.

Yeah, that's exactly what we've been focusing on. So when I say we're making it useful for the enterprise, I mean we're making it an efficient solution so that it doesn't just burn through money and instead is useful in providing value above and beyond what you're paying to use it. So we make very efficient models that can be deployed on a small number of GPUs. We make models that can be privately deployable, right? So that it can access secret private data without leaking that data to your competition. We make models that can be customized. So we work with some of our partners to make models for particular areas, particular languages or particular industries so that they're really good at the things that the companies want those models to do as opposed to being really great at generating images or doing mathematical reasoning which turns out is not the majority of the work that's done out there in the world, right? So we make things that are useful for our customers. We also work a lot on frameworks for using those models. So it's one thing to have a model, it's another thing to have a system that your employees can use to help them with their work. So we work a lot on things like that. Yeah, you mentioned distillation. So distillation is a little bit of an overloaded word these days. It was, I mean when Jeff first introduced it, he was calling it dark knowledge which was a very exciting term. And what he meant at that time was distillation being like logit distillation. I don't know if your audience is mostly machine learning engineers. I assume it's not. So we could break down what logit distillation means.

So a neural network, and even a transformer, we think about it as outputting a word, you know, like one word at a time or people think about it as writing a sentence. It's not really what they do. What they do is give you a probability distribution over what words are likely to come next. So when you give a prompt into a model, the very first output is a probability distribution over all the words that could come next and then you pick one word based on, you can base it on the probabilities themselves or you can just say I'm going to take the most likely next word. That's the sampling techniques. People talked about that a lot in the beginning of this. At this stage we kind of figured it out. You can use a pretty standard sampling technique and that works. Logit distillation is when you train one model based on the output of those probabilities from another model. And the dark knowledge that when Jeff first coined the term, he was able to show that you could take a really big model trained on image classification and then you could train a small model on the output of those probabilities from that big model and it would learn a bunch of things much faster. Like if you're imagining, imagine you're trying to recognize handwritten digits. This is kind of the canonical machine learning thing for a while. And you would have one model that was really good. And so its output probability distribution would say, hey, that three looks a lot, it's a three. It's definitely a three, but it looks a little bit like an eight. And if you trained a model on that probability, the small model would do a lot better than if you trained it with just saying, hey, it's a three. So that was distillation. That's not really done much these days in the world of machine learning, of transformers. In part that's because it's a really computationally intensive process. And if ever you want to train a small model from the logit distribution of a big model, you got to be running that big model all the time. And that takes up a lot of time. Also the probability distribution when you're recognizing handwritten digits, it's either a zero or all the way through nine. But those are the digits. When you're looking at the probability distribution from a transformer, it's like every word. So it's a huge probability distribution. And so it's just not as effective. So we're not really doing anything like that. But your question comes from the right place which is that we're interested in making efficient models and one way we could be doing that is with distillation. We're not, but it makes sense. It's a good thought.

Yeah. A lot of it comes from just going at that angle specifically and figuring out what do we need? What do our customers want the models to do and what is kind of cool but not really relevant to enterprise real world use cases. So generating images is a good example, right? Like we don't make a model, you can't generate a meme with our model and there's a really good consumer application for that. Like I like using generative images to make jokes and it's fun, I like it, it's fun to put up your picture and see it as a cartoon, like whatever, that's a good time. Turns out it's not really relevant for a business. There just aren't that many use cases, certainly not in the industries and the knowledge workers who we work with, just not really a lot of usage. And so if you just don't train your model to do that, you can save a bunch of parameters, you can make the model a lot smaller and still be really great. Now, the flip side of that is there's lots of things that enterprises do really care about, like tool use, like enterprise search, like being able to answer a question based on a really complicated technical document with image input and like graphs and schematics and things like that. And so we train the model to be really good at that stuff. And that's how we're able to make models that are efficient and subsequently much easier to deploy in real world environments. Just to call that out, like the model that we released recently is called Command A Reasoning. It's a reasoning model trained on enterprise reasoning. It requires two GPUs to run. DeepSeek models they take about 16 GPUs. Lots of the other models take significantly more. So there's models that we outperform that were like eight times as easy to deploy from a hardware perspective.

So you could do that. You could take an open source model and then try to fine-tune it. But the reality is most of the underlying capabilities of a model come from the pre-training phase. So we train our own models from scratch. There's about 10, call it 10 companies in the world that make foundational models from scratch. We are one of them. And we are unique in that group in our singular focus on the enterprise.

Well, there's some techniques that we do that I won't tell you about. But the short of it is you just choose an architecture that works. You're principled about the data you put into it. You train it for as long as you possibly can. And you focus on the things that give real ROI to businesses as opposed to cool flashy tricks for consumers.

Yeah. Yeah. Data curation and a handful of other techniques.

Yeah.

So we still train on the open web, on the text that's available for training from the web as a starting point and then further refine it on stuff like I've just described and add that into the pre-training.

How many models? We've been releasing about a new, the whole industry, all of us have been releasing about a new family of models once a year. It kind of seems right.

Yeah. They're more efficient. They're easier to deploy and they outperform on the things that our enterprise customers care about.

That's exactly right. It turns out that an LLM, like a language model, is really only as useful as the data it has access to. For a real world application, it's really only as useful as the data it has access to. And a lot of the useful data is, for very good reason, private secret data. It's data that you can't just send over the wire to some server somewhere. You can't have a company train on that. For various legal reasons, for various strategic reasons. And so when we work with a customer, we don't ask them to bring their data to us. Rather, we bring our model to them. And when we deploy, we deploy in a customer's environment, whether that's the cloud that they're managing or a virtual private cloud or a GPU that they have on a shelf somewhere, you know. And so we'll deploy wherever we need to deploy in order to be useful for that customer. That has opened up a lot of real world applications that you would not be able to use an LLM on otherwise.

Yeah, so a large customer of ours that's using our models and our agentic platform for them is the Royal Bank of Canada, or RBC. They're one of Canada's largest companies. And they're using our models across the company. For things like doing analysis on quarterly earnings reports, right? Like that's a good example. Or working through documentation on a bunch of different things or something, but it's mostly being used by the people who work in that company to do their job behind a computer faster, easier, more effectively.

Yeah. I wasn't there this year, but I have been in previous years. I actually missed this one.

So we are not, those guys are great. I got a lot of, they got a lot of respect for some of the research that they did. And they continue to do cool stuff. We are not working on evolutionary things. I think it's a promising cool research direction and I think one day some useful stuff might come out of it. It's so far not particularly relevant to business but really cool and one day might be, but for now no.

We've definitely done some work on new architectures and we continue to do a lot of research, right? Like we've published more than 100 papers at this point. Our people within Cohere have been collaborators on more than 100. We release the weights of our model into the research community. Our open science initiative, Cohere Labs, continues to collaborate across the industry with people and publish huge amounts of papers. Some of their work has been on new architectures. Some of it has been on efficiency or the impacts of AI or various other training paradigms or the consequences of evaluation and stuff like that. So they're kind of working all over the place. We have a big commitment to open science and I think that's the way that this industry will continue to evolve but our focus is always on making useful stuff.