Cost is by far the largest pain point that open models can jump in and solve. But you know, every business has a vision of getting better control over AI and customizing it for their business. And that's really their north star. You know, cost is something they can solve in the short term, but it then enables them to then go and customize these models for their unique use case.
Early 2024, there was lots of interest in fine-tuning your own custom models. Then it sort of went away and all of that it will just be wasted effort. It'll get stomped by the next model release. Seems like it's coming back now. You have a front seat to all of it. Do you think we're going through like another cycle or is it here to stay this time?
Welcome back to another episode of the Lightcone. Today we're talking to Jeffrey Morgan, co-founder and CEO of Ollama, the easiest way to run open source AI models locally and in the cloud. Ollama is used by 9 million developers, has 178,000 GitHub stars, and is used by 85% of the Fortune 500, which means Jeff knows a lot about the state-of-the-art of AI, what models score highest on benchmarks, and what developers actually download and keep using. Jeff, welcome to the Lightcone.
We're down to here. Like, what is the state-of-the-art? What are you seeing out there?
Well, I think the biggest thing we're seeing is a shift to open models, especially in enterprise, and that's from a mix of US and Chinese origin models. And it's predominantly driven by coding agents and also AI assistants, more co-work cases like OpenClaw and Hermes.
And because you sit in the token flow of like so many tokens, you have really good data on what models people are actually using and how it's changing. What are the trends that you're seeing?
Yeah, you know, it started as a way to run open models on your MacBook or other hardware, Nvidia, AMD, Intel. And earlier this year, we launched Ollama's cloud. And what we're seeing there is that it's predominantly Chinese models right now, of Chinese origin. But they're being accessed by businesses all over the world, especially US, and Germany is actually a big source of where open model tokens are being accessed.
Is it all about cost? Are enterprises coming because they just want to get the cost down, or is there anything more to it?
Cost is by far the largest pain point that open models can jump in and solve. But you know, every business has a vision of getting better control over AI and customizing it for their business, and that's really their north star. You know, cost is something they can solve in the short term, but it then enables them to then go and customize these models for their unique use case.
Is there a particular large enterprise that you can name that has done this?
I think there was a great article in The Information yesterday from AT&T, and it ends up they've already shifted 40% of their token consumption to open models. And that's right now predominantly through US and Europe models, but they're also evaluating the Chinese models.
What kind of workflows do they run?
Predominantly coding agents. I think what we've seen just from the extreme growth and, you know, per developer or per user token usage has predominantly been from coding agents. And then earlier in March and April, we saw OpenClaw take off, and subsequently the Hermes project, the Hermes agent project take off, which has then opened up that ability to automate a huge chunk of work over a long span of time to non-developers too, whether it's like finance or support or marketing or sales.
You had this actually very cool graph on the takeoff exponential for OpenClaw.
Yeah. So earlier this year, you know, this is a graph of token usage by developer on Ollama's cloud. You know, the average amount of tokens they use per week, and we kind of...
This is per developer. So like it looks, this graph looks like it should be an aggregate of Ollama's growth, but this is actually the per user.
This is on an individual user basis. How many tokens are they using a week?
And so there's kind of like two big inflection points. One is that initial run-up at the start of the year, which was driven by coding agents. So we saw Kimi, the GLM models, MiniMax launch. Finally, we had open models that could power coding agents. And then in April, we saw this incredible growth from OpenClaw really, which was then not just developers, but the rest of the world could take a hard problem, give it to an open model, and let it go complete the task, which obviously consumes a ton of tokens as it's figuring out what tools to use, what data to go fetch. We went from a context window of 128K to a million plus with open models. And so all that enabled this explosive growth. So it went from roughly 5x, so somewhere around, I don't know, 15 million tokens before all these core work type of use cases.
I think that's about right for that OpenClaw jump we saw in April. Obviously, in aggregate, it's, you know, in the 10 to 20x, if not more, as a whole. Through Ollama, we saw 150x since the start of the year. And so it's just the big, you know, interesting thing here is this huge surge in demand for open models, whereas open models, I'd say in 2024, 2025, from the large models being served, were mostly being served as custom models. So you'd take an off-the-shelf model like DeepSeek or Kimi and you'd fine-tune it for your use case, like for example, you know, Cursor had famously done. And from there, you know, you could serve that at scale. But seeing out-of-the-box open models being served, that really only took off at the start of this year.
Even since we started this podcast, these things sort of come in cycles, maybe. Like I feel like early 2024, there was lots of interest in fine-tuning your own custom models. Then it sort of went away and it's like all that it will just be wasted effort. It'll get stomped by the next model release. Seems like it's coming back now. You have a front seat to all of it. Like, do you think we're going through like another cycle or is it here to stay this time?
I think the release of these models, the cadence is only speeding up, which makes it ever more harder to, you know, stay on top of that and have a post...
You mean the latest closed frontier models are releasing faster than ever.
I think on the open source side, it's getting faster and faster. You know, this summer we've already seen three iterations of the DeepSeek Flash model, as an example. What used to be more of a six-month cycle.
Yeah. So the gap is like closing essentially.
Closing, and I think that makes it even harder to custom train models. On the flip side, I think the tooling is getting better, and so it allows teams that want to fine-tune their models to stay on top of it.
I mean, we're also entering this moment where AI safety is becoming more and more of an issue at the frontier labs. So the frontier may well slow down to figure out its alignment and containment issues. And then meanwhile, the open source models and open weight models are continuing to grow and get better.
Yeah. And you know, we saw the announcement and the release of the GLM-5.3 model and its capabilities from a cybersecurity standpoint, you know, being extremely impressive. It creates a big opportunity for whether it's startups or existing businesses in the security and governance space to really jump in and help. Because I think if you look at, you know, that AT&T article I was speaking about, largely the blocker to adopting open models is largely around security and safety. But from our experience talking to customers, whether it's in Europe, whether it's here in the US, if you can solve the safety problems, by and large, adopting the Chinese origin model labs is completely on the table, and it's really exciting for these businesses.
There's sort of this interesting moment right now where Hugging Face had to use open weight models to actually even detect the hack from the frontier.
A common question we get is like, well, where can I use open models that are leaps and bounds of an advantage over using a frontier closed model? And one of the key use cases is security testing and making sure that, you know, your software is secure.
Yeah. Because like if you try to get Claude to like pentest your product, it will just refuse to do that.
Correct. Yeah. By and large. Whereas there are literally obliterated security researcher models that you can find on Hugging Face that allow you to do it.
That is true. There are ones that are, you know, custom-trained to be, you know, even more, you know, liberal to go and attack these problems. But even the out-of-the-box models, they do come with safety training, but they're a little better understanding if you're doing this for, you know, a good, you know, use case versus one that's more of a negative or, you know, malicious use case.
A cool thing about is that because you guys are such a key distribution channel for these models, my understanding is that typically the model developers are contacting you before the general release to like coordinate launches and stuff like that. And you often get sort of like previews of what's about to happen, and you get these incredible growth spikes when a new model drops. Like I wonder if you could like tell us a bit about like what it's like to operate this thing at scale.
Yeah, absolutely. And like I said before, the models are coming out faster and faster and faster. And so we've developed a playbook to what does a successful day-zero model launch look like? And there are a lot of things to get right. There's making sure that it's supported in your favorite inference engine, which is generally sometimes a multi-week process to make sure it's fast, to make sure it's accurate, to make sure it's up to spec with the reference. There's also finding the right use cases and harnesses for developers and users to make use of this model. Generally, these models have new capabilities. This morning, you know, there's announced DeepSeek launched their first multimodal model from a large model LLM standpoint. They had previously had some smaller OCR models, which unlocks a whole bunch of use cases. But what they also changed was the DeepSeek harness, which launched recently so that it could support this capability. So step two is then to find the harnesses, make sure they're prepared to actually run this model and to do that effectively. But you know, every model's different. They all have different challenges and architecture changes and tool calling mechanics, and getting all this right is really hard. I think the key thing to do at the end of the day is to run benchmarks against the final product ahead of release and make sure that, you know, it's running as the research team at the lab specified.
So I think one very important part that you play in the whole ecosystem is you kind of create a very legible standard to be able to make sure you have the best way to use the harness for each new model, each new tool call, and all of it is consistent across all the different models, which is pretty hard to do.
Yeah, I think there are three things really that, you know, we try to package together. One is a harness, for example, you know, whether it's an off-the-shelf harness or an SDK to help use the model. Some existing harness that is designed for this. And what's great now, there's so many great open source harnesses. The Codex harness is open source. OpenCode is a great one and one of the most popular harnesses from Ollama users. Step one's getting that right, but then you got to package it with the model, making sure that it's, you know, available, it's reliable. If it's in the cloud, that there's enough capacity for it because day zero tends to be the largest growth day, obviously. And then importantly, there's the hardware and the providers. And that's actually where there's a lot of collaboration to be had, whether it's like an inference provider optimizing the model or, you know, with some of our partners, whether it's Nvidia or, you know, for example, working with the Apple silicon stack, making sure that it actually runs the model really fast. Because if the model's capable but it's really slow, that's not a great experience. So getting those three things packaged into a box, and generally you get the model, you're lucky if there's the ability to access the model a few weeks in advance. A lot of this stuff comes together in the last 24 hours before the model gets released, and so it's generally a fire drill.
So you're kind of almost like an operating system in the old world where you needed to really integrate very tightly with all the drivers, all the hardware, and then at the application level to make sure that all the apps were really tuned up well, and you are the glue for all of it. Right?
I do think the OS, which is generally a cliche analogy to use, is a good one because you've got the drivers for the hardware and the providers and the inference layer, but you also have the application runtime and making sure that the harness works. Gluing that together, it's a very combinatorially large problem to solve. And so doing that well is really difficult. But you know, over time you develop pieces that, you know, allow you to quickly develop that, test it, release it, and kind of have this common runtime that can match any harness to any model. And that's kind of the role we're really playing for developers.
You guys are very hardcore engineers and you fine-tune things all the way from the origins with Apple silicon all the way now to DGX. How do you build such a deep technical bench with that?
I think a lot of our team, you know, we aren't AI researchers by background. We're from VMware and Docker, and from, you know, other networking companies. And so the classic compute problems are kind of reinventing themselves in the inference land. And so largely, you know, that's where we like to focus our time. But I think at the end of the day, you know, it all comes down to the developer experience. What is it like when the developer makes a call to the API and gets tokens back? What happens? And there's more and more happening in that layer right now. And getting that right needs all the layers of the stack to work well together. And I think, you know, one of the challenges with open models has been that hasn't been happening at the rate of what a frontier model lab puts out, where they have, you know, the classic five-layer cake, right, that Jensen mentioned, which is like the apps, you have, you know, the model, you have the infrastructure and inference, you've got the chips, and you got the energy, and they've got all that ready to go for developers on day zero. And that's really the thing that we're trying to reproduce for open models. Of course, we're not going to do every layer of the stack, but we can help orchestrate that.
Yeah. And maybe in open models, there's more than five layers. Like that model layer actually has a lot to it, right? There's the model weights, but there's also a lot of the orchestration components that each model is uniquely good at. There's this developer API layer. There's so many opportunities to build between the model and the application layer that are kind of hidden in today's, you know, five-layer cake stack.
That's interesting. Do you think any of those like hidden layers might get unbundled and become their own companies or providers?
Absolutely. There was a really good talk from some of the Anthropic team, the platform team, and they talked about three big things. One was knowledge. How do you connect your company's data and context to the model? One is coordination. So, as you know, when you make a request in your cloud app or Codex, it goes off and spins out a bunch of sub-agents, some of those in the cloud, some locally, there's a coordination problem. And then lastly, there's an execution problem, which, you know, we talk a lot about as sandboxes, but these agents, more and more of them are moving to the cloud. There's a huge compute problem to be solved there. And if you look at what's happened classically in the cloud business, open source has meant that there are best-of-breed companies for each of those problems. Whereas you might have had kind of the, if you look back to the original generation cloud products, you have like the Herokus of the world, you have Google App Engine, all those things were bundled together. But what developers ended up preferring is best-of-breed products for each one.
It's the bundling/unbundling thesis, which is all things are just bundling or unbundling. Like the frontier model labs want you to be totally in their walled garden of managed agents and their context, their memory layer. And then meanwhile, little tech and all the founders out there and all the open source developers don't want to be caged in, so we're going to make all this other stuff, and it'll be an interesting moment to figure out like what ends up winning. I mean, it'll probably be some mix of both.
I think so. And you know, you've got this abundance of open model tokens that's being created. There are just dozens of open model providers that are able to serve these tokens. The new scarcity, the problems now are what's above the tokens, right? You know, how do you orchestrate an agent from, you know, A to B? These are problems that have tons of new, you know, systems and engineering problems that are just really hard to solve for an individual dev. Like there's no way they're going to build all those layers.
Well, the interesting thing now is because the coding agents themselves are getting a lot better, and ostensibly this is the worst the models will ever be. The classic reason why there was a moat here was it was just too hard to have really well-maintained software that was properly tested that actually satisfied user need. And what if that goes away? Like we're literally at this moment where actually maybe you'll just have a Cursor and it runs a markdown file in some TypeScript and it'll just, you know, there's no lock-in anymore, right? Like you could be using OpenAI's memory system one day and then actually, you know, you could have an agent be constantly syncing that against your own memory system and actually it just works, it's fine. Like it works over MCP. Like there's all, you know, there's plenty of like runtime testing and then there's no lock-in.
Yeah, I think a lot of the, you know, what we know of as harnesses today, a lot of those pieces will go down into the model. But, you know, as you kind of push a lot of the core loop of the model and hooks down to the model itself, there are these pieces that kind of come out, like memory is a great one. And the general guidance we're using is if it's a stateful problem, like there's storage involved, that's something that, you know, in the end can't go into the model because the model's trained and it's, you know, as we know, training runs are now happening on like a monthly basis, but it's still not up to date with the latest data. So generally storing data is like a huge problem space that I don't think will ever make its way down into the model layer.
Like managing credentials and that kind of stuff isn't...
Security, credentials, safety. Open models don't have all the safety tooling that closed model providers give you out of the box, but that's super important, especially for businesses to adopt them.
I'm curious where you see sort of the end or future state for enterprise on the balance between sort of frontier closed models and open source models, especially on spend. Like my, it feels like, you know, initially it was just everyone's just like allocating all of their budget to Anthropic or OpenAI. My sense now is yes, open source is clearly growing, but like so is Anthropic's spend. So like the things seem to be growing together. Like does that continue or do you think there's sort of like a steady state where it's like, I don't know, like half the budget's going to be on the closed source frontier model and half's going to be in open source or something different?
The super majority of tokens, and this is our take, it will be open models within a business. Call it 80, 90%. That doesn't mean 80, 90% of the budget will go to open models. In fact, I think what the open model community is doing incredibly well together is lowering the cost to make it more accessible. And so maybe you'll only pay 10 to 20% of the cost towards open models, but your token...
Most of your tokens will be going through the open models, right?
Which will enable a whole bunch of use cases on top because you have this abundance of tokens. You're not thinking about taking away token access from your team. You're giving more and more access. I think for the hardest tasks, that's reserved for these frontier labs where a lot of the best researchers are. And then from there, there's a whole bunch of problems in the middle, right? Where maybe it's a combination of open and closed models working together. I think the steady state is that most of the software is, most of the models are open.
It's kind of an interesting idea because as the models get more powerful, ideally you just want to delegate to like your smartest model to figure out when to go to like an open model, but the labs who own the models presume you don't want that.
And I think, look, I think that all the labs are aligned in many ways to one thing, which is how do you serve the customer? And I think it will be up to the customer to decide if I have a router where some of the scheduling and harder, you know, orchestration happens through a frontier model, so be it. But a lot of the kind of line-item work can happen through open models and the collaboration of the two together. I think we've seen a ton of projects, whether it's from Sakana AI or OpenRouter, that have combined the two and it's seen really good results.
This is not too dissimilar from a human organization, right? Like you have, you know, like a law firm, there's like a partner and then there's like a bunch of associates and like the partner farms out the work to the associates, right? It's like the same.
We saw the same thing with cloud computing where it was really a blend of proprietary software, some of them provided by the cloud providers themselves. For example, you know, AWS had DynamoDB, which was kind of their proprietary scale-out database. But then a lot of customers use that in conjunction with PostgreSQL. And in the end, you know, what we see is customers will use a combination of the two.
I think it's a very common pattern. This is also the same design for why the Apple silicon is actually more superior. The special accelerators for different kinds of workloads, for let's say image processing versus audio, that's been, or even go way back in the PC era, you had like your standalone audio card, right, graphics card and all that.
And speaking of Apple Silicon, should we talk about local models? Because you're in a bit of a unique position because you have large businesses both in cloud-hosted models and locally hosted models that will run on your laptop. What are you seeing in those two worlds and what do you think is going to happen?
I think it's incredibly exciting because it's similar to the closed versus open model question. It'll be a mix in our mind, and that's what we hear from customers as well, where for easier tasks you could run them locally and with lower latency and of course lower cost when it comes to the per-token cost. Ultimately, you're buying hardware up front and you'll use that in conjunction with these cloud models. What's exciting about this next generation of hardware, which we've had for a few years now, is just how good they are at running the 20 billion parameter to 40 billion parameter range of models, sometimes up to 128 billion parameters.
Yeah. Qwen 3.8 38B is now as good as Opus 4.6 for coding. Is that right?
That's what the benchmarks show.
It's incredibly exciting because you can run that on not the lowest memory MacBook, but the second lowest memory MacBook you can buy from the store. So it's incredible.
And are you seeing your users do that? Like what are you seeing people use the local models versus the cloud-hosted models for in practice?
From the side of which models they're running, we're seeing a really solid mix of US and Chinese trained models being used for local. And we have the incredible models from, you know, the original Llama models, of course, but also the Gemma models from DeepMind. These are great choices for local. But when it comes down to use cases, coding agents by and large are most effective with the large cloud models. You're solving really hard problems. You're writing code, tests, it's really difficult versus some of the document processing workflow use cases that run extremely well locally because they don't have as difficult of a task in the end-to-end, you know, problem you're trying to solve. And so that's where we see this hybrid execution model where some of the easier, more straightforward tasks run locally and then you have a router that can help decide, hey, we need to go to a large cloud model for this. And I think what that means for customers is that you're really dropping the costs even further when you're going from open models. Not just because they're cheaper to run in the cloud, but because now you can run them effectively for free on the hardware you're buying for your business anyways. And what we're seeing ultimately from the cloud coding agent models is it is predominantly Chinese models being consumed today. And for the local models, it's a really strong blend of US, Europe, and Chinese origin models.
Yeah, these two graphs are pretty stunning in comparison. Like basically for local models, the US and Chinese models are neck and neck. We're like tied. And for cloud-hosted models, like the US is like recoloring the x-axis. It's like 100% Chinese models. Basically, we need more US labs to make large models. Is that what this graph is showing?
Effectively. And you know, with the launch of the Nemotron Ultra model, we're seeing kind of the first wave of that, and it's really exciting.
Nvidia as a company is so interesting because their moat is not like trying to start new software businesses or sell, you know, tokens. They seem to be quite interested in just releasing a lot of open source and helping the ecosystem. And then the fact that they do that then helps them stay ahead of the game on the hardware side.
I think so. And you know, ultimately, Nvidia, what's so incredible is they're helping power an ecosystem around open models, whether that's the hardware, the models. You know, we've seen the new DGX Station computers that they're working on, which have GB300 on your desk.
How do we get on that list that isn't deafening loud?
Do you know the price point on that thing yet?
I don't know it off the bat.
Gary's buying it.
I know, right? Well, I looked it up. It's like, I mean, you can probably run a frontier model for like, I mean, very slowly for like two, $300,000. Is that right?
I think it's even more competitive than that. Yeah.
And you can run more than a frontier model at high speeds at a price point that isn't very far off what you can buy from a classic workstation computer.
You think about, you know, quite a few of the customers we talked to, some of them are banks, for example, or industrial businesses. They already have these Nvidia workstation GPUs in every single engineer's desk. Some of them tens of thousands of them. And so this is one.
They've been selling them for all the CAD work.
Well, this is for the kind of original RTX A6000s, but this is the next generation.
Yeah. Yeah. I'm going to have to email Jensen.
This was at GTC. So what we see here was at GTC. And we were one of the first people along with Elon and a few others to receive the DGX Spark as well, which sits on your desk and provides, you know, 128 GB of unified memory to run that kind of 20 to 120B model range. But obviously that's just the beginning of a whole new range of hardware that can run the biggest models.
So did you say you can buy a bunch of these and chain them and actually run a 400B model?
You can absolutely. They have this really fast network link and so you can stack them on your desk almost like a miniature data center rack and...
People do with the Mac minis.
Now this is like the production version of it.
