Open Source AI: Leaders from Hugging Face, SambaNova & Lux Capital on Democratizing AI | Slush 2024

Good morning everyone, you can hear me? That's good to see. Great to see you all. Thank you so much to my lovely panel. My name is Ravi Mattu, I'm the managing editor of DealBook, a newsletter on business and policy at the New York Times. And I can't think of a topic more apt today than AI and open source. As you'll probably be aware, Nvidia is reporting results later today. They're now the second most valuable publicly traded company in the world, above three trillion and challenging Apple for the most valuable publicly traded company, which just gives you a sense of how big the AI boom has become since that moment in 2022 when OpenAI unveiled ChatGPT. But within the AI world, there's a big fight going on, as you'll all be aware, between those who are trying to build proprietary models and probably try to dominate this space, thinking of OpenAI, possibly Google, possibly Anthropic, and raising absolutely enormous sums of money on the back of that, versus another camp which is the open source advocates. We have a few of those here today, but perhaps the most vocal in that camp is Yann LeCun of Meta, who's accused his rivals of trying to scare governments with catastrophizing the effects of AI and trying to capture the effort to regulate and what that will look like. At the same time, the geopolitics which we've been talking about a bit are pretty present. Obviously, we have a new administration in the US, and one tech entrepreneur is at the heart of all of that in Elon Musk, who has his own ideas which occasionally seem to contradict themselves around AI. So we're going to talk about open source with some real pushers of that idea, what works and what doesn't. Thomas, let me start with you. Hugging Face is obviously at the heart really of this kind of revolution in open source AI development. Just to get us kicking, tell me why from the start you've always been very keen that open source is the way to go. Lay out that case for us.

Okay, Klem, I'm going to bring you in on this. I'm going to jump over you, Grace, sorry, forgive me. Klem, Sova in Portuguese, that means new dance, and you had this idea to really hone in on the infrastructure of AI. And you have a very kind of different aspect to the AI equation that you're trying to figure out. And there's this quote that you use about looking forward with a profound mind, the idea that you need to kind of have some of that fundamentals to figure out what you're going to do next. Tell me about your particular angle on where you can fit that open source conversation.

So we were early proponents of open source. One of our co-founders, Chris Re, identified the fact that these open source models were going to be available and were going to compete with the closed source models in terms of their accuracy and their capabilities. And he was absolutely right. And so we leaned in. And as Thomas said, what open source does is it levels the playing field. So it doesn't just level the playing field for the users, it levels the playing field for the infrastructure providers, because then we can focus on how do we make that open source model run as efficiently as possible. And so, you know, going forward, it is about creating the ability to run these open source models very efficiently. Users want to be able to take advantage of the huge amount of compute that goes into training these models and then use those models to focus on their particular problems, fine-tuning those models, and then owning the model, right? Instead of giving your data to OpenAI or to one of the closed source providers, you get to own that model. And so the other piece of focusing on the open source model is that all that work has been put into training the model, and now you want to do inference on that model and actually get that model to give you some benefit at the application level. And so what we're seeing is a lot more compute being moved to what's called inference time or test time. And so this means sorts of things that OpenAI is doing with o1, where you have chain of thought, you've got sequential calling of these models, you've got parallel calling of these models where you've got multiple models which are being accessed at the same time and you compare the results. And so the whole landscape of compute is going to be dominated by inference and high-speed token generation, because these inference-heavy applications, people call them agentic applications or compound applications, are going to require a lot of tokens and they're going to require high token speed. And so that's the sort of infrastructure that we're developing at SambaNova Systems.

All right, Grace, let me come to you. As an investor, you're a backer of Thomas next to me, and you've spoken about open AI being a huge, huge advantage and the appeal of that as someone who is backing companies. And obviously, a lot of startups in the crowd will probably be knocking on your door after this. Tell us why the open bit is so important when you're looking for companies to fund.

Totally. Well, just for introduction, we're a deep tech, frontier tech fund based between New York and the Bay Area. And again, lucky to have led the Series A of Hugging Face. But three main reasons. One, Thomas already mentioned, right? We are seeing it level the playing field, increase transparency and accessibility to create open science in the hands of many instead of the hands of few. Also, from a performance perspective, right? That test time compute, we've actually seen smaller models are outperforming larger models, and that's basically applying more compute at that inference or usage time, like by 14x improvement or 14x larger models, they're doing much better. But third, and really why I'm excited, is it's more efficient. So not just from a compute perspective, but from an economic perspective. So let's just take a little analogy for a second, right? You're seeing these frontier AI models spending billions on compute. You know, we often say internally at Lux, if you're spending a billion dollars on something, there's probably a better way. Take that and apply it to a little bit of a sports car analogy, right? These frontier models are building the Lamborghini, the very luxury supercharged model, and that can be really good for some use cases. But you know, Clem actually was at our AI Summit and said, you don't need a private jet to go to work, right? And often these smaller models, the Toyota car as an example, can get you from point A to point B. And just one more kind of example on that, we're investors in a company called Sakana AI. They're like the DeepMind for Japan, a really cool company doing novel research in evolutionary algorithms, kind of inspired by nature. But their latest model just took eight GPUs to train. That's in big contrast to 75,000 GPUs you're seeing some of the latest frontier models leveraging to be trained on. And Together AI out of Stanford is doing some really cool novel architectures and state space models that are not just really performant but really efficient, you know, 11x cheaper than GPT-4o in many cases.

Just to pick up on that, is part of your bet also though that open will lead to a revenue model that's sustainable more quickly than closed? Because you mentioned the kind of money that some of your rivals are putting into some of the proprietary systems. It's kind of huge, it's kind of hard to comprehend some of this stuff.

Well, it goes back to kind of the performance of open source rivaling closed source. You're actually already seeing these open source models like Llama, Meta's models, becoming a lot more efficient. And if you contrast that to OpenAI GPT-3.5, right, that actually drove down prices 50%. So you're seeing open source actually put pressure on these closed source models. So I think that layer is actually getting more commoditized from an investment perspective, and the value comes from what you're building on top when everyone already has access to these awesome technologies.

That brings me to Thomas. Now, you've obviously, as I said, been at the heart of this kind of exponential growth in this space. I often meet startups that say they kind of employ 10 fewer people just because they pulled some module off Hugging Face to build their new startup or whatever it might be. Tell me about what you're seeing in terms of behavioral change with the clients that are using Hugging Face products, what kind of scale you're looking at, and maybe even project to the future what you're kind of looking at next.

You want to comment on that? Yeah, I think, you know, agents are going to be really big in 2025 and already starting to show that they're worth it. And sort of building an AI application as a thin wrapper around a model is so 2022, right? So going forward, it's going to be about a lot thicker layer of intelligence that uses the model as a module that gets called multiple times. And in fact, each of the models could be different. So you could have specialized models for different purposes that come together in a complex logic that actually delivers the model. And this has two ramifications. One is for startups and application developers like yourselves, there'll be a lot more to do, right, in terms of actually matching the needs of these agentic applications to the business workflows or this particular application use cases. The second will be the role that open source will play in providing this agentic orchestration layer, this programming layer, right? So there are a couple of emerging standards. DSPy is some work that's come out of Stanford, which is a way to combine multiple models together. And then Llama Stack is another approach that people are looking at. And so from the point of view of SambaNova, we're looking at how can we take these orchestration layers and run them very efficiently. Right, as I said, fundamentally, when it comes down to these agentic applications, it will be about how you can generate tokens very quickly, how you can support multiple different models very efficiently, and how you can make sure that the whole environment runs very smoothly. And so those are the kind of components I think are going to drive the future of AI models.

And if I may just quickly add to that, right, it's important to distinguish what open source actually is, right? We have open source models, you have the Llama models, you have the amazing repository of models on Hugging Face. You also have a lot of closed source data sets, right? And that you may be combining them together, whether that's proprietary data in your warehouse, in your research lab, in your factory, and combining those together, we're often seeing folks do in tandem. So open source isn't necessarily a catch-all term. You can still leverage an open source model and train that on your own data set. And one robotics example to finish that thread is a company called Physical Intelligence. They're an amazing team of researchers at Stanford and Berkeley and Google, and they're kind of building a general foundation layer for robotics in AI, kind of like a robotics brain that can work across any different robot with this software AI layer. And they are using an open source model, a VLM, vision language model, under the hood with a proprietary data set that they've collected in-house. So I think we're going to continue to see more of that in the next few years.