Sometimes it scares me. We had a customer, one of these AI clouds. They said, 'We're probably going to need about 500 petabytes over the next three years.' Last week, they came back to us and said, 'We're going to need an extra 2 exabytes on top of that 500 petabytes.' I think in the next 10 years, we'll see more difference than we did in the last thousand years.
Hi, I'm Met. Welcome to the Matt podcast. We talk a lot about AI models on this show and we talk a lot about compute, but today is about the layer in between, the crucial software infrastructure layer that feeds massive amounts of data to all those GPUs. My guest today is Renen Hallak, founder and CEO of Vast Data, a company that is surprisingly under the radar considering it's been most recently valued at $30 billion and powers top AI players like xAI and some of the biggest AI neoclouds in the world. Please enjoy my conversation with Renen Hallak.
Renen, welcome.
All right. So to anchor this whole conversation, I wanted to start with that mental model from Jensen at Nvidia that I find so helpful where when he describes AI, he uses the analogy of a five-layer cake where you have power at the very bottom and then hardware, then software infrastructure, then models, then applications. So where do you sit in that layer cake?
Right in the middle. That software infrastructure layer is the one that we do in collaboration, of course, with the hardware vendors underneath us and with the model builders on top of us.
Yeah. So at a high level, what does that mean, software infrastructure in the context of AI building?
Yeah. So everything you would imagine from the old stack we need for this new stack. We need to manage the compute. We need to manage the storage. We need to manage the networking. And then as you go up the stack, we need to make sense of all of it. Make sense of the data through a database, make sense of the compute through training and inference functions. And as you go even higher up the stack and we shift from training predominantly to inference and agents and reinforcement learning, we need more and more tools to enable this new world. Make it simple, make it safe, make it fast and scalable and resilient. All of that sits in that middle layer. Sometimes I like to call it software infrastructure and other times I like to call it the operating system for this new era.
There is a related concept to that five-layer cake that I think may have originated from Nvidia again but keeps coming back in conversation, which is this idea of AI factory. What does that mean? So if you start with a bunch of GPUs, what do you need to add to create an AI factory?
I think even below the GPUs, the data centers are looking very different today than they did 5 years ago. The old stack you had 10-kilowatt racks. Today you need 500-kilowatt racks. And so you need more power. You need more power density. You need GPUs, of course, versus CPUs. You need extremely fast networks and very large SSDs rather than 10-gig Ethernet and old hard drives. And so every layer of this cake needs to be redone. And from our perspective, you need much more data and much faster access to that data because we're no longer analyzing numbers and columns of a database the way you did with big data in the old stack. We're doing pictures and video and sound and natural language and that takes a lot more space and that's before new generative AI data that gets generated. And of course, all of this compute, all of these GPUs need very fast access to this information whether they're training on it or whether they're running low-latency inference functions. And so the entirety of the stack is new and I think the culmination of this stack is what we call the AI factory.
And is the concept of AI factory mostly for the big labs or like this whole data center plus model kind of thing or is it also for enterprises?
I think it'll be for everyone eventually. The big labs are starting. They're ahead of the pack and they're leading the way for everybody else. But anybody who stays with the old stack with old business information systems will get left behind. And so people need to start leveraging these new AI abilities. Some start by leveraging them out of the clouds, out of the hyperscalers, the neoclouds. Others build within their environments. But in all of these cases, enterprises will need AI factories in the medium term and then after that I think as we shift from people-based organizations to augmenting them with agents, these AI factories become a lot more than what we're used to thinking from just our compute infrastructure.
To play it back, right now if you go to a large enterprise, a non-tech enterprise, so like Pfizer or Goldman Sachs or Walmart, I think the dominant view is that leveraging AI basically means you start partnering with Anthropic or OpenAI and you have APIs coming into your walls and then you build some stuff, initially that was chatbots, now it's agents. Is the concept of an AI factory that you build your own AI? So you have your own data infrastructure but you also rent or buy GPUs and then you do model work. Is that the concept?
I think once you get to a certain level of scale, definitely it's more cost-effective to buy than to rent. And so from that perspective, yes, you should build your own AI factories. But beyond that, over time, I think all of these organizations' IP will be distilled in models, it will be weights, and they don't necessarily need to build the base model. I like to give analogies of the human brain. We were all born with a base model but then over time as we travel the world, we learn new things. We infer during the day, we fine-tune at night, and tomorrow we'll be a little bit smarter than we were today. And my model of the universe is slightly different than your model of the universe because we had different experiences. And so I think in the same way, all of these agents will fine-tune models through reinforcement learning over time. And every organization's IP will be able to be distilled into the models that they own and the weights that they've accumulated.
What does that mean in terms of what those enterprises need to be able to do? You know, like doing model work and weights and GPUs, that's a whole kind of expertise that ultimately few people in the world already own or already know how to do. And then a lot of those companies just don't have that. So how do they go from here to there?
I think again I like analogies. The analogy I would give here is you go back to the 1970s, you needed a PhD in computer science to operate a computer. Today everybody can do it. And the reason everybody can do it is not because we learned. It's because operating a computer became much simpler. The operating system made it intuitive, made it easy for everybody to use. It also made it safe for everybody to use. These enterprises are under a lot of regulation that they need to comply with. And so it is our job as the ones building that software infrastructure layer to make these new technologies accessible to the enterprises such that they don't need to think about it too much and that they can start adopting them at pace rather than get left behind and have new companies with AI experts take their place.
Maybe to push back a little bit or maybe to just dig in to make sure I understand, that's the whole premise of the cloud is to make things incredibly simple and use those functionalities as a service. The concept of an AI factory revolves around owning a lot of this. So it sounds like very kind of like on-prem private cloud which historically has been a lot more complicated. So is effectively what you're saying that there's an opportunity to abstract away all of this in a way that feels cloudish in experience?
I think so. I think the physical location of this AI factory can be on-prem, can be in a colo, can be in an AI cloud where you rent it from somebody rather than manage it yourself. Eventually my guess is that the hyperscalers will also enable these AI factories to be built within their premises. But so long as it's within your control and so long as you're the one that owns it in terms of really I think owns the information, owns the models, owns the weights, owns the agents. I think that's the distinction that I'm trying to make here. Doesn't necessarily need to be physical equipment within a building that belongs to you.
Okay. Great. Great. Do you want to spend more time on what you alluded to a minute ago about agents and agents doing their own fine-tuning? I mean, right now the way I think most people think about agents is that you do reinforcement learning to enable the agents, but I think you're saying agents do it.
I again like to think of agents as artificial people and over time I think we're all going to have agents that work for us, whether we're people or organizations, for different tasks. And we'll have teams of agents that can help us develop software or personalized medicine or whatever it is that we want done, we will have them do on our behalf. And I think over time we will get attached to our agents and we will expect them to know things that happened yesterday or that happened a year ago and that will allow us familiarity with our agents but that requires them to fine-tune models based on experiences that they had with us. And of course that also makes them smarter and able to do more for us and within teams they can learn from each other and I think that generates the next level of artificial intelligence versus where we are today where everybody is based on some base model from one big lab or another and every few weeks we get an upgrade to that model based on what they did in their environment. In this new way, each one of us, each organization owns the IP generated through these fine-tuning mechanisms rather than letting the big model builders basically own everything over time.
Okay. So just to play it back, so the knowledge loop happens within the enterprise and that becomes the new IP. And so we're saying you cannot do that with Anthropic or whoever, you don't want to because you don't want to expose your proprietary information, your processes, your secret sauce out to somebody else such that they can take it and run with it.
Which is super interesting, right? Because people used to say that about data but now we're saying this about intelligence.
Intelligence, processes, and memory, and therefore intelligence. Okay, does that mean that open source becomes the play here? Because if you want to fine-tune or do RL against a model, having access to an open source model enables you to do that. So is that your vision of the world?
Not necessarily. I think we need to build a way for people to monetize these abilities that they worked so hard and paid so much to get to. And so we need a way where on the one hand an enterprise does not expose their data, on the other hand a model builder does not expose their weights. And again, the software infrastructure layer is the right place to enforce that type of trust. And over time as the enterprise starts to generate their own models or fine-tune on top of base models, they should also be able to monetize the skill sets that they develop. And so if I run a carpentry shop and my carpenter robots are the best in the business, I should be able to lease to you that brain module such that you can build a chair or whatever it is that you're trying to do. But I'm only giving it to you for a week and at the end of that week you don't have access to it anymore or maybe I'm selling it to you but I still don't want you to see those weights. And so there should be ways for us to build an intelligence marketplace.
Okay. And I guess that goes to your big announcement of this week. Do you want to mention the principle of it and then we'll go into the details in a minute but at a high level, how does what you guys just announced address this specific problem?
Sure. So basically what we're trying to do is fill out that software infrastructure layer and what we've realized was that the five-layer cake that we discussed is almost complete. The models don't really live on top of the software infrastructure. Models over time have become a resource just like a GPU is a resource or an SSD is a resource and resources need to be managed. And so this announcement is all about model management. How do we as the operating system now get an understanding of this model is really good at coding and that model is really good at generating images and this one is expensive and that one is inexpensive and that way we can understand which tasks belong under which models. And that's today when we have four or five of them. As we start to run reinforcement learning and fine-tune, as we start to have hundreds and thousands of agents and over time generating new derivatives of those models, that's going to be a much bigger task.
One specific aspect of model management which we're announcing and we have a lot of the model builders as partners join the announcement is around confidential computing and enabling that ability of enterprises to run inference on-prem without exposing the model builder's weights. And so both sides can rest assured that my data is not exposed and your weights are not exposed to me. And so with help from Nvidia and encrypted memory underneath, we're able to make sure that no one sees what they're not allowed to see and to hopefully increase the total addressable market of these model builders into regulated industries, into large enterprises that are careful with their sensitive information.
Let's put a pin in this and we'll revisit in a few minutes and go into how that actually works. But taking a step back, I would love to talk about your journey a little bit and then how it led to you to build Vast, which is this incredible company which is valued at $30 billion that, you know, interestingly perhaps because you're right in the middle of the cake, not everybody has heard about yet. So, I think you were telling me, growing up in Israel, that you had a particular affinity for math. So, my words, not yours, but like you were a bit of a whiz kid in math.
Affinity for math for sure. Whiz kid, definitely not. It's funny when I came up here, the math museum is downstairs, and we had some big news this week about math. When I was in school 20 years ago, I spent 6 months of my life trying to figure out if P equals NP. Which I hope your audience knows, but for those who don't, it's the question of is it easier to verify a problem than it is to solve the problem, which intuitively it is. It's much easier for us to review a document and say, 'Yeah, that works,' than to write that document or to look at a picture and say that's beautiful than to paint that picture. But mathematically it's unknown if that is the case or not. And interestingly this is one of seven millennium problems that the Clay Institute back in the year 2000 put a million-dollar bounty on. And I remember spending those six months trying to solve it because I thought if I prove that P does equal NP, then I also solve the other six and I'll get $7 million. And if I prove that it's not equal, then at least I get one. And after 6 months, what I realized was that I'm not nearly smart enough to do it.
Do you want to provide the full context about what happened this week just for anybody that didn't follow? Another one of those problems, Navier-Stokes, was solved by OpenAI, or at least they claim that they've solved it. And the rumors are, hopefully by the time that this is aired, some of those rumors will come true that there's at least two more of the seven problems that OpenAI and Anthropic are trying to solve and are close to solving using AI. And these are problems that have been open for nearly a hundred years. And for the last 26 years, there was a million-dollar prize associated with them and still people were not able to solve. And so for the first time, I think we're at a point today where AI has proven that it can do things that we don't know how to do.
But going back 20 years, it was clear to me that if we could get computers to help with these hard problems, that's the way to solve it. Because people are slow and the evolution of our knowledge is slow. And if we are to see any of these things get solved within our lifetime, we need computers to speed us up. We didn't know how to do it back in 2006. And so I went along my way and joined a few technology companies. The last one I was with was a company called EMC which built large storage systems.
And if I recall correctly, so you joined a startup as employee number one.
And that company was acquired by EMC.
And then you helped run the R&D organization within EMC for that product.
And that acquisition happened in 2012. And by 2015, it became clear that there is a shot at computers helping us think and helping us solve problems. Because neural nets, which again when I was in school 10 years earlier or 15 years earlier were a curiosity. They didn't really weren't able to do anything. Now they were for the first time starting to show that they can do something. They were able to recognize which videos had cats in them and which videos did not have cats in them which was again astonishing to me because suddenly we can try to mimic the human brain without really understanding the human brain. And so immediately I tried to figure out how this happened and it was very clear that it wasn't necessarily new algorithms. It was giving those neural nets much faster access to a lot more data that enabled them to perform these very simple tasks.
So that was the intuition behind the founding of Vast.
2015, beginning of 2016 we started.
Yeah. And then obviously that was at a minimum three years before the Transformers paper came out and another two years before the ChatGPT moment. So what was the idea then because there were not a lot of customers using massive amounts of data for AI at the time?
There was no generative AI at the time. Google acquired DeepMind a year previous to that. But it was clear that none of the systems, none of the infrastructure that we had available to us would be scalable enough, would be performant enough to enable this next revolution. The architectures were built on an assumption that things needed to be fast or big. And here you needed something that was both fast and very, very large. And so we needed to build a new underlying layer to enable the success of this next revolution if it were to come to fruition. In the early days, of course, our customers were not doing generative AI, they were doing large-scale analytics. They were doing autonomous driving projects. They were doing medical imaging analysis, genomics analysis.
Traditional scientific computing. Oh, hedge funds obviously. Right.
Yeah. Hedge funds were trying to get signal from news feeds and natural language. And so it was the beginning of AI but before ChatGPT.
And actually as a way of getting into what Vast actually does, do you want to explain in super simple terms what you just said? So you started from storage and then storage as you just said was, the alternative at the time was either slow and inexpensive or fast but expensive and the whole scientific kind of breakthrough is to break this, right? And was it what you called DASE?
Disaggregated shared everything.
All right, so explain that in super simple terms.
So the old way to build scalable systems is based on a concept called sharding. What's known as shared-nothing systems where you have a lot of nodes. Each node is responsible for a piece of the pie. They collaborate with each other in order to serve up application requests. And as you add more nodes, you get more performance. You get more capacity. It works well up to a certain limit. And so once you have too many nodes, the communication between them starts to show diminishing returns. And so if you're analyzing text, it may work. If you're analyzing numbers, it works well. If you're analyzing columns of a database, that's fine. Once you grow beyond that, and today the amounts of data are four, five orders of magnitude beyond where they were back then, soon they will be seven, eight orders of magnitude bigger. It doesn't work. It breaks down because the communication grows quadratically within the cluster. And then from a resilience perspective also when one part fails everybody needs to recover that takes time. You can't really have more than 100 nodes in one of those systems. To build AI we need systems with many millions of nodes. We have one system at one of our customer sites that is today multiple exabytes within a single cluster delivering tens of terabytes per second. This is tens of thousands of nodes even today before the agentic revolution. And so to do that, we needed to build an architecture that ended up being the opposite of the way shared-nothing systems work. Instead of having the SSDs, the media directly attached to the CPU, it's on the other side of the network. And that through a very fast network and through new protocols like NVMe over fabrics and through new media types that allow us to save not just data on the other side of the network but also metadata, we were able to have all of the nodes see all of the information as if it was directly attached. And that's what led us to being able to do what we call shared everything. All of the nodes now don't need to communicate with each other because they share access to all of the information.
That's the basic idea behind our architecture. And it proved really, really good for a new type of storage system for AI and then for a new type of database for AI and then for a new way to orchestrate compute for AI. And we realized over the years, our customers forced us to realize that it's not just storage that needs to be redone, but the entire part of that stack that needs to be redone.
As an aside, I'm smiling. You must have been a lot of fun to pitch the storage company in 2016 to VCs.
Terrible. Storage was probably considered at the time the least sexy part of the entire stack. I owe you guys a lot of our success because as hard as it was to pitch storage to VCs, now we're reaping the benefits of having a lot less competition than we otherwise would have had. And so, thank you.
Yeah. Is that part of why that middle layer is less known? There's just less companies pitching it. I mean like the, and you don't have to comment necessarily but like the only companies that come to mind are like Weka and there doesn't seem to be a lot of companies doing what you do.
To our point, the ones that we're competing with started three years before us or 30 years before us and when we built this new architecture we were basing it on forward-looking technologies. When we started in 2016 the underlying parts weren't there yet, they became available throughout 2017, 2018, 2019. And so anybody...
So you had to believe that they were going to appear.
And so it wasn't just difficult to raise money from VCs, in those early board meetings when they kept asking me what do we do if this part doesn't come to life I kept telling them we don't have a plan B which they didn't like. And so I was not their favorite person in those days. Hopefully that changed over time, they made some money. But yes, there was no way for a company that started before us to make those forward bets. And so that's our advantage today. That's our moat, the fact that we're the only ones with this architecture. And it's true that there aren't many other companies in our layer. I think mainly because it's not sexy. It's a lot more appealing to build a model company or an application company than it is to build the plumbing.
And today's architecture is still at the very basis of what you do, all products currently.
That's right. We have one product that's this operating system, but all of the parts that we're adding are based on DASE. And that's part of the reason why we can't take open source and bolt it on top. We have to write most of it ourselves.
Why did you start building databases on top? What was not addressed by the market that you needed to build?
Same as with storage, we realized that there was a trade-off between price and performance and scale and resilience and ease of use in the world of database in the same way that there was in the world of storage. We realized that because our customers told us that. We have a big customer for example in Singapore and they have a lot of camera feeds coming in and all of those feeds need to be vectorized as the images come in and that results in trillions of vectors that need to be placed in a database. There isn't a database that was built for trillions of rows. There isn't a database that was built for thousands of agents analyzing different aspects of what's happening in the system at the same time. Old-stack databases were built for big amounts of data which today seem very, very small. They were built for people running queries and doing the analysis. And again that old architecture, the shared-nothing architecture, is not scalable enough to the levels that we need today.
So it's scale, right? I mean just to push back a little bit or probe, you know, I'm sure like Databricks or like Snowflake would say, 'Oh no, no, no, we're massively scalable and we can just run massive analytics on data and including in some in real time with the newer products.' And then you know, I think some people would say, 'Well, storage was solved, like that's S3, that's infinitely scalable.' But what we're saying here is that all of this may be true for certain use cases but not for what we're talking about here.
I think both of those companies that you mentioned started again about 5 years before us and so they too are built on that older architecture and they started top-down. They are based on underlying cloud services like S3 which were not built for this era of AI and we're seeing that today. That's a big part of the reason these AI clouds are starting to pop up because the traditional hyperscalers that built S3 built the old stack and these new AI clouds are building the new stack and now the older clouds need to adapt in order to stay relevant.
So is the ask to customers as they build those AI factories to basically move all the data to Vast?
They don't have to start by moving all the data to Vast. Usually we start by just serving the new applications, just serving the new AI applications. Usually what happens after that is that the customer, the enterprise, the organization realize that this is actually useful and I want it to access the entirety of my data set rather than just this subset. The good news is that we support all of the old-stack interfaces. And so if you're talking storage, S3, we support file systems, we support block devices. If you're talking databases, we support SQL. If you're talking streaming, we support Kafka. And so it's very easy to plug in those old-stack applications onto our platform in parallel to the new agentic workloads and AI applications. And in that sense, we become a bridge between the old world and the new world as companies make that journey.
