Renen Hallak from Vast Data. Thanks so much for joining me on the Ground Level AI Podcast.
Of course, thanks for having me.
Really good to talk to you. I saw you guys and your team for dinner a few months ago. And not to say that I didn't think that I would enjoy talking to you guys, but I didn't quite realize how fascinating the company was, and I was eager to talk to you again. So, thanks. I really appreciate it. We have so much to get into, but one of the things I wanted to start with is right off the bat what I was most profoundly surprised and interested by when I met you guys, and that is how much you are Vast Data. You know, obviously we're thinking data infrastructure, but it hadn't really occurred to me that we're talking about infrastructure, physical infrastructure as well as digital. And that winds right back to the data center. I think just a lot of people, you know, in their mind, they imagine GPUs and AI models being trained and inference. We don't really think about the data infrastructure that is necessary also alongside in the same place to make models do what they do and make AI do what it does. Could you explain that stack a little bit because I thought the physical nature of it was really interesting and the idea that most people don't even think about it.
Of course. So I think data is the defining characteristic of this new era of AI. It's surprisingly relatively old algorithms, neural nets, that have been around for many, many decades, but they haven't been very successful until recently because we couldn't give them enough access to enough information. And so all of these large training runs and now as we do model serving and inference and agentic workloads, that's all a data-centric process. And what that required was data infrastructure that on the one hand scales in capacity six, seven, eight orders of magnitude bigger than what we had before, and then on the other hand, of course, to feed these hungry GPUs we need to be extremely fast. And that breakoff between scale and performance is one that we needed to break in order to enable this revolution. And so the stack is very much centered on compute and data at the physical layer, but then on top of that you need a software layer that manages everything. And so in some sense it's compute and data, in another sense it's hardware and software. And we try to be that data software part of the stack. And over time that's evolved to be a lot more than just data management. Although that's where we started. Today we're managing compute, we're managing models, we're managing agents, we're adding safety and security aspects into the platform. And really it's transformed from just a data system to what we now call the AI operating system.
Are you trying to be the Nvidia of data? Do you ever use that word or is that totally off base?
We don't say that, but Nvidia is definitely our best partner. If you look at the other end of that yin and yang, from a compute perspective obviously they're the leader in GPUs and from a also software versus hardware perspective, they build all of the hardware that is needed for these AI factories and we always try to complement whatever Nvidia brings such that we have a full solution.
Take me into a data center, what would Vast Data's infrastructure look like? Would it look like? I mean, you know, I've been in big data centers where Nvidia's GPUs are humming along in these big racks of servers and there's all the networking cables that go into it. What does the data infrastructure part look like?
It's the same. You have servers in there with a lot of very large SSDs connected to very fast networking. And our part is not the hardware, but rather the software that manages all of it. And so our platform has what we call the data store, which is our storage solution to manage all of the bits of information. We have our database, which is a little bit higher up the stack to understand all of that data and make sense of it and generate insight from it. And then we have the data engine which manages the compute side and orchestrates functions, training functions, inference functions across various processors. Our data space is what we call our global namespace, which basically means that our system spans not just one data center but can build one global namespace across multiple geographies. Those are the parts that we had traditionally. We're now adding again an agent engine and policy engine and model management. As we go up the stack on the compute side, there's a lot more to do for this new stack to get fully built out.
Yeah, I totally want to get back to that. And sort of the confusing nature of today's AI stack, but I'm also just wondering how Vast Data got started. Can you give me the origin story?
Sure. It's actually a really good week to do that because we just had a big breakthrough. Not us, but AI in general, with OpenAI showing us that AI can actually solve problems that we don't know how to solve with this millennium problem breakthrough that they had a little bit earlier. When we started, the most AI could do was recognize cats on YouTube videos.
Which was when? What year was that?
2016. We started very, very beginning of 2016. That was I think an amazing breakthrough because for the first time we could see that neural nets can be useful. When I went to school 20 years ago it was an interesting idea that nobody really knew what to do with.
So even just cats was exciting.
Extremely exciting. Suddenly computers could do something that before we thought only people could do. Today computers can do something that nobody could do before, 10 years later. As we evolved those neural nets and gave them more access to more information.
And this week what you're talking about is OpenAI and the mathematics problems.
Can you connect that a little bit?
Of course. So when I was in school 20 years ago, I spent six months trying to solve a mathematical problem called the P versus NP problem, which is another one of these millennium problems that have that million-dollar prize from the Clay Institute. After spending six months on that problem, I realized that I'm not smart enough to do it and I need to move on.
You just said this is impossible. Well, I'm wasting my time kind of.
Well, at least for me, maybe others could, but 25 years later, we see that others weren't able to solve these problems either. And again, at the time, I thought if there was a way to get computers to help us with these hard problems, not necessarily the math problems, but with medical problems, with energy problems, with climate problems, the ones that are too difficult for us to figure out in time, then that's a mission that's worthwhile to pursue. But there was no thread to pull on in 2006. And so I just went and did other stuff for a decade. And then in 2015 it became clear that there's a thread to pull on because suddenly Google just bought DeepMind and they were showing these initial results in image recognition. But it was clear that there was a new way, a new avenue with which computation can start to evolve. And it was also clear that you needed extremely large systems, what we now call AI factories, because you needed large amounts of data and you needed very fast access to that information. And I was at the time at EMC, which was the largest storage company in the world, and I looked around at all of the different systems that we had available, and none of them were going to be good enough for this next revolution.
And that's the case even though like at that time in the mid-2010s, that was when the term big data had its moment.
And yet none of it was big enough.
If you look at big data, it's mainly numbers. It's columns of a database. And so it was big relative to what we had before, but today we're analyzing pictures and video and sound and genomes. And each of these pieces of information are many times bigger than the numbers that we were analyzing 10 years ago. And so yeah, that's part of the reason we're called Vast Data, to indicate that big data was not big enough.
I get it. Okay. I didn't realize that.
So, fast forward to actually more like 2022, 2023. When did you see that what was happening, you know, even before ChatGPT but certainly after that, this was suddenly Vast Data's moment?
So in the early days before generative AI, our customer base or the organizations that were doing the preliminary phases of AI, it was a lot around image recognition for example, autonomous driving car companies or medical images and genomics type projects or a lot of hedge funds doing quantitative trading and wanting to rely on news feeds. So natural language processing, those were the places, some government agencies, those were the places where you had very, very large clusters of in some cases CPUs, in some cases GPUs, but that required fast access to a lot of information. And those companies, our customers, our partners at the time, they showed us the future and they kept asking us for more and more functionality to enable them to make that pivot into what we now call generative AI post-ChatGPT. And of course once ChatGPT hit, everybody understood that there's a new thing here. And since then our customer base evolved into all of these new model builders and AI clouds. And I would say over the last few months we're starting to see a surge also in enterprise adoption of AI. And so those are the three pillars of customers that we serve.
Following up on that, I do think that still today a lot of people when they think of the data feeding AI model training, you know, they think of the scraping of the internet that sort of supercharged initial versions of chatbots, you know, text. But as you said, we're not at all like self-driving cars. We're talking about video. We're talking about photos. We're talking about genome. We're talking about medical data, enterprise data and things like that. How do you characterize that? Like what data is valuable in this day and age? I mean there was even this talk in the news that you know, a bankrupt company's data is so valuable now, you know, it could be bought for millions and millions of dollars. Yeah, how do you characterize data today?
I think all of it is valuable. I don't know how to put a price on it, but in terms of training these computers, they have so we have billions and billions of years of evolution that has built up our base model of our brain. And then each one of us travels the world and fine-tunes our model of the universe every day or every night that we sleep. During the day we infer, during night we fine-tune so that we wake up a little bit smarter than the day before. And the computers, they have some catching up to do and so the more data we could feed into them the better those frontier-based models become. And as we start to do reinforced learning and each organization, each enterprise will have tens of thousands, hundreds of thousands of agents, those agents need to take in all of the proprietary data that these organizations own and that will turn into their IP. If you think about it, a company's IP is a combination of its information and the processes that are in the brains of the people working there. If those people are augmented by agents, as we fine-tune models, those weights, the weights of the model becomes that organization's intellectual property.
So in addition to you know the news this week about the math problems and that sort of like jump ahead in capabilities, you know, of course another big headline making news over the past month or two or three has been you know, security of agents, of models and of data. I mean enterprises are terribly, terribly concerned about the security of their data. What do you think of that? Like is that possible as the agents get more powerful? There's a lot of talk of agents gone amok.
It is the biggest challenge that we have. I think the two biggest challenges that we have in order to make AI more widespread is security and simplicity. When you talk to these large enterprises, they're both under regulation not allowed to expose their information and also they don't want to expose their information. They've spent a long time accumulating it on the one hand. On the other hand, they don't have a team of 100 or 200 PhDs like an OpenAI does or an Anthropic does to spin up these systems. And so we need to make it very, very simple for them and we need to make it secure and both of those fall into the responsibility of that software infrastructure layer that I discussed before, that operating system layer that we're trying to build. And specifically to data security. There's a big challenge that we see, like you said, enterprises don't want to expose their information to the model builders. The model builders don't want to expose their models to the end users, to the enterprises and we need to find a way systematically to build that trust. And the best way to build trust is to prove again in a mathematical, cryptographic way that nothing is exposed. So part of this announcement that we're doing now is around confidential computing and model management where inference can start happening on-premises at whether it's the physical building of the enterprise or a virtual private cloud of the enterprise without the enterprise ever exposing their information to the outside world and have all of the models, all of the weights encrypted all the way up to the inference application such that the model builders don't need to worry about exposing their intellectual property and that way both parties work.
How are you doing that with outside partners? Is that part of like inside Vast Data doing the security work?
So we do the management of it again at the software layer and so we get the policies set and we understand who is allowed to see what and we make sure to enforce those policies across multiple geographies and across a large customer base. We couldn't do it without Nvidia which we mentioned before because they have the hardware support for memory encryption all the way up to the GPU, which is very critical in this case. We couldn't do it without our OEM partners like Cisco and Super Micro that are packaging these servers and building the physical aspects of these AI factories on-prem. And we also of course couldn't do it without the model builders that are joining this announcement with us to make sure that we have an end-to-end solution that works for all parties.
