The Data Frontier: from Spark to Agent Clouds — Matei Zaharia and Reynold Xin of Databricks

One of the advantages we have is that once you can get the data in the right place, the AI models are becoming pretty good. The generic agents are fairly... I mean, Ali talked about AGI already here. They have pretty good reasoning capabilities. Actually, I think many of the traditional software will be sort of rewritten with this new paradigm, which is just get the data to be there and then let's slap some AGI on top. Magic will come out.

Yeah, but without the right data, you can't really do that.

Before we get into today's episode, I just have a small message for listeners. Thank you. We would not be able to bring you the AI engineering science and entertainment contents that you so clearly want if you didn't choose to also click in and tune into our content. We've been approached by sponsors on an almost daily basis, but fortunately enough of you actually subscribe to us to keep all this sustainable without ads, and we want to keep it that way. But I just have one favor to ask all of you. The single most powerful completely free thing you can do is to click that subscribe button. It's the only thing I'll ever ask of you, and it means absolutely everything to me and my team that works so hard to bring the In Space to you each and every week. If you do it, I promise you we'll never stop working to make this show even better. Now let's get into it.

Matei Zaharia from Databricks. Welcome to In Space.

Thanks for having us.

Thanks for taking time out. You have your Databricks Data AI Summit going on. You were just telling me how the first summit that you guys ran was just 50 people.

Yeah, it was a little meet-up at Berkeley, I think. We put together... we did these tutorials and yeah, just teach people Spark.

Yeah. You know, obviously now it's like I think the headline number is like 100,000 people around the world, 30,000 in person. It's a crazy community. Well, I mean I just saw the keynote.

Ali is just... Did you know, was it obvious back when that Ali would be such a great CEO? Like he's a great presenter.

I mean, I think among our group of founders, it was clear that I think he'd be the best at this. And yeah, it turned out great. He's ramped up on so many topics running a company. He would just go in and study and talk to all the experts. Like, even if you can't hire the person, learn enough about finance and sales and whatever it was, and go from there.

Yeah, I mean, to me the most appealing thing about him is that he's funny.

It's true, yeah.

It's hard to make jokes about data, about serious topics and security and what have you.

Oh yeah, that's for sure.

Yeah. So, you guys launched a whole bunch of things. I'll just name-check briefly the stuff because we're not going to cover everything. Omnigents, your baby. El Tap, your baby. Dream Engine. We're also going to cover Genie, Customer League. You acquired Panther, Open Sharing, and there's Unity AI Gateway. A lot of these I think are things that you would expect a Databricks to do. It's part of the road map. Everyone in your category has similar things. But I think probably the two of you are leading the two most unique and differentiated initiatives in the landscape. Maybe we'll start with Omnigents and then we'll go into it.

I do think that a lot of people are exploring this sort of meta harness concept. What led you to it?

Yeah. There were actually a couple of converging lines, which I think is a good sign that you need something new. So, on the one hand, there's all the coding agent stuff internally. We have a really great dev infra team. They built something called Isaac that's basically a wrapper on Claude Code and Codex and lets you use them either on the web in sandboxes or just on your dev machine or on your laptop. And then they were adding all kinds of stuff there, and we saw all the more advanced engineers building their own workflows with tons of agents and building their own UIs on top of that. And then the other one was us building agents. We shipped this data science agent called Genie on the research team, which I co-lead. We also built a lot of internal ones for various things, and then we have all the customer ones, and all of them were running into this thing of like, oh, I need to switch model and harness every few months. Plus the agent is completely useless if you can't share sessions with someone and have history and have search and all this layer on top for collaboration. I thought a bit about it from both contexts, and at first people thought it was weird, like why are you doing coding agents and custom agents in the same thing? But I said it's basically the same problems, and you just want to build the stuff that lets you deliver the agent, maybe control it if you care about security, and make it portable across things. And then we prototyped some things as experiments. We said, yeah, actually we can make it work, and then we built it for real.

I'm wondering if this kind of, let's call it, architecture maps to anything in your careers in the past. You know, like I always think about how a lot of things actually just tie back to operating systems. A lot of operating systems tie back to databases, or the other way around.

Do we know of any other comments or alternative viewpoints?

Yeah.

I mean, the productivity you gain from all this coding agent stuff is amazing, but have you heard of cloud?

Was it the thing we were working on with the sandboxes, or was this before that?

Okay, so you were...

And so, I was approaching from a very different angle. I wanted to, 'Hey, we're going to have cloud sandboxes that actually don't shut down. You can get one very quickly, but not just for running agentic sessions. It's actually also for running development.' So I was actually personally building that that week, and through building that, I ran into all these issues. And then I wrote actually a document, 'Here's my wish list of what the actual environment should do.' And I think he actually ended up implementing almost every single one of them.

Yeah, I remember Reynold saying, because my first prototype of this had just chats with your agent, and he said, 'I have to be able to open a shell, like my own shell, and list files and tail them and stuff.' So I was...

Is this an SSH into a mainframe?

Yeah, actually it has that. Tailing my logs.

We did that. Yeah, we had a lot of engineers building their own vibe coding setup. But then the other thing they all said is, 'Hey, I built something that's amazing for me, but no one else on the team can use that because I don't have a server to collaborate.' And this is why we tried to set up Omnigen, so you can have a server and have the security set up in there. So you know, log in with Google or whatever and actually securely share stuff. And that's why we've seen a lot of other agents hit things like, people think they prototyped an awesome agent, but it's not allowed to connect to some really important data because of the security team. So yeah.

Yeah. At this point, so for those watching along on YouTube, we're going to bring up an image of the structure here and we can talk through a little bit of the architecture. I think I just want to have people understand, because when we're talking about software, it can be very abstract, and here's actually what we're talking about. You've worked out in open source this entire platform basically, and there's a runner component and server component with a sort of uniform API that you've figured out. Any other sort of element, and obviously you can plug in all these persistence layers and compute layers. This is a whole cloud. This is Agent Cloud.

Yeah, it's got these components to work with it. A lot of the action happens on the machine where you deploy your agent to. So whatever you've got on there, you can run. But yeah, I think it's sort of the minimal thing you want to have for hosted collaborative agents and to have that server. And one of the reasons we open sourced it is anyone building agents, this gives them an app they can start with and customize, which we were seeing in Databricks too. Someone would make a nice agent app and then other teams would ask, 'Oh, can I just use yours for my agent?'

Yeah, you need to basically... people want to take something that works and fork it, and you might as well have something open source. Which also was another question that is interesting for Databricks: what do you choose to open source, what do you choose to make proprietary? I mean this goes back to Spark, right?

Yeah. One of the reasons to open source something is if you think it's a layer that will actually have some network effect. It'll benefit from many people collaborating on it. So for example, with Spark, when Spark came out, we also focused a lot on letting you have libraries on top. So they used to be different distributed computing engines for machine learning and graph computation. We said they should all be libraries that you can compose, and we made it super easy to add connectors to data sources too. And then we benefit because we don't have the time to write connectors to a thousand different databases and file formats. But we can just use the ones people make, and of course they benefit from joining this thing. So that's one way to think about it. Another way is, I can imagine if our thing wasn't open, we had some kind of agent hosting thing, but it's not open, and then there's an open one. Which one's going to win in the long run? So here, because there is this benefit from people writing integrations, it'll be that. And then there are other things that you just can't even deliver as open source that are things the company does. Like for example, how do you make sure your streaming jobs or your lake-based database doesn't lose all your data at night? Well, that requires an operational team that's going to sit there. It has to be a service. So we want to make sure as a company we're really good at those infra services, and then we're as open as we can in terms of what you build on top.

Yeah, Saturday.

Let's see what's going on.

Yeah, you can look at the merge lines. I actually asked someone this morning about the... 400 merges already? Yeah, I think around half are not from my team. But for example, someone added support for running it on Kubernetes. People added many cloud sandboxes. So this can launch a cloud sandbox and run your agent in there, which is great for sharing too, because it's not on your laptop. So yeah, many startups have put those in and we expect to see more of them. We also have more agent harnesses already, Cursor, CLI, and anti-gravity.

Yeah, that's all beautiful. I feel like the last time this happened, there was the rise of the modern data stack. I don't know if it was that useful. I'm actually kind of curious in your postmortem. I think most people will agree that it is finally dead, but maybe this arises to a new modern AI stack that does the same thing.

I mean, I think the modern data stack was a pretty useful thing, probably even up until this day. For the audience who don't actually understand the history, the modern data stack is effectively decomposed into: you need a layer to ingest the data in, you need a layer to transform your data, and then you need a layer to maybe visualize your data. And all of this runs on some sort of data warehouse, or later on, as we're doing, data lakehouse. I think those concepts are all very powerful and very useful. It sort of enabled a lot of workloads. What people eventually run into is a question of unification and consolidation. It's, 'Hey, do you really need to chop all of this into different pieces and work with so many different vendors and platforms in order to get a very simple visualization done?' So over time everybody started realizing that customers are pushing us, so we start building more and more capabilities and trying to consolidate. And at the end of the day now, customers don't have to worry about having to hook up five different systems in order to produce a chart. But I think honestly something like this is probably happening in how many different frameworks do you want to hook up together in order to do a very simple agent.

Just to be clear, I would say the core of this is this common API on top of all the harnesses. So the API is basically like you've got an agent session and you can send in a message or a file. Basically that's what you can send in, and then you get out these streams as it's streaming text or as it's doing tool calls. Or the other thing you can send in is you can tell it to cancel or turn. So that's the API. Now, the thing we did is we could get you that on top of Claude Code running in a terminal, Codex, Phi, OpenAI SDK, all that stuff. We map them all to that same interface. So that is something that you'd have to maintain yourself if you built your own agent orchestrator. And then whenever Claude changes its API, you got to tweak your thing or it's going to lose some messages. So that's the thing that's valuable to maintain. Then on top of that, we built a few apps. I think we built a pretty cool UI, and we built the security and control piece which I'm excited about. But it's that common interface. So it doesn't try to be a stack. And in fact, you could plug in your own UI on top of this server. That's one of the use cases we care a lot about because we want to use this in our own products.

Yeah, it should be everywhere. I think one of those things that is really interesting to me is, well first of all, I'll endeavor to do everything and not call it the modern AI stack because I think we have a name. But yes, one of the first people that told me about compute sandboxing was Nikita from Neon. Because a lot of people think about Neon as, well, it's serverless Postgres with the separation of compute and storage and instant branching and all those things. But actually, every database company is also a compute company. And so he was actually showing me his whole sandboxing solution. I don't think he ever launched it.

Yeah, so this is when you run a coding sandbox. Like if I use the dev infra we had internally at Databricks, there's like many, many tens of gigabytes of data just for all the source code and artifacts and stuff that I built, and I want that to come back next time. So but yeah.

Before the show, we were talking about some statistics that might be surprising at the adoption. It could be internal, could be external, whatever comes to mind. Just to impress people the scale this is happening.

So on the analytics side, I think we launch maybe 50 or 60 million virtual machines a day across our three clouds. So we're one of the biggest compute orchestrators out there. So that's for sure for CPU compute. And all of those process exabytes of data. I joked about, depending on which time zone you are, typically before you have breakfast, Databricks would have processed exabytes of data already on that day. And on Neon, it's actually pretty interesting too, it's launching I think 13 million databases a day now.

Yeah, that to me that was like a big... What do you mean?

And then a lot of those were thanks to agents and branching experimentation. Because we made it so easy and so quick, thanks a lot to Nikita's team, to launch databases. So it's changing the way people use databases.

Yeah. Okay, we're going to go into more database talk in a bit, but I want to make sure we close up anything on Omnigents. You mentioned you're excited about the security and control side. A lot of companies are figuring that out right now, as well as the spend side. What have you found there?

Yeah, so I spent quite a bit of time talking to internal users, developers, security team, managers, and also lots of customers. And there's a few things. First of all, one thing that immediately became obvious for security is there's this tension between usability and security. The way a lot of coding agents work today have very basic things like you can tell me which tool patterns are allowed or disallowed. It's like yes or no. But that puts you in a very tough spot. So just as an example, should my agent be able to read some confidential documents? Or should it be able to install new packages from NPM, which maybe it's compromised? Yes or no. Maybe I want to allow it. Should my agent be able to publish stuff to the company website? Well, if I'm using it on the code on the website, yes. But should it be able to do both? So it can grab a confidential document and be prompt injected and leak it? Probably not. So the thing we decided we need is stateful, or what we call contextual policies, where you keep track of the state of that session. It's not like, is it allowed to push to the marketing site or not? But like, hey, if it did a risky thing, like it installed a one-day-old package from NPM, or it read a thousand confidential docs, then no. Then don't do it. Otherwise, maybe it's okay. That's one example of moving that trade-off. So it's both more secure and more useful by having a more powerful engine essentially. This requires tracking sessions. The other piece that was interesting is there are these very low-level events it's doing and you want some libraries on top that parse them. Like for example, we have an MCP server on Google Drive internally. It's got 60 API calls. Like how do I know which of those will share a document with stuff on the internet and which ones won't? It's annoying. So we designed in Omnigen the policy layer so that it's functions and you can have libraries. Someone can make something that maps the low-level events to high-level ones, and then you write a policy about the high-level things that came out. And that was related to the Panther acquisition. Panther will help with that. Panther is kind of a similar idea on the event processing side and it's Python-based versus a weird custom language. This is sort of more real-time. Those things are happening. So yeah, but these are the cool things. I think the contextual or stateful part and then the way it can be libraries, and that was another reason to make it open source because others will write libraries and our customers can use them. And the final thing, because it's stateful, one of the states we track is how much you've spent in that session. So I can literally say, 'Okay, launch a sub-agent to do this and cap it to spending $5. Ask me for permission if it needs more.' And because we're counting that within that session, it'll pop up and tell me, 'Okay, you spent $5. Do you want to go on?'

So for more context here, Matei spent the last five years, a lot of his time was architecting Unity Catalog at Databricks, which is the governance layer for data.

That's right. Yeah.

And it's sort of combining expertise at that layer together with all the AI governance here.

Yes. But I also spent a lot of time being annoyed by coding agents and getting burned. And also as the CTO, I don't want to end up on the front page as like, I installed some weird NPM package and leaked all the code. So I'm especially paranoid, but also I have very little time. So I don't want to sit there approving, 'Do you want to run a 20-line bash script? Yes or no?' So that's why I spent a lot of time figuring out how can I make it as safe as possible and not annoying.

Yeah. Is safety and let's call it security a bigger concern than token maxing or token budgets? Which one is like...

Oh yeah, they're both there. I guess it depends on the type of company you are. So I think some companies, the budget is limited and they really care about that. Or I mean, you can be Uber and still be concerned. Yeah, oh yeah, totally. For us, security is absolutely critical as a cloud provider. It's the most important thing. And token maxing, we're not so worried about it yet, but I've seen... for example, I talked to some consulting companies. They have like 100,000 employees who are all coding for customers. If those each spend an extra $1,000 a month, that's not fun. We have like only a few thousand engineers.

What's the policy at Databricks? Is it just unlimited or...

Unlimited, but we do... we use our own product to analyze the traces and stuff, and we have a team that's looking to optimize and to see if anyone's doing something weird. And we actually had some really cool insights just from analyzing current traces, like which models are better at say Rust versus TypeScript or whatever. So yeah, at least in our code base.

Yeah, amazing. Obviously, I have to ask the token maxing question. Obviously I think it's a key thing, but yes, security and control above that, and figuring out a sane layer that you can have some autonomy, but not too much.

Yeah, and we want to make it super easy. As an engineer, you should set the thing. So in Omnigen, you can ask your agent to set up a policy on itself to do this.

If there's anything I should be showing, I don't see it on the GitHub, but you know, this is...

It's in the docs there. So you can look at it later. Just look in the docs on contextual policies if you want to see.

I just like to show people...

Policies. Yeah, if you want to follow up on this, this is exactly where to look, right? Yeah. Uh yeah, and the story of these is I just wrote a doc with like 10 ideas for things before as you were working on them. That was like my wish list of things people asked, and I told the team, 'Hey, can you do at least five of these for the launch?' And then they just got back with all of them.

Oh wow.

So you can come up with more, but then some of them are just meant to be examples. Really, you can intercept any event the agent is making, and you can then either block or force it to ask the user or allow, and you can update state to keep track of stuff.

Yeah, because ultimately, I think of you as a systems designer, you let people plug in, right? That's the whole modus operandi of what you do.

Yeah, and we care a lot about also composability. Like, can someone else write a library that others use, which this is meant to...

There's also a batteries-included philosophy here, probably very similar to how you did Spark, which is you could just start using.

Yeah, that's right. It has to be good out of the box at certain things, and then you can build your own things on top. In Spark, if you just want to read a table or do an aggregation, it should be awesome out of the box.

Yeah. People want to catch up on Omnigen, they should watch your keynote, they should go through the GitHub and the docs. If they wanted to contribute or they want to build on this ecosystem, where would you call out as the most high-leverage places to get involved?

Yeah, do get involved in the Discord and then GitHub. Our team is there monitoring, and some of the things people ask for we just build ourselves. Some of them, we're collaborating with them to build that. And also tell us how you would like to use that, because I think especially for developers, everyone wants it to work their own way. And a really good developer tool, you have to hear the feedback on all the ways and figure out the abstractions and how to let people customize. So we love to hear if you think, hey, I don't want it to work this way, tell us. We really just want to get that compatibility layer across agents and then let you do stuff on top.

Yeah. Is there any... in terms of the startup side, I'm a founder, I see an opportunity, I want to get in front of you. What's your request for a startup? Like, I wish someone was working on this.

Oh, for a startup?

Yeah. Like, you got your own startup, it's doing well, but if you weren't working on your own startup, what is obvious that you should... You advise many startups too, obviously.

I mean, I do think just as a company with a lot of engineers, anything that helps me make sense of how people are using coding agents and spend, but also quality, or like, you should add this skill, or you should write this thing, or your agents are really horrible at tasks involving this service, or go spend time. That would be nice.

Yeah, the closest I found is this team Get AI. They started with, we will just do code and human attribution, but they're basically building the analytics layer on top of that. I do think there are a bunch of... Artificial Analysis is obviously doing super well with their stuff. So there will be people, I think this is the domain of consultants first, but then people will actually build software that has the management plane for coding agents.

Yeah, I think there'll be a lot of insights there. You have it in other areas.

Okay, well, and then the other big thing is your Dream Engine. If you want to tell the story of OLTP. And our background was, I'm going to make people listen to our Ankur Goyal episode where we talked about SingleStore, HTAP, and all that history.

To be the database podcast that I'm forcing people to learn your databases, guys. You cannot vibe code with just markdown files.

Yeah, which is for people... Elasticsearch is like a big... Yeah, so some of them actually get into Elasticsearch for log analysis. A lot of our customers obviously get into Databricks to run more sophisticated things. And there's this term called CDC. CDC, change data capture.

You know, my explanation is like everybody became a $5 billion company just doing CDC.

I mean, there's all sorts of tricks that you can do, like you add in some versioning or whatever.

Which, like by the way, every database person ever has always dreamed about this.

Yeah.

Yeah, I think in the way that humans are generally intelligent and want to have the ability and access to query anything, even while they do the work, they also need history and they need context, and where else do they get context? That's an analytical workload.

Yeah, and I remember when we had incidents with our databases, and the engineer said, well, I can't just run a giant query on it to see what's going on because that's going to bring down the database and hurt it even more. Like that's the kind of stuff that this gets rid of, because you spin up a whole separate fleet of machines that's doing the analytics. You're not overloading the main database that's still trying to serve stuff.

Yeah. So, this has been a dream for a while. What had to get done in order to get to today? Like, I feel like you have announced variants of this several times, but it wasn't as clear as LTab.