From Lakehouse to Agent Cloud — Matei Zaharia and Reynold Xin of Databricks

Thanks for having us.

Yeah, thanks so much.

Yeah, it was a little meet-up at Berkeley, I think. We put together—we did these tutorials and yeah, just teach people Spark.

What do you think? I mean, I think among our group of founders, it was clear that I think he'd be the best at this. And yeah, it turned out great. I mean, he's ramped up on so many topics running a company. He would just go in and study it and talk to all the experts. Even if you can't hire the person, learn enough about finance and sales and whatever it was. And you know, go from there.

Yeah. There were actually a couple of converging lines, which I think is a good sign that you need something new. So, on the one hand, there's all the coding agent work internally. We have a really great dev infra team. They built something called Isaac that's basically a wrapper on Claude Code and Codex and lets you use them either on the web in sandboxes or just on your dev machine or on your laptop or whatever. And then they were adding all kinds of stuff there and we saw all the more advanced engineers building their own workflows with tons of agents and building their own UIs and stuff on top of that. And then the other one was that us building agents—we shipped this data science agent called Genie on the research team which I co-lead. We also build a lot of internal ones for various things and then we have all the customer ones and all of them were running into this thing of like, oh I need to switch model and harness and so on every few months. Plus the agent is completely useless if you can't share sessions with someone and have history and have search and all this layer on top of it for collaboration. I thought a bit about it from both contexts and at first people thought it was weird and like why are you doing coding agents and custom agents in the same thing but I said it's basically the same problems and you just want to build the stuff that lets you deliver the agent, maybe control it if you care about security, and make it portable across things. And then we prototyped some things as experiments. We said, yeah, actually we can make it work and then we sort of built it for real.

A lot of operating systems tie back to databases, so or the other way around. So the thing I do think it ties a lot to is network protocols, you know, internet protocol. We also did stuff with data sharing also which is probably—most viewers probably won't know unless they—

Open sharing. Yeah, so it's like you have a company, you maintain some kind of table—like let's say Walmart or something. They have the inventory and what's been sold in each store. And then you also have suppliers and they would love to produce more things and ship them exactly the moment you need them. So they would love real-time access to your table. So instead of sending emails around or Excel sheets or phone calls, why can't you share a view of that table in real-time with them? Then they query, they join it with their data and they decide what to send. So it's one of these things where you might ask—today since we can vibe code anything so fast, why do we even need to design protocols or APIs or software? Why can't you just vibe code things on demand? But actually for this type of interoperability where multiple parties that are moving at different speeds are building stuff and you still want some layer on top to coordinate, you do want to design it and build it. So it reminds me of that—agents talking to each other and users talking to agents and tools.

It was crazy to me. Was the thing we were working on the sandboxes or was this before that?

Yeah, I remember Reynold saying—because my first prototype of this had just chats with your agent—and he said, I have to be able to open a shell, like my own shell and list files and tail them and stuff. So I was—

Yeah, actually it has that.

We did that. Yeah. We had a lot of engineers building their own vibe coding setup. But then the other thing they all said is—hey, I built something that's amazing for me, but no one else on the team can use that because I don't have a server to collaborate. And this is why we tried to set up Omnigents so you can have a server and have the security set up in there. So, you know, log in with Google or whatever and actually securely share stuff. And that's why we've seen a lot of other agents hit things like—people think they prototyped an awesome agent, but it's not allowed to connect to some really important data or whatever because of the security team. So, yeah.

Yeah, it's got these components to work with it. A lot of the action happens on the machine where you deploy your agent to. So, whatever you've got on there, you can run. But yeah, I think it's sort of the minimal thing you want to have to host collaborative agents and to have that server. And one of the reasons we open sourced it is anyone building agents, this gives them an app they can start with and customize, which we were seeing in Databricks too—someone would make a nice agent app and then other teams would ask, oh, can I just use yours for my agent?

Yeah, you need to basically—people want to take something that works and fork it, and you might as well have something open source.

Yeah. One of the reasons to open source something is if you think it's a layer that will actually have some network effect. It'll benefit from many people collaborating on it. So, for example, with Spark—I don't know if you know—but when Spark came out, we also focused a lot on letting you have libraries on top. So they used to be different distributed computing engines for machine learning and graph computation. We said they should all be libraries that you can compose, and we made it super easy to add connectors to data sources too. And then we benefit because we don't have the time to write connectors to a thousand different databases and file formats. But we can just use the ones people make, and of course they benefit from joining this thing. So that's one way to think about it. Another way to think about it is—I can imagine our thing wasn't open. We had some kind of agent hosting thing, but it's not open, and then there's an open one. Which one's going to win in the long run? So here, because there is this benefit from people writing integrations, it'll be the open one. And then there are other things that you just can't even deliver as open source that are things the company does. Like, for example, how do you make sure your streaming jobs or your lake-based database doesn't lose all your data at night? Well, that requires an operational team that's going to sit there. There's no way—it has to be a service. So we want to make sure as a company we're really good at those infrastructure services, and then we're as open as we can in terms of what you build on top.

Yeah, Saturday. Yeah, so someone—let's see what's going on. Yeah, you can look at the merge lines. I actually checked some metrics this morning about the—

Yeah. I think quite—I would guess around half are not from my team. But for example, someone added support for running it on Kubernetes. People added many cloud sandboxes. So this can launch a cloud sandbox and run your agent in there, which is great for sharing too because it's not on your laptop and someone's running code on there. So yeah, many startups have put those in and we expect to see more of them. We also have more agent harnesses already—Cursor, CLI, and Anti-Gravity also.

Just to be clear, I would say the core of this is this common API on top of all the harnesses. So the API is basically—you've got an agent session and you can send in a message or a file. Basically, that's what you can send in and then you get out these streams as it's streaming text or as it's doing tool calls. Or the other thing you can send in is you can tell it to cancel or turn. So that's the API. Now, the thing we did is we could get you that on top of Claude Code running in a terminal, Codex, Phi, OpenAI SDK, all that stuff. We map them all to that same interface. So that is something that you'd have to maintain yourself if you built your own agent orchestrator. And then whenever Claude changes its API, you got to tweak your thing or it's going to lose some messages. So that's the thing that's valuable to maintain. Then on top of that, we built a few apps. I think we built a pretty cool UI and stuff, but that's—and we built the security and control piece which I'm excited about. But it's that common interface. So we—it doesn't try to be a stack. And in fact, you could plug in your own UI on top of this server. That's one of the use cases we care a lot about because we want to use this in our own products.

Yeah.

Yeah, so this is when you run like a coding sandbox. Like if I use the dev infra internally at Databricks, there's like many tens of gigabytes of data just for all the source code and artifacts and stuff that I built and I want that to come back next time. So but yeah.

Yeah, to me that was like a big—

Yeah, so I spent quite a bit of time talking to internal users, developers, security team, managers, and also lots of customers. And there's a few things. First of all, one thing that immediately became obvious for security—there's this tension between usability and security. And the way people do—a lot of coding agents today have very basic things like you can tell me which tool patterns are allowed or disallowed or whatever. It's like yes or no. But that puts you in a very tough spot. So, just as an example—should my agent be able to read some confidential documents? Or let's say should it be able to install new packages from NPM, which you know, maybe it's compromised? Yes or no. Maybe I want to allow it. Should my agent be able to publish stuff to the company website? Well, if I'm using it on the code on the website, yes. But should it be able to do both? So it can grab a confidential document and be prompt injected and leak it? Probably not. So the thing we decided we need is stateful or what we call contextual policies, where you keep track of the state of that session. It's not like is it allowed to push to the marketing site or not? But like, hey, if it did a risky thing—like it installed a one-day-old package from NPM, or it read a thousand confidential docs—then no. Then don't do it. Otherwise, maybe it's okay. That's one example of moving that trade-off. So it's both more secure and more useful by having a more powerful engine essentially. This requires tracking sessions. The other piece that was interesting there is—there are these very low-level events it's doing and you want some libraries on top that parse them. Like for example, we have an MCP server on Google Drive internally. It's got 60 API calls. Like how do I know which of those will share a document with stuff on the internet and which ones won't? It's annoying. So we designed in Omnigents the policy layer so that it's functions and you can have libraries. Like someone can make something that maps the low-level events to high-level ones and then you write a policy about the high-level things that came out. And that was related to the Panther—Panther will help with that. Panther is kind of a similar idea on the event processing side and it's Python-based versus a weird custom language. This is sort of more real-time. Those things are happening. Yeah. So yeah, but these are the cool things. I think the contextual or stateful part and then the way it can be libraries—and that was another reason to make it open source because others will write libraries and we and our customers can use them. And the final thing—because it's stateful, one of the states we track is how much you spent in that session. So I can—I've asked an agent to debug something and it spent $500 because it decided to read a lot of log files and burn a lot of tokens. But I can literally say, okay, launch a sub-agent to do this and cap it to spending $5. Like ask me for permission if it needs more. And because we're counting that within that session, it'll pop up and tell me—okay, you spent $5. Do you want to go on?

That's right. Yeah. But I also spent a lot of time being annoyed by coding agents and getting bombs. And also as the CTO, I don't want to end up on the front page as—I installed some weird NPM package and leaked all the code. So I'm especially paranoid, but also I have very little time. So I don't want to sit there approving—do you want to run a 20-line bash script? Yes or no? So that's why I spend a lot of time figuring out—how can I make it as safe as possible and not annoying.

Oh, yeah, they're both there. I mean, I don't know. I guess it depends on the type of company you are. So I think some companies—the budget is limited and they really care about that.

Yeah, oh yeah, totally. For us, security is absolutely critical as a cloud provider. It's the most important thing. And token maxing, we're not so worried about it yet, but I've seen—like for example, I talked to some consulting companies. They have like 100,000 employees who are all coding for customers. If each of those spend like an extra $1,000 a month, that's not fun. We have like only a few thousand engineers.

Unlimited, but we do—we use our own product to analyze the traces and stuff and we have a team that's looking to optimize and to see if anyone's doing something weird. And we actually had some really cool insights just from analyzing current traces—like which models are better at say Rust versus TypeScript or whatever. So yeah, at least in our code base.

Yeah. Yeah, and we want to make it super easy. As an engineer, you should set the thing. So in Omnigents, you can ask your agent to set up a policy on itself to do this.

Put in the docs there. So you can look at it later. Just look in the docs on contextual policies if you want to see.

Policies. Yeah, if you want to follow up on this, this is exactly where to look, right? Yeah. Yeah. And the story of these is—I just wrote—you know, I wrote a doc with like 10 ideas for things before as you were working on them. Well, that was like my wish list of things people asked. And I told the team—hey, can you do like at least five of these for the launch? And then they just got back with all of them. So, oh wow. So you can come up with more, but then some of them are just meant to be examples. Really, you can intercept any event the agent is making, and you can then either block or force it to ask the user or allow, and you can update state to keep track of stuff.

Yeah. And we care a lot about composability. Like, can someone else write a library that others use? Which this is meant to—

Yeah, that's right. It has to be good out of the box at certain things, and then you can build your own things on top. You know, we don't want to do—but in Spark, if you just want to read a table or do aggregation, it should be awesome out of the box.

Yeah, do get involved in the Discord and GitHub. Our team is monitoring and some of the things people ask for we just built ourselves. Some of them, we're collaborating with them to build. And also tell us how you would like to use that because I think especially for developers—everyone wants it to work their own way and a really good developer tool—you have to hear the feedback on all the ways and figure out the abstractions and how to let people customize. So we love to hear—if you think, hey, I don't want it to work this way, tell us. We really just want to get that compatibility layer across agents and then let you do stuff on top.

Oh for a startup. I mean I do think—just as a company with a lot of engineers—anything that helps me make sense of how people are using coding agents and spend but also quality—or like you should add this skill or you should write this thing or your agents are really horrible at tasks involving this service or go spend time. That would be nice.

Mhm. Oh cool, yeah.

Yeah, I think there'll be a lot of insights there. You have it in other areas.

Replication—replicate them into the analytic systems.

Yeah, so some of them actually get into Elasticsearch for log analysis. A lot of our customers obviously get into Databricks to run more sophisticated things. And there's this term called CDC—

You know, my explanation is—everybody became a $5 billion company just doing CDC.

I mean there's all sorts of tricks that you can do—like you add in some versioning or whatever. But—

Which like by the way every database person ever has always dreamed about this.

Yeah, I think you know—in the way that humans are generally intelligent and want to have the ability and access to query anything. Even while they do the work, they also need history and they need context—and where else do they get context? That's an analytical workload.

Yeah, and I remember when we had incidents with our databases and the engineer said—well, I can't just run a giant query on it to see what's going on because that's going to bring down the database and hurt it even more. Like that's the kind of stuff that this gets rid of because you spin up a whole separate fleet of machines that's doing the analytics. You're not overloading the main database that's still trying to serve stuff.