One of the advantages we have is that once you can get the data in the right place, the AI models are becoming pretty good. The generic agents are fairly... I mean, Ali talked about AGI already here. They have pretty good reasoning capabilities. Actually, I think many of the traditional software will be sort of rewritten with this new paradigm, which is just get the data to be there and then let's slap some AGI on top. Magic will come out.
Yeah, but without the right data, you can't really do that.
Before we get into today's episode, I just have a small message for listeners. Thank you. We would not be able to bring you the AI engineering science and entertainment contents that you so clearly want if you didn't choose to also click in and tune into our content. We've been approached by sponsors on an almost daily basis, but fortunately enough of you actually subscribe to us to keep all this sustainable without ads, and we want to keep it that way. But I just have one favor to ask all of you. The single most powerful completely free thing you can do is to click that subscribe button. It's the only thing I'll ever ask of you, and it means absolutely everything to me and my team that works so hard to bring the In Space to you each and every week. If you do it, I promise you we'll never stop working to make this show even better. Now let's get into it.
Matei Zaharia from Databricks. Welcome to In Space.
Thanks for taking time out. You have your Databricks Data AI Summit going on. You were just telling me how the first summit that you guys ran was just 50 people.
Yeah, it was a little meet-up at Berkeley, I think. We put together... we did these tutorials and yeah, just teach people Spark.
Yeah. You know, obviously now it's like I think the headline number is like 100,000 people around the world, 30,000 in person. It's a crazy community. Well, I mean I just saw the keynote.
Ali is just... Did you know, was it obvious back when that Ali would be such a great CEO? Like he's a great presenter.
I mean, I think among our group of founders, it was clear that I think he'd be the best at this. And yeah, it turned out great. He's ramped up on so many topics running a company. He would just go in and study and talk to all the experts. Like, even if you can't hire the person, learn enough about finance and sales and whatever it was, and go from there.
I mean, he's obviously very high IQ and has very high EQ, but Ali today is quite different from Ali from like 10 years ago. I think there's a lot of work that he put in to get to this point.
Yeah, I mean, to me the most appealing thing about him is that he's funny.
It's hard to make jokes about data, about serious topics and security and what have you.
Oh yeah, that's for sure.
Yeah. So, you guys launched a whole bunch of things. I'll just name-check briefly the stuff because we're not going to cover everything. Omnigents, your baby. El Tap, your baby. Dream Engine. We're also going to cover Genie, Customer League. You acquired Panther, Open Sharing, and there's Unity AI Gateway. A lot of these I think are things that you would expect a Databricks to do. It's part of the road map. Everyone in your category has similar things. But I think probably the two of you are leading the two most unique and differentiated initiatives in the landscape. Maybe we'll start with Omnigents and then we'll go into it.
I do think that a lot of people are exploring this sort of meta harness concept. What led you to it?
Yeah. There were actually a couple of converging lines, which I think is a good sign that you need something new. So, on the one hand, there's all the coding agent stuff internally. We have a really great dev infra team. They built something called Isaac that's basically a wrapper on Claude Code and Codex and lets you use them either on the web in sandboxes or just on your dev machine or on your laptop. And then they were adding all kinds of stuff there, and we saw all the more advanced engineers building their own workflows with tons of agents and building their own UIs on top of that. And then the other one was us building agents. We shipped this data science agent called Genie on the research team, which I co-lead. We also built a lot of internal ones for various things, and then we have all the customer ones, and all of them were running into this thing of like, oh, I need to switch model and harness every few months. Plus the agent is completely useless if you can't share sessions with someone and have history and have search and all this layer on top for collaboration. I thought a bit about it from both contexts, and at first people thought it was weird, like why are you doing coding agents and custom agents in the same thing? But I said it's basically the same problems, and you just want to build the stuff that lets you deliver the agent, maybe control it if you care about security, and make it portable across things. And then we prototyped some things as experiments. We said, yeah, actually we can make it work, and then we built it for real.
I'm wondering if this kind of, let's call it, architecture maps to anything in your careers in the past. You know, like I always think about how a lot of things actually just tie back to operating systems. A lot of operating systems tie back to databases, or the other way around.
So the thing I do think it ties a lot to is network protocols, you know, internet protocol. We also did stuff with data sharing, which probably most viewers won't know unless they... Yeah, open sharing is the term. So it's like you have a company, you maintain some kind of table, like let's say Walmart or something. They have the inventory and what's been sold in each store. And then you also have suppliers, and they would love to produce more things and ship them exactly the moment you need them. So they would love real-time access to your table. So instead of sending emails around or Excel sheets or phone calls, why can't you share a view of that table in real-time with them? Then they query, they join it with their data, and they decide what to send. So it's one of these things where you might ask, today since we can vibe code anything so fast, why do we even need to design protocols or APIs or software? Why can't you just vibe code things on demand? But actually for this type of interoperability, where multiple parties that are moving at different speeds are building stuff and you still want some layer on top to coordinate, you do want to design it and build it. So it reminds me of agents talking to each other, and users talking to agents and tools.
Do we know of any other comments or alternative viewpoints?
I think by the way, we had a debate exactly about this with Matei a lot. And I think around the time we decided to do this thing, I was telling Matei, 'Hey, it just happened to be there's a particular week that I was coding non-stop from the moment I woke up to like the moment I went to bed.' I was looking at my cloud sessions, my Codex sessions, and one of the things particularly annoying was having to keep my laptop open. I was actually driving to a doctor's appointment, and I remember because I wanted to make sure the whole thing continues working. By the way, it's so comforting to hear you say that because I'm like, I don't know if I'm a clown doing this or not. Honestly, I was driving and I was tethering my laptop to my phone, keeping it on the side, and whenever I hit a red light, I started looking at what's going on on my laptop. And I just felt that was ridiculous.
It felt like we went back to the dark ages of programming.
I mean, the productivity you gain from all this coding agent stuff is amazing, but have you heard of cloud?
Was it the thing we were working on with the sandboxes, or was this before that?
And so, I was approaching from a very different angle. I wanted to, 'Hey, we're going to have cloud sandboxes that actually don't shut down. You can get one very quickly, but not just for running agentic sessions. It's actually also for running development.' So I was actually personally building that that week, and through building that, I ran into all these issues. And then I wrote actually a document, 'Here's my wish list of what the actual environment should do.' And I think he actually ended up implementing almost every single one of them.
Yeah, I remember Reynold saying, because my first prototype of this had just chats with your agent, and he said, 'I have to be able to open a shell, like my own shell, and list files and tail them and stuff.' So I was...
Is this an SSH into a mainframe?
Yeah, actually it has that. Tailing my logs.
And also, another thing I asked was, I still use Cursor for the sole purpose of rendering markdown files. 'Give me a way to see my markdown files and render them properly. I don't need a separate tool anymore.' Yeah, I think you also built that in.
We did that. Yeah, we had a lot of engineers building their own vibe coding setup. But then the other thing they all said is, 'Hey, I built something that's amazing for me, but no one else on the team can use that because I don't have a server to collaborate.' And this is why we tried to set up Omnigen, so you can have a server and have the security set up in there. So you know, log in with Google or whatever and actually securely share stuff. And that's why we've seen a lot of other agents hit things like, people think they prototyped an awesome agent, but it's not allowed to connect to some really important data because of the security team. So yeah.
Yeah. At this point, so for those watching along on YouTube, we're going to bring up an image of the structure here and we can talk through a little bit of the architecture. I think I just want to have people understand, because when we're talking about software, it can be very abstract, and here's actually what we're talking about. You've worked out in open source this entire platform basically, and there's a runner component and server component with a sort of uniform API that you've figured out. Any other sort of element, and obviously you can plug in all these persistence layers and compute layers. This is a whole cloud. This is Agent Cloud.
Yeah, it's got these components to work with it. A lot of the action happens on the machine where you deploy your agent to. So whatever you've got on there, you can run. But yeah, I think it's sort of the minimal thing you want to have for hosted collaborative agents and to have that server. And one of the reasons we open sourced it is anyone building agents, this gives them an app they can start with and customize, which we were seeing in Databricks too. Someone would make a nice agent app and then other teams would ask, 'Oh, can I just use yours for my agent?'
Yeah, I think we had like five or six different agentic frameworks built by every different team. They all do more or less the same thing.
Yeah, you need to basically... people want to take something that works and fork it, and you might as well have something open source. Which also was another question that is interesting for Databricks: what do you choose to open source, what do you choose to make proprietary? I mean this goes back to Spark, right?
Yeah. One of the reasons to open source something is if you think it's a layer that will actually have some network effect. It'll benefit from many people collaborating on it. So for example, with Spark, when Spark came out, we also focused a lot on letting you have libraries on top. So they used to be different distributed computing engines for machine learning and graph computation. We said they should all be libraries that you can compose, and we made it super easy to add connectors to data sources too. And then we benefit because we don't have the time to write connectors to a thousand different databases and file formats. But we can just use the ones people make, and of course they benefit from joining this thing. So that's one way to think about it. Another way is, I can imagine if our thing wasn't open, we had some kind of agent hosting thing, but it's not open, and then there's an open one. Which one's going to win in the long run? So here, because there is this benefit from people writing integrations, it'll be that. And then there are other things that you just can't even deliver as open source that are things the company does. Like for example, how do you make sure your streaming jobs or your lake-based database doesn't lose all your data at night? Well, that requires an operational team that's going to sit there. It has to be a service. So we want to make sure as a company we're really good at those infra services, and then we're as open as we can in terms of what you build on top.
I mean, speaking from a benefits perspective, I think we're already seeing pull requests and the ecosystem integration, even though it was only released on Saturday.
Let's see what's going on.
Yeah, you can look at the merge lines. I actually asked someone this morning about the... 400 merges already? Yeah, I think around half are not from my team. But for example, someone added support for running it on Kubernetes. People added many cloud sandboxes. So this can launch a cloud sandbox and run your agent in there, which is great for sharing too, because it's not on your laptop. So yeah, many startups have put those in and we expect to see more of them. We also have more agent harnesses already, Cursor, CLI, and anti-gravity.
Yeah, that's all beautiful. I feel like the last time this happened, there was the rise of the modern data stack. I don't know if it was that useful. I'm actually kind of curious in your postmortem. I think most people will agree that it is finally dead, but maybe this arises to a new modern AI stack that does the same thing.
I mean, I think the modern data stack was a pretty useful thing, probably even up until this day. For the audience who don't actually understand the history, the modern data stack is effectively decomposed into: you need a layer to ingest the data in, you need a layer to transform your data, and then you need a layer to maybe visualize your data. And all of this runs on some sort of data warehouse, or later on, as we're doing, data lakehouse. I think those concepts are all very powerful and very useful. It sort of enabled a lot of workloads. What people eventually run into is a question of unification and consolidation. It's, 'Hey, do you really need to chop all of this into different pieces and work with so many different vendors and platforms in order to get a very simple visualization done?' So over time everybody started realizing that customers are pushing us, so we start building more and more capabilities and trying to consolidate. And at the end of the day now, customers don't have to worry about having to hook up five different systems in order to produce a chart. But I think honestly something like this is probably happening in how many different frameworks do you want to hook up together in order to do a very simple agent.
Just to be clear, I would say the core of this is this common API on top of all the harnesses. So the API is basically like you've got an agent session and you can send in a message or a file. Basically that's what you can send in, and then you get out these streams as it's streaming text or as it's doing tool calls. Or the other thing you can send in is you can tell it to cancel or turn. So that's the API. Now, the thing we did is we could get you that on top of Claude Code running in a terminal, Codex, Phi, OpenAI SDK, all that stuff. We map them all to that same interface. So that is something that you'd have to maintain yourself if you built your own agent orchestrator. And then whenever Claude changes its API, you got to tweak your thing or it's going to lose some messages. So that's the thing that's valuable to maintain. Then on top of that, we built a few apps. I think we built a pretty cool UI, and we built the security and control piece which I'm excited about. But it's that common interface. So it doesn't try to be a stack. And in fact, you could plug in your own UI on top of this server. That's one of the use cases we care a lot about because we want to use this in our own products.
Yeah, it should be everywhere. I think one of those things that is really interesting to me is, well first of all, I'll endeavor to do everything and not call it the modern AI stack because I think we have a name. But yes, one of the first people that told me about compute sandboxing was Nikita from Neon. Because a lot of people think about Neon as, well, it's serverless Postgres with the separation of compute and storage and instant branching and all those things. But actually, every database company is also a compute company. And so he was actually showing me his whole sandboxing solution. I don't think he ever launched it.
So our sandbox solution, the reason we could have built it so quickly was because we realized if you just take the actual lakehouse architecture and remove the database from it, by the way, it was coming from you. Now there are some differences. For example, the ones that support this particular workload, it's important to have local persistence because you want your state to persist. Your libraries, you don't have to install your library every time, right? Whereas the Neon architecture, because of the separation of storage from compute, you don't need persistent local disk. So there's some differences. But at the end of the day, yeah...
Yeah, so this is when you run a coding sandbox. Like if I use the dev infra we had internally at Databricks, there's like many, many tens of gigabytes of data just for all the source code and artifacts and stuff that I built, and I want that to come back next time. So but yeah.
Before the show, we were talking about some statistics that might be surprising at the adoption. It could be internal, could be external, whatever comes to mind. Just to impress people the scale this is happening.
So on the analytics side, I think we launch maybe 50 or 60 million virtual machines a day across our three clouds. So we're one of the biggest compute orchestrators out there. So that's for sure for CPU compute. And all of those process exabytes of data. I joked about, depending on which time zone you are, typically before you have breakfast, Databricks would have processed exabytes of data already on that day. And on Neon, it's actually pretty interesting too, it's launching I think 13 million databases a day now.
Yeah, that to me that was like a big... What do you mean?
