Hi. My guest today needs almost no introduction in the data world. He created Apache Spark, co-founded Databricks, and is now its CTO. He's also a professor at UC Berkeley and behind massive projects like MLflow, Delta Lake, DS Pipe, Unity Catalog, and more recently, Omnigen. Matei, great to have you here.
Great. Thanks for having me.
And of course, my famous co-host, Quentin.
Hey. Hey, Matei. Thanks for taking the time. I wanted to start with Omnigen because, I mean, we all were in pink, obviously, and that's what everyone is talking about. And can you tell us more what is it? And what problem does it actually solve? And also, I have to ask because I heard the origin story involves Reynold sending you a wish list before Christmas, and because he was tired of keeping his laptop open while live coding, but that's not the only—
Uh, it wasn't before Christmas. It was like three months ago that we started building this. Yeah. So, Omnigen is what we call a meta harness. It's basically an orchestrator and control layer on top of agents. You have many coding agents today that you can use. You build lots of custom agents. But then you want to work with a bunch of them together. And you also want certain features across all your agents like governance, control, or even just collaboration with people. If every agent and every app builds those in a different way, you just have a big mess. And you know, it's very hard to make things work. So, we built this layer that kind of wraps existing agents and lets you easily work with many of them, whether you're composing many agents to solve the same task, or you're trying to control them more centrally, or trying to give them some nice features like real-time collaboration with other humans and stuff like that. The main use case that we started with is for coding because all the developers we saw that really got into coding agents ended up building very custom ways to manage lots of agents. So it's an easy way for you to use any of the coding agents out there and create a collaborative workflow with your team and with multiple people as well and multiple agents. But we're also using it to build custom agents. It makes it really easy to build an agent and swap out the harness and swap out any subcomponents of it. So we think it's kind of the next layer up for working with AI basically.
It actually kind of started from a few different places. So first of all, there were all the engineers coding at Databricks and as I said, anyone who really got into coding agents was soon building their own way to manage lots of them. People were building their own desktop apps and stuff because they didn't like using a terminal. And also people were mixing different agents like Codex and Claude Code and so on. So we saw everyone's kind of solving the same problem. If we build a single way to manage these then they can share the work and they can build security once and build collaboration once. And that was one angle. And that's where Arnold was saying, well, I don't think you can do it unless I can also see the terminal version of the agent because if it's just a chat app like I'm missing too much. You know, sometimes I want to run stuff in a terminal. Well, lots of engineers had very different opinions on what they would like and that's how we came up with the idea of wrapping existing coding agents and building something around them instead of trying to replace them all with one new thing.
And then the other angle was building custom agents. So, there are all these teams building custom agents like Genie or internal agents, and they all ran into the exact same problems. Like, I built my agent, but now I need an app with sharing and collaboration. Now I need history. Now I want to try a different harness. Like, new GPT model came out, so I want to try that or whatever. So we thought, hey, actually, we could also build something there. This idea of a layer above the harnesses where you could change.
So, it means, let's say I'm using Claude Code or Codex on a daily basis, and then I want to switch—I'm using Claude, and I want to switch to Codex—then I can just stay on Omnigen, and I can just choose the new one, and I keep everything here like all my—
Yeah, you can move your session from one to the other. So, for example, a lot of the feedback we're getting from users is, this is great because, you know, when I finish my daily quota of Claude, I just take the same exact chat, and just switch the agent at the bottom, and now I can continue in Codex or vice versa. And the other thing you can do is you can use an orchestrator agent that uses multiple of these together. So, we actually include one called Poly, but you can also easily make your own or just tell it like go launch a sub-agent that's using Codex or whatever. So, for example, whenever I write code with this, I tell like, you know, use one agent to write the code, then use a different one to review it. Or if I have a bug, I just say, hey, like, let's kick off two or three of these and look for the bug. Because, you know, one of them is basically you just need to try a lot of ideas until you figure out what caused the bug. So, it's better to have models with different biases, and you'll find the bug faster that way.
And can it help also because I think like a commit team is based on what you want to do. Maybe you need a smaller model or a bigger one. So, could you also like split that into smaller—
Yeah, definitely. It's very easy to just put in the prompt for the orchestrator agent, you can tell it when to use different ones. And they are better at different things. Like for example, Claude is widely considered to be better at fine-end. GPT is like widely considered to be better at debugging. So you can write your own rules there. We're also now shipping what we call intelligent routing where we basically have trained some heuristics where there's an API you can call based on your prompt. It'll suggest which model and which agents to use. And we think that's going to get better over time, too. This pattern of orchestrating multiple agents is really going to pick up. Most people with coding agents, they start with one long chat session with one agent, and then they keep asking it things to like build this, now build that, now debug this. But that's actually very wasteful and not a very effective way to use the agent, because you just get a huge context. When the context window gets bigger, the model gets stupider because it has to pay attention to everything else, and you're paying more. Like you're paying for this million-length context window all the time. And then when it compacts it, you forget things.
Whereas if you kick off sub-agents and you give them tasks, they each have like a fresh start. They just do that. And then you can have this planner or orchestrator maybe above them. This move towards multi-agent, which you see in Claude Code and Codex too, actually makes that much easier to do routing on a per-task basis and to actually mix and match and get the best of all the models out there.
That's very interesting because as you mentioned, some developers used to stick with the same LLM and actually this is bad when you're reviewing the app with the same LLM that built this app. And how does Omnigen architecture work behind the scene? So I guess you have a server that's where we can store the history and everything, the artifacts and the gateway. But is there anything else? And I heard something about Neon—had something? Like a component that was brought on to Omnigen?
There's a server which you can use for all the collaboration features or if you want central management you can set policies there on all your agents. We didn't talk about those yet but that's the other feature that a lot of people are excited about. And then there's something we call the runner that wraps existing agents. So for example if you just type Omni Claude or Omni Codex it launches those, you see exactly that thing in the terminal but it also wraps them, like listens to all the events and injects hooks and tools in there that let you use the Omnigen features such as collaboration through the server. So these are the two big pieces. Both pieces are open source and it's designed to work with any database really, so that it definitely works with Neon. It just uses SQLAlchemy. And also it supports launching sandboxes. So we announced Databricks Sandbox, which is a VM in the cloud that spins up quickly so you can code in an isolated environment. We also support like Daytona and Modal and Kubernetes and other ways of launching these. So we try to make it easy to integrate with any infrastructure you have.
So how does it work with the sandbox? If I type—so I'm on my laptop, I go to the server and I type something in Omnigen? And then it's sending the query to—I don't know—Claude?
So, the sandbox is a way to launch the whole session in a cloud VM so you can close your laptop and do other stuff. And so you can have maybe better security and isolation around it. Actually, one of the cool things you can do with a sandbox is you can launch it and then you don't have your personal credentials and files and stuff in there. You can actually just share it with another developer when they're debugging or whatever. Like you can actually do collaborative coding or review that way. But even for yourself, you know, instead of trying to keep your laptop running, you got this reproducible environment. So, yeah, most of our engineers I think code in sandboxes. Now, the only thing some people like to do better on the laptop is fine-tuning, because the developer loop is very fast, but I think even there about half the people use sandboxes.
So, it means you also load all the skills and everything into the sandbox.
