Introducing Omnigent: an open meta-harness – Matei Zaharia, Co-founder and CTO, Databricks

Hi.

All right. Hi everyone. Welcome to the second day of keynotes. So, I'm excited to kick off the morning today and talk about a new open source project we launched over the weekend. We just released it Saturday. This is a new layer from our engineering team for working and building with agents, and it's based on patterns and challenges that we saw coming up everywhere where people are using agents. And we realized that we're going to need this new layer on top of the current agentic stack to make it really effective. So, I'll talk about what that means, but we call it a meta harness, a harness of harnesses, if you know what an agent harness is.

Okay. So, as everyone probably knows, agents are really starting to work. Coding agents are the first type that were really successful. Pretty much every developer has probably used them at this point, but the same is true in other fields and you saw with Genie and with other agents that you can make very effective agents. And especially because coding is a lot easier now, you might wonder do we even need agent infrastructure at all? If I don't like the current agents, can I just write my own code? So, it seems like hey, maybe we're kind of done with at least simple software and we're set and this is what the world should look like. You know, agents are everywhere, things are working. But actually when you use agents, it can be quite annoying if you're a manager, it can be very scary because of security issues and stuff like that. So, you know, the world isn't really perfect yet. So, there is a problem and actually I didn't realize until I was making the talk and looked very carefully at this image that we even see part of the problem in the image. It's the harness, the thing around our little agent there.

Okay, so what's an agent harness? The agent harness is basically the software around the language model that is the agent's interface to the world. So this is something like Claude Code as a coding agent. There's Claude the model, that's a thing that just outputs tokens, and then there's Claude Code the harness, which connects it to files and a user interface and security permissions and stuff like that. And then there's Codex, and then there's any custom agent you make, you have some kind of software around it called the harness. And because it's so easy to build harnesses and because it actually helps to co-design the harness with the language model, everyone is building them and indeed everyone has. So there are so many agent harnesses here. These ones here are just some of the popular ones for coding, and then if you make custom agents, there are lots of agent SDKs and stuff like that. It's just a ton of these out there. Actually this weekend I was at the pet store and this reminded me a lot of it. Like pick the harness for your agent, any shape, any color, you can put it in there.

So what's the problem with these? Well, the issue is that agents sometimes have to interoperate, and there are also a lot of common challenges that the harnesses have where it helps to have a common layer that lets us interface across them. And that requires people to sort of build the layer and agree. If everyone is just vibe coding their own thing, they will still be incoherent. So you know, no amount of faster coding will solve this problem. So what are some of the challenges with interoperability of these? The first is composing agents. When I'm doing coding and when a lot of the engineers at Databricks do coding, they use many coding agents at the same time. If there's a bargain, you know, one of the agents is stuck, they ask another one. They give them different inputs. They ask them to work on different things in parallel. So, how do you compose agents with different harnesses? It's easy to do within one of them like Claude Code or Codex, but like what if you want to use both? And there's a lot of other techniques for using multiple models to get better cost, better quality, and so on.

The second one that's interesting is collaborating across humans and agents. You know, if I'm using one coding agent and someone else is using another one, how do we collaborate? Because ultimately we have to get stuff done together. So, it's actually a huge problem that we think we can improve on. And then the final one is I've got all these agents, all these different pieces of software, the harnesses, how do I control all of them for cost and security? So, we saw these problems repeatedly in Databricks actually from two angles. First, the engineering side where we have our own infra around coding agents. And all the thousands of engineers are using them. And also on the customer-facing side where we build agentic products like Genie and we have to keep changing how they work and we build custom agents for our customers. So, let me dive into each of them a little bit more.

So, composition. There's so many patterns emerging now for how to combine agents to get a better result. So, if you've looked at coding, maybe you've heard of loops. This is how you have some agents maybe supervise each other or running in a loop to try to finish a task. And there's all these patterns for loops. Another pattern that's emerging that's really interesting is called advisors. So, you can have a small, fast, cheap model that's doing a bunch of the work and then an expensive model and maybe the expensive model is like driving the small one, getting it to do the busy work, or maybe the small one is calling to the expensive one when it's stuck. So, Harvey AI had this blog post on custom train model with a frontier model as an advisor. Anthropic has a post about it. Claude Code actually uses multiple sizes of models for different things. Actually, my research lab at UC Berkeley, we wrote a paper on advisor models, too.

And then there's many other patterns of composing them. We have a post about how in Genie, if you use multiple LLMs, you can get better quality and better cost. So that's the composition that does kind of continue to be an issue as agents and models evolve. Second one is collaboration. You know, whose desktop looks like this? This is what my life is like all the time. I've got all these agent windows open plus things like Slack and Google Docs. And all I'm doing is copy-pasting text across them. This is pretty ridiculous. It's you know, there should be an easier way to collaborate, but each agent has its own harness, its own app, and it's stuck there. And again, your company managing this, it's also a huge pain to manage everything. So just as a simple thing, what I am working with with one of these coding agents, and I have to go away or whatever, why can't I just hand my session to another engineer and have them continue with the full context? That's a simple thing that any harness would benefit from.

And then the final one is control. Agents can be hacked through prompt injection. So you know, you shouldn't really trust them. Agents will also make mistakes even without being hacked like deleting databases and so on. With Lake Base, you could just revert the database, but in general, it'll be an issue. And they can also cost a lot. And people might say, 'Okay, well, agents are very dangerous. Let's ask users for permission.' Here's the average permission prompt when Claude asked me something. It's like, 'Do you want to run this hundred line Python script? Yes or no?' So, you know, it's a huge pain as a user. Just waiting, hoping that users will review every action is not enough either. So this control and security layer is another common problem across harnesses that we want to solve.

So we think that we need a new layer above the harness level to manage and build with agents. And we also think the layer should be open. So anyone can plug in their own harnesses, their own workflows, their own tools. And we call this layer the meta harness. So that's what we decided across teams at Databricks. And then we decided we have to make it open source because that's going to be much better if anyone can contribute and it works with the latest things. And that's what we just released with Omnigen. So how does Omnigen work? At the bottom you've got existing agents. It works with coding agents actually. Just since Saturday we added two more, Antigravity and Coarser, that it supports because people were asking for them. It also works if you're building custom agents with something like OpenAI Agents SDK, you get a lot of the same features on top. Now all these harnesses are different pieces of software sometimes tailored to that LLM. But what we do is on top of them all a common interface so we can do the collaboration and composition and sharing that Omnigen tried to do. And the interface is basically you can send messages in, you can get a stream of events out. They can call tools. That's what it is.

So Omnigen then has two pieces. First there's a runner which wraps around your agent and can enforce security and other types of controls by sandboxing it and just monitors it and makes sure it's running. And then you can also optionally connect Omnigen to a server where you get collaboration and central policy features. So with your server if you connect your agents to that, you get a central history for everything. You can share agents with people. You can set up MCPs and skills centrally. You can also set up security policies and share artifacts out. And then on top of this, no matter what kind of agent you have, you also get a uniform interface. So, you still get the agent's terminal UI if it has that, but any agent is also accessible on the web, native app, mobile, and also REST API. And Omnigen is open source. It does not require Databricks in any way. We developed it so you can just run it anywhere. And we can use it both at small scale and large scale. So, it just uses standard technologies like Docker and Postgres and OpenTelemetry to run the server.

Okay. So, what does this give you now that you have this layer? There's three different things. First for composition, you can now easily combine agents that use multiple harnesses and LLMs. You can actually just write a little YAML file to define an agent that's using multiple of these together. It's also super easy to switch agent harnesses while the agent is running and continue with a different one, which I'll show in a bit. Second, you get collaboration. So, we support live collaboration. You can even invite other people to write to your session and talk to the agents and comment on files and things in real time. So, this makes it way easier to work with these. Once this is available, all my work I try to do through this. And then finally, for control, we looked a lot at how can we make security around agents better? And we actually came up with what I think is a much nicer, more powerful model than what you have today. Today, in most agent tools, the controller is always just for each type of tool call, do I allow it or deny it or ask the user? They're very static. We support contextual policies where the permissions vary based on what the agent did before. So, for example, if the agent had a confidential document, maybe we don't let it send an email outside the company anymore. But, if it only had public marketing docs, then it's okay to send an email. Tracking state makes that much more powerful. And we also have policies on cost, which none of the agent frameworks do. So, you can tell it, you know, don't spend more than $5 on this, for example. We also built a very flexible OS sandbox to protect the agent from harming your machine. So, that's three things that it does. To show you a little bit of this in action, I'd like to show a demo.

All right, cool. So, I'm here with Omnigen setup. This is actually my dev box inside Databricks. It's like a machine that I develop on. And then there's my laptop. So, the first thing I'll show you with it is how it just wraps an existing agent. So, I'm just going to start Claude, but with this Omni in front of it. And that's just going to start it. It's starting this liner. And then, as you'll see, we'll basically just have the standard Claude interface. So, this is actually the instance of Claude in Databricks that's been configured with all our settings. It's got all these MCPs and different things in there specifically for Databricks. And I can ask it to do stuff. So, currently I'm in the Omnigen project. So, what is this project? I can ask it. And you'll see it's doing... And even we customize these things that Claude says. Archiving. If you don't know what archive is, don't worry. So, there it is. It means something in Databricks. Okay. So, now I'm just running Claude. So, that doesn't seem that exciting. But what do I get through Omnigen even as an individual developer with an existing agent? One of the things I get is a web UI for my agent, too. So, the same session I was doing, I have it on the web. I can see all the files and things in here if I want. And I can see it here in a nice chat form. I can even see the same terminal in here if I want, so it's the same thing. But let me ask the agent to do something like... Actually, since I was doing this before, I'm going to check out the readme first. Oops, sorry. Because I was trying this before, so let me do add a joke to... So, I'll just tell it to add a joke to the top of the readme. Okay? So, I sent a message from the web. Now, in the terminal, you still see it here. So, we do this mapping of user input and output across the two interfaces. And okay, it's saying, 'Hey, I've added a joke in there.' If I look here, it says that it added it. Now, the other nice thing about the web UI is of course it's much more visual. So, if I look here, I can see there's a changed file, the readme. I can see there's a joke at the top, but this is just a markdown file, so I can just view the whole thing. I'm going to close this to make it a little bit bigger here for you. All right, so this is it. I can view the diff. And I can even put in comments about it. So, for example, all right, I guess there were a few because I was doing this before, but maybe I can tell it like... I'll just tell it to delete the jokes. Okay. And then I don't know. I could try to change other things, but I can leave some comments in here and tell it to address all of the comments. And this is just sending it. So, I get this nice visual interface, and it's doing its thing. The other thing we built for it that's very helpful, I'll just tell it to add one. The other thing we built is actually a native app. So, in my browser, if I go here, in my browser, I've got all these tabs. It's hidden all the time. I closed all my tabs this morning. But in the native app, it's very easy to find it, and you get notifications and stuff from the system. So, let's look at this again. Okay, we got a joke in there. Okay. So, what else can I do? The other thing I can easily do is I can switch agents. So, I can take the same session and say, 'Hey, let's actually continue from here with Codex.' And just fork the session, and it's going to start Codex here, and you'll see that it has the same history. Okay. So, now in the terminal, this is actually OpenAI Codex, and I can just tell it, 'Add a second joke too.' And you'll see it adding that in a second. This is a little bit slower, but it's going to be in there. So, this is pretty cool. If I'm stuck with one agent, I can switch to another one, and then there are two of them. But then the super interesting thing that is really enabled by composition is I can also use a multi-agent coding. So, we actually include an example here, Polly, which is a multi-agent coding supervisor that can use both Claude and Codex to do sub tasks. And I'll start it in the same folder and tell it, 'Ask these guys to both rate the jokes at the top of the readme.' And you'll see when it starts that it's going to kick off a couple of them. It's checking to see if I have Claude and Codex installed. It's launching some workers, okay? So, in here you'll see some workers will pop up. There's a Claude worker and then it's going to start a Codex worker, too. And it's telling them to rate these things. So, these are again, these are just Claude Code and Codex. They do the same things you could do otherwise. I can click on these guys and even see what they're doing. And here there's the terminal. They're trying to rate things, etc. If I go back at the top, you'll see that the supervisor hears. And eventually it's going to say, 'Okay, so first one, pretty good. Second one, not quite as good.' So, that's what Polly thinks. Okay. So, obviously I could do more things. I use these to do all kinds of other stuff. Like, for example, thing I was doing last night is I asked it to find all the open GitHub issues about something and then investigate them in parallel and see if they've been fixed in the code. So, there's a lot of cool stuff you can do with this that I don't have time to show. But the final thing is, so that was composition of agents. The other thing you can do is collaboration. So, let's say I think I have a great change to the project here and I want to get an engineer to review and commit this. I'm just going to call one of our engineers. Oops, sorry. And this is Ankit. He's on the AI team and I think he's a good reviewer for this readme. So, I can just invite him to it. And let me just give him a call.

Hey man, you busy?

Okay, I got this important change to the readme. Can you take a look and tell me which version you prefer? Thanks, yeah, I shared my session with you.

So as you can see, thanks Ankit. That was very easy. So I invited him, he can see the same session, he can leave comments and things, and he can send a message in there and tell my agent to do things. So I can hand off work to someone else. And obviously all the history, all the sessions, any agent, whatever, are also stored on your server, so you can analyze them later and do things. Okay, so final thing I'll show is policies. Actually, I guess I can show it in here, too. So the stateful policies. So notice at the top here, we actually track the cost of each agent, how much it spent, and we also allow you to set policies. And these are contextual, they can depend on the current state. So one of the policies we allow is set a budget on it as it's spending in this session. Like you don't let it spend more than say a dollar without asking the user. So I'll just make something very silly here. It already spent 25 cents or whatever, so I'll say ask me if you're trying to spend more than 20 cents. And then I'll tell it to do something that's going to involve reading a lot of files. And you'll see as it's thinking, eventually when it begins to do stuff, it's going to ask me for permission because it's out of budget. So this is pretty nice. You can also tell it like launch a sub agent, but only let that agent spend a dollar. There's all kinds of things. And if you're managing this for a team, you can set policies centrally. And if I approve it here, it's going to continue. And eventually when it gets to $5, it's just going to stop. So this is a really cool feature. Again, on top of any agent, Claude, Codex, whatever it is. And it's just scratching the surface. You'll hear more about these soon. Cool. So that's the demo. So let me just wrap up with a couple more things in the slides.

Cool. So, one thing you may be wondering is there's a lot of functionality in this. Why are we open sourcing it? So, we really think working with agents has to go above the level of a single harness and SDK and so on. We think this meta harness layer is going to be needed and it'll benefit from being open the same way that data formats and sharing protocols and stuff like that benefit from being open because it's an ecosystem where people sort of have to agree on a way to do it to make it work well. So that's why we're making it open source. We use it a lot internally. We hope you use it as well. And let's figure out together how to build things with this. We already got a lot of really great contributions since Saturday. The other cool thing about open sourcing this is everyone who starts building an agent wants an app like what I showed. Maybe not for coding, but with collaboration, sharing, security, all that stuff. So, now you got an open source app you can use to build any agent with a great interface. I didn't show it, but you can even get a mobile interface to your agent through this. So we have a lot more coming up, more ecosystem support, also auto optimization of the agents, which you'll hear about hopefully in future events. So that's Omnigen. We really think that AI needs a layer above the harness. It makes it way easier to work. Suddenly, our team has been working with this all the time. We, you know, developing and not just the engineers, but product managers and designers and so on. And we invite you to check it out. It's early on, but we think it'll be an important layer. Thank you.