Databricks Omnigent Deep Dive with Matei Zaharia: The Collaboration and Control Layer for AI Agents

Thanks for having me, Josh. Great to see you.

Yeah. So, Omniant is a layer for working with agents. We call it a meta harness because it's a layer above the individual agent softwares and the harnesses, which is basically the software to interact with an LLM. So for example, Omniant can run and manage coding agents like Claude Code and Codex and OpenCode. It can also run on top of custom agents you build with SDKs such as OpenAI SDK. And what it does is basically these issues that come up above the level of a single agent. So for example, composition of agents, how do I get a bunch of them to work together to do something? Collaboration, a big part of it is the UI which allows humans and agents of different kinds to collaborate. And then control, which includes security, cost control, and other things that you want to do in a common way across all your agents. And it's an open-source project. It's something we built internally to help with coding agents. And we're also increasingly using for custom agents internally.

Yeah, so basically Omniant can give you a richer set of functionality on top of Claude Code. So one of the things you can do with it is you can take Claude Code, you know, work with it, but then share your session with another person, you know, save it for later use, like let them clone it and run it themselves. So there's very rich kind of collaboration features on top. The other thing is you can easily switch to another provider. So nowadays, I think we actually did a survey, many engineers are using multiple coding agents, especially because, you know, there are all these sort of promotions and usage tiers and so on that you have from each one. So you want to try them and the agents are also good at different things. And then the other piece is if you want richer security controls and cost controls around your agent. So, for example, one thing Omniant can do is when you launch an agent, you can say, 'Hey, run this, but tell it to ask me like for every $10 it spends.' You know, pause it every time it hits 10, 20, 30, and just ask me if you want to continue. That's something that none of these individual ones do. But it's very useful as a developer. Like my experience is I get runaway costs sometimes when the agent decides to do something sort of stupid like reading a giant log file or whatever. So this is what it does. What we found at Databricks is a lot of the engineers who are using agents heavily, they built their own UI and their own orchestration around them and so they really found it useful to do the composition and to have a nicer way of interacting with the agents and then they got stuck on how do I invite others to collaborate on it because like everyone has their own UI and their own model of doing it or how do I make it really secure. So this is what we try to solve.

Yeah, we wanted to make sure it's awesome at coding initially but people are using coding agents for many things and we also support building custom agents. You can just write a YAML file with sort of the tools and skills and sort of sub-agents and so on that you want for your agents. So I use it like I basically talk to all my agents through this now. It's kind of nice to have a single app where I can do like sort of the web research type tasks and coding and data analytics and so on. And a lot of other folks are doing it the same way. We focused a lot on like the human interface of how do you manage lots of agents together and how do you share and collaborate with people but it's also interesting for building custom agents. What we found internally, we ship all these agents like Genie and then some smaller ones for working in the product and every team building those ended up building a YAML-based config language to specify their agent because they wanted to change the tools and the skills and so on all the time. And then every team ended up building a chat UI where they needed the same feature history, sharing, permission requests, artifacts that it can display, the ability to comment on them. So they were building a lot of common stuff. So Omniant is also a good starting point to build custom agents and custom apps for that purpose.

Yeah, we think it's really like, you know, the way I think about it, like LLMs alone were sort of like maybe a little bit like the assembly language or like the processors for compute, like they each have their own different formats sort of that they expect input in and different ways you call them. Then harnesses like OpenAI SDK or Claude SDK or whatever were kind of the software layer, the compiler that makes them all look similar, right? So in your compiler you don't worry about whether you're targeting an Intel processor or an ARM processor but like Intel makes sure the compiler gives instructions in the right format for that and you know ARM does it for their thing. And now that you have these, so like the harnesses are mainly about translating like your human intent into the type of format and the type of API calls that that LLM is optimized for. But now above the harness you have this natural language interface. So anything you do with like composing agents or controlling them or exposing them to humans on top of that natural language interface is a common sort of problem you have across all the harnesses. So you can do it the same way for all of them. And what I found with this is it makes it super easy for me also to switch between agents or kick off multiple agents on a thing. Whenever I'm researching something actually I ask two or three agents to do web searches on their own and then I ask them to compare. So just generally can improve the results of a lot of things and can let me build my own workflows.

Yeah. One of the things it means is flexibility. As you build internal agents you can easily swap between the latest models and harnesses or you can even combine them into a big agent that does different things with sub-agents. I think for coding specifically people are also really excited about the collaboration and the cost and security control. Like coding agents are very scary from a governance sort of perspective because you can spend a ton on them. They might just decide to like read a big file or something and you spend like a thousand dollars by the time you're done. And also they're working with code so they often might read stuff from the web or like install packages that might be compromised. They might push stuff into your repos. You know, engineers will want to connect them to production tools and like query the data and stuff. So very easy for things to go wrong and since we focus on doing this security in a consistent way across any type of agent, you can set up those policies once. You have this open-source project where many people are contributing to improve them to give you more policies, to give you sort of a library of security and cost tools to choose from. And then you can apply them to any of these and even if your developers want to switch, you know, from Claude to Codex or to Copilot or whatever, they work in a uniform way.

No, you don't. We try to make it very easy to deploy on just common infrastructure that any developer has access to. So you can run it all with Docker and the database like Postgres. We made it really easy to deploy onto like Railway and some of these hosting platforms. Fly.io is another one. We also integrate like OpenRouter, Ollama, like all these LLM providers. It supports like subscriptions you have to existing agents or API access. So we tried to make it really easy to deploy anywhere. And the reason for it is we think we and we think the industry will benefit from having a common layer to do these things above the level of individual agents. It's hard for any one company to keep up to date. Like for example, how do we build integrations with all the 50 plus coding agents that are out there and maintain them? Well, with an open-source project, if they're all contributing, they can make sure those integrations are good. How do we make sure we have the best, easiest to use security policies that understand like the latest MCP servers and so on, like say the latest API from Google Drive or something. Again, if it's open source, people can build those. It's kind of like with Apache Spark because anyone can build connectors for it and anyone can build libraries on top, people benefit from that. Like Databricks doesn't have to build say a MongoDB connector. Maybe MongoDB builds it on their own or like someone else does and MongoDB doesn't have to build like a machine learning library because someone else contributed that and they can just run it. So we think these problems above the level of a single agent are big enough that you need a layer there and especially the security and governance ones, they're deep problems, they'll keep evolving because the attacks and threats and concerns will keep growing. So there's a lot of depth. So it makes sense to do that together. And same with the connectivity and collaboration between lots of different agent types.

Yeah, good questions. Yeah, so the managed versus open-source ones, the features are the same. Mainly we just run it for you. Like in Omniant there's this server, so you have to run a server somewhere if you want collaboration and central control of your agents. Then there's the individual agents you're running and that could just be on someone's laptop or you deployed it as an app somewhere or whatever. So the managed one is this hosted server so you don't have to host your own. It already understands all the MCPs and stuff in Unity Catalog. It also can get capacity through the AI Gateway if you want to get LLM capacity through us or to manage it through the gateway, but it's basically the same server and the same UI and the same experience for users. The other piece we launched that's also exciting is what if I want to run an agent in a sandbox, like in a VM somewhere else that's just for that, that has restricted permissions. So we also announced sandboxes on Databricks and much like apps, you can restrict sandboxes in terms of access to all the data and all the tools and stuff that you registered in Databricks. So for example I can say like launch an agent in this sandbox and using the Unity Catalog controls I can say like only let it access this catalog or whatever. So that's another feature. Again, if you just want a very reliable place to host a lot of CPU VMs, we're also good at that because we launch like millions of VMs every hour at this point probably for all the data workloads that we do. But all the other things, the collaboration UI, security policies, cost management policies are open source and we want to keep building them that way and benefit from the community figuring this out because no one knows like the very best way to say do intelligent routing for coding agents or whatever yet.

Yeah. So on agent, it's primarily for coding. You can also use it for custom agents like custom knowledge work agents. So it's really closer to sort of the Agent Bricks space of like build your own agents plus this coding one which is really new. We didn't have anything for coding but the combination of Omniant plus the sandboxes can make us a pretty good platform for development. And internally we use these things for our engineers, like most engineers work in a sandbox. Engineers were already using like the predecessor to Omniant, which is our wrapper around like Claude Code and Codex and Cursor and so on and then now many of them are using the new thing. So it's sort of a new product line. So I think in terms of end users, Genie, Genie Code, these are integrated into the data analytics sort of interfaces, like you can even go in a dashboard and ask Genie a question about like a specific data point on a chart. In terms of like, hey, I'm making a custom agent and I want a nice interface and hosting for it or I want an environment to share and collaboratively code on things with my team, that's what Omniant does. And then the underlying technology actually more and more of our agents are based on Omniant also so we hope to basically consolidate the stack in there and then we benefit in all these. Like for example, the security policies and so on, they'll apply the same way to Genie and to coding agents and to custom agents once you write them.

Yeah, I think, yeah, so definitely open-source models have gotten a lot better recently, especially for coding, which is one of the biggest use cases of AI, probably one of the biggest centers of AI spend overall. So really worth a look at. Now it'll continue to be a race and the private companies are well funded. You know, they'll keep improving things. And I'm sure the open-source ones will as well. So in practice we actually have some engineers that are trying and mostly using open-source models to see how they work and we actually put out a blog post just recently where we built a benchmark of like internal tasks our engineers were doing and like basically a pull request someone does and tried to do them with different agents and we found that GLM for example was up there like basically in the same ballpark as Claude Opus and as GPT-5.5 and like the latest top tier models in terms of success rate on our tasks. So they're becoming very competitive, people are using them. I think stepping back though, like for sure with the fast-moving space here, whether you're using agents or building your own agent, you probably want flexibility to be able to move to the latest and best LLM when it comes out or to use them for different things because they're better at different things. Actually nowadays with post-training, you know, before like models were, most of the compute was in pre-training and they're all trained on the same data on the internet so they all sounded the same. But with post-training actually we're finding that they have very different skills. Like pretty much everyone thinks Claude is better at front-end and website design than GPT. On the other hand, GPT is quite good at debugging and reasoning. So you basically as a user of these things you want flexibility and Omniant is meant to make that really easy. So you can move your settings, actually move entire conversations from one to the other, share them with people and if you're writing a custom agent it's just a YAML file and you can change like a couple of lines in there to change the harness and run it on a different agent. And I think that's really important. Even if people end up using mostly closed models and closed model providers, they'll want that flexibility to move between them that prevents lock-in and makes it better for them long term. And I think the open-source models also obviously are very strong if you want to control exactly where they're served and hold on to the data or customize them. And I think they'll continue to play a role. We're seeing huge demand for them.

Yeah, I'll just show you a couple of the things that I do with it and then the collaboration which is one of my favorite features. Really the whole team is using this to develop Omniant and other things. And then I think Elise is going to also show us intelligent routing which is one of the new things we're launching. So we're very excited about that. Let me just...

Yeah. And part of that is based on like at least for us internally, we have lots of engineers. We care about lowering the cost. So a lot of it is based on measuring how different models do at different types of tasks. So I do recommend checking out the blog post on the benchmark actually if you want to.