Welcome to MongoDB.localenyc. Our session will begin in 5 minutes.
Welcome to MongoDB.localenyc. Please take your seats. Our session is about to begin.
Welcome to MongoDB.localenyc. Please take your seats. Our session is about to begin.
Welcome to MongoDB.localenyc. Please take your seats. Our session is about to begin.
Welcome to MongoDB.localenyc. Please take your seats. Our session is about to begin.
Welcome to MongoDB.localenyc. Please take your seats. Our session is about to begin.
Good morning and welcome to MongoDB.localenyc. Now please put your hands together for Dev Ittycheria.
How are you? It's nice to be back. I'm back. Felt like I never left. But seriously, I'm thrilled to be back up here with all of you, and I appreciate the fact that you're spending the day with us because I know all of you have very busy schedules, a lot of things that you have to do. So the fact that you're spending this time with us, we really appreciate, and my team and I really appreciate the fact that we can spend this time with you and tell you all the great things that we're doing. So to that end, we have some amazing new product announcements that we're going to go into a lot more detail. We have a lot of hands-on sessions. There's a lot of technical breakouts, and this show is really about helping people move AI from experimentation into production. You're going to hear from a great number of speakers, and I feel really honored that these people are going to talk about how they're using MongoDB and AI to transform their business. These leaders are from some of the most technically sophisticated organizations in the world, so you're going to hear some really groundbreaking insights. You're going to hear from founders of fast-growing AI-native companies and large Fortune 500 organizations who are really doing some very, very sophisticated work. So I want to thank them for also spending the time and sharing what they're doing with MongoDB. I also would encourage you to try and talk to them one-on-one between sessions just to get a sense of who they are. None of this happens alone. I also want to take a moment to thank our sponsors. Our sponsors include AWS, Google, Azure, IBM, and Accenture, who are global sponsors. And there's also a number of sponsors who made this show happen. So please take a moment to go by their booths. They can also talk to you about how they're helping work with us to better serve you and how we're kind of extending the MongoDB ecosystem. And most importantly, I really want to thank all of you for being here, right? Even in the rows you're sitting in, you're going to find people from a whole host of different companies, organizations, and industries, and I encourage you to talk to them about how they're thinking about using MongoDB. And essentially, what I always find is that these sessions are great to learn from your peers about what they're doing because there's some great things that everyone's doing that we can all learn from each other. And frankly, you're the people who push us. You're the people who give us feedback on what to do, what to do better, what to do more, what to do that maybe we're not doing. So that feedback really gets internalized in our organization. So hopefully what you see today is a reflection of all the things that you've been asking for from us. Sorry. So I just want to take a moment to step back. You know, MongoDB has always, I would say, been a contrarian thinker. And what I mean by that is that when we first launched as a company, we came up with this novel idea that you don't have to store all information in a table. In fact, it doesn't make sense to store information in a table because information and data is so complex and comes in a variety of shapes and forms. And the document model, we believe, was a much more natural and much more intuitive way to organize and work with data. Now, that was pretty controversial, and a lot of people and purists were like, 'Ah, that's never going to work.' And I think what we spent the first few years really trying to prove, not just to all of you but to us, is that we could deliver and support the most mission-critical use cases with a document-based database. And frankly, we did that. We have some of the world's most sophisticated organizations doing that. Then we heard from you that people said, 'You know, I don't want to always have to manage MongoDB myself. I just want to be able to consume it in a way that's easy for me to work with your technology.' So we launched a cloud service. We launched that around 2016. I will tell you it was about one year before we went public. And when we went public, that business was only 2% of revenue. And people were highly skeptical that we could build a cloud service, go head-to-head with the hyperscalers, and offer a really differentiated capability. Well, we proved that as well. Atlas is our fastest-growing business, and it's something that the majority of our customers actually use today. The third thing we did, we heard was customers said, 'Okay, this is great. I'm glad you have a database. I'm glad I can consume it on demand, but I need more. I want to be able to address a broader set of needs.' So then we listened to that feedback and basically built things like full-text search, vector search, and then through an acquisition of a very innovative company out of Palo Alto, Voyage AI, we have the world's best re-ranking and embedding models in the industry. So today, we're more than just a database. Today we are truly an intelligent data platform, and as the requirements for the AI era have evolved, we believe that we are well positioned to address those requirements. So let me talk to you about what those requirements really are, at least the way we see it. First is that we think it's really important for these AI models to have fresh, accurate data. You can have the most powerful model in the world, and these models are coming out fast and furiously almost every week, but if it doesn't have access to the freshest information, it's going to make bad decisions. It's going to generate bad outcomes. So with our document model and our agent memory, where you can now embed business logic — and by the way, we're going to go into a lot more detail into this the rest of the day — we believe that we can provide the freshest information and the most accurate information for the models. But the other thing is, okay, if you have all this data and this data is fresh, how do I know which data to choose to be able to reason with and basically have a good outcome? Well, as I mentioned, we have the best embedding and re-ranking models. And with Voyage and full-text search and vector search, we can enable you to find the right information in the most precise way at the right time to make the right decision. And that is something that's super important. The third requirement is that agents, you know, are very unpredictable. They don't go home at night. They don't take coffee breaks. They don't stop working over weekends. And so the demand from traffic of agentic workloads is highly unpredictable. It's bursty. It can go up and down, sometimes ephemeral. So you needed a platform that can massively scale up and down to serve the needs of these new agentic workloads. What you're going to hear again about MongoDB 9.0 and Infinite, I think, will be super exciting about how we're pushing the envelope on performance and scale. And I'll touch on that in a little bit more.
The next point is pretty obvious. You can have all this power and all this capability, but if you can't trust it, you're not really going to use it. You know, if you can't trust the answers, you don't know how secure it is, you don't know the governance model. Well, you're going to hear a lot of things about what we've done to beef up the security of our architecture and also provide the guardrails for runtime environments to ensure that when you do build and deploy agents, that they do what you want them to do. And then last but not least, we have heard loudly and clearly from all of you that you want the flexibility to deploy your workloads anywhere you want — not just in the cloud, not just on-prem, not just in an air-gapped environment depending on your organization's needs, but be able to do it across them and be able to, in fact, work in a hybrid environment. This is something that's become super important, and we know that a lot of customers are nervous about models potentially training on your proprietary data. So we hear loudly that people want to work with open-source models, and we also hear loudly that they don't want to run everything in the cloud because people want to feel comfortable about running their workloads also on-prem. So the foundation of all this has been happening over the last few years, and this is just a small chart. In fact, there's a chart that we have that shows all the capabilities we've built over the last few years, but this is just some examples of the things that we have built over the last few years to set the stage for basically positioning us for the AI era. This year alone, we introduced text search and vector search to our Enterprise Advanced product, which is a product that people use to self-manage MongoDB. And that was something that was in huge demand from a lot of our financial institution customers and other customers who really want to run workloads on-prem. We just recently came out with our managed MCP connector that's become incredibly popular, and I'll talk a little bit about that. And also, we've just made enhancements to our Voyage embedding and re-ranking models that again you're going to hear a lot more about. But the key point here is that we are removing the friction around how you can work with MongoDB, and the customers are starting to notice. The big announcements that you're going to hear about today are one, MongoDB 9.0. It is truly the most performant database version we've ever produced. 8.0 was super performant, and the uptake by our customers was super — in fact, the adoption of 8.0 was the fastest ever for a new release, and I think you're going to be really impressed with 9.0. So I encourage all of you to try it out. You're going to hear a lot more about this. Atlas Infinite is probably one of the most profound architectural changes we've made in Atlas since probably when we launched Atlas and when we introduced asset transactions. And you're going to hear a lot about this, but basically we're decoupling storage and compute in a very sophisticated way because in a distributed database that's not easy. And because of the way we manage and charge for Atlas, it actually is going to make your life so much easier in terms of using MongoDB for very, very cost-sensitive workloads. And last but not least, we're introducing Atlas Engine, which is a memory runtime and governance stack for running agents in production. And the key point here is you can embed your business logic into the memory, not just data. So you can offload all that heavy lifting you had to do yourself to create agents to run in your production environment. So I strongly believe that MongoDB, more than ever, is the best place to run and build your next generation of applications, including your AI workloads. So what I would like to do now to get the show started is to introduce Jim Sharp, our CTO, who's going to go into a lot more detail about these new products. Thank you very much.
Hello, New York. It's an exciting time to be in technology. So I've spent my entire career building and operating high-scale, mission-critical systems. And I've continued that focus over the last three years at MongoDB. In fact, I just crossed my three-year mark the other day. I knew that before enterprises will hear you out on shiny new products or features, they first need to trust the foundational technology they're running on. It just has to work every time at scale, 24 by 7, 365 days a year, because customers depend on us. So a couple of years ago we launched MongoDB 8.0, and 8.0 was the best database version this company ever produced. We focused on performance, scale, and reliability. Because of that investment, over 80% of our customers now run on MongoDB 8. And I'm happy to hear from customers that they're seeing the benefits. So today, I'm really excited on behalf of the team here to announce MongoDB 9.0. Thank you. So with 9.0, we took the foundation of 8.0 and doubled down on it to further improve. Now there are a lot of great improvements that you're going to hear about today in 9.0, but I want to just cover a few. One is yet again improved performance. Improved performance helps all customers for all workloads. The next is we've invested in what we call intelligent workload management, and this protects the database regardless of whatever traffic comes its way. Now, much like how modern cars can detect unexpected traffic even if the driver didn't respond and automatically slow down the car to protect the contents they're in, likewise, IWM protects the database clusters regardless of what traffic hits them. And last, we're really excited about queryable encryption, a truly incredible industry first. Most databases store data encrypted at rest, and they may encrypt it in transit, but within the database, it's in plain text. With queryable encryption in MongoDB, you can encrypt the data even within the database but still query it as if it was a normal query. It's truly remarkable and really important for security-conscious workloads. So the team and I are really excited to put this in the hands of all our customers today. We're committed to running this anywhere available in Community, Enterprise Advanced, and in Atlas. Now, I've only scratched the surface rather on improvements that we added in 9.0. So later today, you can learn more about 9.0 and hear from our customer Coinbase live on stage, who actually worked with us to help develop intelligent workload management. So 9.0 is awesome, but we asked ourselves, hey, how could we take this one step further? How can we leverage the power of the cloud to provide Atlas customers even more benefits? So the team re-imagined Atlas unconstrained by existing architecture. And what we came up with was Atlas Infinite. So today, we're excited to announce that Atlas Infinite is in public preview. And let me be clear, this is the biggest advancement in the Atlas architecture since it first launched over a decade ago. And what Atlas Infinite allows you to do is you can still scale MongoDB horizontally as you always could from the beginning. In fact, MongoDB really distinguished itself from the outset as being a horizontally scalable distributed system. Now, but for some workloads, customers told us that they'd like to scale compute separate from storage rather than in fixed blocks horizontally as they scale. So simply put, we separated the storage from the compute again. So when storage and compute are together, you're forced to compromise because you either need to scale both resources together based on the configuration of the server you're using. But let's say you don't want to do that. So there are lots of use cases where you might want to scale compute independent from storage. Let's say you had a high-traffic use case and you had a lot of queries coming in and a lot of customers on a relatively static set of storage. There you would want to scale the compute and keep the storage fixed. Conversely, there are use cases, maybe a logging use case, where you have a fairly small query volume but your data set is going to grow perhaps to vast amounts of terabytes and terabytes. There you would want to scale storage independent from compute, and this is what Atlas Infinite allows you to do. Now, this is not something that was easy to implement. Changes like this don't happen overnight. You can't easily just put this on top of an architecture in an OLTP database. You have to build it from the ground up. And it's not just a technical challenge. With a large customer base like we have, we need to do it in a way that guarantees minimal disruption without lowering the bar for security. Atlas Infinite is the only multi-tenant database designed so that the infrastructure underneath it can never read what you store there. It's really important. Now, we have a full session planned for later today where you can learn and see a live demo of Infinite. Plus, we're happy to have our customer Okta on stage who will join us to explain it more. Now the team and I could not be more excited to get this into your hands to see what you do with it. So I've covered 9.0 and Infinite, and we have one more product to tell you about, and for that I'd like to invite Ashish on stage to walk you through Atlas Agent Engine.
Thank you, Jim. Hey everyone, I'm Ashish, and as technical fellow, I am very lucky. I get to work on some cool innovations that we're bringing to market. I've been here for three years as well and focus on technical strategy and architecture across our line of products. Everything Jim just walked through is about load. It's about how much, how fast, how suddenly. Today, enterprises are building agents that act on user requests in line. This requires reads and writes at scale in real time. See, delayed analytical-type products are not going to work here. That's why what you heard Jim talk about for 9.0 and Infinite is so critical for agentic AI. This is the part of agents that you can put a number on. The harder part is making sure that the agents do exactly what they're supposed to and only what they're supposed to. See, agents act on their own over and over. And given the autonomy they have with LLMs, they might choose to take actions that you want to block. Even after a run, someone may want to know what happened and check if the agent stayed inside the limits you set. Most teams answer that in pieces. Context sits in one system and guardrails in another. Identity and audit logs live somewhere else, each with its owner and its own review. None of them share context. So the team becomes the glue. Stitching it together takes weeks just to get through a security review. Across over a hundred customer conversations since last year trying to understand what our customers are doing about agentic AI, two things became clear to me. One, every enterprise is eager to deploy agentic use cases and they want it yesterday. They're all experimenting with various frameworks, tools to stitch something together that would provide them trusted, accurate, repeatable execution while ensuring safety and auditability of all interactions. This combination of the flexible JSON document model that we provide along with our industry-leading vector search and Voyage embeddings meant that customers were already treating us as the default storage for memory. But on top of this, they were struggling to stitch together memory aggregation, policy guardrails, security and observability frameworks amongst others to get that repeatability, timely, safe execution for agents they wanted to deploy. See, we built plumbing. We built Agent Engine so that you can focus on the business value your agents create and leave the plumbing to us. Agent Engine offers memory as an automatic first-class primitive so that all agents you build automatically get both short-term and long-term memory out of the box. No coding or pipelines needed. And when you access that memory, you're provided safety controls and the security and identity checks to make sure that only the right data is accessed and block invalid accesses as they're requested. Also note that you own this memory. It lives in your MongoDB instance that you can learn from or decide to port away at any time. Observability comes out of the box so you can audit what the agent did and why. And finally, for customers that are looking for scalable ways to run their agents and tools securely, Agent Engine provides a hardware-isolated runtime to run your agents and tools securely. Agent Engine is built to be open. It supports major open standards, whether that's A2A, MCP. It allows interoperability with the investments you've already made. We also integrate with the best-of-breed providers on areas like observability, evaluations and others so that Agent Engine can fit in your enterprise within the policy and standards that you've adopted. And Agent Engine runs everywhere on all public clouds, on premises, and on your laptop. Write your agent once and deploy it where you need to, close to your data. It's in preview today. In the Atlas Agent Engine session, right after the keynote, you'll watch a store manager supply agent work through supply orders. A payment fails, the agent finds another route, and by the end it has changed how it pays the supplier. Nobody needs to quote that behavior. If you're the finance controller, that's the moment you sit up. The agent just changed how your company pays a supplier, and you'd want to know why. On this platform, you can find out. There's a trace of the moment the agent recalls the failure and makes that call. See, we're not going to ask you, any of you, to trust something with your end users that we haven't already tested on our business first. As it happens, we have exactly that proof. Our own CIO, Deepa, had a real problem to solve. But rather than me telling you about it, let me bring her up to show you what she built. So, please welcome Deepa to the stage.
Thank you, Ashish. All right. Hello, New York. Six months ago, I got this email from one of our sales reps. I'm not exaggerating. The subject line literally read, 'Please make the robots talk to each other and share information.' That was the whole ticket, which translated meant that none of these tools share context. They don't even know they're working on the same account. I think that was the best requirements doc I got all year. So the problem was, let's say a seller wants to approach a particular customer and position MongoDB as the intelligent data platform, which of course we know is the best data platform out there. To make that case, they need to go through a whole bunch of manual steps. They first need to conduct account research and use four different AI tools to understand the account itself. Then they have to look internally and source MongoDB proof points. And once they've done that, they have to weed out all that Postgres noise. So once that's done, they have to track down the right prospects to reach out to and manually enter all that information back into the CRM, toggling between seven different tools that have zero shared context. And these tools are supposed to save the seller time. But when none of them can talk to each other, it's on the seller to manage all this by themselves, and that becomes a second job on top of their real one. So we're talking about millions of dollars of seller time evaporating into mundane copy and paste between tools. And it's kind of ironic that MongoDB spent a decade telling the world that their data shouldn't live in 12 different systems. Turns out our own sellers were buried under these tools. So we set out to fix this problem, but not just patch it for this one use case. We wanted to solve this for scale. And we imagined this amazing agentic experience where the sellers could prompt, interact with it, and get their job done in a fraction of the time. But I needed the right infrastructure for this. There were three basic essential requirements that I was looking for. First and most critical, the architecture needed a sophisticated memory layer for that shared context. And I want to give you an analogy on shared context. Think about a patient who sees five different specialists for their symptoms. You're literally starting from square one with each of these doctors having to re-explain the same history all over again. And you're the one — each specialist is brilliant in their own right, but without that shared context, you're the one stuck threading it all together. And that's exactly the problem our sellers were living with. They had disparate tools, brilliant tools, but with the collective memory of goldfish. And that was exactly what we solved for. Now the second key requirement I needed was an architecture that is open and flexible, allowing me to run these agents anywhere on the hyperscaler of my choice, and I did not want us to be locked into any models. I know if any of you are building AI applications, this will resonate. We wanted our developers to pick the framework of their choice, and we needed this architecture to scale to thousands of agents that could run within the enterprise, not having to place our bets on any single model, any framework, or any hyperscaler. And finally, we needed an architecture that is secure and governed by default with the right level of cost visibility. If you're all building with AI, that token spend is the bill that you don't want to open. And we needed an architecture that would bring that visibility, that governance, and we could trust our agents to behave within the guardrails that we specify. So my team and I, we looked at a few existing products in the market. None of them truly met all of these three needs. So we decided to build it all ourselves at MongoDB. And that became our Atlas Agent Engine. It checked every single box, everything we needed, fully integrated, not bolted on. And now we had this foundation to build on. And our seller solution became our customer zero. So allow me to introduce you to Holly — that's what we named her — your own personal sales assistant, who delegates work to a whole bunch of sub-agents, each of which understands the organization's context, the workflows, processes, your enterprise intelligence, so you don't have to feel like you're having to repeat yourself over and over again. So think of it as an orchestrator agent that delegates work and gets the job done with that shared context. So you can better understand how it works and why this needed that sophisticated, scalable architecture. I want to show it to you live. And since we are right here in New York, what better example than to put an iconic New York City brand, Tiffany, here. I'm going to log in, and this is the point where I pray to the demo gods because it's live in production here. So I've logged into Holly, and I'm a seller, and I'm going to position MongoDB as the intelligent data platform for all the AI agents at Tiffany & Co. And fingers crossed here. So what's happening behind the scenes is these agents, Holly and the sub-agents, are actually doing the steps of planning, reasoning, and figuring out the tasks that need to be performed. So as you can see here, it's identifying what are the right moves, and it's identified eight steps, eight tasks that need to be performed here. The first one being setting the goal, which the seller is trying to position MongoDB as the platform, and then doing all that account research that I mentioned they were doing manually, synthesizing all that information, creating the pain hypothesis, and consolidating it back together. I'll show you the completed product here. This is a synthesized view of what the seller receives from all of those agents acting. So you can see for this prospect account, Tiffany & Co, the goal is set — the three W's: why should Tiffany care, why MongoDB is the right choice for Tiffany, and why now is the right time for Tiffany to invest. It's done the account intelligence mining and found the pain points for this company. It's found that the problem statement is data fragmentation — how the vector and the operational data store, there is no shared operational data store, no agent memory, and why MongoDB is the right choice here. And it's generated and synthesized all these outputs into a customer-specific pitch that the seller can take when they are meeting with Tiffany. And finally, you will see here that Holly has identified six targets that you're seeing up here in a sec. It's not coming up. Oh, you see it there. Okay, perfect. So these six targets are contacts that Holly has identified — the chief digital officer — and it doesn't end there. It actually mines the enriched information about these contacts from LinkedIn and a whole set of data sources, including our CRM, and creates a targeted, personalized email that the seller can just use to outreach. So that's four to six hours of tool tax per account per deal cycle that completely disappears, and Holly becomes that end-to-end agentic experience from signal to outreach. Now none of what we just showed you was a bet we made for this event stage. It's not an AI experiment. This is live in production, being used by all our sellers at MongoDB. So we're not asking, as Ashish said, we're not asking you to trust something that we haven't tested ourselves. So Holly is customer zero. It's proof that an enterprise can build, deploy, and run thousands of agents on our own Atlas Agent Engine, and it's ready for you today. Each agent that we add onto this foundation is faster and cheaper to build. And the best part for me as CIO is the cost control and governance. Atlas Agent Engine lets us get that token spending under control through prompt caching, picking the right model for the right task, and being really disciplined about what context is being shared with each sub-agent. It has a memory layer for that shared context, so no one has to start from square one. An architecture that is open and flexible, and it's secure and governed so we can trust our agents to operate within the guardrails we set. That's Atlas Agent Engine.
And see, I told you we wouldn't ask you to trust something that we hadn't already trusted with our own business. So if your stack still looks like seven browser tabs, no shared memory, and a prayer, talk to us about Atlas Agent Engine. Deepa, thank you so much for customer zero. Thank you for building. And to close out our keynote, please welcome Dev back to the stage.
Thank you, Ashish and Deepa. Ryan, when's the forecast for Tiffany coming in? So I think you've seen what we're trying to do. The next set of speakers are going to be super fantastic. And what I am super excited — I'm not going to steal their thunder — but these are founders of some of the most cutting-edge AI companies today. So they're going to talk to you about how they're working with MongoDB and what they're doing to push the envelope to help you really deploy AI workloads. So please join me in welcoming the next group, and they're going to come on the stage right now. Thank you.
MongoDB.localenyc continues right now. Please welcome to the stage MTS host Sophia Doo.
So funny. You're the brief differently woman. Awesome. Awesome. Hello. Welcome, guys. Hello, everyone. Welcome. We have an amazing panel today, and we'll be
Discussing what it actually takes to run AI agents in production. I am joined by some amazing folks here. We have Maxim Fativ, who is the co-founder and CTO of Temporal. We're also joined by Louis Vichi, who is the co-founder of Open Router. And finally, we're joined by Walden Yan, who is the co-founder and CPO of Cognition. So, as you can see, we have some amazing folks right now on the panel who are joining us to discuss what it actually takes to run AI agents in production. So, first I wanted to get each of your perspectives because you work with some amazing enterprise customers all across the stack. And so what would you say is the biggest challenge maybe that surprised you about what it actually takes to run these AI agents at scale? I'd love to start with you.
I think the main challenge is not even technical. I think it's social. In a lot of companies, AI is done by AI teams, and AI teams are usually traditional kind of data engineers, which understand how to do ML, how to do models, data, and then they think they kind of prototype something around agents, and then they need to put stuff in production. And reality is that agents are usually very large-scale production systems, which also have failure scenarios, and these teams are not really thinking about it because it's just not in their DNA to productize something at scale. And this is where I see a lot of disconnect: that kind of type of talent and type of even interest is not about like every time you talk to them and like, okay, something can fail, or like you can have outage, you need to retry something, or what happens if this system goes down. It's just not something they even think up front. So we saw kind of a lot of examples when people put stuff in production and then, oh wow, it's a distributed system, we need to do something about it.
Yeah, Louis, I'm curious to get your perspective here.
I think that so when we reach to enterprise customer profile, I think that the bigger the bag, the less technically sophisticated the buyer is, and so they care more for the control plane and not so much the agent themselves, actually. So the buyer now is your CISO, the CTO, and the CFO is not the user of the agent, actually. So the governance part is quite crucial. Like how can we... so the CFO kind of wants two dashboards: one shows the cost went down and efficiency go up, right? So how can we classify the tasks as done and actually complete? And then the CTO wants to understand what are the workloads and how can they group them up and combine them, right? Because CTO always wants to refactor stuff, and the CISO wants to see policy, guardrails, access. And so a lot of these control planes are quite crucial to actually run them in production so that the leadership know where to, you know, like route and actually allocate resource, right? So I think that's kind of the part where, you know, like initially Open Router started as just a router layer with the API, right? And, you know, and it went very much to PLG and builders, but as we sell more and more to enterprise, as our customers get much, much larger, the need for a control plane is like paramount. And the funny story is like two years ago, right? I'm sure many of you guys did the same thing where many of you hired this guy who is the AI enabler into the company, and then the guy came in and he just put in Claude Code, you know, like even your Cursor, Devin, everything, right? He just put everything in the company, and now you have access everywhere, you have cost split across all of these platforms. There's no single point of observability, no single place for the leadership to understand, okay, the cost go up and is the efficiency go down, go up as well. So I think that's like the challenge.
Thanks. Well, then curious to hear your perspective here.
Yeah, absolutely. You know, we've seen some pretty sick demos of things you can do with AI. Some of you may know, a week or two ago, seems like forever ago, OpenAI solved an open millennium prize problem, and they just threw tens, hundreds of thousands of dollars of compute at one of these problems and managed to solve it. You know, unfortunately, a lot of our customers find that you can't just throw like $100,000 at a tough programming challenge and actually have it build production software. And the real difference is like the reason you can go solve an open math problem is because the goal is very well defined. You effectively have a test at the very end of, okay, does this proof actually check out? Most people aren't doing that with their AIs today. And so really the problem we saw is that people need their AIs to also go actually test the software that they're building and really QA it. So when we go and build Devin and use Devin, you'll find that there's a lot of emphasis on after it writes the code, it spends just as long also running the code, testing it, sending you verified proof that it works, and also setting up telemetry so that you can watch, okay, after deployment, how does it actually go out? And you need all these systems that you can set up with AI to make sure that everything you're building is actually secure, stable. But if you just throw your code at it without any of this kind of autonomous harness around it, you'll really be stuck.
Yeah, it's really clear from everything that each of you guys are saying that there's so much more when it comes to actually running an agent in production than maybe you'd see in a demo. You can see a demo, something really efficient, and go, 'Wow, this is amazing. It works great.' But then when you're trying to run it at a company, you run into all these different issues. Each of you touched upon a different issue. Louis, I wanted to talk a little bit about what happens when at one company you have different executives who care about different things. How do you actually decide what to route to? Tell me a little bit more about that problem.
Well, I think that there's a routing for... Okay, there's actually per department, each and every employee or each team might have a certain agent they care for, right? Like engineers probably use Devin or Claude Code or Cursor. Non-engineers probably using a Slack bot or, you know, like MongoDB Atlas maybe, or even like Rockbot, right? Because those are non-engineer kind of bots that people like to use, even Open Claw, which is quite dangerous. So I think that for each of those profiles, there's the agent routing which they need to understand what people are using, and then there is the tool routing which also correlates to model routing. So like with, I think for now Open Router does have a lot of data on model routing, so I can talk more about that because the agent we're starting to have more data on it, but we don't have clear session trace to like, you know, actually dig into it. But like on the model, I think that, like I just pulled a stat by the way before this chat, so if I remember correctly, most customers who spend more than $1,000 on Open Router monthly use more than three models.
So people clearly switch models because of cost optimization, right? It's very clear that there's a ton of open source models out there. Given a very clear set of tasks and workflow, what can match Opus performance, like the large model performance, because the task is quite deterministic. And over time you build an eval set, right? I mean, this is how you would tune your workflow. Like everyone can do this too, right? Just send a prompt to Claude. It's like, 'Hey Claude, here's my workflow, or go through my code, find places where I'm using an AI model and find ways to improve the prompt and evaluate.' So it's not even like a fixed eval product. It is a sub-agent. You can send Claude Code or Codex or even Devin to where it literally will look through your code or your product, find the prompt, find the model, it will build out the golden dataset based on your session log or, you know, whatever telemetry log you have, and then it will run eval. And then at the end of the day, you'll get like, okay, this workflow, you'll get, you know, like maybe a GLM or Qwen might be much better, and the error is like, I don't know, 2% compared to your golden set, and it turns out it's safe. Like we do this internally too, right? So we don't have to talk much about even external customers, but internally for our GTM team, we have this agent that was running on Sonnet and Opus and was burning like 1,600 bucks a day.
Oh my gosh. And we ran the eval, and then, you know, it just, you know, like it tweaked the prompt a little bit because, you know, a ton of time people spin up like three skills when it should be just one single skill with like three steps, right? So like it turned into a single skill. So reduced three invocations to two invocations, and then it changed the model to like GLM, and the cost went down to like maximum 80 bucks a day, and the amount of workload is the exact same or even more as people piling up more work to it.
No, it's super fascinating because as we talk about agents who are starting to run even longer horizon tasks, you run into more and more issues. Even as we talk about alignment and security, a lot of these issues come from being run in long horizon environments. And that's why, Maxim, I really wanted to get your perspective here because you care a lot about durable execution. That's one of the core principles of what you guys are working on with Temporal. And so when it comes to agents that run long horizon, how do you actually ensure durable execution? What is the most important thing to think about when it comes to security?
So security is important independently of horizon, I would say. So let's start from what durable execution is. For some of you who never heard about it, the idea is that you preserve full state of your code execution all the time. So if your process crashes, we just move that execution to another machine automatically, and your code keeps running as nothing happened. So if you block on an API call and the process crashes and the reply comes back three days later, it's still relevant and it still will be blocked on the same API call and it will just continue executing. From a developer point of view, it just waited for three days. Imagine if you have a tool call which is beyond a few seconds. Traditionally, you can still have an open HTTP connection. So if your process crashes, you will lose that tool call. And if this tool call is a payment, you probably don't want to be in that situation. So as we are moving from agents which are just doing queries and practically not performing real work, we need to start thinking about: do I really want to lose tool calls, for example? Do I want to only have tool calls which are short-lived? And every short-lived call which requires retries becomes long-lived because something can be down. And the moment you run millions of those agents and any failure happens once in millions, you end up with a situation when you have failure all the time. And then you need to start thinking about compensations because if you do a money transfer, for example, using an agent and then it failed, you really want to make sure that because, for example, you cannot deposit to another account, you want to run a compensation flow. And agents don't do it by themselves. So you need to make sure that the harness and the kind of surrounding code knows which actions should be rolled back. It needs to make sure that these actions are recovered, and you not only have visibility but also proactively can fix those things. And then going to security is the same thing. Security right now is mostly focused around agents, but an agent is not just an agentic loop. Agents are really much more than that. Practically, for example, a lot of people call agents just static flows which call LLMs. But there is no clear way to secure code, especially agent-generated code these days, because the harness can call guardrails, but what about code which is generated by an agent and is not part of the harness? And this is what we're kind of trying to provide. We are providing ways to actually secure even agent-generated code, and which means by secure, we provide the ability to put guardrails on that and put humans in the loop or security scanners which will validate any command and action before it's even executed, because the traditional sandbox approach does it only after the command is already running. It just can only intercept a call and kind of cut it, but it cannot do it proactively.
Yeah. Walden, I see you nodding your head here. Anything you wanted to add?
Definitely. I mean, we think a lot about security. I actually have a pretty scary story from the other day where I was running an agent on my machine. I was telling it to do some data analysis. It comes back, shows me some beautiful graphs. You know what you basically expect. And then I had this moment where I was like, wait a second, I never gave it the keys to pull this data. How did it have access to the store database and pull this analytics? So I asked it, hey, can you tell me, you know, how did you get this information? And it turns out it actually read through my laptop's clipboard history. It had found when I had copied the secret key from my admin dashboard the previous day and had used that without telling me to go get the information. We have incredibly persistent agents today. They can hack into governments. They can hack into, you know, live companies. So if you just give them even a simple task, they won't stop at anything to get access to what you need. And so this was—I wasn't using Devin that day—and this reminded me why we do what we do at Devin, which is actually create cloud sandboxes where nothing sits on the sandbox that you don't want it to have. It's all network controlled and gated. All these controls are super important. You guys are safe because you guys use Devin, and this type of stuff won't happen to you. And I actually think we under-index on classical controls and classical systems to stop AI from going rogue and breaking into things. And on the note of classical systems, yesterday we announced a partnership with MongoDB. You guys can clap that. Yeah. A lot of the inspiration for this partnership is there are a ton of companies on old systems and they want to modernize over to MongoDB. This is not a thing that you can just say, hey, AI, just go do this entire thing for me. There's an incredible suite of classical tools that the team has built out to help you understand your databases, what is necessary to migrate. And when we're partnering with them, we're actually helping them figure out how do you let Devin use these classical tools that MongoDB has built to do this migration. And I really think without those classical controls, you're not going to have the confidence you need to actually do the full end process. So I'm super excited about that.
Yeah. Interestingly, I actually had the exact same experience with what he had, where the agent somehow found a production key to like look up customer data, which is quite scary. And it actually happened with the Devin cloud because this is the environment we set on Devin, right? So because even if you set secrets on Devin, sometimes you can set multiple secrets and some of them might, you know, some engineer might add a production key, which is like, I think there's still a factor of human error that's still heavily accounted for, which I think to me, security is, or with anything, right, you have to track it, then you can improve it. There's no way to improve anything without tracking. So I think it's very important for a CISO or for, you know, like security team to have like a clear backlog or traces of all the network calls, right? You can put in a sandbox, but the moment you get it, you know, like connect to the external system, you need every single network call to understand what the hell it's doing and what key and permission it is using. I think it's very paramount. And then you can set up alerts, right? Like sure, it will be noisy at first, but that's the whole point of alerts.
Totally. And I think another human error that's quite interesting though is, I think that's to what Max was talking about, is actually the humans themselves are telling the agents, right, to do things, and then sometimes it can be quite... it's not a malicious, you know, problem, but to the means to solve it, sometimes it breaches into data that you don't want to access. Like, you know, like for some reason they might need data of all the other customers that's similar to this customer you're trying to serve, right? That's not quite good because now you have breaching into, you know, like customer data, another customer's data to serve, you know, your current customer. So I think there's a whole like AI guardrail that's needed at the agent level.
Yeah. And you guys are at the frontier and you're willing to do that experiment of let's give it a bunch of permissions and see what we have to remove later. And there's a lot of orgs you work with that go the other way, which is let's start with no network permissions, no secrets on the box, and incrementally add things. And so you can kind of see where, I mean, earlier you were talking about governance, who can do what with your agents. You know, some orgs that are at the frontier, you let anyone do anything with your agents, and then other orgs that we work with, they'll say, okay, only administrators and managers can put things on the machine. And so really depending on like how aggressively your org wants to test these things, you can kind of set different permissions there.
I think for the most part, slowly we are... I think the best way to treat agents is almost like think of them as a human being who is not smart yet, right? The model is still not good. So I think the bottleneck is still the model because you can see the jump pretty clearly between like, you know, o3.7 to o4 to now us 5, right? You can do more and you can actually remove more of your code, but it actually is, I think, the human loop that needs to control the system better.
