I'm going to talk today about some very recent work we have done. There is no published paper. There is no blog. As a result, you should expect some rough edges. So let me start.
Today AI coding agents are generating code and tests at order of magnitude faster and cheaper than any humans can do. Unfortunately, the failures are piling up as fast.
In this talk, I am going to make three points. First, I am going to show that despite these failures appearing to be different, they share a common underlying structure. Second, I will discuss the fundamental challenges they raise and finally suggest some approaches to address these challenges. And I am going to start with a story, our agentic story.
So as many others, we are inspired by some results which came last year, more than last year, one year ago from DeepMind. The FunSearch paper showed that you can use agentic loops to improve the state-of-the-art algorithms. Further, after a few months, Alpha Evolve in June 2026 showed that you can use agentic loops to also optimize systems. And a few weeks after that, Open Evolve, an open source version of Alpha Evolve was released and of course even then these agentic coding systems were becoming more and more powerful.
So what we done, that you know, I'm a faculty so I can use the students. We did a seminar over the summer and we asked the students to use these tools for their own work, not to beat a particular benchmark but to use these tools for their work.
So we picked 12 projects. Some of them were papers they recently submitted at top conferences, systems, databases, networking and AI. A few of them are work in progress and or just submitted papers. And the results for us are astonishing. In 10 out of 12 cases, these tools are able to improve on their results, on the state of their results. And also this was quite cheap. We only spent a few tens of dollars on each for each of these projects.
So obviously we are very excited about that. And since then we did quite a lot. I co-taught two graduate classes at Berkeley. We wrote position papers. We launched this blog series. ADRS stands for AI-driven research on systems. And obviously we wrote a lot of papers. We built a lot of systems to improve this kind of agentic loop like you know, Japa, maybe some of you use it, ad evolve, evex and many more.
So it started with this very great, great expectations that AI will disrupt the way we do system research. Now what is the reality? Well the reality is that AI significantly improved our productivity of doing research. It did become a big part of our research like I've shown you all this paper. However, did it disrupt the way we do research? Did it allow us to build new systems from scratch which were never built before? Did it allow us to come up with new research ideas otherwise you couldn't have come? Okay. So, not really.
And to give you an example of why was that, let me start with a recent work we have done. And this work is about synthesizing from scratch, not optimizing a system, building from scratch, a new system. It's a simple system, a key value store. Many of you work and maybe build systems. It has two primitives. You can insert a value associated with a key. And given a key, you retrieve the associated value. And we took a pretty simple setting, not distributed, only one node, multi-threaded, store data on the disk, cache it in memory, maximize throughput, and for evaluating we use this kind of traditional benchmark, Yahoo Cloud Serving Benchmark or YCSB to evaluate it.
The way you build this kind of systems is the same way even when you use agents like we did for decades. It's a loop which we call implementation verification loop in which a developer or agents writes a program and then submits to an evaluator. An evaluator can be like a test suite and the evaluator says whether the program passes or not. If it doesn't pass, you are going to iterate until it passes. It's accepted by the evaluator, for instance passes a test, and once it passes a test, you are going to deploy it to production. Right? That's it.
Of course, when you develop the program, you start from some bunch of requirements like what you are given by your PM in the product requirements document describing the system, describing in our case for instance what are the consistency semantics of the data, performance goals and so forth. Also, when you write the test for instance, evaluator, you have to assume some kind of environment right which is development environment which you hope is a good reflection of the real world. Like in this case you are going to have multi-ready server, you are going to the workload right, you are going to use what kind of failures you expect to encounter and so forth.
Okay so now what happens is that one solution which was generated by this agent was six times faster obviously when you than everything else. So obviously when that happens you try to understand why, what is magic about it. Well it turns out that these benchmarks basically generate values by computing a hash on the key and a seed. Now the agent found this out and stopped storing the values. When it gets a get for a particular key, it uses a hash to generate the value on the fly. And now because you don't need to store the values, you can cache a lot more keys in memory. So you're going to get a lot more hits. So you're going to be much faster. Now this is what is known as reward hacking.
And fundamentally it happens because there are these two gaps which are one of the key topics of my talk. It's a gap between the intent of the users or stakeholders who wrote the requirements and her intent. In this case, the agent was required to return a put value and it did exactly that. All the tests passed. However, the user intent was to store that value. Also it's a second gap is kind of an environment gap because the benchmark which are given to the agent are predictable, the values are predictable but in real world they are not. The values which are provided are supplied by a client are arbitrary. And here I'm kind of depicting the customer intent, the real world and requirements and environments are an approximation of intent and this real world respectively. And between them you are going to have these gaps.
So why do you have the requirement gap? The requirement gap exists because the intent is broader than the requirements. For instance from the requirements you may miss positive assertion like store arbitrary client values or negative assertion never expose customer data or say there is a trade-off and you don't specify how that trade-off should be resolved. Say you have an SLA of 200 milliseconds and now you have a request which hasn't returned in 200 milliseconds. What you are going to do? Are you going to wait? Are you going to return a partial value or return a failure? Or there can be conflicts we need to specify. Maybe you haven't specified how to be resolved. Like for instance, you want personalization, you need chat history. However, maybe you have a privacy rule which prohibit keeping that history.
The environment gap exists for the same reasons. The real world is just broader. Example in our case you have like workload, the predictable workload we provided in the environment but in the real world is not. Or failures, say you are in development environment you assume certain failures but in the real world you may get this kind of Byzantine failures like for instance the memory being corrupted or dependencies. You often are going to develop against a particular API, you are going to have a mock API however in real world you may have rate limits right, you may that API may change or you can see new attacks and the or may evolve in the way you didn't anticipate.
Okay now the next claim I'm going to make that these two gaps are responsible for the main challenges you see today in the agentic systems. One is reward hacking and the other one is hallucination. The reward hacking exploits omissions in the environment and this requirement gaps, things which are not specified. Right? A requirement in intent in the user intent is missing from the requirement document like the intent of storing the value or a condition from the real world is missing in the environment like the real world clients submit arbitrary values.
In contrast, hallucinations expand these gaps by manufacturing things which are not in environment or the requirements. Right? For instance, it can hallucinate by adding a requirement like duplicate refunds request shouldn't be dropped, something that the user may never have thought about or it can hallucinate and add non-existing APIs. Right?
These gaps also can be used to explain some of the recently high-profile breaches. In this you read over the past few weeks in this OpenAI Hugging Face example. The agent was running a cyber capability evaluation on the exploit gym benchmark and somehow the agent reached a production system and stole the test solutions. This is reward hacking and an environmental gap because here we implicitly assume that the agent doesn't have access to the solutions right otherwise what sense has the exercise right? And but it didn't, this actually the agent had access to the solution somehow.
The second example is this kind of cloud, you know, Anthropic published that and it was a big deal about which three cloud agents run security evaluations and they were told they have no internet access. However, there was a sandbox that actually connected online. So they leverage that sandbox to bridge three organizations. This again they were hacking because the assumption is not connected to the internet but it was connected to the internet. Actually in this case there is also an example of hallucination because the agents added this kind of requirement that the real systems were part of the exercise. So then convince themselves they continue hacking.
Now these gaps are not new. They are in the software engineering field for decades. Okay. What is new is that the agents make them much more consequential, worse, much worse and this happens for three reasons. One there is kind of the sometimes missing context. A developer typically has more local context about the system, about the organizations or context accumulated over years of experience what we call common sense. The second one, agents are relentless. They generate code order of magnitude faster than humans. So if there is a gap, they are going to find it. A human much slower. Even if they wants to exploit a gap, they may not find the gap. And finally, if we do automatic deployment at scale, for instance, accept changes and deploy it. All of this impact, negative impact of the gaps will amplify right and can spread before you catch it.
Now you may ask okay these gaps whatever you know like okay maybe I believe you so just close them. Well, it turns out that closing these gaps is fundamentally hard. It has been a long line of research not only in software engineering, in political sciences, in kind of and economics, right? So why is closing gaps is hard? Let me give you some examples here.
Would you spec, would you if you design a key value store would you think to specify that it should store the values? Now you know you should. We didn't. Why? Because it's called key value store. So store means that you need to store things there. Here is another example. You design a load balancer. Would you think to tell to say that the solution should actually process a request? Maybe not because that's a function of the load balancers. However, in a when we optimize an expert parallel load balancer actually one of the solutions in order to increase the throughput figure that a good way to do that is by dropping the request.
Or think about vacuum. You design the software for a vacuum and use AI agents. Would you think to specify that he shouldn't spill garbage on the floor? Why would you? That's the definition of the vacuum. He should get the garbage from the floor, pick it up. However, if you give to the agent kind of like the goal to maximize how much garbage you pick from the floor, then a solution, easy solution is to dump the garbage, pick it up and do it again. It certainly is going to maximize the amount of garbage picked from the floor.
Okay. So now what are the possible solutions? Let's take from the other side. One you can ask the user, the stakeholders, whenever a new outcome, an action creates an outcome in the real world, judge it, say it's good or bad. The other one is enumerate all possible outcomes of all the executions in the environment and the associated judgments and then you check okay. Or another one is to have a simulator. You simulate the stakeholder, the human right at least for what is relevant for the problem. And that model, you know, that build a model that predicts every user judgment for every possible relevant outcome. Of course, this is generally infeasible and even if you do that, it's not enough because the intent can change in time.
Like for instance, you design a file system and for instance, you have a bulk delete operation and it deletes a bunch of files by mistake. What you are going to do? So you are going to modify your intent and next you are going to say well I want to be prompted before such the operation can proceed.
The same goes for the environment gap. It's a long line of research. And to give you a sense about why it's hard to close consider this kind of example. This is a 1985 paper the limits of correctness. And this is the opening paragraph. It says on October 5th, 1960, an American ballistic missile early warning which was stationed in Thule Greenland indicated a large contingent of Soviet missiles headed towards the United States. Fortunately, common sense prevailed as the informal threat assessment conference that was immediately convened decided not to launch a counter strike, right? because it's a new system, the political environment actually was kind of was better between the US and Soviet Union and so forth.
So what was the problem? Any guesses? It's unfair question. The problem was the moon had risen and was reflecting radar signals back to earth. The main point here is that it's kind of this lunar reflection for a particular position of the moon hasn't been predicted by the system designers. Again, if you want to close this gap, you can enumerate all conditions in the real world that might affect the outcome. Or you can have a simulator of the real world or the model of the real world. Again, this is generally infeasible, right? At least we don't know how to do it now. And again, the world is open and changing.
Actually, there is a third gap which is called evaluation gap. And this evaluation gap is basically it's a question is that does the evaluator acceptance of the program guarantees that that program is going in all for all outcomes for all executions going to satisfy the requirements. In general the answer is no because when you do for instance you test test suite you only generate you only evaluate you only test for certain inputs right you cannot say much about the inputs for which you didn't test right you don't have a proof for that.
Now this gap can be closed if you use formal methods because it's formal method if you have a requirement formal specification for requirements and for the environment you can generate the code and the proof that the code satisfies the requirements. However, when you do that you may increase other gaps because when you specify when you use formal specs for the requirements or environment you are going to constrain compared to the natural language. So therefore the gap, the requirement gap and the environment gap may grow.
Now another question, okay I'm telling all these gaps and so forth are they really important? Well I cannot give you a proof but what I can tell you that the domains in that agentic loops have most success are the ones which have narrow gaps.
One example is math right you heard about all these kind of new theorems which are proven by the first time with the help of AI. In this case there is no environment gap because the real world is a mathematical system within you prove the particular result so the environment and the real world are the same. In this case there is still a requirement gap because you need to translate a mathematician what it's in head into a theorem. Or think about hardware systems, you have narrower gaps because a word is a instruction set architecture and it's kind of it's pretty bounded you know you don't bound the physical effects but it's still much narrow gap and also you have strong verification method in hardware which may close or at least narrow significantly the evaluation gap.
Or system optimization versus system synthesis. It turns out that optimizing the systems is much easier than building system from scratch. Why is that? Because with optimizing the system, you have the system. The system itself, the code you have anchors, a bunch of things like interfaces, edge cases and so forth. If you synthesize a system, you need to specify explicitly everything.
And finally, everyone lot of discussions about the word models. But what is a word model? Right? A word model is basically training a model to reflect more accurately the real world. Right? So it's closing that environment gap.
So how can we narrow? You cannot close these gaps at least requirements and the environment gap. What can you do about them? Of course you only thing you can do is narrow them. So how can you narrow them? Well, in this case, you are going to have a second loop which already exists there in some shape or form in which you have the stakeholder, the user observe the outcome and if the outcome violates its intent is going to diagnose it and is going to revise the artifacts in the verification implementation loop. May add a new requirement, may make a more realistic environment or update the evaluator.
Okay. Now these two gaps are mapped on two bottlenecks. The first requirement gap maps on a human bottleneck. Why? Because the intent belongs to humans. So they have to be in the loop to close that requirement gap. The environment gap maps to kind of more of a compute bottleneck. Why? Because if you want to have an environment which is high fidelity reflects the real world can be expensive. You can always deploy it in the real world and so forth although it can be also expensive for other reasons and can take a lot of time.
Assume that you build a database or your distributed systems may take hours to test it. So even if you have only 100 iterations in an agentic loop it may take you over one month. So here there is inherent tradeoff between the fidelity or the gap size and the overhead. So there are a bunch of direction here, future directions about how can you improve things.
Well you know a few of them is like for instance obviously the human is a bottleneck you try to alleviate the bottleneck. How you do that? You are going only to use humans for the most consequential to answer most consequential questions. Like for instance, is this particular behavior consistent with your intent? What is trade-off is acceptable? What consequence or exposure are acceptable? Right? Like for instance, you may want to deploy first the system to your lowest tier customers and only to a percentage of those, right? Other things are besides selective escalation, iterative disambiguation. You can have an agent looking at the requirements try to do the implementation and whenever it detects some ambiguity some gaps you may ask the user hey you know what about this try to close the gap.
It can be you know that you remember about this trade-off between the fidelity and the size of the gap and the overhead for the evaluator. Well, you can do things like cascading evaluation. The evaluator can start with pretty approximate course approximation of the real world which are very low overhead like analytical models. Then you can go to simulators, emulators and real system itself. And then you have this kind of impact bounded control which is about to expand the exposure in the system. So to roll out to more and more users as your confidence grows. And the last one is continuous learning. Another hot topic. But if you think about what continuous learning is about, it's about continuously learning about the world. Continuous the real world and about the users about the user intent. It is about narrowing these gaps. And obviously for all of this you should use agents aggressively.
Now over the past year we have done a lot of work at different areas of this kind of two loops frameworks. The first one is verification implementation inner loop. The second one we call assurance revision loop is the outer loop. New agentic loops like I mentioned to you like japa, evolve, evex, we close this kind of evaluation gap by using formal method. And for instance on the right hand side you can see when you get how do you get information evidence outcomes from the systems you get logs in many cases but these logs can have thousands and thousands of lines. So how can you extract the information we have this kind of system to automatically taxonomize the failures and make it much easier to understand not only by the human by the agents.
And I'm going to end with this example which is we have this kind of experimental internal AnyScale optimizer. And here what you see on the right hand side we have a bunch of tasks and on the last column you see what are the results improving the performance of this kind of workloads by using the latest version of cloud and everything which is available publicly, documentation, blogs, GitHub issues, everything. And in the middle the AnyScale, the blue column AnyScale optimizer it shows the improvements. If you also use the data collected by AnyScale about a variety of workloads it has seen so far right as you can see the AnyScale optimizer is doing much better. So of course the difference is this data new additional data use but fundamentally what does what the data does it adds more information about the real world in which these workloads are going to be deployed right and are running. So they close, not they close, they narrow this environment gap.
Okay. So what are the takeaways? First, environment and requirement gaps are fundamental and cannot generally be closed, probably closed especially in open and changing environments. We can only narrow them. There is a third gap which is kind of evaluation gap which can be closed by using formal method but at the cost it's expensive sometimes it's hard to use and can increase the environment and the requirement gap as we discussed. The third one is that you see the humans are needed to close the requirement gap. So I would expect that for foreseeable future the humans to remain in this kind of outer assurance and revision loop.
Finally what are the what you can do? What are the actionable items so to speak? The first time I think that you know AI is amazing. It's very powerful. However, it comes with challenges like we've seen hallucination, reward hacking and so forth. So, one thing you can do is to assume that AI generated code is malicious by default. And the good thing about it, there are a lot of best practices developed over decades, safety, security and so forth, defense programming and so forth you can use and you should use, right? You don't almost need to invent much. Just use the best practices.
So, the last one which I cannot go back unfortunately. Okay. So, the last one is basically if you want to make your agents more and more useful, right? I think one thing to focus on is on narrowing that gaps. Yes, these gaps are hard to narrow but they are the most consequential to improve the performance of your agent. Thank you.
