So we're very excited about the power of AI agents, particularly. We started in June 2026, and our interest was driven by recent papers from DeepMind, FunSearch, which demonstrated early on that you can use AI to improve the state-of-the-art performance of algorithms. Then was followed by AlphaEvolve showing that coding agents can be used to improve existing systems and techniques. And then OpenEvolve was an open source version of AlphaEvolve which was released around the same time. And obviously we have all these kind of agents, coding agents like Plot Codex and Cursor. So, you know, triggered by this kind of events, we started to run a special seminar over the summer, last summer at Berkeley, where the students were using these AI evolutionary agents and applied them to their own systems, to their own research. In particular, we went through 12 different projects here. And the results were quite impressive or better than expected. So these 12 projects included around seven published work by the students in top conferences in systems or AI, included also some work in progress, and some research just published on arXiv. And in 10 out of these 12 cases, we got results which are better than SOTA. And not only that, but it was very cheap to use, you know, the best models then, Gemini and OpenAI GPT, to get the results. So it was quite cheap and quite fast. So this is a lot of excitement, and also since then, we've done a lot of work in the space. In particular, we taught two graduate courses the previous two semesters, we wrote position papers, we launched the ADRS blog series where ADRS stands for AI-driven research for systems, and of course we like we are from academia, so we published and we built a bunch of systems and published a bunch of papers. So everything started with great expectations, and the expectation was that AI will disrupt systems research the way we know it. So what happened actually over the past year? What was our experience? What was the reality?
Now, if you take a step back, the key goal of an AI coding agent is to develop a software system that meets a certain level of assurance with acceptable levels of risks as fast as possible and as cheaply as possible. Here, the software assurance is an ongoing process to establish and maintain some justified confidence, which is supported by credible evidence from running the software, that the software system will satisfy some specific claims, will have some specific properties in its deployment environment. So fundamentally, you need to do that by using the resources at your disposal, and the resources come in three forms: humans, agents, AI agents, and of course compute. So the question is how to achieve our goal by best using these resources. And if you think about this as a framework or a system, then the key question to answer is about what are the bottlenecks, and then once you identify the bottlenecks, you alleviate them or you remove them if you can, and then you are going to speed up the development.
So let me start with the motivating examples. So this is work we done recently, and the goal here is to synthesize a key value store system from scratch. And you know, as everyone knows, it's a pretty simple system, pretty simple interface actually where you associate a key with a value and you put that pair key value in the storage, and then you can given the key retrieve the value. In this particular case, it was a pretty simple setting. We have only one server, so no distributed environment, although the server was multi-threaded. So you have concurrency. So basically you store the data on the disk and cache in memory to retrieve it faster, and the goal is to maximize the throughput. And to evaluate it, we use YCSB which is a well-known benchmark for such workloads. So now in the process of using these evolutionary AI agents to synthesize the solution, we got one solution at some point which was six times faster than all the previous solutions. A huge improvement. So why was that, right? Of course, you are curious to understand what led to such a big improvement. Well, it turns out that these benchmarks generated values in a predictable way. So when you generate the value in the benchmark, you know, you need to say what the value is. And in this particular case, it generated the value by hashing from the key associated with that value and a seed. Now the agent found this out. So, and instead of storing the values, it basically computed the values using this predictive function on the fly. Which meant that because you don't need to store up the values, you only need to store up only the keys, you can store a lot more items in memory. So, the retrieve is going to be much faster. Hence the improvement we've seen. Now everyone knows this is kind of, you know, reward hacking.
So if you look about this reward hacking, you see it is actually due to two gaps, and this is the main topic of my talk. One gap we call it requirement gap. So you see in this case the agent in the specification was required to return a value which was put associated with a particular key. However, the user intent, the stakeholders we are going to call intent, was to store that value, right? Not only to return that value. So here is a gap. And then there is a model gap in this particular based on the workload in the development versus deployment environment. In the benchmark in the development environment, it contains predictable values. However, in the real world deployment, you have arbitrary values. So here is another gap. So now how is a solution typically working in this case? So let's just take a step back here. So you have a bunch of requirements which basically describe the system needs to be built, performance goals, and so forth. And you have a model which tries to abstract away the deployment environment. And this is about where in the environment you are going to target the deployment, the workload which you think is representative of the real workload, kind of the failures and things like that. And now a developer or an AI agent takes this requirements and the model and generates a program, and then this program is evaluated, right? And evaluated you can have unit tests, end-to-end tests, integration tests, or you can have contracts or formal methods. Now if it passes this evaluation, if it is a evaluation, sorry it doesn't pass, then you provide the feedback to the developer and to the AI agent, and then you are going to continue in a loop until the program passes the evaluation, and then you are going to deploy it. The problem is again that the requirements and the models are abstractions on one hand of the user intent, what the user wanted the system to do, and the real world where the system is going to be deployed, and these are the delta between those are the requirement gaps you are talking about. So more being a little bit more specific, the requirement gap is there because the intent of the users is broader than the requirement. And here are some examples. There are omissions, things that the system should do but is not specified, like store arbitrary client values. There are exclusions like things system should not do and are not specified, like never expose customer data. Trade-offs like for instance a search has a 200 millisecond SLA; if this SLA is exceeded, should you return partial results or should you return a failure? And there are conflicts which are not specified how to be solved, like for instance you want personalization but this chat history, but privacy rules prohibit retaining the chat history; how you are going to resolve this trade-off, it has to be specified. Now the model gap. It's the same you have for the same reason because the real world is broader than the model, the model gap. So examples are the workload predictable for instance the workloads have predictable benchmarks during the development versus arbitrary values during the deployment. You have failures like fail-stop tests during the testing and Byzantine faults during the deployment. You have dependencies like stable mocks versus changing the APIs in the deployment case, or you can have adversarial drift new attacks or workloads which happen after release.
So these two gaps are responsible for the reward hacking, and the reward hacking is arguably responsible for many of the things you have seen recently in the news, like these agents escaping their environments and going rogue. It's also you can trace them back to alignments or misalignments and hallucinations. And a key challenge here is about under-representation. And, you know, because you can only judge from available evidence and the evidence is often incomplete. New evidence might invalidate acceptance of a program P during deployment. Examples are implementation P developed under the model M which assumes first of failures, then you deploy this in practice and as long as you have only face of failures in the deployment environment, you are good. The stakeholder accepts the implementation, but once you get a failure like a Byzantine failure like a corrupted memory, that's no longer acceptable. Now why is closing these requirements gaps? So, you know, gaps are hard. First of all, intent might be tacit, contested, or changing. Fundamentally, you don't know what you don't know, right? You cannot think about all the corner or future cases until you see them. Right. And this has been noticed for the past decades in many domains: software engineering, AI, economics with contracts, politics, and so forth. And you can also think about this from a pure solution perspective. If you think about how you can close them, well, one way is to ask people whenever there is a material change on the deployment, you can ask people is this good or not? Or you can create an oracle, a model or a simulator that predicts every material judgment of the stakeholder. The same is true for the model. If you really think about you want a good model, you need to prove that every omitted aspect from the real world in the model, what the model misses, is irrelevant for acceptability of that implementation and you keep the claim, and the other one this has to, the claims have to be valid as the world changes. Okay. And again, has been a lot of work here over the past decades pointing out to this point and this kind of fundamental gap between the model using development and the real world. Okay. So again here to illustrate why it's hard, you can ask people for instance every time on is a changing environment to say okay is this change materially important for the model? Or you can predict, you can simulate the real world, right? But obviously all this direct giving the user direct access and to or complete having complete simulations are not feasible actually. Even if you have the direct access, it's on one hand, the cost and latency can be prohibitive. But on the other hand, it's like you cannot you only can reason for the existing evidence. You cannot reason for things you have never seen, you haven't thought about. And obviously for the deployment, for the real world, you can't have a complete simulation. You can seem to have a simulation. But if once you have the simulation, that's another model. So you are back in square one because you need to demonstrate that the simulation there is no gap between that simulation and the real world.
Now the other thing I want to make and is that now why we are talking about this because we have been, you know, like I said these gaps are there and have been there for decades, people identify them, recognize them. Well, one reason we are emphasizing this because agents amplify the gaps, and these are for several reasons. Is context asymmetry. A developer typically has more local and system and organizational context about the task than the agent. The agent which brings broader context but has less specific context about that task. The other one which is very important: AI iterates much faster under a fixed requirements model and a fixed evaluator, which means they discover and exploit exploitations much faster. Actually, you know, the developers typically give up or they are not, you know, to find any of this exploitation even if they want, but the model because AI agent is much faster, they are going to find it what humans will never find. And is deployment scale especially if you are going now to have an automatic deployment of these programs in production. Um, let me skip now. I want to say about there is a third gap and which I want to mention is and this is between the evaluator and the requirement on one side and requirement and model on the other side. So the question this gap, you know, illustrated by illustrate this gap is that you have an evaluator. So by passing this evaluator does it guarantee that the implementation is going to satisfy our requirements under model, right, in all possible executions? And in general the answer is no because for instance you use kind of unit tests and so forth. So it's only a subset of tests you can points you can show that for these points, for these data points, for these values, you satisfy the requirements under RAM, but actually there are ways you can do it and you close the gap, and this is formal methods, and that's why we are not focusing as much on this gap right now. However, the reason I'm mentioning this gap is because using the formal specs actually can widen requirements and model gaps because the formal specs are going to be less expressive say than kind of natural language written requirements and the models.
Um so finally, you know, like I will say, you know, does narrowing these gaps help? Uh yes, most actually most successes reported so far are in domain with narrower gaps and there are significant work ongoing work to narrow gaps in other domains and here are some examples. Formal mathematics. In formal mathematics, model gaps is closed because there you have formal semantics, you have axiomatic systems which fully define the world in which the properties have to be, the theorem has to be true. Requirement gap still remains because informal to formal translation still remains, capturing what is your intent in a theorem and so forth. Hardware systems, you know, for hardware systems you have instruction set architecture, you have RTL which bounds the relevant behavior, of course it doesn't close a gap to physical effects which are remaining outside the model. Strong verification also may close the evaluation gap. System optimization, we are talking about system optimization versus system synthesis. With system optimization, you are going to be anchored in a code and in this code is going to capture the interfaces, API, functionality. So it's kind of narrowing model and requirement gaps as compared with the system synthesis who lacks this anchor and must infer more from incomplete requirements and models. And finally, world models, right? World models try to achieve is to narrow the model gap by learning the models part of the model by interacting with the deployment work. Okay. So two more minutes and I'll be done. So, you know, so how can you close this gap in general? Well, to close this gap in general, you need another loop which actually includes the intent, the user intent and the real world. And this is the first step here is to have to observe and try to detect what are kind of the issues, unexpected issues, misbehaviors. Then you need to diagnose them. Then you need to revise, and you are going to revise the requirements. Hopefully improve some requirements, improve the models, improve the evaluators, which should reduce the gaps, and you do this until you are going to achieve your level of assurance and risks. And let me get back because I'm, so, you know, skip a few slides. So we've done quite a bit of work in this area, and but of course much more remains to be done. And here are some pieces of work we've done, for instance for the loop for development loop, evolutionary loop for AI agents, for reducing the evaluation gap, for diagnosing, you know, tracing the intent and from the traces which represent a user interaction with the systems or the agent interaction with the real world and obviously directly trying to reduce, to narrow the requirement, the model gaps. So what are the takeaways? So here the takeaways from this presentation. First, the model and requirement gaps are fundamental and cannot be certified as closed in an open and changing environment. Again, because you don't know what you don't know. So therefore, you can only narrow them. The second one is like the humans will remain in the assurance loop because they are needed at least to narrow the requirement gap because they own the intent and likely because they are going to be in this assurance loop, they are going to be the bottleneck. And the evaluation gap can be closed using formal methods but the cost can be increasing the model and requirement gaps because the lack or the limited expressivity of the specification which are needed by the formal methods. Finally, these are not new problems like I mentioned. However, AI agents exacerbate them because of the context asymmetry, speed and scale. They are going to find and exploit these kind of gaps much better, much faster than humans. So therefore, what do we need to do? We need to use the best practices in software engineering and security and to do to we need to be more cautious and meticulous than ever before, and of course do a lot of more and interesting research to close these gaps. Thank you.
