Hello everyone, and very glad to be here. Thanks for having me. So, I'm going to talk next about some work we have done recently at Berkeley, and it's about reliability in AI in general, and in particular reliability for agentic systems. So, first of all, what is reliability? Right? And again, there are many definitions. So, I pick the one which is on NIST website. This is National Institute of Standards. And what it says that the reliability is the ability of a system or a component to function under stated condition for a specified period of time. Okay? And what does it mean to function? It's to meet some properties like accuracy, correctness, consistency, predictability, robustness, and safety.
So now, the reliability in AI has received a lot of attention over the years. This is a paper coming up of those of Stanford. Dario Amodei saying in an interview 1 year and something ago that it's, you know, actually think the most challenging thing about AI agent is making sure they are safe, reliable, and predictable. And in the same interview, he's basically saying that it's what holding us back are not the model capabilities. And this is over 1 year ago. Of course, we have a lot of improvements in the model capability in the past year and so. But what's holding us back is the fact that what we are going to deploy in productions is going to work reliably. And this is some state-of-the-agent engineering published recently by LangChain. The same thing is saying that as we enter 2026, organizations are no longer asking whether to build agents, but rather how to deploy them reliably and efficiently at scale. And a few others kind of headlines. The last one is a little bit over 1 month ago. We also have done with collaboration with some of the partners, IBM and others, a study of the agent in production. And we wrote this paper, which you can find it on archive, and we look at 86 deployments over 20 different use cases and five domains. And it's the same conclusion when you interview people, they're basically saying, you know, reliability remains the top development challenge.
So, now this is not something new, right? We had this in systems forever. It's software engineering. You have that everywhere, right? For your fridge, for your car, and so forth. Reliability is a big thing. And the one thing I want to note in software engineering is like you first build a prototype to demonstrate the capabilities. And then you have to productize this prototype. Now, it turns out that the most of the work is going to productization. It's anywhere between 10 and 50x. So, this is the famous book by Fred Brooks, The Mythical Man-Month, which is basically saying from product prototype to production you need to spend nine times more effort. Okay? Now, this was in '60s, '70s, so smaller code base, maybe people are more careful, but there are more recent code, so to speak. Here is John Miller, which was architect, engineer at all these kind of top large companies like Amazon, Google, Microsoft. And he was saying basically that going from prototype to production, it takes the effort is over 50 times more than just building the prototype. So, why am I telling you this? What does it mean to go from prototype to production? It is basically making that software artifact reliable, right? Meaning that it has to work for any user, every user input, in all kind of use cases, maybe on different kind of hardwares, all of this, right? And to just work, like Steve Jobs used to say, right? Now, going into more details, what does it mean to productize a feature? You need to test it, debug it, you need to care about compliance, governance, when you deploy it, you need to have observability, and all of this stuff, right?
And if you are going to go to testing and debugging, it's like how do you do it, right? Typically, a system consists of small multiple components, then you try to figure out which components, where is the error, and then to figure out what the error is, make the changes, and then hopefully it's going to work. Now, the one thing which is very important, and I'm going to come back towards the end, is that you can do that because each of these components, as well as the entire system, you have very clear expectation what it's going to do. Okay? So, when it's an error, when you have a deviation from these expectations, and that expectation is specification, right? Always you have that. Now, the question is about how precise it is. Because the specification can be in the form of PRD, the product requirement documents, or formal specification using some languages like Lean or Coq and so forth. Okay? And then debugging, once you identify the component, like I mentioned, you fix the problem by changing the code and repeat the process until hopefully all tests pass. Okay?
Now, debugging an AI agent is harder. Okay? And by the way, going to the software engineering examples, I just want to make sure we are talking about the software agents, right? Which generates code and everything. The point I want to make here is that for a long time and in many situations, generating the code is not the biggest problem. It's just making sure that the program is working. Again, reliability, which I just mentioned to you, takes much more time. Right? So, that's kind of I just want to put things into perspective. That's where the problem is. And it's not now, it has been for decades. Right? This is what people do. This is where software engineers spend time. So, for agent, like I mentioned, things are more difficult. And by the way, just to make sure that we define the agent, the agent is a system in which an LLM acts as a central reasoning engine and in a loop which repeatedly interacts with the environment using some external tools, gets the observations of these interactions, and iterates until a stopping condition becomes true. Like task is done, budget exceeded, or there are some errors. And the reason it's much harder is because the central part of an agent is a large language model. A large language model is a black box. And unlike software, every software component when you debug it, it's like you have the source code, it's a white box. You can go and identify the instruction which has a problem then you modify it. Here it is much harder. You know, just to give you a sense, it's like you would debug a program without looking into it, just observing the inputs and outputs. Right? Just much harder.
Now, in the past year or so, you see all of this kind of transition from agent loops to multi-agent system. It's a good transition. Right? It's actually in some sense, the agent resembles more kind of the system when you have different components which communicate between them and which have clearer interfaces. And the reason you have multi-agent systems, it's actually similar reason of why you have a system which is composed of multiple components. You have content isolation with kind of total isolation. Each component, each of these agents or sub-agents knows only about its prompt, about what it has to do. Like specialization, right? Each sub-agent is doing only one thing which is much easier than one agent handling many tools with a very large prompt and also sometimes more cost-efficient. It's parallelism. If you have tasks which can be executed in parallel, you can have each sub-agent executing its task. Like for instance, you want to process a lot of documents in parallel, you do a lot of searches and then you want to aggregate the response and so forth. And it's modularity, right? If you want to change the system, you don't need to change the entire system. You can change one component at a time, right? You have an agent which is doing better verification. You can change that one, and the rest can remain the same, right? It's a more modular way to build systems.
This is an example about ChatDev, and here what you have is, you know, you model the agents by kind of a software company organization. You have a CEO which comes with the idea. You have a CTO which basically figures out what to do and how to do it, provides a plan or requirements documents or something like that to the programmer, and the programmer is implementing, then you are going to have kind of iteration between the programmers and you have a reviewer for the program, and then you have testing the program, right? That's kind of what it is. So, the next part is going to be what I'm going to focus. It's about one step in all of these kind of process about how to make system reliably. And that's the first step is say you have to actually identify the errors, right? The failures. And I'm going to talk about two projects, MAST and Atlas. So, MAST is multi-agent system failure taxonomy, and we made this study around 1 year ago, and we published this paper a few months ago. And we look at different agentic frameworks, multi-agentic frameworks, like MetaGPT, ChatDev, HyperAgent, and so forth. And we try to run them on pretty regular benchmarks at that time. And the main point here is that each bar is basically saying blue is success, yellow is failure. So you can see the majority of cases you have failures. So failures are prevalent.
So the next question is why do they fail? So you look into that and we come up with, you know, the goal is to identify and classify, come up with a taxonomy of these issues, of these failures. And how did we do it? We took 150 traces and the students manually annotated all the failures to fit in some kind of categories. Then we took 1,000 or more traces and then we develop a system using the LLMs to categorize these failures and then we use the data from manually generated by users to make sure that these LLMs are going to classify the same way. The agreement between the human classification and LLM classification 95%. And then you generate this kind of taxonomy. We look at seven the most popular at that time agentic frameworks and this is what we got. We got three categories which is specification issues, inter-agent misalignment, and task verification, and each of them has another three subcategories. The one point to notice here is that there is no dominant failure category. They are pretty balanced. I'll give you a few examples next.
So specification issues. Let me just... So specification issue is that the failure is like here it's about ambiguous prompt specification and things like that. But here are some kind of failures: if the agent disobeys specification, disobeys what you are going to tell it to do. Disobeys role specification. If you are going to say, 'Hey, you are going to play the role of a verifier,' it's not going to do only that, it's going also to change the code, for instance. Step repetition. If you are going to run a loop, doing the same thing over and over again. Of course, you don't want that. You just going to spend more tokens. And conversational loss. You lose things in translation. The information is not carried on across iterations. And agent unaware of terminal condition. I don't know when it's going to terminate. Here is one example, very simple example, in ChatDev. So here, the CEO basically say, 'Okay, we are going to discuss about what to do and how to do it. And when we all agree, then we are going to do it.' And in this case, the CPO is basically saying, 'Okay, I will do it.' and just go to the website to do it. So this is an example disobeying the role specification because the CPO shouldn't take any action before the CEO decides what the action is going to be taken. Okay? So the interesting thing here is that you may think, 'Okay, I'm going to use a better LLM, better prompts, and so forth.' But one other way you can fix this one is to basically improve the process. Like you'll be in a company, you improve the process, right? You don't change people, you improve the process. So here is equivalent with changing a little bit the system and inserting another step in the process. In this case, you have a step in which the CEO is going to decision step, right? You discuss, but then it's an explicit decision step on you are going to make the decision what to do next. And by doing just that, not changing anything else, you can get an improvement in the accuracy by 9%, which is pretty high. Okay?
Inter-agent misalignment. So, here you can have a lot of things. It's like conversation this set, failure to ask clarification questions, information withholding, ignoring the input, and so forth. Here is an example. So, here you have a supervisor, you have a user which tries through an agent to make an app to take an action. Like for instance, you want to use an agent to buy your flight ticket. Okay? So, here you say, 'Okay, this is a task I want you to do. And by the way, these are my credentials because you may want to log in from a site to perform that action.' The first thing happening here is that the phone agent, this is kind of the agent which is doing action on your behalf, is just sending to the Python executor, like this is the site where you buy the ticket for instance, is just sending, 'Hey, I want to buy this ticket.' without sending credentials. Right? So, it's just information withholding. Now, the side of Python executor is saying back, 'Okay, you need to log in. You have to log in. And for that, you need to give me the username and your account is a phone number.' Okay? Now, I didn't notice that the supervisor, as the agent to do the action on his behalf, he did provide credentials, but look at the credentials. The credential was a username, it was an email. Right? While this Python executor asked for a phone number. Okay, fine. Now, what it turns out that this phone agent in this case ignored the input and basically sends to the Python executor, ignored the input which was asking for a phone number as a username. And it just sends a username as an email. Okay, so that's... Now, the Python executor basically says, 'Hey, invalid credentials.' Okay? And what happens right now is like the phone agent is going to go back to the user and basically say, 'Hey, the login credentials are incorrect.' But it doesn't say what was incorrect. It doesn't say, 'Hey, you should provide actually the phone number, not the email.' Right? So, it's again information withholding. As a user you don't know what to do about it. Okay? So, the inter-agent alignment are kind of more subtle and you cannot do this kind of easy changes. You need in some sense more deeper social reasoning to solve these problems.
Let me talk a bit about task verification. The task verification is the last category of failures and these are premature termination, incomplete verification, or simply stating that something is working when it doesn't work. And here is one example. I am telling, you know, I want to build a chess game, right? And this is what the CEO tells to the programmer. And then I have a verifier to verify whether the program is working correctly. Okay, so looks good to me. However, in this case, the inputs do not follow the standard chess notation like king at K6, like king at, you know, king E8, right? The king is E8 position. And then the pawns can move backwards, okay? So, with task verification, there is no silver bullet. You need more testing, multi-level verification, and so forth.
Now, let's understand the user profiles for some of these cases. So, this is again ChatDev. So, specification issues, you have... these are beta specification, step repetition, and then agent unaware of termination. And then here you have the other categories, inter-agent alignment and task verification. Now, if you look across all these frameworks, you see it's kind of diverse. In particular for Aport, each of the categories you have more or less the same errors. It's pretty well balanced. However, if you look at Open Manus, it has much more skewed. Most of the problems are specification issues. You have very few task verification issues. So, this raises a question. Is one taxonomy good for all? Right? And obviously the answer is probably not. So, you want to customize your taxonomy of failures based on agentic architecture or domain, like coding, finance, and so forth.
And then the other question is that, okay, I'm going to have this taxonomy, what am I going to do about it? How is it going to help me? Right? Can I use it to improve the system? So, it's the next work, Atlas. I have only a few slides. And this is another way to look at. It's about here what you are going to see is an overlap between the failure modes for different domains. Each benchmark is a different domain. As you can see, it's very low. Right? One is total overlap, zero no overlap. Everything is like 0.1, 0.2. The max is 0.3. Okay? So again, failure modes are kind of different across domains, which you should expect. So Atlas is going to build a customized taxonomy for a domain and kind of framework. How you do that? You take the traces, then you are going to classify the traces in system, agent, and domain errors. Then you are going to clean a little bit that, you eliminate the duplicates, you make sure that each failure is indeed a failure. Then you have a group of agents which are going to try to figure out a taxonomy and agree on that, and then you end up with a taxonomy.
So the other question, the last question I mentioned, is that I have this kind of these taxonomies. What can I do with them? Well, it turns out that you can use them to improve this multi-agent system reliability. How you do it? So this is how the MAST logs look like. In this case, the first one is a prompt for the CEO in ChatDev. Then you have an error, then this next prompt is about for the CTO. Okay? Now, so this is user system prompts, chat history, logs, different logs, and code execution errors. It's lots of unstructured and messy data. You can have thousands of lines of that. So even if I give you the programmer, 'Hey, this is the logs. Find the problem,' it's very hard. Right? So what you can do here is use this taxonomy to make a little bit of order, to produce a structured feedback around the failure modes. Okay? And you can do that, for instance, providing the prompt. 'Hey, these are kind of the taxonomy failures,' or you are going to do it as a skill. Like you have another agent, sub-agent, using that sub-skill to check and how to debug the problem. And here you have compared self-reflection. This is you are going to SWE-bench for two frameworks and harnessings. And the only thing, you know, you have self-reflection. This is a baseline. And then you have MAST using the MAST fixed taxonomy, and using then Atlas in pattern A as in the prompt and as a skill. And as you can see, by having, first of all, taxonomy helps. Both MAST and Atlas are better. The second thing is that Atlas also customized taxonomy helps even more. This is another example. You have terminal bench to zero. And here we do it as an LLM as a judge. In many cases, use LLM as a judge to decide whether an output is good or not. Right? This is how you do. Now, in this case, I'm going to tell the LLM that there's a judge, I'm going to give this taxonomy for that domain, and that really helps. You again, you see, Comba pass one and LLM as a verifier, they are baselines. MAST judge using the fixed taxonomy, Atlas is using customized taxonomy, and you can see the best of five Oracle. This is Oracle. It's actually, you know, the best you can do, assuming that you know the correct answer.
So, in summary, reliability is critical for AI success and must... and at last, there are two projects which try to understand why AI agents fail and can use failure categories to improve reliability. And this is just starting, many challenges. Some parting thoughts. So, again, reliability is important. It's important in every domain. It's important in software. It's important for your car. It's important for your fridge. Everything. So, reliability will be very important for AI as well. There is no question. It's also reliability that's a staple of engineering disciplines. All of the things I'm talking about are engineering disciplines. Okay? And in AI, it's going to be true and is going to be even more difficult to solve. And one thing which matters a lot is specification. We are talking about verification. There is no verification without specification. Right? What are you going to verify? You always verify that something behaves according to some expectation, according to a specification. That's very important. And specification not only allows you to verify, allows you to debug, to decompose the system in smaller pieces, to reuse a component if you know exactly what a component is doing and it's very well specified, you can replace it with a component which matches the specification, and it also helps you automate decision making because if you trust the specification, then you don't need maybe in some cases a human in the loop. So, okay? So, the last is our some papers in this area we have done. So, this is my last slide and sorry for taking so long. So, specifications help make agents verifiable. But alone, they don't ensure correctness. So, that's why you want to use hardware to enforce the correctness. Right? Like you heard, right? Using the hardware, you can ensure, you can guarantee where the code is running, what code is running, what data is accessed, and what rules and policies have been enforced. And not only that, but using cryptographic proofs, it can provide you probable evidence that all of these things were correctly followed by your agent in production. Thank you.
