Agentic AI by Andrew NG

When I coined the term 'agentic' to describe what I saw as an important and rapidly growing trend in how people were building LLM-based applications, what I did not realize was that a bunch of marketers would get hold of this term and use it as a sticker and put this on almost everything in sight. And that has caused hype on agentic AI to skyrocket. The good news, though, is that ignoring the hype, the number of truly valuable and useful applications built using agentic AI has also grown very rapidly, even if not quite as rapidly as the hype. And in this course, what I'd like to do is show you best practices for building agentic AI applications and open up a lot of new opportunities to you in terms of what you can now build. Today, agentic workflows are being used to build applications like customer support agents, or to do deep research to help write deeply insightful research reports, or to process tricky legal documents, or to look at patient input and render or to suggest possible medical diagnoses. On many of my teams, a lot of the projects we build just would be impossible without agentic workflows. And so knowing how to build applications with them is one of the most important and valuable skills in AI today. It turns out that one of the biggest differences I've seen between people that really know how to build agentic workflows compared to people that are less effective at it is the ability to drive a disciplined development process, specifically one focused on evals and error analysis. And in this course, I'll tell you what that means and show you what allows you to be really good at building these agentic workflows. Being able to do this is one of the most important skills in AI today and will open up a lot more opportunities, be it job opportunities or opportunities to just build amazing software yourself. With that, let's go on to the next video to dive more into what are agentic workflows.

So, what is agentic AI and why are agentic AI workflows so powerful? Let's take a look. The way that many of us use large language models or LLMs today is by prompting it to say, write an essay for us on a certain topic X. And I think of that as akin to going to a human, or in this case going to an AI, and asking it to please type out an essay for me by writing from the first word to the last word all in one go and without ever using backspace. And it turns out that we as people, we don't do our best writing like that, by being forced to write in this completely linear order, and nor do AI models. But despite the difficulty of being constrained to write in this way, our LLMs do surprisingly well. In contrast, with an agentic workflow, this is what the process might look like. You may ask it to first write an essay outline with a certain topic, then ask if it needs to do any web research. And after doing some web research and maybe downloading some web pages, then to write the first draft, and then to read the first draft and see what parts need revision or more research, and then revise the draft, and so on. And this type of workflow is more akin to doing some thinking and some research, and then doing some revision, and then doing some more thinking, and so on. And with this iterative process, it turns out that an agentic workflow can take longer, but it delivers a much better work product. So an agentic AI workflow is a process where an LLM-based app executes multiple steps to complete a task. In this example, you might use an LLM to write the first essay outline, and then you might use an LLM to decide what search terms to type into a web search engine, or really what search terms to call a web search API with in order to get back relevant web pages. Based on that, you can feed the downloaded web pages into an LLM to have it write the first draft, and then maybe use another LLM to reflect and decide what needs more revision. And then depending on how you design this workflow, perhaps you may even add a human-in-the-loop step where the LLM has the option to request human review, maybe of some key facts, and based on that it may then revise the draft. And this process results in a much better work output. One of the key skills you learn in this course is how to take a complex task like writing an essay and breaking it down into smaller steps for an agentic workflow to execute one step at a time to then get the work output that you want. And knowing how to decompose the tasks into steps and how to build the components to execute the individual steps well turns out to be a tricky but important skill that will determine your ability to build agentic workflows for a huge range of exciting applications. In this course, a running example that we'll use and something that you build alongside me is a research agent. So here's an example of what it will look like. You can enter a research topic like, how do I build a new rocket company to compete with SpaceX? But I don't personally want to compete with SpaceX, but if you want to, you can try asking a research agent to help you with background research. So this agent starts with planning out what research to do, including calling a web search engine to download some web pages, and then to synthesize and rank findings, draft an outline, have an editor agent review for coherence, and then finally generate a comprehensive markdown report, which it has done here: building a new rocket company with SpaceX, with an intro, background, findings, and so on. And I think it points out appropriately that this is going to be a tough startup to build. So I'm not personally planning to do this, but if you want to tackle something like this, maybe a research agent like this could help you with some initial research. And by finding and downloading multiple sources and deeply thinking about it, this actually ends up with a much more thoughtful report than just prompting an LLM to write an essay for you would. One of the reasons I'm excited about this is because in my work, I've ended up building quite a few specialized research agents, be it in legal documents for complex legal compliance, or for some healthcare sectors, or some business product research areas. And so I hope that working through this example, you not only learn how to build agentic workflows for many other applications, but that some of the ideas in building research agents will be directly useful to you if you ever need to build a custom research agent yourself. Now, one of the often discussed areas of AI agents is how autonomous are they? What you just saw here was a relatively complex, highly autonomous agentic AI workflow. But there are also other simpler workflows that are incredibly valuable. Let's go on to the next video to talk about the degree to which agentic workflows can be autonomous, and this will give you a framework to think about how you might go about building different applications and how easy or difficult they might be. See you in the next video.

Agents can be autonomous to different degrees. A few years ago, I noticed within the AI community that there was a growing controversial debate about what is an agent. And some people are writing a paper saying, 'I built an agent,' and others say, 'No, that's not really a true agent.' And I felt this debate was unnecessary, which is why I started using the term 'agentic,' because I thought if we use it as an adjective rather than a binary — is it either an agent or not — then we're going to just acknowledge that systems can be agentic to different degrees, and this is called agentic, and move on with the real work of building these systems rather than debating, you know, is this sufficiently autonomous to be an agent or not. I remember when I prepared a talk on agentic reasoning, one of my team members actually came to me and said, 'Hey, Andrew, we don't need yet another word. You know, we have agent. Why are you making another word, agentic?' But I decided to use it anyway, and then later on wrote an article in DeepLearning.AI's newsletter, The Batch, and then also posted on social media saying that instead of arguing over which work to include or exclude as being a true agent, let's acknowledge that different degrees to which systems can be agentic. And I think this helped move past the debate on what is a true agent and let us just focus on actually building them. Some agents can be less autonomous. So take the example of writing an essay about black holes. You can have a relatively simple agent to come up with a few web search terms or web search queries. Then you can hardcode in that you call a web search engine, fetch some web pages, and then use that to write an essay. And this would be an example of a less autonomous agent with a fully deterministic sequence of steps, and this will work. Okay. In terms of notational convention, throughout this course, I'll use the red color as you see here on the left to denote the user input, such as a user query in this case, or in later examples maybe the input document into an agentic workflow. The gray boxes denote calls to an LLM. And the green boxes, like the web search and the web fetch boxes that you see here, indicate steps where other software is being used to carry out an action, such as a web search API call or executing code to fetch the contents of a website. Then an agent can be more autonomous, where given a request to write an essay about black holes, perhaps you let the LLM decide: does it want to do a web search, or does it want to search recent news sources, or does it want to search for recent research papers on the arXiv? Based on that, maybe in this example, the LLM — not the human engineer, but the LLM — chooses in this case to call a web search engine. And then after that, you may let the LLM decide how many web pages does it want to fetch, or if it fetches a PDF, does it need to call a function or also call the tool to convert the PDF to text? And in this case, maybe it fetches its top few web pages, then it can write an essay, decide whether or not to reflect and improve, and maybe even go back to fetch more web pages, and then to finally produce an output. And so even for this example of a research agent, we can see that some agents can be less autonomous, with a linear sequence of steps to be executed determined by a programmer, and some can be more autonomous, where you trust the LLM to make more decisions, and the exact sequence of steps that happens may be even determined by the LLM rather than in advance by the programmer. So for less autonomous systems, you will usually have all the steps predetermined in advance, and any functions it calls, like web search — and we'll call that tool use, as you learn in the third module in this course — might be hardcoded by the human engineer, by you or me, and most of the autonomy is in what text it generates. At the end of the spectrum would be highly autonomous agents, where the agent makes many decisions autonomously, including, for example, deciding what is the sequence of steps it will carry out in order to write the essay. And there are some highly autonomous agents that can even write new functions, or we sometimes say create new tools that it can then execute. And somewhere in between are semi-autonomous agents, where it can make some decisions, choose tools, but the tools are usually more predefined. As you look at different examples in this course, you'll learn how to build applications anywhere on this spectrum of less to more highly autonomous. And you'll find that there are tons of applications in the less autonomous end of the spectrum. They're very valuable, being built for tons of businesses today. And at the same time, there are also applications being worked on at the more highly autonomous end of the spectrum, but those are usually less easily controllable, a little bit more unpredictable. And also a lot of active research as well to figure out how to build these more highly autonomous agents. And with that, let's go on to the next video to dive deeper into this and to hear about some of the benefits of using agents and why they allow us to do things that just were not possible with earlier generations of LLM-based applications.

I think the one biggest benefit of agentic workflows is that it allows you to do many tasks effectively that just previously were not possible. But there are other benefits as well, including parallelism that lets you do certain things quite fast, as well as modularity that lets you combine the best-of-breed components from many different places to build an effective workflow. Let's take a look. My team collected some data on a coding benchmark that tests the ability of different LLMs to write code to carry out certain tasks. The benchmark used in this case is called HumanEval. And it turns out that GPT-3.5 — this is a model that the first publicly available version of ChatGPT was based on — if asked to write the code directly, to just type out the computer program, gets 40% right on this benchmark. This is a positive K metric. GPT-4 is a much better model. Its performance leaps to 67% with this also non-agentic workflow. But it turns out that as large as the improvement was from GPT-3.5 to GPT-4, that improvement is dwarfed by what you can achieve by wrapping GPT-3.5 within an agentic workflow using different agentic techniques, which you learn about later in this course. You can prompt GPT-3.5 to write code and then maybe reflect on the code and figure out if you can improve it. And using techniques like that, you can actually get GPT-3.5 to get much higher levels of performance. And similarly, GPT-4 used in the context of an agentic workflow also does much better. So even with today's best LLMs, an agentic workflow lets you get much better performance. And in fact, what we saw in this example was the improvement from one generation of model to another, which is huge. It's still not as big a difference as implementing an agentic workflow on the previous generation of model. Another benefit of using agentic workflows is that they can parallelize some tasks and thus do certain things much faster than a human. For example, if you ask an agentic workflow to write an essay about black holes, you might be able to have three LLMs run in parallel to generate ideas for web search terms to type into a search engine. Based on the first web search, it may identify, say, three top results to fetch, and based on the second web search, it may identify a second set of web pages to fetch, and so on. And it turns out that whereas a human doing this research would have to read these nine web pages sequentially or one at a time, when you're using an agentic workflow, you can actually parallelize all nine webpage downloads and then finally feed all these things into an LLM to write an essay. So even though agentic workflows do take longer than purely non-agentic workflows, or by direct generation by just prompting a single time, if you were to compare this type of agentic workflow to how a human would have to go about the task, the ability to parallelize downloading lots of web pages can actually let it do certain tasks much faster than the non-parallel, sequential way that a single human might process this data. To build on this example, it turns out one of the things I often do when building agentic workflows is look at the individual components like the LLM and add or swap out components. So for example, maybe I look at the web search engine I use up here and I might decide that I want to swap in a new web search engine. When building agentic workflows, there are actually multiple web search engines, including Google, which you can access via Serper, as well as others like Bing, DuckDuckGo, Tavily, Exa. There are actually quite a lot of options for web search engines designed for LLMs to use. Or maybe instead of just doing three web searches, maybe on this step you can swap in a new news search engine so they can find out what's the latest news on recent breakthroughs on black hole science. And lastly, instead of using the same LLM for all of the different steps, I will often try out different large language models and maybe try out different LLM providers to see which one gives the best result for different steps of this system. So to summarize, the main reason I use agentic workflows is it just gives much better performance on many different applications. But in addition, it can also parallelize some tasks that humans will otherwise have to do sequentially. And the modular design of many agentic workflows also lets us add or update tools and sometimes swap out models. We've talked a lot about the key components of building agentic workflows. Let's now take a look at a range of agentic AI applications to give you a sense of the sorts of things people are already building and the sorts of things you'll be able to build yourself. Let's go on to the next video.

People and businesses do a lot of stuff. How do you take this useful stuff that we do and break it down into discrete steps for the agentic workflow to follow? Let's take a look. Take the example of building a research agent. If you want an AI system to write an essay on a topic X, one thing you could do is prompt an LLM to have it generate an output directly. But if you were to do this for topics that you want deeply researched, you may find that the LLM output covers only the surface-level points. So maybe it covers only the obvious facts but doesn't go as deep into the subject as you wanted to. In this case, you might then reflect on how you as a human would write an essay on a certain topic. Would you just sit down and start writing, or would you take multiple steps, such as first write an essay outline and then search the web, and then based on the input from the web search write the essay? And as I take a task and decompose it into steps, one question I'm always asking myself is: if I look at these steps one, two, and three, can each of them be done either by an LLM or by a short piece of code or by a function call by a tool? And so in this case, I think an LLM can maybe write a decent outline on many topics that I would want it to help me think through. So I say probably okay on the first step. And then I know how to use an LLM to generate search terms to search the web. So I would say the second step is also doable. And then based on web search, I think an LLM could input the web search results and write an essay. And so this would be a reasonable first attempt at an agentic workflow for writing an essay that goes deeper than just direct generation. But if I were to then implement this agentic workflow and look at the results, maybe you find that the results still aren't good enough. It's still not yet as deeply thoughtful. Maybe the essays feel a little bit disjointed. This has actually happened to me. I once built a research agent using this workflow, but when I read the output, it felt a bit disjointed. You know, the start of the article didn't feel completely consistent with the middle, didn't feel completely consistent with the end. In this case, what you might do is then reflect on how you would change the workflow if you as a human found that the essays were a bit disjointed. One thing you could do is take the third step and further decompose 'write the essay' into additional steps. So instead of writing the essay all in one go, you might instead have it write a first draft and then consider what parts need revision and then revise the draft. And this would be how I as a human might go about it to not just write the final essay at my first attempt, but write the first draft and then read over it, which is another step that an LLM is pretty decent at. And then based on my own critique of my own essay, I'll revise the draft. So to recap, I started off with direct generation, just one step, decided it wasn't good enough, and so broke it down into three steps. And then maybe decided that still isn't good enough, and took one of the steps and further broke it down or decomposed it into three more steps, resulting in this more complex, richer process for generating an essay. And depending on how satisfied you are with the results of this process, you may choose to even modify this essay generation process further. Let's look at a second example of how to decompose complex tasks into smaller steps. Take the example of responding to basic customer order inquiries. The first step that a human customer service representative might carry out might be to first extract the key information, such as who is this email from, what did they order, and what is the order number. And these are things that an LLM could do. So I could just say, let's have an LLM do that. The second step would be to then find the relevant customer records. So to write and generate the relevant database queries to pull up the order of what the customer had ordered and when it shipped, and so on. I think an LLM with the ability to call a function to query the orders database should be able to do that. And then lastly, having pulled up a customer record or the customer order record, I might then write and send a response back to the customer. And I think with the information we pulled up, this third step is also doable with an LLM, if I give it the option to call an API to send an email. So this would be another example of taking a task of responding to customer email and breaking it down into three individual steps where I can look at each of these steps and say, 'Yep, I think an LLM or an LLM with the ability to call a function to query a database or send an email should be able to do that.' Just one last example for the invoice processing. After a PDF invoice has been converted to text, the first step is to pull out the required information: the name of the biller, the address, the due date, the amount due, and so on. And an LLM should be able to do that. And then if I want to check that the information was extracted and save it in a new database entry, then I think an LLM should be able to help me call a function to update the database record. And so to implement this, we implement an agentic workflow to carry out basically these two steps. When building agentic workflows, I think of myself as having a number of building blocks. One important building block would be large language models, or maybe large multimodal models if I want to try to process images or audio as well. And LLMs are good at generating text, deciding what to call, maybe extracting information. For some highly specialized tasks, I might also use some other AI models, such as an AI model for converting a PDF to text, or for text-to-speech, or for image analysis. In addition to AI models, I also have access to a number of software tools, including different APIs that I can call to do web search, to get maybe real-time weather data, to send emails, check calendar, and so on. And I might also have tools to retrieve information, to pull up data from a database, or to implement RAG, or retrieval-augmented generation, where I can look up a large text database and find the most relevant text. Or I might also have tools to execute code. And this is a tool that lets an LLM write code and then run the code on your computer to do a huge range of things. In case some of these tools seem a bit foreign to you, don't worry about it. We'll go through the most important tools in much greater detail in a later module. But I think of a lot of my work when I'm building an agentic workflow as looking at the work that a person or business is doing and then trying to figure out with these building blocks, how can I sequence these building blocks together in order to carry out the task that I want my system to carry out. And this is why having a good understanding of what building blocks are available, which I hope you have a better sense of by the end of this course as well, will allow you to better envision what agentic workflows you can build by combining these building blocks together. So to summarize, one of the key skills in building agentic workflows is to look at a bunch of stuff that maybe someone does and to identify the discrete steps that it could be implemented with. And when I look at the individual discrete steps, one question I'm always asking myself is, can this step be implemented with either an LLM or with one of the tools such as an API or a function call that I have access to? And in case the answer is no, I'll then often ask myself, how would I as a human do this step? And is it possible to decompose this further, break this down into even smaller steps that then maybe is more amenable to implementation with an LLM or with one of the software tools that I have. So I hope this gives you a rough sense of how to think about task decomposition. In case you feel like you don't fully have it yet, don't worry about it. We'll go through many more examples in this course and you'll have a much better understanding of this by the end of this course. But it turns out that as you build agentic workflows, you find that often you build an initial task decomposition, initial agentic workflow, and then you want to keep on iterating and improving on it quite a few times until it delivers the level of performance that you want. And to drive this improvement process, which I found important for many projects, one of the key skills is to know how to evaluate your agentic workflow. So in the next video we'll talk about evaluations or evals, a key component of how you can build and then also keep on improving your workflows to get the performance that you want. Let's talk about evals in the next video.

I've worked with many different teams on building agentic workflows and I found that one of the biggest predictors for whether someone is able to do it really well versus be less efficient at it is whether or not they're able to drive a really disciplined evaluation process. So your ability to drive evals for your agentic workflow makes a huge difference in your ability to build them effectively. In this video, we'll take a quick overview of how to build evals. And this is a subject that we'll actually go into much deeper in a later module in this course. Let's take a look. After building an agentic workflow like this one for responding to customer order inquiries, it turns out that it's very difficult to know in advance what are the things that could go wrong. And so rather than trying to build evaluations in advance, what I recommend is you just look at the outputs and manually look for things that you wish it was doing better. For example, maybe you read a lot of outputs and find that it is unexpectedly mentioning your competitors more than it should. Many businesses don't want their agents to mention competitors because it just creates an awkward situation. And if you read some of its outputs, maybe you'll find that it sometimes says, 'I'm glad you shopped with us. We're much better than a competitor.' Or, 'Sure, refund. Unlike rival co, we make returns easy.' And you may look at this and go, 'Gee, I really don't want this to mention competitors.' This would be an example of a problem that is really hard to anticipate in advance of building this agentic workflow. So the best practice is really to build it first and then examine it to figure out where it is not yet satisfactory and then to find ways to evaluate as well as improve the system to eliminate the ways that it is still not yet satisfactory. Assuming your business considers it an error or mistake to mention competitors in this way. Then as you work on eliminating these competitor mentions, one way to track progress would be to add an evaluation or an eval to track how often this error occurs. So if you have a named list of competitors like comco code, rival code, the other code, then you can actually write code to just search in your own output for how often it mentions these competitors by name and count up as a number, as a fraction of the overall responses, how frequently it mistakenly mentions competitors. One nice thing about the problem of competitor mentions is it's an objective metric, meaning either the competitor was mentioned or not. And for objective criteria, you can write code to check for how often this specific error occurs. But because LLM outputs are free text, there are also going to be criteria by which you want to evaluate this output that may be more subjective and where it's harder to just write code to output a black-and-white score. In this case, using an LLM as a judge is a common technique to evaluate the output. So for example, if you're building a research agent to do research on different topics, then you can use another LLM and prompt it to maybe say, assign the following essay quality score between one and five, where one is the worst and five is the best essay. Here I'm using a Python expression to mean, you know, copy-paste the generated essay into this. So you can prompt the LLM to read the essay and assign it a quality score. Then I might ask a research agent to write a number of different research reports. For example, on recent developments in black hole science or using robots to harvest fruit. And then in this example, maybe the judge LLM assigns the essay on black holes a score of three, the essay on robot harvesting a score of four. And as you work on improving your research agent, hopefully you see these scores go up over time. It turns out, by the way, that LLMs are actually not that good at these one-to-five scale ratings. You can give it a shot, but I personally tend not to use this technique that much myself. But in a later module, you'll learn some better techniques to have an LLM output more accurate scores than just scores on a one-to-five scale. Although some people will do this maybe at an initial cut as an LLM-as-a-judge type of eval. Just to give a preview of some of the agentic AI evals you'll learn about late in this course. You've already heard me talk about how you can write code to eval objective criteria such as did it mention the competitor or not, or use an LLM as a judge for more subjective criteria such as what's the quality of this essay. But later you'll learn about two major types of evals. One is end-to-end, where you measure the output quality of the entire agent, as well as component-level evals, where you measure the quality of the output of a single step in the agentic workflow. And it turns out that these are useful for driving different parts of your development process. And one thing I do a lot as well is just examine the intermediate outputs, or some call this a trace of the LLM, in order to understand where it is falling short of my expectations. And we call this error analysis, where we just read through the intermediate outputs at every single step to try to spot opportunities for improvement. And it turns out being able to do evals and error analysis is a really key skill. So we have much more to say about this in the fourth module in this course. We're nearly to the end of this first module. Before moving on, I'm just going to share with you what I think are the most important design patterns for building agentic workflows. Let's go take a look at that in the next video.

We build agentic workflows by taking building blocks and putting them together to sequence out these complex workflows. In this video, I'd like to share with you a few of the key design patterns, which are patterns for how you can think about combining these building blocks into more complex workflows. Let's take a look. I think four key design patterns for building agentic workflows are reflection, tool use, planning, and multi-agent collaboration. Let me briefly go over what they mean, and then we'll actually go through most of these in depth later in this course as well. The first of the major design patterns is reflection. So I might go to an LLM agent and ask it to write code. And it turns out that an LLM might then generate code like this. You know, it defines here a Python function to do a certain task. I could then construct a prompt that looks like this. I can say, here's code intended for a certain task, and then copy-paste whatever the LLM had just output back into this prompt and then ask it, check the code carefully for correct style and efficiency and give constructive criticism. And it turns out that the same LLM prompted this way may be able to point out some problems with the code. And if I then take this critique and feed it back to the model to say, looks like a bug. Could you change the code to fix it? Then it may actually come up with a better version of the code. To give a preview of tool use, if you're able to run the code and see where the code fails, then feeding that back to the LLM can also cause it to be able to iterate and generate a much better, say, V3, version three of the code. So reflection is a common design pattern where you can ask the LLM to examine its own outputs or maybe bring in some external sources of information, such as run the code and see if it generates any error messages, and use that as feedback to iterate again and come up with a better version of its output. And this design pattern isn't magic. It does not result in everything working 100% of the time. But sometimes it can be a nice bump in the performance of your system. Now, I've drawn this as if it was a single LLM that I'm prompting, but to foreshadow multi-agent workflows, you can also imagine instead of having the same model critique itself, you can imagine having a critic agent. And all that is is an LLM that's been prompted with instructions like, your role is to critique code. Here's code intended for a task, check the code carefully, and so on. And the second critic agent maybe to point out errors or run unit tests. And by having two simulated agents where each agent is just an LLM prompted to take on a certain persona, you can have them go back and forth to iterate to get a better output. In addition to the reflection pattern, the second important design pattern is tool use, where today LLMs can be given tools, meaning functions that they can call in order to get work done. For example, if you ask an LLM, what's the best coffee maker according to reviewers, and you give it a web search tool, then it can actually search the internet to find much better answers. Or a code execution tool. If you ask a math question like, if I invest $100 at compound interest, what have you, it can then write code and execute code to compute an answer. Today different developers have given LLMs many different tools for everything, for math or data analysis, to gather information by fetching things from the web or for various databases, to interface with productivity apps like email, calendar, and so on, as well as to process images and much more. And the ability of an LLM to decide what tools to use, meaning what functions to call, that lets the model get a lot more done. The third of the four design patterns is planning. This is an example from a paper called HuggingGPT, in which if you ask a system to please generate an image where a girl is reading a book and a pose is the same as a boy image, then please describe the new image in your voice. Then a model can automatically decide that to carry out this task, it first needs to find a pose determination model to figure out the pose of the boy, then to pose the image to generate a picture of a girl, and an image-to-text, and then finally text-to-speech. And so in planning, an LLM decides what is the sequence of actions it needs to take. In this case, it is a sequence of API calls so that it can then carry out the right sequence of steps in the right order in order to carry out the task. So rather than the developer hardcoding the sequence of steps in advance, this actually lets the LLM decide what are the steps to take. Agents that plan today...

are harder to control and some are more experimental, but sometimes they can give really delightful results. And then finally, multi-agent workflows. Just as a human manager might hire a number of others to work together on a complex project, in some cases it might make sense for you to hire a set of multiple agents, maybe each of which specializes in a different role, and have them work together to accomplish a complex task.

The picture you see here on the left is taken from a project called ChatDev, which is a software framework created by Chen and collaborators. In ChatDev, multiple agents with different roles, like chief executive officer, programmer, tester, designer, and so on, collaborate together as if they were a virtual software company and can collaboratively complete a range of software development tasks.

Let's consider another example. If you want to write a marketing brochure, maybe you'd think of hiring a team of three people, such as a researcher to do online research, a marketer to write the marketing text, and then finally an editor to edit and polish the text. And so in a similar way, you might consider building a multi-agent workflow in which you have a simulated research agent, a simulated marketer agent, and a simulated editor agent that then come together to carry out this task for you.

Multi-agent workflows are more difficult to control since you don't always know ahead of time what the agents will do. But research has shown that they can result in better outcomes for many complex tasks, including things like writing biographies or deciding on chess moves to make in the game. You'll learn more about multi-agent workflows later in this course as well.

And so with that, I hope you have a sense of what agentic workflows can do, as well as of what are the key challenges of finding building blocks and putting them together, maybe via these design patterns, in order to implement an agentic workflow, and of course also developing evals so you can see how well your system is doing and keep on improving on it.

In the next module, I'd like to share with you a deep dive into the first of these design patterns, that is, reflection. And you'll find that it is a maybe surprisingly simple to implement technique that can give the performance of your system sometimes a very nice bump. So let's go on to the next module to learn about the reflection design pattern.

In this module, I'd like to share with you practical tips for building agentic AI workflows. I hope that these tips will enable you to be much more effective than the typical developer at building these types of systems. I find that when developing an agentic AI system, it's difficult to know in advance where it will work and where it won't work so well, and thus where you should focus your effort.

So very common advice is to try to build even a quick and dirty system to start, so you can then try it out and look at it to see where it may not yet be working as well as you wish, to then have much more focused efforts to develop it even further. In contrast, I find that it's sometimes less useful to, you know, sit around for too many weeks theorizing and hypothesizing how to build it.

It's often better to just build a quick system in a safe, reasonable way that doesn't leak data. Kind of do it in a responsible way, but just build something quickly so you can look at it and then use that initial prototype to prioritize and drive further development.

Let's start with an example of what might happen after you've built a prototype. I want to use as our first example the invoice processing workflow that you've seen previously, where the task is to extract four required fields and then to save it to a database record. After having built such a system, one thing you might do is find a handful of invoices, maybe 10 or 20 invoices, and go through them and just take a look at their output and see what went well and if there were any mistakes.

So let's say you look through 20 invoices. You find that invoice one is fine, the output looks correct. For invoice two, maybe it confused the date of the invoice, that is, when was the invoice issued, with the due date of the invoice. And in this task, we want to extract the due date so we can subsequently make payments on time.

So then I might note down in a document or in a spreadsheet that for invoice two, the dates were mixed up. Maybe invoice three was fine, invoice four was fine, and so on. But as I go through this example, I find that there are quite a lot of examples where it had mixed up the dates.

So it is based on going through a number of examples like this that, in this case, you might conclude that one common error mode is that it is struggling with the dates. In that case, one thing you might consider would be to, of course, figure out how to improve your system to make it extract due dates better, but also maybe write an eval to measure the accuracy with which it is extracting due dates.

In comparison, if you had found that it was extracting the biller address incorrectly, who knows? Maybe you have billers with unusual-sounding names, and so maybe it struggles with billers, or especially if you have international billers whose names may not even all be English letters, then you might instead focus on building an eval for the biller address.

So one of the reasons why building a quick and dirty system and looking at the output is so helpful is it even helps you decide what do you want to put the most effort into evaluating.

Now, if you've decided that you want to modify your system to improve the accuracy with which it is extracting the due date of the invoice, then to track progress, it might be a good idea to create an evaluation, an eval, to measure the accuracy of data extraction. There are probably multiple ways one might go about this, but let me share with you how I might go about this.

To create a test set or an evaluation set, I might find 10 to 20 invoices and manually write down what is the due date. So maybe one invoice has a due date of August 20th, 2025. I write it down in the standard year-month-date format.

And then to make it easy to evaluate in code later, I would probably write the prompt to the LM to tell it to always format the due date in this year-month-date format. And with that, I can then write code to extract out the one date that the LM has output, which is the due date, because that's the one date we care about.

So this is a regular expression pattern matching, you know, four numbers for the year, two for the month, two for the day, and extract that out. And then I can just write code to test if the extracted date is equal to the actual date, that is, the ground truth annotation I had written down.

So with an eval set of say 20 or so invoices, I'll be able to make changes to see if the percentage of time that it gets the extracted date correct is hopefully going up as I tweak my prompts or tweak other parts of my system.

So just to summarize what we've seen so far: we've built a system, then looked at outputs to discover where it may be behaving in an unsatisfactory way, such as due dates are wrong. Then to drive improvements to this important output, put in place a small eval with say just 20 examples to help us track progress.

And this lets me go back to tweak prompts, try different algorithms, and so on, to see if I can move up this metric of due date accuracy. So this is what improving an agentic AI workflow will often feel like: look at the output, see what's wrong. Then if you know how to fix it, just fix it. But if you need a longer process of improving it, then put in place an eval and use that to drive further development.

One other thing to consider is if after working for a while, if you think those 20 examples you had initially aren't good enough, maybe they don't cover all the cases you want, or maybe 20 examples is just too few, then you can always add to the eval set over time to make sure it better reflects your personal judgments on whether or not the system's performance is sufficiently satisfactory.

This is just one example. For the second example, let's look at building a marketing copy assistant for writing captions for Instagram. Where to keep things succinct, let's say our marketing team tells us that they want captions that are at most 10 words long.

So we would have an image of a product, say a pair of sunglasses that we want to market, and then have a user query like, 'Please write a caption to sell these sunglasses,' and then have an LM or large multimodal model analyze the image and the query and generate a description of the sunglasses.

And there are lots of different ways that a marketing copy assistant may go wrong. But let's say that you look at the output and you find that the copy or the text generated mostly sounds okay, but maybe it's just sometimes too long. So for the sunglasses input, it generated 17 words. If you have a coffee machine, 'is okay, stylish' is okay. Blue shirt, 14 words. Blender, 11 words.

So it looks like in this example the LM is having a hard time adhering to the length guideline. So again, there are lots of things that could have gone wrong with a marketing copy assistant. But if you find that it's struggling with the length of the output, then you might build an eval to track this so that you can make improvements and make sure it's getting better at adhering to the length guideline.

So to create an eval to measure the text length, what you might do is create a set of tests. So market a pair of sunglasses, a coffee machine, and so on, and maybe create 10 to 20 examples. Then you would run each of them through your system and write code to measure the word count of the output. So this is Python code to measure the word count of a piece of text.

Then lastly, you would compare the length of the generated text to the 10-word target limit. So if word count is equal to 10, num correct plus equals 1.

One difference between this and the previous invoice processing example is that there is no per-example ground truth. The target is just 10, same for every single example. Whereas in contrast, for the invoice processing example, we had to generate a custom target label, that is, the correct due date of the invoice, and we're testing the outputs against that per-example ground truth.

I know I used a very simple workflow for generating these captions, but these types of evals can be applied to much more complex generation workflows as well. Let me touch on one final example in which we'll revisit the research agents we've been looking at.

If you look at the output of the research agents on different input prompts, let's say that when you ask it to write an article on recent breakthroughs in black hole science, you find that it missed some high-profile result that had a lot of news coverage. So this is an unsatisfactory result.

Or if you ask it to research renting versus buying a home in Seattle, well, it seems to do a good job. Or robotics for harvesting fruit, well, it didn't mention a leading equipment company. So based on this evaluation, it looks like sometimes it misses a really important point that a human expert writer would have captured.

So then I would create an eval to measure how often it captures the most important points. For example, you might come up with a number of example prompts on black holes, robotic harvesting, and so on, and for each one come up with, let's say, three to five gold-standard discussion points for each of these topics.

Notice that here we do have a per-example annotation because the gold-standard talking points, that is, the most important talking points, they are different for each of these examples. With these ground truth annotations, you might then use an LLM judge to count how many of the gold-standard talking points were mentioned.

And so an example prompt might be to say, 'Determine how many of the five gold-standard talking points are present in the provided essay.' You have the optional prompts, the essay text, gold-standard points, and so on, and have it return a JSON object with two keys: that scores how many of the points, 0 to 5, did it score, as well as an explanation. And this allows you to get a score for each prompt in your evaluation set.

In this example, I'm using an LLM as a judge to count how many of the talking points were mentioned because there's so many different ways to talk about these talking points. And so a regular expression or code for simple pattern matching might not work that well, which is why you might use an LLM as a judge and treat this as a slightly more subjective evaluation for whether or not, say, event horizons were adequately mentioned.

So this is your third example of how you might build evals. In order to think about how to build evals for your application, the evals you build will often have to reflect whatever you see or you are worried about going wrong in your application. And it turns out that broadly there are two axes of evaluation.

On the top axis is the way you evaluate the output. In some cases, you evaluate it by writing code with objective evals, and sometimes you use an LLM as a judge for more subjective evals. On the other axis is whether you have a per-example ground truth or not.

So for checking invoice date extraction, we were writing code to evaluate if we got the actual date, and that had a per-example ground truth because each invoice has a different actual date. But in the example where we checked marketing copy length, every example had a length limit of 10, and so there was no per-example ground truth for that problem.

In contrast, for counting gold-standard talking points, there was a per-example ground truth because each article had different important talking points. But we used an LLM as a judge to read the essay to see if those topics were adequately mentioned because there's so many different ways to mention the talking points.

And the last of the four quadrants would be an LLM as a judge with no per-example ground truth. And one place where we saw that was if you're grading charts with a rubric. This is when we're looking at visualizing the coffee machine sales. And if you ask it to create a chart according to a rubric, such as whether it has clear axis labels and so on, there is the same rubric for every chart, and that would be using an LLM as a judge but with no per-example ground truth.

So I find this 2x2 grid as maybe a useful way to think about the different types of evals you might construct for your application. And by the way, these are sometimes also called end-to-end evals because one end is the input end, such as the user query prompt, and the other end is the final output. And so all of these are evals of end-to-end system performance.

So just to wrap up this video, I'd like to share a few final tips for designing end-to-end evals. First, quick and dirty evals is fine to get started. I feel like I see quite a lot of teams that are almost paralyzed because they think building evals is this massive multi-week effort, and so they take longer than would be ideal to get started.

But I think just as you iterate on an agentic workflow and make it better over time, you should plan to iterate on your evals as well. So if you put in place 10, 15, 20 examples as your first cut at evals and write some code or try prompting an LLM as a judge, just do something to start to get some metrics.

Then that can complement the human eye at looking at the output, and then it's a blend of the two that can drive your decision-making. And as the evals become more sophisticated over time, you can then shift more and more of your trust to the metric-based evals rather than needing to read over your hundreds of outputs every time you tweak a prompt somewhere.

And as you go through this process, you'll likely find ways to keep on improving your evals as well. So if you had 20 examples to start, you may then run into places where your evals fail to capture your judgment about what system is better.

So maybe you update the system and you look at it and you feel like this got to work much better, but your eval fails to show the new system achieving a higher score. If that's the case, that's often an opportunity to go maybe collect a larger eval set or change the way you evaluate the output to make it correspond to your judgment as to what system is actually working better. And so your evals will get better over time.

And lastly, in terms of using evals to gain inspiration as to what to work on next, a lot of workflows are being used to automate tasks that, say, humans can do. And so I find for such applications I'll look for places where the performance is worse than that of an expert human. And that often gives me inspiration for where to focus my efforts or what are the types of examples that I'm able to get my agentic workflow to work better than it is currently.

So I hope that after you've built that quick and dirty system, you think about when it would make sense to start putting in some evals to track the potentially problematic aspects of the system, and that that will then help you drive improvements to the system.

In addition to helping you drive improvements, it turns out that there's a method of evals that helps you hone in on your entire agentic system what are the components most worth focusing your attention on, because agentic systems often have many pieces. So which piece is going to be most productive for you to spend time working to improve?

Turns out being able to do this well is a really important skill for driving efficient development of agentic workflows. In the next video, I'd like to deep dive into this topic. So let's go on to the next video.

Let's say you've built an agent workflow and it is not yet working as well as you wish. And this happens to be all the time, by the way. I'll often build a quick and dirty system and it doesn't do as well as I wish it would. The question is, where do you focus your efforts to make it better?

Turns out agentic workflows have many different components, and working on some of the components could be much more fruitful than working on some other components. So your skill at choosing where to focus your efforts makes a huge difference in the speed with which you can make improvements to your system.

And I found that one of the biggest predictors for how efficient and how good a team is is whether or not they're able to drive a disciplined error analysis process to tell you where to focus your efforts. So this is an important skill. Let's take a look at how to carry out error analysis.

In the research agent example, we had carried out an error analysis in the previous video, and we saw that it was often missing key points that a human expert would have made in writing essays on certain topics. So now that you've spotted this problem, that is, sometimes missing key points, how do you know what to work on?

It turns out that of the many different steps in this workflow, almost any of them could have contributed to this problem of missing key points. For example, maybe the first LM was generating search terms that weren't great. So it was just searching for the wrong things and did not discover the right articles.

Or maybe it used a web search engine that just wasn't very good. There are multiple web search engines out there. In fact, there are actually quite a few that I tend to use for my own agentic applications, and some are better than others.

Or maybe web search was just fine, but when we gave the list of web search results to an LM, maybe it didn't do a good job choosing the best handful to download. Maybe web fetch has fewer problems in this case, assuming you can fetch web pages accurately. But after dumping the web pages into the LM, maybe the LM is ignoring some of the points in the documents we had fetched.

So it turns out that there are teams that'll sometimes look at this and go by gut to pick one of these components to work on, and sometimes that works, and sometimes that leads to many months of work with very little progress in the overall performance of the system.

So rather than going by gut to decide which of these many components to work on, I think it's much better to carry out an error analysis to better understand each step in the workflow. And in particular, I'll often examine the traces, and that means the intermediate output after each step, in order to understand which component's performance is subpar, meaning, say, much worse than what a human expert would do, because that points to where there may be room for significant improvement.

Let's look at an example. If we ask the research agent to write an essay about recent news in black hole science, maybe the LM outputs search terms like the search for black hole theories, Einstein, event horizon, telescope radio, and so on. And I would then have a human expert look at these and see are these reasonable web search terms for writing about recent discoveries in black hole science.

And maybe in this case, an expert says, 'These web searches look okay. They're, you know, pretty similar to what I would do as a human.' Then I look at the outputs of the web search and look at the URLs returned. So web search would return many different web pages, and maybe one web page returned is that an elementary school student claims to track a 30-year-old black hole mystery from Astro Kid News.

And this doesn't look like the most rigorous peer-reviewed article. And maybe examining all of the articles that web search returns causes you to conclude that it's returning too many blog or popular press types of articles and not enough scientific articles to write a research report of the quality that you are looking for.

It'd be good to just look through the outputs of the other steps as well. Maybe the LM finds the best five sources it can. You end up with Astro Kid News, Space Blog 2000, Space Fun News, and so on. And it is by looking at these intermediate outputs that you can then try to get a sense of the quality of the output of each of these steps.

To introduce some terminology, the overall set of outputs of all of the intermediate steps is often called the trace of a run of this agent. And then some terminology you see in other sources as well is the output of a single step is sometimes called a span. This is terminology from the computer observability literature where people try to figure out what computers are doing.

And in this course I use the word trace quite a bit. I'll use the word span a little bit less, but you may see both of these terms on the internet. So by reading the traces you start to get an informal sense of where might be the most problematic components.

In order to do this in a more systematic way, it turns out to be useful to focus your attention on the cases that the system is doing poorly on. Maybe it writes some essays just fine and the output is completely satisfactory. So I would put those aside and try to come up with a set of examples where, for whatever reason, the final output of your research agent is not quite satisfactory, and just focus on those examples.

So this is one of the reasons we call it error analysis, because we want to focus on the cases where the system made an error, and we want to go through to figure out which components were most responsible for the error in the research agent output.

In order to make this more rigorous, rather than reading and getting an informal sense, you might actually build up a spreadsheet to more explicitly count up where the errors are. And by error I mean when a step outputs something that performs significantly worse than maybe what a human expert would have given a similar input as that component.

So I'll often do this myself in a spreadsheet. So I might build a spreadsheet like this. And so for the first query, I'll look at recent developments in black hole science. And I see that the search results say too many blog posts, popular press articles, not enough scientific papers.

And then based on this, it is true that the five best sources aren't great. But here I won't say that the five best sources did a bad job, because if the inputs to the step for selecting the five best sources were all non-rigorous articles, then I can't blame this picking the five best sources for not picking better articles, because it did the best it could have, or did nearly as well as any human might have given the same selection to choose from.

And then you might go through this for different prompts: renting versus buying in Seattle. Maybe it missed a well-known blog. Robotics for harvesting fruit. Maybe in this case we look at it and say, 'Oh, the search terms are too generic and the search results also weren't good,' and so on.

And then based on this, I would count up in my spreadsheet how often I observe errors in the different components. So in this example, I'm dissatisfied with the search terms 5% of the time, but I'm dissatisfied with the search results 45% of the time.

And if I actually see this, I might just take a careful look at the search terms to make sure that the search terms really were okay and that poor choice of search terms were not what led to poor search results. But if I really think the search terms are fine, but the search results are not, then I would take a careful look at the web search engine I'm using and if there are any parameters I can tune to make it bring back more relevant or higher quality results.

There's this type of analysis that tells me in this example that maybe I really should focus my attention on fixing the search results and not on the other components of this agentic workflow.

So to wrap up this video, I find that it's useful to develop a habit of looking at traces. After you build an agentic workflow, go ahead and look at the intermediate outputs to get a feel for what it is actually doing at every step, so you can better understand if different steps are performing better or worse.

And a more systematic error analysis, maybe done with a spreadsheet, can let you gather statistics or count up which component performs poorly most frequently. And so by looking at what components are doing poorly, as well as where I have ideas for efficiently improving different components, then that would let you prioritize what component to work on.

So maybe a component is problematic, but I don't have any ideas for improving it. So that would suggest maybe not prioritizing that as high. But if there is a component that is generating a lot of errors and if I have ideas for how to improve that, then that would be a good reason to prioritize working on that component.

And I just want to emphasize that error analysis is a very helpful output for you to decide where to focus your efforts, because in any complex system there are just so many things you could work on. It's too easy to pick something to work on and work on it for weeks or even months, only to discover later that that did not result in improved performance of your overall system.

And so using error analysis to decide where to focus your effort turns out to be incredibly useful for improving your efficiency. In this video, we went over error analysis with the research agent example. But I think error analysis is such an important topic, I want to go over some additional examples with you. So let's go on to the next video where we'll look at more examples of error analysis.

An agentic workflow may comprise many different types of components, and so your tools for improving different components will be pretty different. But I'd like to share with you some general patterns I've seen. Some components of your agentic workflow will be non-LLM based. So it may be something like a web search engine or a text retrieval component if that's part of your RAG or retrieval augmented generation system.

Some people use code execution or maybe a separately trained machine learning model, maybe for speech recognition or detecting people in pictures, and so on. So sometimes these non-LLM based components will have parameters or hyperparameters that you can tune.

So for web search, you can tune things like the number of results or maybe the date range that you ask the web search engine to consider. For a RAG text retrieval component, you might change the similarity threshold that determines what pieces of text it considers similar, or the chunk size. Often RAG systems will take text and chop it up into smaller chunks for matching.

So the main hyperparameters you could use, or for people detection, you might change the detection threshold. So how sensitive it is and how likely it is to declare this is a person. And this would trade off the false positives and false negatives.

If you didn't follow all the details of the hyperparameters I just discussed, don't worry about it. The details aren't that important, but often the components have parameters that you can tune. And then of course you can also try to replace the component. I do this a lot in my agentic workflows where I'll swap in different RAG search engines or swap in different RAG providers and so on, just to see if some other provider might work better.

Because of the diversity of non-LLM based components, I think the techniques for how to improve it will be more diverse and dependent on exactly what that component is doing.

For an LLM based component, here are some options you might consider. One would be to try to improve your prompts. So maybe try to add more explicit instructions, or if you know what few-shot prompting is, that refers to adding one or more concrete examples of an example of an input and a desired output.

And so few-shot prompting, which you can learn about from some DeepLearning.AI short courses as well, is a technique that can give your LM some examples to hopefully help it get better performing outputs.

Or you can also try a different LLM. So with AI or other tools, it could be pretty easy to try multiple LLMs, and then you can use evals to pick the best model for your application.

Sometimes if a single step is too complex for one LLM to do, you can consider if you want to decompose the task into smaller steps, or maybe decompose it into a generation step and then a reflection step. But more generally, if you have instructions that are very complex all within one step, maybe a single LLM has a hard time following all those instructions.

And if you break the task down to smaller steps, that may be easier for, say, two or three calls to an LLM to carry out accurately.

And lastly, something to try when the other methods aren't working well enough is to consider fine-tuning a model. This tends to be quite a bit more complex than the other options. So it can be quite a bit more expensive as well in terms of developer time to implement. But if you have some data that you can use to fine-tune an LLM on, that could give you much better performance than prompting alone.

So I tend not to fine-tune a model until I've really exhausted the other options, because fine-tuning tends to be quite complex. But for applications where after trying everything else, if I'm still at, you know, say 90% performance or 95% performance and I really need to eke out those last few percentage points of improvement, then sometimes fine-tuning my own custom model is a great technique to use.

I tend to do this only on the more mature applications because of how costly it is.

It turns out that when you're trying to choose an LLM to use, one thing that's very helpful for you as a developer is if you have good intuitions about how intelligent or how capable different large language models are. One thing you can do is just try a lot of models and see what works best. But I find that as I work with different models, I start to hone intuitions about which models work best for what types of tasks.

And when you hone those intuitions, you can be more efficient as well in writing good prompts for the model as well as choosing good models for your task. So I'd like to share with you some thoughts on how to hone your intuition on what models will work well for your application.

Let's illustrate this with an example of using an LLM to follow instructions to remove or to redact PII, or personally identifiable information, to get out to remove private sensitive information.

For example, if you are using an LLM to summarize customer calls, then maybe one summary is on July 14th, 2023, Jessica Alvarez with a social security number at a certain address, support ticket, and so on. So this piece of text has a lot of sensitive personally identifiable information.

Now let's say we want to remove all PII from such summaries because we want to use the data for downstream statistical analysis of what customers are calling about and to protect customer information. We want to strip out that PII before we do that downstream statistical analysis.

So you might prompt an LLM with instructions to identify all cases of PII in the text below and then return the redacted text with redacted colon and so on.

It turns out that the larger frontier models tend to be much better at following instructions, whereas the smaller models tend to be pretty good at answering simple factual questions, but are just not as good at following instructions.

If you run this prompt on a smaller model, the open-weight Llama 3.1 model with 8 billion parameters, then it may generate an output like this. It says the identified PII is social security number and address. And then it redacts it as follows and so on.

And it actually makes a few errors. It didn't follow the instructions properly. It showed the list, then redacted the text, then returned another list which it wasn't supposed to. And in this list of PII it missed the name, and then I think it also didn't redact part of the address. So details aren't important, but it didn't follow these instructions perfectly and maybe it missed a little bit of PII.

In contrast, if you use a more intelligent model, one that's better at following instructions, you may get a better result like this, where it actually correctly listed all the PII and correctly redacted all of the PII.

And so I find that as different LLM providers specialize on different tasks, different models really are better for different tasks. Some are better at coding, some are better at following instructions, some are better at certain niche types of facts. And if you can hone your intuition for what models are more or less intelligent and what types of instructions they are more or less able to follow, then you'll be able to make better decisions as to what models to use.

So to share a couple tips on how to do this, I encourage you to play with different models often. So whenever there's a new model release, I'll often go try it out and try out different queries on it, both closed-weight proprietary models as well as open-weight models.

And I find that sometimes having a personal set of evals might also be helpful, where there's a set of things you ask a lot of different models that might help you calibrate how well they do on different types of tasks.

One of the things that I do a lot that I hope will be useful to you is I spend a lot of time reading other people's prompts. So sometimes people publish their prompts on the internet and I'll often go and read them to understand what best practices in prompting looks like.

Or I often chat to my friends at various companies, including some of the frontier model companies, and kind of share my prompts with them, take a look at how they prompt. And sometimes I'll also go to open-source packages of, you know, written by people I really respect and download the open-source package and dig through that open-source package to find the prompts the authors have written in order to read it and hone my intuition about how to write good prompts.

This is one technique that I encourage you to

Consider is by reading lots of other people's prompts. That will help you get better at writing prompts yourself. And I certainly do this a lot, and I encourage you to do so too. And this will hone your intuition about what types of instructions models are good at following and when to say certain things to different models. In addition to playing with models and reading other people's prompts, if you try out lots of different models in your agentic workflows, that also lets you hone your intuition. So you see which models work best for which types of tasks, and either looking at traces to get it in a formal sense or looking at either component-wise or end-to-end eval can help you assess how well different models are working for different parts of your workload. And then you start to hone intuitions about not just performance but maybe also price and speed tradeoffs for the use of different models. And one of the reasons I tend to develop my agentic workflows of AI suite is because it then makes it easy to quickly swap out and try out different models. And this makes me more efficient in terms of trying out and assessing which models work best for my workflow.

So, we've talked a lot about how to improve the performance of different components to hopefully improve the overall performance of your end-to-end system. In addition to improving the quality of the output, one of the things you might want to do in your workflows is to optimize the latency as well as cost. I find that for a lot of teams, when you start developing, usually the number one thing to worry about is just are the outputs sufficiently high quality. But then when the system is working well and you put it into production, then there's often value to make it run faster as well as run at lower cost as well. So in the next video, let's take a look at some ideas for improving cost and latency for agentic workflows.

When building agentic workflows, I'll often advise teams to focus on getting high quality outputs and to optimize cost and latency only later. It's not that cost and latency don't matter, but I think getting the performance or the output quality to be high is usually the hardest part. And then only when it's really working, then maybe focus on the other things. One thing that's happened to me a few times was my team built an agentic workflow and we shipped it to users, and then we were fortunate enough to have so many users use it that the cost actually became a problem, and then we had to, you know, scramble to bring the cost back down. But that's a good problem to have. So I tend to worry about cost usually less. Not that I ignore it completely, but it's just lower down my list of things to worry about until we have so many users that we really need to bring the cost down per user. And then latency, I tend to worry a bit about it, but again, not as much as just making sure the output quality is high. But when you do get there, it will be useful to have tools to optimize latency and cost. Let's take a look at some ideas on how to do that.

If you want to optimize the latency of an agentic workflow, one thing I will often do is then benchmark or time the workflow. So in this research agent, it takes a number of steps, and if I were to time each of the steps, maybe it takes 7 seconds to generate the search terms. Web search takes 5 seconds, this takes 3 seconds, this takes 11 seconds, and then writing the final essay takes 18 seconds on average. And it is then by looking at this overall timeline that I can get a sense of which components have the most room for making faster. In this example, there are maybe multiple things you could try if you haven't already taken advantage of parallelism for some steps like maybe web fetch. Maybe it's worth considering doing some of these operations in parallel, or if you find that some of the LLM steps are taking too long. So if this first step takes 7 seconds, this last LLM step takes 18 seconds, I might also consider trying a smaller, maybe slightly less intelligent model to see if it still works well enough for that, or if I can find a faster LLM provider. There are lots of APIs online for different LLM interfaces, and some companies have specialized hardware that allow them to serve certain models much faster. So sometimes it's worth trying different providers to see which ones can return tokens the fastest. But at least doing this type of timing analysis can give you a sense of which components to focus on in terms of reducing latency.

In terms of optimizing cost, a similar calculation where you calculate the cost of each step would also let you benchmark and decide which steps to focus on. Many LLM providers charge per token based on the input and output length. Many API providers charge per API call, and the computational steps may have different costs based on how you pay for server capacity and how much the service costs. And so for a process like this, you might decide in this example that the tokens for this step on average cost 0.04 cents. Each web search API maybe costs 1.6 cents. Tokens cost this much. API call costs this much. PDF to text costs this much. Tokens for the final essay generation cost this much. And this will maybe again give you a sense of are there cheaper components you could use or cheaper LLMs you could use to see where the biggest opportunities for optimizing costs are. And I found that these benchmarking exercises can be very clarifying, and sometimes they'll clearly tell me that certain components are just not worth worrying about because they're not that material a contributor to either cost or latency. So I find that when either cost or latency becomes an issue, by simply measuring the cost and/or latency of each step, that often gives you a basis with which to decide which components to focus on optimizing.

So we're nearly at the end of this module. I know we've covered a lot, but thank you for sticking with me. Let's go on to the final video of this module to wrap up.

We've gone through a lot of tips for driving a disciplined, efficient process for building agentic AI systems. I'd like to wrap up by sharing with you what it feels like to be going through this process. When I'm building these workflows, I feel like there are two major activities I'm often spending time on. One is building, so writing software, trying to write code to improve my system. And the second, which sometimes doesn't feel like progress, but I think is equally important, is analysis to help me decide where to focus my build efforts next. And I often go back and forth between building and analyzing, including things like error analysis. So for example, when building a new agentic workflow, I'll often start by quickly building an end-to-end system, maybe even a quick and dirty implementation. And this lets me then start to examine the final outputs of the end-to-end system or also read through traces to get a sense of where it's doing well, where it's doing poorly, based on even just looking at traces. Sometimes this will give me a gut sense of which individual components I might want to improve. And so I might go tune some individual components or keep tuning the overall end-to-end system. As my system starts to mature a little bit more, then beyond just manually examining a few outputs and reading through traces, I might start to build evals and have a small data set, maybe just 10, 20 examples, to compute metrics at least on end-to-end performance. And this then further helps me have a more refined perspective on how to improve the end-to-end system or how to improve individual components. As it matures even further, my analysis then becomes maybe even more disciplined, where I start to do error analysis and look through the components and try to count up how frequently individual components led to subpar outputs. And this more rigorous analysis then lets me be even more focused in deciding what components to work on next or inspire ideas for improving the overall end-to-end system. And then eventually, when it's even more mature to drive more efficient improvements at the component level, that's when I might also build component-level evals. And so the workflow of building an agentic system often goes back and forth. It's not a linear process. We sometimes tune the end system, then do some error analysis, then improve a component for a bit, then tune the component-level evals, and I tend to bounce back and forth between these two types of techniques. And what I see less experienced teams often do is spend a lot of time building and probably much less time analyzing with error analysis, building evals, and so on would be ideal because it's this analysis that helps you really focus where to spend your time building. And just one more tip: there are actually quite a few tools out there to help with monitoring traces, logging runtime, computing costs, and so on. And those tools can be helpful. I sometimes use a few of them. And quite a few of DeepLearning.AI's short course partners offer those tools, and they do work well. I find that for agentic workflows I end up working on, most agentic workflows are pretty custom, and so I end up building pretty custom evals myself because I want to capture the things that work incorrectly with my system. So even though I do use some of those tools, I also end up building a lot of custom evals that are well fit to my specific application and the issues I see with it.

So, thanks for sticking with me this far to the end of the fourth of five modules. If you're able to implement even a fraction of the ideas from this module, I think you'll be well ahead of the vast majority of developers in terms of your sophistication at implementing agentic workflows. Hope you found these materials useful, and I look forward to seeing you in the final module. We'll talk about some more advanced design patterns for building highly autonomous agents. I'll see you in the last module of this course.

Welcome to this final module where you learn about design patterns that let you build highly autonomous agents where you don't need to hardcode in advance the sequence of steps to take but can be more flexible and decide for itself what steps it wants to take to accomplish a task. We'll talk about the planning design pattern and then later in this module how to build multi-agent systems. Let's dive in. Suppose you run a sunglasses retail store and have information on what sunglasses are in your inventory stored in a database. You might want a customer service agent to be able to answer questions like, do you have any round sunglasses in stock that are under $100? This is a fairly complex query because you have to look through the product descriptions to see what sunglasses are round, then look at what is in stock, and then finally see what's under $100 in order to tell the customer, yes, we have classic sunglasses. So, how do you build an agent to answer a broad range of customer queries like this and many others? In order to do so, we're going to give the LLM a set of tools to let it get item descriptions such as look up if different glasses are round, check inventory, maybe process item returns, which is not needed for this query but may be needed for other queries, get item price, check past transactions, process item sale, and so on. In order to let an LLM figure out what's the right sequence of tools to use to respond to the customer request, you might then write a prompt like this: you have access to the following tools, and then give it a description of each of the say six tools or even more tools that the LLM has, and to then tell it to return a step-by-step plan to carry out the user's request. In this case, to answer this query, a reasonable plan that an LLM might output might be to first use get item descriptions to check the different descriptions to find the round sunglasses, and then use check inventory to see if they're in stock, and use get item price to see if the in-stock results are less than $100. After an LLM outputs this plan with three steps, what we can do is then take the step one text that is this text written here in red and pass that to an LLM, maybe with additional context about what are the tools, with the user query, with additional background context, and task an LLM to carry out step one. And in this case, hopefully the LLM will choose to call the get item descriptions to get the appropriate descriptions of items. And the output of that first step can let it select which are the round sunglasses. And that output of step one is then passed together with the step two instructions that would be these instructions I have here in blue to an LLM to then execute the second step of the plan. Hopefully, it will then take the two pairs of round sunglasses we found on the previous slide and check the inventory. And the output of that second step is then used in another LLM call where you have the output of the second step as well as the instructions of what to do for step three. Pass the LLM to have it get the item price. And finally, this output is fed back to them one last time to generate the final answer for the user.

In this slide, I've simplified a lot of details a little bit. The actual plan typically written by the LLM is more detailed than these simple one-line instructions. But the basic workflow is to have an LLM write out multiple steps of a plan and then task it to execute each step of the plan in turn with some appropriate surrounding context about what is the task, what are the tools available, and so on. And the exciting thing about using an LLM to plan this way is that we did not have to decide in advance what is the sequence in which to call tools in order to answer a fairly complex customer request. If a customer were to make a different request such as, I would like to return the gold frame glasses that I had purchased but not the metal frame ones, then you can imagine an LLM similarly being able to come up with a different plan to figure out based on what they had purchased previously which glasses they had bought based on get item descriptions, where the gold frame ones they want to return, and then maybe call process item return. So with an agent that can plan like this, it can carry out a much wider range of tasks that can require calling many different tools in many different orders. One more example of planning. Let's take a look at an email assistant. If you want to be able to tell your assistant to please reply to that email invitation from Bob in New York and archive his email, then an email assistant may be given tools like this to search email, move an email, delete an email, and send an email. And you might write an assistant prompt saying you have access to the following tools. And again, please return the step-by-step plan. In this case, maybe the LLM will say the steps for this are to use search email to find the email from Bob that mentioned dinner in New York, and then generate and send an email to confirm attendance, and then lastly move that email to the archive folder. Given this plan, which looks reasonable, you would then again task an LLM step by step to carry out this plan. So the text from the first step shown here in red will be fed to the LLM with additional background context and hopefully trigger search email. Then the output of that can be given to an LLM again with the step two instructions to send an appropriate response. And then finally, assuming the email was sent successfully, you can take that output and have the LLM execute the third step of moving the email from Bob into the archive folder.

The planning design pattern is already used successfully in many highly agentic coding systems where if you ask it to write a piece of software to build some fairly complex application, it might actually come with a plan to build this component, build this component, build this component to almost form a checklist and then do those steps one at a time to build a decently complex piece of software. For many other applications, the use of planning is still maybe more experimental, is not in very widespread use. And one of the challenges of planning is it makes the system sometimes a little bit harder to control because you as a developer, you don't really know at runtime what plan it will come up with. And so I think outside highly agentic coding systems where it actually works really well, adoption of planning is still growing in other sectors. But this is exciting technology and I think it will keep getting better and we'll see it in more and more applications. The cool thing about building agents that can plan for themselves is you don't need to hardcode in advance the exact sequence of steps an agent may take to carry out a complex task.

Now I know that in this video I've gone over the planning process at a fairly high level with it outputting a list of steps and then tasking an LLM to carry out the steps of the plan one step at a time. But how does this actually work? In the next video, we'll take a deeper dive to look further into the guts of what these plans actually look like and how to string it together to have an LLM plan and execute a plan for you. Let's take a look at that in the next video.

In this video, we'll look in detail at how to prompt an LLM to generate a plan and then how to read, interpret, and execute that plan. Let's dive in. This is a plan that you saw in the previous video for the customer service agent, and I have presented this plan at a high level using simple text descriptions. Let's take a look at how you can get an LLM to write very clear plans that go a little bit beyond these simple high-level text descriptions. It turns out that many developers will ask an LLM to format the plan it wants executed in JSON format because this allows downstream code to parse what exactly are the steps of a plan in a relatively clear and unambiguous way, and all the leading LLMs are pretty good at generating JSON outputs at this point. So the system prompt might say something like this: you have access to the following tools, and then create a step-by-step plan in JSON format, and you might describe the JSON format in enough detail with a goal of getting it to output a plan like that shown here on the right. So in this JSON output, it creates a list where the first list item has clear keys and values that say the step one of the plan has the following description and it should use the following tool with the following arguments passed to that tool. Then after that, step two of the plan is to carry out this task and then use this tool and so on. So this JSON format, as opposed to writing the plan in English, allows downstream code to more clearly parse out exactly what are the steps of the plan so that it can be reliably executed one step at a time. Instead of JSON, I also see some developers use XML where you can use XML delimiters or you use XML tags to clearly specify what are the steps of the plan and what step number it is. Some developers, I feel like fewer developers will use markdown, which is just sometimes slightly more ambiguous in terms of how we parse it. And I think plain text is maybe the least reliable of these options. But I think either JSON, which I'm showing here, or XML would be good options for how to ask the LLM to format a plan unambiguously.

So that's it. By outputting plans in JSON, you can then parse it and have downstream workflows execute different steps of the plan more systematically. Now, in terms of getting LLMs to plan, it turns out there's one other really neat idea that lets an LLM output very complex plans and get them executed reliably, and that's to have an LLM write code and to have code express the plan. Let's take a look at this in the next video.

We've talked a lot about how to build a single agent to complete tasks for you. In a multi-agent or multi-agentic workflow, we instead have a collection of multiple agents collaborate to do things for you. When some people hear for the first time about multi-agent systems, they wonder why do I need multiple agents? It's just the same LLM that I'm prompting over and over, just one computer. Why do I need multiple agents? I find that one useful analogy is even though I may do things on a single computer, we do decompose work in a single computer into maybe multiple processes or multiple threads. And as a developer thinking even though it's well one CPU on a computer, say thinking about how to take work and decompose it into multiple processes and multiple computer programs to run, that makes it easier for me as a developer to write code. And in a similar way too, if you have a complex task to carry out, sometimes instead of thinking about how to hire one person to do it for you, you might think about hiring a team of a few people to do different pieces of the task for you. And so in practice, I found that for many developers of agentic systems, having this mental framework of not asking what's the one person I might hire to do something, but instead would it make sense to hire people with three or four different roles to do this overall task for me, that helps give another way to take a complex thing and decompose it into subtasks and to build for those individual subtasks one at a time. Let's take a look at some examples of how this works. Take the task of creating marketing assets. Say you want to market sunglasses. Can you come up with a marketing brochure for that? You might need a researcher on your team to look at trends on sunglasses and what competitors are offering. You might also have a graphic designer on your team to render charts or nice-looking graphics of sunglasses. And then also a writer to take the research, take the graphic assets, and put all together into a nice-looking brochure or to write a research article. You might want a researcher to do online research, a statistician to calculate statistics, a lead writer, and an editor to come up with a polished report or to prepare a legal case. Real law firms will often have associates, paralegals, maybe an investigator. And we naturally, because of the way human teams do work, can think of different ways that complex tasks can be broken down into different individuals with different roles. So these are examples of when a complex task will already naturally decompose into subtasks that different people with different skills can carry out. Take the example of creating marketing assets. Look into detail into what a researcher, graphic designer, and writer might do. A researcher might have the task of analyzing market trends and researching competitors. And when designing the research agents, one question to keep in mind is what are the tools that the researcher may need in order to come up with a research report on market trends and what competitors are doing. So one natural tool that an agentic researcher might need to use would be web search, just as a human researcher asked to do these might need to search online in order to come up with their report. Or for a graphic designer agent, they might be tasked with creating visualizations and artwork. And so what are the tools that an agentic graphic designer might need? Well, they may need an image generation and manipulation APIs, or maybe similar to what you saw with the coffee machine example, maybe it needs code execution to generate charts. And then lastly, the writer has transformed the research into report text and marketing copy. And in this case, they don't need any tools other than what an LLM can already do to generate text. In this, in the next video, I'm going to use these purple boxes to denote an agent. And the way you build individual agents is by prompting an LLM to play the role of a researcher or a graphic designer or a writer depending on which agent it is part of. So for example, for the research agent, you might prompt it to say you are a research agent expert at analyzing market trends and competitors, carry out online research to analyze market trends for the sunglasses product and give a summary as well of what competitors are doing. So that would allow you to build a researcher agent, and similarly by prompting an LLM to act as a graphic designer with the appropriate tools and to act as a writer. That's how you can build a graphic designer as well as a writer agent. Having built these three agents, one way to have them work together to generate your final reports would be to use a simple linear workflow or a linear plan in this case. So if you want to create a summer marketing campaign for sunglasses, you might give that prompt to the research agent. The research agent then writes a report that says, "Here are the current sunglasses trends and competitive offerings." This research report can then be fed to the graphic designer that looks at the data the research has found and creates a few data visualizations and artwork options. All these assets can then be passed to the writer that then takes the research and the graphic output and writes the final marketing brochure. The advantage of building a multi-agent workflow in this case is when designing a researcher or graphic designer or writer, you can focus on one thing at a time. So I can spend some time building maybe the best graphic designer agent I can while maybe my collaborators are building research agents and writer agents. And then in the end, we string it all together to come up with this multi-agent system. And in some cases, I'm seeing developers start to reuse some agents as well. So having built a graphic designer for marketing brochures, maybe I'll think about if I can build a more general graphic designer that can help me write marketing brochures as well as social media posts as well as help me illustrate online web pages. So by coming up with what are the agents you might hire to do a task, and this will sometimes correspond to who are the types of human employees you might hire to do a task, you can come up with a workflow like this with maybe even building agents that you could choose to reuse in other applications as well. Now what you see here is a linear plan where one agent, the researcher, does his work, then the graphic designer, and then the writer. With agents, you can also, as an alternative to linear plan, you can also have agents interact with each other in more complex ways. Let me illustrate with an example of planning using multiple agents. So previously you saw how we may give an LLM a set of tools that they can call to carry out different tasks. In what I want to show you, we will instead give an LLM the option to call on different agents to ask the different agents to help complete different tasks. So in detail, you might write a prompt like you're a marketing manager, have the following team of agents to work with, and then give a description of the agents. And this is very much similar to what we're doing with planning using tools, except the tools, the green boxes, are replaced with agents, these purple boxes that the LLM can call on. And then we'll ask it to return a step-by-step plan to carry the user's request. And in this case, the LLM may ask the researcher to research current sunglasses trends and report back. Then it'll ask the graphic designer to create the images and then report back. Then ask the writer to create a report and then maybe will choose to review or reflect on and improve the report one final time. In executing this plan, you would then take the step one text of the researcher, carry out research, then pass that to the graphic designer, pass it to the writer, and then maybe do one final reflection step and then you'd be done. One interesting view of this workflow is as if you have these three agents up here, but this on the left is actually like a fourth agent. There's a marketing manager that is a manager of a marketing team that is setting direction and then delegating tasks to the researcher, the graphic designer, and the writer agents. So this becomes actually a collection of four agents with marketing manager agent coordinating the work of the researcher, the graphic designer, and the writer. In this video, you saw two communication patterns. One was a linear one where your agents took actions one at a time until you got to the end. And the second had a marketing manager coordinating the activity of a few other agents. It turns out that one of the key design decisions you may end up having to make when building multi-agentic systems is what is the communication pattern between your different agents. This is an area of hot research and there are multiple patterns emerging. But in the next video, I'm going to show you what are some of the most common communication patterns for getting your agents to work with each other. Let's go see that in the next video.

When you have a team of people working together, the patterns by which they communicate can be quite complex, and in fact designing an organizational chart is actually pretty complex to try to figure out what's the best way for people to communicate to collaborate. It turns out designing communication patterns for multi-agent systems is also quite complex. But let me show you some of the most common design patterns I see used by different teams today. In a marketing team with a linear plan where first the researcher worked, then the graphic designer, then the writer, the communication pattern was linear. The researcher would communicate with the graphic designer, and then both the researcher and the graphic designer maybe pass the outputs to the writer, and so this very linear communication pattern. This is one of the two most common communication plans that I see being used today. The second of the two most common communication plans would be similar to what you saw in this example with planning using multiple agents where there is a manager that communicates with a number of team members and coordinates their work. So in this example, the marketing manager decides to call on the researcher to do some work. Then if you think of the marketing manager as getting the report back and then sending it to the graphic designer, getting a report back, then sending to the writer, this would be a hierarchical communication pattern. If you're actually implementing a hierarchical communication pattern, it'll probably be simpler to have the researcher pass the report back to the marketing manager rather than the researcher pass results directly to the graphic designer and to the writer. But so this type of hierarchy is also a pretty common way to plan the communication patterns. We have one manager coordinating the work of a number of other agents. And just to share with you some more advanced and less frequently used but nonetheless sometimes used in practice communication patterns. One would be a deeper hierarchy where same as before if you have a marketing manager send task a researcher, graphic designer, writer, but maybe the researcher has themselves two other agents that they call on such as a web researcher and a fact checker. Maybe the graphic designer just works by themselves, whereas the writer has an initial style writer and a citation checker. So this would be a hierarchical organization of agents in which some agents might themselves call other sub-agents, and I also see this used in some applications, but this is much more complex than a one-level hierarchy, though used less often today. And then one final pattern that is quite challenging to execute but I see a few experimental projects use it is the all-to-all communication pattern. So in this pattern, anyone's allowed to talk to anyone else at any time. And the way you implement this is you prompt all four of your agents in this case to tell them that there are three other agents it could decide to call on. And whenever one of your agents decides to send a message to another agent, that message gets added to the receiver agent's context. And then the receiver agent can think for a while and decide when to get back to that first agent. And so if they all collaborate in a crowd and talk to each other for a while until say each of them declares that it is done with this task and then it stops talking, and maybe when everyone thinks it's done or maybe when the writer concludes it's good enough, that's when you generate the final output. In practice, I find the results of all-to-all communication patterns a bit hard to predict. So some applications don't need high control. You can run it and see what you get. If the marketing brochure isn't good, maybe that's okay. You just run it again and see if you get a different result. But I think for applications where you're willing to tolerate a little bit of chaos and unpredictability, I do see some developers using this communication pattern. So that I hope conveys some of the richness of multi-agentic systems. Today there are quite a lot of software frameworks as well that support easily building multi-agent systems, and they also make implementing some of these communication patterns relatively easy. So maybe if you use your own multi-agent system, you find some of these frameworks helpful for exploring these different communication patterns as well.

And so that now brings us to the final video of this module and of this course. Let's go on to the final video to wrap up.

Welcome to the final video of this course. It feels like we've been through a lot together, just you and me, and we've gone through a lot of topics in agentic AI. Let's take a look. In the first module, we talked about what are the applications you can build with agentic AI that just were not possible before. And we started then to look at key design patterns including the reflection design pattern, which is a simple way to sometimes give your application a nice performance boost. And then two, using function calling, which expands what your LLM application can do, with code execution being one important case of that. And then we spent a lot of time talking about evaluations as well as error analysis and how to drive a disciplined process of building as well as analyzing to be efficient in how you keep on improving the performance of your agentic AI system. This fourth module is some of the material that I think you will find most useful as you keep building agentic AI systems, I hope for a long time. And then in this module we talk about planning and multi-agent systems that can let you build much more powerful, although sometimes harder to control and harder to predict and advance types of systems. So with the skills you learn from this course, I think you now know how to build a lot of cool, exciting agentic AI applications. When my team or I see other teams as well interview people for jobs, I find interviews often try to assess whether or not candidates have pretty much the skills you're learning about in this course. And so I hope that this course will also open up new professional opportunities for you and you just do more. Whether you're doing these things for fun or for professional practical settings, I think you'll enjoy this new set of things you can now build. So just to wrap up, I want to thank you again for spending all this time with me, and I hope you will take these skills, use them responsibly, and just go build cool stuff.