The Future of AI Agents with Andrew Ng | Interrupt 26

It's good to be back. This has always been such a cool gathering that you put together. I was telling you this earlier, but Andrew's fireside chat last year was the most liked and the most watched on YouTube afterwards. So, we're thrilled to have him back for year two.

So I think the hype has exceeded my expectations. And then also the doomsaying narratives also got more traction unfortunately, I would have guessed including the job apocalypse which I don't think is going to be a thing. And on the more positive side, coding agents probably took off faster than I would have guessed, and the frontier coding agents. I know the hype is that everything AI changes every few months or whatever and that hype isn't entirely true, but that does feel true for coding agents where it feels like the frontier of what we can do with coding agents and it's still competitive. I think six months ago I was almost all cloud code. These days I'm still a lot of cloud code but also increasingly open AI codecs with a mix of Gemini CLI and want to support open code as well because it's open codes. So the mix of coding agents we use has been changing rapidly. I wouldn't have guessed a year ago that I'd be coding so much on my phone as well. These workflows and like many of you I have a Mac mini in my office, so all these workflows changing really rapidly. And then agentic workflows are really starting to make their way into enterprises. That feels pretty good as well.

So about a year ago I was writing about the product management bottleneck, which is this observation that if building becomes much faster then deciding what to build or the product management work of scoping the project maybe getting customer feedback deciding what to build becomes the bottleneck. It feels like over the last year the product management bottleneck has become much worse in a good way because software building has become much faster. But it turns out when writing software becomes 10 or 100 times faster, not only is there a product management bottleneck, pretty much everything else becomes a bottleneck. So some of my teams have seen the marketing bottleneck because we can build so many features, our marketers are scrambling to figure out what on earth the engineers did in order to figure out how to communicate it. Previously if you had a product that needed legal compliance, if you took three months to build it, waiting a week for legal to sign off is maybe okay. But now if you're building a day, then wait a week for legal to sign off, that's a legal compliance bottleneck. And there's a design bottleneck and so on. So I've been thinking a lot about how software engineering teams in the future will be organized and I don't think I know the answer but increasingly I find myself setting up very small parts, anywhere from one to 10 engineers in a team of often generalists, high context, highly empowered generalists. They're given a set of very wide guard rails within which they can just run like crazy and build and ship code and even drive decisions like writing marketing copy that are traditionally outside the purview of engineering. By definition, if you have a team that needs software engineering, product management, a little bit of new terms of service, some marketing copy, some design, say you need five functions represented in a team of two humans, then by the pigeonhole principle these two humans have to play more than a single role per human. The good news is it turns out I don't think I'm a very good marketer, but when I use AI, I'm still not a good marketer, but I'm slightly less bad compared to if I had to do without AI assistance. So I find these small teams of high context generalists that are deeply technical, but able to use frontier technology to do a little bit of other roles as well. Every engineer can use AI to take the first draft of a terms of service to then take to a lawyer for the lawyer to finally polish it before it goes out. But I find that these processes allow teams to move much faster. So that's been really exciting.

Yeah, so a lot of the people I work with most closely have deep engineering expertise. And then also they are highly empowered slight generalists that step into other roles and acquire this mix of skills often with AI supporting where they didn't have that training. I think people can succeed from any direction. I've seen product managers become much stronger at coding and then participate in these teams. I think just because so much of AI coding and engineers maybe had a natural advantage understanding the frontier tech. So I see engineers play this role successfully the most often but there's definitely a smaller number of product managers. I've seen marketers learn to code in pretty effective ways. I've seen operations people start to build more and more products. I think it's actually possible for people of any background to learn to do a lot of this but the largest number of bodies doing this well right now seems to be people that came from engineering background. But I really encourage everyone from whatever background to see if you can play these roles.

Yeah. So there's one way I've been thinking about the future of software engineering. This is a mental model I have in my head: because of a lot of providers of many tools like RAG, agentic frameworks, evals, guardrails, it turns out that these are AI building blocks, and there are also non-AI building blocks like user interface components, identity mechanisms, frontend, backend, persistence databases and so on. So I think that in computer science we've always had a wonderful set of building blocks and with agentic coding, building blocks are proliferating because more and more people are building open source or proprietary API-based building blocks. So they're just wonderful building blocks all around us. And so I find that developers that have a good mastery of enough building blocks can often put them together in combinatorially many ways to very rapidly build software. If you picture a building with Lego bricks, if all I have is a white colored Lego brick, I can build some stuff but it's not that interesting. But I mix in a black Lego brick, a yellow one, a brown one, a green one, and throw in some squiggly pieces of Lego, then what I can build grows combinatorially or grows exponentially as a function of the number of Lego bricks I have. And I think of a lot of the building blocks we now have access to as being akin to that too. So I find that the developers are able to have a good sense of what these building blocks do and DeepLearning offers a lot of short courses to help people master these wonderful building blocks provided by tons of people. And then the other challenge is to use coding agents to rapidly assemble these building blocks into the software that you want. One challenge that coding agents have is a lot of building blocks are so new that the coding agents do not know how to use them. Until recently the leading coding agents were released after the knowledge cut off dates of a lot of leading models. So they just had no idea how to call the Nano Banana API or even knew that Nano Banana existed. So one project that one of my friends Roy Prasad and I have been passionate about is Context Hub, which is like a Stack Overflow for AI agents, for AI agents to get the latest documentation on what are the latest APIs, SDKs, building blocks they can use, as well as a way for agents to give feedback to the documentation to try to improve it for everyone. And so it turns out there are a number of APIs that I personally use that I find the syntax slightly annoying to remember. But I find that by using Context Hub to load the latest docs, I let the coding agent accurately make all of these API calls for me, and so it has actually helped my own coding work accelerate quite a bit.

Yeah, so we're trying a lot of ways to improve the education experience. In terms of training, the thing that's clear is that what people have to learn has changed significantly. I think for developers we have to learn coding agents, learn these building blocks, maybe learn some product management or these type of general skills to make us more effective. So what we learn is changing. And DeepLearning and Coursera try to provide a lot of that training. But separately from how to what to learn, there's the delivery of training and it feels like we've been thinking about how will learning transform for a long time and it feels like it's actually not quite here yet. One thing we launched just several weeks ago is a new website in preview called codeream.ai in which the vision was rather than taking an online course, come and have a conversation. So it's not a course, it's a conversation where the experience we tried to build was for you to come and get on a simulated video call with me. If you feel like leaning back and listening to me talk and present to you in a one-on-one simulated video call about Context Hub and coding agents, you lean back and I just show you some ideas. Or if you want to interrupt me or my AI and ask a question, you could do so at any time. And one thing I actually personally played around a lot with is replacing videos and slides with JavaScript. What that means is instead of a video when I present when I demo something in a video, if you could click into the video and type your own prompts or type your own queries into the video window. So instead of a static video that just plays, the video area is interactive and this creates more ways for people to interact. So check out codeream.ai if you want and have a kind of a conversation with me or with my AI where I'll present to you in AI form how to use these coding agents. But I think we're still iterating and improving these experiences every day.

Oh, that's live. So imagine if instead of me screen sharing in a video, I am JavaScript sharing in a video and so you can interact with whatever I'm sharing with because it's running JavaScript code, not a canned video. So hope so, if you play with it we would love your feedback. But it feels like the transformation of education has been overhyped. I think something is coming. But today, taking online courses, I wish we had something way better than online courses. And I will say we have something better than the courses we had 10 years ago. They're much more interactive. For a lot of our courses, we actually this week just launched a new course on transformers taught by AMD Sharon Job. And I find that rather than just having videos, which is mostly what we had a decade ago, we now built much more interactive visualizations, there's a lot more fun stuff to play with than courses had a decade ago. But that bigger transformation, I actually spent a lot of time thinking about that.

Yeah. So, I think every enterprise, I'm guessing all of yours, are excited about AI adoption. One thing we've seen is over the many businesses, turns out one of my teams, AI Aspire, which is an AI advisory firm that my business partner Kirsty Tan and I co-founded. We talk to large businesses all the time, Fortune 50, Fortune 500, G2000 businesses about AI strategies and transformation. So some consistent themes: all of us have invested in the bottom-up innovation, let a thousand flowers bloom strategy, and for the most part it is not paying off. So CEOs and boards are asking where is the ROI for AI. I think we should keep on investing in bottom-up innovation. Let's keep on doing that. But it turns out that bottom-up innovation often results in point solutions that drive incremental efficiency gains which are actually a good thing but not the broader transformation that AI has been promising us and that I think we should work to deliver. To illustrate this as an example, my team's work with a number of banks, we actually do a lot of work in financial services. If you think about the process of underwriting a loan, maybe five steps: market a loan product, get the application, review and approve the loan, do the final diligence, and execute the loan. A number of teams have noticed that the step in the middle of loan approval we could use AI to do that, and if we can automate that, then instead of a human spending an hour reviewing the loan application, we could have AI do it. That's great, we should absolutely do that. But it turns out that if your entire process underwriting the loan stays the same except for automating what was previously one hour of human time, that's a small incremental efficiency gain. So what a number of banks have said is, instead of doing this efficiency gain which is worthwhile, let's rethink the entire workflow and market a get-approved-in-10-minute loan product because rather than waiting around for a week for a human to be free for an hour, we can send the loan application to the AI right away for a decision. But the challenge with implementing this in a lot of businesses is this takes someone with the broader scope to rethink and redesign the entire workflow because now you have to market a get-approved-in-10-minutes product. You have to route the application to approval not in a day but right away. So marketing, data, infra needs to be involved. Then yes, AI can make the initial decision and then final diligence execution pro needs to scale up as well. So I find that bottom-up innovation is really valuable, generates lots of ideas, but often it has to be complemented with a top-down motion of having someone with a broader scope to change how all of these steps operate to then create growth. And I find that many businesses talk about cost savings, that's fine if worth doing, but I try to push for more imaginative things we could do with AI, which is drive growth because we can only save so much money but growth has almost no practical ceiling. So the more exciting ideas I find usually relate to driving business growth rather than savings.

Yeah. So the bank example is a real one, we work with a number of banks on that and financial institutions. I don't know, maybe one other pattern: customer service contact centers often viewed as a cost savings thing, and that's fine, cost savings are nice, but when you can automate customer service or augment part of it, then the ability to serve customers many more much faster delivers a more delightful customer experience and drives growth. I think that actually speaking with a number of businesses that have been working on automating the drive-thru voice application, the drive-thru order process, I think that also results in a more delightful customer experience and drives growth. So I'm seeing that there are more and more of these examples popping up in different places of the economy. There's some I know about that I don't have permission to talk about, but I'm actually pretty confident with the number of things businesses are working on that there'll be more and more examples of these things.

Yeah, I wish I knew. I think part of challenge is businesses are so diverse. So measuring ROI is like measuring business, which is very hard for a one-size-fits-all answer. But there is one thing I feel like the projects that excite me most are measurable, should be measured, deserve measurement, but there are some things where we swing for the fences and create so much value that we're not debating this drive 2% growth and then minus the 1% cost of implementation; it's just glaringly obvious this is transforming the business. Of course we still need to measure it especially if we're a publicly traded company, but I find that those things are to really... There's actually one thing I've learned: sometimes driving incremental gains is harder than driving transformative gains because if you tell someone to improve their business results by 2% next year, they think my boss is telling me to work 2% harder or 5% harder. But if you try to search for ways to drive 20% business growth or 50% business growth, then you can't just get everyone to work 50% harder, and you have to come up with more creative solutions. One lesson I've learned at AI Aspire: we've had many businesses literally send us spreadsheets with hundreds of ideas. One financial institution sent us a spreadsheet with over 300 ideas, and just to help them sort out of these 300 ideas which ones to put real capital behind, it turns out that the analysis is really difficult. I wish I was smart enough to glance through it and say, "Oh, this idea and this idea." But I find that when faced with that many ideas, often brainstorm in a top-down and bottom-up motion, it's actually a lot of work to do the technical analysis to figure out what is possible, and then the business analysis to figure out which ones could drive meaningful change. Then it's actually a lot of work on the technical and business analysis to narrow down to a small handful of bets to put very meaningful resources behind.

Yeah. I find that businesses hopefully not take one wild swing for the fences, but more a portfolio of a handful of thoughtful bets where if anyone pays off it will be meaningful for the business. But it turns out one of the things I love about agentic coding is we run tons of experiments, prototype all the time. So the cost of prototype has plummeted. But sadly you can't do everything on a $100,000 budget, and at some point putting meaningful resources behind every one of a small portfolio of a handful of projects does make sense. But because of the resource allocation needed at that level, it often takes a little bit more of a top-down motion to allocate the amount of resources needed.

Yeah. It's been fascinating how it's remained persistently, I'm going to say, about six to nine months behind the frontier models. But the frontier models are expensive enough that for many use cases my teams use a lot of the open weight models, sometimes fine-tuned, sometimes without fine tuning. So I hope we can all keep on supporting the open weight models. Over the last two weeks there have been concerning noises out of the White House about inspecting models before their release. So I'm actually quite concerned about that and in touch with a few friends in the administration about this. And I feel like there a lot of war on open source open weight models is still being waged sometimes in the name of US-China, sometimes in the name of whatever arguments people are coming up with. But I feel like if we can all protect open source open weights, it will make the world much richer and also help all of us preserve optionality.

Yeah, this is a great question. So when at AI Aspire, when we speak of large businesses, one very common pain point is rethinking the data architecture because over the last 10, 20 years we put so much effort into organizing our structured data, the tables, relational data, the spreadsheets, and that's great, that's clearly important. But now that AI can process unstructured data, text, images, PDF files, audio, maybe video, organizing that to get it to AI or agents in the right time and right place for it to create value is suddenly much more valuable than it used to be. And candidly, I've looked at the market, there are many vendors starting to talk about dealing with unstructured data but I've not been able to find a single good solution that I'm particularly satisfied with. So within my team's AI fund, AI Spire, we've been running a bunch of crazy experiments in building rearchitectures of our own data. If it ever works, I'll probably talk more about it, but I spend a lot of time thinking about how to rearchitect our own internal unstructured data to get it to the agents for the agents to use at the right time. I actually foresee that just as many businesses had very large data architecture problems, a data architecture kind of work to reorganize and structure data over the next few years, there will be very large, tens of millions maybe hundreds of millions of dollars types of projects in many businesses to rethink their data architecture to make their data more AI ready or more agent ready.

Boy, fragmentation, governance, data is all over the place. No consistent schema, some of this sitting on someone's laptop. The permissions were designed for humans, not for agents. So does the agent inherit my permissions? How do we manage governance and observability? I feel like we've all seen so many businesses have massive buckets of tons of PDF files that no one's looked at for the last 20 years. You see in financial services a lot of documents are retained for compliance reasons. So previously there was no point looking at it because no one had the time, but sorting data for AI to look at turns out to be really valuable. Oh, one small thing, this doesn't cover the whole data architecture, but I think CJ is speaking later. One little lesson I've learned for AI coding: I personally use MongoDB a lot. I love relational databases, right? But I find that when I'm iterating and prototyping rapidly, the need to redesign the database schema is so annoying, and we've all had that one in a hundred times that we asked AI to do a database migration and did something clever like erase my whole database instead. It almost never happens but the fact that it almost never happens but doesn't never happen is a little bit annoying. So I find that having a NoSQL where I can dump whatever data I want into a database and then figure out the schema when I'm reading it rather than when I'm writing to the database lets me iterate much faster. NoSQL doesn't always scale to the largest production workload, so very large production workloads enterprise rate eventually I use more relational databases, more scalable solutions, but I think NoSQL is more scalable than most people realize these days and it drives that pace of iteration. I get so frustrated if I've designed some database schema and then I think, oh shoot, I want to add a field, it's so annoying to have to refactor the entire database. So I find that these are examples of the workflow that we're all making to drive faster iteration to take advantage of the fact that AI agents can code really fast. So let's not get slowed down by these other things either.

Thank you. Thank you. Thank you.