I'm so grateful to be here. I'm from Stanford GSB, so I'm from a business background. I have zero coding experience. I seriously considered coding two years ago and I studied it many, many times. Today I had a great experience because I have no idea what a project would look like until I see it. When I was listening this morning, I really resonated with you talking about the modularization of the agent. We think about prompts, we think about MCP, we think about all kinds of things, right? Eval. So I'm thinking about, there's evaluation for agents, is there evaluation for agentic fasted programming? What is the best practice? Because everybody is using different tools, everybody uses spec-driven, and people are talking about context engineering, not prompt engineering, right? We come up with our own working process, but sometimes it turns out very good, but sometimes it's not. So we really wanted to clear your perspective in terms of what we can, you know, do better.
Right, so maybe share. I think best practices for coding, frankly, it's evolving so quickly. Even today, listening to the panelists or chatting to people in the hallway, I'm learning as well. So I don't know that there's any single world best practice, it's evolving so quickly. But I think some of the best practices I know about, you know, test-driven development. But one thing I found is there's some components when we know we build on top of, so test those much more rigorously. The things that you're not building on top of, like front end, I test that less rigorously. But backend database, I really test that because to avoid really weird bugs I can't find later. The spec is important. I find that fast iterations really helps drive product progress. Yeah, I don't know. I think the question of how do you orchestrate multiple agents to work together? I see lots of different people doing lots of different things to coordinate it. It feels like wild west. Very exciting. Wild west. I think there's a lot of issues with MCP was a great step forward. The challenges of MCP is it's a long flat list of APIs. And so I think eventually we need to evolve the protocols to more hierarchical discovery. So there's a lot of this going on. So I think the AI... oh, so one tip I have as coding, actually one thing I do a lot is try to read other people's prompts. It turns out that one of the ways I learned to be a decent prompter, hopefully, is whenever some system prompt is leaked, I will go and read that. Whenever there's a research paper with really cool results, I will sometimes go and download the open source and go and read the prompts. Because I find that a lot of people... yeah, the only way to be good at prompting is to read other people's prompts. One thing about agentic workflows, they're not very mature. One of the biggest factors that determines your ability to build an agent workflow is the ability to drive a disciplined error analysis process. And this is surprisingly... but the best practices is, some agent workflows, you have an AI agent that, you know, for example, download a PDF, extract the text, do a web search to check if you're working with someone that's sanctioned if it's compliance work, with database and pricing, and email. So these agentic workflows, I find that the ability to quickly build an end-to-end workflow and then have eval, and then to also do error analysis so that when the AI agent doesn't do what a human expert would do, to really carefully attribute the gap between the agent and the human to which component is misbehaving so you can focus attention on the right component to work on. Right now I see still very large differences between the team skill at driving the disciplined error analysis process. Then it's not, you're not guessing what to work on, you're the confident agent being diagnosed where the hot spots are and where it's underperforming and really focus attention on that. And it comes from having an eval set and being very systematic and looking where's the AI agent performing short of what human experts would do and using that to focus attention. It just makes... I wish you could teach that to all people, but it makes a huge difference to the velocity of the team.
What are the limits right now for this AI coding? Like, we can see it's creating SaaS applications which we have been seeing, but what about like large back-end software? Like, you know, how RDBMS is made. Do you see AI... because it may not take, you know, like decades to make databases, you know what I mean? Do you think it's possible to get a lot of help from this?
So I think Dario from Anthropic, right? I think he said that a lot of Anthropic researchers haven't written a line of code by hand for a long time. So I think we are seeing really serious software being written with AI coding agents. What I'm seeing is that for the more common tasks, like, you know, normal front-end, back-end development, there's more training data and so it's really great at that. Providing really weird research, it turns out, by the way, it's pretty good implementing a transformer network, right? There's enough data on that. But for really weird corner cases or some types of, you know, low-level GPU programming, there's less training data and so they're actually less reliable. Or for real corner cases with building very weird research algorithms that no one else has implemented, you know, you need to go low-level in your prompt. Whereas for a front-end, there's so many React front-ends, right? You can high-level prompts, usually pretty sensible interpretation. And then it turns out that, I don't know if you look at a lot of the recent progress, the models are getting really good. So I think Claude is an amazing model, GPT-5 is an amazing model. But I find that a lot of the power of these is not just the model, it's the wrapper that, you know, the Claude Code like CLI or Replit or whatever was wrapping around these models. No, no, I think in the report, the Ving study showing that... I don't remember the citation, there's a really cool study showing that you look at tasks and humans can do, kind of two years ago, AI could auto-complete, could maybe complete tasks that humans would take, you know, 30 seconds to do, it could do that reliably. And what's been happening is tasks that take humans a minute, 5 minutes, 10 minutes, 20 minutes has been kind of doubling of how difficult a task is as measured by how long a human needs to do it, that now AI can inclusively, autonomously do. So that has been a very exciting trend, that kind of a doubling, rapid doubling of how much code can a human write in a certain amount of time that now AI could do.
So from like a coding, AI is the new electricity, all the way back to deep learning and deep unsupervised learning using graphic processors. What's another Andrew Ng belief that the world hasn't caught on to yet?
By the way, you know, part of me misses the days that I would say something and then nobody would believe me, right? And then I remember when I said we got to scale up, you know, machine learning efforts more, like a lot of people did not believe me, right? And then I just worked on it quietly for myself for a long time and eventually people came around. Now I actually worry that I might say something dumb and then people believe me. I don't know. Yeah, I think for the next few years there's a lot of work to be done to take a lot of enterprise workflows and implement them as agentic workflows. There's one thing I think is exciting. I think people still underestimate the power of the voice stack. I think in the future there'll be a lot more voice-based applications than is widely appreciated now. I think visual AI is also coming a little bit slowly. Landing AI is seeing that agentic document extraction. Turns out surprisingly the most valuable type of image is images of PDF files. Someone told me that humanity generates more than a trillion pages of documents per year. And so agentic doc extraction is really taking off like that, right? The ability to extract, separate PDF files with downstream processing, but really seeing kind of classic, you know, really rapid growth, organic or not. Yeah, so it seems like a lot of work to do. Oh, maybe one other thing. Oh, one thing that people don't believe me on yet, I think everyone should learn to code, not just software agents.
Do you mean learning to code by yourself without... or doing it with...?
Oh, right. Just, I'll repeat the question. So people should learn to code by hand or with AI assistance. Learn to do it with AI assistance. Yeah. But just on my team at AI Fund and DeepLearning.AI, you know, everyone's learned to code. I see a lot of people whose job function is not software engineer do things that are much more effective. They're not becoming software engineers, and nor do I believe want everyone to. But you know, when non-technical people in non-technical worlds learn to code, you're a better marketer, you're a better recruiter, you're a better HR professional, you're a better finance professional. So I think the world will still need quite a bit of time to figure that out.
Thank you, Professor. So a lot of these models that are used today have been trained on software that's already been written. Now, in the future, how will new patterns of software building emerge and propagate into these models and become widespread? What's that mechanism going to look like?
You know, one thing I wonder about, so in computing we build layers of abstractions and then sometimes there's some low-level abstractions that are, you know, okay, right? Not great, like pretty good, but we just leave it alone and we just build on top of it. So for example, I think Linux is amazing. On the other hand, I don't know, we're probably stuck with it maybe in perpetuity, right? Like it's okay. Maybe there's an even better OS to be invented than Linux, but there's so much built on top of it. Probably not worth trying to change it at this point. So part of me wonders... and you know, like x86 assembly, right? Frankly, you know, like we're probably stuck with it for a long time, but it's okay. So part of me wonders with so much code on the internet, with Python and JavaScript, will Python and JavaScript, you know, be the abstraction layer and maybe C++ and Java or whatever for a long time? And I think it'll be okay if it is because it's not that bad, right? But if someone invents a new programming language, that's okay too. I do see what I mentioned just now, a lot of struggle with lower-level GPU programming because the data isn't there. And so, yeah, I think... oh, one interesting thing. So the Llama paper, the Llama 3.1 or 3.3 paper, and Surf using an earlier version of Llama and an agentic workflow to basically generate coding puzzles. The AI thinks for a long time to generate coding puzzles and solutions, and this generates data to train the next generation of model that where it goes to solve the same coding puzzle very quickly, right? They just generate the output directly without as long an agentic loop. So I think as AI systems with the agentic rules can take a long time to solve puzzles, generates data to train the next gen of AI to try to solve that more quickly. So there's a path to synthetic data generation, which I think is quite exciting as well.
Thank you, Andrew. I have one question. It is very open-ended. So when we talk about LLMs on a very fundamental level, it is the prediction of next token given the previous tokens, right? But when we talk about AGI, okay, nowadays people are like craving about AGI is coming. And when I sense, like when I compare it with the human minds, okay, we basically think based on gut, based on beliefs, and there are a lot of other factors involved. But in order to reach AGI, where my mindset goes and like basically the bottleneck arrives is the consciousness in terms of like LLM or evolving it to the AGI. How do you think that this bottleneck, where it is right now we see it as a next token prediction, is going to evolve in a way where the model can surpass this bottleneck of just predicting the next output into predicting something that we as a human do based on beliefs, consciousness? Like, are we too far?
