Generative AI in the Real World: Andrew Ng on where AI is headed. It’s about agents.

Thanks, Ben. It's always good to see you.

Now I would say back then the controversial thing I did was I ran around saying we need to scale up deep learning algorithms. So that's why I started Google Brain because I saw data in my lab at Stanford that building bigger neural networks with bigger computers, first with more CPUs and later GPUs, that drove performance. And that's a very controversial statement back then. So when I tell my friends I'm going to start Google Brain because I want to use Google's computers to scale up deep learning, some of my friends including very senior people in AI at the time would say, 'Hey Andrew, this is weird. Why are you scaling up deep learning? Focus on the algorithms. The algorithm is where the action is at.' Fortunately, I had enough data and I had internal conviction and I did that and that worked out. But it's actually pretty controversial back then when I tell people we should scale up models.

Yeah, I think my team has started. I think we wrote the first research paper saying that we should use CUDA to scale up deep learning. We weren't the first to look at GPUs for AI but CUDA was the tipping point because before CUDA it was really difficult to do GPGPU, general purpose GPU, but to me CUDA was the tipping point. So we started publishing papers saying, 'Hey everyone, take a look at CUDA, let's take a look at GPUs.' I think that, to me one of the biggest compliments is about your work that used to be controversial is everyone just knows about it so no one bothers to cite you anymore. And I actually feel quite happy that now, of course you use GPUs, of course you scale up deep learning, that it's good that what we used to push that was controversial is now like an obvious fact. I have another controversial one, speaking of GPUs, which is I think many people still underestimate the importance of inference. So GPUs are fantastic for training. I'm actually trying to think through myself, is the hardware for inference going to be different than the best hardware for training? So there are actually multiple businesses at this moment in time working on better inference hardware. And that's actually exciting as well.

Yeah. It's been really nice to see how many people are working and pushing forward data-centric AI in a more systematic way. And I feel like I was at NeurIPS in person last December. I was actually very happy NeurIPS had these big banners on the big topics. So there's one for deep learning, one for, I don't remember, reinforcement learning, and there was a big banner on data-centric AI. So I was actually happy that I was able to play a small role, kind of try to coalesce the community of people that have realized, sometimes for decades, that curating the data is really important, sometimes even more important than curating the model for some application. So seeing so many people, it's been a few months since I checked but the last time I checked the number of research papers that say data-centric AI looked like it's going up very rapidly. So I think it's an important topic. It's good to see so much work on this.

So large language model based agents are able to do things that no prior piece of technology has been able to. In fact, the way that most people use GenAI, say ChatGPT or Claude or Gemini as a consumer, is you write a prompt and have AI write an essay, generate output for you. But that's a bit like if you were to go to a person or in this case go to an AI and say, 'Please write an essay for me on a topic, but I need you to do so by typing from start to finish in one go without ever using backspace.' And you know, you can write like that, probably don't do your best writing that way. And for AI, despite the difficulty of this task, I think large language models do this surprisingly well. But for humans doing writing, you know, who writes a book like that? Hopefully no one. You instead write an outline, critique the outline, do some online research, improve the outline, write a first draft, critique the draft, and it's a much more iterative process. We are writing, thinking, writing, thinking. And that's what agentic workflows let us do, is we're using an LLM over and over to have it write a draft, critique the draft, use a tool to do research, critique it, and iterate a lot more. And this results in a way better work product.

So when my team looked at some data on a coding benchmark called HumanEval, we found that while the improvement in performance from GPT-3.5 to GPT-4 was significant, it was great that we moved from GPT-3.5 to GPT-4, the improvement from 3.5 to 4 is dwarfed by the improvement you get even using 3.5 but adding this type of iterative agentic workflow on top. And what at AI Fund we've actually been seeing is for, I don't know, well half a year, maybe half a year, a year now, we've been helping many of the companies we work on, that we help build, use agentic workflows and it just gets much better performance than zero shot prompting, which is writing from start to finish. So that's why with agentic workflows, which feels like it's kind of in its infancy, the set of tasks we get AI to do is expanding rapidly.

And I think since I wrote that series in the newsletter, The Batch, I thought this was important enough. I just wrote for like five weeks in a row about this, right? Five weeks in a row. I was trying to explain different parts of agentic workflow. I think one of my friends told me, 'Guess what, Andrew? Now they've done this, every company's going to use the word agent and in six months the term will be totally useless and it won't get anything.' It does feel like the term AI, you know, everyone's using AI, I don't know the term is that special. I think in the next six months the term agents will come, everyone will say agents, but that's a sign of progress too. I'm not complaining. I think it's good if everyone has tried their own version of agents.

I think agents can be used for RPA, I think it's actually going to be even bigger than that. So you know, I think that a lot of people had a ChatGPT moment when they used GPT maybe for the first time and saw it significantly surpass expectations for what AI could do. I think a lot of people are going to have an agentic moment when you use a single agent or multi-agent system and it just goes and does the task, maybe by planning out how to do it in a way that was not how you would have tasked it to do, that will amaze and surprise and delight you. So I think the agentic workflow platforms, so I think Joao Moura did a great job building CrewAI, Tinbo and Chuang did a great job building AutoGen, I think Harrison Chase is doing a great job building LangGraph. So I think I see these AI agent platforms developing that makes it much easier for people to build agents on top.

And to me, I've had a few really delightful moments when I had implemented, like actually often the multi-agent system to do a task and it does it in a way that was not at all what I was expecting. Like it decides to do a web search or summarize or use a tool or whatever that I really didn't tell it to take those actions but it chose to take those actions and accomplish the tasks. And I think that so this feels different than RPA because we give the agents planning capabilities, you know, they start to decide on actions to take without us needing to code them up. And I don't want to say this works reliably 100% of the time. You know, frankly, it's like immature technology, not totally predictable, but boy, there are some moments that go, 'Wow, that was amazing.'

Yes. Yeah, there will be. In fact, with agentic workflows, a lot of enterprise applications, for example reading over very complicated documents and drawing inferences accurately, for specific business verticals we find that agentic workflows work way better. In fact, one example of a simple design pattern is reflection, where you have an agent generate output, then you have it please double check your own output, comment on the strengths and weaknesses, where might you be right or wrong, then take its own feedback to just rewrite it. It's such a simple design pattern to implement. I'll be really surprised at how this simple design pattern, where you now prompt an LLM three times, generate, reflect, critique, and then use the critique to generate a new result, this generates significantly better results than zero shot prompting. And if you let it do this a few times, sometimes we've seen for some really tricky document understanding this gives much more accurate results. And I find that the portfolio of agentic design patterns is really letting us build capabilities that I would really struggle to build any other way.