Welcome to the second stage for your interactive session. That was an amazing talk as we always expect from you. This is my first session that I'm hosting today, so I'll introduce myself to the audience as well as to you, Andrew. I'm John Krohn, I'm Chief Data Scientist and co-founder of an AI startup called Nebula, but I'm perhaps best known as the host of Super Data Science, which is the world's most listened-to data science podcast. And I'm delighted that for three years in a row now I've been hosting sessions here at ScaleUp AI, so thanks Inside Partners for inviting me back again. Andrew, in your talk you discussed how your team found that with GPT-3.5 and an agentic workflow it can outperform a more advanced foundational model such as GPT-4 with a zero-shot approach. How should companies balance their investments between pursuing more powerful models versus leveraging more effective agent architectures?
I think almost all companies, with the exception of a few giants, should be focusing on building applications using agentic workflows. If you have an extra few billion dollars to spare, by all means go compete with OpenAI, Anthropic, and Gemini or whatever. I think that in all seriousness, you want to do it, go for it. You know, just plan to spend a few billion dollars maybe. But I think that for most businesses, there are so many opportunities to build applications. And it turns out that if you look at the use of generative AI, the cost of using these models is falling rapidly. So over the last year and a half, it's fallen by maybe about 80% year on year. So I find that, you know, kind of two years ago there were teams worried about, oh, GPT-4 is kind of expensive, but the prices are falling so quickly that I would advise to worry much more about building something valuable. And then I think there's a good chance that the use of these APIs or to use GenAI would just become cheaper over time. And there are companies spending many millions of dollars on GenAI APIs, so it can get expensive, but the vast majority of businesses I see have generative AI bills. If you build an application on top of your OpenAI, Anthropic, or these other APIs, I see so many businesses getting so much value out of it, and frankly the bill that's sending, you know, is so small that you'd be surprised at how small the number is. You won't be surprised, John, I think many people would be surprised.
Well, it makes perfect sense. And yeah, so unless people do have billions of dollars to spend, let me move on to a related question. Assuming that people aren't going to be trying to train their own LLMs themselves, if you're an enterprise, should you be thinking more about always trying to use the latest and greatest LLM, or be thinking about grabbing the best agentic workflows? It seems like there's kind of a trade-off there between cost and efficiency, because yes, while costs have gone down dramatically, say by 80%, you could save a lot of money by working with GPT-4o mini instead of GPT-4o. And so if I can be using that cheaper GPT-4o mini and getting better results by leveraging a more effective agentic workflow, it seems like, do you think that's the way to go?
For the most part, you know, I would say don't worry about... I feel like as a general suggestion, I would say don't worry about the price of LLMs to get started. And I think for development purposes, it's actually, you know, it's not impossible, but honestly, so I still, you know, do a fair amount of coding myself, right? And sometimes I would be spending all day on the weekend coding, you know, just experimenting, and then I find at the end of the day, I just ran up like a $5 OpenAI bill. Yeah, right. And now it is possible, there are some workloads that can get more expensive. It is possible to run up, you know, tens of dollars, maybe low hundreds of dollars, but it's actually cheaper than you would think. And so what I advise most teams is, the hardest thing is just building something that works, that's still pretty hard. So use the best model, build something that works, and after you build something, if we're so lucky to build something so valuable that its usage is too expensive, that's a wonderful problem to have. A lot fewer people have that problem than I wish, but when we have that problem, we then often have tools to solve the cost. But I think that a lot more people are worried about the price of using these generative AI APIs than is necessarily the case. And the most important thing is, so I would say use the best model, use the latest best model, just build something that is valuable, and only after you succeed at that, and only if it turns out to be expensive, then work on the cost optimization.
Okay, nice. And then so if you are lucky enough to get to that stage, maybe there's a balance of experimenting both with lower-cost options like moving to GPT-4o mini, say, or experimenting with different agentic workflows and just trying to see which gets you the best results for your use case.
Yeah, yes. And just to be clear, there are teams that have, you know, found that they were spending too much money on these and then spend time optimizing it. So you can use cheaper models, you can take a smaller model and do something called supervised fine-tuning to optimize it for your own workload. So there are multiple tools, but I think using these other tools to optimize cost before you've first built something valuable, I think that will most likely be premature optimization, and I would shy away from that.
Nice, that's a great answer. Digging into agentic AI a bit more, which was a big theme in your talk and something that you said is maybe the technology we should be most excited about at this time. Going back to a Wired article in 2013, you mentioned how in the early days of AI the prevailing opinion was that human intelligence derived from thousands of simple agents working in concert. This is what MIT's Marvin Minsky called the Society of Mind. But then you mentioned later in this Wired article that you stumbled upon the single algorithm theory popularized by Jeff Hawkins, which led you to deep learning. Now, 11 years after that Wired article, are agents and multi-agent systems in particular marrying both of these concepts together, where, you know, we kind of have both ideas now blending together and providing powerful tooling?
Yeah, that was an interesting question. Boy, I totally forgotten about that. Well done taking it up. So I think that what's been remarkable about the large language model revolution is how much of it is because of one or a relatively small number of algorithms, you know, mainly the Transformer network. And it turns out that the reason large language models, which are based on the neural network called the Transformer network, the reason they can demonstrate such amazing capabilities is, I want to say a lot of it is because of the richness of the data we feed in. So there's this hypothesis, not proven, but this hypothesis that even human intelligence, you know, the human brain, a lot of human intelligence is due to one or very, very small number of algorithms that when fed all the richness of data from the world allows us to learn to do all of these amazing things that humans can do. And then as children grow up into adults, they also really, to a large part, I think because of the data they were fed, maybe a little bit of genetics, maybe a little bit of other, but really a lot of data. You know, the same infant brain could grow up to become a doctor or an architect or software engineer or whatever, and that's the data and the interactions. And I think agentic workflows, always dangerous to make analogies between AI and humans, but I think there is a little bit of that getting the AI models to specialize a little bit for different tasks based on how we prompt it or how we feed it additional data to do specific jobs.
Nice. Yeah, and I appreciate that you loved us digging that out. Wish I could take credit, I have an amazing researcher, Serge, who I have to give credit to for pulling out that question and that idea. I want to move on to large vision models now as a topic, which follows from that five key AI trends slide that you kept coming back to in your talk. So this is particularly related to the image processing revolution that's coming that you mentioned on that five key AI trends slide. So Landing AI has a product called Vision Agent that led the way in this image processing revolution trend. Can you elaborate on why planning using multiple tools and code generation are so crucial for building effective vision AI applications and how Vision Agent addresses these challenges?
