[Music]
Hello again for a panel discussion around gen AI: whose future is it? Debating future directions for AI in the enterprise. Again, I'm Ren, and I would love to start with everyone introducing yourselves. Do you have a go-to karaoke song?
Don't Stop Me Now, that's it. That's my... okay.
And you have to sing yours too. Oh man, you don't want to have to listen to me. So, who are you?
I'm Michael Balant. I work at Nvidia. I'm a director. I focus on product architecture, so building new products at Nvidia.
I'm Ren. You just heard me ramble on for 10 minutes. I love a ballad. Love Coldplay, so my reference was Coldplay, 'The Scientist'. And if you get me drunk enough, I will sing it. Okay, this is a challenge.
I would say probably anything Madonna. Yeah, maybe some Spice Girls.
Yeah, so I have some questions, but we will be opening up, so if you have questions, please shout them down and we'll have time to ask those as well. I'm going to start with you, Miriam. When thinking about the marketplace of LLMs, it seems so easy to just simply plug in an API. But what are the considerations that organizations need to deal with in choosing to rather self-host? Why are people doing that?
I think APIs are really, really fantastic, and if you can get away with your use case using them, I think it's on the whole a really good thing. Broadly, there's three reasons why our clients in particular would choose to self-host. The first one is security and privacy. If you want to deploy something in your own VPC or in your on-prem data center, like a lot of our European clients are on-prem still, then you don't really have another option but to self-host. If you really need that total privacy and control, then you have to self-host. The second reason, potentially counterintuitively, is performance. The best of the best model in general terms is something like a Claude or something like a GPT-4, but when you're looking at domain-specific models, you can get better performance by self-hosting smaller and specialized models. The third reason why people choose to self-host is scalability and cost at scale. When you're starting at a very small scale, using API-based models is much cheaper because you pay by the token, but when you start deploying it at mass enterprise scale, that per-token cost starts really adding up, and deploying a much smaller model in your own infrastructure can be much cheaper at scale. So that's privacy and control, performance, and also scalability and cost factor, the three main reasons.
And from Nvidia, Michael, do you have anything to add there?
No, I think that's the way we think about it too. We use different words. The second category I'd call it more like accuracy. We can go down a rabbit hole talking about performance and what we mean, but I think the ability to customize, to adapt models to a particular domain, that's a really important factor. That's why internally at Nvidia, a lot of what we run internally is heavily adapted models. And yes, price per performance is a big factor.
Yeah, great. And how many of you would say you're in a highly regulated industry in the room? A lot of people, yeah, majority. And we're also in California, so there we go. Next question to you, Florian. Are we moving towards an ecosystem of models dominated by a few very large foundational models from large companies, or are we flourishing into a diverse ecosystem around open, closed, specialized, generalized, everything, everywhere?
I think it's a very deep question. When I look at the patterns, as Miriam pointed out, there is so much benefit today in the ability to build your own specialized smaller model, at least when you build version two of your app or you try to get more accuracy, that there will be good benefit of having open-weight models and building on top of that, even long term, not just short term. Second, I actually think that for many of our use cases that are of interest to us, which are a lot about text analytics, finding rightful information in documents, and so forth, down the line in two or three years, models will be better and potentially there will be some form of a plateau of performance. So you will have a variety of models you can use to do that. I actually think that because of that, you'll have a variety of models, specialized or not, that you will be using in the enterprise, not just one or two providers calling the shots, at least for text-based analytics. For building VR or fantasy worlds, I don't know.
Hence all of us being gung-ho about the mesh, right? Michael, if you had to simplify, what are the workflows or tooling you think are essential for differentiated AI?
Well, certainly for gen AI or for LLMs, there's a whole new ecosystem we see around LLMOps. There are a lot of new tools, but they largely focus on what I'd classify as the model ops portion of MLOps. How do you pick a particular foundation model? How do you customize it in some way or adapt it and build a flow around it? It could be RAG, it could be some agentic type of flow. And then how do you wrap that up into an application that you deploy and scale? That's the general flow I see across multiple different domains. We work with a lot of different verticals and a lot of different use cases, internally at Nvidia and also externally. So it's pretty common flow with different pieces, different solutions. What's really important is that there's some tool you're using to evaluate how you're progressing. I heard this recently and I stole it from someone else, but I think it's brilliant: you don't want to develop your model by vibes. You don't want to just shoot in the dark. There's a tendency to do that as you're developing AI. It's much better if you have some sort of rigorous approach, and the way you can do that is by evaluating your results every time. If you have a tool that you're using to do that, that's great. We provide something at Nvidia called the NeMo model eval tool, so you could think of using that, especially if you're customizing. And also, especially when you go to production, when you deploy your model or deploy your app, you should consider how you're deploying it and how you're going to support it in production. So I think those two components are especially important.
Yeah. I'm going to ping that back to Florian as well. When we think of this kind of treadmill AI, the AI treadmill we talked about earlier, where it's become a commodity people can go buy off the shelf, but to right-size that differentiated AI, we're essentially becoming curators. Anything to add in terms of the essentials as we think of tackling differentiated AI from your perspective?
Yeah, I think that if you fast forward a few years, it's reasonable to imagine that many among you, including us at Dataiku, will have to build and maintain maybe 50 or 100 types of gen AI apps for information extraction or various specialized chatbots in the enterprise. When you start having this type of portfolio, it's a lot about the complaint factor. By complaint factor, I mean the number of people in your organization that can complain about it and how strong the complaint can be. If you've got 50 chatbots in your organization, the complaint factor is probably high. Many people will complain at some point; they'll say it's not the right answer, and maybe it's their fault, your fault, you don't know. So I think that long term, the job of us doing analytics, data science, and what else will still involve some very basics that are actually true today: you need to make sure that you can get the data in properly with data quality to build your embeddings and have strong workflows there. Because whatever the quality of the models, it will always be your job. You need to have proper evaluation frameworks to have an understanding of the quality, the issues, and your ability to troubleshoot. This is true today, as you pointed out, to build a strong and robust development environment. But even fast forward two years, that will still be the job to deliver quality in the enterprise.
Thank you. Miriam, you wrote on a recent LinkedIn post, and I think it was to your engineers, you said, 'Why do we need more A100s? Everyone wants more A100s and it is an incredible technology, but there is something to be said about efficiency.' Could you expand on what you were talking about?
Well, I think what I was referring to was that my engineers kept asking me for more GPUs, and you guys sold them all to OpenAI, so I can't get anymore. I think what I posted was something like, 'Why do we need so many A100s? Your CV said you were good at maths.' But I think what I think about when it comes to GPUs is that we focus a lot on the headline numbers for how expensive these GPUs are per hour. And I think the thing we need to focus on when we're thinking about LLM deployment, because it does require these really expensive and large and powerful GPUs, is that your North Star should be GPU utilization. It actually doesn't really matter how expensive the per-hour price is if you're utilizing it properly. GPUs are so efficient that your unit cost for an actual inference is incredibly low. So that's what I like. Well, not all of the work, we have an entire team dedicated to improving GPU utilization. How can we make our models faster? How can we make models smaller so we can pack more models onto a single GPU? Because if you utilize this resource efficiently, then actually it's a reasonably cheap way of doing compute. So utilization is absolutely everything. You don't necessarily need thousands of the most expensive GPUs; you just need to use them really effectively.
Sorry that tanked your share price.
Actually, backing up, we should talk. I can get you some A100s if you want. You might be surprised, but that's actually the philosophy internally at Nvidia, and this is very much led by Jensen, is to be efficient. Much of what we do internally and what we try to deliver externally is how can you get as much as possible out of the hardware. Certainly our mission isn't to try to scam everyone or have inefficient workloads and use more GPUs. The intention is how can we get as much as possible. We have lots of discussions internally about clusters we have, like are we getting the right amount of use in training and inference? And everything we learn we try to bake into a solution that we also share externally. So it's about democratization, sharing what we learn along the way. But I could definitely see why others may not see it that way.
