Navrina, thanks so much for making the time. I was laughing out loud in my hotel watching your conversation with Ina on our stage from a year ago because you were talking about how all of these businesses are rushing to adopt AI, but they're just not thinking about safety. They're not thinking about trust, and that feels hilarious in the context of this particular moment in the conversation around AI. How big of a sea change has it been in terms of the focus on AI safety and governance?
So, Megan, thank you so much for having me. So, you know, it's interesting, just literally a year back I was talking to Ina that everyone needs to be thinking about trust and governance. So, they've started to think about it. Are they doing something about it is a very different question, and to the level of effort, whether it's investment or your entire org change, is that actually going towards governance and trust building? Not yet. And I think we are at this consequential stage in the AI ecosystem where the enterprises are really deeply thinking about what would the ROI from this consequential technology look like, but we are in very early innings of that.
That's fascinating because obviously, let's do the first Hugging Face mention of our conversations here today. After that, I imagine there's so much focus on safety, but it sounds like from your point of view, it's not always sticky. It's something people think about and then maybe they put in their back pockets. To what extent do you think that these breaches, incidents happening at these AI labs, are becoming frequent enough that customers are going to really dedicate the capital and resources to prioritize safety?
So, Megan, maybe a little bit more context around Credo AI's work should give you a little bit of a sense of where we sit in the ecosystem. So, Credo AI is a leading provider of AI governance software. What that means is we help enterprises who are buying a lot of frontier capabilities, who are building around open source, and are building their own applications, they want to have control and trust within their applications. So, given our role in the ecosystem, what's happening right now is obviously the capability on the frontier is increasing, but can that be safely deployed in enterprise context? That is a massive gap. And I truly believe that applied AI is such an under-hyped space right now, because you can increase all the capabilities, but if you cannot manage the cost, if you cannot manage control, and if you cannot apply that to consequential workflows within your enterprise, whether you are in insurance, healthcare, wherever you are, the frontier doesn't matter as much. So, I think the thing that we try to work with, whether it's with the labs or whether it's with the enterprises, is how do we bring these great capabilities but with the right infrastructure of trust and control to actually impact those consequential workflows?
So, what are some of those aspects of preparing your infrastructure, preparing to have control, that are, you know, good case studies for people in the room who want to deploy AI safely within their enterprises?
Yeah, you know, I think there's sort of an equation we are coming up to, which is everyone's doing tradeoffs between capability, cost, and control. So, one of the largest telcos I was speaking with last week, they've actually switched away from frontier model providers, and they're going completely towards open models. And the reason for that was cost is a big factor as they are thinking about the capabilities. The capabilities of open source have increased pretty much or they are at par with the frontier labs. So, the question that this telco is asking is, 'Okay, if the capability differential isn't as much, can I actually optimize on cost, and then I will put the infrastructure of trust and control around it so that my infrastructure is still resilient, it is still governed, it is still controlled.' So, I think that's a very interesting aspect of our work where, you know, creating that infrastructure of trust, which is, you know, a lot about oversight, it is a lot about accountability. And then we basically work with your data, AI, security stack to go and gather the evidence, because at the end of the day you have to prove that you can actually control an agent. You have to prove that you can actually scale your AI applications.
But this is so fascinating to me as someone who covers the frontier AI companies, because it feels like a massive potential hit to their revenue. If more and more customers are saying, 'We don't need the frontiers of the frontier AI capabilities. We care about control and cost.' To what extent are you hearing more broadly from customers that they don't need something like a GPT-5? They just want, you know, other things out of their AI tools. How broad is that?
Again, you know, we live in nuance, and I think nuance doesn't get well captured on social media or in media. So, a little bit more detail here is context matters. At the end of the day, when you are trying to build an application that is going to serve, you know, patients versus if you're building a fraud detection system versus if you're building a marketing system, within all those three systems, you might have very different parameters of what the capability of the model should be versus what is the infrastructure I need to put in place so that this workflow actually works the way I want. And so, as a result of which, I think right now we are still in very early innings in something that I, you know, love to provoke my team with is, 'What if this frontier is not truly the frontier? What other new capabilities are on the horizon?' You know, a very interesting research report from Andreessen Horowitz last week stated that most of the AI used right now is by AI and not by humans. Humans are the minority users of AI based on the OpenRouter data.
Agents using other agents.
Agents using other agents. So, imagine that world when humans are actually the minority users of this powerful technology. How do you actually think about a lot of the questions that we are discussing right now?
Absolutely. It makes me think about the various examples of Meta's AI Studio tools on X that have gone viral of, you know, certain CEOs' banking information getting leaked by his AI agent, for example. I would hate to have that happen to me. I'm sure you all relate to that. What do you tell clients about using these autonomous agents and the risks associated with it?
So, I think we at Credo AI are fundamental believers that this technology is what we need to really get to this next frontier of humanity. And to get to that next frontier of humanity, you have to really embed trust. So, a few things that have shifted for us in the last three months. One is the unit of governance has deeply shifted. It's not just about governing an application or an agent, it is actually governing authority. Like, what are you allowing your agents to do? What are you allowing them to access from a data perspective? What actions are they allowed to do? And I think that is a pretty massive shift, right? The second is I think the definition of what do you actually measure. Well, the definition of good hasn't emerged because a lot of people are waiting for some form of regulation or some folks are waiting on some form of embedded evaluators. I think there's a spectrum of trust. And based on whether you're a foundation model provider or whether you're a deployer, you need to start thinking about all those mechanisms in place to define what you need to measure to really believe the system and that the system is going to perform as you expect. And that becomes even more complex when you have agents using agents and then agent swarms actually collaborating and coordinating, especially attacks. So, what really needs to be true in terms of infrastructure there? So, again, we are in very early innings of this technological wave and I don't think we are at the frontier yet.
