Palo Alto studio connection, Silicon Valley and Wall Street. I'm John B here with Dave Volante, my co-host.
Welcome back to the Cube studio here at the New York Stock Exchange. I'm Gemma Allen, host of NYC Wired's Mixture of Experts, and we are going to have a conversation now about how AI is getting very good at just about everything. It can make decisions, interact with customers, move information, write code, and increasingly act autonomously inside a business. But here's the uncomfortable question: who is actually keeping track of what all these AI systems and agents are allowed to do? Navrina Singh has spent the last six years building Credo AI around that very problem. And now she's taking it to the next frontier, governing AI agents as they start to act on our behalf. So when AI goes from giving us answers to actually making decisions, who sets the rules? Navrina, welcome to NYC Wired.
Thank you so much for having me, Gemma.
So I mean, we are certainly at a very interesting time in the world of enterprise tech and enterprise AI, right? Not just from the perspective of tech economics, but from the perspective of risk, efficiency. There's a lot of narratives out there, you know, that are contested and non-contested. But I think what we can all agree on is that we are entering the next frontier from a risk perspective, right? We have seen that over the last couple of weeks with Hugging Face, with Mistral, maybe may or may not be true just last week. You're essentially saying that we need to have very strong, very intelligible governance in place, both technically from the perspective of a platform and also from a policy perspective, I guess. Help me unpack that. Like, help me understand what exactly Credo AI can do for an enterprise.
Absolutely. So Gemma, we've been on this quest for the past six years to really build the infrastructure of trust, because a thesis that we had was as AI systems become extremely capable, which we've seen, you know, from answering questions to now fully making autonomous decisions, what becomes central is how do you ensure that within your organization's goals and intentions, these systems are behaving as they should be. So when you think about governance, governance is not just about risk management and compliance management. They are really important components. But a big part of governance, especially in Fortune 500 companies, is do I know where AI exists? If I know where AI exists, can I put the right intentions and guardrails and policies around it so that it behaves in accordance to my objectives? And when it doesn't, can I intervene? So governance really is how do I define what good looks like for my business and operationalize that. So where this confluence of policy and technical capability comes in is, as you can imagine, and we've seen this especially in the last week, do you need a federal framework or do you need independent evaluations, or should you be doing your own testing and evaluations yourself? So there's a spectrum of trust that is getting created from the responsibility that you as a foundation model provider have, to you as an independent evaluator have, to you as an enterprise has as you are building capabilities around these frontier models. And so within all those different parameters of trust, you have to define those policies and operationalize it. That is what Credo AI does as governance as code. We take your intention, we codify that into controls, and we make sure wherever your AI is being built, bought, and deployed, you're guiding those systems appropriately.
I mean, if we are to take the philosophy of Jensen, right, he thinks that companies, frontier labs, and also enterprises should govern themselves, right? First and foremost, that was kind of his key takeaway from that regulation debate last week. But when we think about governing yourself as an enterprise, not just in the world of AI, but over the last 15 to 20 years, right? A lot of it is actually lost at the discovery phase alone. There are a lot of enterprises that don't fully understand the scope of their environments from a tech perspective, right? Because you see so much scope creep. Even we saw that in the move to cloud. That alone was a challenge, right? Folks need to understand what exactly they had and what they could migrate. Yeah, when we think about AI, talk to me a little bit about what sort of discovery is happening and what sort of rogue activity is happening. You know, we have folks come on, they say we have builders and they're, you know, like using these AI-generated tools to like create and they want API keys and they're creating some phenomenal content and I'm like, wow, there's also a phenomenal level of risk there. Like, so help me understand what's actually happening inside enterprise. Are companies trying to figure out, okay, what exactly is in scope here and who's building what and where?
Yeah. So let's unpack that. So one, you know, on Jensen's comment, I think there's a spectrum of trust that everyone's sort of missing and it's really important to nail that down. I don't think the answer is just testing and evaluation that an organization does or having a regulatory framework. There's an entire set of things that you need to be doing to make sure that these highly capable systems actually reap the benefits and the ROI that we all are expecting, and we can unpack that a little bit more in a bit. But what's really happening within an enterprise is a great question because enterprises are not just taking in these frontier capabilities. They are asking themselves within the context of my use. If I'm a healthcare organization and I'm using agents to really help with patient and patient decisioning, that's a very different set of guardrails than if I'm going to be using these frontier models or even open source models to build a marketing application or to build a customer service application. So the first thing that's happening when the frontier hits enterprise is context becomes paramount because within the context you can actually define what that good looks like. So let's take this healthcare example. You've built a patient-serving agent and this patient-serving agent is going to give you some guidelines on how your, you know, what kind of healthcare benefits that you have. The question a business is asking is what is this agent allowed to do? Can it just guide the patient around all its healthcare benefits? Can it actually prescribe those benefits? Should it be accessing patients' proprietary information or should it not be? So all those decisions around access, identity, what it is allowed to do, when it does do something which is outside the bound, should you escalate, should you block, should you allow it to happen? All those configurations basically come in from a policy. And so Credo AI is trying to translate those policies into actual configurations that these agents can follow and we can take appropriate actions and intervene when the agents are performing anything outside of that.
So if I'm correct, this is essentially a command center, right? It gives you full visibility over what's happening across your entire stack, your environment, let's say in the world of enterprise tech. It also gives you a certain amount of control. Does it give you, you know, the opportunity to, I don't want to call it a kill switch, but you know, can it help you prevent potential challenges or, you know, threat vectors? Like it's not a cybersecurity blade per se, right? It's one step back from that. Help me understand how exactly the technology works, like where does it draw the line, I guess.
Yeah, great question, Gemma, and this is where the market landscape is not only very vibrant but also getting confusing. So governance, or as you describe, command center, is basically that layer of trust that sits on top of your data AI security stack. So we have to be able to orchestrate across everything like Azure AI Foundry all the way to maybe security that you might be getting from Cisco Splunk, and we aggregate all that information to really understand one, where is AI, whether it's shadow or sanctioned AI being used in your organization. So this is where we'll plug into your security tooling to say we've just discovered some agents that Gemma built. Should these be sanctioned to use within the context of New York Stock Exchange? Right after that discovery, Credo, because of the signals we are able to get from your agent platforms, we can do first level of risk analysis, what kind of risk that this agent, you know, potentially can cause. Is it an identity risk? Is it potentially a fairness issue because the agent hasn't baked in demographic parity? So we do that initial risk analysis. And then the third thing, which is really important, is, and then we were just discussing this, when you think about multinational organizations, you are deploying agents all over the world, but when they enter the boundaries of those different countries, new set of regulations and new set of standards now apply to it. So Credo AI is now able to surface for this agent, for this system that you've built, here are the set of requirements you absolutely need to meet if you're launching the system in Europe, if you're launching the system in Australia. And then we do this continuously, and this is where we have to hook into monitoring platforms because once you've launched the agent, how do you make sure that all the policies that you've provided to that agent are actually happening in the operational realms? So we'll hook into your monitoring platforms. We're taking signals back from those monitoring platforms to really see did the agent at any point violate the conditions that we have configured it for, and then take appropriate action.
So I'm going to ask you a very, what might seem like a basic question, but help me understand this. Let's say healthcare. You're a nurse. You have a Muse. Okay, we heard a lot about Muse this week. You say to it, hey, go in and check the roster and see if there's any overtime opportunities for me. That system is interacting with, you know, your work system through like a certain level of credentials and authentication, right? What is the unique authentication by which you track agentic and agent activity? Is it like on a form factor device? Is it based on a domain? How are you actually really finding and catching these operators, these actors within environments?
Very difficult problem. So one of the things to again distinguish is we are not on the threat side. We are on the governance and oversight side. What that means is when you think about a three-layered cake, your organization level, your use cases or entity level, and then your runtime level. What does good look like? That's what governance defines. The actual threat analysis happens in your ops layer, which is the security and the monitoring layer. And that's why there's a better together that needs to happen. But one of the things that I do want to address in your question is what is becoming even more paramount is the need for standardization. So for example, there's a specification that's been created by Google called A2A that defines very specifically what your agent card or the metadata that needs to be associated with every agent needs to be. This has, okay, is this Gemma's agent? Yes. How do we associate it with Gemma? What is it allowed to access? Which databases it should not be allowed to access? So all the critical metadata that defines what is that role of that agent. So think about, you know, that agent sort of like an employee that you're bringing in. You want to make sure that you are not only onboarding that employee with all the policies that are important for that employee to be successful, but to be able to effectively onboard and in some cases offboard them if the agent or the employee doesn't perform right. And the only reason you'll be able to offboard or fire that employee is if they've done something wrong. So violation of those policies is something that Credo AI monitors.
Wow. I mean, it is certainly an interesting world if you think about just identification alone. But okay. So moving on then, Credo. So you've been in this business before we all were saying ChatGPT, generative AI. You've been kind of like in it from the very, very beginning, right? But a lot has changed in a relatively short space of time. Talk about the business model here. Like, is this usage-based? Is it environment scope? How do you actually go out to an enterprise customer or a prospect and say, okay, here is, you know, the scope of what we can do for you? Because it seems though it's pretty monolithic from some respects. And then talk a little about what you're seeing from the perspective of TAM. I mean, it seems though right now everyone's claiming that they are going to meet the agentic future tomorrow, right? How real is that? Like, how much is it driving your TAM, Navrina?
