I would say the core thesis for us was not really what could go wrong with AI, but the core thesis was what would go right when we actually get this technology right. And I think that's what's really inspired our mission of making sure AI is in service of humanity.
Welcome to Made to Do, the podcast exploring the ways people can still be the most important part of the transformation of our economies and society. Every episode, I sit down with a global leader to uncover human-centered transformation insights for all of us who are navigating a positive, feasible future in the age of AI, climate, global fragmentation, and societal change. I'm your host, Floren Hoffman. Listen on to find out more. Let's get right to it.
Who and how do we make sure that AI is being controlled by people? As AI agents start taking real-world actions rather than just generating outputs, who's responsible when something goes wrong or a company's being hacked by agents? Legal and governance analysts are increasingly framing this as a live debate over whether AI agents should be treated as legal actors bearing duties or something closer to legal persons. My next guest has spent her career trying to make sure AI doesn't run ahead of accountability. Navrina Singh is the founder and CEO of Credo AI, which she built after nearly two decades leading AI and enterprise products at Microsoft and Qualcomm. She advises the White House, the OECD and the UN on AI policy. And last year, Time named her one of the 100 most influential people in AI. What distinguishes Navrina is not only that she's an absolutely inspirational and kick-ass leader, she also lived on both sides of the table: inside big companies actually shipping AI, and inside the rooms where the rules for AI get written. Back when Navrina started her governance business six years ago, this category didn't even exist. Today, it's one of the most discussed topics. Here's my conversation with Navrina Singh. Enjoy.
My pleasure, not at all. It's amazing to have you with us. Thanks so much for taking the time and having blown away all the young entrepreneurs down there. That was fantastic.
As an incredibly inspiring entrepreneur, you built a category — AI governance. What is that?
Yeah. Well, thank you so much for having me, Florian. You know, startups are hard and creating a new category is even harder. So AI governance is essentially a set of processes and requirements that are required to make sure that the AI that you're either buying or building is compliant, it is safe, it's reliable, and more importantly, it's human-centered. So imagine like you're a Fortune 500 company and you're buying all the latest and greatest AI tooling that's available right now in the world, whether it is from foundation model providers or from the open-source ecosystem. How do you ensure that as you're bringing that within your enterprise to use it for different applications, that it actually meets the requirements that are important for your business? And similarly, if you are one of the enterprises or a company building customer-facing applications, you really want these systems to be trusted by the end users. You want these systems to be compliant to regulations. You want these systems to actually always work reliably in a secure manner. So that's what an AI governance product and platform enables.
So break that down for me as a non-techie. So this means I'm buying, I don't know, Copilot or my Anthropic business cloud solution, and then I'm buying your software on top to check on what Claude is doing or what Microsoft is doing? Or how does it actually work?
Yeah, great question. And so imagine that you've already bought Anthropic and you're using Claude to build a customer-facing system, and that customer-facing chatbot, for example, answers everything reliably about your business. So you'll be using Credo AI, which enables you to do use-case-level governance. What that means is at the point of use of the Claude system for this customer service, we will look at your customer service agent and say what is important for your business. For example, for your business it might be really important that this customer service chatbot is not toxic, that it serves all the different demographics in the right, fair way, that the language is very aligned with your brand voice. So Credo AI basically will take all that business intent, convert that into code — it's called governance as code — and then use that code at runtime to make sure that your conversational agent that is built on Claude actually ends up doing the things it's supposed to do.
And if the agent doesn't, is it being shut down? Is it being corrected by Credo AI's code? Or how is the intervention?
Yeah, great question. So the intervention happens throughout the AI life cycle. So you assume that you're in the design and development phase. At the design and development phase, obviously there are checks and balances. The minute you go out of the risk threshold or compliance threshold, our system will alert you: you're now risky or you're not compliant with a certain regulation or standards. Now you take that system and you put it in production. So in production, we will connect into your monitoring platforms, whichever platform you're using, and we'll be taking the signals back from that monitoring platform to check whether your system is still risk managed or compliance managed. And the minute it goes out of those boundaries, Credo AI actually creates an incident, and based on that incident, it will provide you a prescription on how you need to reduce or mitigate that risk.
And so when you started out, you've been doing this for a while, right? Seven years. That wasn't really a topic. Now it's on everybody's mind. Does that mean we don't need to worry about AI and where it's going?
Well, we are working hard to solve that problem. So I started the company almost six and a half years ago, created the AI governance category. I would say the core thesis for us was not really, you know, what could go wrong with AI, but the core thesis was what would go right when we actually get this technology right. And I think that's what's really inspired our mission of making sure AI is in service of humanity. The product that we've built, the capabilities that we've built, the forward-deployed advisory function that we've built to really uplevel enterprises around AI governance — I think we are well on the way to make sure that you can adopt third-party AI systems but also build your own AI systems with the highest level of safety and trust. And so we are very well, I would say, ahead on making that mission come to life, but still with the AI capabilities on the rise, there's so much more work for us to do each day.
And is it actually possible if you imagine that all the big LLMs are putting out a new model every two, three months that is shifting significantly what it can do? How does it work? Do you have to then build a new model every time with it, or are you able to still capture a new model as well? What's the risk potential then there?
Yeah. Great question. So this is where we actually work — we have a very unique place in the ecosystem. We not only work with on one side the AI builders, the foundation model providers, the enterprises, but on the other side we work with the policy makers, regulators, standard setting bodies, independent researchers. And the reason that role is really important is, to your point, we can't figure out all the risk within all the emerging AI systems. So we work with all the ecosystem providers, whether it is the MIT Risk Repository, whether it is OWASP, whether it is MITRE, to really understand what are the known risks — so things like fairness or security, known threats — right? What are the unknown unknown risks? These could be emerging threats like psychopensy, which we don't even know what could happen. Or what are the even more emerging threats, especially when you have multi-agent systems where you could result in data poisoning or agent poisoning? So we actually monitor risks from known inherent risks, proliferation risk — which is when you actually are using a foundation model or an open-source model and that risk proliferates into your application, what do you do about it — and then last is emergent threats. And many times for emergent threats we might not have mitigations at that time, but we can alert the enterprises to these new kinds of risk. So I would say it's every day the finish line of this marathon keeps extending, because as the capabilities increase, new threats are emerging. So we have to be a great ecosystem partner to figure out what mitigations or at least awareness we can bring to these risks.
And ultimately the question is so big that it goes beyond any one single company, right? And I know that you've been advising now multiple White House administrations and are in touch with lots of different governments. So one of the feedback that I hear quite a lot between private sector leaders but also people in office is that obviously government is working much slower than this incredible speed that AI is now pushing. And on the one hand everybody is sort of trying to achieve supremacy over the other models and also sort of chasing the returns. So how do you look at that dynamic and what do you think right now are the biggest risks in that, just society speaking?
Yeah, great question. So one of the things I think that's getting missed in this AI conversation is it's not just about the model. It's actually what are you trying to serve, and we call it the context or the use case. Right? So the kind of guardrails that you might have for a system that is helping you make marketing flyers versus a system that is helping you with claim processing, the guardrails are going to be drastically different. So our role in the ecosystem here — and this is one of the reasons we work very effectively and close with policy makers and standard setting bodies like NIST, like ISO — is there are guardrails that are coming from this amazing body of research, policy, foundation model providers that you want to see: can these actually be codified and used within the context that primarily enterprises care about? Because we serve Fortune 500s. So we are basically the operationalizing layer that might take a policy — so for example the EU AI Act — we'll take the EU AI Act, we'll codify that in Credo AI as governance as code, and check for the enterprises that are using Credo AI to get compliance to the EU AI Act, does our software actually meet their needs? Or imagine you are trying to build a very good risk-management posture aligned with ISO 42001. So we will take ISO 42001 and we'll work with ISO to really make sure that the codification and then the implementation actually aligns with not only their objectives but is actually getting operationalized within an enterprise.
And so as you can imagine, for us this role of a translator becomes really critical, where we are translating what the AI builders are building as the next level capabilities and then what actually can be put in place as a standard or potentially a regulatory regime that actually makes sense for these enterprises. And that role, as you can imagine, needs to be able to speak tech and it needs to be able to speak policy. And having said all that, I think one of the key things that I keep bringing back to everyone is model capabilities are going to increase, but it always depends upon what is that context and do you have the right guardrails for that context? Do you have the right benchmarks and evaluation created for it? And more importantly, once you've done the testing and evaluation, once you've looked at that use case, can you actually create the right governance artifacts, which can be a transparency report, it could be a risk report, it could be a compliance report, it could be a disclosure, so that you can show that within the context of what you know as a business, you've done everything to be risk managed and compliance managed.
And if you go a little bit for a moment beyond the compliance and risk management lens, and you're dealing with so many CEOs and board members, all of them are trying first now to create adoption with AI internally and then the next step is how do I get and create innovation for the customer. What distinguishes a leadership team that does the integration well, including the risk mitigation and the compliance piece, from a CEO leadership team that is doing AI transformation?
I think the biggest thing that I see falls in two categories. One is they are seeing AI governance as an enabler versus a hindrance. So if you talk to a board or the executive leadership of a company that just has this mindset that I don't want to build a foundation of risk and compliance because it's going to slow my AI innovation down, you should already count them out, because what we are finding is when you actually have the foundation of governance, you can adopt AI faster with eyes wide open — like, is this new system from this XYZ vendor, does it meet my needs, should I be procuring that? So we are seeing faster adoption of third-party AI, but we are also seeing faster development of your AI systems, because once you know potentially what could go wrong and if it doesn't meet your consumer's need, you can actually make changes much more effectively. And also, lastly, we are seeing that the companies that actually embrace AI governance, they are able — because they're able to bring in more AI — the productivity increase among their employees is so much more, I would say, at a higher level than a company that did not, because now you're enabling your employees to get access to this very powerful technology with eyes wide open, again with the governance and the oversight much faster.
So one thing, as I mentioned, is really using AI governance as an enabler rather than a hindrance. And then the second distinguishing factor is AI literacy, where the enterprises and the leadership that are actually investing in AI literacy — not just for their employees, not just for their board members, but also for their customers — actually are winning. Because what you are doing in that case is, I'm a big believer that if you can build with AI, you can actually govern AI better. And if you learn AI, you should know how to build AI, right? And I think we are seeing that the more you can provide AI literacy to all these stakeholders — your employees, your consumers, your board members — the better informed you are about all the new technological innovations that are coming your way.
And if you know from that perspective of being so steeped into the risk question most of your day, if you listen to a historian like Yuval Harari who talks about the end is coming and we're giving out our own agency to a machine, how do you look at these types of narratives around us basically taking a significant step into the wrong direction as humankind, you know?
So this is — I'm a big proponent of agency, but with the right oversight and governance, and that's one of the reasons I created Credo AI. And when you think about the spectrum of folks who are very worried about existential risk to the folks who are just worried about maybe some of the more known risks, we somehow fall right in the middle because we want to be solution providers. We know how powerful this AI technology is going to be, not just for enterprises, not just for SMBs, not just for startups, but for humankind. And we really want to get this technology right. So most of the time that we are spending is actually not in thinking about risk, but we are really thinking about how do we get this technology right, by managing risk, by making sure it's compliant, by making sure it's human-centered. So do I worry about a Skynet scenario right now? No. Because I think a lot is still in our control. However, this is why governance and investment in governance matters right now, because if we do not have that investment, I think we'll have a very different conversation and we reach a fork in the road.
What about the scenario of AGI, like if you have general intelligence? How would an oversight over something like that look like?
I think we are in very early innings of figuring out what governance for AGI might look like. But this is exactly the reason why we should be thinking about all the different scenarios. That's why we should be investing more in AI governance, why we should be investing more in AI safety and trust ecosystem as well as startups, because I think we just don't know so much, especially on — we're already on this exponential in this technology. And I think, you know, we at least, we do get access to the frontier more often than the others, and we know that we are much nearer to some of these scenarios around AGI on the frontier. So the key question for us is now we should be investing equally in oversight and governance so that we can control it.
And who would be ultimately the actor that would make you feel the safest? Is it like, should governments basically prescribe this? Are we giving this as self-governance to businesses? How do we, from a societal point of view, how would we best navigate this or manage this?
You know, this is such a great question. I've thought a lot about it and I've looked at some historical context. Like if you think about even the creation of the New York Stock Exchange — the New York Stock Exchange didn't even come to being maybe 200 years since the first time trading started. Similarly, the FAA didn't come to being like 30 years since the first flight took off. When the technology is advancing so fast, you have to sometimes let the innovation play out, and the people who are building that technological wave need to also guide it in the right direction. And I think we are right now in that place where a lot of what these LLMs are able to do and what the systems or agent tech systems are able to do — unless you've built those systems and used those systems, you really don't know what good governance guardrails you can put in place, let alone what policy and what regime you can create. And so that's why you're seeing a lot of private sector coming together to create the right assessment mechanisms. So a senior bureaucrat in some capital somewhere around the world just doesn't have the tools or understanding to actually be able to effectively govern these systems.
And that's why educating the other side becomes so critical, and we are big believers in AI literacy, because over time, I think the best way that we can make sure that these systems actually work for humanity is the private and public sector has to come together. You know, there was a recent very interesting call to action by Demis Hassabis from Google, really talking about a FINRA-style agency that can provide oversight especially to the frontier-class models, and I think we are going to see something like that come to fruition, especially for frontier class, where things or systems that are so powerful and so capable that potentially are important for us to watch out for, from US competitiveness or from national security risk. You will need this public-private sector coming together and doing evaluation and testing and making decisions around it. But the key question is how does that sort of foray into enterprise-class models which are being used in healthcare and insurance? Do you need an agency like that? I think those are exactly the conversations we are having right now in terms of regulatory regime as well as thinking through the right guardrails. Context matters so much.
