Thank you. She already gave away most of my presentation, so that's—as an AI, she could be AI. It's really good summary of the kind of things that we're going to talk about.
So I'd like to talk to you about trust. Now trust is important because it drives the adoption of our technologies. Now people will not use tools that they doubt. Yeah? Now question is, do you trust me? Yeah? You should not. Yeah? You've never met me. Yeah? You cannot verify a single thing that I'm saying here to you. Yeah? And you're granting me your trust based on a name, based on my association with a company, based on the fact that I'm standing here on stage. Yet that is not a good way to trust people. Yeah? You should be able to verify what I'm saying, and this is how most of the world, by the way, this field works.
Yeah? I want to go back a little bit in time. You know, trust is actually one of humans' oldest technologies. Yeah? That's why it matters. Trust is older than money. Yeah? Trust is what allows strangers to cooperate, to trade with each other. You can trade with people that you've never met before. Yeah? By the way, you can board an airplane, and it's flown by someone that you've never met, yet you trust it. Yeah? Without trust, nothing works—not commerce, not government, not science, not this room. Yeah? In the next half hour, I will try to show you some solutions how to handle the trust issue in our systems.
But first, we need to look at how bad things have become. Yeah? At scale, trust is failing. Yeah? More than half the people today cannot understand whether the information that they see is genuine. The majority cannot separate real from a fake. And this means we can't fix this by just debunking fakes one at a time or by telling people to look harder. Fakes are free and infinite. So our eyes have already lost. Yeah? We need to fix the systems instead.
Now, let me give you a good example. This is a story about Hong Kong, January 2024. A finance employee at Arup, which is the architecture firm behind the Sydney Opera House, receives an email from his CFO asking for a confidential money transfer. It feels off. Yeah? So, he does exactly what we train people to do: he gets suspicious, asks for a video call. The CFO joins, some of his colleagues join, they talk it through, and reassured, he transfers $25 million. Yeah? Every single face on that call was fake. Yeah? The video call was not the safeguard we would expect; it was a highly sophisticated weapon. And before you think that you would have caught this, research shows that only one person in a thousand can reliably spot a deepfake.
Now, it is crucial, because if people cannot trust what AI says and does, enterprises will lose profit. But you in this room lose much bigger things: impact, trust in your community. The communities you serve will not use your systems because they won't trust you.
Another area is betrayal. You know, forgery is one way that trust breaks, but betrayal is another one. In Afghanistan, humanitarian organizations registered millions for aid using iris scans and fingerprints. And for a decade, that worked perfectly fine. But then Kabul fell and the devices were captured by the Taliban. And the registry that was originally built to feed the vulnerable became a tool to identify them. Now, if your password gets leaked, you rotate it. Your iris scan cannot be rotated. Yeah?
So, in general, COVID has delivered to us a declining trust in medical institutions. The trust actually here, especially in medical systems, is measured in lives. In the Democratic Republic of Congo, health teams are turned away by communities who believe Ebola is a fabrication and the vaccine is the real threat. And this distrust doesn't come from nowhere. I mean, during COVID, vaccine confidence fell in 52 of the 55 countries in research. Yeah? This distrust now arrives at every new outbreak. It stops healthcare workers from reaching the patients that need to be reached. And if trust dies, people die. This is incredibly important. It's not just a matter of whether your bestseller list is out of date. It is the impact that you have on vulnerable communities.
So, how do strangers decide who to trust? You trust through signals: a face you recognize, a stamp on a document, a row of five stars. Yeah, we've always trusted through signals, and over time the signals have changed. So, let me take you through a number of them.
Yeah, trust used to be local. For most of history, trust was that, and the signal was presence. Yeah, you trusted a trader because you saw him every day. But as trade grew, merchants in Amsterdam, my hometown, had a problem: how do you trust what this trader is saying, the weighing of the goods? Yeah, so the city built something called De Waag, which is a weighing engine. So in public, goods were being weighed. Yeah, every transaction was weighed on the city scales in public, in the open. Yeah, and everyone could watch and actually trust what the outcome would be. Remember, Amsterdam did not ask the merchants to be honest. Yeah, they built a mechanism, a standing public inspection running every day of the week for every trade.
Well, at some moment, trade outgrew the town square. Yeah, when your cargo is on a ship from Asia, you can no longer stand in the weighing house and watch. So, trust was scaled by delegating it to institutions. Amsterdam again: in 1602, the world's first stock exchange was created so that strangers could own pieces of voyages that they would never see. Yeah, and then banks and courts and universities, newspapers, all say the same thing: "Trust this because we say so." And it worked because authority was brutally expensive to counterfeit. Marble costed millions.
And then the internet arrived. Yeah, and strangers began transacting at planetary scale, far beyond what any institution could vouch for. So, we found a third answer: we trusted the crowd. And think about how you do this today. Yeah? You're getting into a stranger's car because other strangers rated him five stars. Yeah? You sleep in a stranger's house. You eat food cooked by strangers and delivered by strangers. You run code written by volunteers that you will never see. All held together by reviews and ratings and reputation scores, and it works. A product with five stars sells three times as much as products without any stars. Trustpilot became a fundamental internet service. Yeah, the signal stayed honest for the same old reason: faking a thousand customers used to require a thousand actual customers. Or at least it used to.
Now, notice what all these three areas have in common. Yeah? Why does a bank build its lobby out of marble? Marble does nothing for banking, but marble is expensive. Yeah, it signals. It signals that a fly-by-night operation cannot afford this overnight. Marble says, "We have money. We're here to stay." Yeah? And the expense is the message. Every single signal works like this. Yeah, the expense is its message. Presence—showing up every day to market—takes time. A degree costs years of your life. Yeah, tens of thousands of reviews cost tens of thousands of actual customers. Expensive to produce and therefore honest. Every era of trust rested on that one assumption: that it was expensive to fake.
But that has changed, because two years ago the cost collapsed. Generative AI did not break the trust system itself; it broke the assumption underneath it. It broke the assumption on all three of them at once. As soon as we digitized our signals, they could be disrupted. Presence: the fake face on a call. Deepfakes will cost you a few dollars. Yeah, authority: credentials can be faked in seconds. Consensus: reviews can be manufactured by the millions. The expense that kept our signals honest for 500 years is simply gone.
Let me give you a realistic example of this. You see this number? 275 fake reviews—275 million reviews in one year were removed by Amazon from its systems. One company, Amazon alone. One trust signal. This is the new economy of fakery.
So the question is, can we build systems that can be trusted, yeah, when every trust signal now can be faked? Yeah, there is a trap in the obvious answer because we'll say, "Oh, we put humans in the loop." Yeah. Humans are after all sort of the gold standard of trust because it's expensive. But human capacity is limited. It becomes the ultimate bottleneck. Human eyes cannot scale to the velocity of silicon. A system that creates at machine speeds and verifies at human speed collapses under its own speed. So, to survive an era of cheap creation, we must change the currency of trust. We do not just need faster engines for writing and coding. We need automatic verification. We must use the machine to check the machine.
And here's the answer: the only trust that survives is trust you can verify. Yeah, for 500 years, we took signals on faith because they were expensive to fake. That era is over. The communities you serve will not accept, "Trust me." And they should not. So, your answer is, "Here is how you can check." We call this verifiable trust, the fourth era. And the rest of this talk will be about how can you build this?
Yeah, so I've built many large-scale systems over the years, over now—what is it? 22 years at Amazon. By now, we've built systems that can scale and be trusted. And they work not because every component is perfect, but because the architecture tolerates failure. And stop asking whether you can trust the model; you cannot. Yeah? And you do not need to. Build a system around the model that you can trust. And that system has three stages: verify what goes in, verify what comes out, and with our agentic systems, verify what it does.
So, first and foremost, you have to audit the data that comes in. Yeah, the first layer is our data; AI learns from that. And let me tell you why this matters. This is another example of my home country. In Rotterdam, the local city wanted to build a system to manage welfare fraud. But they were very careful not to put anything in the input dataset that said anything about immigration status, yeah, so there would be no discrimination based on that. Unfortunately, they left one thing in: how well does this person speak Dutch? And that turned out by level of indirection a discriminatory item. It turned out that immigrants were way more often investigated for welfare fraud than non-immigrants. And not because they had to, not because there was proof of that; it just was built in the model in itself. Nobody intended to discriminate. The data had the discrimination inside. The same trap sits in these automated hiring tools that you have today. If you train it on your current workforce, you get exactly that: you get the current workforce back in your training systems. Yeah? So, the first rule is audit what goes in before you trust what comes out.
By the way, we launched yesterday a new Model Context Protocol server, an MCP server. So, at Amazon we have well over 1,000 open datasets. And of course, you can go onto the website, look at what these datasets are, but the Model Context Protocol server allows you from any AI agent to type in and describe what is the kind of data that you're looking for, and then investigate that data, get samples back, look at that. So, in this way we can integrate all of our open data that we have accessible through the AI systems that you use on a daily basis. It's available today.
Now, remember LLMs are optimized for plausibility, not for truth. Yeah, and often I get asked, why do these systems hallucinate? Well, it's because that's what they're built for. Well, they don't want to hallucinate, of course, but remember it's a system built for plausibility. Yeah, if you're talking about a table and in the context there's chairs, it's probably a physical table. But if rows and columns are in the context, that table might be an Excel spreadsheet or a database. Yeah, a large language model is mathematically optimized for plausibility, not for truth. It produces the most likely sounding next word, not the most accurate one. So, plausibility is not truth. So, if the model can't tell you whether their own answer is right, the system around it has to check it. The question becomes how do you check at the speed that machines generate?
And the strongest form of checking is mathematics itself. Yeah, if a model claims that this triangle—and this is just a simple example—if a model claims that this triangle is a right-angle triangle, then it needs to meet Pythagoras: a² + b² = c². If it doesn't, it's not a right-angle triangle. And that's just a simple one. Yeah, you describe the systems that you have in a formal way and we can use provers to take care of that. We internally at Amazon use this all the time. For example, in our security services, we need to prove to ourselves that we are protecting you. As such, we formally describe exactly what these systems do and then make use of automatic reasoning to prove to ourselves that we're doing the right thing. When Amazon's S3 storage service went from eventual consistency to strong consistency, we needed to prove to ourselves that every edge case was handled correctly. You do that by using automatic verification.
Well, how can you do that? Imagine you're subject to regulatory requirements. You can encode those requirements or clinical guidelines, and you can express those in machine-checkable logic. And then you feed them into model checkers that test the claims against the encoded rules, but also the output that these systems have, and for example, return proof of compliance or flag violations. Yeah, remember, if you're subject to any regulatory requirements and the AI makes a mistake, it's you that's on the hook, not the AI. Yeah, as such, you're responsible for the verification of this.
Well, that was all fine—these systems generate text output, you as a human can look at it. But agentic systems are something different. They take actions on your behalf, and actions have consequences. Yeah? They book the flight for you. They move your money. They deploy the code. They email your donor. A wrong answer sits on the screen and you have your judgment on it. The risk is no longer just saying the wrong thing; it's doing the wrong thing. And the consequences grow with every degree of autonomy that you hand over to these agentic systems.
Now, we've built systems like this forever. Let me go back—let me try and guess—about 35 years ago, kind of signals how old I am. 35 years ago, I worked on factory floor automation for airplanes, for trains, things like that. Your Airbus A320 is not flown by one computer; it's flown by five computers. Huh? Each of those computers is from a different manufacturer, uses a different chipset, uses different software generated by different teams. Airbus calls this dissimilar redundancy. If these five systems disagree, the disagreeing one is taken out of service, and the other systems will take over. More modern systems make use of quorum voting, meaning that these five computers all produce an output; if they agree, then they will take the action that the quorum actually gives them. Again, important is that each of those systems are different—built by different companies, built by different groups, using hardware that is dissimilar to each other.
And so, if you build agents, and these agents have high risk, don't trust the agent; trust the quorum. Right? So now, almost 40 years later, we can apply the same ideas to AI: don't run one agent. Yeah? And before any high-stake actions, they have to reach a quorum. If they agree, they run. If they don't, stop it and a human looks. Yeah? And probably a human can take the three different outputs and actually feed it back into it and explain to me why it's different. Yeah? 30 years of distributed systems have told me, though, that these models need to be different. Yeah? Each of those agents will need to use a different LLM to actually produce the output, and then you compare and see whether those three actually agree with each other—or five. Yeah? They all need to be odds because otherwise you can't reach a quorum.
If you look at GitHub today, 40% of code that is committed to GitHub today is written by AI. However, 96% of engineers don't trust the code—and by the way, rightfully so. Yeah? And so only 48% actually verifies it. And why? The honest answer is time. When a human writes code, the understanding of the code comes with typing in the code. Yeah? But however, when a machine writes it, the understanding must be rebuilt afterwards. And almost nobody budgets that in their time management. This is what's called verification debt, and it's the backlog of unchecked work.
So, what do we do about this? Slow everything down to human review speed? Then why automate at all? Do we skip verification? Then the debt becomes a catastrophe. The answer is actually neither. Trust is the other side of risk. The more risk an output carries, the more trust it has to earn before you act on it. So, treat verification like every scarce engineering resource: budget it, spend it where risk really lives.
And think about that now: not all work requires the same scrutiny. So, put every task on the scale. A meeting summary that is wrong will cost you minutes: automate a check, move on. Marketing copy costs money: build workflows that provide human approval and reviews. A benefits decision can cost someone their rights: you must not only build systems that verify decisions, but ensure there's human accountability and look into mathematical verification. A medical diagnosis can cost a life: the proof can be proven, and an expert or human owns the decision.
So, because agents make decisions and take actions, there's risk that is fundamentally higher. You cannot manage that risk unless you want to ask four questions about the agents in your system: Who is it? Identity, permissions, boundaries. What is it allowed to do? Not what it can do, but what is it allowed to do—what do you decide that it may do? Is it working correctly? And can we prove that?
That's why at Amazon we're building a whole set of tools that actually help you build the right infrastructure for trust, but I won't go into that.
What is the case is that good intentions don't work. You need to create mechanisms to actually ensure that you can trust your systems. A mechanism is a ritual that you keep running after launch. These three mechanisms: Inspect—review real outputs on a schedule every week forever, not just when you launch it, but forever. And be the customer—use your own system the way that the most vulnerable user uses it: their phone, their network, their bandwidth, their language. And audit usage—watch using the system for what it does, not just what it produces.
So, make your system verifiable. You have to design systems that are auditable from the start. Most of you in this room will not build these systems, but will buy these systems. Force your vendor: demand verifiability from your vendors of their systems. And if your regulatory requirements—if your regulator requires proof, not just paperwork, require proof of what the system does.
Here in Switzerland, last year an LLM was launched that is an ethical LLM. It basically is based 60% on English, but 40% on files in other languages. It meets the Swiss regulatory requirements around copyright. It meets the AI Act. And more importantly, it is completely open: the dataset on which it has been trained is open and available for you to inspect. That really results in an auditable infrastructure.
Now, we go back to trust. You know, trust began local, and it was earned face-to-face because you're there in the marketplace or in the weighing house. Then it became institutionalized through authority at places like banks, and then distributed through the consensus of strangers' reviews. And now verifiable, earned through proof. Every era was humanity's answer to a new scale of strangers. Yeah? This one is ours to write.
If you have no trust, AI doesn't matter. Yeah? No trust, no AI. Trust is the constraint in our technologies; it is also what makes it possible.
Now, trust in technology in general is not enough. I'm reminded that every day through our CTO fellowship that I run. Take Jacaranda Health, whose CTO is a fellow. They reach 3 million expecting mothers or mothers that have already given birth through an AI-powered health system. A mother can send an SMS to this system in their own language. The system answers in their own language—not in English, not in the official language of the country, but in its local language. That creates trust. It is trained on real questions from real women, shaped among the community that she's in. So, when a message arrives, the system knows what to do and how to answer it. But trust is very important here. What the system also does is that if there's certain keywords in the message that may signal an emergency—bleeding, for example—it doesn't feed it into the automated LLM; it will immediately escalate this to a doctor to review. That's the reality: humans are still in the loop in this to make sure that they make the right decisions.
So, the CTO fellowship now has five different cohorts. The first two after two years are graduating this year. And I would like to actually introduce you to the healthcare and education cohorts of the CTO fellowship. Let's watch.
