Great. Yeah, love to hear it. I'll give you guys a quick tour of how we made Trail of Bits AI native so far. Okay, so just to level set, I think this is a really common problem across the industry. Back in February there was a big survey of thousands of CEOs and the result was that AI was having no measurable impact on employment or productivity. At the time people were calling it a new Solow paradox, which is a reference to this guy Robert Solow, an economist in 1987, who said you can see the computer age everywhere except in the productivity statistics. And I think the same thing is happening with AI right now. Obviously I don't think AI doesn't work. I think most companies are doing it wrong. They see these flashy tools and they give people the flashy tools, but they don't actually change the way they work. So they'll hand out ChatGPT licenses and Claude licenses and then leadership waits for the magic to happen and it just doesn't. And that's the big gap between companies that are trying to just get some quick efficiency gains by being AI assisted versus companies that are doing more AI native redevelopment. One of them is a tool and the other is an operating system. So the rest of the talk is about what it took for Trail of Bits to build the operating system and transform the company and achieve some of these benefits. So to level set, a lot of people throw around these terms like AI native. Everyone's going AI native and it's like a marketing gimmick, but I want to put actual words to what this means. So to me, I think about AI assisted where everybody starts. You give people access to ChatGPT, they draft some emails, they summarize some documents, it's just a productivity tool and your organization doesn't change. Like your workflows are the exact same as they were before. You just have a little buddy that helps you with a couple of tasks. AI augmented is where you start redesigning workflows. That's where you're not just using AI as a tool, but you're putting agents in the loop. So like maybe AI does the first pass on a code review and then the human does the second. So the process itself starts to look different. And then AI native is structural shift. That's where you've redesigned the company's various workflows from the ground up assuming that AI is going to be there and that it's a poor participant. So that's not really a tool. That's more thinking about AI as a teammate. That's when you structure your knowledge management, your delivery model, and your expertise all designed to be consumed and amplified by agents. So for me, what this means is that I want our security expertise to compound as code.
Every engagement that we do, all the skills, the workflows, everything that we build makes the next engagement we work on faster and better. And I want every engineer at Trail of Bits to operate with an arsenal of specialized AI agents that are built from our 14 years of audit knowledge. So that's not we're using AI, that's AI is on the team. It's a core integral part of the work. So in order to get there, no surprise, problem is people. People resist technology adoption. They always do. It's the same thing when there's any new trend. You know, people resisted cars and microwaves and the web and cell phones and whatever. It's people's unwillingness to accept something that might be better. It's just sort of a common problem and this has been studied. Psychologists have broken down a lot of these obstacles and biases that we have against adopting new technology and I think there are four of them that really play a key factor in adopting AI. There's a whole universe of these biases but these four are the ones that really make it for me.
The first one is self-enhancing bias. This is the tendency to perceive oneself in an overly favorable manner. This is what you do to maintain your own self-esteem. This means you take credit for your success, but you blame external factors and failures. And this self-enhancing bias is really tightly related with the identity threat. Basically, the more senior that you get, the more you think that your intuition is what got you there. You don't believe that a model, that a machine, that an automated process could do the work that you do because that machine didn't go to university like you did, didn't spend 20 years working like you did. It didn't accumulate all this hard work and all this knowledge like you did. So you don't believe it. The identity threat is that especially when we frame AI in terms of efficiency gains, it looks like someone is being replaced instead of augmented by the technology. So there's a pretty good study out of I think it was Stanford around a cooking study where they gave groups of people this kitchen automation device. Think like a pressure cooker or something. And they advertised it to these groups of people in two different ways. On one hand, they advertised it as it does the cooking for you, and on the other hand, they advertised it as it helps you cook better. It's the same device, but in this group of people that were receiving the device, the people that identified as cooks rejected the first version and accepted the second. So this augmentation story really helps alleviate the identity threat but so few people approach advertising new technology that way. And there's another dimension to this too where work can sometimes be imbued with symbolism. So like imagine trying to get a tattoo from a tattoo robot. People definitely do not want a robot giving them a tattoo. It's supposed to be a human craft. But they're fine usually if you ask them the opposite question, they're fine with a robot removing one because that's more of an instrumental process. There's no symbolism involved in it. And what I found at Trail of Bits and with like most knowledge work in particular is that it's a deeply symbolic act. Like yeah, you feel that there's a human craft involved. It is not an instrumental process. You're not just doing the thing. It has a lot of meaning behind it. So especially for security auditing this is definitely the case. So from my perspective I have to frame AI as something that makes you a more dangerous auditor, not it does the audit for you.
There's also problems with opacity where for a lot of complex knowledge work, the subject doesn't really have a lot of understanding of what's going on with it. But because there's a human, you can sort of identify with it. You go along with it, you trust their judgment. But if you have an automated process performing some kind of knowledge work, like let's say a medical diagnosis, let's say you have an AI that is determining if you have skin cancer, even if you don't understand a word the doctor's saying, you still trust them for the most part. But if you have an automated process do the same thing, you're suspicious, you don't have introspection to how it reached the decision. So instead, what you really have to do is blow up the process. You have to explain your reasoning. And that's something that AI is really bad at out of the box. We don't know how it arrived at a decision. You just press the enter button and it gave you one. And most humans are unwilling to take that mental leap. And then finally, there's an intolerance for imperfection. If you have a machine that predicts the weather and you have a human that predicts the weather, the human's allowed to make mistakes every once in a while because a human is a human and it's just sort of expected. But if you have an AI that's predicting the weather and it makes one mistake and it tells you it's going to rain on a day that it didn't actually rain, you will never trust that algorithm ever again. People are very intolerant of mistakes from machines. So generally the fix for this is you need to give people the control to modify parts of the algorithm. If you have even a single adjustable parameter that's enough for a human to overcome this aversion.
So with knowledge of these biases, I had to construct a system in Trail of Bits that works with the biases to overcome them. And this is what the process mostly looks like. For the self-enhancing bias, we made a maturity matrix. Nobody likes being told they're at level one, but that's kind of the point. You can't argue that you're already good enough when there's a visible ladder that you have to climb. And it also makes the conversation concrete, showing what you can do with it instead of just saying, 'I don't think AI is useful.' It also creates social proof. So when you see a peer at level two or level three, a lot of people in the passive 50% start moving. And I think this is actually, I forgot to mention, when we started on this journey at Trail of Bits, pretty much everybody in the company fell victim to these biases. We have an extremely talented, skilled workforce and the more senior you are, the more you resist against these new technologies. So I think when I announced our initial sort of journey that we would be going on last year sometime, that we were all in on AI and that we're going to be using it across all of our workflows and we're going to redesign the way that the company operates. I'd say only about 5% of the company was with me. 95% was resisting, and I'd say about 20% was actively resisting. And I think what happens in a lot of these cases is it creates fear. This identity threat especially creates fear, and they will go along with it in public but in process they'll sabotage the process and they won't do it and they'll just speak to it but they'll hope that if they keep their head low that this will pass over them and that 3 months from now we'll move on to a different initiative, that management's focus will change and it won't be a problem anymore and we can get back to doing what we're doing. That's where the majority of people land when these initiatives happen.
So anyway, going back, the identity threat was the hardest one. We needed to get people to accept skill-allowing automation, but we were very cognizant that it's easy to view this as a skill-replacing automation. So we never asked people to stop being a security expert. We just wanted to give them a new way to express that identity. So we used public skill repositories as a clear example of where a security engineer could write a really complex Claude plugin and they're not being replaced because when that Claude plugin goes into the company repository, they get credit that their expertise is encoded and reusable and now there's people across the company that are thanking them for the skill that they use every day. We also looked at the maturity matrix that we evaluated people with where the highest level of the maturity matrix is not somebody who uses AI the most. It's somebody who invents new ways to work and builds tools with AI. So the identity of the expert shifts from 'I don't need AI' to 'I'm the one that makes AI useful for the company.' We also did hackathons instead of mandates. I never told people they had to do this. We didn't force adoption. We just created a space where people could experiment and see the benefits and learn the pitfalls on their own terms. So adoption felt more like exploration. It didn't feel like compliance against a company initiative. For intolerance for imperfection, if somebody's first experience with AI is bad, you have lost them for months. One error and people abandon the whole algorithm. So instead, we invested heavily in preventing ways that AI could fail embarrassingly. We gave people curated skill marketplaces. We gave them out of the box configurations. We gave them sandboxing so that things like Claude code can't just accidentally delete your work. So all the guard rails and foot gun reductions meant that we had a lot less stories floating around the company of people saying 'AI did something stupid' because those stories are toxic.
For opacity, we wanted to make it transparent about what was safe to use, what was approved, and where data could be put. Because that eliminates a lot of the uncertainty that would have created paralysis. So we created an AI handbook. We showed people what was approved, and we explained why. We didn't just say trust us, but we opened our decision-making. And we didn't just list tools, but we described the risk model that the company had. And then finally, I think the most important one is we wanted to make adoption visible and fast. So the deferred benefits kind of kill adoption. When you have to invest like six weeks of work into a new tool, you are going to bail out of that process. We wanted people to reduce the setup time so that they could achieve a benefit within the first hour of using the tool. You don't want the first result to be mediocre. You don't want to confirm everybody's prior thinking that 'ah, this is useless.' So we made copy-pasteable configurations, one command setup, we standardized in a tool chain and we made sure the first experience was fast and good. And then I was the first person through the door. My voice as the CEO matters a lot more than people think. The passive 50% of the company that isn't sure if this initiative is going to be successful or useful, they're watching to see what leadership actually does, not what it says. So I made sure I was visibly using the tools every day for my job and I wanted to signal to the company that this is not a fad. It's not going to stop. There's no end to this process. This is part of what the company is now.
So at a high level, that's the problem. This is the approach. There are five major parts and each one addresses the barriers that we talked about. So I'll walk through each one in detail. But this is the overview and I just want to reiterate this is not a strategy deck. This is what we actually did. Everything that we did is something that we built and that we open sourced and that we iterated on in production inside of a 130 person company while we were doing real work. We never slowed down to stop to reinvent what was going on. This was iterative and incremental over a months-long period. So let's go through each one. Obvious first one, you can't get momentum unless you standardize on some tools. We got everybody on Claude code and we treat it the way that we treat any other enterprise tool. We have supported configurations. We have known good defaults. And I wanted to have a clear path to 'this is how we do it here.' If you let everybody use every tool, then you're going to run into these imperfection problems. Everyone's going to try a tool and it's going to stink and then they're going to think that AI is awful and it doesn't work. And if somebody learns something, they can't transition that knowledge easily to another person. So we needed to make sure we're on a common platform. You can't build anything else unless you're on a common platform. You'll end up with 40 different workflows and no leverage. So we chose Claude.
Next, we wrote an AI handbook. This was not just to teach people how to prompt, but it was to remove ambiguity. So the key part is a usage policy. We need to know what tools are approved, what isn't. Especially for sensitive data inside of a security firm. And we explicitly call out things like where a tool can't be used, where meeting recorders are disallowed and where we want people to use a specific alternative. And we do this because we don't want endless debates on Slack. We want to give all of our project managers and engineers a shared answer, not just for ourselves but also for our clients. Like if somebody hires Trail of Bits, they need a clear and consistent answer for how is my data going to get used inside your company. So this helped us tremendously in that process. Now once we have a policy then you can start to push more hard on adoption. That's where we built an AI maturity matrix. We already have a skills metrics. We track about 50 different skills for all of our engineers. They're things like Python software development, Git, Rust, various security auditing sub capabilities. But we wanted to explode out AI usage into its own matrix where we clearly define what the stages were and make it into a first-class professional capability. So this is a clear and consistent ladder. There are clear levels, we have expectations, there's a clear path up. You know what you need to do to get to the next level. And we also include consequences for staying stuck. Now in the matrix, what I mean by consequences is that levels one, two, and three are a skill issue. Just means you haven't spent enough time with the tool to get good at it. If you're capable means that you just had a few hours of experience. And you need more time to experiment to get to level two. If you're in level zero though, if you're not engaged, that means you're fighting back against the company. It means that you have not adopted the vision and the direction that the company is now going in. If you dismiss AI as hype, if you refuse to use AI for security work, this is a disagreement on principles, not on skills. So for people that were stuck in the not engaged category, we had hard conversations with our HR team and there were people that left the company because they were stuck at that stage. But for anybody that's at level one, two or three, we just treat this as like okay, you need more time to play.
Now the important part about the AI maturity matrix is that it avoids two big failure modes. The first is I don't want to split the company into AI people and everybody else. The real way that we get momentum on this is to make it a whole of company initiative. We're looking at this as a company transformation not as a couple of teams like a small team of experts. And the other thing that it helps avoid is that we're just giving out a tool. Leadership is just providing licenses to a tool and then wishing adoption into existence. We needed to give people clear pathways to push. So this turns AI into a normal part of performance growth and resource allocation. Next thing we did is in order to push people through the capability ladders, we use hackathons as a management system. So we run these short focused sprints. They're usually two or three days with a single objective. Now there's a problem with AI like everyone knows this. Everyone knows that on Twitter if you pay attention, from noon to 2 there's a big AI release. Then from 2 to 4 there's another big AI release and then you pop open the Wall Street Journal and there's another big AI release and you go read your newsletters and there's more tools everywhere, new features, new versions, and it's just constant, the pace is crazy. So the process we had to use for knowledge management was not to buy people courses or to send them to school or to purchase training. It's really about teaching people how to learn on their own. And we wanted this to be something that was self-directed, that was a little bit more intrinsic instead of something that was from the top down that we push people through. So we created these hackathons with that in mind. And the hackathons they always have an objective. It is not just like 'oh go around and play around with whatever you want.' From a management perspective as an expert myself I was thinking through what are the critical skills that we need people to learn. And one really good example of a critical skill we wanted people to learn was how to use autonomous agents. So we constructed a hackathon based around autonomy. And we forced people to do the unsafe thing with guard rails inside of the hackathon. You were not allowed to use Claude code in this hackathon by pressing enter on every single instruction. You had to let it run in a fully autonomous mode. And we constructed the hackathon to use only public projects, we gave people sandboxing tools to use ahead of time and we gave people guidance about how to structure work so that agents can succeed on it. And with that learning objective in mind, that's what we challenged people to do in the span of two to three days. So we gave them enough structure but not so much that they couldn't experiment. We also had people work in pairs. We wanted everybody to have a buddy that allows their work to get reviewed so they have some quality bar that they understand they have to meet. So this is what allows us to move fast but still have quality control baked in.
Now the next thing, these hackathons obviously produce a voluminous amount of code. We make a ton of knowledge. A lot of people have little light bulbs go off. We need to capture that. So we turn our learnings into artifacts that scale our expertise. And the single most important way we do that is through our skills repository. We do this in a variety of ways. We want to collect every single bit of knowledge so that the next person that shows up can stand on our shoulders. We're iterating on our AI handbook, we're refining the configurations, the tools that we have, but the most important lever is our skills repository. So these are the reusable structured workflows that an AI agent can follow. They're usually examples of how to solve a problem. They're constraints of how we want it done and they're usually some way to verify the output. We actually have three different skills repositories. We have an internal one for company specific workflows. We have an external one that the broader community can use to validate and improve on what we're building. And then we have a curated skills repository that includes third party skills that other people have made. But I'll get back to that in a second. So the ones that we publish, this gives us external validation that we truly care about. It keeps us honest and it forces us to write things that other people can use. Not just other people outside the company, but inside too. It really helps us think about the tribal knowledge that's baked into the tool, the little weird gotchas, and it improves our product sense around the skills to make them more usable when we know the entire public has access to them. Security marketplace. I don't know if anybody follows security stuff with like OpenClaude for instance, but there have been a lot of news articles including some really cool techniques that have come out of Trail of Bits ourselves about how to write malicious skills. It is very easy to fool an AI agent into doing things it's not supposed to, into inserting hidden instructions that a human doesn't see but that an agent does in order to take over a computer, to leak private data, and so on and so forth. So just like any good enterprise, we had to make a curated marketplace that has known good tools for third party skills. I don't want to tell my employees to start using AI and have them download code from all over the internet, run it on their computer. So if you want adoption, you need a safe supply chain and this was one of the ways that we achieved that.
The defaults. So as we saw people using Claude code, we saw the ways that they ran into corners. We saw the ways that Claude code fails and we took an iterative process to learn every single time Claude didn't do something we wanted. We would bake it into a set of global copy-pasteable defaults, known good settings, recommended patterns, and we accumulate all this, I call it scar tissue, into a repository we call the Trail of Bits Claude code config, which is 'this is how we do it here.' We want to add anything that we can standardize so that people don't have to reinvent it. If I hire somebody new tomorrow, I don't want them to have to go through the entire discovery process of the last year of Trail of Bits to figure out how to use the tool right. So everything we learn goes in this repository. And then we started to refine the sandboxing tools. If we want autonomy, we want to allow the agents to operate at machine speed to accomplish objectives that don't require our oversight or that operate within guard rails that we've constructed. And as part of that, we have to construct those guardrails. So we made, we're now up to three of these. We have a dev container solution. We have a Digital Ocean droplet solution and we have something else called Coupe that we just released I think a day or two ago. Effectively we're not prescribing a solution when it comes to sandboxing but we're giving people options because a developer might really like a dev container. A more casual user might go with the natively integrated sandboxing that is already built into Claude code. Some more power users for autonomous processes in the cloud might like our Coupe solution. But the point isn't that everybody uses the same sandbox. The point is that everyone has a safe sandbox to use and that it's easy for them to do it. So we gave people options and these were investments that we made but they're not super difficult ones.
We also looked at other safety risks. Another problem with AI, with agents in particular, is that they just write code and they suck down dependencies from all over the internet. It's not just about the skills they follow, but also the dependencies they use and the code they import. So we hardened our defaults through our MDM system. The main thing, like we do all these hardened defaults, but I think the main thing that matters is that we enforce a 7-day cool down policy on all of our dependencies. So right now the trick is that there are dozens of security companies that are scanning the internet trying to find a new cool blog post they can write about malicious code that's hiding on the Python package index or on npm, and all of those companies are competing against each other to discover these things and they usually do it pretty quick. They usually figure out that there's a supply chain security issue within hours that they come out. So what we do is we just delay all the packages that Trail of Bits uses, and generally all the malicious stuff gets picked up before we ever even get a chance to run it. So obviously we have more controls than this, but I think the real master stroke for our security is just enforcing this delay. And like keeping these package managers, they get more secure as time goes on. You want to actually use the latest and greatest version of npm. It's much safer than the version that came out last year.
And then finally, now we've got policy, we've got guardrails, we've got sandboxing, we've got skills. We want to connect the agents to real tools we have. And the way we do that is through MCP servers. So we started writing MCP servers for a lot of our internal tools. And hardly anybody here, probably nobody here knows what Slither is, but that's not important. What is important is that MCP turns your internal tools into something that agents can use reliably and that your organization can govern. There are some companies like Stripe, they have a system they call Minions, which is really foundationally built on a single large MCP server that has hundreds of functions where they can apply all kinds of granular permissions and controls on every single function. It's connected to every single database in the company and that's their control point when they think about how do we make AI safe. That's really the process that I think a lot of companies are going through right now. So the results are fantastic. Like right now we have an internal and a public skills repository. These numbers are slightly out of date. I updated them about a month ago. But we have more than 100 plugins. We have more than 200 skills. We have more than 100 specialized agents. We have hundreds of reference files that encode our domain expertise and that's all compounding on every engagement. Every time that we do work for a client, it adds to that arsenal. And it's also about breadth. This is not just an auditing toolkit. We have skills for writing sales proposals, tracking project hours, onboarding new hires. Anything that you can imagine the company does is now assisted by an agent. And we also have depth on our core skill set. We have at least 30 plus plugins that target specific vulnerability classes that use our specialized domain knowledge to do it better than anyone else in the world.
Um, and all these packages, they encode expertise that used to live up in somebody's head and now there's something that every person across the company can invoke. We try to do benchmarking and track KPIs on this and the numbers that we have today are that about 20% of the bugs that we find for our clients are initially discovered by AI in some form. They're bugs that we hand review that we sometimes write proof of concept exploits for and they go into real client reports so we know they're valuable and the AI is surfacing things that humans would have missed or they wouldn't have had the time to look for in the span of a short engagement. I'd say on the sales side we also have good information that our go to market team is outperforming the industry through some of the use of AI. We're averaging something like $8 billion per sales rep where the consulting industry benchmark is something like two to four. And this is about one year into building this, right? The models get better every month. Our skills repository gets bigger every month. And we have better agent systems and harnesses that we can deploy. Things like Claude tag just came out. We've invested in building custom tools around like Langchain's deep agent system. So, a year from now, these numbers are going to look very different and the workflows the company has will look very different. I'd say if I step back, a couple things that we've learned over the last couple months specific to security is that discovery has gotten cheap. Discovery of bugs has gotten cheap. So, when I think about the impact this is having on the industry, AI is generating offensive capabilities and bugs faster than we can process them. We have great examples of this. Like there's a contest every year called Pwn2Own. There's one that happened in Berlin about two or three months ago and for the first time in 19 years the company running that Pwn2Own sort of bug bounty thing rejected a bunch of zero-day exploit submissions because they ran out of contest slots. So the most prestigious offensive security contest in the world broke down under the load. So our capability to process this sort of attack capability has not kept up with the capability to defend against it. We're also looking at so this is creating a situation that a friend of mine Katie Msurus has coined 'vulnera'. It's that this disaster that's coming it's not the bugs that are coming out it's the patch tsunami that's going to follow. So most companies can't keep up with the volume of fixes let alone the number of findings. You know downstream of this issue a lot of organizations are going to be deploying a lot more patches a lot more quickly.
