Thank you. Thank you, Gadi. Let's get this up over here. I can see that. Woohoo. All right. Great. So again, thank you for the introduction. I am Dan Guido. I'm the CEO and co-founder of Trail of Bits. And I'm here to talk about how we made Trail of Bits AI native so far. It's a journey. We're not there yet. So just to start out, this article came out about two weeks ago. There were thousands of CEOs surveyed and the result was that AI had no measurable impact on employment or productivity for them. Economists are starting to call this the Solow paradox which is a phrase that was coined in the 80s saying this economist called Robert Solow that said you can see the computer age everywhere but in the productivity statistics. So the same thing is sort of happening with AI right now that a lot of people think they're using it and they find out that it doesn't work and the projects get dropped. So obviously I don't think AI doesn't work. I just think most companies are doing it wrong. They give people tools without changing the system. So everyone gets a ChatGPT license and then leadership waits for the productivity numbers to move and they don't. So that's the gap between AI assisted and AI native. One is a tool, the other is an operating system and the rest of this talk is about how I'm building the operating system at Trail of Bits.
So before I go on, I want to define what we're even aiming for. So AI native gets thrown around a lot. It means a lot of different things to different people and the way I think about it is that there are basically three levels. Most companies start at level one. AI assisted is where everyone starts. You give people ChatGPT, they draft emails, generate boilerplate, you know, the organization doesn't change, the workflows don't change. You just do the same things a little faster. AI augmented is where things start to get interesting. That's where you start slowly, incrementally redesigning workflows. You're not just using AI as a tool, but you're putting agents in the loop. You're changing how work actually flows. And maybe the AI does the first pass on your code review and then the human does the second. So the process is slightly different. And then finally AI native is where you get to a structural shift. This is where you've designed the organization from the ground up assuming AI is a co-participant in the work. It's not a tool. It's a teammate. And that's your knowledge management, your delivery model, your expertise. All of them have to be designed to be consumed and amplified by agents. So at Trail of Bits, what this means is that our security expertise should compound with code. Every engagement we do, all the skills, all the workflows we build make the next engagement faster. Every engineer operates with an arsenal of specialized agents built from our 14 years of audit knowledge. That's not we use AI, that's AI is on the team.
So candidly, I had a lot of trouble getting here. When I first launched this initiative inside Trail of Bits, there was an incredible amount of pushback. Like you would not believe. I think that I had about 5% of the company with me, about 95% against. At least 50% were passively resisting. There were probably a good quarter that were actively resisting. I'm sure people thought about quitting. Just totally like cards on the table here. And this isn't really an uncommon issue. Whenever new technology comes around, people resist adopting it. And it can be the most senior, most expert people that resist the most. So I had to work through a couple of common psychological barriers that are well documented in research and build systems that worked within those psychological conditions to get to success. The core insight here is the problem's not the technology, it's the people. So these are four psychological barriers that are I think key among all the other ones in getting people to adopt new technology inside of a company like Trail of Bits.
First was self-enhancing bias. This is the tendency to perceive oneself in an overly favorable manner. So you obviously want to maintain your own self-esteem and this bias means that you take credit for your success and you blame external factors on failures. Now there are some funny interactions with this. Like in some research by Paul Meehl and Robin Daw, you can take the variables an expert says they use and build a crude linear model and the model outperforms the expert almost always. And it's not because the model is smarter. It's because it's consistent. Humans don't operate the same every day. Some of you are very hung over right now and I'm sure you could agree with me on this. And this gets worse with seniority. The more that you are an expert, the more that you believe that your intuition is what got you there.
There's also an identity threat. This really attacks the core of who we are. That like I spent 20 years doing this. A machine can't do 20 years of experience and like get to be me. And this shows up in things like there was Stanford ran a cooking study. They used this device, some cooking system, and they advertised it two different ways. One way is it does the cooking for you. The other way is it helps you cook better. And it's the same device, but for people who identify as cooks, they reject the device if it's advertised the first way. There's another dimension to this as well based on the symbolism of what the activity represents. So for instance, if you had a tattoo robot, people do not want robots to give them tattoos because it's symbolic. There's a human craft to it. But they're fine with a tattoo removing one because that's an instrumental process. There's no symbolism there. So for us, security auditing is deeply symbolic work and it's important to frame the AI as making you a more dangerous auditor, not something that does the audit.
There's also challenges in opacity. So in this case, the subjective understanding of human judgment is high and AI judgment is low, but objectively you actually understand neither. Like when a doctor tries to communicate to you that you might have some medical condition or cancer or something, you might think you understand the words that are coming out of their mouth, but you absolutely do not. But the level of trust you have in the doctor is way higher than if a machine just prints out a receipt and says, here's what you got. So, the way you typically fix this is, you could modify the same AI to show you, well, why is this not a benign skin condition? And if you give a visual comparison to a benign mole then trust starts to equalize. So we had to think about how to build this into our process as well.
And then finally humans have an intolerance for imperfection. So again well-known academic study participants watch an algorithm outperform a human forecaster. The algorithm is better but the second that the algorithm makes a single error humans switch back to preferring the human. The fix in this case is generally to let people modify the algorithm. If they have the perception of control then they start to trust the process again.
So what did we do? So for the self-enhancing bias we built a maturity matrix. This was controversial internally. Nobody likes being told they're at level one but that's the point. You can't argue with something that you're good enough when there's a visible ladder. And it also makes the conversation concrete instead of I don't think AI is useful. It also creates social proof. When you see your peers escalating up the ladder, the passive 50% of the company starts to move with them. The identity threat was extremely difficult. You had to get people to accept skill-allowing automations and reject skill-replacing automation. Same tool, different framing. So we never really asked anyone to stop being a security expert but we gave them a different way to express that identity. So the skills repos that I hope you all are familiar with are the clearest example of that. When we have a senior consultant on the team write a constant time analysis skill. They're not being replaced. They're becoming permanent inside the company. And that means their expertise is encoded and reusable by the whole team. That's an identity upgrade, not a threat. So the maturity matrix reinforces this. Level three isn't who uses AI the most. It's who invents new ways and builds new tools. So the identity shifts from I don't need AI to I'm the one who makes AI dangerous.
We also did hackathons instead of mandates. I never forced anyone to adopt these tools, but we created a space where people could experiment on their own terms. For the intolerance for imperfection. Everyone, if somebody's first experience with AI is bad, you've lost them for months. So one error and people just abandon the system. So we invested a lot in ways to reduce the ways that AI can fail embarrassingly. We curated a marketplace so that people weren't getting random plugins. We created sandboxing tools so Claude Code didn't accidentally delete your work. We put guardrails and footgun reduction systems in place so that there were a lot less of like you know AI did something stupid stories circulating in Slack. For opacity we clearly articulated what tools were allowed and which ones were not and under what conditions. We described those conditions and why we decided to adopt those risk levels. And then we made adoption fast and visible. So this really underpins everything else. If the setup takes an hour and the first result is mediocre, then you've confirmed everyone's priors. So we have copy-pastable configurations, one command setup, we have a standardized tool chain and the first experience is fast and good.
And then I think finally I matter to this more than most people think. The passive 50% of the company is going to watch what leadership actually does, not what it says. So if I'm using these tools visibly every single day, it signals that it's not a fad. We're actually all in.
So that's the problem and the approach. Here's the actual system we built. So five parts. Each one addresses barriers we just talked about. I'm going to walk through each of these in detail with screenshots and examples. But this is the overview. You could think of it like a checklist. But you need all five for it to compound. And importantly, this isn't like some strategy deck that I'm presenting here. This is everything that we built ourselves and most of it has been open sourced so that all of you can use it right now. This is what I use for a 140 person company doing real client work. So let's go through each one. First one, super boring. We had to standardize. We got everybody on Claude Code and we treat it like any other enterprise tool. We have supported configs, no bad defaults, and a clear path to this is how we do it here. If you skip this step, you can't build anything else because you end up with 40 different workflows and zero leverage on it. So, for better or for worse, we settled on Claude Code. I think it was for better.
Next thing we wrote an AI handbook. And this wasn't to teach people how to prompt, but it was to remove ambiguity. What tools are approved and what isn't, especially for sensitive data where we have a lot of that. So, we explicitly call things out like where a tool can't be used on client code, where meeting recorders are disallowed depending on engagement type, where we want people to use a specific alternative. And that stops the endless Slack debates. It gives project managers and engineers a shared answer, and it makes client conversations consistent. When we have highly sensitive clients ask us questions about, hey, what are you using on the code that you've given you? I can give them a clear answer that everybody knows. So once you have a policy, you can push harder on adoption.
Then I will post this on Twitter because no one can read it. But we built an AI maturity matrix. This makes AI a first-class professional capability. It's just the same as you can use git or you can write tests or you can audit C. It's not a vibe. It's a ladder. So there are clear levels. There are expectations for getting one for escalating and there are consequences for staying stuck. It starts at level zero and if you're stuck at level zero and you're not engaged in this process then there may be a transition out of the company. And this is how you avoid two different failure modes. You don't want leadership to wish adoption into existence and you don't want the organization splitting into AI people versus everybody else. This is a whole company need. Everybody needs to be on board. So this turns it into a normal part of performance growth, resource allocation, and bonuses.
Now the fun part. So hackathons, hackathons became a management system for us. We run short focused sprints, two to three days with a specific objective, one objective, and that's how we keep pace when the ecosystem changes every single week. So here's one of the kickoffs for one of them. We did this one or actually well these are both these are the last two I think. Which one I think the one on the right is our autonomy focused one. The most important thing is that we put a constraint on the hackathon. You had to run Claude Code and bypass permissions mode. This was to break the barrier on people's hesitance to giving full control over the tools and build systems that allow it to work under those constraints. So that was intentional. It forces everyone to learn how to use sandboxing guardrails and how to structure their work so that the agents can succeed on their own.
Other design choices that we made, we focus on public repositories so that we can move fast and show real outcomes. We measured success on activity. I wanted people to put in the reps. So we're talking about issues fixed and filed, PRs reviewed and merged, branches created and deleted. We're not so worried about like the meaning behind all that code. I just want activities that they put in the reps. And then we had people work in pairs. Every change gets reviewed by a buddy. So even with the move fast sort of sprint style work, we had some quality control built in. So the hackathons create motion, but the motion doesn't compound unless you capture it.
And that's where we started collecting these artifacts as skills. So a skill most of you know is a reusable structured workflow usually with examples, constraints, and a way to verify the output. So we maintain three separate skills repos, two of which you all have access to. We have an internal one that has a lot of company specific workflows and a lot of our most sensitive vulnerability research related skills but we also have an external one so that the broader community can validate and improve on the work that we're doing. The third one is we have a curated marketplace which I think I'm talking about next. Yeah. So for a firm like ours I appreciate the external validation. It keeps us honest and it forces us to write things that other people can actually use. That's a core value of Trail of Bits. I don't want to just release a Black Hat talk that gets me a Wired article or whatever and disappear the next day. We measure success based on downloads of a specific tool or stars of a specific tool. I want continued usage and enhancement.
So then we have another problem. You know, you tell people to go use skills and plugins and they go use skills and plugins and they get a whole bunch of random stuff that's like AI slop filled with emojis and you know steals your cryptocurrency. So we created a curated marketplace and this is just standard enterprise stuff, right? We made a known good place for third party skills. You want a third party skill, there's a command built into the curated marketplace that goes out and grabs it from somewhere else, filters it, and then puts it into ours. So if you want adoption, you need to save supply chain. Basic enterprise thinking.
Then we made the defaults dead simple. So after we went through a couple of rounds of this, we started to get to a really known good Trail of Bits style Claude Code installation. So we publish a repository that has our recommended Claude Code configuration. I want to eliminate tribal knowledge. We want to have specifics around this is how we do Claude Code at Trail of Bits. So we put known good settings, a known good CLAUDE.md and anything else that we want to standardize for how people use it. We also included some usage instructions there. Now this is freely available but there is a whole separate section that describes how to use it for bug hunting that we're keeping internal. So if you're wondering where that part is, that's why it's missing. So it's very developer focused.
Then the real constraint obviously the more control that you give to these agents, you're going to want a sandbox. So we give people choices. We don't just tell people to do one thing or enforce the Claude built-in sandbox, which is quite good. But we give people choices and we invested in building some of our own. We built a dev container option. We made another one that automatically interacts with DigitalOcean to create single-use virtual machines in the cloud. And then we documented the native Mac OS sandboxing option. So the point wasn't that everybody uses the same sandbox. The point is that everyone has a safe sandbox to use. You know, we have a variety of different things going under one roof at Trail of Bits. We have engineering engagements, research engagements, audits, we have non-technical staff, operations staff, and they all need different solutions. I can't mandate one.
Then as you give people more autonomy, these agents start to go haywire in different ways. They start to pull in third party dependencies that have security issues. So the one step that we took that I think is extraordinarily effective besides the ignore scripts in npm was implement a mandatory package cooldown policy. So, nobody installs packages that are less than seven days old for any package manager across the company. We have it enforced with MDM. That's the I think the easiest way to reduce risk and make the safe path the default path. There's more that we've done here, but this is something that I'm preparing to release. We're going to start open sourcing our entire JFrog configuration stack for how we secure some of these agents with MDM.
And then finally, once you have the policies, the guardrails, the sandboxes, the skills, you can connect agents to real tools. So one example that we've already wrapped inside of an MCP server is Slither. And if you don't care about Slither, which is a static analyzer for Ethereum smart contracts, the point is that MCP turns your internal tools into something that agents can use reliably and that your org can govern. So this is another way that we can inject policy into how these tools get used.
So now the results. So, you know, you've seen some of our open source work, but internally I can share that we have 94 separate plugins accumulating 2,014 specialized agents and 400 reference files encoding all of our domain expertise. And that's the compounding effect. So every single engagement, every auditor, every experiment adds to that arsenal. And what's important is the breadth, right? We're not just talking about an auditing toolkit. I have skills for writing sales proposals, tracking project hours, onboarding new hires, prepping conference blog posts. We have 20 plus plugins targeting specific vulnerability classes. But the important thing is each package packages expertise that used to live in somebody's head and it turns into something that every auditor on the team can now invoke.
On the delivery side, we have certain clients where the codebase and the scope allow it. We've gone from finding 15 bugs a week to finding 200 bugs per week. And that's not a faster human. That's an auditor running a fleet of specialized engagements that do targeted analysis across the codebase in parallel and then validate the results. So across most of our engagements at Trail of Bits, the number that we're about at is that 20% of all the bugs we report to our clients were initially discovered by AI. On the business side, we also have our sales team averaging about $8 million in revenue per representative, which is about double what the industry averages for sales consulting. So yeah, so AI doesn't just help our engineers it also helps with proposal drafting, competitive positioning, conference prep, lead enrichment, same system, same repositories, same compounding effect.
And this is maybe a year into building the system seriously. Claude Code came out what was about a year ago. The models get better every month, the skills grow every week and a year from now these numbers are going to look very different. Yeah.
So let me close with some of the essential starter path. This is the minimum viable AI native org design. This is what we've done so far and it's changed how fast we can ship and how quickly we adapt. If you want to copy this, you have to copy the system, not the tools.
We do have some open questions. You know, there's obviously some things we're still working on. Obviously, Gadi made an allusion to a conversation I had with him about private inference. You know, we don't want to always send our queries to the cloud. Sometimes we want to do them on a machine that we control. But these local models, they're good, but they're not amazing. There's definitely this big model smell where you can tell the difference between, you know, Opus 4.6 and a Haiku 4.6, no matter what day it is. So right now we're investigating a couple of paths here and I'm sort of reticent to spend money on a local server because a lot of the private inference systems are getting really good. We've been evaluating tinfoil.sh, but there's a huge variety of other private inference servers that are now available where I can confidentially run my queries in somebody else's cloud. So these are where we're doing some research to figure out the right solution for us.
The prompt injection problem obviously not solved. We have a real blunt instrument for solving it today, which is for certain clients, we just deny access to the web. And for a lot of our internal tools, we don't allow them access to external resources. So things we're playing with here are agent native shells. We're looking at things like the no-shell, agent.sh that are going to wrap the agent execution in something that can be protected with kernel level system calls instead of, you know, discretionary guidance thrown into a CLAUDE.md. And then something else we don't have solved yet is policy enforcement and continuous learning. So, we push a lot of settings via MDM, but what do we get back? Because there's a huge amount of data that's accumulated on each person's device about how they're actually using AI and how they're doing their job that I would love to be able to collect and synthesize and use to enhance our process. And we don't have a solution there yet either, but it's something we will probably soon. You can take a look at things from Trail of Bits like our We Audit Visual Studio Code extension that might offer a solution there.
You know, longer term we also need some sort of policy architecture. You can see Stripe with their Minions paper they have one giant MCP server that all their agents call into and that they can use to enforce policy and that might be a good long-term answer. And then finally the business is really shifting. I think consulting today may not last that long. We're looking at billing by hours today, but in a future where everybody has an arsenal of agents and some people can outperform others by 200x, I think you have to start billing by expertise and by results. So the nature of how Trail of Bits offers our services to the world is probably going to change in the next 6 to 12 months as well. And again, no answers there yet because we're still on this journey just like everyone else.
So I wanted to offer a few links to these references. These are all the tools that I talked about during this talk. I also want to mention that yesterday as a gift to you all I open sourced 10 additional plugins. We have one for checking false positives for bugs that your Claude might find and you want to give it the ability to check. We have a fun one called Let Fate Decide that teaches your agent how to deal tarot cards. As well as a whole bunch of others that are up on my Twitter and that are up in the skills repo right now. And then I would also be remiss to mention we have a bunch of jobs open and I would love it for someone to work with me to carry this exact process forward. If you want to help me continue to instrument Trail of Bits and make us further along in this journey, please contact me. Thanks.
