Dan Guido, Trail of Bits

I mean, I know their GDP is something like $25 billion a year, and if they're stealing $1 to $2 billion a year, like this is a huge material impact to how much money their entire country has. Um, like, yes, the insecurity of blockchain protocols is funding nukes in North Korea.

Thanks for having me.

So, Trail of Bits is a research lab that combines really high-end consulting with genuine research and tool building. Um, we were founded to make fundamental progress in software security. And today we have services across government research, across software engineering and security tool building, as well as our consulting practice that most people in the blockchain industry probably know. Uh we do work in application security, cryptography, blockchain security, AI. We work on hard problems for tech, defense, and finance companies primarily. And it's really just an all-inclusive place. Like if you have a problem with software security, then Trail of Bits probably has an answer for it.

So Trail of Bits was founded back in 2012, way before blockchain was ever a thing. I think a lot of people in the blockchain industry think like, oh, Trail of Bits. They obviously named it because they're doing blockchain stuff, right? And that's what the majority of our public reports are about because blockchain people are radically transparent. So, you could be easily fooled and think that's the majority of the company, but it's actually only about 20% of what we do. The way we got started is a fun story. My partner Alex, his wife Amber worked at JP Morgan on a startup within a startup doing blockchain stuff for them and kept telling us year after year like hey this is going to be a big deal, Ethereum thing, it's brand new, it's programmable money, you know, you can manage state, it's so much different than just value transfer with Bitcoin, like you guys should get into it. And eventually she and Alex convinced the rest of us to buy into the DAO.

So we all bought DAO tokens and then we all got robbed and it was this really funny light bulb moment when we were like oh my god did another hacker just steal money from us like what are we going to do about this and we all looked at each other like we're going to make money. So we started consulting with other blockchain industry people right around that time. Uh there was a guy Rick that introduced us to some of our first clients in the blockchain industry and we brought all of our research tools to bear on this problem and started building all the first foundational tools that the industry had. Um, and a lot of them actually came from our government research. We were using like government sponsored hacking tools to help secure Ethereum in 2017. Um, so that's where all this stuff came from. We've just been having fun ever since.

I lost $100, man.

Yeah. No, I think the blockchain industry is really unique because there is an actual immediate risk of failure. Like I'd call it FAFO. They like... huge honeypot. It's not just a huge honeypot. Like when you're in the tech industry and you get hacked, the consequences are disconnected from the hack. Like they usually trickle in over weeks or months or even years and a lot of it is like the risk gets transferred. Like you pay somebody for credit card monitoring and your legal team takes care of it and the industry doesn't really hold you accountable because your data is still at that company. Like I could still log into Dropbox so I can still download my data. I don't care if they lost some of their marketing information. Like what does that actually matter? And you find this in the entire tech industry that when people get hacked there is an immediate drop in their stock price and then it rockets back up 3 months later. 3 months is the maximum amount of time that people have memory of these things. Uh it's sort of why I kicked myself really hard for not buying CrowdStrike stock two summers ago. Um but in the blockchain industry when you get hacked people find out about it on Twitter in real time and the consequences are that your company's over and that drives a lot more interesting conversations from blockchain companies and it makes it a perfect environment for Trail of Bits to succeed because we're trying to sell people on perfection. Like I don't really care about vulnerability mitigation. And I feel like side note, the security industry is a really fun industry because we sell people all these services and we don't actually fix any problems, right? It's like it's fantastic. Um, but for blockchain people, you have to fix the problems and they know they have to fix the problems and they're desperate to find people who can live up to the ideal of yes, we can fix the problem. And for an environment like that, Trail of Bits thrives. So that's why I love working on blockchain projects.

So I went to a public school in Long Island. I went to Mineola public high school and I was a smart nerdy kid. Like I went to the Johns Hopkins Center for Talented Youth Program.

Oh, you did? Oh, that's awesome. Um, what classes did you take?

Okay. I think I was like 13, 12 or 13. And this was at... not at Johns Hopkins, I think I did a summer at CTY at Johns Hopkins and I did a summer in Dickinson College. Yeah, that's great. Yeah, I did it for four summers and I took all computer science courses. And I was like Oh. And I took algebra. I took self-paced algebra. And every single time I took one of these courses, I came back to my high school and they just they didn't accommodate me at all. Like I took the entirety of algebra 1 in 3 weeks by myself by reading a textbook over a summer. I came back and I'm like great, I can take, you know, calc. I can take algebra 2. I can do like something more advanced. And they're like, no, you're going to take algebra 1. Like oh okay. So I did that for nothing. And then the same thing happened with computer science. The same thing happened with every class after that. So I was just getting really frustrated as a kid cuz I wanted to use these skills that I had and I was getting pretty resentful over the administration, the teachers, like the help that I was getting. Um and at the same time, I just found it I like to show up and do unexpected things. So you know, kids want to pull pranks and stuff. They want to mess with their teachers. So, I had all this knowledge. I had all this like pent up not I wouldn't call anger but just like energy and I wanted to have a little bit of fun with it. So, I started having some fun with it. Um I got my first taste when I just guessed the password to the school and every single computer had the same one. The password was Mineola.

And then the like strength of the reaction from that is what told me like, Oh man, I'm on to something like this gave me a bunch of dopamine. I'm going to keep doing it. Uh so it escalated until like you know I got the principal's email password. I got the Windows domain admin password and I used stuff like PWdump and I figured out how to do wireless hacking. I got like a junk laptop and I put an Orinoco Gold card in it with like hacked firmware in order to get raw packets off the sky, off the internet. So I kept doing that stuff and then eventually the school wanted to come down hard on me, right? Uh, and they created all these new rules when I was in 11th grade that like I had to sign away a bunch of my privacy and they were going to spy on all my activity and they wanted to like institute all these sorts of punishments if I did things that were wrong. And they gave me this form to sign that told me all this. And I'm like, I go to a public school. You have to give me all the same access to everything that every other student has. Like I pay taxes and that's all right. And I refused to sign it. Um, so they wouldn't let me have access to a computer at all for my last two years of high school.

I was furious. Um, I actually I went to the ACLU at the time and I'm like, Can you help me with this? This is completely not legal. Um, and I got no help at all. Uh like they I couldn't even get a competent like legal person to like look at what I was saying and it really soured my perspective on the ACLU for the rest of my life. Um but yeah, so like that's kind of how I got started is completely did it on my own. Uh really did it to have a little bit of fun. Did it so that I could play Need for Speed 3 in the computer lab during my off periods, which I couldn't do without the admin password for the computers. Um, and yeah. Yeah, that's kind of how it started.

So, I had no community to reach out to until I got to college. The only thing that I had when I was in high school is I would go down to Barnes & Noble and sometimes I would pick up a copy of 2600 magazine, which was distributed in Barnes & Noble at the time, which is just insane to think about. Uh, so I remember getting a couple of copies of that. Um, but to be honest, a lot of the content then was a little bit above my head and a lot of the stuff that I learned was literally just like reading Linux man pages and brute forcing it on my own.

Yeah, so I applied to a bunch of different places. I took the one that gave me the most money. It worked out pretty well. I went to Polytechnic University in downtown Brooklyn. And the reason why I chose that is they also had a scholarship program called Scholarship for Service or the CyberCorps program that would give you it would completely cover your education in return for working for the government in a cyber security related career. So I knew when I was 17 years old that this was a spark that I had and I wanted to keep going with it. So I joined Poly and the very first thing I did when I walked in the door was I wanted to make an entrance. I wanted to like introduce myself to the security lab there. There were about 30 kids that were doing that program. It was one of the first cyber security programs in the US. Um they had actual courseware. They had like more than one course. They had like five courses in cyber security and a whole bunch of kids that were doing a bunch of self-study. And they had a PHPBB web forum that was set up for all the kids in the program to communicate and I thought it would be great if I defaced it and said, Hello, I'm Dan. So, I did that. I wrote...

Oh, yeah. Well, everybody loved it except for I think his name was Constantine, the guy who was the admin for it. He was a little grumpy. Uh but yeah, so I defaced it and I was like, Hi, I'm new here. Like, you know, I'd love to meet you all. Whatever. And then I walked in the door. I'm like, I'm Dan. and they're like, Oh my god, you're that guy. Um, that was fun. There was a public exploit for PHPBB that I like refactored and used, but like I had to learn SQL in order to pull that off and I had to manipulate a bunch of other details that were specific to his installation. Um, so I was a little bit of a script kiddie, but like I was doing some work for it.

I think it was you know just like the blockchain field there's immediate feedback right like you know you write your exploit you press enter you get a shell and that rush you really can't compare anything to it. It's not like playing on defense right like you know playing on defense like you do all this security stuff and then you sit on your hands and you wait for 6 months and you hope that you don't get hacked. Kind of boring. But when you're like identifying security flaws and exploiting them and then teaching other people how to fix them. It is like you can't argue with it. You're right. And that like black and white nature of it is really appealing. I have this drive where like I want to know stuff. I want to seek out new information. I want to find the right answer. And it's really easy to find the right answer when I can test. Yeah. Yeah. When I can test it. Um so that was always really fun. That was really addictive. Uh plus I also just really like to teach people stuff. Uh you know eventually while I was at Poly this turned into me being an adjunct there and I was teaching classes before I even graduated. Um I took over their top level cyber security programs so I was teaching a course in penetration testing and vulnerability analysis, application security. Um I ran at least one like self-study course with a bunch of other students and I just really like showing people things that they don't already know and that was another way to do it like you know if I'm hacking into... It is irrefutable that I have discovered something that you missed. Let me help you discover the ways that these systems compose together into something that you didn't expect. Um, and it's kind of a great teaching tool.

It was a win.

It was a stressful time, but it was a win.

Yeah. So, when you're in SFS, you have to pay off your time with the government. And I don't want to work for like the veteran affairs or something, you know, like I'm not going to be an IT janitor. No, you know, no offense to people out there that need to do assistance stuff. Like, it's important work, but I really want to push the limits on the field. I want to be there at the tip of the spear. And when you look at where the tip of the spear is in the US government, if I'm going to pay off my time, I want to work for the NSA. So, I got some opportunities to do that. And I think it was really formative for my career. Like I think a lot of people in security, just like my little story about defense, like they do a bunch of stuff, they sit on their hands, they wait to get hacked, and it's you don't get feedback. You know, if you don't have an active group of attackers going after your company, how do you know what you do is even useful? You need that testing and most people never get tested. But when you're at NSA and you see how people really get hacked and you participate in that, now you know the curtain has been lifted. Like you understand how security really works. And I think there's so many people in security that have never had that experience and they end up telling you to fight ghosts. But I think I know more about what the real story is because I had that early in my career.

Yeah. So I worked in two different roles when I was down there. The first was one called TAO. So TAO is Tailored Access Operations. That's like the group that performs more surgical kind of signals intelligence collection. That's a good leave it. Yeah. Yeah. A little hand wavy. Um and I developed tools for them. Tools. Um and then on the other end of it, I worked for something called the SNAC which was the defensive operations for cyber stuff. They would do they were actually sort of like internal consultants. They would go to various usually military networks and help them identify exposures and fix them before a foreign motivated adversary hacked in. Um but obviously like both of these sides of the house were pretty tightly integrated, right? Um especially after like you know post 911 stuff. Um there was a little bit more information sharing and I got a really good perspective about everything that was going on. Um, so I had fun down there. I spent a long time down there, but ultimately I burned out because, you know, what you get to do is different. Like, yes, there's no other place where you can get to be authorized to hack into foreign countries, right? Kind of a unique job experience. Um, but there's also like the actual techniques that you're using for that job are not unique. Um, so I could go work for a consulting company in New York or I could go work for a bank or I could go work for another like security firm and I could use a lot of the same techniques, use a lot of the same technology but just have a different mission.

right? And that's crazy for NSA for like I think we've got to admit that on the cyber end of things, the reason why an organization like NSA is so good is because they have the motivation to be, they hire people to build systems. They are slow and steady and they keep their eyes on the target and they always get there. But the actual mechanics of like, well, how do I write an exploit for this? What are the flaws that I'm going to exploit? Those are all well-known. Like we kind of know we're on the same page about that in public and in private. So, I couldn't justify that with the downsides of also living in rural Maryland and commuting to work in a fort in a car I didn't want to drive.

How so? Like who didn't innovate?

Oh sure I mean but like how come you know why does anybody get hacked? There's you know people take their eye off the ball they have pressure there's business needs that like oh you know I have to deploy this new technology. Um, you know, security is not always the top priority. I'm a pragmatist when it comes to this stuff. I actually I advise a bunch of startups. A lot of people trying to go from zero to one, seed, series A, whatever. And it's always funny to be on the advisory board for a company like this. They're like, Oh, Dan, tell us how to be secure. I'm like, I don't I the most important thing for you is to get users. Screw all this security stuff. Like, you need to be alive first for us to have this conversation around how to not get hacked. Um, and that's just life. Uh, it's also a massive resourcing issue, right? Like even if you decide you want it, can you? Um, and with NSA, you know, you might have a team that's been assigned to like study some target for like weeks or months or years or whatever, and there's just one dude at that other company that's like trying to make sure that everybody's computers are patched. It's not a fair fight. Um, so yeah, like I think not to like skip ahead or anything, but this is really the promise of AI is equaling the playing field a bit. It has huge advantages for defense that it doesn't give to offense because the talent constraint and the cost of getting help is now starting to equalize a little bit every day.

Um, I can't provide an assessment of what the NSA is capable of doing right now. It's been 15 years since I saw anything and I couldn't say anyway. Um, but I can say that just like objectively I am truly impressed with what North Korea's been able to pull off. um they are like normally from countries like that you're seeing people recycle pentesting tools using off-the-shelf exploits and techniques that are well understood. And what you see from North Korea is they are performing intrusions. They are assessing the environment that they have access to and on the fly writing novel tools to take advantage of it. Um they're writing like smart contract code. They're writing malware. They're writing like they're doing things that are really quite sophisticated. Um but that's because they've chosen to specialize. Like I actually I don't think they're doing a lot. Like it does seem like their focus right now is almost like 80% on blockchain protocols because it's showing so much success. Um and they have a deep bench of knowledge and talent to be able to do that. They have like an MO that works that they can redeploy every single time. And every single time they learn something new. Uh so yeah, I'm impressed. They did a good job. Um as an attacker like I have to appreciate good game when I see it, right? Um on the other end of it, Iran's a little nuts. Um that's the thing like I think there's a lot of countries out there that have really amazing capabilities, but do they want to use them? And Iran does and North Korea does. Like that's what makes them scary because we know that everything's insecure. We know that everything's unsafe. We know that like if I ask some guy on my team to sit down and study name a company, you know, pick anyone out of a hat, we'll be able to get in as long as we spend enough effort and that's just one guy on my team. Whereas instead, if there's some country that has a budget and you can put somebody on it 40 hours a week for 6 months, like yeah, they can blow it up. But will they? And with the NSA or with somebody else in the US, they're going to have reams of lawyers that are going to pull out some huge book of like title 50, title 10 stuff, be like, I don't know, are we allowed to do that? They have so much restraint, but like North Korea and Iran don't, right? They have nothing to lose.

I mean it's really there's people that know for sure and it's going to be people in the government right um so I can only make assessments from the outside. And from the outside like I would bet it's probably a pretty like startup mentality. It probably looks a lot like Trail of Bits. It probably looks like, you know, they have an internal wiki where they keep notes about things that worked and things that didn't. They have a bunch of git repos where they've got forks upon forks of like five different malware toolkits where they've got process documentation for here's how to pull off this attack and this attack and this attack. And they've got a target list with like priorities on it that are evaluated according to both like impact, you know, amount of money they can steal versus the difficulty and effort required to get there. Yeah. Um, and it's just a systematic thing. And I'm sure that a lot of it is sort of like tribal as well. You get people that have been around for a long time. They've done it before. You take the new kid, you let him sit in the driver's seat, and you know, like that's probably what it looks like. And the numbers that I've heard thrown around is that we're looking at somewhere between like 20 and 50 or so people that are doing this on like a semi-full-time basis. And that's all it takes to steal several billion dollars a year from crypto companies.

Yeah. These days. I mean, I know their GDP is something like $25 billion a year. And if they're stealing one to two billion dollars a year, like this is a huge material impact to how much money their entire country has. Um, like yes, the insecurity of blockchain protocols is funding nukes in North Korea. Just so we're clear about what the impact is.

Yeah. If you fail at securing your blockchain protocol, the world is going to end. So like please do a better job. I know that's a very strong incentive if there wasn't enough more already.

I was down there like it covered a two calendar year span of time. Um a little bit on and off in between. Uh I actually weird tangent like you know you can include this. I have Crohn's disease. I had a really big flare up when I was down there. I had to take like six months off. I had to delay my school. Um I lost a bunch of weight. Uh, I got shingles. That was awesome. Yeah. Just like huge health problems that year. Um, but I eventually I really just realized I want to be in a big city. I need to be around more people to exchange ideas with, to go out with, to like, you know, have access to culture. And um, I really desperately wanted to go back to New York. Uh, so I started looking around for opportunities to pay off my time from that scholarship in New York. And the place that I landed was the Federal Reserve Bank of New York. Um and I worked for the whole system on an incident response team, which again kind of like aligned with my interest of I want to be where the action is. Like so much of security is just sitting around waiting for nothing to happen. But when you're a big target, when you're like the Federal Reserve, people hack you. And if I'm the head of incident or not, I wasn't the head of incident, I was a guy on the incident response team, the IR team. Uh that means I had something to do every day and it was fun and novel and interesting. So we got hit by they didn't have a name for it when I was doing this. This must have been like 2009. We hadn't even coined the term APT yet, the advanced persistent threat which was effectively just this moniker to say China without saying China at least in the beginning. um they would target the bank like every week and that was way better than working at any tech startup. Um so I got I think the Federal Reserve it's not quite government but it's kind of government but like in a place like that it's very easy for somebody with ambition and you can very easily accumulate a bunch of authority just by showing up with a plan. Yeah. And motivation. And that's what I did every day. I showed up with a plan. I showed up with motivation. I came up with techniques to detect these guys inside the bank. Uh I was always on the ball. I respond I think my resume says that I responded to a 100 security incidents. Um which is like one every 3 days that I was there. Oh no, not like four or five days. Um there were weeks where I slept at the office. Uh there are weeks when I found incidents at 4 in the morning. uh we caught a lot of interesting stuff and it just served to better validate what had already started when I was at NSA of like this is how it really looks. This is how companies get hacked. This is how hackers hack and all that fed into this good like clear view of the world that informs everything I do now of like I don't want to waste your time. I don't want to tell you to fight ghosts. I don't want to, you know, give you help that misleads you. I want to give you the right answer and I know the right answer because I've been on the front line. So that was like also a pretty good choice in retrospect of place to work.

Yeah, I'm a moderately capable researcher, but I definitely say I'm more of a practitioner. I'm a doer, right? Um, but when it comes to research, I think the thing to remember is that even if you're a tech company or a bank or whatever and you adopt all the best practices, you know, you do all the things that everyone tells you to do, you're still going to get hacked. And the reason is that like we just don't have the techniques available to actually adequately secure against attacks. Like we have to make fundamental progress. We have to come up with new techniques. We have to advance the field, which also informs a lot about what Trail of Bits does. Like the status quo is not acceptable, right? Uh so I have to I'm like effectively the CTO of Trail of Bits and I have to plan out our research and development vision and the kinds of technologies and things that we create and select from like a huge list of hey we could do all these 50 things. What's the three that are the most important? Um and because I have like a good knowledge of well what's going to help the most? It helps me be pretty fine-tuned about it. But yes, I'm much more on a practitioner end I think.

Oh no. I mean sometimes we get our best research from the consultants. Uh you know they're under the gun. They have to deliver. They have clients that need their help and we deeply empathize with the problems that our clients have. We feel this compulsive need to do the things that they need us to do or rather to do the things that they need to do. Like it's at the end of the day it's not our responsibility but like we try to keep as much weight on our shoulders as possible and under those intense times of pressure that's where a lot of the best ideas come from. Um, so all of our consultants, we have like a reasonably low utilization rate. Most consulting firms, you know, you go to like Deloitte or McKinsey or something.

Yeah, it's total sweat shop. You're going to be like 80, 85% utilized and the 15% you're not those are going to be the minority of vacation days that you get. Uh whereas Trail of Bits we try to hit like 65, 70 which gives us a lot of time across the business where our consultants effectively work for us and we can use that time to build tools that we need to reflect on what we learned from different projects and improve our understanding of the problems the clients have.

Yeah.

I'd say because the consultants have so much alignment with our clients, that really enables us to build custom-fit, perfect alignment tools and techniques that help with them. But the researchers are on a different spectrum, right? The researchers typically work on DARPA programs. We do some ungodly amount of work for DARPA. We're on a rotating staff of like four to five DARPA programs at any given time. They usually last for two to four years. We have two to five people that work on each one. And we're trying to figure out where security should be 10 years from now. And the kinds of solutions that you make for 10 years from now aren't always useful for what you need today, right? It's not necessarily commercially important now, but it's setting the foundation for...

Yeah, so I mean DARPA's claim to fame is they invented the internet. But basically they're like a venture capital firm that has no responsibility to make back the money they give away. They get billions of dollars in budget. They hire in about 200 program managers, which is like a partner at a VC. And each of those program managers is able to create, let's say, about four programs, which are four moonshot ideas that they can give away, let's say, 20 to 50 million dollars to see through. And they get a time horizon usually of about four years to make it happen. So they get to work on... it's crazy, like if you become a DARPA PM you get 200 million dollars. It's prestigious. It's time limited. You're only allowed to be there for about four years. You get maybe 200 million dollars and they tell you to spend it on whatever the heck you feel like. There's very limited oversight. I mean, there is... that's not true. There's some oversight. They have a tech council that's extremely rigorous where they have to vet all the ideas that you come up with. They have these hail-mary questions to validate that your idea is good and that it's moonshot-y enough. But at the end of the day it's one of the only places in the world where you get a chance to take risks that big. So they do a lot of things in cyber. They do... through this subgroup in DARPA called I2O. We've worked with people in I2O continuously for our entire existence. The first money that Trail of Bits ever received was from DARPA. We worked on something called the Cyber Fast Track, which was a scaled-down version of these big programs that enabled us to get three research programs in 2012, six months after we were created. That got Trail of Bits off the ground. So yeah, DARPA is incredible. You have to really... it is not easy to get money from DARPA. It is not easy. It's not easy to come up with ideas that are DARPA-level. That's another thing I think a lot of people think about, 'well I have this great idea to do this research thing' and what you're talking about isn't actually research in DARPA's eyes. Like they're really aiming for where do we need to be 10 years from now, not where do we need to be next week or six months from now or even a year from now. So DARPA is a federal agency whose specific focus is to fund essentially venture-style moonshot bets that can improve America's technological stance over the next few decades. Clear it's part of the DoD and the specific aim is to help us win wars. It's to improve the warfighter. So a lot of these capabilities like for cyber really filter into like how do we improve signals intelligence? How do we improve the security of the US versus these kind of cyber threats? But there's a lot more there, like they've had incredible success in fields related to biology for how do we ensure that when people get hurt on the battlefield that they don't die. They've had incredible innovations in that. They've done all kinds of things with prosthetics, a lot of brain-computer interfaces. They work on amazing stuff. It's really crazy.

Oh, yeah. There's so many. I get them all on Twitter all the time. Every time we tweet something about DARPA, we get like a huge list of crazy people that tweeted us about it.

Yeah, I took a pit stop at this really great little company called iSEC Partners. When I was at the Fed, you know, incident response is great, but I never felt like it was more than a pit stop in my career. I didn't want to be in incident response forever because you're really downstream of where the problem is, reacting. Yeah. Like I had to argue, and this is where I got to be really persuasive, I had to argue up the hierarchy to say, 'hey we've had 50 incidents of this type, here is the security control that if you put in place I can go to sleep at night. So, like please, for the love of God, turn on DEP in Adobe Reader so that we don't have to deal with PDF exploits anymore.' Yeah, that was the thing back in 2010 was PDF exploits. So I said, 'Okay, I want to be upstream. I want to work on software security. I want to consult with people that are developers, not with people that are IT owners.' So, I joined a firm. And this helped because like I was doing this at night. I was still teaching at Poly. I was teaching there for seven years, man. So I was teaching kids how to hack into computers. I was teaching them software security, how to audit code for bugs, how to do it in C++, how to do it in web apps, how to write fuzzers. I had this whole crew of my friends and we would go in and give these kids a class that honestly they had no idea how good it was. But yeah, it was an incredible opportunity. So I kept my software skills sharp from that. And I joined iSEC Partners. iSEC Partners was effectively the software security consultants to the stars. They were Silicon Valley based and everybody in Silicon Valley that needed high-touch security consulting talked to iSEC Partners. They were... you know everything in security is very tribal. It's a very small community. There's very few people that actually know security very well. And we all kind of know each other. So, the field really started way back in the late 90s, early 2000s with a company called AtStake. Which I could go back further, but we'll just start at AtStake. And AtStake eventually got acquired by Symantec, an antivirus company. And then a bunch of hackers decided they didn't want to sell antivirus software, so they all quit. And when they did that, it was this huge diaspora of brand new companies that started overnight of all the people that loved what they were doing at AtStake but wanted to work for themselves. So iSEC Partners was the San Francisco office and there was another one in the New York office, Matasano crew. And there were a few more around, but like I felt like the world was basically divided into there was iSEC Partners and Matasano and that was like the coolest places to be. So I ended up at iSEC. And I did that because iSEC wanted to start a New York office. They didn't have anybody here and I would be their first employee. And that was a huge opportunity for me to build something instead of join something. And I feel terrible about this because I love the Matasano people. There's a whole another life I could have had if I went with them. But iSEC got me in front of all of Silicon Valley. They taught me how to master the art of software security and showed me again, like pulled back the curtain on how does real software actually get made and what are the kinds of problems that people have and just generally gave me this window into the world that was compressed in this about year and a half, two-year period that I would never have been able to get if I just worked at one firm.

Yeah. So I loved that job. I loved that job and I love the people I work with. But the problem was that every single engagement you do, you know, some whatever PayPal hires you in to do a security audit and then 12 months later you go back to PayPal and you're like, 'what do you find?' All the same bugs. And it's like, 'did you listen to us at all? Did you like read the report that I wrote?' Like I spent so much time on this, stressed out about it. Didn't sleep the night before the report was due. And what did you do? So that was really frustrating. The other frustrating part was that we really didn't have... it was very iterative. It was very like, 'okay we did this assessment. We have a little bit of time afterward to study and think about what we learned and we're going to build a couple of tools every once in a while that help us get our job done.' But there's no fundamental change to the way that we're offering services to companies or to our clients. And that showed up in like year after year we'd find the same bugs and it showed up in year over year we're using the same techniques to find them. That really motivated... so now you see like the history that I've had all starts to come together of like I know the status quo is not acceptable. I want to hold my clients accountable. I want to use research to advance the field, but I still want to get out there and help people. So we mushed all this stuff together and that's how it turned into Trail of Bits. We wanted to do fundamental research to build new tools to help people with like government-sponsored hacking tools and that's kind of where we started in 2012.

I think so too. I mean I'm kind of shocked that more security consulting firms don't operate the way that we do because I think like we're very radically transparent. And 90% of the work we do ends up on our blog or on GitHub or something like that. I'd say like the three pillars of Trail of Bits are that we actually have a research lab mentality. We do fundamental research. Second is we build foundational tools. We actually work on filling in major gaps where there is no coverage and you can't get access to like some attack surface or whatever. And then last is we learn and share... we help people learn and share all the knowledge that we get.

Yeah, I think we have this weird portfolio of different businesses under one roof. And I think that most people just choose one and stick with it. And you find a lot of pure-play consulting firms that offer great services but don't do research and then they're caught flatfooted when they're working on emerging technologies where they actually need the research backing to do it right. Or they do a lot of work but you never hear about them because they haven't found a way to discuss what they do publicly. They haven't like browbeat their clients to say, 'we need to share the work we've done for you. It's going to help the world.' Or they do some research and they share it but don't actually improve the fundamental capabilities of everybody else. Like they find a really great bug and they talk about their really great bug but at the end of the day that's a drop in the bucket. Hooray for you. You found that great bug. But like how can anybody follow in your footsteps without tools to do it? So we combine all this stuff together and I think those three characteristics really make us unique. I think many other companies have one, maybe have two, but nobody has three.

Oh man, we did it poorly. We did it very poorly. So the company really started on like January 1st, 2012 and we had no idea what we were doing. It was me, Alex Sotorov, and Dino Dai Zovi, and the three of us were just playing business. We couldn't even incorporate the company. We couldn't choose a name. That was great. Like we showed up, it was April, and we're like, 'Oh, no, maybe it was February, whatever.' We were working in an office for a month. We're like, 'Great, we have a company.' And the only thing we could call it was like 'the company.' So we sat down. And in my true style, I said, 'Guys, we have a deadline by Monday. I want all of you to come to the table with at least 10 ideas for names. We're going to choose one of the 30 that we come in with and that's going to be the name. We're just going to commit and stick with it.' And we did that and we showed up the next week and all of us had zero names. So, at some point Dino was like, 'Well, I already have trailofbits.com. What if we just do that?' We're like, 'Sold. Let's do Trail of Bits.' And then we couldn't figure out how to incorporate it because he had registered an LLC. We needed to reincorporate it as an Inc, which meant that he had to wind down the LLC and then the lawyer screwed up the paperwork and we had to amend it and restate it in August that year. We didn't even have like a real company. But I think the more the answer that you're looking for is none of us took salaries for nine months. Like we started this company, we wound down our savings, we worked and worked and worked through 2012 to get money in the bank and then the first money that we spent was just a teeny bit on each of us to get modest salaries so we could pay our rent and we plowed it into salaries for employees from day one. So we ended that year with I think six or seven employees.

Yeah, we brought on a couple of people. And we also stupidly began development of a product that we had no business writing and no skills whatsoever to sell. So, it was a huge money sink. But that was like, you know, the exciting stuff like, 'yeah, we're going to be big. We're going to solve problems for the industry.' And yeah, I think I was making... I think I paid myself 100K after like October in 2012. So, I went like 10 months. Which was I don't know. I just thought that's what you had to do. I thought that's what you know, that's like in the rule book, right? Like you start a company, you don't get paid and save the money for what you want to spend it on.

Yeah, we were writing proposals to DARPA. We were doing government research.

Yeah. Yeah. Yeah. So, it was a special moment in history. First I did have connections because I worked for the DoD for a bit when I was down at NSA. So I did know how to navigate that bureaucracy and I understood what the delivery requirements were and like what the deliberation process looked like internally, you know, that was good orienting. But what really happened was there was this one guy, his name is Mudge, Peiter Zatko, and he joined DARPA as a PM with the aim to scale down the government contracting process to the minimum viable contracting process. He wanted to allow individual hackers who he thought, and I think is right, are the best cybersecurity talent that the country has to offer and enable them to rapidly work with the government with zero friction. So this is a weird DARPA program, right? DARPA programs are usually, you know, I remember I saw a DARPA program one time for a stealth motorcycle, like crazy stuff like that, right? But in this case the DARPA program was can we accept a proposal and get money into the bank account of an individual who's never worked with the government in five days or less. That's a DARPA program. So that was called Cyber Fast Track and we knew Mudge quite well and this was a big thing in the industry. So we were just writing proposal after proposal. We're like, 'All right, we're just going to brute force this thing and find a way to work with them.' So we ended up getting selected for three different proposals. We did one that was sort of a code analysis tool called CodeSonar, one that was a bug finding tool called Pointsto, I think the last one was an obfuscation toolkit called MAST. And the three of those together funded the company.

They were small in size, you know, each... I can't remember the exact dollar amount so it was somewhere between like 200k and like 700k each. And that enabled us to get our initial set of technology, we hired a couple employees to work on each one and it got us exposure to the government and it gave us like this escalation path where like, 'okay we did Cyber Fast Track, what's next? Now we know how to write a small government contract, can we write a bigger one?' And it got us access to partners because we weren't a prime contractor. So, we're going to work through a bigger company. Yeah. So, we partnered with like Leidos and Raytheon and Mantech and all these other defense contractors. And they always had these big programs that were on that were like 100 million dollars, but they required specialized expertise that they didn't have. And we were the specialists. We're like, you know, SEAL Team 6, whatever. Like, call us in for the crazy stuff. And we would work on these little carve-outs in these larger programs to do insane software security backflips. And that sustained us for about two years.

No. So Raytheon had a DARPA program. So, Raytheon would, as a prime contractor, they're enabled to directly contract with the government. So, they get like a 10 million dollar contract from DARPA to work on some PM's crazy plan for like, I don't know, an interplanetary internet network, right? Whatever. And then Raytheon's on the hook for delivering that. But I don't think there's many good hackers that want to work for Raytheon. So, they bring you guys in, they assemble a team, and they bring in a bunch of subcontractors where they might have like a team of five different subcontractors and the way they carve it up they're going to find like, 'oh okay like we need to do something weird with LLVM to do compilers, the right guy, here's the right team for that.' Yeah and we would get that little piece of it. And what's nice about this too is some of these programs are classified, DARPA does most stuff unclassified but some stuff classified, and even if the program was classified we were able to get carve-outs where we could do our work in the open. We were... I mean I was aggressive about this from day one.

Because I annoyed them. I would call them. I would email them. I'd find every little regulation I possibly could to get them to allow us to do work in the open. That was a strategy from day one. That actually led to some really funny situations like we worked on... so from CodeSonar, that was our code analysis tool. It effectively like would... you could give it like a block of code and ask 'what does this do' and it tells you the answer. Yeah. We turned that into something that was like a decompiler called McSema, terrible name but we love to beat it up, but yeah McSema and we worked on McSema for two other DARPA programs afterward. It was this incredibly flexible analysis tool for binary code and we really wanted to release it and we had to drive through this release process for like two years in order to put it on our GitHub and it was crazy because I think internally the way it worked is it's called the DIAR process. They had to print out all the source code, give it to two people and they had to read it for classified information then scan it back onto a CD and mail it to us. And we had to go... I remember going to Best Buy, our office was in Union Square. We walked over to Best Buy. We bought a CD player for the specific reason to like get the government authorized public disclosure code back out and put it up on GitHub. But like we went through that work and not every other company would, especially defense contractors like around the beltway kind of people, they have no motivation for that. They're just going to work on another DARPA program afterward or another government program afterward. Whereas with Trail of Bits, I'm trying to bring these things to Silicon Valley. And I can't do that if it's like my customer is not just the government. So the McSema thing was our first taste of that, our first success at that I'd say. And we've continued to do that ever since. So, a lot of the government work, I'd say 80% of the government work we do, I have fought hard to make sure people outside the government can see it. And I like it that way.

That's what they're paying for, right? To help America, help private industry, all on the same page, man. It's a frustrating... the whole clearance system, classified system is very frustrating, you know. But I found a good way to navigate it. So I'm proud of that. And I think that among defense contractors, you know, we've also got kind of a reputation inside DARPA now where if you want the work you're funding to have applicability outside of the DoD, you talk to Trail of Bits. And that's a great reputation for us to have.