When you talk to customers, how are they thinking about their risk assessments with all this AI adoption? So more and more businesses are using AI models in their products. What are you seeing there?
A lot of what we see is, I think, the universe is cooling. By that I mean the big enterprises have concerns about how their data is used or how these products might work. The software vendors and people building with AI want to satisfy those concerns, but no one quite knows exactly like the magical incantation to say to, you know, either ask for or provide to provide a bunch of certainty. And so in that we're starting to see some of the standards bodies create new regulations: NIST AI RMF in the US, ISO 42001, and I think more of these will gain traction over time as we just kind of figure out how to talk about some of this stuff. But again, right now we like don't quite have all the words, and so the conversations can be like a little bit longer, if that makes sense.
Thank you so much for coming on, Christina, and I really appreciate you coming on today. Vanta is an incredible company, created and defined the automated security and compliance category. We're going to talk a lot today about how AI has become part of Vanta's story. But before we get into any of that, would love to get to know you two a little bit. Christina, you have this amazing story. You led a very curious life that led you from Ohio to following your love of technology to Stanford, from a venture role at USV to teaching yourself to code, from product management at Dropbox to founding Vanta. So, so you got a lot done. I heard that early on this wild path you were flipping Beanie Babies when you were 11 as a young entrepreneur. As a former Beanie Baby lover, I'm just curious what caused you to do that? What compelled you as a kid to start Beanie Babies?
I think it was like the puzzle of it, or I've thought of, I think it was a puzzle bit, and honestly I think it was having only an allowance that was $5 and like wanting to have things that cost more than $20 a month maybe, right? $5 a week allowance, which I'm very grateful to have, but I think wanted a bit more. And there were Beanie Babies. Thank you, creative. We see this in venture too. Yes. But I think also, it was this like puzzle to figure out on the internet, and this was early eBay. And there were people who were doing this, and it seemed to be working for them. And so it's like, okay, this is possible. Like, can I figure out how to do this? And the answer was like kind of, or like I did the things. I don't think I made any money. Might have even lost some like allowance money, but it was fun. But for the love of the puzzle. Yeah.
Do you have a favorite Beanie Baby? Do you remember?
Yeah. Oh, yeah. Well, I was like really into the camel. It was like an older retired one. His name was Humphrey. I didn't have him. He was, I think, $1,200 at the time. So I did not have him. And then about 10 years ago, I walked into a thrift store and I found him and I bought him. It's totally random, but I'm serious. He's actually on the shelf behind me. We'll circle.
Okay, great. So Humphrey's on the podcast, too. Yeah, mine was Hope the Bear. I remember I had like four Hope the Bears. So, I love that. But you were a very young entrepreneur, we'll call it. At some point in time, you believed that you needed to be Zuckerberg coding since you were six to start a company. I'm curious when you changed your mind about that, and specifically when you decided to build Vanta.
Yeah, longer process, but I think the short version was actually working at Union Square Ventures, which is an early stage venture capital firm in New York, and like really luckily to get that job. And a lot of what that job was was meeting people who were starting companies. You know, probably like 10 a week kind of on average. And so over the course of two years, you just meet a lot of founders and would-be founders and see there's lots of ways to do it. And I think that was really helpful because you saw the people who'd been coding since six and you saw the people who hadn't been, and you know, you saw different, just like all sorts of people in all sorts of different ways to make it kind of be successful or make it work for them. And so I think it was something over those two years where it was like really helpful. And then, you know, at that moment it was like, oh, okay, maybe if I, you know, work really hard and, you know, do things and get lucky, I could like start a company. And then Vanta itself was not quite 10 years after, but like five or six years after that, like the idea in particular.
Why was Vanta the company that you decided to build? How would you describe what Vanta does?
Yeah. So Vanta, we're a trust management platform. We help companies build out and understand and improve their security, and then go get credit for all that security work with their customers through compliance certifications like SOC 2 or ISO 27001, through security questionnaires, through trust centers. So sort of this dual mandate of helping companies understand and improve their security, and then use that to grow their business.
There's definitely a puzzle part. So you mentioned, but prior to Vanta, I was a product manager at Dropbox. I was working on a new product at the time, Dropbox Paper, and did not know it needed to be compliant or, you know, kind of secure per customer contracts until we were about to launch and the legal team walked over and asked me what the heck I was doing launching and violating these contracts. And so that was sort of the beginning of when I first learned about this world, and I think I was struck that it was clearly really valuable, right? We wanted it, it was the only way to get to, you know, this new product out, we had to get to customers, and yet it was going to take us a year of engineering time to get the product to market. And so here was this like really valuable thing we needed for product market fit, we needed to like make Dropbox successful, but there was like a year of manual work that kind of only the engineers could do between it, and that just felt strange, for lack of another word.
Yeah, I love that. Yeah. You're like, why slow down? We want to bring this product to market. We want to bring something into the world, but we just can't do it because we need to check this box that is really hard to actually check. I love it. So, you know, Itcha, you've been in engineering leadership at a slew of impressive companies, and you joined Vanta about a year ago as the VP of engineering. What compelled you to join the company?
Yeah, two things were like really important for me when I was looking for my next company. One was like a company solving like a really important problem, so like having solid product market fit, and the second was, you know, having solid leadership at the company. So the first one about like the product market fit, like Christina was telling about her Dropbox journey, like I've done the whole get, you know, get SOC 2 at, you know, my previous company when GDPR came out. I was like one of the engineers sitting with a room full of lawyers trying to figure out what to do with my product. I've been there too. It's very painful. I know. Yeah. And when, you know, I heard about Vanta and them, you know, taking a very modern approach, an automated approach to solving this problem space, I'm like, okay, this is a real problem. I know it's painful. I know it takes a lot of work to go solve, and so there's something here that Vanta is on to. So that was important. And second, you know, having worked at different places, I know it's important to have a solid leadership team who are smart, no BS, you know, all on board to roll up their sleeves and solve this problem together. And I saw that at Vanta and in Christina, and so I was like, yes, this is where I want to go next.
Yeah, I love it. So one of the bets that you made as a leadership team was to lean in pretty early into AI. So, Christina, I saw that you launched Vanta AI October 2023. What was the moment that either you or Vanta as an org became interested in incorporating AI and LLMs as part of the product?
Yeah, I think different places, the company kind of earlier, and I think for me, and I'm probably a little bit of a slow adapter, but for me it was probably GPT-3.5's launch, so kind of a year prior, and that was like a model that really I think caught a lot of people's attention, mine included, and you just started to see the possibility. And kind of the way I think about it now is a lot of these models and LLM let you build product experiences that are like really magical and we couldn't build before, and we had the idea to build some of them before, and we tried in some cases, but it like didn't quite work. And I think 3.5 was kind of first time for me personally where you could start to see like, oh, this isn't quite there, but this is so much closer than everything else has been. I know, Itcha, you probably have another different earlier take.
Yeah, actually I'm curious when you took the role, how much of it you thought would be AI-centric.
It's such a funny story. Like, AI was just popping up and everyone was like, is this the next crypto? Like, is this just going to like blow up? And I had some background in ML, like back in undergrad, grad school, my thesis was multi-document summarization using probabilistic mathematical models. That's very relevant. Yes. We're just basically like LLMs are like a step leap from there, you know. And so I was more bullish on like, okay, AI is not like another crypto. There's like more to it. And I know Vanta was an early adopter, which is like a positive sign. I was looking for, you know, a space where we are willing to try new technologies, but it kind of surpassed my expectations, to be honest. The rate of innovation in the space has been quicker than what I expected, and I knew it would be a part of my job, but like I did not really realize how much more part of my job it will be. And honestly, I'm actually glad. It's very fun and exciting to be a part of this wave in the industry.
I think that's true of a lot of engineering orgs as the models get better and as you see like you have real products out there that people are using, and so you invest more team and more time into it.
It would actually be helpful, Christina, could you just walk me through the parts of Vanta today that do use AI? How is it part of the product?
For sure. And you should jump in on stuff I forget. But one of the first places we launched it, kind of to the point about you can now build experiences you wanted to but couldn't in the past, was we launched a questionnaire automation product. So one part of enterprise software often is if you're selling a tool to someone else, going through their security review, and often that security review starts with a long list of questions that might be in a spreadsheet, you know, do you encrypt data at rest, how do you do that, what do you do with data in transit, like times 100 questions. Generally no one really likes those. Yeah, you like it's one of those things where you want to get it right, but it's pretty tedious, and yeah, you don't think there's much room for error. And so folks have tried to answer these questions kind of automatically before, and yeah, like there was the BERT models from Google in like 2020, but nothing was like really that good, where I think users would have to like fight with these models. And so it wasn't clear if it actually saved any time. And so when these things started to get good, this was one of the first obvious places to apply the technology because Vanta knows so much about the company setup and its policies, and you can take the old questionnaires and so go and give a rough draft answer for, you know, someone who can then go through and say, yep, I agree, I disagree, you know, I want to fix that. Anyway, that was the very first product we launched just over a year ago. And we've seen kind of great uptick on that, both from an end user perspective, and it's a nice kind of place to measure accuracy and quality too because you have these discrete question-answer pairs.
Yeah. And it sounds like in that case you actually have someone on the end saying this is a good answer, this is not a good answer.
Exactly. Yeah. And you know, Itcha can talk more to this, but like how she's kind of instrumented that, and we track those metrics and like have that, and you know, kind of think about, well, how do they fluctuate based on different types of users, different types of data, different models, right? Like there's kind of a bunch there. But that was one of the first places. Over time we've just been pulling it into more and more parts of the product. So a lot of compliance or a GRC workflow is kind of literally mapping different pieces of information together. So you have this kind of policy document. It's sort of your, it's sort of like legalese. Here's, you know, how we think about information security or cryptography, and you have those documents, and then you want to make sure you're actually doing what's in the document. So like that legalese has to match your AWS config, has to match the like control you put in front of an auditor, like the rule you say you follow. So much of a GRC person's workflow is in fact making sure all of these different things are in sync all the time. It's really hard. And so we've like started to apply LLM to different pieces of this wider problem so that again we can kind of flag for someone and say, hey, you know, your document says you're doing XYZ, your cloud says you're doing ABC. Is that on purpose? Do you want to change one of these?
Yeah. And this makes a lot of sense because LLMs are, they're good at moving around different structures of data. So you can do that matching much more easily.
Yes. And we're adding it to more parts of our product. So maybe by the time this comes out there'll be another AI application, like from vendor risk management to like syncing policies to our trust center. So everywhere there's data, there's opportunity to make that data more accessible to our end users.
Yeah, I love it. So, you know, I'm curious on the engineering side, security and compliance are very high stakes fields, and false positives or false negatives can be costly. You had mentioned, Christina, the example where you have humans in the questionnaires checking, you know, before you say this is the truth, you have someone going and checking and saying yes, like I'm giving a thumbs up here, and so you have a little bit of a safety net there. But how in general does Vanta ensure that AI-driven automation is reliable in these spaces where it's really so important?
Yeah, I think a big part of this is having a human in the loop in the design, right? So AI is here to assist you and make it easier for you and not necessarily to replace you. So intentionally in all our workflows you will see like here's a suggestion and do you want to accept it or do you want to reject it. Or the other aspect of this is we also make sure there's like transparency in wherever any responses are generated by AI. So if you go to our chatbot, for example, in the trust center, it says powered by Vanta AI, AI can sometimes make mistakes, so check your responses. So the end user is educated where like LLMs or AI is at play. And I think the third part of this is explainability, which is, so for example in the questionnaire automation example, whenever we, you know, generate a response for you, we also give you references, which is like, hey, this is how we generated this answer for you. You know, we based it off of this like knowledge base that you linked or this document that you have.
Yeah, like citations effectively.
Citations. Exactly. So that there's more trust built with the user that like, okay, hallucination is not at play, or like, okay, I can go verify this source and even adjust it when I need. So building trust with the end user is like very important in tools like these by providing them that information.
Do you think there'll always need to be that human in the loop?
That's such a good question. I think in high stakes types of activities, yes. I think if you have lower stake automation activities where the stakes are less, that's where agentic workflows can take more of a place. But I feel like especially in security and compliance, we have a while to go where I think always having a human in the loop will be necessary.
That's super helpful. Tell me a little bit about how you've built up an evaluation system. So you, you know, you mentioned earlier that you're tracking some of these metrics over time. Maybe for the questionnaire example, how many look good? How many get edited and changed? So what is the eval system that you've built, and what are you measuring?
Sure. So we have eval at different phases of building an AI product, right? So there's like the build phase. You know, you have to make sure that you have a really good golden data set or multiple golden data sets, which is basically canonical representation of what kind of like customer data or input you can get. And then we use prompt engineering tools, like, you know, trying out different prompts, evaluating them against these golden data sets. In places we've built LLM as a judge concept, which is basically you use LLMs to evaluate some of your responses to be like, okay, you know, this is working good, or this tweak that you made is not working good, and using that as a signal. And kind of the most interesting part for me, at least, that I'm pretty proud of what we've built, is we've integrated this into the existing engineers' workflows in their CI/CD systems. Like, you know, when I'm coding and I push up code, which, you know, impacts an AI feature, automatically in our CI/CD, LLM as a judge runs evaluations against your like different golden data sets to give the engineers a signal that, hey, your AI quality may be dipping or it may be doing better with this change that you made. So this is all like offline eval or like evaluation done without live customer input.
And let me just ask on the golden data set part. Can you just tell me a little bit about the construction of that golden data set? Because all those downstream things that you mentioned, like you want to do LLM as a judge on your answers, you want to basically do regression testing at CI and know, hey, if I change this prompt, do I break a bunch of evals? But it all kind of hinges on having an actually good golden data set. So how do you do it?
You picked the right question to ask. Building a golden data set is actually a pretty hard problem because, you know, you want to make sure you have representative information and you're constantly keeping it up to date. So we have GRC, which is governance, risk, and compliance SMEs, who are subject matter experts who, you know, are like deep experts in the space of compliance. So they actively work with our engineering team and our product team to build these golden data sets for us, and they're partners along this journey with us. They also help us in things like doing human evaluation sometime to be like, okay, we push this change out in the wild, customers are thumbs down in this, like, okay, this is why, and they, you know, we go tweak that back in the golden data set. So there's like this whole feedback loop happening. So our SMEs are quite honestly the biggest key to helping us build our golden data sets.
And then the SMEs or subject matter experts, you know, is it natural for them to think in this way? They're used to poking holes in pretty challenging security and compliance issues, but of course when you're dealing with AI and thinking about all the corner cases, it's kind of like non-deterministic unit testing. How do you like train up SMEs or get them in the mindset to build the right data sets?
So we actually embed them with our team. So we have like a SME who's like working closely with our AI team and our AI engineers. So they have like pairing sessions that they do together where like the engineer and this SME are sitting together, and this SME is like, okay, this doesn't look right. And the engineer is like, huh, I wonder if I shifted this prompt it would do better. And this, and you know, they kind of run it together. Does this look better to you? You know, SME. And so they work very closely with each other rather than in isolation. And that way, you know, it's kind of cool for the SME, they're learning more about how AI systems work and sometimes can play around with these prompt tools on their own.
Yeah. So you'd say they're excited about it. They're like, we get to be part of the AI features.
Yes. And their job isn't all day just clicking and giving him an email. Yeah. Yeah. So it's a part of their job is helping this team. A part of their job is working in other parts of Vanta's security org.
Yeah. Do you use any external labeling or is it mostly internal SME for creating the data sets?
You know, one thing I am curious about is how much those golden data sets are being updated over time.
Yeah. So, you know, I spoke a lot about our offline eval system, but like our online eval, which is basically based on customer feedback. Yes. Is, you know, there's multiple ways to get customer feedback. Like a lot of people know about a thumbs up, thumbs down, but in real life, not a lot of customers are willing to even give you that extra second to do the thumbs up or thumbs down. It's like a very small percentage. So we're also gathering additional signal, like are they accepting your answer? Are they editing your answer? Are they rejecting your answer? And if they make edits, are they just converting it to like UK English instead of, you know, American English? And we're getting all this like detailed like feedback loop into our system that a combination of like our product managers, our engineers, our SMEs are looking at to take this feedback and be like, okay, where should we represent this change back in the system? And yes, it plays a feedback loop back into like should we go update our golden data set, should we go update prompts, or should we like add more features to give customers more control. So that feedback loop is happening continuously.
