Episode 08 - The Audit Advantage: Ensuring Trial Success with Elizabeth Nelson

Thank you so much for having me. It is an absolute pleasure.

Okay, great. Thank you so much again for having me. So, what I do in clinical research is I am an auditor, a consultant, and a trainer. I like to balance all of those things. I'm the person that comes in and finds your problems that you may or may not be aware of, but really wants to work with you to help address those issues so that we're all moving forward in quality.

Actually, the way I got into clinical research is different than how I got into the quality piece of it. I got into the quality piece, unfortunately, because we were involved in a large global study at my site where I was working, and we ended up having to turn the sponsor and two CROs into the FDA. It was unfortunate. We had a former FDA investigator come in and do our site qualification visit, and so we kept in touch with her. When I contacted her, I said, 'Listen, I just have some questions for you. This is what I'm seeing. This doesn't make sense. This is what we're being told. We're not going to do that. And now we're being pressured to do this.' And she was like, 'I got you.' So, she kind of came in, and we actually went through the Ombudsman program at the FDA. She liked how I wrote and thought, and she took me under her wing. I worked with her for a few years doing consulting work and doing audits. I found that for me, a lot of what we see with audits wasn't because people were trying to do the wrong thing—there are certainly those folks out there—but it was because no one had ever taught them how to do the right thing. They had left them to their devices, like, 'Oh well, you're a doctor, you're a key opinion leader, certainly you know how to do this.' Sponsors would be selecting investigators based on their prescription capabilities and their status, but we're required to select people as experts to study our product. So, I found that when I had folks that were non-compliant, oftentimes it was because they didn't know the right way—'Well, my monitor told me to do it this way'—and they didn't even know what they had signed up for in many cases. That's just evolved for me in terms of, you know, you can audit and you can go in and you can wag your finger at folks, but if we're really trying to get good data from protected participants and support the public's health, then help them. Help them get there. That's how I got into clinical quality, and that's why I stay there.

I did. I started out—and that's also an evolution. When I first started in clinical research, we couldn't go and get a degree in clinical research or clinical quality management or things like that. It didn't exist. So, I actually came from a laboratory background. I had been working at the CDC doing biochemistry and pathology, and before that, I was at the Department of Energy doing some analytical chemistry. I needed to finish my degree—it took me 10 years to get through my undergrad degree. I had one last class because I worked full-time. I had one last class to take, and I was working in a neuroanatomy lab. Then, I found this job. A friend reached out and said, 'Listen, I'm leaving this post. Would you be interested in this? I think you'd be a good fit.' So, I moved over into pediatric cardiac anesthesia, sedation, and pain management research for 10 years. I literally walked in the first day to the NICU, and I thought, 'Okay, what am I doing?' I've always been very methodical in my approach. Fortunately, I worked with a fantastic clinical investigator. He was a tenured professor doing research for all the right reasons, trying to get access to his patient population for novel therapeutics, and he wanted to be part of it. So, I worked with him for 10 years, and I learned an awful lot. It's been an evolution. I love the fact now that folks can go out and get a degree in this. So, yeah, it was very circuitous for me, but it seems to have worked out.

Yes, we had a main CRO, a sub-CRO, and the sponsor. Unfortunately, they were not behaving themselves. They were sending folks in, and they were walking away with source, and they were altering information. They were telling us to alter stuff because they were... that was my kind of baptism, if you will, into clinical quality. I'm like, 'I know this isn't right, but how do I anchor this?' They were trying to get a patent extension, and they had decided they were going to move forward at all costs because we were keeping them from getting that patent extension. And unfortunately, we did keep them from getting that patent extension because the data that they were submitting was not the data we had generated.

The FDA stepped in and did their job. They did. And like I said, my investigator was fantastic. He was like, 'No,' because we were very disappointed—the product actually worked, and we wanted to be able to offer that. But he's like, 'I'm not going to do it this way. We're not going down this path.' So, it was very contentious. He's a great guy. It was very, very contentious. We proffered to them, 'We will have independent people come in and watch what we do so that we can make sure that the data that's getting reported to you is from us, without your monitors coming in literally manipulating information.' They wouldn't go for that. They said it was too expensive. I'm like, 'Okay.' So, their patent clock was ticking. I learned hard lessons about the industry side of it. I went into clinical research because—and at the time, I wasn't a nurse, and everyone said, 'How can you do this? You're not a nurse.' Well, I was a scientist. My boss really liked that, so we approached it that way. But unfortunately, it didn't work out for that particular study. But it also gave me—and my boss was great, because I started working with this former FDA investigator, and he supported that too. He's like, 'Go learn as much as you can, bring it back to us and to this institution.' So, that was 20-some years ago.

That's a great question. It kind of comes in waves. At this point in time for the year—we're coming at the end of September—the training's really picked up because everyone's trying to use their training budget. But it's been a pretty heavy auditing year. So, we do see the industry kind of wax and wane. But I do try to keep a mix of it because I feel if all I did was audit, I think... and a big part of training is having those real-life experiences to be able to share with others to really hit home some of the points. So, I like to keep that mix. Lately, I would say though, I've been doing a lot of consulting. Particularly during the pandemic, a lot of remote audits, but SOP gap analysis, training gap analysis, things like that. So, it's all those pieces that come together. Some years are more training-heavy, others are more audit-heavy, but I do try to balance those out.

That's one example where sponsors will contact me. I've been really fortunate over the time that I've had my own business that it's word of mouth, and folks will get in touch. So, either a sponsor contacts me, I've had IRBs contact me, clinical investigator sites contact me, and CROs too. I've had CROs contact me to audit their sponsor because they're having trouble with the sponsor. So, it's just making sure that you can have that independent eye on what's happening and understand—we're not attached to any one entity as auditors. Wherever we can help and bridge that quality gap, I'm more than happy to do so. So, all-comers, so to speak, we will get contacted and do the best we can to help support them.

That's a great question, and I appreciate that because I think sometimes we don't stop and ask ourselves that. We just go on autopilot, and we do it because it's what we do, and we can do it with our eyes closed. Why I do it, when I sit and think about it, is because this is how I feel I use my public health degree. Certainly, as I get older, I become more of a consumer of these therapies. But I want to make sure that these products are safe and effective. That's what we're doing. And so, if we can put a product out there—it's very gratifying to see where even if you see something advertised, as a nurse now—I went to nursing school during the pandemic—I see where we're prescribing things for folks, and I'm like, 'Hey, I was part of that, and it's working for you and it's helping you.' So, that's really gratifying. I like to make sure that if things are being done—which we know they are, clinical research isn't going to stop—that I have some part in making sure they're done properly. And because I have those 20 years of experience, and some of it has been trial by fire, I want to share that with others. 'Hey, did you look here? Did you think about this?' Just this morning, I was working with a group and they were talking about, 'Hey, we have a Phase 1 study, and we're not required to register on ClinicalTrials.gov, so we might not.' I'm like, 'Yeah, but your Phase 1 study is for people who are affected by the disease. You're enrolling them in your study, and ClinicalTrials.gov is a patient-facing portal, so you're doing them a service when you post that. It's not like you're just gathering up a bunch of undergrad students and having them be in your Phase 1.' So, really thinking about those experiences, bringing it in, and trying to share it with other people. They can take it for whatever they want, but I just feel it would be a waste if I had this information tucked away in my head and I didn't inflict it on others.

Yes. I think that anytime the stakes are really high in what we do, to have that independent oversight—even as auditors too, you get in a groove and you just think, 'Well, this is the way I've done it, this is the way I've always done it, I'm just going to keep doing it.' But things change. People change, the regulations change, the guidances change, and just the environment—the industry has changed significantly since I started out in 1997. So, I think that independent quality assessment—we're the assurers of quality, if you will—it's really important because you do need to have those outside eyes saying, 'Did you think about this? Did you see this?' You might think things are working because it's what you've always done or everyone on your team is in agreement, but it's not always a pleasant place to be. Not everyone loves an auditor, but if we can come in and just get you to think differently about things—there is no requirement for clinical quality assurance in the regulations, but there is a requirement that you follow those regulations. So, when we can help you interpret those and maybe fit those into your management systems and style better, we've done our job.

Right. So, 86% of all participants across five studies had the same blood pressure and mean arterial pressure. The equipment was new, but the person who was operating it didn't know how to use it, and they were embarrassed to tell their investigator. So, they Googled what was a good blood pressure and put that in for everyone because they thought, 'These aren't sick people; they just have mental illness, so that's a good blood pressure.' But I did see variation when someone other than that individual was the one taking blood pressures, and so that's how I knew something was up. I mean, you just know something's up when almost everybody has the same blood pressure. That's something we have the opportunity to do as auditors—we look at patterns. That's why central monitoring, risk-based monitoring, data management is looking for that too. We're just that extra layer of, 'Hey, this doesn't make sense to me,' where your monitors are going in and they're looking at blood pressure for Visit 1, Visit 9, what have you—'Hey, the blood pressure is there, it's good.' We're pulling back and looking at things a little bit differently.

Right. And if you remember, he had already done all the work for me because he had his own source document worksheets that had the vitals for every visit. It was all sitting right there. It was just having those different eyes looking at it, and that's what we do.

Sure. That is frustrating. My soft spots revolve around vulnerable populations. I'm going to do my job for anyone, but particularly when you have the elderly, those who maybe have cognitive impairment, children, socioeconomically or educationally disadvantaged—that's just not a population you want to be messing with. So, I find that yes, I can be upset about that. My job is to protect people. That's the way I look at it. I look at myself as a steward of that participant's health, well-being, rights, and safety. It does get frustrating, and you have people who are like, 'What's the deal? Not a problem,' or 'I'm just going to do it this way because my monitor doesn't have a problem with it. I don't know who you think you are coming in and telling me this.' What I know about myself is that as upset as I might be and the injustice I feel and the way I want to advocate for these people and protect them, my passion is going to impair that. So, I need to step back and make sure—whether it's taking a walk in the parking lot or talking myself down—is that the information I gather is factual and objective. And I need to put that into a report in a way so that others can pick that up and be like, 'That doesn't sound right. That's not cool at all.' Because if I say, 'This person is belligerent,' my opinions and my emotions stand in the way of them being able to say, 'That's just that auditor's opinion.' It is our opinion when we write an audit report, but we bring that opinion because we have factual, objective evidence to support it. And I think I told you about going into that closeout meeting one time where someone said, 'Oh, be prepared for a fight,' and I'm like, 'There is no fighting in auditing. You cannot fight what I find.' What I find stands on its own, and that's what all of us auditors are trying to do. I think if we allow our emotions to come into the mix, then we do a disservice to the people that we're trying to protect.

I've been auditing 22 years. I've been involved in clinical research for 27 years.

Right. Exactly. I do. That's been really refreshing. Some people are kind of disingenuous about it, but for the most part, when you walk in and start an audit and do your opening meeting, people are like, 'We're so glad you're here. We want to do a good job, and it's good to have your eyes. Tell us this and that.' That's great. Now, going back to your question about sponsors—some of them are right. Some of them are really kind of clueless. We've got all these great startups, we've got all these different kinds of products out there, and they have a really good lab background. But helping them transition into the clinical—they're like, 'Oh, I had a friend that used to work for this big company 20 years ago, and they did it this way.' I'm like, 'Yeah, they also got fined two billion dollars, so they don't do that anymore.' There's a reason for that. So, let's bring it back to the here and now. We have had companies that have been fined a lot because of what they were doing with clinical research, and that put some people back in line. But I do see that there are a lot of these companies that are trying to step out on the right foot, simply because of their fund structure and the fact that their company is built around one product, and they want to make sure they get it right. So, certainly, that ethical, happy part of me is like, 'Yeah, we all want to do the right thing,' but sometimes doing the right thing comes because, 'Hey, we only have one chance to do this, and we can't afford to get it wrong.' So, whatever it takes—I don't care. As long as you're willing to be compliant, I don't care what leads you there. I'll take it.

I think one thing that comes to mind—and it's hard to really replicate this—but I always like that light bulb moment where people are like, 'Oh no, I didn't know that was my responsibility.' I was doing some remediation training for an investigator. He's very well-known in his field, very well-known internationally. We were doing some remediation training because he had had some issues with his clinical research. I asked him, 'Okay, this happened. Tell me about it. What happened?' For him, he was very grateful because I wasn't coming in and saying, 'I know you messed up, and we're going to do this.' I'm like, 'I want to understand why we got here. What do you see?' There were some gaps in his understanding of what was required of him as a clinical investigator versus a clinician. So, we talked about that. He was rationalizing why he did what he did. Then, when I started providing him with warning letters and NOPO letters, and talking to him about, 'This is the path that they went down. Do you see any similarities here?'—the look on his face—we were doing it over Zoom, and he just looked up, he's like, 'I'm going to jail.' I'm like, 'Well, that's why we're here, right? So, let's not get there.' So, that's always refreshing. I check in on him. I make sure it's not just a one-and-done thing. 'How's it going?' And I check in with the people that contracted me. They're like, 'He's really towing the line.' So, that's really gratifying when we see that change, because we're always looking for executive management being engaged. That's good.

There's also that satisfaction of getting a product approved and seeing it in use. I volunteer here locally, and I was up at the hospital last weekend or the weekend before. I saw that some of the things that were being used—I'm like, 'Oh my gosh, they did it right.' Something that I had worked on was now being used, and that's really... it is. That was something I had done a mock inspection for because they knew they were going to have an FDA inspection. It went well—the mock inspection—we got our stuff done. They said the inspection went well. You go off and do other things. I just happened to see this product on the cart for this company, and I was like, 'Good, good for them.' And I knew that it was being used to help these neonates. I was really, really grateful for that. So, there's a lot of reward in what we do. There's certainly, like I said, grumpy folks sometimes, but if you think about, 'Why am I really doing this?' that will help you get through those bumps.

I haven't audited ethics committees as they're termed, but IRBs I have. Yes. And we also talked about how I think that's something sponsors overlook too much because there's the expectation that your IRB is compliant. And if your IRB is not compliant, you as a sponsor are not compliant, your investigator is not compliant. And as we also talked about, it's the easiest audit to do remotely because they are mandated to have certain procedures. You ask for those procedures and see if they're following them. So, IRBs absolutely can be inspected. As a matter of fact, we talked about that the FDA knows that if there's a local IRB and they're investigating an investigator, they can say, 'Oh, if that IRB hasn't been inspected in the last five years, we're going to add them to our list.' So, I think that's one of the most overlooked pieces. I was just having a conversation yesterday with a group about this—that IRBs do all sorts of nonsense just like the rest of us, and we have this magical bubble around them that, 'Oh well, the IRB approved it, everything's good.' So, we have a system of checks and balances in clinical research. We, as auditors, can check all of y'all and see how you're doing, and you should be checking on each other as well. So, I get sent out by IRBs, and I get sent to IRBs.

Well, they should—the monitor should know enough about what's expected. The investigator should know enough about what's inspected from these IRBs. And you have telltale signs, right? Like you said, they have the wrong thing stamped. Or I've seen IRBs approve a consent with another IRB's contact information that gets used and filed away. So, in my experience, going to local IRBs—now, certainly I've done IRB audits for vendors. Sponsors will send you out to go to your vendor IRBs, your centrals. But for the local IRBs, it's because something has been surfaced during a clinical investigator review or information they've been providing. Historically, the information sheets that the FDA has out for IRBs and investigators has put it as the responsibility of investigators—like if we're going to a local IRB, we go through the investigator to check that IRB. But now it's kind of everyone's available. Those are like from 1990, 1991. When I've gone into a local IRB, we have asked the investigator to introduce us—'Hey, we're coming.' But honestly, if you're selecting a site and they're using a local IRB, they're your vendor too. Your site, your investigator, your IRB—they're your vendors. So, we should make sure as sponsors that we're making sure they're up to the task. We sign the back of the 1571 as sponsors, ensuring the IRB is compliant. The investigator signs it on their behalf, that the IRB is compliant. We better make sure of it.

Oh, sure. They can ask, 'Have you been inspected? What was the outcome? And is there any attendant documentation?' They can ask for written procedures, table of contents. They can ask for specific written procedures. They can ask for clarity in approval letters that are getting more and more vague—'What did you actually review and approve? This is what was submitted, but is there documentation that you reviewed it?' Now, sponsors from a business risk perspective will typically not take an IRB approval letter and say, 'No, we'd like you to do it over, we'd like you to clarify it,' because that's going to hold us up in terms of enrollment. But you spend a lot of time on the back end if you can't prove that you had IRB approval for your study to move forward. So, it's just better to have those systems in place up front. Check those things, make sure everyone's on board. Work with other entities that understand the importance of it, that aren't looking at you as being difficult or just asking for things that aren't relevant, because it is relevant. Someone's going to ask you for it eventually. And like you said, it's better that it's us than the FDA.

But if you can start out proactively and having these things in place—like we talked about the Federalwide Assurance number and how that's not relevant for us in industry-sponsored research—understanding and knowing those things, and to your point, everyone trying to do the right thing on the front end is going to make things a lot easier for us and smoother for the study.

So much easier. It's so much easier to do it right from the beginning than to go back and fix things. And I think sometimes when these sponsors and CROs make these decisions, they don't think about there's several different types of impact—so not only time to fix things, but the financial impact to fix things, the risk you're putting your study at. Think of all the millions of dollars you've spent on it just to put it at risk because of a silly decision.

That's right. Absolutely. That's my mantra there. And I think too, if we can step forward in quality, it's easier to keep it going. It's pretty clear in terms of what the expectations are from the FDA that they don't expect us to kind of tidy things up just because they're coming. They expect us to be functioning in quality and inspection-ready at all times.

Right. And a lot of folks bank on the fact that they don't have the resources to do that—to just show up. Oh, surprise visits. But I think too, they're going to ask, 'How do you do this? How do you follow what's required of you? How do you make sure you do it in a uniform, compliant manner?' And the proof's in the pudding. At the end of it, when everything comes out and you're having to scrub your data and clean and try to storyboard your way out of trouble, you know that there's definitely a problem. And you get to see it all from a Trial Master File perspective, right? Because everything filters down to you.

So, I know sometimes it's worrisome when I see things that get to me and think about how many steps it had to go through to get exactly to me and no one noticed it. That's one of the reasons, honestly, I now specialize in working with smaller sponsors, and I really like that because I do feel like there's a lot more attention to detail. Because as you know, when you get into those larger pharmaceutical companies, those larger CROs, they're just so busy, and there's so many people and so many pieces that it's so easy to lose track of things and documents and such and to keep things happening in a uniform fashion.

It's a little bit of everything, you know, wherever I'm needed. Standard training, like we did the course together—standard GCP training, investigator meeting training. Going in and training IRB staff, going in and training CROs and sponsors, training whole departments. So, yeah, go to academic medical centers, really wherever the need is. Whenever someone—I was just reaching out to someone I know that they are working on starting launching clinical trials with—she's part of an association, a disease association. But I said, you know, I'm happy to do pro bono work. I know that some of these folks are going to be new to this, but I would love to lay down because this is near and dear to me—not only is this my friend, but this disease is near and dear to me. So, I'm happy to help anybody that doesn't have access to this, because most institutions have access to CITI training, and they can do that, which is a great starting point, but not all of them do. So, where do you go and find this? And what we find going to the training piece is that people will reach out and look at the free online stuff, which is great, it's great that it's there, like the NIH is nigh on to it, but it may not be fit for purpose for the type of product you're studying or the regulations you need to be compliant with. So, it's good if we can customize that out of the gate and look at what are your strengths, what are the areas that could use some strengthening. We're required as sponsors to provide investigators with all the information they need to properly conduct that study. Training is a big part of it. I just had a client this morning say, 'Well, if they can give us a certificate, do we need to train them on GCP?' Unless you know the content of that training and you can review it, then yes, you do need to train them, because you're relying on someone else telling them what they think is going to be important for your study and your population. Training—it's interesting. It's the knee-jerk reaction whenever we have a CAPA, but it's also, starting out, the most overlooked piece of it.

Right. And not only that, but when they say training—are they following through on it? Did they do the training? Did they follow through on it? And then did it actually fix the problem? Exactly. It's all around that, and I've seen that myself. 'Oh, we'll just train them.' Okay, but was that really the root of the issue? Or is the process not working right? And what did you train them on? And how did you train them? And who did the training? Do they even know? You can train the right people on the wrong thing, the wrong people on the right thing. You really need to understand issues. Again, if we've given people all the information they need—meaning we've tailored the training to that disease state, to that therapeutic area, to that investigational product type, to their skill set—then we level-set. So that when something happens, we can say, 'Oh, then apparently it wasn't the training, it was maybe something else.' But like I said, we immediately go back to, 'Oh, we'll just retrain them.' Well, you could—for some people, the spaghetti never sticks to the wall. So, you can go ahead and keep doing all the training you want, but it's not going to work. So, you really need to understand why.

Right. And how do we usually present training? It's read and acknowledge, which tells me you know how to read for the most part, but not that you understand it and how to apply it and implement it. So, we really could bridge that gap in our industry. But I think we think of training—like I said, if you're willing to accept somebody else's training certificate, you're looking at training as an item to be filed away in the Trial Master File, but not really assessed for is this the information you need?

Right, right. Because your job is to do the work, and your job is to make sure that you can show that they've done the work. But we don't always dig deep with that. Just recently, I had a group who wanted to—because it was free, they were a sponsor, but because it was free, they wanted to use this free training module. I'm like, 'Okay, but that's all investigator responsibilities. It's focused for investigators.' And they said, 'Well, that's fine, we'll get a certificate.' And I said, 'Yes, you will get a certificate that says that you've been trained on investigator responsibilities.' They're like, 'But then we'll write a note to file that we read 21 CFR 312, and so we understand sponsor responsibilities as well.' I'm like, 'Oh my God, okay. I caution you on that.'

No, no. It depends on how we're hired to do. I can't, right? Because as an auditor, I can't have my fingers in the pie. But I do, and my husband's like, 'I can't imagine why people would hire you,' because I do tell folks, 'Listen, I expect you to do the right thing.' And if not, 'I will do the right thing for you because that's my job.'

Right. I always ask folks, 'Are you looking for a short-term or a long-term return on this investment?' Because if you put the effort in up front, it's going to carry you forward. But like you were talking about, trying to clean up everything at the end, you put all those resources there—no guarantee it's going to work out for you.

Right. Like we talked about, I'd rather take away your matches than put out your fires. Let me build that quality up front. And I appreciate sponsors and CROs that are truly trying. That doesn't mean it's going to be perfect, and you aren't going to run across things. Like you said, it's not like they're purposefully not trying to do things right; it's just they may not realize it. But I feel like some of them are starting to get that, or at least key people are, and then when they're in the right place within an organization, they can have that impact. Exactly. And you know, as long as you can justify why you did what you did—as long as it gets you to compliance, even if it's not the most efficient way or the way that we're used to seeing—if it works, it works. And that's really what it comes down to. You have to be able to—when you're standing in judgment from a regulatory agency—'I did it this way because I thought this was the way I should do it, and I think it worked.' So, I feel confident with this. And that's all we can hope for. We're not infallible. I always say it's that perfect data that makes me twitch. I'm glad when I see, 'Oh, we messed this up, but this is how we got ourselves back to where we needed to be.' That's all you can hope for.

Oh, sure. If there are any questions, let me know. You can call me or text me. My number is 304. So, Suzan, I want to thank you so much. It was a pleasure having you in class because you had such great insight and questions, and I really appreciate that. And this is amazing to me that you do a podcast. As you know, I have a hard time just downloading the app, so this is incredible. And what a great way to share—not only my experience, which I'm grateful for you inviting me, but others' experiences. We need as much handholding and help as we can. It's a community; we're all working towards quality. It's good to be like, 'Hey, have you thought of this?' or 'Don't go that way.' So, this is a great service you're doing too. So, thank you so much.

Alright, you take care. Have a great weekend.