Senior Vice President & Chief Accounting Officer, COMMUNITY HEALTH SYSTEMS INC
AI PHI Affinity Group - Presentation by Jason Johnson and Renato Umeton September 2024
Topic Unlocking the Potential of Large Language Models in Healthcare: A Case Study from Dana-Farber Cancer Institute In this ...
About Jason Johnson
Jason Johnson, Senior Vice President and Chief Accounting Officer at Community Health, presented in September 2024 on the use of large language models in healthcare, describing a case study from Dana-Farber Cancer Institute. He stated that the institute prohibited direct clinical use of its AI tool outside of controlled studies, citing concerns about regulatory maturity and patient safety. Johnson noted that the institute worked with Microsoft to deploy a HIPAA-compliant model inside its firewall and that after one year of use, ethics and compliance concerns had decreased, with inaccurate output remaining the primary issue. Johnson has also hosted a podcast on water crises in Black communities, discussing the financial burden of bottled water, the difficulty small towns face in accessing federal grant money, and the impact of the Supreme Court's decision on the Clean Water Act. In a 2020 video message as a school superintendent, he addressed challenges related to COVID-19 social distancing requirements and budget constraints, stating that the school district was at high risk of insolvency due to low funding. In a 2011 TEDx talk, he discussed the decline in book reading and proposed integrating social features into digital books to create a more interactive reading experience.
Transcript (13 segments)
J
Jason Johnson0:00
It's a great pleasure to be with you today, in Hawaii or wherever you are. This is the closest I'll get for a long time, I would imagine, to being there. It's good to be able to share our experience and thinking about large language models and generative AI. Over the last couple years, we spent time thinking about how to do this well and how to get this used by our community. We'll talk a little bit about that and some of the challenges, and there'll be plenty of time for questions at the end, so save them or interrupt us as you want. Either one is fine. I'll say a little bit about Dana-Farber Cancer Institute before I start. It's a national comprehensive cancer center in Boston. We are a Harvard Medical School teaching hospital. We have about 50% research and 50% clinical care. It's one of very few places that has this kind of balance. So a lot of what we're trying to do is bring the best research into practice and the best data from clinical practice to our researchers. My team is informatics analytics. Our department sits in between and we help enable research, but we also help enable clinical and operational work as well.
I will jump in. Let's see, some disclosures. The most relevant of these is I have a few thousand dollars of stock in Microsoft, and I'm going to talk about Microsoft. I just wanted to say that Dana-Farber and the Cancer Center are the two farthest apart cancer centers in the nation. We couldn't be farther apart, only in distance. I hope we hope to converge. Yes, there is a direct flight to Honolulu. I think it's like 12 hours long. That's right. I have not taken it yet, but someday I would like to. Okay, so we'll talk through some of the background and history of the project. This is really a deployment project. We thought about the decisions we were making, a few of the decisions that we used to accelerate the process. I'll talk about three aspects of the project that I think were particularly important along the way, and then Ronaldo will take over and talk about how our users are using the tool that we've put in place and where we're headed in the future. There will be plenty of time for questions.
Starting back a couple years ago, a little less than a couple years ago, when ChatGPT was released, I think it opened lots of people's eyes to the potential that AI would have, that large language models in particular would have in every industry, but certainly true for cancer research and clinical care and patient experience and operations. I don't think it is hype. I mean, there is a lot of hype, but I think it's largely going to bear out that this is going to be a transformative technology for all these areas of healthcare, for healthcare in particular as an industry that has so much inefficiency in it and so much opportunity. I think for AI to help remove work that isn't directly providing value to patients, there's a huge opportunity here. You all are aware of that. That's partly why you're here. In this series, there's enormous potential. Oncology, as you probably also already know, is particularly dependent on things like text and images that are suitable for generative AI. We take a lot of our information that matters the most—diagnosis, treatments, outcomes—to how we treat disease. I'm not a clinician, so I'm not treating disease, but how we as an institution treat disease and how we store the information and then how we abstract it back out for research, it really is about text. We look in images to understand if a cancer has progressed, and the results of that—whether a cancer has progressed or not, or stayed the same, or regressed, or is cured—is sort of stuck in text in the medical record or in the imaging report or in the pathology report. Then we pull it out again to use for research purposes, for regulatory reporting, or reimbursement. So we put it in the text and then we take it back out again. This is going to change, and large language models are particularly useful for text and images. I think these combined tools will be particularly interesting to use. So there's a lot of potential in oncology specifically for these kinds of tools to help. There's also a great risk that's different than other kinds of AI. Generative AI doesn't have the nice clean properties that you can understand where it comes from and that it gives you consistent answers. It has the benefit of creativity, but creativity isn't always what you want. There's also in the public tools, it's unclear where the models come from and whether they're consuming copyrighted information or other things in order to build them. What text you get out may have come from anywhere, so it's a little bit hard to use for clinical care, for example. We are concerned about our clinicians using ChatGPT or other public tools and unknowingly violating HIPAA. We don't really want them to break the law. We also don't want people putting in their research discoveries, rewriting their reports in some way, and accidentally disclosing that information. Putting information into a public tool may equate to disclosure of intellectual property, so you may be publishing your information when you don't intend to by using these tools. We also are a highly regulated space. Grants are regulated, journals have rules, and we don't want our research community to violate those rules, especially without understanding that they're doing that. So we saw significant risk and significant benefit from generative AI in the beginning of the year of 2023. That began our journey, recognizing that this was potentially transformative and also kind of a problem.
We looked at different options. We could ban large language models, which was our first thought. Maybe we should just put out a ban that we shouldn't do this. But as an academic medical center, innovation is our brand. How would we do that? We felt like people would probably ignore it anyway. Off the record, I guess this is recorded, but it's hard to tell researchers what they can't do. It's better to give them something that they can do and steer them towards that than to try to tell them what they can't do. We thought about building a private large language model either ourselves or with a partner, but that seemed to be very expensive and likely to be slow, and we'd be behind the curve. We also explored using open source options, and many of those are quite promising. We were worried about having to do too much fine-tuning and having to do that fine-tuning iteratively over time. It might take us a whole team to set up these models, and the cost of maintaining that would be relatively large. We also might not get the same versatility and value and up-to-dateness as if we could just somehow use the tools that were being put out in the public domain. So what we wanted to do, the question that we asked, was: can we deploy a commercial large language model inside our firewall in a private and secure way that would be HIPAA compliant, that we could make available to our entire workforce, and hopefully affordable as well for broad use? We have about 6,000 employees at Dana-Farber. They could use it for PHI, PII, and intellectual property, or those kinds of use cases that were more sensitive. Would that be possible? So we asked this question, and Ronaldo took the question and ran with it really. The result is what we will show you today. The idea was quite simple: can we take GPT-4 and make it available inside with some warnings and welcomes? This is our policy on the front, but it's basically as simple and easy to use and as fast as ChatGPT itself, ideally. Originally we called it GPT for PHI because that was the concept, or the most sensitive aspect of the study was: can we do this in a way that would be HIPAA compliant? That was the main question.
We first looked at feasibility. We had a business associates agreement with Microsoft that covers our cloud infrastructure and online services, including the Azure environment that we used. The tools within that environment were HIPAA compliant, or at least some of them were HIPAA compliant and approved. We could indicate which ones were and which ones weren't. We worked with Microsoft, and they happened to have this new partnership with OpenAI to make sure that GPT-3, 3.5, 3.5 Turbo, and 4 were inside and available in that Azure environment as services that would be HIPAA approved. With that in place, that partnership with Microsoft, in a sense, we were able to imagine at least that this would be theoretically secure, HIPAA compliant, and covered in our BAA with Microsoft the same way that we use other tools from Microsoft like Outlook and OneDrive without worrying about the privacy. This would run inside our Azure data centers, isolated from OpenAI, which we wanted to be the case. We didn't want our text and content to be shared with OpenAI or built into any new models or accidentally shared with other companies or institutions. So that was the idea. We also looked at what engineering and expertise would be required to set up the platform. Luckily, Ronaldo's team had that expertise and could configure the toolkit that was provided to meet the requirements that we imagined we had. We didn't know exactly what we needed to do, but we had the expertise to do it. Then we made some estimates about how much time it would take, how many people it would cost, and what it would cost to run it. They seemed pretty reasonable, and those estimates have held true over time. I'm happy to report that they weren't—if anything, we're running a little cheaper than we thought it would cost, so that's good. I'll talk about three decisions that we made that helped accelerate the process. Maybe I'll start with the middle one, which is the choice of a commercial partner because I already mentioned this one. Working with Microsoft helped us to accelerate because of this mature environment. We basically already had a fairly mature data environment with computational tools and security there. It's part of our partnership with Mass General Brigham; we share some IT infrastructure, and they deserve credit for this environment. Microsoft's partnership with OpenAI was particularly convenient in this regard. We could have been with other partners, but it happened to be convenient that they had this partnership. We had the environment. We do have environments set up for Google and other partners as well, but this was particularly helpful in this regard. We were able to configure a ChatGPT-like tool using the ChatGPT LLMs underneath, but meeting our own requirements and specifications and configuring it ourselves. Microsoft was very helpful in making their technical teams available for debugging and review. They understood in a way that we've rarely seen, the constraints of healthcare clients. They got what we were trying to do and why, and they understood HIPAA things like that. They were pretty flexible and responsive and supportive as they could be with quota requests too. They had many companies and institutions that wanted to be using these tools, and it was great that we got a little bit. Over time, as we went through the project, we were able to add more and more computational power to the project. We couldn't have started deploying it to 6,000 people on day one; they didn't have the capability, and neither did we. So it was helpful to build the relationship with them over time. This was an accelerating choice of partner. We didn't give away anything to them either. We pay them for the services, and they don't get access to our data or anything else. So it was a good partnership.
The number one decision I would say that we made is to start with governance. I have this slide which I probably should have put in, which has governance at the bottom of the pyramid. It's really the basis that you build everything on when you're doing an IT project or an informatics project or any piece of work. What is the governance of it? How do you set this up? I was a sponsor of this idea, and we also had support from our CIO and our general counsel that we wanted to explore this. There was significant need to do this. There were a lot of people that wanted to use these public tools. We were worried about the risk, so there was quite a bit of interest from the community in having something. We knew we wanted to do something. We as sponsors recommended that we create an AI governance committee, which was called a Generative AI Governance Committee at the time. That's still the major emphasis because of the difference between generative AI and other AI. We don't feel the need to have a special committee to regulate yet AI tools that have very clear inputs and very clear outputs and operate in the same way. They still need regulation, but it's very similar to other kinds of tools, and we have experience and other governance processes to deal with them. Whereas generative AI brought in some new challenges and twists into this, as you're aware. So we were focused on this, and we wanted there to be a broad representation of departments and functions, from clinical to operational to business areas to patients and providers, so that we could have lots of input into the problem and the opportunities and to help us think through the risks and benefits and compliance and ethics. This was about a year and a half ago that we created this committee to look into this. The first function of the committee was education for ourselves, largely about the legal, technical, and privacy issues, all the different kinds of risks and how they were being dealt with by other institutions. We did a lot of literature reviews and searches. We had presentations on different kinds of risk and what they meant and how relevant they might be to our environment. That was the first little while, first few weeks anyway, it was largely about that. The committee decided to kick off two work streams in parallel, partly because of the urgency of this project. We knew we really wanted to get something into the hands of people because they were going to start using or already starting to tinker with public tools, and we wanted to head that off by giving something else. So we gave Ronaldo the job of coming up with a secure technical implementation that would have content filters to protect our users from copyright or from violent language or other things that we didn't really want to get into and didn't want to have happen, and with full logging capabilities and auditing capabilities. Those were part of this configuration model. At the same time, we started working on the policy for usage: what would we allow, what would we not allow, how would we communicate with our community? I think pretty early on, we sent out a communication reminding people not to paste in parts of medical record into a public LLM, that that would be bad. That was some of our early communications. But how would we communicate? The hope was that we'd have a technical implementation and a policy ready more or less at the same time, so that in this way we could save some time. There was lots of discussion, lots and lots of discussion.
The third decision that we made was the one that really helped us move out of the discussion phase and into the deployment phase. That was prohibiting clinical use. We decided that we didn't want direct clinical care to be part of what the tool was used for, or direct patient use of the tools outside of controlled studies. By controlled studies, we either mean an IRB-approved clinical trial, which is a perfectly appropriate way to start to understand the risks and benefits of a new technology, and we wanted to encourage that. But we didn't want sort of random clinical use in a variety of different ways. We also wanted to support dedicated pilots that would be done with the IT infrastructure, so something like ambient AI for clinical notes, drafting clinical notes or something, that was also conceived of as supported by what we would try to do, but we do it in a very controlled institute way with the IT process. So when we set aside the clinical use cases, we set aside a lot of high-value use cases. It was a bit of a sad moment because we were setting aside things that we knew would help patients, we knew would help clinicians. But we felt like we would get there eventually, and we didn't feel like our decision-making frameworks were mature enough to be able to navigate these use cases, provide guidelines to our clinicians and patients how to use the tools. We didn't feel like our regulatory infrastructure was mature enough to guide use of these things, and there wasn't really much evidence that the general trained large language models trained on the internet as a whole would really adapt that well to broad sets of clinical oncology use cases. Some things would work pretty well, and some things wouldn't. A lot of it would depend on how good your prompt was or how you iterated, and it was really hard to predict what would work and what wouldn't. Some things work great, and some things didn't. How would you know if something was going to work or not? If you were translating something into a language you didn't understand, how would you know if it was correct or not? What are the risks of that? So we set aside these use cases that we knew would have a lot of value, but we were thinking that we just didn't have the decision-making framework mature enough. The regulatory process had a lot of work to do. There was going to be a lot of work for each of these use cases to really ensure a productive integration into the clinical workflow: how we would protect patient dignity, how they would consent, what would the consent say, would they have to consent, how would the autonomy of patients work, how would we limit and prevent bias, how would we mitigate it, how would we prevent drift, how would we give guidance to our clinicians for each of the use cases? There's just a ton of work to do for any particular use case, let alone the kind of open-ended thing. So this was a fairly big decision, but it allowed us to accelerate quite rapidly towards the other kinds of use cases that our community would benefit from: operational, research, and administrative use cases. It wasn't that long after that decision that we were able to release a version of the tool because we had done those in parallel, working on the technical with a guess at what the policy would be, or at least an idea that we'd have a policy ready. We were able to put it out and have it be used. This is about a year ago to have a generally available tool. We didn't market it broadly because we didn't really have the compute quota to support the whole community yet, and we wanted to get feedback and do this iteratively by releasing it to different departments and understanding the value proposition and what kinds of support they needed, what kinds of training they might need, what kinds of policy clarifications might be needed. But that's what the first release looked like.
The first six months of the project began with conversations and the idea phase, then the feasibility phase in month two. Our first meeting of the committee was in the second month, where we imagined what it might look like and talked about the pros and cons. This is where we also had the feasibility assessment in month two and started thinking about technical requirements and security and quotas, and began meeting with Microsoft to explore that. In month three, we had a working prototype, and we started thinking more deeply about how we would communicate the policy. That's where we got sort of stuck thinking about well, what could you do, what you can't do, how are we going to manage this? That led to clinical care being ruled out. If we wanted to get this deployed quickly, we were going to have to set aside clinical care. That led us to the initial release and some support for the community that went with it. We had a very small team working on this, so a few people working on it. There was an ability to get out and talk to every user about it and educate them, so we had to have materials that would help us do that. Over time, we released it to more and more groups and continued to work on the policy towards a formal final policy. We called this alpha or beta testing or something. We had different people using it and testing it, and anybody that was super anxious to be using it was using it at this point. In month five, we had an institute-wide communication and guidance sent out about using large language models and some guidance on it. Then in month six, we were able to add more users. We were always sort of balanced between what the compute infrastructure could support and releasing it, so we gradually worked to add more users and have them have a good experience when they were in the tool, and kept iterating and learning and adding improvements. Then we had terms of use language approved by the governance committee by the end of month six. It was a sort of six-month project to get there. We had a lot more work to do at that point though. After that, we continued to have monthly meetings of the governance committee, and we did not have a final policy. We had sort of guidance documents but not a final policy, and we continued working on that over time, adding training. We had weekly technical sprints still trying to work out some of the bugs and kinks. We did a survey that Ronaldo will show you of our early users: what was their experience, what were they worried about? We'll show you that soon. We gradually released some more users while monitoring the performance, but anybody that asked for it, we gave them access. At this point in week 13, we had an institute-wide communication with a link to the tool. Before that, we had released it department by department, and we'd said don't use the public LLMs, but we hadn't really sent out the link to the new tool. Here we did, and we then had thousands of queries per week where we had hundreds before. We started then monitoring the load and the usage, and eventually got to the final policy approved.
I'll talk through a few other aspects of the project that I found important. The first one is an ethical framework or thinking about how you do this in a process framework. A few of our colleagues had published this paper a year before, about how you would ethically deploy AI in oncology, in JCO in 2022. This was about a year before, and it was focused largely on the clinical aspects of AI, but the framework was also pretty useful to us. So we took the major aspects of that and tried to be guided by this. We also included some of the authors who were part of our steering committee as well. They included making sure that the decision-making process, the reasons, the materials used to make decisions were relevant and important, that we kept a transparent process along the way, that we would iterate in a transparent way and revise based on feedback that we got, that we would include the people in the process that were affected by the process including patients and clinicians, and that we would have clear policies and ways of auditing and checking that those were being followed. Part of our configuration was to make sure that we had these in place as a way to check the environment. The second aspect was how we did training. We embedded support materials inside the tool so people could learn what generative AI was in the tool. Many of our users didn't really understand how it worked or what it was or what it could do, what it couldn't do, what were the risks, what did the words mean. We could also give them prompt engineering strategies and technical training and best practices through these tools that were embedded because we knew we wouldn't be able to get to all users this way. Because we're an academic center and learning and exploring was really a big part of this project, we wanted to get the tool in the hands of people to learn and to try use cases that we weren't going to think of, and to test things out and share with each other what was working and what wasn't working. This Teams channel is another example of this kind of socialization by crowdsource socialization: 'Today I used AI for something, what was it and how does it work?' We could share other things through these channels. We also went department by department, starting with the more technical groups and moving towards more clinical and business groups, to make sure that we had presentations that were tailored to their types of use cases and interests. We were learning through the process how to make a good presentation and how to anticipate what their questions would be, building better and better training materials over time. This iterative listening process I think was key to the success of the project. There was a ton of enthusiasm from the community to be using something, and we tried to harness that enthusiasm to help create materials that would be needed to have input from people about what they needed and what they were worried about. We listened to power users and tried to listen to all of the different clients that we had. We had a ticketing system and an intake form which helped us with the IT aspects of the platform, which have been quite smooth. This iterative approach really allowed us to accelerate, and that was true really after we excluded clinical use that we could accelerate. We've continued monthly meetings with Microsoft to make sure that that partnership is working. At this point, I will hand off to Ronaldo to talk about how our community is using the tool. Thank you very much, Jason.
R
Ronaldo33:37
Thank you very much, Jason. So we have really an opportunity here to show you the uses of this tool because we have had it for over a year, yesterday now in production. This is how people are using it. These are the early adopters. The very first survey, a few dozen users, they said that the most useful contribution for GPT for DFCI was for extracting searching information in notes, in reports, in other documents, and files in general, as well as answer general knowledge questions. Fast forward to now, one year later, what we see is very interestingly, we have a lot of other use cases also appearing: drafting, editing emails, minutes, presentations. That remains king together with answering general knowledge questions, so basically a solid alternative to Googling information. But what we see is that also the other use cases are now starting to pick up. In other words, our users are becoming more savvy. They are really understanding this tool better, and they are able to create prompts that are more complex and that reflect a deeper understanding of the technology as well as the integration that they can do with that technology. When it comes to concerns, this is what we reported initially. The first one was ethics and compliance with our policies, as well as inaccurate false output, so what generally is referred to as hallucinations in LLM lingo. What's interesting is that after we have had our policy and one year of usage now, we see that inaccurate or false output is the only one that now really remains strong. So hallucination is really a worry that people have. They're also interested now in additional things, so they want more training as an example, and there are still limitations around the usage such as they want uploads directly from certain sources, they don't want to copy-paste the content. So these are all features that we are considering. It's important now that ethics and compliance with policy, because of the acceptable usage policy in place, is not spiking all the way to the top. It's actually on par with other requests. So this is actually very good to see. It means the policy is working, it's enforcing, and it's disambiguating a lot of queries that people had in their mind.
We also use the GPT for DFCI to analyze its own logs, right? Why not? So we used the GPT-4o mini, very inexpensive, still very efficient model, to analyze all the prompts that were fed into the application. We analyze what you're seeing here is the past six months. About half of the use, even a bit more than that, is around help with technical coding as well as summarizing, paraphrasing, editing, and critiquing text. So that's a very good insight that we had from analyzing the logs. Second, about a quarter is related to operational clinical research type of queries. Then also having this tool as an on-demand thinking partner is something new that we started to see. So basically, as you Google things, what happens is that you start opening some links and then you come back, you revise your query. On the other hand, the thinking partner guides you through this process. You describe what is the status of your current brainstorming, and it guides you through a revised thinking. That was very nice to see. So as of today, we have 1,800 users and we run about 500 sessions per day. That's really nice. So what are we looking at in the future? Number one, we are opening up the API, so the back end that supports the web app you have seen in screenshots. We are opening that up to researchers so that they, being our primary consumer, can run complex prompts, not just one or two, but run hundreds of them, run them against different A/B scenarios, and really run with this innovation and have it inform their research. The pilot phase is complete, and we are opening it up to more and more users this month. The second thing, and this is regarding that hallucination point I made earlier, is a capability. I have to thank my colleague Brent Manahan for suggesting we go this route: having a capability of querying through an LLM the repository of documents that we do have across the institute. So that could be a OneDrive folder, a SharePoint location, and so on and so forth, my past emails. There are many, many connectors. We could actually index in a vector database and use the LLM in a RAG fashion. We're really building that as a capability. We have done numerous pilots. One was published in research. We ingested the last two and a half years of papers that were published in the context of computational pathology, and we were then querying this database of knowledge that frankly I don't think anyone will ever have time to read. It was over 14,000 pages of highly specialized text, and now we had it all available for querying at our fingertips. 'What is the best method to figure out that biomarker in that context?' It would read a bunch of papers for us, compile an answer, and show in a RAG fashion what documents, meaning what papers, it had read to compile that answer. That was very nice to see, and we'll be doing more of that. Next up will be really streamlining existing procedure and going after those use cases that are higher ROI and lower risk compared to clinical AI. As Jason said, we took off the chart clinical use, direct clinical use of generative AI, because yes, it is high, but it is also very high risk. So we're going after those other use cases that are somewhat boring but very high ROI. Think about revenue cycle, right? How many reimbursements you could support if you had an LLM to support your filings with an insurance company, and all of that, just to give you an example. Then the last one that is still in scoping is 'Talk to My Data Live.' That means being able to ask questions like, 'Hey, what is the KPI status for my department financials around project ABC?' and take also into account the seasonality of that. Only an expert would know what all the acronyms I just used mean and how to design the SQL queries, what documents to look at. So we are building in the future a piece of LLM that will have access to our metadata, our data, our acronyms. It's writing the SQL, it's consulting the documents, and it's also creating charts. Again, this is in scoping phase.
To wrap up, what are the key summaries that you can take from this talk? As Jason said, it was a sad but really excruciating decision to head off clinical immediate use. What do we mean by that? We use the definition that the FDA and the IMDRF use for that. So that basically means four things: to treat, to diagnose, to drive the clinical management, or inform it. Those four use cases were banned. They were banned to protect our patients, to protect our practice, to protect our providers, to really keep everybody safe. It's the same thing for drugs. When a new company invents a new molecule, you don't just go give it to everyone. You do a clinical trial. So we thought this is a terrific piece of innovation, we have to use a solid framework. Clinical trial is the way to assess this. You have a use case for treatment, go for it, wrap it around the clinical trial and explore it through research, clinical research. So basically this decision allowed us to enforce some of the best practices that we already had, shake off some of the early excitement, and at the same time be very pragmatic, very deliberate about the use cases that we were chasing. Again, it opened up hundreds of new use cases around basic research, clinical research, and operations and efficiency around that. Another thing I wanted to highlight is the companion AI models that come with it. When you interact with the UI through the web interface, there are a few dozen other models that run in the background against every text that you put in and every text that is trying to come out of the models. If any of that text contains any mention of violent language, sexual content, harmful content in any way that is skilled, it is either not shown to the user or is not sent to the AI. It's still logged, but basically it's stopped in its track. Same thing for jailbreaking attempts. As you know, there are certain prompts that you can use to reverse engineer some of the models or mess around with the model. All of that, we have now a supportive AI that allows us to spot that type of prompts and highlight that behavior for further investigation by the internal authorities that really have an eye on these, as well as copyrighted text and code. As Jason said, most likely some OpenAI models were probably trained on GitHub, and the license is not clearly stated when it outputs a piece of code. So when a piece of code is output, we always run against these additional models to make sure that there is no copyright infringement unintentionally done there. Again, the logs are fully auditable by the designated personnel and officers at the Farber. We have these in our secure Azure premises. Our focus was really learning through this experimentation, and only later down the road opening it up to clinical use. We're going to use the same exact framework that we use for any drug: the clinical trial. So if you want to read more, please consult the paper we published in NEJM AI. We also open-sourced all our code, our training material, the broad strokes of the policy. All of that can be found linked in the GitHub together with the paper. If you actually follow the link from GitHub, you'll get access to the PDF of the paper directly. With that, this wraps our presentation. Jason and I will be very happy to take any questions while we keep on the screen this slide because there are really dozens of people who contributed to this, and Jason and I were really honored to work with them throughout this journey. Again, happy to take any questions.
H
Host46:12
Well, thank you to you both, Jason and Ronaldo. Excellent presentation on a timely topic.