Model Performance Monitoring and Why You Need it Yesterday // Amit Paka // MLOps Coffee Sessions #42

Thanks so much to Major. I think it's been amazing to watch the like engaged communities on ML ops, and it's so insightful to be part of these conversations, right, to drive all these learnings. Given it's still a wild west with ML ops, it's like kind of communities like these are where standards get defined, workflows get defined, people help each other to make their lives easier in this wild wild west. So we are really happy that you and the team have actually picked up this kind of tough job on maintaining this kind of community. It's a lot of work, and we're happy to actually play a small part of it.

And so we were using machine learning models for product recommendations. And so if you went to a Samsung mobile app and you looked at the list of products, those were recommended by a machine learning model. And I was observing a lot of challenges that the data team was facing just operationalizing these models, right? So just starting off with managing versions of these models, things that we take for granted on Git, was very hard for them. Doing an A/B test on these models was incredibly hard. The way the team would do an A/B test is they would run a previous model for a week and then they would stop running the model, they would run a new model for the next week. Based on the output of these models, one of the things I was asked to do is I was testing the hypothesis that Samsung users would be open to buying a Samsung product via a chatbot. So if you speak to a Samsung chatbot using Facebook Messenger, you'd go through this conversation script and then you'd buy it. I was testing that hypothesis. To actually test the hypothesis, we ran an ad campaign, a million dollar ad campaign. The target of this ad campaign, 300,000 users, was called from 21 million users in the database that came from a machine learning model. And of course, as a product manager, you want to understand why those 300,000 from the 21 million. Was that the wrong audience that we went after, or was it something else? So the lack of transparency in the decisioning of these models played a key role, and it came up again and again. And so I connected with Krishna, who's the founder CEO, who I had gone to grad school with and overlapped at Microsoft. And he was facing a similar issue at Facebook. He was working on the news feed and he joined after the US elections, and there was a lot of angst about potential bias in the news feed. And it's very hard to dispute that given it's a complex ensemble. And so he worked on trying to help internal teams learn why you're seeing a news story using explainability, and that was also a challenge. And it really keeps us going. So that led to the genesis of the company. And it's been great and almost validating to see how our worldview has, close to two and a half years back when we started, is becoming more mainstream. The trust in AI is becoming not just a nice to have but a must-have, especially in light of this new EU regulation where Margrethe Vestager comes out and says that trust is a must-have. That really is one of the things.

Yeah, I was just going to add that at Fiddler, I look at ensuring that our products stay true to the mission of not just giving you visibility into model behavior at train time, but also post-deployment, and kind of weaving that into an experience that's very unique to ML ops, given machine learning models are unique probabilistic entities that need constant new iterations given their actual performance can decay constantly.

Yeah, yeah, yeah, great point. It's a pretty, I just wanted to start off by saying, Nikki, this is something we were discussing. It's a pretty detailed proposed regulation. It's around 80 pages, and I am surprised how much time the team spent on crafting this. It's pretty well written. The goal that the EU seems to be going for with this proposed regulation is they really want to go after helping teams build human-centric and trustful AI. That's typically how proposed regulations get voted on, and then it does take a couple of years for it to go into effect, and there might be some changes that will happen from now until the one that gets voted on and accepted, and there will be a couple of years before it really goes into effect, just like GDPR did. But to summarize what the proposed regulation is doing with trust at its core, it first classifies AIs into four categories. It says the first category is unacceptable risk applications that are banned. These are applications like behavior manipulation, for example. They're banned. The second one, and now we're getting into high risk applications, and we're going to talk about what this oversight is. The third one is limited risk applications, and these are the likes of chatbots, and there are some transparency requirements for these limited risk. For example, it says if you're talking to a chatbot, you should know that you're talking to AI, really simple guidelines. And the last one is minimal risk application, and this they believe will be the majority of AI applications, and there's no proposed intervention here. An example of this is a spam filter. Limited minimal risk application that is high risk. And so high risk applications are really ones that have implications on public safety, services, there are actual opportunities. Examples of these are credit scoring, loans, recruitment, self-driving cars, robotic surgery. So these are high-risk applications. So the first thing is, do you fall in this high-risk application? And the next thing is, what does high-risk application mean? What are these new guidelines? And so what the proposed regulation stipulates is that you can go back to it. There are requirements on transparency, knowing how this model was built, how it operates, and included in this transparency is fairness and bias implications on certain protected attributes like race and so on. And human oversight. It uses human oversight a little bit differently from transparency. The ability for a human to come in and control this. Of course, the human needs transparency, but they need the ability to actually control it. And finally, it says that once you know how this model is working, it's transparent, high quality data you need to have. It was trained well, but it ran into live data that completely makes it a hazard. That should not happen. You should be able to monitor it. And finally, there's robustness aspects, accuracy aspects, and security aspects to get this operational visibility. So that's the high level of what the proposed regulation means, and we can deep dive into what it means in the context of transparency and what it means in the context of operational visibility.

My sense is there's a little bit of an unknown there. They probably have not really run into a lot of military applications, so they don't know what they don't know, and they probably want to keep that door open. But it feels like there would be potential pushback on that, and there might be language in other iterations of this regulation that might... but it feels like they wanted to have that option, not knowing how they're going to deploy, to just make sure that it doesn't hold them back in situations of emergencies.

What's been really interesting is in the US, the DoD has actually come out with more focus on trust in AI. So there has been a team formed called JAIC, and that team is now looking at how do you bring in transparency and trustworthiness to AI. So the US has kind of absorbed the implications of black box AI, especially in the context of DoD. So I think the DoD had a principles document that they kind of came out with towards the start of this year. So it's kind of interesting to see that in the US, DoD is actually focusing on increasing trust within AI.

The diversity of explainability is broad, and we get to hear a whole lot of that from prospective customers across a variety of verticals. The urgent need for explainability comes from verticals that are facing the problem today on the ground and cannot run their business. And so these typically tend to be businesses that want to or are using AI, but they need to make sure that they're meeting some guidelines, some regulations. For example, in banking in the US, there's a stipulation of SR 11-7 that the Fed laid out at the end of the financial crisis, which mandates that you need to have explainability for models after deployment so that you don't take down the financial system again. It was meant for quantitative models, but it is also applicable to AI models. So the need for explainability on the ground, the ones who have the budget and are buying, are where they want to use AI and are hitting into regulations that mandate transparency today. That's one part. And now there are also companies who want to be good corporate citizens and are also seeing the potential pitfalls of not being transparent. For example, there have been recent issues where facial recognition software was not recognizing people well if they have darker skin tones, and because of that problem, a lot of companies exited the facial recognition market due to the groundswell of protest by people. So the second need is coming from these group of companies who want to be good corporate citizens or want to ensure that they don't have any PR issues with the AI that they're deploying. And so the diversity there is pretty broad depending on how widely used their AI is and what kind of potential brand impacts they might face.

Yeah, it's very interesting now as teams go beyond just looking at explainability, beyond the actual regulated use cases like lending, healthcare, recruitment. There are other use cases which are high value, like in oil and natural gas, where a single decision can be worth 10 to 20 million dollars. So explainability there is more like a need to ensure that you're making decisions with the right ROI. But what we're also seeing is that teams are now coupling explainability with monitoring. So it's not enough to know how the models are behaving, but also ensuring that you know how they're behaving when they're running, operational aspects of it, not just pre-deployment but also post-deployment. Because given the nature of machine learning models which are really probabilistic, they can decay over time. The data that they see might be different than the data they were trained on.

Yeah, it's a good question. I think different companies in this domain are talking to their customers and they're kind of converging on the same problem that they're seeing. So I think different companies are actually coming up with similar solutions, so they are mostly similar. The premise of MPM, really for folks who are hearing about this for the first time, is that within operationalization, you have to focus on understanding and improving model performance, and you do that through visibility across all the parts of the model life cycle. So that is MPM: how do you improve understanding and model performance through visibility across the life cycle? And what does it mean in different parts of the life cycle? At train time, when you have training sets that you're using for your model, you can assess them for bias, you can assess them for any data quality issues. You can also get feedback from your current model champion running on certain slices of the data. This could mean slices of low performance, or if you're a lending model, slices where you're making loans to men versus women or loans in one state to another. It could be slices that are off, like business interest, or it could be slices that have performance implications. At deployment time, this could mean that you're recording traffic that your model is about to serve and you're about to do a champion-challenger or maybe one champion and multiple challenger tests. At monitoring, of course, and this is where a lot of focus seems to be, but it's one part of MPM. You're assessing this traffic that you're recording for drift within life and trying to analyze that maybe in the context of the slices that you just validated the model against at train time. You can use the same slices and then that gives you a new world view that feeds back into the training site. So this iterative process where constantly the model is converging and diverging on its actual performance because data is changing. The management of this across multiple versions of your models is MPM. And you could also view this in the context of if you want to store everything in one location, it could be an evaluation store. We believe it's like a meta layer that allows you to store all the information about your model or slices of the model so you can be better equipped to ensure that your model is operating at peak performance.

Control theory really means that you not just measure it but you can influence it, so you can actually keep it at a desired state. You've got sensors that not just tell you what it's doing but you can control the speed based on the input. So if your desired speed is 25 miles per hour, the sensor tells you 27, now you can control the sensor and say come down to 25. In the case of models, that means that, for example, you're not really going for highest accuracy in your model, but you're trying to ensure that the model has a certain acceptable level of bias across protected attributes. And so when you first have visibility that that is being breached, you feed whatever that new data representation is back into your training set so you can create a new representation of that model that meets that new threshold, the new reality that you're seeing within production. So it is an extension of control theory given that you first have to have visibility into how a thing behaves. A whole lot of teams don't even know how their models are behaving when they deploy, so that's number one, not having any operational insights. And once you have operational insights, how do you ensure that it's maintaining your desired metrics? And that is where the control theory comes in.

Today, teams typically have some data, they train a model, either it could be handcrafted or it could be using AutoML, and it's likely tuned for highest accuracy. And then you just take this model. If it's through AutoML, you don't really know what kind of relationships it dropped. You then go and deploy it, and then you don't monitor it because you assume that the reality that your model faces meets the hypothetical training set. So this is what teams do today. In the MPM world, what you would do is when you get that data, you would assess it for data quality. You would ask: should you have more samples of one versus the other based on the fairness metric that you're trying to assess? How many instances? What is the distribution of the data? Does it have a whole lot of missing values where you want it to grab relationships without the missing values? And so on. Then you go on to the post-model training phase where when your model is trained, how do you learn what relationships the model is grabbing today? That's typically not what a data scientist does. You might do some ad hoc testing on a test data set, but you don't really look at what are the feature importances, what are the important features of this model, what are the lowest performing regions of this model, how is this model behaving in the context of certain protected classes or certain slices that might be relevant to data, for example loans made to certain states. That is not typically done today, and that's one of the aspects of MPM. So teams would be logging this model in the context of the data and running the analysis to ensure they know how the model is behaving. At the monitoring part, you would then be looking at not just in the context of the data that the model is facing in live, you would look at it in the context of slices that you just validated the model against. If for example you have set a threshold for bias, or if loans made in California are really important, you would look at it from the lens of that. So you would need tooling not just to help you monitor broadly but also at a very specific level if your bias thresholds, if you have certain slices that you want to monitor, certain metrics. It might be that you could have trained the model for accuracy, you want to ensure that you're meeting the accuracy. At times you might not even have ground truth. If you have data integrity issues happening, then you use that representation of data that your model is facing in live and feed that back into your training set so you can train a model that accurately represents the reality versus the hypothetical. None of these things are done by a typical data scientist today, and that's what they would need to do within an MPM workflow.

It's not just an ML engineer or a software engineer focused on machine learning, but it's more actually the model development experts themselves. So it's a mix of data scientists and machine learning engineers. In fact, what we're also seeing in some teams is the traditional DevOps folks who monitored web apps are being repurposed to monitor machine learning apps. That's also happening. But our typical users are data scientists who want to know more about this model that they've trained, ensuring that it's not biased, ensuring that they know if there are regions of lower performance. And machine learning engineers as well. This is where I think the job functions are still being formalized across data science and this world of ML ops. Especially within ML ops, we've not seen a whole lot of purpose-built ML ops teams that handle these deployments. They're being repurposed from DevOps. We have seen ML engineers also kind of do the deployment but maybe not stay on top of the operational model.

Yeah, we've got secondary users beyond just data scientists and the actual MLE. These are folks who are decision makers running the business who want to know how the ML is working. Then there are folks within ops who are taking calls from customers on why they're seeing the output they're seeing. And then there are customers themselves who want to know more and build trust with these models that impact them. So while there are primary users, there's a whole host of secondary users, and we've built the product to be able to be consumed easily by this range of users who might not be as technical as the primary users.

Well, I think banking is still one where you would expect, 'Hey, there's regulation and there's a little bit of an unknown, so that really causes teams to take a step back and maybe become a lot more conservative about the new technology.' However, I was surprised that they all wanted to push the boundaries of what AI models can do. We've seen as we work with banks, we've seen complex models, not just your typical ML models like XGBoost or logistic regression, we've seen deep learning models with complex data types. So the surprise was how much they want to adopt AI and how much they want to push that boundary and ensure that they're still abiding by the regulations and not just staying on the sidelines and saying 'We'll wait.' So banking, which has historically been very conservative with high regulations, in the domain of AI they are forming teams, for example, they are forming AI governance teams where the teams are formulated only to validate machine learning models. So they try to crack machine learning models using tools like ours and validate them. So they're rapidly forming these new teams and doing a lot of investments. Very interesting.

Different functions have different incentives that move them towards this goal. For machine learning teams that deploy models and use cases, given the investment that you need to make in building machine learning teams and tooling, the use cases that they deploy to first are the highest value. So you've made these investments in machine learning, and you built this model. Do you want to make sure that your business metrics are still working well and that you stay on top of it in real time? You know that if your model has operational issues, you can jump on top of it and address it. Or if you're in a lightly regulated or highly regulated domain, you have visibility into how your model works so you actually protect it against not just fines but also PR risks. So that resonates with a business buyer or a user on the ground. Within DevOps, within the actual management part of these models, you can call it old-school DevOps or new school ML ops. They have seen the pain of managing just five models. So how do you manage this massive scale of AI models that is coming your way? If you have five models, save your time. You don't want to spend your time babysitting these models, and that resonates to that audience that's managing these deployed models, whether it's an MLE in newer teams or old-school DevOps. And for data scientists, the thing that resonates is helping business teams understand how their models that they're working on are behaving. Whether you need to explain it to a regulator or a compliance officer, but beyond that, if you're not in a regulated domain, knowing how your model works can help you unlock potential performance benefits, can help you debug the model. Models are incredibly hard to debug. You're just training a model for high accuracy but you don't know how the accuracy is actually distributed. So the thing that resonates with a data scientist is helping others understand how the model is doing, helping them understand it so they can debug it or improve the performance. Those are the three kind of groups and the three incentives that resonate. So for a data scientist who's looking to get started, you've got operational insights. Do you have tools to help you ensure that you stay on top of when these models go bad or go awry? And then go and talk to the actual business owner and ensure that they have the right tools to ensure that the performance of these models in high ROI cases are still being maintained, and they have the right tooling because typically the actual budgets might come from somewhere else but the user might be somebody else. So bringing this all back full circle to where we started the conversation.