What is an IoT Analytics Platform? | #AskIoT | Grid Dynamic's Ilya Katsov

Hello.

Yep, sure. So my name is Ilya Katsov. I'm VP of Technology at Grid Dynamics, which is basically a consulting company that works with large manufacturers and brands on different data programs that involve data science, advanced analytics, collection of data from IoT devices and so on. So my personal background is predominantly in advanced analytics, data engineering lab at Grid Dynamics basically around our technology practices, including data platforms and data science capabilities.

Well, actually, we started with like data engineering and advanced analytics in several different areas, working with technology companies, you know, like largely technology giants and some other verticals that we came to IoT space essentially like a couple of years ago and we invest in aggressively into this area trying to build up these capabilities.

Yep. Absolutely. So typically, we view IoT analytics platform as like extended versions of general purpose analytics platform. You know, like all companies, they are building some sort of data leaks or analytics platform that include typical standard capabilities. Basically, you need to ingest the data. You need to establish processes for data transformation, data quality control, data governance, data access layers and so on. Also, you would build some ML platform capabilities on top of that to develop and manage the sort of different types of statistical models and so on. But these are generic capabilities, but when it comes to IoT space, generally you need to develop a number of extensions. Of course, it includes more specialized data collection capabilities because you obviously need to collect to like sensors, IoT devices. It requires to deploy typically some parts of this platform on edge and manage the services for data collection. Then it's also in many cases, companies have built in specialized capabilities to process IoT events specifically, build in like cool engines, or build in specialized services for that facilitate the development of specialized models such as predictive maintenance. So you can control. And then these models need to be deployed back in many cases to the edge devices or directly connected devices. So this like all this capabilities together like in my view, they constitute these IoT analytics platforms essentially.

Yeah, you know, definitely you know, in my view there are differences between these cases because when we are dealing with like industrial use cases meaning that we have let's say some physical facilities like factories. There are some implications of that that influence the design of the IoT analytics platform. Let me let me give you like an example. In the cases when yeah, when like large, especially large manufacturing companies build their IoT platforms, they face the problem that they need to connect to multiple facilities that can be operated by different third parties, like suppliers or partners. And these facilities can be located in different countries. You know, it's very common, for example, to, you know, have these facilities, let's say in Asia, and there are different protocols and rules how these facilities can be accessed in terms of data collection. You know, in case of, you know, for some suppliers, you can directly connect to specific devices, some equipment for some suppliers, you cannot do that. You can only like send specifications essentially companies to you. And by this reason, in the case of like industrial IoT, it's like at least in our experience, it's very common to expend significant effort and make design decisions that facilitate this, like global integrations and there are security aspects of course as well. And you know, all these capabilities, they're not necessarily the keys for other like verticals which are in IoT space. For example, if you just collecting, you know, data from just, you know, your local, you know, let's say, you know, actually the opposite example was collecting data from consumer devices, which is like totally different, right? You would not have this problem. You can use cloud services for that more efficiently. And so it would be much of difference, of course.

Yep. I think like in our practice, we actually seen several good strategies how you can go about it and the specific choice of the strategy I think it influenced by a few factors, including the scale of the company. Let me again give you like a couple of examples, probably. So one possible way, one possible strategy, how you can go about it if you, let's say, reach like small to medium size, like company with relatively few facilities and relatively few devices is that you can start with and basically like data collection and doing some exploratory data analysis so that all the data that you potentially can collect from you know your sensors and devices aggregated, you can actually monitor this data, you can establish then some basic capabilities such as monitoring some exploratory data analysis, some basic analytics, and then you can start to improve your operational decisions, build some maybe rule engines and gradually increase the complexity, then build maybe more like predictive models and so on, and to try to cover a broader range of use cases and so on. But this approach is actually, in my view, it works well only to a certain scale because if we operate like if we are talking about large companies, like large manufacturers, they cannot really start with collecting all possible data from, you know, all facilities and all devices and all the sensors, it's just not feasible. And even if they are able to collect all this data, it's not really possible to do meaningful analysis of all this data at the same time and productionalize it. And it leads us to a different strategy that is basically use case driven. In that case, the better approach is typically to identify some priority use cases. And this is also not a trivial problem. There is typically, you know, some methodology, some approach for doing that. But then once this use case is identified and the business value, potential business value for this use cases is estimated, several things can go in parallel. First, in many cases it makes sense to start with some prototyping that's not necessarily related to building a complete platform as such, it's just around building some proof of concept models or something like that. And then parallel with that, you can start to establish more like, you know, foundation for productization of this building a more complete platform that addresses, you know, all these different concerns that we talked about, like data collection, deployment on edge devices and so on and so forth, and then at some point, these different workstreams they come together. Of course, the POC will ultimately be productized and deployed on this on this platform and then you can start to onboard more and more use cases. This is what we've seen for, basically larger companies.

Well, I think there are actually several aspects to it. Obviously there is some like business aspect. How do we pick the use cases and what you basically invest in the term, how you evaluate the impact and so on. This is one part of it. But I think your question is more about technical aspect, how you operate this platform and from that perspective. Also, there are like a few best practices, I think, just to, you know, to provide a few examples. First one is that like the quality and understanding of proper understanding of the data, semantics is very important because if you cannot ensure a good quality of data and in like IoT use cases, this is like particularly a problem because of different connectivity issues and so on. But it can basically invalidate all downstream analysis or whatever you do with this data. And understanding the semantics of the data, I think it's also very important because for some use cases such as anomaly detections, it's very easy, for example, to build a, let's say anomaly detection model that actually mismatches the actual semantics of the data. And that does not detect let's say meaningful anomalies. So this understanding of data completeness, doing data quality checks, making sure that data semantics is well understood in the downstream analysis, I think it's it's very important. And then the second example, yeah just the second example that they wanted to collect is like upstream mobility of, of data collection and inference processes. I think it's also very important, especially in use cases that involve like automatic real time decision making, is of course important to, you know, to collect data not only from IoT devices, but also collect data that characterize how models or whatever logic you deploy is performing, control, you know, different types of drifts or some anomalies in this processes to make it sure that that you trust the decision making, that the decision making automation processes are correct.

Well, typically, you know, typically like in my experience, this is like two layer approach. You basically in most cases you would have some sort of infrastructure or platform operations team that basically is responsible for each of these foundational capabilities, like general data quality control, deployment capabilities, observability capabilities. And this team, they can include more like operations type of resources that like doing day-to-day monitoring and so on. And of course, there will be some counterparts, maybe physical locations like virtual facilities. But I think it's kind of separate. And then on top of that, the second layer are teams that actually like implement specific specific use cases. And these teams, like in many cases, they also are responsible for like doing at least like second level, third level of support of production solutions and make incremental improvements on top of that. So this is this is not the only possible approach, but I think it's most typical one, this, you know, this two layer.

Well, I think different use cases related to automated quality control are really like important and it's quite used because these use cases are directly related to different costs, either cost of quality control, cost associated with, you know, recovery from different defects and, you know, failures. So we see that currently, you know, like many companies, that's not only manufacturers, but companies that, you know, somehow wants to share the physical operations, they, like investor law team, this type of use cases can include anomaly detection in numerical metrics, also quality control in different levels are very, very popular. So this is one large group, and I think this is probably like the most the most important trend, the most common one. But also there are other areas that I think are gaining traction, such as, let's say, automated dynamic process control. You know, you can collect different metrics or like images, videos, do some analysis based on that and then dynamically make adjustments to parameters of agreement to ensure a certain level of quality or stability of the outcome. So this is also very important from the business value perspective because it helps to achieve quality levels and stability of quality that cannot be achieved using, for example, normal parameters, management or some other techniques.

Well, I think there are like several ways, like several types of resources that they can use because there are different types of solutions. How this problem can be approached. First one is that there are some third party products that position themselves as IoT platforms, specifically, and these are resources that, of course, you know, one should check and be familiar with what are the capabilities of this product. But this is just one possible, possible approach. And like in our experience, capabilities of this platform are somewhat limited or specialized. Like more general approach is to build solutions using like cloud native services provided by hyperscalers like Amazon or Google or Azure. I think, you know, these hyperscalers, they're currently are very good in providing IoT related capabilities like specialized services specifically for IoT deployment and so on, and again, I would of course recommend, you know, anyone who wants to understand this space to get familiar with, you know, what is provided by hyperscalers. And then, of course, there are a number of different publications that describe how to build specific use cases and all these and custom models how to address this data quality and so on. Yep. So in particular, you know, we have quite a good library like publicly available, library of articles and my own papers on this topic.

Yeah. For sure.

Yeah, thank you. I appreciate the opportunity to have this conversation.