Artificial Intelligence (AI) Driven Appearance Search with Dr. Mahesh Saptharishi

So why is search important? A good example would be to think about how we use the internet today through a web browser. Imagine that you could use the internet but had no access to a service like Google, enabling you to search for specific things, to find sites and information. You can imagine how things would be that much more time consuming and harder to do without search engines available to you, but also knowing that an abundance of information is at your fingertips if only you could search for it, if you could find the needle in the haystack. Well, that's why we need search.

We're able to store valuable information for much longer because storage is becoming more affordable. Collecting more data and storing it for an extended period is useful only if you can actually take advantage of it, to use it in some manner. Only 5% of video is ever actually used; 95% is never touched or looked at. The reason is that human attention is at a premium; it takes time to find the information we want in video. It's like having the power of the internet at your fingertips without Google at your disposal. Everything takes longer, and given that human attention is a premium, time is important if we want to make security not just a reactive task but also a proactive one involving decisive action. Decisive action requires understanding who, when, where, and what associated with any given event. We want to understand that quickly, but most importantly, when thinking about events that span a site, you don't just want to know what's happening in the field of view of a single camera but across your entire site. So being able to search for events and actors fast, so you can figure out what happened and what action to take, is critical.

One way to augment that attention gap, that attention deficit, is by adding the power of artificial intelligence. The goal of AI in this case is to filter out everything uninteresting so human attention can focus only on things that are truly important, the things that need the human brain to say, 'This is an important event; this is someone doing a bad thing at my site; I need to react proactively.' I need to understand the intention of that user. Those are the things we want human attention focused on, not arbitrary things irrelevant to the important event.

Time-based searches have been a basic feature available in pretty much every video surveillance system for years. Motion-based searches, probably the next most sophisticated, are also available in nearly every system. More recently, with the common occurrence of video analytics, we can search not just by motion but by activity involving not just anything that moves but persons and vehicles, and understand whether they are doing something interesting, like loitering, congregating, or coming together.

Moving from camera-based searches to site-wide searches involves being able to search for the presence of a single actor involved in an important event not just within one camera but across all cameras in the site, searching by any given actor. In many cases, we don't have the resolution or detail to facilitate a biometric search like face recognition on every individual. But we do understand what a person looks like; you might be able to describe a person, and based on that description, someone else may find that individual or everyone who looks like them. We want to facilitate the same thing, but instead of a manual process, you can ask, 'What did this individual do in my site?' rather than just within the field of view of a single camera. The second question is important, but it's not as powerful as asking across the entire site. That's what site-wide intelligence is for, and appearance search is a foundational step to enable it.

Enabling site-wide intelligence and appearance search requires a full solution where computation and intelligence are spread throughout, not just in bits and pieces. It starts with data collection; we must collect data at high enough resolution and clarity to understand and describe it with AI. Layer one of three is capturing high-resolution video. At Avigilon, we have cameras ranging from 1 MP to 7K or 30 MP.

Layer two is using artificial intelligence to describe what the camera sees. That's where analytics, specifically high-definition self-learning analytics, comes in. It's not just traditional analytics but one that takes advantage of high definition and is self-learning, so with a hundred cameras, you don't have to manually configure each one. The output of these two layers is a camera that can stream video and metadata describing the video content. With those pieces, you can start talking about artificial intelligence-based searching.

With AI, we can efficiently search across the entire site using analysis done in each camera, which is only efficient if cameras describe what they see. This is where appearance search comes in. It's the first step in Avigilon's sequence of AI capabilities. It starts with understanding who in the scene showed up across your entire site, either who did a bad thing or where they are now, across all cameras. That's what appearance search is about. Sometimes it's hard to describe a person in words, so instead of typing a description, you can use a visual example from any of the cameras in the entire site.

As two quick examples: you can click on a person in the scene and say, 'Find all instances of this person in the past' or forward in time, or ask, 'Where is this individual now?' Similarly, with a vehicle, clicking on it enables you to find it anywhere in your site across all cameras very quickly and efficiently, so you can make a proactive decision. This is fundamentally an AI-driven system powered by fairly sophisticated neural networks, at least by today's computational capabilities.

Such systems really shine when we have to look through lots of video where human attention is at a premium, ensuring that things easily missed by a human are caught. Not because humans are incapable, but because in the effort to find something quickly, they might miss a small occurrence. In this example, we're looking for an individual highlighted in the corner, who shows up way back in the scene. If playing the video quickly, you'd easily miss them, but with appearance search, you get instances of that individual even in the far background.

Being able to do this quickly and efficiently, not just within a single camera but across all cameras, enables the power of forensic and proactive intelligence, and that's enabled by appearance search. I can't do it justice by describing it, so I encourage you to come to our booth, stand 2A16, and get a demo to see exactly what we have in store. Thank you very much.