How to Select Video Analytics
Selecting video analytics can be difficult because many factors impact the choice. In this report, we examine 8 important questions that should be addressed when evaluating video analytics.
The easy part is usually determining the type of video analytic you want (perimeter violation, abandoned object, etc.). The hard part is assessing how it works with your existing system, facility and operations.
Here are the 8 recommended questions:
- How does your existing equipment impact choice of video analytics?
- How will camera positioning impact the cost of video analytics?
- How tolerant are you to false alerts?
- How tolerant are you to missing a real event?
- How many cameras will use video analytics?
- How will you monitor the video analytic results?
- How can you determine the performance of a video analytic system?
- What is the total cost of the video analytic solution?
How does existing equipment impact choice of video analytics?
Most end users have existing cameras and video management systems (DVRs, NVRs, IP video software, etc.) currently deployed. Before you can consider what vendor to purchase video analytics from, you must appreciate the abilities and limitations of your existing cameras and video management system.
Your existing systems will greatly determine how expensive the video analytics solution will be and how well it will perform. Only a minority deploying totally brand new systems can avoid this question.
Existing Camera
Check what video analytic capabilities your existing cameras offer. No analog camera provides on-board video analytics. Despite the marketing brochures, only a very small percentage of current IP cameras support real video analytics.
If you are using analog cameras, you can connect the camera to an encoder that runs video analytics next to the camera or you can deploy a server that encodes many analyzes many camera feeds at the headend. If you are using IP, some video analytic systems may allow you to remotely process/analyze the video but some require using their own cameras.
Existing Video Management Systems
Only a few video management systems provide built-in video analytics. This may be provided as a software upgrade or on a separate server. This is often the least expensive path and provides clean integration with the security operator's monitoring client. The drawback is that you are often limited to the type of video analytics and the performance that your existing manufacturer provides. Frequently, built-in video analytics perform poorly for demanding applications.
If you are looking to use a different vendor's video analytic, you must verify if the alerts or video analytic events can be monitored in your video management system's monitoring client. Alternatively, you can check if the video analytics can be monitored in your access control or central station software (depending on what your organization uses).
Support of video analytics in security monitoring clients is limited. Since most security managers want (or demand) monitoring from a single client this can be an important operational issue.
Edge vs Core
A frequently debated topic in the video surveillance industry is whether video analytics should be put at the camera (edge) or the server side (core). The operational reality is that for any given implementation, the biggest factor in selecting an analytic will be what works with your existing systems. Once you determine what your system will allow you to do, you are generally limited. Similarly, you may find that vendor that meets your performance requirements only provides edge or server based video analytics.
In this tutorial, we examine the pros and cons of different video analytic architectures including edge vs core.
How will camera positioning impact the cost of video analytics?
A routinely overlooked but key aspect to using video analytics is camera positioning. Not simply an implementation issue, camera positioning requirements can have a significant impact on the cost and design of video analytic systems. This is true whether you are analyzing license plates, people, faces or intruders.
In the Fine Print
In the user's manual of almost any video analytics systems are guidelines and requirements about how cameras need to be positioned. They almost always include points about minimum mounting heights, avoiding direct exposure to sunlight and minimizing the angles of incident from the camera to the target (see painful restrictions on one analytic offering). If you ignore these directions, performance can suffer dramatically.
Do This Before Buying
You need to factor this in during the design phase. If you wait until deployment, it can become a very significant problem. You may realize that you need to:
- Move an existing camera
- Add a new camera of the same type
- Add a new, more expensive, camera
At this point, the cost will go up, often dramatically. Worse, you may realize that logistical barriers exist (such as not being allowed to put cameras in places the video analytic system requires).
If you do this before the project, sometimes it will stop the project from being deployed. However, if you do this during the project and hit a problem, not only may the project fail but the end user can look bad and the integrator can lose a lot of money.
Why This is the Case
Handling a variety of lighting conditions or different angles can dramatically increase the computing resources needed to analyze video. Even if technically possible to perform such adjustments, it's often not feasible to implement them in commercial products because of the increased hardware cost this requires.
While cost of computing resources will certainly continue to fall and vendors will make improvements, these elements will continue to be an important constraint on video analytics for years to come.
Practical Examples of This
While different types of video analytics have specific requirements, you can certainly see the same general pattern of requirements for lighting and angles. Here are resources for 3 different video analytics:
Impacting Test Data
Be careful about the accuracy of test data as the tests may be performed with camera positioning that is unrealistic for real world deployments. For instance, see this image example from i-LIDS:
It's very unlikely that a camera could ever be deployed in this position. One, placing a camera on the exterior of a facility risks the theft or destruction of the camera. Two, placing a pole near a fence is a security risk. Three, often it's not even possible to place a camera in such a position because there's a road on the outside.
This position is misleading because it provides a perfect, unobstructed direct view of the target. This will result in better performance than the type of positioning that real world deployments require.
As such, test data or performance expectation needs to be evaluated in the context of the deployments that can be achieved.
See this image of an actual deployment at the Vatican:
The camera positioning is not theoretically ideal but given the layout probably the best that can be done.
Once you determine the constraints on camera positioning for your site, the cost and operational issues involved should become much clearer.
How tolerant are you to false alerts?
False alerts are a well known problem of video analytics because they undermine the confidence in and use of video analytic systems.
All video analytic systems have false alerts. Video analytics system differ in how many false alerts they generate and under what conditions.
Every video analytic user must tolerate some level of false alerts. The key question is how many false alerts can a user tolerate before the system becomes unusable? Is it 5 a day, 100 per week? Does it differ by the time of day? If you are an integrator of a video analytic system, you definitely need to know this and set the expectation up front with the end user because false alerts can kill video analytic projects. In my experience, they are the #1 killer.
Making the problem more challenging is that video analytics often obscure the total number of false alerts one can expect per day. You often see statistics that say the system is 99% accurate but this does not address how many times per day the video analytics triggers a false alert. A 99% accurate system that triggers a false alert every hour is probably going to be a significant nuisance for most users. On the other hand, even vendors who are being fair have real difficulties providing estimates because the number is so dependent on environment and local conditions.
Furthermore, end users do not really know how many false alerts they can tolerate. As we explore in a later question, this is often a function of how you plan to monitor video analytic alerts.
Minimally, users should consider how many false alerts they are willing to deal with per day and how those alerts would impact their ongoing operations.
How tolerant are you to missing an actual event?
Video analytic systems can miss actual events that they are supposed to alert on (called false negatives because the system falsely reports that nothing happened).
Some facilities demand systems never or almost never miss an actual event. This is typical if you are in a war zone or other critical facility where a missed alert can result in massive casualties or the destruction of a critical asset.
If your application can basically never miss an event, you will need to commit more on better equipment and environmental modifications to remove maximum the performance of the system. Two common examples of this for military deployment is the use of thermal cameras and/or positioning cameras in ideal locations (even if it occurs additional expense). It is also common for these scenarios for very rigorous testing in varied environmental conditions to be done to determine which video analytic systems are the least likely to miss true events. Finally, some users deploy other sensor systems along with the video analytics to minimize the likelihood that an actual event is missed. In this configuration, false alerts will increase but may be acceptable since it reduces false negatives.
However, most video analytic users tend to be fairly tolerant to false negatives. It is common for video analytics to be used to supplement or add security to facilities that previously lacked monitoring at all. In these scenarios, false alerts will be the biggest problem.
In applications of video analytics for retail or operations, missing a small percentage of actual events may not be tolerable. In these applications, users often care mainly about the general trends (rising, falling, etc.) and the estimated levels.
How many cameras will use video analytics?
The more cameras you have, the more critical that (1) false alerts become and (2) that the video analytics integrate with your monitoring systems.
More Cameras Increase False Alert Problem
Let's say a camera with video analytics generates 5 false alerts per day. That may be reasonable for a guard to verify throughout the day with their other responsibilities.
Now let's say you have 100 cameras. That's 500 false alerts per day or 1 every 3 minutes. At that point, it's unlikely that anyone could monitor this unless they were dedicated solely to this task.
Pilots with a few cameras may not demonstrate the extent of this factor. While different video analytic systems generate can generate far different number of false alerts (also depending on the environment), the large the system gets the more this becomes an issue.
More Cameras Makes Monitoring Systems More Critical
If you have a small site or only a small number of cameras, you may be able to monitor your video analytic system separately from your video management or your access control. However, bigger facilities almost always demand tight integration between various components and a single user interface. Some organization us the access control system, some use video management and other use PSIM.
Whatever it is that security uses, the more cameras video analytics are being run on, the more critical it becomes to integrate the analytic alerts into that software. And as we discussed in the first question, this can be difficult to accomplish because of incompatibilities and lack of support amongst systems.
How will you monitor the video analytic results?
Monitoring video analytics is a combination of technology and people. The people element is often overlooked as it is the last part of the deployment process, only commenced after the system is installed and optimized.
Nonetheless, ongoing monitoring is critical in ensuring that video analytic alerts are responded to and action is promptly taken.
The practical reality is that almost all alerts from any video analytic system are going to be false alerts. This is not an attack on video analytics any more than stating that almost all burglar alarms are false alarms. It is simply a fact of life.
Central Stations
While video analytics are gaining support in central stations, they are nowhere near as universal as traditional burglar alarms. Additionally, any video analytic system will only be supported by a small subset of central stations, it can be cumbersome or complex to connect them and the monitoring fees can be substantial. However, many organizations simply cannot manage self monitoring of numerous video analytic alerts night and day. As such, inquiring into remote surveillance monitoring options is strongly recommended if you want analytics but cannot handle them on your own.
Self Monitoring
However, today, almost video analytic users need to do in-house monitoring of video analytic alerts. There are 3 common options for this:
- Send alerts to a staff member with a PDA/phone: If the alerts are infrequent enough and the system small enough, you can simply set up an email to be dispatched to an assigned individual. This can work just like the procedures used for small businesses responding to burglar alarms. Of course, it suffers from the same scalability issue. If you have more than a few dozen cameras, this is likely to frustrate and overwhelm the operator.
- Have a security operator check the alerts at their desk: This is the most common approach for mid-size organizations from campus to malls to corporate parks. The big danger is that the number of false alerts overwhelms the guard and the guard stops paying attention.
- Set up your own internal dedicated monitor: This is the most efficient way to do this if you have hundreds or thousands of video analytics at numerous locations. Indeed, at this point, you are essentially running your own internal monitoring station. Very large corporations do have this infrastructure in place.
How can you determine video analytic system performance?
Unfortunately no independent tests of video analytics performance are publicly available. Even if they were, the test results may not represent the performance you should expect in your deployment (as the environmental or camera setup may differ from yours).
Furthermore, I would not accept the statistics from any vendor. There's simply too much ambiguity and or manipulation that a vendor can introduce in their statistics.
The two best steps you can take to determine performance is:
- Ask a reference: While references are always a good idea, you should target two key questions when asking a reference: (1) How many false alerts do they have per day per camera? This is usually pretty easy for the other party to know and something they are likely to share accurately. (2) What is their camera setup? Do they have new cameras or old cameras? Where are the cameras mounted? Given that this reference, their results are probably going to be good (or else the vendor would not have shared it with you). Asking about operational details can help identify some logistical or equipment advantages they have, that you may not.
- Do Your Own Test: The reality is everyone who deploys video analytics has to do their own test. It's not ideal and it is time consuming but it really must be done. This is a good idea not simply because of the products available but because their performance can differ significantly based on your environment and existing infrastructure. Also, tests will help you learn about the types of nuisances you will face in production, helping you to better set expectations on the system.
What is the total cost of the video analytics solution?
In general, recommendations about evaluating total cost tend to unfairly favor more expensive products as they can be excuses for justifying paying premiums. This is especially true in mature product segments where the functionality of products are very similar and the products have little or no side effects.
Because video analytics is not a mature market, the total cost is very important, more so than the individual product cost. Video analytic performance can vary dramatically. Plus, how video analytics works with other systems and can be monitored can differ greatly by vendor.
The cost of the video analytic products themselves range from free to about $1,500 USD (many video analytics are bundled into cameras so I am estimating the premium for the analytics).
In addition to the direct product cost, you should factor in:
- The cost of any new cameras
- The cost of modifying existing cameras
- The cost of integrating your video analytics system with other security or operational systems
- The ongoing operational cost of monitoring the video analytics systems
Ongoing monitoring can be especially challenging unless money or staff is budgeted from the beginning to cover this aspect.
Conclusion
These questions should help you examine various analytic vendors and to consider how analytics will impact your operations.