End User: How Do I Specify Video Analytics Performance / False Alarm Rate?

This is an excellent question. There is not an easy answer so let me first examine the tradeoffs of both extremes:

If you do not specify any specific performance standards (i.e., just specifying "Camera must have cross line detection capabilities"), you do risk having to accept poor performance.

If you do specify rigorous false alarm rate specifications (like "no more than 1 false alarm per week"), you may scare bidders away and force the price significantly higher for the remaining (i.e., the bidders pick thermal cameras and more expensive analytics and more labor time to optimize, etc.).

Also, if you do specify such performance, it will likely need to be over a defined acceptance test after the install, i.e., '1 week' after commissioning as opposed to some sort of service level agreement where the integrator/supplier faced penalties anytime over the next 1, 2, 3 years that the system had such false alerts (which is theoretically possible but will drive the price even higher up - keeping in mind that some times of year may cause more false alarms - rainy season, fog, summer vegetation, winter storms, etc.).

All this said the best approach is to test analytics before you buy so you are comfortable with their performance in your facilities / environment. This way you can then fairly confidently specify that specific analytic provider.

As an engineer for a supplier I can give some insight from my side. 

Sorry this is not performance related but here are some things that I would find particularly useful when responding:

A list of the analytics that are required and/or desirable (optional / future capabilities) clearly marked as such. This helps choosing the best analytics package for the job, combining several products from different vendors can get expensive fast.

Try to explain the purpose of the analytic and the outcome for the operator rather than just naming an analytic. We can often re-purpose analytics that were designed for a different function to meet your need (with the vendor's blessing). For example perimeter protection analytics have a lot of features such as direction control and can be used in place of specific direction control analytics. We might also have some products to suit the need such as VMS integrated RADAR or LIDAR (for example) - but without knowing the purpose or use case for the analytic we can't prepare a detailed response.

Provide as much information as possible. Drawings of the site with a reasonable level of detail, images of the expected scenes even if taken phone. Without this it is very hard to get analytics providers to comment in detail.

You are asking to document my favorite warning.  It’s all about managing the expectation. 

Marketing will tell you to expect 100% detections and alerts with “almost” no false alarms.  But what is a false alarm?

I can guarantee you if the site was in the line of hurricane Harvey, it would experience false alarms.

If you are looking for fence intrusion, usually defined as “line crossing” but typically installed as “entered region of interest” or “loitered in region of interest”, someone maintaining the fence will create false alarms as will a guard in a golf cart patrolling as I experienced once when the end user failed to mention this nightly occurrence.

Analytics in an alert can provide a benefit in information but sometimes you can consume a person with too much information.

1 alert a week, with settings providing 100% detection would seem reasonable.  If you have 100 cameras, you just added 100 actions a week for someone to manage.  How hard will that be to manage?  3 seconds to silence or 5 minutes to dispatch, respond and report?  BTW, that’s 5,200 actions per year. 

1 a month per camera with 90% detection?  Does everyone agree that one event missed may be the 1 you needed?

Comments about specifying incorrectly will create no bids or excessive bids.  If you really want to manage it, learn how to configure the analytics internally and only require the contractor to install and point the correct camera.  From there the expectations and risk/reward will all be on you. 

This is a difficult question to answer without knowing the specifics of your application.

As a VA skeptic to begin with, I would first want to know if VA is the best solution to your problem in the first place. Camera placement and the correct field of view is a crucial component of good VA performance. Many clients design a camera system for general surveillance, and then say, "Oh by the way, I want to use this system to detect intrusions through my perimeter fence line". In my opinion, this is the opposite of the way it should be done - cameras should be placed to provide good VA performance first, and then supplemented by additional cameras as needed to provide general surveillance.

Simply hanging a few cameras on the corners of the buildings and then expecting to get reliable detection at your perimeter fence line is usually unrealistic. I often find that the cost of getting the cameras where they need to be for VA exceeds the budget of all but the most security conscious clients. 

If I did think that VA was a viable solution, I would then want to do some actual on-site testing using the products I was considering. Only after gaining some actual field experience with these products would I begin to develop a specification.

When writing a specification, you not only need to define your expected performance requirements, but also need to clearly define each of the terms that you use within your specification. For example, even the term "false alarm" can mean different things to different people.

With regards to an acceptable false alarm rate, this also varies tremendously depending on application. For example, if I was using VA to detect intrusion in a challenging outdoor environment where I had a 24 hour command center that could assess before responding, one or two false alarms per day might not be a problem. On the other hand, if VA was used at an unmanned facility where the police were dispatched to alarms, a false alarm rate of even 1 per month would be unacceptable.  

Three quick questions:

Can you describe the site in general terms? (IE- This is an oil field, construction yard, ect...)

Will you be self monitoring this install, or will you be hiring a third party to remotely monitor the cameras? 

How much experience do you have with VA? 

The only truly effective solution is to do a proof of concept prior to assembling specifications. Then specify the solution that was proven to work in the environment at hand. Takes a little more time but the end user gets what they want (for a change) 

Some have asked for an example.  I would encourage you to come up with examples that you thought were good and bad.. But here is a generic example as I can not be specific to my location types.

I used the Bosch R&D Facility - Robert-Bosch-Campus 1, 71272 Renningen, Germany

Normal (non-analytic) Typical Coverage requirements:  80ppf coverage of entrance, utilizing multi optic camera cover perimeter of building at 40ppf and cover open areas at 20ppf. 

Analytic Alerts:

Entrance:  People counting coming in and out of the entrance to include demographic data. 

Alert Area near a building to include critical infrastructure:  Alert if a person enters this area and alert if an object is left behind. 

Advanced 1: Metadata search - being able to search across all camera metadata for the man in the red jacket

Advanced 2:  Using an AI model to alert on unusual behavior in the alert area.   

Normal (non-analytic) Typical Coverage requirements:  Cover fence line and open areas with 20ppf

Analytic Alerts:

Fence Line:  Cross the fence line alert - PTZ track violation 

Loitering/medical emergency:  Person on path stop for defined period of time to indicate loitering. 

Advanced:  Using an AI model to alert on unusual behavior in the alert area. 

UI#4 asked:

Can you describe the site in general terms? (IE- This is an oil field, construction yard, ect...) 

Response: Corporate campus environment or office building environment in a downtown area. 

Will you be self monitoring this install, or will you be hiring a third party to remotely monitor the cameras?

Response: Monitoring Center with onsite guard force

How much experience do you have with VA?

Response: I have quite a bit, but the problem is describing the issue on paper. 

Several have suggested testing VA prior to install:  As a government agency we can not do that, we depend in the vendors to propose a viable solution and must be manufacture agnostic in our proposals. 

My suggestion would be to only consider thermal detection if you need to guarantee a false alarm rate of a couple of alarms/day. This takes most of the site and positioning issues out of the equation.  Quality thermal/visible cameras are now available starting in the $1000 ASP range.  If this price/camera is too high then I would not recommend undertaking the responsibility of maintaining the false alarm rate, unless you have built in costs for making a lot of adjustments. Visible only detection will be very subject to camera placement, the environment (rain, lighting changes, etc.), and prone to missing valid detects.  

Plan on using more than one camera for certain areas to gain the required analytic. 

The new SightLogix SightSensor HD Thermal/Visible product line ranges in ASP from $1000 to $2000 and uses the same powerful processing and high quality analytics as the standard SightSensor NS line that is much longer range (150m+) where the ASP in in the $3000+ range.

Think a bit out the box!

In above mentioned sample at Bosch RD building I´d put a single PTZ cam and a 270 degree  / 40m laser sensor at the edge of the building. So you easily get >250 Ppm at the perimeter, configure once by just walking around and sleep well without thinking about false alarms, false positive or false whatever.
It´s absolutly worth the investment especially if there´s a number of situations you want to cover.

Otherwise you will have to engage a human specialist to be "installed" on the project to maintain some several VCA situations. And even in that case, it´s just a question of time till the customer will get angry.

I would work on requiring a walk test at night and specifying the minimum performance for detection to minimize misses. Misses happening are more important than focusing on the false positives. The installed environment introduces more uncertainty than published performance criteria created in a controlled environment to which analytics are and can be optimized that are unrealistic in the real world.