100% Video Analytics Accuracy?

Author: Ethan Ace, Published on Feb 01, 2012

In January 2012, analytic provider SightLogix announced its SightSensor intrusion detection analytics had been validated by the Transportation Safety Administration, providing "100% video analytics accuracy", a bold claim.  In this note, we look at this announcement, the TSA test, the integrator's experience and what users should look for when reviewing manufacturer test reports.

Overview 

While the report was prepared by the TSA, the evaluation of the SightLogix system was performed by the Safe Skies Alliance, which is a non-profit formed in 1997 to support testing of aviation security technology. The SSA receives funding from the FAA, and support from the TSA. This testing was performed at Buffalo International Airport in March 2011.

SightLogix has said that this report will be available to other airport security departments, but not to the public. While we understand the need for redaction, and keeping some of these measures secret, any number of other industries could benefit from this test, as well. As one commentor pointed out, though, it would take TSA involvement to distribute this to even industries outside of aviation, let alone the security industry at large.

Evaluating Claims

SightLogix's perimeter detection system is well respected and higher cost than many other options in the industry. It has also been developed for the specific purpose of intrusion detection, unlike many all-in-one analytic packages (see our review of their new 2012 camera series). Given this, we would not be surprised if detection rates are high, with false alarm rates low. However, users should be careful when reading this test report and press release, as some claims may be misleading.

  • Location and coverage information is redacted, so we're left with no knowledge of how wide the area covered is, nor are detection distances shared.
  • Intrusion techniques (the scenarios) were redacted, so we've no idea if subjects were detected while walking or crawling, from which direction they approached, at what speed, etc. Results for each of these scenarios were also redacted, so performance by intrusion type is unknown.
  • All nuisance and false alarm reports are redacted. It is extremely unlikely that any systems can be free of false and nuisance alarms, so this lack of detail removes information regarding potential system weaknesses.

Integrator Response

In a LinkedIn discussion regarding this release, one of our members who was involved with the integration shared the following clarification:

  • Regarding false alarms: "We had a number of alarms in the beginning and by a process of adjusmtments we were able to tune the system for the customer to an acceptable level. That is the key when installing an analytic system is setting the expectations of the customer correctly. 99% of the alarms are caused be animals and dependant on the size and quantity may or may not be a concern for the airport."
  • Regarding Nuisance alarms: "It does track objects that are not humans, but that was an added benefit to certain airport operations as birds, deer and other animals are not wanted guests on a runway. The biggest benefit to the system is that if there is an alarm it is automatically time stamped and logged, a monitor displays the target and the police officer can, by not touching anything, view to see if the alarm is something that requires attention."
  • Regarding performance: "I can say that the system after being in service for over a year is still providing a useful tool that is reliable and paying for itself."

He also stated that the system has been expanded to a second airport, bus garages, and a light rail system, which would lead us to believe the system is performing satisfactorily.

Conclusion

We found the integrator's insights to be far more useful than the study itself. A key factor was that the integrator was able to share experiences in the optimization stage and how it is used in daily operations. The lesson to take away - If you want a good reference, stay away from 'studies' and find an engineer who was involved in the details.

Comments : PRO Members only. Login. or Join.

Related Reports

Best and Worst ISC West 2018 on Apr 16, 2018
ISC West 2018 had strong attendance, modest overall new products, and a surge in Artificial Intelligence marketing. First, here are 20+...
Alarm.com Business Market Expansion on Apr 13, 2018
Alarm.com has millions of subscribers, but the company has traditionally been mostly a residential/home focused offering.  ADC's new Smart Business...
Eocortex / Macroscop VMS Company Profile on Apr 09, 2018
Eocortex is the international brand of Russian VMS manufacturer Macroscop. Macroscop was founded in 2008, and the Eocortex name created in 2013. We...
VMS New Developments Spring 2018 (Avigilon, Exacqvision, Genetec, Hikvision, Milestone, Network Optix) on Apr 04, 2018
What's new with VMS software? In this report, we examine new features and releases for Spring 2018 to track different areas of potential...
Forced Door Alarms For Access Control Tutorial on Apr 04, 2018
One of the most important access control alarms is also often ignored. "Forced Door" provides a vital and even critical notification against...
70+ New Products Directory Spring 2018 on Apr 02, 2018
We have compiled a list of new products for Spring 2018 in the categories: Video Surveillance Access Control Fire and Intrusion Other...
Favorite Video Analytic Manufacturers 2018 on Apr 02, 2018
Video analytics is one of the biggest trends in 2018, driven by marketing for Artificial Intelligence and Deep Learning. But what do integrators...
Destructive Video Analytics Hype Returns on Mar 27, 2018
It is not just Hikvision's false advertising campaign. With marketing money being pumped into deep learning, we are returning to the bad old...
Startup Image Intelligence 10X Accuracy Guarantee on Mar 26, 2018
An Australian startup, Image Intelligence, is targeting the biggest problem of video analytics, poor accuracy, with quite a claim: Is this...
Axis Z-Wave IP Camera Tested Poorly on Mar 20, 2018
Z-Wave is drawing notable interest for video surveillance use. In IPVM's initial coverage, 84% expressed interest in it, with nearly half being...

Most Recent Industry Reports

Favorite Access Control Manufacturers 2018 on Apr 26, 2018
150+ Integrators told IPVM "What is your favorite access control management software/system? Why? Unlike the 2016 access favorites where a group...
Last Day Save $50 - May 2018 Camera Course on Apr 26, 2018
Today is the last day to save $50 on early registration. Register now (save $50) for the Spring 2018 Camera Course This is the only independent...
Hikvision DarkfighterX Vs Darkfighter PTZ Tested on Apr 26, 2018
Hikvision has focused on improving low-light performance for PTZs, an area that has traditionally been a problem, even more so than fixed cameras,...
Digifort VMS Profile on Apr 25, 2018
Digifort, a Brazilian company, has a strong presense in their home country. In a crowded and mature Enterprise VMS market, will they be able to...
Death Of A Dummy Camera Manufacturer on Apr 25, 2018
5 years ago, IPVM gathered insights from a dummy camera manufacturer, who was then the top selling dummy camera provider on Amazon and 3rd in all...
Hikvision Critical Cloud Vulnerability Disclosed on Apr 25, 2018
Security researchers Vangelis Stykas and George Lavdanis discovered a vulnerability in Hikvision's HikConnect cloud service that: just by...
The Yolo Bro And The Death of Journalism on Apr 24, 2018
There's an old quote: The job of the newspaper is to comfort the afflicted and afflict the comfortable Unfortunately, the opposite is more...
DMP Adds Ring Video Doorbell Integration on Apr 24, 2018
Video doorbells have become one of the hottest items for security systems. After several years with no doorbell, DMP has announced integration with...
Milestone 2017 Financials Examined on Apr 24, 2018
For ISC West 2018, Milestone released ... their financials, touting "strong revenue growth in 2017". However, there were discrepancies with the...
Chinese Manufacturer Kickstarter Campaign Huge Success (EverCam) on Apr 23, 2018
In a week, a Chinese manufacturer's expertly done Kickstarter campaign has received $1.4 million in pledges, an incredible amount for a video...

The world's leading video surveillance information source, IPVM provides the best reporting, testing and training for 10,000+ members globally. Dedicated to independent and objective information, we uniquely refuse any and all advertisements, sponsorship and consulting from manufacturers.

About | FAQ | Contact