100% Video Analytics Accuracy?

By: Ethan Ace, Published on Feb 01, 2012

In January 2012, analytic provider SightLogix announced its SightSensor intrusion detection analytics had been validated by the Transportation Safety Administration, providing "100% video analytics accuracy", a bold claim.  In this note, we look at this announcement, the TSA test, the integrator's experience and what users should look for when reviewing manufacturer test reports.

Overview 

While the report [link no longer available] was prepared by the TSA, the evaluation of the SightLogix system was performed by the Safe Skies Alliance, which is a non-profit formed in 1997 to support testing of aviation security technology. The SSA receives funding from the FAA, and support from the TSA. This testing was performed at Buffalo International Airport in March 2011.

SightLogix has said that this report will be available to other airport security departments, but not to the public. While we understand the need for redaction, and keeping some of these measures secret, any number of other industries could benefit from this test, as well. As one commentor pointed out, though, it would take TSA involvement to distribute this to even industries outside of aviation, let alone the security industry at large.

Evaluating Claims

SightLogix's perimeter detection system is well respected and higher cost than many other options in the industry. It has also been developed for the specific purpose of intrusion detection, unlike many all-in-one analytic packages (see our review of their new 2012 camera series). Given this, we would not be surprised if detection rates are high, with false alarm rates low. However, users should be careful when reading this test report and press release, as some claims may be misleading.

  • Location and coverage information is redacted, so we're left with no knowledge of how wide the area covered is, nor are detection distances shared.
  • Intrusion techniques (the scenarios) were redacted, so we've no idea if subjects were detected while walking or crawling, from which direction they approached, at what speed, etc. Results for each of these scenarios were also redacted, so performance by intrusion type is unknown.
  • All nuisance and false alarm reports are redacted. It is extremely unlikely that any systems can be free of false and nuisance alarms, so this lack of detail removes information regarding potential system weaknesses.

Integrator Response

In a LinkedIn discussion [link no longer available] regarding this release, one of our members who was involved with the integration shared the following clarification:

  • Regarding false alarms: "We had a number of alarms in the beginning and by a process of adjusmtments we were able to tune the system for the customer to an acceptable level. That is the key when installing an analytic system is setting the expectations of the customer correctly. 99% of the alarms are caused be animals and dependant on the size and quantity may or may not be a concern for the airport."
  • Regarding Nuisance alarms: "It does track objects that are not humans, but that was an added benefit to certain airport operations as birds, deer and other animals are not wanted guests on a runway. The biggest benefit to the system is that if there is an alarm it is automatically time stamped and logged, a monitor displays the target and the police officer can, by not touching anything, view to see if the alarm is something that requires attention."
  • Regarding performance: "I can say that the system after being in service for over a year is still providing a useful tool that is reliable and paying for itself."

He also stated that the system has been expanded to a second airport, bus garages, and a light rail system, which would lead us to believe the system is performing satisfactorily.

Conclusion

We found the integrator's insights to be far more useful than the study itself. A key factor was that the integrator was able to share experiences in the optimization stage and how it is used in daily operations. The lesson to take away - If you want a good reference, stay away from 'studies' and find an engineer who was involved in the details.

Comments : Members only. Login. or Join.

Related Reports

Clearview AI Alarm - NY Times Report Says "Might End Privacy" on Jan 20, 2020
Over the weekend, the NY Times released a report titled "The Secretive Company That Might End Privacy as We Know It" about a company named...
IPVM Opens 12,000 Sqft Testing Facility on Dec 16, 2019
IPVM is proud to announce the opening of the world's first video surveillance testing facility that will allow us to significantly expand our...
XNOR.ai Terminates Wyze on Nov 27, 2019
Wyze has shocked the industry again - this time with a major setback. Just months ago, Wyze triumphantly added AI analytics to their ~$20 cameras,...
AI Video Surveillance (Finally) Goes Mainstream In 2020 on Sep 03, 2019
While video surveillance analytics has been promoted, hyped and lamented for nearly 20 years, next year, 2020, will be the year that it finally...
ZeroEyes Gun Detection Startup on Jul 16, 2019
A gun detection video analytics startup, ZeroEyes, is being led by a group of 6 former Navy SEALs, aiming to "save lives" by using AI to assist...
UK Facial Recognition Essex Errors Report on Jul 05, 2019
Facial recognition trials in the UK have generated significant controversy and debate over the past few years. This week, it flared again when Sky...
China Subway Facial Recognition System Examined on Jun 24, 2019
A China city of 6+ million people has installed facial recognition-enabled gates in subways, allowing commuters to enter stations by simply showing...
Carnegie Mellon AI Startup Zensors Profile on Jun 11, 2019
Zensors is a startup formed by Carnegie Mellon graduates from a Carnegie Mellon research project, offering customized models per camera that they...
Covert Facial Recognition Using Axis and Amazon By NYTimes on May 20, 2019
What if you took a 33MP Axis camera covering one of the busiest parks in the US and ran Amazon Facial Recognition against it? That is what the...
AI Video Tester Released on Apr 02, 2019
IPVM has released the world's first AI video tester that lets you see how various AI models (including from Amazon, Google, Microsoft and YOLO)...

Most Recent Industry Reports

Embedded Logix Thermal Temperature Detection System Examined on Apr 08, 2020
Embedded Logix has been producing thermal temperature measurement systems for industry and fire detection for over 10 years. Now, they are entering...
Micron 1 TB SD Cards Aim To Eliminate NVRs on Apr 08, 2020
Micron has boldly proclaimed their latest 1TB microSD "eliminates the need for network video recorders", targeting the growing market of...
US DoD Declares "Can No Longer Do Business" With Contractors Using Dahua, Hikvision, Huawei on Apr 08, 2020
The US Department of Defense has confirmed to IPVM that they fully support and intend to proceed with the NDAA 'blacklist clause' covering Dahua,...
IPVM's 12th Anniversary - Thank You! on Apr 07, 2020
IPVM is proud to celebrate it's 12 anniversary expanding our commitment to providing the industry independent and objective information on video...
Mobotix Thermal Body Temperature Detection Examined on Apr 07, 2020
Mobotix has jumped into the Coronavirus temperature detection market, but how do they compare to thermal incumbents like FLIR or ICI who have been...
Verkada Coronavirus Response: Free Temp Systems For Government and Health Care on Apr 07, 2020
Verkada has built a reputation on giving away things for free - free Yeti Tumblers, free trial cameras and now free temporary systems for...
Hikvision USA Refuses, Dahua USA Drives Forward With "Coronavirus Cameras" on Apr 07, 2020
Both have been federally banned, both sanctioned for human rights abuses but only one - Dahua - is taking aim at the booming "coronavirus cameras"...
China Surveillance Vulnerabilities Being Used To Attack China, Says China on Apr 07, 2020
While China video surveillance vulnerabilities have been much debated in the West in the past few years, China is now saying those vulnerabilities...
USA ICI Elevated Skin Temperature Detectors Examined on Apr 06, 2020
Infrared Cameras, Inc. (ICI) is aiming to help slow the spread of COVID-19 with "pinpoint accurate skin temperature measurement" using their...
Trade Groups Request NDAA Blacklist Delay Citing Coronavirus on Apr 06, 2020
Two trade groups representing government contractors have asked Congress to delay implementation of the NDAA's 'blacklist' clause from this August...