What Does 3VR Face Recognition Test Results Mean?

Author: John Honovich, Published on May 06, 2009

Recently, results from a facial recognition test for the Korean National Police were reported widely by 3VR of obtaining: "85-92 percent accuracy in recognizing and matching faces in a few crowded, highly-trafficked public train stations in Seoul."

The manufacturer claims these results are the best ever in any public video facial recognition study (compared to only 60% historically). One of the security magazine editors suggested that this was reason to "change the standard analytics copy

A Korean National Police spokesman praised the results, expressing interest and confidence in deploying facial recognition widely. It is always impressive to see end users expressing confidence in new surveillance technologies.

The Importance of Test Results

In the security industry, test results are key benchmarks for buyers and integrators in evaluating technologies. Improvements in test results can be important drivers for expanding the use of technologies. As such, accuracy and clarity in test results are essential in ensuring successful deployments.

In facial recognition test results, it is common that detailed reports are released and in-depth quantifications are offered. These tests are often administered by government agencies. (Read a facial recognition overview and examine the 2006 US government study for good examples and background.)

The Korean National Police Test

Three material differences should be considered when comparing the Korean National Police Test to other previous publicly released studies:

  • While the test was sponsored by the Korean government, the tester, SK Networks, is a corporation that sells technology products, not an independent testing agency.
  • The tester, SK Networks, is marketing 3VR products (see SK Networks' marketing website and a 20 page marketing proposal for 3VR products, both dated 2008 [Note: the website has since been taken down]). 3VR reports that they were not aware of these documents and that they have asked SK Networks to remove them from the Internet.
  • No details were publicly provided on false error rates, system setup, or testing methodology. Small variances in any of these elements can result in significantly different accuracy statistics. 3VR reports that a 170 page test result document is available in Korean (NIA pub number NIA 111-RER-07113/2007.12) which they are in the process of translating but have not been given permission to share publicly.

Concerns about The Test

3VR's claim that this test result is the best ever raises concerns. First, since we cannot read the test results, it cannot be assessed how comparable the implementation and design of this test is to leading global scientific studies. Secondly, unlike other facial recognition tests conducted by independent testing agencies, SK Networks sells technology products and has a general interest in promoting the purchase and use of such products.

Setting appropriate expectations for advanced technology projects is crucial, especially when statistics are being cited. This is why the community depends on scientific, independent studies that can be peer reviewed by outside experts.

It is promising to hear end users interested in using facial recognition and pleased with their test results. However, comparable statistical claims require far higher scrutiny and clarity than provided yet for the Korean NPA test results.

Related Reports

Hikvision Hires Crisis Communication Writer on Mar 21, 2017
Hikvision has hired a crisis communication writer as the company ramps up its efforts to deal with the 'crisis' it feels it is facing. 'Crisis...
Simplisafe Warns Customers About Alarm Fines on Mar 17, 2017
Simplisafe markets themselves as a 'better way' than traditional professional alarm companies. However, in one key way, Simplisafe hides the...
Bill Gates Funded Startup Evolv Gets $18M For The "Most Advanced Threat Detection System" on Mar 17, 2017
Evolv Technologies has just raised $18 million, bringing its total to $30M, for developing what the company says is the most advanced threat...
Alarm Panic Switches Tutorial on Mar 16, 2017
Panic switches allow silently triggering an alarm system when it is otherwise disarmed. In this tutorial we explain and contrast the 7 most common...
Genetec Comments on Washington DC MPD Hack on Mar 13, 2017
This January, the Washington DC police video surveillance system was hacked with ransomware, impacting 123 of 187 cameras. Last month, IPVM...
Avigilon Favorability Results on Feb 27, 2017
One of the fastest growing companies has turned into one of the rockiest, as cooling growth, management turnover and a roller coaster stock price...
Washington DC MPD's Surveillance Equipment on Feb 21, 2017
The Washington DC Metropolitan Police Department's surveillance system was hacked in January 2017. Two immediate questions were: Whose...
Hitachi Taking On Security Industry on Feb 09, 2017
Hitachi, bigger than Sony and Panasonic overall, with $89 billion USD 2016 total revenue, is expanding into the security industry. They are...
IPVM For PR / Marketing People on Jan 25, 2017
This post helps PR and Marketing people understand and productively work with IPVM (as much as possible given our independent, often critical,...
Sony Misleading Marketing Hides Cracked Backdoor on Jan 25, 2017
Sony is attempting to deemphasize the severity of the backdoor uncovered in Ipela cameras. Meanwhile, IPVM has verified that the root password for...

Most Recent Industry Reports

Axis Camera Vulnerabilities From Google Researcher Analyzed on Mar 23, 2017
A Google security researcher has reported 6 vulnerabilities for Axis cameras, affecting multiple models and firmware versions. In this report, we...
OpenEye Takes Aim At Exacq on Mar 23, 2017
First Milestone targeted Exacq with a takeover offer, and now OpenEye is gunning for them with an offer to swap out Exacq for their cloud-managed...
Lock Keyways For Access Control Guide on Mar 23, 2017
Lock keyways can be the difference between a lock working or not. Understanding keyways is important for access control. Indeed, a member recently...
Broken Browser Support for Video Surveillance on Mar 22, 2017
Modern web browsers have left the security industry behind. Current Chrome, Firefox, and Microsoft Edge browsers do not support NPAPI plugins,...
ADI Favorability Results on Mar 22, 2017
150 North American integrators provided feedback on 6 distributors, and why they do (or do not do) business with ADI. ADI is clearly a big name in...
1 Million Dahua Devices Exposed To Backdoor on Mar 22, 2017
Statistics show that 1 million Dahua devices are publicly exposed and vulnerable to the Dahua backdoor. Despite this, Dahua has downplayed the...
Hikvision Hires Crisis Communication Writer on Mar 21, 2017
Hikvision has hired a crisis communication writer as the company ramps up its efforts to deal with the 'crisis' it feels it is facing. 'Crisis...
Glass Break Sensor Tutorial on Mar 21, 2017
Burglars often break glass windows to get into a house. Using glass break detectors in conjunction with alarm contacts is a good way to protect the...
Unikey Smart Phone Access Control Platform on Mar 21, 2017
More and more people carry smart phones. Many think this could replace the conventional key or card for access control. However, using a phone...
Hikvision Attacks IPVM on Mar 20, 2017
Hikvision has attacked IPVM repeatedly over the last month, both in the international press and in its dealer communications. Attacks Listed On...

The world's leading video surveillance information source, IPVM provides the best reporting, testing and training for 10,000+ members globally. Dedicated to independent and objective information, we uniquely refuse any and all advertisements, sponsorship and consulting from manufacturers.

About | FAQ | Contact