What Does 3VR Face Recognition Test Results Mean?

Author: John Honovich, Published on May 06, 2009

Recently, results from a facial recognition test for the Korean National Police were reported widely by 3VR of obtaining: "85-92 percent accuracy in recognizing and matching faces in a few crowded, highly-trafficked public train stations in Seoul."

The manufacturer claims these results are the best ever in any public video facial recognition study (compared to only 60% historically). One of the security magazine editors suggested that this was reason to "change the standard analytics copy

A Korean National Police spokesman praised the results, expressing interest and confidence in deploying facial recognition widely. It is always impressive to see end users expressing confidence in new surveillance technologies.

The Importance of Test Results

In the security industry, test results are key benchmarks for buyers and integrators in evaluating technologies. Improvements in test results can be important drivers for expanding the use of technologies. As such, accuracy and clarity in test results are essential in ensuring successful deployments.

In facial recognition test results, it is common that detailed reports are released and in-depth quantifications are offered. These tests are often administered by government agencies. (Read a facial recognition overview and examine the 2006 US government study for good examples and background.)

The Korean National Police Test

Three material differences should be considered when comparing the Korean National Police Test to other previous publicly released studies:

  • While the test was sponsored by the Korean government, the tester, SK Networks, is a corporation that sells technology products, not an independent testing agency.
  • The tester, SK Networks, is marketing 3VR products (see SK Networks' marketing website and a 20 page marketing proposal for 3VR products, both dated 2008 [Note: the website has since been taken down]). 3VR reports that they were not aware of these documents and that they have asked SK Networks to remove them from the Internet.
  • No details were publicly provided on false error rates, system setup, or testing methodology. Small variances in any of these elements can result in significantly different accuracy statistics. 3VR reports that a 170 page test result document is available in Korean (NIA pub number NIA 111-RER-07113/2007.12) which they are in the process of translating but have not been given permission to share publicly.

Concerns about The Test

3VR's claim that this test result is the best ever raises concerns. First, since we cannot read the test results, it cannot be assessed how comparable the implementation and design of this test is to leading global scientific studies. Secondly, unlike other facial recognition tests conducted by independent testing agencies, SK Networks sells technology products and has a general interest in promoting the purchase and use of such products.

Setting appropriate expectations for advanced technology projects is crucial, especially when statistics are being cited. This is why the community depends on scientific, independent studies that can be peer reviewed by outside experts.

It is promising to hear end users interested in using facial recognition and pleased with their test results. However, comparable statistical claims require far higher scrutiny and clarity than provided yet for the Korean NPA test results.

Related Reports

Hikvision Chinese Government Owner CETHIK Exposed on Nov 20, 2017
Hikvision deceives about its Chinese government ownership. Contrary to their claims about 'independence' and simply having 'shareholders' that are...
Panasonic Unified Surveillance Strategy Analyzed on Nov 17, 2017
Panasonic is now a "Unified Surveillance" offering, as their ASIS 2017 booth proclaimed: Looking to make a comeback in the security industry,...
Hikvision China Criticizes The WSJ on Nov 15, 2017
Hikvision, through the Chinese government's authoritative news service, has criticized the WSJ investigation into Hikvision. In this...
WSJ Investigates Hikvision on Nov 13, 2017
The Wall Street Journal (WSJ) has released a detailed investigation into Hikvision's government ownership and cybersecurity problems, hitting the...
Honeywell Crowdfunding Bombs on Nov 08, 2017
100,000+ employees and $40,000,000,000 in annual revenue. That's Honeywell. ~200 backers and ~$67,000 in funding in a week. That's Honeywell...
Dahua USA Removes Head of Marketing on Nov 06, 2017
Dahua USA's marketing and just their overall business organization has been a mess. These guys could not sell life jackets on the Titanic. But,...
Ingram Micro: The Blind Lead The Blind on Nov 02, 2017
Ingram Micro, as a huge as they are overall, with $40+ billion in annual sales, has never been a force in physical security, despite, or perhaps...
The Milestone Personal Demo Marketing Campaign on Oct 30, 2017
Waste of money or emarketing innovation? In a program being heavily promoted on their social media channels Milestone is offering to...
Genetec Citigraf Big Data Platform Analyzed on Oct 26, 2017
A big industry debate is whether VMS software is now a commodity. However, Genetec believes it can charge a substantial premium for higher-end...
Hikvision Admits Backdoor 'PR Issue' on Oct 24, 2017
Hikvision is admitting a problem. The backdoor itself is evidently not the problem for them. The problem, according to Hikvision, is a public...

Most Recent Industry Reports

Camera Multi-Streaming Usage on Nov 22, 2017
IP cameras typically support multiple streams, allowing a single camera to transmit multiple streams at different resolutions, frame rates and even...
Law Breaking Longse Enters USA on Nov 22, 2017
Longse has established itself as world class, at least in spamming the industry, ripping off Milestone and Video Insight as well as Hikvision. But...
Amazon Key In-Home Package Delivery Examined on Nov 21, 2017
Interesting idea or invitation for criminals to rob you? Amazon's recent announcement of Key, a service that will help manage visitors, welcoming...
Top Maglock Provider Warns Against Using Maglocks on Nov 21, 2017
Do not buy my company's product. It sounds strange indeed, but a senior Allegion consultant stated that maglocks should not be used in common...
CBR vs VBR vs MBR - Surveillance Streaming on Nov 21, 2017
How you stream video has a major impact on quality and bandwidth. And it is not simply CODEC choice (e.g., H.264 vs H.265). Regardless of the...
Hikvision Chinese Government Owner CETHIK Exposed on Nov 20, 2017
Hikvision deceives about its Chinese government ownership. Contrary to their claims about 'independence' and simply having 'shareholders' that are...
Dahua Hard-Coded Credentials Vulnerability on Nov 20, 2017
A newly discovered Dahua backdoor is described by the researcher discovering it as: not the result of an accidental logic error or poor...
Panasonic Unified Surveillance Strategy Analyzed on Nov 17, 2017
Panasonic is now a "Unified Surveillance" offering, as their ASIS 2017 booth proclaimed: Looking to make a comeback in the security industry,...
Amazon Cloud Cam Is Poor (Tested) on Nov 17, 2017
Retail behemoth Amazon has entered the surveillance market with the Amazon Cloud Cam, the eyes of its just-announced Amazon Key delivery...
Nest Secure Alarm System Tested on Nov 16, 2017
Google's expansion continues, this time into home security with their Nest subsidiary's move into alarm systems. They paid more than a...

The world's leading video surveillance information source, IPVM provides the best reporting, testing and training for 10,000+ members globally. Dedicated to independent and objective information, we uniquely refuse any and all advertisements, sponsorship and consulting from manufacturers.

About | FAQ | Contact