What Does 3VR Face Recognition Test Results Mean?

Author: John Honovich, Published on May 06, 2009

Recently, results from a facial recognition test for the Korean National Police were reported widely by 3VR of obtaining: "85-92 percent accuracy in recognizing and matching faces in a few crowded, highly-trafficked public train stations in Seoul."

The manufacturer claims these results are the best ever in any public video facial recognition study (compared to only 60% historically). One of the security magazine editors suggested that this was reason to "change the standard analytics copy

A Korean National Police spokesman praised the results, expressing interest and confidence in deploying facial recognition widely. It is always impressive to see end users expressing confidence in new surveillance technologies.

The Importance of Test Results

In the security industry, test results are key benchmarks for buyers and integrators in evaluating technologies. Improvements in test results can be important drivers for expanding the use of technologies. As such, accuracy and clarity in test results are essential in ensuring successful deployments.

In facial recognition test results, it is common that detailed reports are released and in-depth quantifications are offered. These tests are often administered by government agencies. (Read a facial recognition overview and examine the 2006 US government study for good examples and background.)

The Korean National Police Test

Three material differences should be considered when comparing the Korean National Police Test to other previous publicly released studies:

  • While the test was sponsored by the Korean government, the tester, SK Networks, is a corporation that sells technology products, not an independent testing agency.
  • The tester, SK Networks, is marketing 3VR products (see SK Networks' marketing website and a 20 page marketing proposal for 3VR products, both dated 2008 [Note: the website has since been taken down]). 3VR reports that they were not aware of these documents and that they have asked SK Networks to remove them from the Internet.
  • No details were publicly provided on false error rates, system setup, or testing methodology. Small variances in any of these elements can result in significantly different accuracy statistics. 3VR reports that a 170 page test result document is available in Korean (NIA pub number NIA 111-RER-07113/2007.12) which they are in the process of translating but have not been given permission to share publicly.

Concerns about The Test

3VR's claim that this test result is the best ever raises concerns. First, since we cannot read the test results, it cannot be assessed how comparable the implementation and design of this test is to leading global scientific studies. Secondly, unlike other facial recognition tests conducted by independent testing agencies, SK Networks sells technology products and has a general interest in promoting the purchase and use of such products.

Setting appropriate expectations for advanced technology projects is crucial, especially when statistics are being cited. This is why the community depends on scientific, independent studies that can be peer reviewed by outside experts.

It is promising to hear end users interested in using facial recognition and pleased with their test results. However, comparable statistical claims require far higher scrutiny and clarity than provided yet for the Korean NPA test results.

Related Reports

Avigilon Favorability Results on Feb 27, 2017
One of the fastest growing companies has turned into one of the rockiest, as cooling growth, management turnover and a roller coaster stock price...
Washington DC MPD's Surveillance Equipment on Feb 21, 2017
The Washington DC Metropolitan Police Department's surveillance system was hacked in January 2017. Two immediate questions were: Whose...
Hitachi Taking On Security Industry on Feb 09, 2017
Hitachi, bigger than Sony and Panasonic overall, with $89 billion USD 2016 total revenue, is expanding into the security industry. They are...
IPVM For PR / Marketing People on Jan 25, 2017
This post helps PR and Marketing people understand and productively work with IPVM (as much as possible given our independent, often critical,...
Sony Misleading Marketing Hides Cracked Backdoor on Jan 25, 2017
Sony is attempting to deemphasize the severity of the backdoor uncovered in Ipela cameras. Meanwhile, IPVM has verified that the root password for...
PR Firm Pleads Don't Scrap PR Spending on Jan 20, 2017
PR is not dying, warns pleads PR firm. Take 40+ year old industry PR firm LRG, who recently lamented the 'misconceptions' that: Traditional PR...
Dahua UnFavorability Results on Jan 19, 2017
Dahua, the mega-Chinese surveillance manufacturer not primarily owned by the Chinese government has been trying to break out of the shadow of...
ADI Battles Manufacturer Partners on Jan 11, 2017
ADI is battling their manufacturer partners, building up their own competitive house brand W-Box, while manufacturers still fund ADI's business...
Exacq 3 Co-Founders Exit on Jan 10, 2017
Exacq co-founders David Underwood and Daniel Rittman have exited the company. UPDATE: And a 3rd co-founder is now also leaving. In this note,...
How Axis Can Beat Hikvision on Jan 09, 2017
Hikvision has rocked the industry globally, bullying former video surveillance revenue leader Axis even in the Western world. What can Axis...

Most Recent Industry Reports

Hikvision Pyronix Releases 'Huge' New Panel on Feb 27, 2017
Hikvision's 2016 acquisition Pyronix has been touting their 'huge' new large system panel. Indeed, the new Euro 280 claims features that are better...
Bosch Favorability Results on Feb 27, 2017
Bosch is one of the most well known brands in the industry and they have combined recently with Sony in video surveillance. But how has Bosch...
ADT CEO: My Daughters Better At Installs Than Most 20 Year Techs on Feb 27, 2017
Times have changed. At the Barnes Buchanan conference, ADT's CEO Tim Whall made an interesting and, surely to some, controversial observation....
Avigilon Favorability Results on Feb 27, 2017
One of the fastest growing companies has turned into one of the rockiest, as cooling growth, management turnover and a roller coaster stock price...
Honeywell Sues Alarm.com For Violating Anti-Trust Laws on Feb 24, 2017
Is Alarm.com about to dominate the smart home software market? That is what Honeywell alleges in its new lawsuit, first reported by...
Axis: "Everything is IP" - False on Feb 24, 2017
Axis is congratulating itself, with executive Fredrick Nilsson declaring: "Now the conversion is all done and everything is IP and analog is...
Advertising Like Avigilon at the ISC West Airport on Feb 24, 2017
Avigilon has grabbed a lot of attention over the last few years advertising at the Las Vegas airport when attendees fly in. But how does that...
Artificial Intelligence Robot Assistant (ACTi) on Feb 23, 2017
Has artificial intelligence come to the video surveillance industry? ACTi has released 'SARA' which it bills as an 'AI assistant that brings...
Cutting Costs 70% Using Milestone With HD Analog on Feb 23, 2017
HD analog and enterprise VMSes are often thought of as being on opposite sides of the spectrum, with HD analog best for small jobs due to its low...
Dahua 4K HD Analog Cameras Announced on Feb 23, 2017
HD analog has been gaining popularity (even if Axis hopes otherwise). Last year, HD analog's max resolution doubled from 1080p to 4MP (see our 4MP...

The world's leading video surveillance information source, IPVM provides the best reporting, testing and training for 10,000+ members globally. Dedicated to independent and objective information, we uniquely refuse any and all advertisements, sponsorship and consulting from manufacturers.

About | FAQ | Contact