What Does 3VR Face Recognition Test Results Mean?

Author: John Honovich, Published on May 06, 2009

Recently, results from a facial recognition test for the Korean National Police were reported widely by 3VR of obtaining: "85-92 percent accuracy in recognizing and matching faces in a few crowded, highly-trafficked public train stations in Seoul."

The manufacturer claims these results are the best ever in any public video facial recognition study (compared to only 60% historically). One of the security magazine editors suggested that this was reason to "change the standard analytics copy

A Korean National Police spokesman praised the results, expressing interest and confidence in deploying facial recognition widely. It is always impressive to see end users expressing confidence in new surveillance technologies.

The Importance of Test Results

In the security industry, test results are key benchmarks for buyers and integrators in evaluating technologies. Improvements in test results can be important drivers for expanding the use of technologies. As such, accuracy and clarity in test results are essential in ensuring successful deployments.

In facial recognition test results, it is common that detailed reports are released and in-depth quantifications are offered. These tests are often administered by government agencies. (Read a facial recognition overview and examine the 2006 US government study for good examples and background.)

The Korean National Police Test

Three material differences should be considered when comparing the Korean National Police Test to other previous publicly released studies:

  • While the test was sponsored by the Korean government, the tester, SK Networks, is a corporation that sells technology products, not an independent testing agency.
  • The tester, SK Networks, is marketing 3VR products (see SK Networks' marketing website and a 20 page marketing proposal for 3VR products, both dated 2008 [Note: the website has since been taken down]). 3VR reports that they were not aware of these documents and that they have asked SK Networks to remove them from the Internet.
  • No details were publicly provided on false error rates, system setup, or testing methodology. Small variances in any of these elements can result in significantly different accuracy statistics. 3VR reports that a 170 page test result document is available in Korean (NIA pub number NIA 111-RER-07113/2007.12) which they are in the process of translating but have not been given permission to share publicly.

Concerns about The Test

3VR's claim that this test result is the best ever raises concerns. First, since we cannot read the test results, it cannot be assessed how comparable the implementation and design of this test is to leading global scientific studies. Secondly, unlike other facial recognition tests conducted by independent testing agencies, SK Networks sells technology products and has a general interest in promoting the purchase and use of such products.

Setting appropriate expectations for advanced technology projects is crucial, especially when statistics are being cited. This is why the community depends on scientific, independent studies that can be peer reviewed by outside experts.

It is promising to hear end users interested in using facial recognition and pleased with their test results. However, comparable statistical claims require far higher scrutiny and clarity than provided yet for the Korean NPA test results.

Related Reports

Avigilon Touting 'Made In America' on Sep 18, 2017
Canadian manufacturer Avigilon, who completed a US manufacturing facility in 2015, is now running a marketing campaign touting 'Made In America',...
Genetec Launches Community Connect Examined on Sep 14, 2017
Genetec has done best in large-scale, enterprise systems and relatively worse in smaller systems such as SMB. Now, Genetec is launching...
Pelco CEO Is Out on Sep 05, 2017
Less than 2 years after taking the job, Pelco's CEO is out. Pelco's rollercoaster continues. In this note, we examine the move, share feedback...
Neurala Deep Learning Platform Profile on Aug 31, 2017
Find lost children? If Neurala has its way with its recent partnership with Motorola, that is the aim as shown in their video below: Neurala has...
BBC Featured Deep Learning Face Recognition Analyzed (Digital Barriers) on Aug 30, 2017
The UK's largest broadcaster, BBC, recently featured Digital Barriers (also based in the UK) on a segment highlighting face recognition technology....
Favorite SMB Camera Manufacturers 2017 on Aug 25, 2017
Small to Medium Businesses (SMB) have been hotly pursued by lower-cost entrants over the past few years. Nowhere has that been as strong a factor...
Vivotek Acquisition Offer From Delta on Aug 10, 2017
One of the largest remaining independent video surveillance manufacturers has received an offer to become majority owned by Delta Group, a Taiwan...
Competing Against ADT on Jul 20, 2017
ADT is one of the biggest players in the security industry, with ~$4 billion revenue. In 2017, they were acquired / merged with Protection...
Hikvision USA Head of Cybersecurity Exits on Jul 18, 2017
Hikvision USA's Head of Cybersecurity has exited the company. In this note, we review the move, share Hikvision's feedback and examine the...
Nest Cam IQ Tested on Jul 10, 2017
Nest has released their latest entry in their camera line, the Nest Cam IQ, touting 4K "Supersight", facial recognition, "HD audio", invisible IR,...

Most Recent Industry Reports

Genetec Launches Cloud Access Control (Synergis SaaS) on Sep 21, 2017
Genetec's cloud everything expansion continues, with their announcement of Synergis SaaS edition, joining their cloud video offering Stratocast,...
Genetec CEO Warns Against Insider Threats on Sep 21, 2017
With Dahua and Hikvision cybersecurity issues becoming indisputable, a new counter has emerged. Just put them behind a firewall, buy cheap...
Automatic Door Operators For Access Tutorial on Sep 20, 2017
Opening and closing doors might sound simple, but it takes a high-tech piece of door hardware to pull it off. Integrating automatic door operators...
'Clowns' Allege Ubiquiti 'Completely Fraudulent' on Sep 20, 2017
A short seller has alleged Ubiquiti is 'completely fraudulent'. Ubiquiti's CEO has responded calling them 'clowns'. Here is the short...
Avigilon 'Blue' Cloud Entry Examined on Sep 19, 2017
Avigilon is moving to the cloud. The company announced their Avigilon Blue platform, designed to be a web-managed surveillance system, utilizing...
HID Buys Mercury Security on Sep 19, 2017
One of the biggest access control deals in years. Mercury Security, the most widely used access hardware OEM, and partner to 20+ manufacturers,...
Hikvision Backdoor Exploit on Sep 18, 2017
Full disclosure to the Hikvision backdoor has been released, allowing easy exploit of vulnerable Hikvision IP cameras. As the researcher, Monte...
Avigilon Touting 'Made In America' on Sep 18, 2017
Canadian manufacturer Avigilon, who completed a US manufacturing facility in 2015, is now running a marketing campaign touting 'Made In America',...
Cloud Guy Prints Book, Misses Irony on Sep 15, 2017
On-premise security systems are dead. But $75 print books are alive and well. Such are the lessons from Brivo's CEO new book "The Five...

The world's leading video surveillance information source, IPVM provides the best reporting, testing and training for 10,000+ members globally. Dedicated to independent and objective information, we uniquely refuse any and all advertisements, sponsorship and consulting from manufacturers.

About | FAQ | Contact