What Does 3VR Face Recognition Test Results Mean?

Author: John Honovich, Published on May 06, 2009

Recently, results from a facial recognition test for the Korean National Police were reported widely by 3VR of obtaining: "85-92 percent accuracy in recognizing and matching faces in a few crowded, highly-trafficked public train stations in Seoul."

The manufacturer claims these results are the best ever in any public video facial recognition study (compared to only 60% historically). One of the security magazine editors suggested that this was reason to "change the standard analytics copy"

A Korean National Police spokesman praised the results, expressing interest and confidence in deploying facial recognition widely. It is always impressive to see end users expressing confidence in new surveillance technologies.

The Importance of Test Results

In the security industry, test results are key benchmarks for buyers and integrators in evaluating technologies. Improvements in test results can be important drivers for expanding the use of technologies. As such, accuracy and clarity in test results are essential in ensuring successful deployments.

In facial recognition test results, it is common that detailed reports are released and in-depth quantifications are offered. These tests are often administered by government agencies. (Read a facial recognition overview and examine the 2006 US government study for good examples and background.)

The Korean National Police Test

Three material differences should be considered when comparing the Korean National Police Test to other previous publicly released studies:

  • While the test was sponsored by the Korean government, the tester, SK Networks, is a corporation that sells technology products, not an independent testing agency.
  • The tester, SK Networks, is marketing 3VR products (see SK Networks' marketing website and a 20 page marketing proposal for 3VR products, both dated 2008 [Note: the website has since been taken down]). 3VR reports that they were not aware of these documents and that they have asked SK Networks to remove them from the Internet.
  • No details were publicly provided on false error rates, system setup, or testing methodology. Small variances in any of these elements can result in significantly different accuracy statistics. 3VR reports that a 170 page test result document is available in Korean (NIA pub number NIA 111-RER-07113/2007.12) which they are in the process of translating but have not been given permission to share publicly.

Concerns about The Test

3VR's claim that this test result is the best ever raises concerns. First, since we cannot read the test results, it cannot be assessed how comparable the implementation and design of this test is to leading global scientific studies. Secondly, unlike other facial recognition tests conducted by independent testing agencies, SK Networks sells technology products and has a general interest in promoting the purchase and use of such products.

Setting appropriate expectations for advanced technology projects is crucial, especially when statistics are being cited. This is why the community depends on scientific, independent studies that can be peer reviewed by outside experts.

It is promising to hear end users interested in using facial recognition and pleased with their test results. However, comparable statistical claims require far higher scrutiny and clarity than provided yet for the Korean NPA test results.

Related Reports

ADI Battles Manufacturer Partners on Jan 11, 2017
ADI is battling their manufacturer partners, building up their own competitive house brand W-Box, while manufacturers still fund ADI's business...
Exacq 3 Co-Founders Exit on Jan 10, 2017
Exacq co-founders David Underwood and Daniel Rittman have exited the company. UPDATE: And a 3rd co-founder is now also leaving. In this note,...
How Axis Can Beat Hikvision on Jan 09, 2017
Hikvision has rocked the industry globally, bullying former video surveillance revenue leader Axis even in the Western world. What can Axis...
Introduction To Burglar Alarm Systems on Jan 04, 2017
While alarm systems are popular, balancing between the right level of protection, the appropriate components and an acceptable price can be very...
Dahua New Marketing Head Hired From Hanwha / Samsung on Jan 04, 2017
Dahua is bad at marketing. And they are finally taking a step to correct that, hiring Janet Fenner, who joins after 6+ years at Samsung / Hanwha...
Axis Launches A Blog on Dec 22, 2016
Is it 2007 or 2017? Because Axis Communications has just launched a blog. Are they 10 years too late or just right? In this post, we examine...
FLIR Acquires Drone Manufacturer For $134M on Dec 01, 2016
FLIR has acquired Prox Dynamics, a Norwegian maker of small military-grade drones, for $134M.  FLIR president Andy Teich provided additional...
3VR Sales And Marketing Execs Out, Company Shifting on Nov 22, 2016
Two executives are out at 3VR, both after relatively short tenures at the company. 3VR's CEO says this is part of a shift to cover new markets....
Top Manufacturers Gaining and Losing 2016 on Nov 21, 2016
Hikvision has been the top gaining manufacturer in IPVM's annual integrator survey the past 2 years (see 2015 results and 2014 results). Can they...
Sony and Bosch Combine Forces on Nov 15, 2016
Two of the biggest brands in the video surveillance industry are joining forces. Following a wave of industry consolidation and the race to the...

Most Recent Industry Reports

Top ISC West 2017 Booth Trends on Jan 16, 2017
There are significant moves among many major manufacturers on the ISC West show floor, and very few prime spaces left unoccupied. In this report we...
Genetec Favorability Results on Jan 16, 2017
In the race to the bottom and flight to 'solutions', Genetec has taken a contrary path. The company remains independent, focusing up market,...
Free VMS Software Directory on Jan 13, 2017
Many Video Management Software (VMS) providers offer free versions, either open source, for a limited number of cameras or for a limited amount of...
Milestone Essential Subscription Removed on Jan 12, 2017
Subscriptions may be the future for Milestone and VMSes but not right now. Responding to negative feedback, Milestone has removed subscriptions...
Alarm Reporting Formats Overview on Jan 12, 2017
Alarm reporting formats are methods for communicating complicated information quickly, using plain old telephone service. It is still the main...
Amazon Sales of Hikvision China Cameras on Jan 12, 2017
Hikvision has become widely available in the US, including on popular retail outlets like Amazon, with over 2,600 results: 4MP Hikvision outdoor...
Last Day IP Networking Course January 2017 on Jan 12, 2017
This is the only networking course designed specifically for video surveillance professionals plus it includes live training, personal help and...
ONVIF Favorability Results on Jan 11, 2017
ONVIF has been one of the most debated aspects of the video surveillance industry. On the one hand, its aim to increase interoperability has been...
Honeywell Ademco Vista Intrusion System Tested on Jan 11, 2017
One of the biggest brands in security holds one of the most common intrusion lines. We bought and tested a Honeywell Vista 15P intrusion...
ADI Battles Manufacturer Partners on Jan 11, 2017
ADI is battling their manufacturer partners, building up their own competitive house brand W-Box, while manufacturers still fund ADI's business...

The world's leading video surveillance information source, IPVM provides the best reporting, testing and training for 10,000+ members globally. Dedicated to independent and objective information, we uniquely refuse any and all advertisements, sponsorship and consulting from manufacturers.

About | FAQ | Contact