What Does 3VR Face Recognition Korean Results Mean?

By: John Honovich, Published on May 06, 2009

Recently, results from a facial recognition test for the Korean National Police were reported [link no longer available] widely by 3VR of obtaining: "85-92 percent accuracy in recognizing and matching faces in a few crowded, highly-trafficked public train stations in Seoul."

The manufacturer claims these results are the best ever in any public video facial recognition study (compared to only 60% historically). One of the security magazine editors suggested that this was reason to "change the standard analytics copy

A Korean National Police spokesman praised the results, expressing interest and confidence in deploying facial recognition widely. It is always impressive to see end users expressing confidence in new surveillance technologies.

The Importance of Test Results

In the security industry, test results are key benchmarks for buyers and integrators in evaluating technologies. Improvements in test results can be important drivers for expanding the use of technologies. As such, accuracy and clarity in test results are essential in ensuring successful deployments.

In facial recognition test results, it is common that detailed reports are released and in-depth quantifications are offered. These tests are often administered by government agencies. (Read a facial recognition overview and examine the 2006 US government study [link no longer available] for good examples and background.)

The Korean National Police Test

Three material differences should be considered when comparing the Korean National Police Test to other previous publicly released studies:

  • While the test was sponsored by the Korean government, the tester, SK Networks, is a corporation that sells technology products, not an independent testing agency.
  • The tester, SK Networks, is marketing 3VR products (see SK Networks' marketing website and a 20 page marketing proposal for 3VR products, both dated 2008 [Note: the website has since been taken down]). 3VR reports that they were not aware of these documents and that they have asked SK Networks to remove them from the Internet.
  • No details were publicly provided on false error rates, system setup, or testing methodology. Small variances in any of these elements can result in significantly different accuracy statistics. 3VR reports that a 170 page test result document is available in Korean (NIA pub number NIA 111-RER-07113/2007.12) which they are in the process of translating but have not been given permission to share publicly.

Concerns about The Test

3VR's claim that this test result is the best ever raises concerns. First, since we cannot read the test results, it cannot be assessed how comparable the implementation and design of this test is to leading global scientific studies. Secondly, unlike other facial recognition tests conducted by independent testing agencies, SK Networks sells technology products and has a general interest in promoting the purchase and use of such products.

Setting appropriate expectations for advanced technology projects is crucial, especially when statistics are being cited. This is why the community depends on scientific, independent studies that can be peer reviewed by outside experts.

It is promising to hear end users interested in using facial recognition and pleased with their test results. However, comparable statistical claims require far higher scrutiny and clarity than provided yet for the Korean NPA test results.

Related Reports

Hikvision ColorVu is Smart Marketing on Oct 03, 2019
Hikvision ColorVu (see IPVM test results) is smart marketing, a lesson to be learned by competitors and a rising trend. Inside this note, we...
Critiquing Carnegie's AI Surveillance Paper on Sep 25, 2019
The Carnegie Endowment has issued an ambitious paper on the Global Expansion of AI Surveillance. While its aim is applaudable, the paper has...
Directory of 70 Video Surveillance Startups on Sep 18, 2019
This directory provides a list of video surveillance startups to help you see and research what companies are new or not yet broadly known. 2019...
ASIS GSX 2019 Final Show Report on Sep 12, 2019
IPVM went to Chicago for ASIS GSX 2019, with many exhibitors disappointed about traffic and the exhibitor schedule changing next year. However,...
UK Facewatch GDPR Compliance Questioned on Aug 27, 2019
Even as the GDPR strictly regulates biometrics, a UK company called Facewatch is selling anti-shoplifter facial recognition systems to hundreds of...
Suprema Biometric Mass Leak Examined on Aug 19, 2019
While Suprema is rarely discussed even within the physical security market, the South Korean biometrics manufacturer made global news this past...
Biometrics Usage Statistics 2019 on Aug 13, 2019
Biometrics are commonly used in phones, but how frequently are they used for access? 150+ integrators told us how often they use biometrics,...
Hikvision Global News Reports Directory on Aug 11, 2019
Hikvision has received the most global news reporting of any video surveillance company, ever, ranging from the WSJ, the Financial Times, Reuters,...
"Stats Don't Lie" Says Deceptive IFSEC on Jul 30, 2019
While IFSEC has declared #statsdontlie and trumpeted seemingly skyrocketing visitor numbers, they are decieving about their show's problems. On...
ZeroEyes Gun Detection Startup on Jul 16, 2019
A gun detection video analytics startup, ZeroEyes, is being led by a group of 6 former Navy SEALs, aiming to "save lives" by using AI to assist...

Most Recent Industry Reports

Camera Focusing Tutorial on Oct 14, 2019
Camera focus is fundamental to quality imaging. Mistakes can significantly reduce details, making cameras less effective. In this guide, we...
"UL Has Blood On Their Hands" Alleges The Interceptor / Keith Jentoft on Oct 14, 2019
"UL has blood on their hands" alleges Keith Jentoft of "The Interceptor Project". We examined The Interceptor in-depth last year, see: The...
Access Control Course Fall 2019 - Save $50 Last Chance on Oct 14, 2019
Register Now - Fall 2019 Access Control Course. Save $50 through October 10th. Thursday, October 17th is the last day to register. IPVM offers...
Axis HD Analog Encoder Tested on Oct 11, 2019
Two years after declaring "Everything is IP", Axis has released their first HD analog encoder, the P7304, with support for AHD, CVI, TVI, and SD...
Dahua Celebrates PRC 70th Wearing Communist Party Hammer and Sickle on Oct 11, 2019
Dahua celebrated the PRC's 70th anniversary with a video of various Dahua employees wearing China Communist Party hammer and sickle pins as shown...
Last Chance - Register Now - October 2019 IP Networking Course on Oct 10, 2019
Last Chance - Register Now - Fall 2019 IP Networking Course. The course starts next week. This is the only networking course designed...
Network Optix NxWitness 4.0 Tested on Oct 10, 2019
Network Optix released Nx Witness 4.0, proclaiming new features like a deep learning analytics metadata SDK, increased H.265 support, and UX...
HID Fingerprint Reader Tested on Oct 09, 2019
HID has released their first access reader to use Lumidigm optical sensors, that touts it 'works with anyone, anytime, anywhere'. We bought and...
Hikvision And Dahua Sanctioned For Human Rights Abuses on Oct 07, 2019
In a groundbreaking move that will have drastic consequences across the video surveillance market, Dahua and Hikvision have been sanctioned by the...