What Does 3VR Face Recognition Korean Results Mean?

By: John Honovich, Published on May 06, 2009

Recently, results from a facial recognition test for the Korean National Police were reported [link no longer available] widely by 3VR of obtaining: "85-92 percent accuracy in recognizing and matching faces in a few crowded, highly-trafficked public train stations in Seoul."

The manufacturer claims these results are the best ever in any public video facial recognition study (compared to only 60% historically). One of the security magazine editors suggested that this was reason to "change the standard analytics copy

A Korean National Police spokesman praised the results, expressing interest and confidence in deploying facial recognition widely. It is always impressive to see end users expressing confidence in new surveillance technologies.

The Importance of Test Results

In the security industry, test results are key benchmarks for buyers and integrators in evaluating technologies. Improvements in test results can be important drivers for expanding the use of technologies. As such, accuracy and clarity in test results are essential in ensuring successful deployments.

In facial recognition test results, it is common that detailed reports are released and in-depth quantifications are offered. These tests are often administered by government agencies. (Read a facial recognition overview and examine the 2006 US government study [link no longer available] for good examples and background.)

The Korean National Police Test

Three material differences should be considered when comparing the Korean National Police Test to other previous publicly released studies:

  • While the test was sponsored by the Korean government, the tester, SK Networks, is a corporation that sells technology products, not an independent testing agency.
  • The tester, SK Networks, is marketing 3VR products (see SK Networks' marketing website and a 20 page marketing proposal for 3VR products, both dated 2008 [Note: the website has since been taken down]). 3VR reports that they were not aware of these documents and that they have asked SK Networks to remove them from the Internet.
  • No details were publicly provided on false error rates, system setup, or testing methodology. Small variances in any of these elements can result in significantly different accuracy statistics. 3VR reports that a 170 page test result document is available in Korean (NIA pub number NIA 111-RER-07113/2007.12) which they are in the process of translating but have not been given permission to share publicly.

Concerns about The Test

3VR's claim that this test result is the best ever raises concerns. First, since we cannot read the test results, it cannot be assessed how comparable the implementation and design of this test is to leading global scientific studies. Secondly, unlike other facial recognition tests conducted by independent testing agencies, SK Networks sells technology products and has a general interest in promoting the purchase and use of such products.

Setting appropriate expectations for advanced technology projects is crucial, especially when statistics are being cited. This is why the community depends on scientific, independent studies that can be peer reviewed by outside experts.

It is promising to hear end users interested in using facial recognition and pleased with their test results. However, comparable statistical claims require far higher scrutiny and clarity than provided yet for the Korean NPA test results.

Related Reports

2020 IP Networking Book Released on Jan 06, 2020
The new IP Networking Book 2020 is a 280 page in-depth guide that teaches you how IT and telecom technologies impact modern security systems,...
AI Video Tester Released on Apr 02, 2019
IPVM has released the world's first AI video tester that lets you see how various AI models (including from Amazon, Google, Microsoft and YOLO)...
Hikvision AI Problems Criticized By Chinese Publication on Apr 09, 2019
Hikvision's facial recognition works poorly, causes delays and worsens learning, according to a new investigation by one of China's leading...
Axis ARTPEC-7 Chip Release Examined on Apr 08, 2019
For years, Axis essentially de-promoted their own chips. Now, they are reversing course. Axis has announced ARTPEC-7, their latest chip, which...
IBM Shuts Down Intelligent Video Analytics on May 02, 2019
Abruptly, IBM has discontinued its Intelligent Video Analytics solution, it confirmed to IPVM. IBM's video analytics have faced recent media...
Manufacturer Favorability Guide 2019 on Jun 12, 2019
The 259 page PDF guide may be downloaded inside by all IPVM members. It includes our manufacturer favorability rankings and individual...
China Subway Facial Recognition System Examined on Jun 24, 2019
A China city of 6+ million people has installed facial recognition-enabled gates in subways, allowing commuters to enter stations by simply showing...
UK Facial Recognition Essex Errors Report on Jul 05, 2019
Facial recognition trials in the UK have generated significant controversy and debate over the past few years. This week, it flared again when Sky...
AI Video Surveillance (Finally) Goes Mainstream In 2020 on Sep 03, 2019
While video surveillance analytics has been promoted, hyped and lamented for nearly 20 years, next year, 2020, will be the year that it finally...
Critiquing Carnegie's AI Surveillance Paper on Sep 25, 2019
The Carnegie Endowment has issued an ambitious paper on the Global Expansion of AI Surveillance. While its aim is applaudable, the paper has...

Most Recent Industry Reports

Dahua CEO Is Out on Feb 28, 2020
Dahua CEO Li Ke has resigned less than 3 years after he was brought in from Huawei to professionalize management. Inside this note, we examine...
Hazardous & Explosion Proof Access Control Tutorial on Feb 27, 2020
Controlling access to hazardous environments requires equipment meeting specific ratings that certify they will not start fires or will not...
Motorola / Avigilon Drops ISC West on Feb 26, 2020
Motorola Solutions has pulled out of ISC West 2020 effective immediately, because of coronavirus concerns, IPVM has learned. This is done amidst...
Cancel or Not? Industry Split Over ISC West on Feb 26, 2020
The industry is split, polarized, over whether ISC West 2020 should run or be canceled. New IPVM survey results of 400+ respondents show heated...
Coronavirus Hits Sony, Bosch Says Switch on Feb 26, 2020
Sony's fall in video surveillance has been severe over the past decade. Now, they may be done. In this note, we examine Bosch's new...
Video Surveillance Cameras 101 on Feb 25, 2020
Cameras come in many shapes, sizes and specifications. This 101 examines the basics of cameras and features used in 2020. In this report, we...
Favorite Video Analytic Manufacturers 2020 on Feb 25, 2020
Video analytics is now as hot as ever, driven by the excitement of advancing deep learning offers. But what are actually integrator's...
Latest London Police Facial Recognition Suffers Serious Issues on Feb 24, 2020
On February 20, IPVM visited another live face rec deployment by London police, but this time the system was thwarted by technical problems and...
Masks Cause Major Facial Recognition Problems on Feb 24, 2020
Coronavirus is spurring an increase in the use of medical masks, which new IPVM test results show cause major problems for facial recognition...
Every VMS Will Become a VSaaS on Feb 21, 2020
VMS is ending. Soon every VMS will be a VSaaS. Competitive dynamics will be redrawn. What does this mean? VMS Historically...