What Does 3VR Face Recognition Korean Results Mean?

By: John Honovich, Published on May 06, 2009

Recently, results from a facial recognition test for the Korean National Police were reported [link no longer available] widely by 3VR of obtaining: "85-92 percent accuracy in recognizing and matching faces in a few crowded, highly-trafficked public train stations in Seoul."

The manufacturer claims these results are the best ever in any public video facial recognition study (compared to only 60% historically). One of the security magazine editors suggested that this was reason to "change the standard analytics copy

A Korean National Police spokesman praised the results, expressing interest and confidence in deploying facial recognition widely. It is always impressive to see end users expressing confidence in new surveillance technologies.

The Importance of Test Results

In the security industry, test results are key benchmarks for buyers and integrators in evaluating technologies. Improvements in test results can be important drivers for expanding the use of technologies. As such, accuracy and clarity in test results are essential in ensuring successful deployments.

In facial recognition test results, it is common that detailed reports are released and in-depth quantifications are offered. These tests are often administered by government agencies. (Read a facial recognition overview and examine the 2006 US government study [link no longer available] for good examples and background.)

The Korean National Police Test

Three material differences should be considered when comparing the Korean National Police Test to other previous publicly released studies:

  • While the test was sponsored by the Korean government, the tester, SK Networks, is a corporation that sells technology products, not an independent testing agency.
  • The tester, SK Networks, is marketing 3VR products (see SK Networks' marketing website and a 20 page marketing proposal for 3VR products, both dated 2008 [Note: the website has since been taken down]). 3VR reports that they were not aware of these documents and that they have asked SK Networks to remove them from the Internet.
  • No details were publicly provided on false error rates, system setup, or testing methodology. Small variances in any of these elements can result in significantly different accuracy statistics. 3VR reports that a 170 page test result document is available in Korean (NIA pub number NIA 111-RER-07113/2007.12) which they are in the process of translating but have not been given permission to share publicly.

Concerns about The Test

3VR's claim that this test result is the best ever raises concerns. First, since we cannot read the test results, it cannot be assessed how comparable the implementation and design of this test is to leading global scientific studies. Secondly, unlike other facial recognition tests conducted by independent testing agencies, SK Networks sells technology products and has a general interest in promoting the purchase and use of such products.

Setting appropriate expectations for advanced technology projects is crucial, especially when statistics are being cited. This is why the community depends on scientific, independent studies that can be peer reviewed by outside experts.

It is promising to hear end users interested in using facial recognition and pleased with their test results. However, comparable statistical claims require far higher scrutiny and clarity than provided yet for the Korean NPA test results.

Related Reports

Bias In Facial Recognition Varies By Country, NIST Report Shows on Jul 15, 2020
While many argue that face recognition is inherently racist, results from one...
Facial Recognition 101 on Mar 18, 2020
Facial recognition interest, use and fear is increasing. This guide aims to...
Dahua, Hikvision, ZKTeco Face Mask Detection Shootout on Jun 19, 2020
Temperature tablets with face mask detection are one of the hottest trends in...
London Live Police Face Recognition Visited on Feb 13, 2020
London police have officially begun using live facial recognition in select...
Avigilon Face Mask Detection Tested on Jun 24, 2020
Face mask detection or, more specifically not wearing a face mask, is an...
HID Presents Signo on Apr 23, 2020
HID Global presented their new Signo readers they claim read farther and...
Face Masks Increase Face Recognition Errors Says NIST on Aug 04, 2020
COVID-19 has led to widespread facemask use, which as IPVM testing has shown...
Verkada People And Face Analytics Tested on Aug 16, 2019
This week, Verkada released "People Analytics", including face analytics that...
China Jaywalking Facial Recognition Guide on May 27, 2019
News reports touting the PRC's AI prowess often showcase facial recognition...
Hikvision Dual Lens Face Recognition Camera Tested on Nov 19, 2019
Hikvision's Dual Lens Facial Recognition camera, claims that it "adopts...
TVT / InVid Facial Recognition Tested on Mar 25, 2020
Facial recognition is frequently sold for thousands of dollars per channel...
Dartmouth College Deploys K3 Temperature Screening on Sep 29, 2020
While Dartmouth College has a $6+ billion endowment, the College has bought...
Integrated IR Camera Shootout 2020 - Avigilon, Axis, Bosch, Dahua, Hanwha, Hikvision, Panasonic, Uniview, Vivotek on Jan 30, 2020
The best and worst cameras tested in this IPVM shootout showed major...
30 Million Criminal Face Database Tested (Captis Intelligence) on Apr 27, 2020
30 million criminal mugshots are now available for facial recognition...
Hanwha Face Mask Detection Tested on Jul 01, 2020
Face mask detection or, more specifically lack-of-face-mask detection, is an...

Recent Reports

New Products Show Fall 2020 continues tomorrow with Genetec, Milestone, Avigilon, Microsoft and more! on Sep 29, 2020
IPVM's sixth online show continues tomorrow and will feature New Products...
Avigilon / Motorola VS Virtual ISC West on Sep 29, 2020
ISC West has historically been so dominant that no player would think of...
Dartmouth College Deploys K3 Temperature Screening on Sep 29, 2020
While Dartmouth College has a $6+ billion endowment, the College has bought...
Hanwha AI Object Detection Tested on Sep 28, 2020
Hanwha has added detection and classification of people, cars, clothing...
Favorite Access Control Manufacturers 2020 on Sep 28, 2020
200+ Integrators told IPVM "What is your favorite access control management...
OnTech Smart Services Partners With Google and Amazon To Compete With Integrators on Sep 25, 2020
A pain point for many homeowners to use consumer security and surveillance is...
The Future of Metalens For Video Surveillance Cameras - MIT / UMass / Immervision on Sep 25, 2020
Panoramic cameras using 'fisheye' lens have become commonplace in video...
Hikvision Sues Over Brazilian Airport Loss on Sep 24, 2020
Hikvision was excluded from a Brazilian airport project because it is owned...
China General Chamber of Commerce Calls Out US Politics on Sep 24, 2020
While US-China relations are at an all-time low, optimism about relations...
Verkada Disruptive Embedded Live Help on Sep 24, 2020
Call up your integrator? Have someone come by the next day? Verkada is...
IP Networking Course Fall 2020 - Last Chance - Register Now on Sep 23, 2020
Today is the last chance to register for the only IP networking course...
Drain Wire For Access Control Reader Tutorial on Sep 23, 2020
An easy-to-miss cabling specification plays a key role in access control, yet...
Norway Council of Ethics Finds Hikvision Human Rights Abuses "Ongoing" on Sep 23, 2020
Hikvision's involvement in "serious human rights abuse" in Xinjiang is...
IPVM Camera Calculator User Manual / Guide on Sep 23, 2020
Learn how to use the IPVM Camera Calculator (updated for Version 3.1). The...