What Does 3VR Face Recognition Korean Results Mean?

By John Honovich, Published on May 06, 2009

Recently, results from a facial recognition test for the Korean National Police were reported [link no longer available] widely by 3VR of obtaining: "85-92 percent accuracy in recognizing and matching faces in a few crowded, highly-trafficked public train stations in Seoul."

The manufacturer claims these results are the best ever in any public video facial recognition study (compared to only 60% historically). One of the security magazine editors suggested that this was reason to "change the standard analytics copy

A Korean National Police spokesman praised the results, expressing interest and confidence in deploying facial recognition widely. It is always impressive to see end users expressing confidence in new surveillance technologies.

The Importance of Test Results

In the security industry, test results are key benchmarks for buyers and integrators in evaluating technologies. Improvements in test results can be important drivers for expanding the use of technologies. As such, accuracy and clarity in test results are essential in ensuring successful deployments.

In facial recognition test results, it is common that detailed reports are released and in-depth quantifications are offered. These tests are often administered by government agencies. (Read a facial recognition overview and examine the 2006 US government study [link no longer available] for good examples and background.)

The Korean National Police Test

Three material differences should be considered when comparing the Korean National Police Test to other previous publicly released studies:

  • While the test was sponsored by the Korean government, the tester, SK Networks, is a corporation that sells technology products, not an independent testing agency.
  • The tester, SK Networks, is marketing 3VR products (see SK Networks' marketing website and a 20 page marketing proposal for 3VR products, both dated 2008 [Note: the website has since been taken down]). 3VR reports that they were not aware of these documents and that they have asked SK Networks to remove them from the Internet.
  • No details were publicly provided on false error rates, system setup, or testing methodology. Small variances in any of these elements can result in significantly different accuracy statistics. 3VR reports that a 170 page test result document is available in Korean (NIA pub number NIA 111-RER-07113/2007.12) which they are in the process of translating but have not been given permission to share publicly.

Concerns about The Test

3VR's claim that this test result is the best ever raises concerns. First, since we cannot read the test results, it cannot be assessed how comparable the implementation and design of this test is to leading global scientific studies. Secondly, unlike other facial recognition tests conducted by independent testing agencies, SK Networks sells technology products and has a general interest in promoting the purchase and use of such products.

Setting appropriate expectations for advanced technology projects is crucial, especially when statistics are being cited. This is why the community depends on scientific, independent studies that can be peer reviewed by outside experts.

It is promising to hear end users interested in using facial recognition and pleased with their test results. However, comparable statistical claims require far higher scrutiny and clarity than provided yet for the Korean NPA test results.

Related Reports

Bias In Facial Recognition Varies By Country, NIST Report Shows on Jul 15, 2020
While many argue that face recognition is inherently racist, results from one...
Facial Recognition 101 on Mar 18, 2020
Facial recognition interest, use and fear is increasing. This guide aims to...
Dahua, Hikvision, ZKTeco Face Mask Detection Shootout on Jun 19, 2020
Temperature tablets with face mask detection are one of the hottest trends in...
London Live Police Face Recognition Visited on Feb 13, 2020
London police have officially begun using live facial recognition in select...
Avigilon Face Mask Detection Tested on Jun 24, 2020
Face mask detection or, more specifically not wearing a face mask, is an...
HID Presents Signo on Apr 23, 2020
HID Global presented their new Signo readers they claim read farther and...
Face Masks Increase Face Recognition Errors Says NIST on Aug 04, 2020
COVID-19 has led to widespread facemask use, which as IPVM testing has shown...
Hikvision Dual Lens Face Recognition Camera Tested on Nov 19, 2019
Hikvision's Dual Lens Facial Recognition camera, claims that it "adopts...
FaceFirst Problems And Layoff on Oct 01, 2020
FaceFirst, a US company and one of the oldest ongoing facial recognition...
Verkada People And Face Analytics Tested 2019 on Aug 16, 2019
This week, Verkada released "People Analytics", including face analytics that...
TVT / InVid Facial Recognition Tested on Mar 25, 2020
Facial recognition is frequently sold for thousands of dollars per channel...
Dartmouth College Deploys K3 Temperature Screening on Sep 29, 2020
While Dartmouth College has a $6+ billion endowment, the College has bought...
China Jaywalking Facial Recognition Guide on May 27, 2019
News reports touting the PRC's AI prowess often showcase facial recognition...
Integrated IR Camera Shootout 2020 - Avigilon, Axis, Bosch, Dahua, Hanwha, Hikvision, Panasonic, Uniview, Vivotek on Jan 30, 2020
The best and worst cameras tested in this IPVM shootout showed major...
30 Million Criminal Face Database Tested (Captis Intelligence) on Apr 27, 2020
30 million criminal mugshots are now available for facial recognition...

Recent Reports

Deceptive Meridian Temperature Tablets Endanger Public Safety on Oct 21, 2020
IPVM's testing of and investigation into Meridian Kiosk's temperature...
Honeywell 30 Series and Vivotek NVR Test on Oct 21, 2020
The NDAA ban has driven many users to look for low-cost NVRs not made by...
Ubiquiti Access Control Tested on Oct 21, 2020
Ubiquiti has become one of the most widely used wireless and switch providers...
Mexico Video Surveillance Market Overview 2020 on Oct 20, 2020
Despite being neighbors, there are key differences between the U.S. and...
Dahua Revenue Grows But Profits Down, Cause Unclear on Oct 20, 2020
While Dahua's overall revenue was up more than 12% in Q3 2020, a significant...
Illegal Hikvision Fever Screening Touted In Australia, Government Investigating, Temperature References Deleted on Oct 20, 2020
The Australian government told IPVM that they are investigating a Hikvision...
Panasonic Presents i-PRO Cameras and Video Analytics on Oct 19, 2020
Panasonic i-PRO presented its X-Series cameras and AI video analytics at the...
Augmented Reality (AR) Cameras From Hikvision and Dahua Examined on Oct 19, 2020
Hikvision, Dahua, and other China companies are marketing augmented reality...
18 TB Video Surveillance Drives (WD and Seagate) on Oct 19, 2020
Both Seagate and Western Digital recently announced 18TB hard drives...
Watrix Gait Recognition Profile on Oct 16, 2020
Watrix is the world's only gait recognition surveillance provider IPVM has...
Intel Presents Edge-to-Cloud Ecosystem for Video Analytics on Oct 16, 2020
Intel presented its processors and software toolkit for computer vision at...
Best Manufacturer Technical Support 2020 on Oct 16, 2020
5 manufacturers stood out as providing the best technical support to ~200...
Microsoft Azure Presents Live Video Analytics on Oct 15, 2020
Microsoft Azure presented its Live Video Analytics offering at the September...
Worst Manufacturer Technical Support 2020 on Oct 15, 2020
4 manufacturers stood out as providing the worst technical support to ~200...