What Does 3VR Face Recognition Test Results Mean?

Author: John Honovich, Published on May 06, 2009

Recently, results from a facial recognition test for the Korean National Police were reported widely by 3VR of obtaining: "85-92 percent accuracy in recognizing and matching faces in a few crowded, highly-trafficked public train stations in Seoul."

The manufacturer claims these results are the best ever in any public video facial recognition study (compared to only 60% historically). One of the security magazine editors suggested that this was reason to "change the standard analytics copy

A Korean National Police spokesman praised the results, expressing interest and confidence in deploying facial recognition widely. It is always impressive to see end users expressing confidence in new surveillance technologies.

The Importance of Test Results

In the security industry, test results are key benchmarks for buyers and integrators in evaluating technologies. Improvements in test results can be important drivers for expanding the use of technologies. As such, accuracy and clarity in test results are essential in ensuring successful deployments.

In facial recognition test results, it is common that detailed reports are released and in-depth quantifications are offered. These tests are often administered by government agencies. (Read a facial recognition overview and examine the 2006 US government study for good examples and background.)

The Korean National Police Test

Three material differences should be considered when comparing the Korean National Police Test to other previous publicly released studies:

  • While the test was sponsored by the Korean government, the tester, SK Networks, is a corporation that sells technology products, not an independent testing agency.
  • The tester, SK Networks, is marketing 3VR products (see SK Networks' marketing website and a 20 page marketing proposal for 3VR products, both dated 2008 [Note: the website has since been taken down]). 3VR reports that they were not aware of these documents and that they have asked SK Networks to remove them from the Internet.
  • No details were publicly provided on false error rates, system setup, or testing methodology. Small variances in any of these elements can result in significantly different accuracy statistics. 3VR reports that a 170 page test result document is available in Korean (NIA pub number NIA 111-RER-07113/2007.12) which they are in the process of translating but have not been given permission to share publicly.

Concerns about The Test

3VR's claim that this test result is the best ever raises concerns. First, since we cannot read the test results, it cannot be assessed how comparable the implementation and design of this test is to leading global scientific studies. Secondly, unlike other facial recognition tests conducted by independent testing agencies, SK Networks sells technology products and has a general interest in promoting the purchase and use of such products.

Setting appropriate expectations for advanced technology projects is crucial, especially when statistics are being cited. This is why the community depends on scientific, independent studies that can be peer reviewed by outside experts.

It is promising to hear end users interested in using facial recognition and pleased with their test results. However, comparable statistical claims require far higher scrutiny and clarity than provided yet for the Korean NPA test results.

Related Reports

Canon Launches World's Most Expensive IP Camera (ME20F-SHN) on Jan 09, 2018
Canon has launched the ME20F-SHN , likely the world's most expensive single imager, non-thermal, IP camera at ~$20,000. And Canon subsidiary...
Chinese Government Attacks Western Reports on Jan 03, 2018
The Chinese government is angry at the BBC and WSJ's reporting on Chinese video surveillance (see BBC Features Dahua and WSJ Investigates China's...
Washington DC Surveillance Hackers Arrested on Dec 29, 2017
The US Department of Justice has announced that "Two Romanian Suspects Charged With Hacking of Metropolitan Police Department Surveillance Cameras...
WSJ Investigates China's Total Surveillance State on Dec 26, 2017
The WSJ is continuing its investigation into Chinese video surveillance. Following up on last month's WSJ Investigation of Hikvision, the WSJ is...
2018 Top Sales Interest For Integrators on Dec 20, 2017
130+ integrators answered: This is a key question, especially for manufacturers, as it impacts what areas to focus more on and what...
BBC Features Dahua on Dec 13, 2017
Hikvision is not the only mega-Chinese video surveillance manufacturer getting global attention. Last month, the WSJ investigated Hikvision and now...
Testing $20 WyzeCam, The Money Losing Amazon Vet Startup on Dec 12, 2017
This startup is perfecting the old adage: We lose money on every sale, but make it up on volume But it is no joke. The company, Wyze Labs, is...
2018 Video Surveillance Cameras Overview on Dec 11, 2017
This report concisely explains the developments for surveillance cameras offered in 2017 and the state of offerings going into 2018, including...
Lighthouse Deep Learning Camera Tested on Dec 07, 2017
A Silicon Valley startup, Lighthouse, with a Stanford PhD CTO, has released a deep learning AI camera with 3D sensors for just $300. The company...
ASIS Dumps 'ASIS' For Show on Dec 06, 2017
After 60+ years, ASIS is dumping its eponymous show name and replacing it with 'GSX'. This is a classic marketing mistake. For a show struggling...

Most Recent Industry Reports

Chinese Government Hikvision Surveillance System On US Government Network on Jan 18, 2018
Hikvision, the Chinese government-owned manufacturer, has publicly claimed that their products are running on a US government network. Moreover,...
Camera Course - Last Day - Save $50 on Jan 18, 2018
Today is the last day to save $50 - register now. Learn video surveillance and get certified. Save $50 on the course, ending this Thursday the...
VSaaS Usage Statistics 2018 on Jan 18, 2018
VSaaS has been a 'next big thing' for more than a decade. The prospect of managing, storing and streaming video from the cloud rather than...
Vivint Streety Video Strengthens Door Knocking on Jan 17, 2018
Vivint is famous (or infamous depending on your perspective) for mastering large scale door to door selling. The company has skyrocketed from a...
Axis: "It’s A Question Of Trust And Who You Want To Be Associated With" on Jan 17, 2018
Who do you trust? Who do you want to be associated with? Axis is raising hard questions to start 2018. In this note, we examine these questions,...
Software House Vulnerability Allows Inside Attacker To Open Doors on Jan 17, 2018
A vulnerability in Software House IP-ACM modules allows an attacker to potentially unlock doors, or perform other actions, on affected systems....
'Defiant' Hikvision 'Strikes Back' At WSJ And US on Jan 16, 2018
The fight is on. Hikvision and their owner, the Chinese government, 'strikes back' against the Wall Street Journal and US politicians raising...
The 2018 Surveillance Industry Guide on Jan 16, 2018
The 300 page, 2018 Video Surveillance Industry Guide, covering the key events and the future of the video surveillance market, is now available,...
Edward Snowden Haven App Tested on Jan 16, 2018
Global coverage followed the December 2017 announcement that Edward Snowden was leading a team developing Haven, an app "that leverages on-device...
This High Schooler Is Excited About His Future Security Career on Jan 15, 2018
A common lament is that smart, young people have little interest in surveillance systems. In fact, discussions like Should Talented Young People...

The world's leading video surveillance information source, IPVM provides the best reporting, testing and training for 10,000+ members globally. Dedicated to independent and objective information, we uniquely refuse any and all advertisements, sponsorship and consulting from manufacturers.

About | FAQ | Contact