What Does 3VR Face Recognition Test Results Mean?

Author: John Honovich, Published on May 06, 2009

Recently, results from a facial recognition test for the Korean National Police were reported widely by 3VR of obtaining: "85-92 percent accuracy in recognizing and matching faces in a few crowded, highly-trafficked public train stations in Seoul."

The manufacturer claims these results are the best ever in any public video facial recognition study (compared to only 60% historically). One of the security magazine editors suggested that this was reason to "change the standard analytics copy

A Korean National Police spokesman praised the results, expressing interest and confidence in deploying facial recognition widely. It is always impressive to see end users expressing confidence in new surveillance technologies.

The Importance of Test Results

In the security industry, test results are key benchmarks for buyers and integrators in evaluating technologies. Improvements in test results can be important drivers for expanding the use of technologies. As such, accuracy and clarity in test results are essential in ensuring successful deployments.

In facial recognition test results, it is common that detailed reports are released and in-depth quantifications are offered. These tests are often administered by government agencies. (Read a facial recognition overview and examine the 2006 US government study for good examples and background.)

The Korean National Police Test

Three material differences should be considered when comparing the Korean National Police Test to other previous publicly released studies:

  • While the test was sponsored by the Korean government, the tester, SK Networks, is a corporation that sells technology products, not an independent testing agency.
  • The tester, SK Networks, is marketing 3VR products (see SK Networks' marketing website and a 20 page marketing proposal for 3VR products, both dated 2008 [Note: the website has since been taken down]). 3VR reports that they were not aware of these documents and that they have asked SK Networks to remove them from the Internet.
  • No details were publicly provided on false error rates, system setup, or testing methodology. Small variances in any of these elements can result in significantly different accuracy statistics. 3VR reports that a 170 page test result document is available in Korean (NIA pub number NIA 111-RER-07113/2007.12) which they are in the process of translating but have not been given permission to share publicly.

Concerns about The Test

3VR's claim that this test result is the best ever raises concerns. First, since we cannot read the test results, it cannot be assessed how comparable the implementation and design of this test is to leading global scientific studies. Secondly, unlike other facial recognition tests conducted by independent testing agencies, SK Networks sells technology products and has a general interest in promoting the purchase and use of such products.

Setting appropriate expectations for advanced technology projects is crucial, especially when statistics are being cited. This is why the community depends on scientific, independent studies that can be peer reviewed by outside experts.

It is promising to hear end users interested in using facial recognition and pleased with their test results. However, comparable statistical claims require far higher scrutiny and clarity than provided yet for the Korean NPA test results.

Related Reports

Why 3VR Failed on Feb 16, 2018
3VR destroyed transformed ~$65 million in VC funding into a $6.9 million exit. The reason they failed is simple. They bet on analytics. They...
Convergint Damage Control on Feb 07, 2018
Convergint needs you to understand. When Convergint acquires companies, as they continuously do, it is an acquisition, e.g.: But when...
Chinese Police Wearing Facial Recognition Are Here on Feb 06, 2018
This is a very interesting and highly atypical usage of facial recognition that the Chinese government touted this week: It is a PRC police...
Canon Launches World's Most Expensive IP Camera (ME20F-SHN) on Jan 09, 2018
Canon has launched the ME20F-SHN , likely the world's most expensive single imager, non-thermal, IP camera at ~$20,000. And Canon subsidiary...
Chinese Government Attacks Western Reports on Jan 03, 2018
The Chinese government is angry at the BBC and WSJ's reporting on Chinese video surveillance (see BBC Features Dahua and WSJ Investigates China's...
Washington DC Surveillance Hackers Arrested on Dec 29, 2017
The US Department of Justice has announced that "Two Romanian Suspects Charged With Hacking of Metropolitan Police Department Surveillance Cameras...
WSJ Investigates China's Total Surveillance State on Dec 26, 2017
The WSJ is continuing its investigation into Chinese video surveillance. Following up on last month's WSJ Investigation of Hikvision, the WSJ is...
2018 Top Sales Interest For Integrators on Dec 20, 2017
130+ integrators answered: This is a key question, especially for manufacturers, as it impacts what areas to focus more on and what...
BBC Features Dahua on Dec 13, 2017
Hikvision is not the only mega-Chinese video surveillance manufacturer getting global attention. Last month, the WSJ investigated Hikvision and now...
Testing $20 WyzeCam, The Money Losing Amazon Vet Startup on Dec 12, 2017
This startup is perfecting the old adage: We lose money on every sale, but make it up on volume But it is no joke. The company, Wyze Labs, is...

Most Recent Industry Reports

Why 3VR Failed on Feb 16, 2018
3VR destroyed transformed ~$65 million in VC funding into a $6.9 million exit. The reason they failed is simple. They bet on analytics. They...
"Fear Mongering": Hikvision USA Cybersecurity Director Dismisses Chinese Government Ownership Concerns on Feb 16, 2018
The facts are: The Chinese government created Hikvision and is Hikvision's controlling shareholder. Hikvision's Chairman, a Communist Party...
16:9 vs 4:3 Video Aspect Ratio Statistics on Feb 16, 2018
What aspect ratio do security integrators prefer? The 'standard' 4:3 or the 'wide' 16:9 one? 100+ integrators told us what they preferred, with...
Mercury Releases New Series 3 Redboard Access Panels on Feb 15, 2018
Mercury Security has their first major product release post-HID buyout, and things literally look different. The Series 3 SIO boards now are red...
Last Chance February 2018 Camera Course on Feb 15, 2018
This is the last chance to get into the Winter camera course, starts next Tuesday. Register now. IPVM provides the best education, live online...
Hikvision DeepInMind Tested Terribly on Feb 15, 2018
While Hikvision is heavily marketing deep learning and 'AI' as their next big thing, new IPVM test results of their DeepInMind NVR shows their deep...
Genetec CEO: You Cannot Buy Trust on Feb 14, 2018
Genetec's CEO, Pierre Racz, delivered a direct message at their channel partner conference: Racz has become a focal point in the industry debate...
Assa's Lowest Power Draw Maglock: Securitron M680E Examined on Feb 14, 2018
Securitron produces some of the most extreme maglocks on the market, including massively strong maglocks and even ones with integrated CCTV cams...
Hanwha Wisenet X 5MP Camera Tested (XNV-8080R) on Feb 13, 2018
Wisenet X is Hanwha's high-end camera line. We tested their Wisenet X 1080p camera last year. Now Hanwha is offering 5MP cameras listing super low...
Top Problems For Integrator Project Management on Feb 13, 2018
Security projects routinely encounter issues that jeopardize deadlines, create confusion, and shrink profits. Unfortunately, there are common...

The world's leading video surveillance information source, IPVM provides the best reporting, testing and training for 10,000+ members globally. Dedicated to independent and objective information, we uniquely refuse any and all advertisements, sponsorship and consulting from manufacturers.

About | FAQ | Contact