What Does 3VR Face Recognition Test Results Mean?

Author: John Honovich, Published on May 06, 2009

Recently, results from a facial recognition test for the Korean National Police were reported widely by 3VR of obtaining: "85-92 percent accuracy in recognizing and matching faces in a few crowded, highly-trafficked public train stations in Seoul."

The manufacturer claims these results are the best ever in any public video facial recognition study (compared to only 60% historically). One of the security magazine editors suggested that this was reason to "change the standard analytics copy

A Korean National Police spokesman praised the results, expressing interest and confidence in deploying facial recognition widely. It is always impressive to see end users expressing confidence in new surveillance technologies.

The Importance of Test Results

In the security industry, test results are key benchmarks for buyers and integrators in evaluating technologies. Improvements in test results can be important drivers for expanding the use of technologies. As such, accuracy and clarity in test results are essential in ensuring successful deployments.

In facial recognition test results, it is common that detailed reports are released and in-depth quantifications are offered. These tests are often administered by government agencies. (Read a facial recognition overview and examine the 2006 US government study for good examples and background.)

The Korean National Police Test

Three material differences should be considered when comparing the Korean National Police Test to other previous publicly released studies:

  • While the test was sponsored by the Korean government, the tester, SK Networks, is a corporation that sells technology products, not an independent testing agency.
  • The tester, SK Networks, is marketing 3VR products (see SK Networks' marketing website and a 20 page marketing proposal for 3VR products, both dated 2008 [Note: the website has since been taken down]). 3VR reports that they were not aware of these documents and that they have asked SK Networks to remove them from the Internet.
  • No details were publicly provided on false error rates, system setup, or testing methodology. Small variances in any of these elements can result in significantly different accuracy statistics. 3VR reports that a 170 page test result document is available in Korean (NIA pub number NIA 111-RER-07113/2007.12) which they are in the process of translating but have not been given permission to share publicly.

Concerns about The Test

3VR's claim that this test result is the best ever raises concerns. First, since we cannot read the test results, it cannot be assessed how comparable the implementation and design of this test is to leading global scientific studies. Secondly, unlike other facial recognition tests conducted by independent testing agencies, SK Networks sells technology products and has a general interest in promoting the purchase and use of such products.

Setting appropriate expectations for advanced technology projects is crucial, especially when statistics are being cited. This is why the community depends on scientific, independent studies that can be peer reviewed by outside experts.

It is promising to hear end users interested in using facial recognition and pleased with their test results. However, comparable statistical claims require far higher scrutiny and clarity than provided yet for the Korean NPA test results.

Related Reports

Lighthouse Deep Learning Camera Tested on Dec 07, 2017
A Silicon Valley startup, Lighthouse, with a Stanford PhD CTO, has released a deep learning AI camera with 3D sensors for just $300. The company...
ASIS Dumps 'ASIS' For Show on Dec 06, 2017
After 60+ years, ASIS is dumping its eponymous show name and replacing it with 'GSX'. This is a classic marketing mistake. For a show struggling...
Security 101 / Axis Charity Contest on Nov 30, 2017
Integrator Security101 and manufacturer Axis Communications have teamed up for what they call the "Gift of Security", saying they "will...
Hikvision Chinese Government Owner CETHIK Exposed on Nov 20, 2017
Hikvision deceives about its Chinese government ownership. Contrary to their claims about 'independence' and simply having 'shareholders' that are...
Panasonic Unified Surveillance Strategy Analyzed on Nov 17, 2017
Panasonic is now a "Unified Surveillance" offering, as their ASIS 2017 booth proclaimed: Looking to make a comeback in the security industry,...
Hikvision China Criticizes The WSJ on Nov 15, 2017
Hikvision, through the Chinese government's authoritative news service, has criticized the WSJ investigation into Hikvision. In this...
WSJ Investigates Hikvision on Nov 13, 2017
The Wall Street Journal (WSJ) has released a detailed investigation into Hikvision's government ownership and cybersecurity problems, hitting the...
Honeywell Crowdfunding Bombs on Nov 08, 2017
100,000+ employees and $40,000,000,000 in annual revenue. That's Honeywell. ~200 backers and ~$67,000 in funding in a week. That's Honeywell...
Dahua USA Removes Head of Marketing on Nov 06, 2017
Dahua USA's marketing and just their overall business organization has been a mess. These guys could not sell life jackets on the Titanic. But,...
Ingram Micro: The Blind Lead The Blind on Nov 02, 2017
Ingram Micro, as a huge as they are overall, with $40+ billion in annual sales, has never been a force in physical security, despite, or perhaps...

Most Recent Industry Reports

Imperial Capital Security Investor Conference Review on Dec 08, 2017
Investment bank Imperial Capital holds an annual Security Investor Conference where 60+ companies present, including this year: IPVM bought a...
Integrator GPS Vehicle Tracking Statistics and Success Examined on Dec 08, 2017
GPS vehicle tracking is a growing but somewhat controversial topic. On the plus side, tracking may increases productivity by providing greater...
Hikvision NA Biggest Sale of 2017 on Dec 07, 2017
Hikvision North America has been relatively disciplined the past 5 months, reducing the number of sales and the breadth of what is on sale. No...
Security Integrator IT Expertise Statistics on Dec 07, 2017
20 years ago, putting physical security systems on IP networks was just emerging. Today, almost every system is networked in some way, IP cameras...
Lighthouse Deep Learning Camera Tested on Dec 07, 2017
A Silicon Valley startup, Lighthouse, with a Stanford PhD CTO, has released a deep learning AI camera with 3D sensors for just $300. The company...
Access Control Course Winter 2018 on Dec 07, 2017
Learn more below about the Winter 2018 IPVM Access Control Course. Register here. IPVM offers the most comprehensive access control course in...
Broken Hikvision App Exposes Hypocrisy on Dec 06, 2017
While Hikvision talks about a commitment to cybersecurity, their broken app and their insecure 'solution' exposes not only their engineering...
'Catastrophic Problem' For Videofied App on Dec 06, 2017
Less than 2 months after closing their DIY division DragonFly, Videofied has been hit with a problem the company calls 'catastrophic'. Now the...
ASIS Dumps 'ASIS' For Show on Dec 06, 2017
After 60+ years, ASIS is dumping its eponymous show name and replacing it with 'GSX'. This is a classic marketing mistake. For a show struggling...
Risks Of Managing End User Passwords (Statistics) on Dec 05, 2017
Integrators know admin passwords for nearly all end-user systems, according to IPVM statistics. But how do they manage them? How do they ensure...

The world's leading video surveillance information source, IPVM provides the best reporting, testing and training for 10,000+ members globally. Dedicated to independent and objective information, we uniquely refuse any and all advertisements, sponsorship and consulting from manufacturers.

About | FAQ | Contact