What Does 3VR Face Recognition Test Results Mean?

Author: John Honovich, Published on May 06, 2009

Recently, results from a facial recognition test for the Korean National Police were reported widely by 3VR of obtaining: "85-92 percent accuracy in recognizing and matching faces in a few crowded, highly-trafficked public train stations in Seoul."

The manufacturer claims these results are the best ever in any public video facial recognition study (compared to only 60% historically). One of the security magazine editors suggested that this was reason to "change the standard analytics copy

A Korean National Police spokesman praised the results, expressing interest and confidence in deploying facial recognition widely. It is always impressive to see end users expressing confidence in new surveillance technologies.

The Importance of Test Results

In the security industry, test results are key benchmarks for buyers and integrators in evaluating technologies. Improvements in test results can be important drivers for expanding the use of technologies. As such, accuracy and clarity in test results are essential in ensuring successful deployments.

In facial recognition test results, it is common that detailed reports are released and in-depth quantifications are offered. These tests are often administered by government agencies. (Read a facial recognition overview and examine the 2006 US government study for good examples and background.)

The Korean National Police Test

Three material differences should be considered when comparing the Korean National Police Test to other previous publicly released studies:

  • While the test was sponsored by the Korean government, the tester, SK Networks, is a corporation that sells technology products, not an independent testing agency.
  • The tester, SK Networks, is marketing 3VR products (see SK Networks' marketing website and a 20 page marketing proposal for 3VR products, both dated 2008 [Note: the website has since been taken down]). 3VR reports that they were not aware of these documents and that they have asked SK Networks to remove them from the Internet.
  • No details were publicly provided on false error rates, system setup, or testing methodology. Small variances in any of these elements can result in significantly different accuracy statistics. 3VR reports that a 170 page test result document is available in Korean (NIA pub number NIA 111-RER-07113/2007.12) which they are in the process of translating but have not been given permission to share publicly.

Concerns about The Test

3VR's claim that this test result is the best ever raises concerns. First, since we cannot read the test results, it cannot be assessed how comparable the implementation and design of this test is to leading global scientific studies. Secondly, unlike other facial recognition tests conducted by independent testing agencies, SK Networks sells technology products and has a general interest in promoting the purchase and use of such products.

Setting appropriate expectations for advanced technology projects is crucial, especially when statistics are being cited. This is why the community depends on scientific, independent studies that can be peer reviewed by outside experts.

It is promising to hear end users interested in using facial recognition and pleased with their test results. However, comparable statistical claims require far higher scrutiny and clarity than provided yet for the Korean NPA test results.

Related Reports

Search More Important Than Live Monitoring - Statistics on Oct 18, 2017
Search is overall more important than live monitoring to integrators, according to new IPVM statistics.  The key themes found in integrator...
Long Time Industry Exec Leads New Security Franchise Offering on Oct 12, 2017
John Nemerofsky previously built and sold a $150 million dollar integration business, and then was VP of Niscayah from its spinout of Securitas,...
Exporting Video Surveillance Tutorial on Oct 05, 2017
Exporting video surveillance is important when incidents or crimes occur. However, there are multiple ways to export video which have their pros...
Vivotek Delta Acquisition Concludes on Oct 02, 2017
On August 8, 2017, Delta Group made an offer to acquire Vivotek. Now, the acquisition process has concluded. Inside this note, we examine the...
Knightscope Financials Reveal Huge Increase In Expenses And Incorrect Accounting on Oct 02, 2017
Knightscope's H1 2017 financials are out, in the midst of their ongoing crowd funding approaching $17 million. In this note, we analyze...
ASIS Show 2017 Final Report on Sep 27, 2017
ASIS is in Dallas for 2017 and this is our final show report (compare to our 2016 ASIS show report). When walking in, one is greeted with Dahua's...
Avigilon Touting 'Made In America' on Sep 18, 2017
Canadian manufacturer Avigilon, who completed a US manufacturing facility in 2015, is now running a marketing campaign touting 'Made In America',...
Genetec Launches Community Connect Examined on Sep 14, 2017
Genetec has done best in large-scale, enterprise systems and relatively worse in smaller systems such as SMB. Now, Genetec is launching...
Pelco CEO Is Out on Sep 05, 2017
Less than 2 years after taking the job, Pelco's CEO is out. Pelco's rollercoaster continues. In this note, we examine the move, share feedback...
Neurala Deep Learning Platform Profile on Aug 31, 2017
Find lost children? If Neurala has its way with its recent partnership with Motorola, that is the aim as shown in their video below: Neurala has...

Most Recent Industry Reports

Uniview Recorder Backdoor Examined on Oct 20, 2017
A Chinese research group has identified a vulnerability in Uniview recorders that allows backdoor access in a method similar to the Dahua...
Hikvision Access Control Tested on Oct 19, 2017
Hikvision aggressive pricing and marketing combined with generally reliable hardware and free software has made them a major player in video...
Verkada, Silicon Valley VSaaS Startup, Targets Enterprise on Oct 19, 2017
Verkada says they are building an enterprise-class VSaaS offering, calling it "The new platform for video security". This is a departure from the...
Exacq Unbreaks Avigilon Integration on Oct 18, 2017
For nearly 4 years, Exacq had broken and effectively blocked use with Avigilon cameras, as IPVM reported in January 2014. Now, Exacq has...
Search More Important Than Live Monitoring - Statistics on Oct 18, 2017
Search is overall more important than live monitoring to integrators, according to new IPVM statistics.  The key themes found in integrator...
Axis 'Sold Out' P3707-PVE Multi-Imager Tested on Oct 18, 2017
Axis faced significant product shortages over the summer. Perhaps the most notorious and significantly sold out model was the Axis P3707-PE 8MP...
Dahua Removes Auto Rebooting on Oct 17, 2017
For years, Dahua has automatically programmed its IP cameras to reboot weekly, a highly atypical and questionable practice. Following IPVM...
Deep Learning Tutorial For Video Surveillance on Oct 17, 2017
Deep learning is a growing buzzword within physical security and video surveillance. But what is 'deep learning'? In this tutorial, we explain...
Multipoint Lock Access Control Tutorial on Oct 17, 2017
Doors are notoriously weak at stopping entry, and money can be misspent on wrong locks that leave doors quite vulnerable. While closed and locked...

The world's leading video surveillance information source, IPVM provides the best reporting, testing and training for 10,000+ members globally. Dedicated to independent and objective information, we uniquely refuse any and all advertisements, sponsorship and consulting from manufacturers.

About | FAQ | Contact