What Does 3VR Face Recognition Test Results Mean?

Author: John Honovich, Published on May 06, 2009

Recently, results from a facial recognition test for the Korean National Police were reported widely by 3VR of obtaining: "85-92 percent accuracy in recognizing and matching faces in a few crowded, highly-trafficked public train stations in Seoul."

The manufacturer claims these results are the best ever in any public video facial recognition study (compared to only 60% historically). One of the security magazine editors suggested that this was reason to "change the standard analytics copy

A Korean National Police spokesman praised the results, expressing interest and confidence in deploying facial recognition widely. It is always impressive to see end users expressing confidence in new surveillance technologies.

The Importance of Test Results

In the security industry, test results are key benchmarks for buyers and integrators in evaluating technologies. Improvements in test results can be important drivers for expanding the use of technologies. As such, accuracy and clarity in test results are essential in ensuring successful deployments.

In facial recognition test results, it is common that detailed reports are released and in-depth quantifications are offered. These tests are often administered by government agencies. (Read a facial recognition overview and examine the 2006 US government study for good examples and background.)

The Korean National Police Test

Three material differences should be considered when comparing the Korean National Police Test to other previous publicly released studies:

  • While the test was sponsored by the Korean government, the tester, SK Networks, is a corporation that sells technology products, not an independent testing agency.
  • The tester, SK Networks, is marketing 3VR products (see SK Networks' marketing website and a 20 page marketing proposal for 3VR products, both dated 2008 [Note: the website has since been taken down]). 3VR reports that they were not aware of these documents and that they have asked SK Networks to remove them from the Internet.
  • No details were publicly provided on false error rates, system setup, or testing methodology. Small variances in any of these elements can result in significantly different accuracy statistics. 3VR reports that a 170 page test result document is available in Korean (NIA pub number NIA 111-RER-07113/2007.12) which they are in the process of translating but have not been given permission to share publicly.

Concerns about The Test

3VR's claim that this test result is the best ever raises concerns. First, since we cannot read the test results, it cannot be assessed how comparable the implementation and design of this test is to leading global scientific studies. Secondly, unlike other facial recognition tests conducted by independent testing agencies, SK Networks sells technology products and has a general interest in promoting the purchase and use of such products.

Setting appropriate expectations for advanced technology projects is crucial, especially when statistics are being cited. This is why the community depends on scientific, independent studies that can be peer reviewed by outside experts.

It is promising to hear end users interested in using facial recognition and pleased with their test results. However, comparable statistical claims require far higher scrutiny and clarity than provided yet for the Korean NPA test results.

Related Reports

Competing Against ADT on Jul 20, 2017
ADT is one of the biggest players in the security industry, with ~$4 billion revenue. In 2017, they were acquired / merged with Protection...
Hikvision USA Head of Cybersecurity Exits on Jul 18, 2017
Hikvision USA's Head of Cybersecurity has exited the company. In this note, we review the move, share Hikvision's feedback and examine the...
Nest Cam IQ Tested on Jul 10, 2017
Nest has released their latest entry in their camera line, the Nest Cam IQ, touting 4K "Supersight", facial recognition, "HD audio", invisible IR,...
Top 5 Improvements For Manufacturer Sales People on Jun 28, 2017
Manufacturer sales people are very important to integrators, but it takes more than just punching a clock to be successful and truly valuable to...
No Personal Opinions About Work on Jun 26, 2017
One rising trend is the tendency for people to disclaim their statements on work related topics as their own 'opinions' or 'personal...
Hikvision: IPVM Is "Destined To Fail" on Jun 14, 2017
Hikvision has accused IPVM of 'cyberbullying' them, declaring IPVM 'destined to fail.' This is the 3rd anti-IPVM Hikvision post in 2 weeks,...
NLSS / ATV Partnership Examined on Jun 13, 2017
NLSS has been one of the most ambitious security / surveillance startups. After selling his last startup, Sypixx, to Cisco, Pete...
ShotSpotter IPOs, Facing Low Revenue and Losses on Jun 09, 2017
A rare event for North American security manufacturers has happened. ShotSpotter has IPOed on the NASDAQ, raising $30.8 million. US IPOs,...
Avigilon New COO James Henderson Profile on May 23, 2017
It has been nearly 2 years since the infamous Bryan Schmode 'resigned' as Avigilon COO. Now, Avigilon once again has a COO, promoting James...
Hikvision Marketer Caught Spamming, Fails at Coverup, Fired on May 23, 2017
A Hikvision marketing employee was caught by IPCamTalk trying to surreptitiously disparage IPVM and IPCamTalk. This is an outgrowth of Hikvision's...

Most Recent Industry Reports

Security Robots Are Just Entertainment on Jul 21, 2017
Great entertainment, no real security value.  That is the happy (or sad) state of security robots in 2017. Knightscope robot's drowning, the...
Wireless Burglar Alarm Sensors Guide on Jul 21, 2017
Wireless sensors for burglar alarm sensors are an increasingly common option for the historical labor intensive wired alarm systems. However,...
Competing Against ADT on Jul 20, 2017
ADT is one of the biggest players in the security industry, with ~$4 billion revenue. In 2017, they were acquired / merged with Protection...
Hikvision Launching Deep Learning Recorders on Jul 20, 2017
Hikvision has become a common choice for super low cost NVRs. Now, Hikvision is aiming to move up market, with deep learning NVRs that claim far...
PR Campaign Exploiting Manufacturer Cybersecurity on Jul 20, 2017
Manufacturers increasingly have a bulls-eye on their back. As cyber security solutions providers grow, they realize a great way to get publicity...
Axis Door Station Tested (A8105-E) on Jul 19, 2017
Axis continues their push into niche markets, especially audio, with network speakers, an IP horn, and video door stations. We bought and tested...
Manufacturer Favorability Guide on Jul 19, 2017
This 120 page PDF guide may be downloaded inside by all IPVM members. It includes our 20 manufacturer favorability rankings and 20 manufacturer...
$8 Billion Utility Georgia Power Enters Surveillance Business Offering Avigilon And Genetec on Jul 19, 2017
Utilities are typically considered major customers of surveillance integrators but one utility, Georgia Power, with $8+ billion in annual revenue...
Knightscope Laughs off Robot Drowning on Jul 18, 2017
A day after a Knightscope robot drowned, Knightscope has issued an 'official statement' making fun of the issue: The implied message is that...
Microsoft Video AI Cloud Services Examined on Jul 18, 2017
Microsoft has released one of the most amazing video analytics marketing videos ever. In it, they detect oil spills, track individual people giving...

The world's leading video surveillance information source, IPVM provides the best reporting, testing and training for 10,000+ members globally. Dedicated to independent and objective information, we uniquely refuse any and all advertisements, sponsorship and consulting from manufacturers.

About | FAQ | Contact