What Does 3VR Face Recognition Test Results Mean?

Author: John Honovich, Published on May 06, 2009

Recently, results from a facial recognition test for the Korean National Police were reported widely by 3VR of obtaining: "85-92 percent accuracy in recognizing and matching faces in a few crowded, highly-trafficked public train stations in Seoul."

The manufacturer claims these results are the best ever in any public video facial recognition study (compared to only 60% historically). One of the security magazine editors suggested that this was reason to "change the standard analytics copy"

A Korean National Police spokesman praised the results, expressing interest and confidence in deploying facial recognition widely. It is always impressive to see end users expressing confidence in new surveillance technologies.

The Importance of Test Results

In the security industry, test results are key benchmarks for buyers and integrators in evaluating technologies. Improvements in test results can be important drivers for expanding the use of technologies. As such, accuracy and clarity in test results are essential in ensuring successful deployments.

In facial recognition test results, it is common that detailed reports are released and in-depth quantifications are offered. These tests are often administered by government agencies. (Read a facial recognition overview and examine the 2006 US government study for good examples and background.)

The Korean National Police Test

Three material differences should be considered when comparing the Korean National Police Test to other previous publicly released studies:

  • While the test was sponsored by the Korean government, the tester, SK Networks, is a corporation that sells technology products, not an independent testing agency.
  • The tester, SK Networks, is marketing 3VR products (see SK Networks' marketing website and a 20 page marketing proposal for 3VR products, both dated 2008 [Note: the website has since been taken down]). 3VR reports that they were not aware of these documents and that they have asked SK Networks to remove them from the Internet.
  • No details were publicly provided on false error rates, system setup, or testing methodology. Small variances in any of these elements can result in significantly different accuracy statistics. 3VR reports that a 170 page test result document is available in Korean (NIA pub number NIA 111-RER-07113/2007.12) which they are in the process of translating but have not been given permission to share publicly.

Concerns about The Test

3VR's claim that this test result is the best ever raises concerns. First, since we cannot read the test results, it cannot be assessed how comparable the implementation and design of this test is to leading global scientific studies. Secondly, unlike other facial recognition tests conducted by independent testing agencies, SK Networks sells technology products and has a general interest in promoting the purchase and use of such products.

Setting appropriate expectations for advanced technology projects is crucial, especially when statistics are being cited. This is why the community depends on scientific, independent studies that can be peer reviewed by outside experts.

It is promising to hear end users interested in using facial recognition and pleased with their test results. However, comparable statistical claims require far higher scrutiny and clarity than provided yet for the Korean NPA test results.

Related Reports

FLIR Acquires Drone Manufacturer For $134M on Dec 01, 2016
FLIR has acquired Prox Dynamics, a Norwegian maker of small military-grade drones, for $134M.  FLIR president Andy Teich provided additional...
3VR Sales And Marketing Execs Out, Company Shifting on Nov 22, 2016
Two executives are out at 3VR, both after relatively short tenures at the company. 3VR's CEO says this is part of a shift to cover new markets....
Top Manufacturers Gaining and Losing 2016 on Nov 21, 2016
Hikvision has been the top gaining manufacturer in IPVM's annual integrator survey the past 2 years (see 2015 results and 2014 results). Can they...
Sony and Bosch Combine Forces on Nov 15, 2016
Two of the biggest brands in the video surveillance industry are joining forces. Following a wave of industry consolidation and the race to the...
Samsung Brand 'Phasing' Out For Hanhwa Techwin on Nov 14, 2016
The phasing out of the Samsung brand for video surveillance products continues. First, it was the corporate name changing from Samsung Techwin to...
Milestone Launches $50,000 Developer Contest on Oct 26, 2016
Milestone wants independent developers to enhance XProtect. In a hackathon-style contest Milestone is soliciting developers to submit ideas for a...
"WTF?!?!? Who is Brian Karas?!?" Exclaims Knightscope on Oct 21, 2016
Knightscope co-founder Stacy Stephens emailed us: He may not have intended to send it to us and he probably can figure out who Brian Karas is,...
IHS Sells Credibility To Manufacturer Sponsorships on Oct 18, 2016
A valuable manufacturer opportunity or a conflict of interest? Trade magazines have long offered promotional whitepapers and webinars to...
No Marketing Gimmick, Sharp Security Robot Targets Professional Market on Oct 17, 2016
Security robots are getting a lot of attention. Unfortunately, this is often for wrong reasons, most spectacularly highlighted by the Knightscope...
The 'Last Chance to Save' On Hikvision Is Here on Sep 29, 2016
It is over. After at least 8 across the board price cuts in the past 10 months, including an unprecedented back to back 20% off, Hikvision has...

Most Recent Industry Reports

Knightscope - $122,509 Revenue, $2.5 Million Loss Seeks $20 Million Investment on Dec 09, 2016
The robot that ran over a child, Knightscope, wants money and they need it. Investors can invest as little as $1,000 to participate and...
'Solution' Manufacturers Threaten Integrators on Dec 09, 2016
The race to the bottom has driven manufacturers to become 'solution' providers, threatening integrators. Axis shift to 'solution' sales might be...
The Russian SMP Security Robot on Dec 08, 2016
A Russian manufacturer, SMP, has a commercially available outdoor security robot, at a lower price and with much less marketing than their main...
How Hikvision Beats Its OEMs on Dec 08, 2016
Hikvision GM declared that they are not aggressive with their competitors. But some of their own OEM partners disagree. Inside, we reveal a key...
Dahua Discontinuing H.264 Only Products on Dec 08, 2016
Dahua has taken a stand for H.265 and is discontinuing its H.264 only products. We examine the shakeup inside this...
IP Networking Course January 2017 on Dec 08, 2016
This is the only networking course designed specifically for video surveillance professionals plus it includes live training, personal help and...
Hikvision vs Dahua Mobile Apps Tested on Dec 07, 2016
With smartphone use and low-cost video recorders surging, many user's main interface to their surveillance system is their phone. With mobile video...
Paxton Drops US Reps, Plans Major Expansion on Dec 07, 2016
Paxton is gearing up to make a big run at  US access control success. The first step they have made is to cut all US Rep Firms, in anticipation of...
Axis Partner Elder Care Video Analytics (Smartervision) on Dec 07, 2016
Can video analytics be used to improve the care of the elderly? Axis and a video analytics startup, Smartervision, are working together to do so....
Sony IP Camera Backdoor Uncovered on Dec 06, 2016
A backdoor has been uncovered in ~80 Sony IP camera models, attackers can remotely enable telnet on the camera, and then potentially login as root,...

The world's leading video surveillance information source, IPVM provides the best reporting, testing and training for 10,000+ members globally. Dedicated to independent and objective information, we uniquely refuse any and all advertisements, sponsorship and consulting from manufacturers.

About | FAQ | Contact