What Does 3VR Face Recognition Test Results Mean?

Author: John Honovich, Published on May 06, 2009

Recently, results from a facial recognition test for the Korean National Police were reported widely by 3VR of obtaining: "85-92 percent accuracy in recognizing and matching faces in a few crowded, highly-trafficked public train stations in Seoul."

The manufacturer claims these results are the best ever in any public video facial recognition study (compared to only 60% historically). One of the security magazine editors suggested that this was reason to "change the standard analytics copy

A Korean National Police spokesman praised the results, expressing interest and confidence in deploying facial recognition widely. It is always impressive to see end users expressing confidence in new surveillance technologies.

The Importance of Test Results

In the security industry, test results are key benchmarks for buyers and integrators in evaluating technologies. Improvements in test results can be important drivers for expanding the use of technologies. As such, accuracy and clarity in test results are essential in ensuring successful deployments.

In facial recognition test results, it is common that detailed reports are released and in-depth quantifications are offered. These tests are often administered by government agencies. (Read a facial recognition overview and examine the 2006 US government study for good examples and background.)

The Korean National Police Test

Three material differences should be considered when comparing the Korean National Police Test to other previous publicly released studies:

  • While the test was sponsored by the Korean government, the tester, SK Networks, is a corporation that sells technology products, not an independent testing agency.
  • The tester, SK Networks, is marketing 3VR products (see SK Networks' marketing website and a 20 page marketing proposal for 3VR products, both dated 2008 [Note: the website has since been taken down]). 3VR reports that they were not aware of these documents and that they have asked SK Networks to remove them from the Internet.
  • No details were publicly provided on false error rates, system setup, or testing methodology. Small variances in any of these elements can result in significantly different accuracy statistics. 3VR reports that a 170 page test result document is available in Korean (NIA pub number NIA 111-RER-07113/2007.12) which they are in the process of translating but have not been given permission to share publicly.

Concerns about The Test

3VR's claim that this test result is the best ever raises concerns. First, since we cannot read the test results, it cannot be assessed how comparable the implementation and design of this test is to leading global scientific studies. Secondly, unlike other facial recognition tests conducted by independent testing agencies, SK Networks sells technology products and has a general interest in promoting the purchase and use of such products.

Setting appropriate expectations for advanced technology projects is crucial, especially when statistics are being cited. This is why the community depends on scientific, independent studies that can be peer reviewed by outside experts.

It is promising to hear end users interested in using facial recognition and pleased with their test results. However, comparable statistical claims require far higher scrutiny and clarity than provided yet for the Korean NPA test results.

Related Reports

Hikvision: IPVM Is "Destined To Fail" on Jun 14, 2017
Hikvision has accused IPVM of 'cyberbullying' them, declaring IPVM 'destined to fail.' This is the 3rd anti-IPVM Hikvision post in 2 weeks,...
NLSS / ATV Partnership Examined on Jun 13, 2017
NLSS has been one of the most ambitious security / surveillance startups. After selling his last startup, Sypixx, to Cisco, Pete...
Avigilon New COO James Henderson Profile on May 23, 2017
It has been nearly 2 years since the infamous Bryan Schmode 'resigned' as Avigilon COO. Now, Avigilon once again has a COO, promoting James...
Hikvision Marketer Caught Spamming, Fails at Coverup, Fired on May 23, 2017
A Hikvision marketing employee was caught by IPCamTalk trying to surreptitiously disparage IPVM and IPCamTalk. This is an outgrowth of Hikvision's...
ShotSpotter To IPO, Facing Low Revenue and Losses on May 17, 2017
A rare event for North American security manufacturers is upcoming. ShotSpotter is planning to IPO on the NASDAQ, aiming to raise $34.5...
Shark Tank Startup Guard Llama Mobile Panic Tested on May 16, 2017
A Shark Tank TV show appearance has attracted big attention for Guard Llama, a security startup touting a 'panic button' paired with mobile app...
Mobile Video Installation Statistics on May 02, 2017
Mobile video systems - those mounted on buses, trains, cars, boats, etc. - are a niche within video surveillance with its own particular...
Duress Codes For Alarms Systems on May 02, 2017
An alarm system can call for help in the event of an attempted break in, but only if it is armed. If an adversary forces an authorized user to...
Burglar Alarm Zoning Guide on Apr 28, 2017
The function of an alarm panel is to gather information from sensors and respond to this information by triggering actions. While it is possible to...
A Marketing Home Run For Knightscope - Man Attacks Robot on Apr 27, 2017
We criticize Knightscope regularly - their lack of revenue, their trying to fool mom 'n pop investors, their associating themselves with a clueless...

Most Recent Industry Reports

Importance of Sales To Integrators - Statistics on Jun 23, 2017
One of the top trends in the industry over the past few years has been the rise of across-the-board sales (e.g.: Hikvision Sales, Dahua Sale,...
Deep Learning Surveillance Startups Deep Problem on Jun 23, 2017
The undeniably good news for the video surveillance market is that we are seeing the rise of more startups than in many years. The cause of this...
Avigilon Announces RADAR-Based Presence Detector on Jun 22, 2017
RADAR is gaining momentum within physical security. Two months after Axis announced a network radar detector, Avigilon has announced a RADAR-Based...
Covert Cloud Camera Service Launching (KJB) on Jun 22, 2017
Cloud IP cameras, for consumers, has become increasingly commonplace. However, covert cameras, lag there, with few options. Now, North America's...
Manufacturers Shipping Unlicensed H.265 Products on Jun 22, 2017
While H.265 support in video surveillance is growing, IPVM's research shows that most surveillance manufacturers are shipping H.265 products with...
Uniview Low-Cost Bullet PTZ Tested on Jun 21, 2017
Uniview is offering a HD zoom bullet camera, the IPC742SR9-PZ30-32G, with an integrated pan / tilt positioner, for the price of a low-cost...
QSR Video Surveillance Best Practices on Jun 21, 2017
Fast food restaurants or QSRs (quick service restaurants), are frequent victims of crime and fraud. Because they are open late, deal with cash, and...
45 Drives 'Lowest Cost' Enterprise Storage Company Profile on Jun 21, 2017
45 Drives claims the "lowest cost per Hard Drive Slot in the industry." But who or what is '45 Drives'? What started as a product design to...
No Hack, Still Liable, Court Finds ADT on Jun 20, 2017
Recently, ADT has been in the news for a $16 million settlement for a cyber security vulnerability class action suit. One of the most important...
Resolver / PPM 2000 Incident Management Platform Profile on Jun 20, 2017
You might have seen the company whose employees wear hockey jerseys at trade shows and wondered "what do they do?" PPM 2000 has been active in...

The world's leading video surveillance information source, IPVM provides the best reporting, testing and training for 10,000+ members globally. Dedicated to independent and objective information, we uniquely refuse any and all advertisements, sponsorship and consulting from manufacturers.

About | FAQ | Contact