What Does 3VR Face Recognition Test Results Mean?

Author: John Honovich, Published on May 06, 2009

Recently, results from a facial recognition test for the Korean National Police were reported widely by 3VR of obtaining: "85-92 percent accuracy in recognizing and matching faces in a few crowded, highly-trafficked public train stations in Seoul."

The manufacturer claims these results are the best ever in any public video facial recognition study (compared to only 60% historically). One of the security magazine editors suggested that this was reason to "change the standard analytics copy

A Korean National Police spokesman praised the results, expressing interest and confidence in deploying facial recognition widely. It is always impressive to see end users expressing confidence in new surveillance technologies.

The Importance of Test Results

In the security industry, test results are key benchmarks for buyers and integrators in evaluating technologies. Improvements in test results can be important drivers for expanding the use of technologies. As such, accuracy and clarity in test results are essential in ensuring successful deployments.

In facial recognition test results, it is common that detailed reports are released and in-depth quantifications are offered. These tests are often administered by government agencies. (Read a facial recognition overview and examine the 2006 US government study for good examples and background.)

The Korean National Police Test

Three material differences should be considered when comparing the Korean National Police Test to other previous publicly released studies:

  • While the test was sponsored by the Korean government, the tester, SK Networks, is a corporation that sells technology products, not an independent testing agency.
  • The tester, SK Networks, is marketing 3VR products (see SK Networks' marketing website and a 20 page marketing proposal for 3VR products, both dated 2008 [Note: the website has since been taken down]). 3VR reports that they were not aware of these documents and that they have asked SK Networks to remove them from the Internet.
  • No details were publicly provided on false error rates, system setup, or testing methodology. Small variances in any of these elements can result in significantly different accuracy statistics. 3VR reports that a 170 page test result document is available in Korean (NIA pub number NIA 111-RER-07113/2007.12) which they are in the process of translating but have not been given permission to share publicly.

Concerns about The Test

3VR's claim that this test result is the best ever raises concerns. First, since we cannot read the test results, it cannot be assessed how comparable the implementation and design of this test is to leading global scientific studies. Secondly, unlike other facial recognition tests conducted by independent testing agencies, SK Networks sells technology products and has a general interest in promoting the purchase and use of such products.

Setting appropriate expectations for advanced technology projects is crucial, especially when statistics are being cited. This is why the community depends on scientific, independent studies that can be peer reviewed by outside experts.

It is promising to hear end users interested in using facial recognition and pleased with their test results. However, comparable statistical claims require far higher scrutiny and clarity than provided yet for the Korean NPA test results.

Related Reports

Ambitious Mobile Access Startup: Openpath on May 24, 2018
This team sold their last startup for hundreds of millions of dollars, now they have started Openpath to become a rare access control small...
Amazon's "Dangerous New Face Recognition Technology" Says ACLU on May 23, 2018
The ACLU has caused a stir, with a new report Amazon Teams Up With Law Enforcement to Deploy Dangerous New Face Recognition Technology,...
Hikvision Source Code Transparency Center Examined on May 14, 2018
Following criticism of Hikvision's Chinese government ownership and Hikvision's IP camera backdoor, the company has responded with a series of...
March Networks Targets Cannabis Market on May 10, 2018
Will the next March Networks customer appreciation event be held a steakhouse or at a Taco Bell at 2 am? Can March sell the types of systems to the...
Integrators Divided On Website Importance to Business (Statistics) on May 04, 2018
Are websites important? Still quite a number of integrators have no websites.  New IPVM statistics show that nearly half of all integrators find a...
Amazon Enters Home Security Services on Apr 26, 2018
Move over ADT? It is certainly what investors have feared for ADT, especially since Amazon acquired Ring. [Update: We spoke with Amazon and have...
Digifort VMS Profile on Apr 25, 2018
Digifort, a Brazilian company, has a strong presense in their home country. In a crowded and mature Enterprise VMS market, will they be able to...
Favorite Biometrics 2018 on Apr 23, 2018
Biometrics are on the rise, or at least integrator opposition to them is declining, according to new IPVM integrator statistics.   Almost half of...
Dahua and Hikvision Win Over $1 Billion In Government-Backed Projects In Xinjiang on Apr 23, 2018
Dahua and Hikvision have won well over $1 billion worth of government-backed surveillance projects in China’s restive Xinjiang province since 2016,...
Rare Video Surveillance Fundraising - Verkada $15 Million on Apr 19, 2018
Fundraising in video surveillance (and the broader physical security market) has been poor recently. Highlights are few and far in between...

Most Recent Industry Reports

BREAKING: US House Passes Bill Banning Gov Use of Dahua and Hikvision on May 24, 2018
Today, the US House of Representatives has passed H.R. 5515, a bill that includes a ban on the US government's use of Dahua and Hikvision. This...
Hanwha Wisenet X Analytics and VMD Test on May 24, 2018
Continuing our updated testing of camera analytics, we tested Hanwha's Wisenet X analytics for over two weeks in multiple scenes, indoors and out,...
Ambitious Mobile Access Startup: Openpath on May 24, 2018
This team sold their last startup for hundreds of millions of dollars, now they have started Openpath to become a rare access control small...
Amazon's "Dangerous New Face Recognition Technology" Says ACLU on May 23, 2018
The ACLU has caused a stir, with a new report Amazon Teams Up With Law Enforcement to Deploy Dangerous New Face Recognition Technology,...
Software Only VMS vs NVR Appliances on May 23, 2018
Should you buy your own PC/server and load VMS software on it or get a turnkey appliance (both hardware and software, e.g., NVR, Hybrid DVR) from a...
Buy Arecont: Top Bid $10 Million Cash on May 22, 2018
Last year, Arecont had a deal for a purchase price of $170 million (see Failed Arecont China Acquisition). This year, Arecont has a deal for a...
Installing Box Cameras Indoors Tutorial on May 22, 2018
This tutorial starts our physical installation for video surveillance series, starting with Box Cameras, one of the oldest and most basic types....
The Hikvision Smart Classroom Behavior Management System on May 22, 2018
Hikvision's rapidly growing offering of analytics, which we most recently examined with Hikvision's ethnic minority analytics, is now going into...
Dahua Intrusion Analytics And VMD Tested (Poorly) on May 21, 2018
Dahua ships basic analytics on practically all their cameras, ranging from low cost to high end. To see how these analytics work in real world...
Exacq Improving Technical Support, Responding To Integrator Complaints on May 21, 2018
Exacq had been a long-term favorite of integrators, but since their 2014 Tyco acquisition, Exacq has fallen in IPVM integrator studies (though...

The world's leading video surveillance information source, IPVM provides the best reporting, testing and training for 10,000+ members globally. Dedicated to independent and objective information, we uniquely refuse any and all advertisements, sponsorship and consulting from manufacturers.

About | FAQ | Contact