What Does 3VR Face Recognition Test Results Mean?

Author: John Honovich, Published on May 06, 2009

Recently, results from a facial recognition test for the Korean National Police were reported widely by 3VR of obtaining: "85-92 percent accuracy in recognizing and matching faces in a few crowded, highly-trafficked public train stations in Seoul."

The manufacturer claims these results are the best ever in any public video facial recognition study (compared to only 60% historically). One of the security magazine editors suggested that this was reason to "change the standard analytics copy"

A Korean National Police spokesman praised the results, expressing interest and confidence in deploying facial recognition widely. It is always impressive to see end users expressing confidence in new surveillance technologies.

The Importance of Test Results

In the security industry, test results are key benchmarks for buyers and integrators in evaluating technologies. Improvements in test results can be important drivers for expanding the use of technologies. As such, accuracy and clarity in test results are essential in ensuring successful deployments.

In facial recognition test results, it is common that detailed reports are released and in-depth quantifications are offered. These tests are often administered by government agencies. (Read a facial recognition overview and examine the 2006 US government study for good examples and background.)

The Korean National Police Test

Three material differences should be considered when comparing the Korean National Police Test to other previous publicly released studies:

  • While the test was sponsored by the Korean government, the tester, SK Networks, is a corporation that sells technology products, not an independent testing agency.
  • The tester, SK Networks, is marketing 3VR products (see SK Networks' marketing website and a 20 page marketing proposal for 3VR products, both dated 2008 [Note: the website has since been taken down]). 3VR reports that they were not aware of these documents and that they have asked SK Networks to remove them from the Internet.
  • No details were publicly provided on false error rates, system setup, or testing methodology. Small variances in any of these elements can result in significantly different accuracy statistics. 3VR reports that a 170 page test result document is available in Korean (NIA pub number NIA 111-RER-07113/2007.12) which they are in the process of translating but have not been given permission to share publicly.

Concerns about The Test

3VR's claim that this test result is the best ever raises concerns. First, since we cannot read the test results, it cannot be assessed how comparable the implementation and design of this test is to leading global scientific studies. Secondly, unlike other facial recognition tests conducted by independent testing agencies, SK Networks sells technology products and has a general interest in promoting the purchase and use of such products.

Setting appropriate expectations for advanced technology projects is crucial, especially when statistics are being cited. This is why the community depends on scientific, independent studies that can be peer reviewed by outside experts.

It is promising to hear end users interested in using facial recognition and pleased with their test results. However, comparable statistical claims require far higher scrutiny and clarity than provided yet for the Korean NPA test results.

Related Reports

How To Become an 'Expert' In An Hour on Sep 02, 2016
If you want to be perceived as an expert in your market, speak at every event possible... the math is simple: when someone stands in front of an...
Axis Takes Over Canon Surveillance Sales and Marketing on Sep 01, 2016
The wasteful Axis / Canon conflict is ending. After more than a year of Canon's video surveillance group competing with Axis, Canon...
How Much Axis and Avigilon Pay For Sales And Marketing Per Camera on Sep 01, 2016
When you buy an Axis or Avigilon camera, how much of that goes to pay sales and marketing expenses? Is it trivial like $2 or $3? Or is it far...
Get End User Leads for Just $997 A Month on Aug 31, 2016
Scam or opportunity? As trade mags continue to suffer from the decline of print media, they are scrounging for new ways to make money. Rather than...
Integrators Vs Manufacturers Direct To End Users on Aug 31, 2016
Many manufacturers have increasingly large and hungry sales forces that call on more and more end users. One recent example is Axis ~$20 Billion...
Milestone Declares End-To-End Is A Dead-End on Aug 24, 2016
While Axis celebrates their end to end system win at top 50 global retailer H&M, their Canon sister company Milestone is proclaiming that...
Hikvision 20% Price Cut Twice In A Month on Aug 23, 2016
Days after celebrating the Chinese government's commitment of up to $3 billion USD more in funding, Hikvision USA is now launching up to 20% across...
Axis Lists Itself First Among VMSes on Aug 19, 2016
When you think of VMSes, who do you think of first? Well, Axis, the camera manufacturer, thinks of itself. Axis declared: There are a variety...
Dahua Fakes Being American on Aug 10, 2016
Robbed from the Hikvision playbook, now Dahua takes it turn. Dahua, China's second largest manufacturer, is ramping up its...
IPVM Site Goes All HTTPS, Largest 3 Manufacturers Do Not on Jul 12, 2016
IPVM.com now serves all pages over HTTPS to improve security and privacy. However, a number of video surveillance manufacturers do not, including...

Most Recent Industry Reports

Nest Cam Outdoor Tested on Sep 23, 2016
After years of claiming an outdoor model was "coming", addressing their biggest user demand, Nest has finally released their Outdoor Camera, an...
ACTi Refuses Race To The Bottom, Shifts To Solutions on Sep 23, 2016
The original low cost IP camera disruptor was ACTi. Back in the 2008 - 2010 time frame, Taiwanese manufacturer ACTi challenged the Western and...
You Get Robbed, Canary Will Pay You Up To $1,000 on Sep 22, 2016
Canary is trying to break the status quo in DIY security, first by raising over $40 million, and now a revamp of their monthly services package...
Milestone Ends Development of "Enterprise" VMS on Sep 22, 2016
Milestone 'Enterprise' was one of the first enterprise video management software offerings, selected by many early adopters of IP video. However,...
History of Video Surveillance on Sep 22, 2016
This is a concise history of video surveillance covering the past decade.  The goal is to help professionals newer to the industry understand...
Access Control Course Fall 2016 on Sep 22, 2016
IPVM offers the most comprehensive access control course in the industry. Unlike manufacturer training that focuses only on a small part of the...
Totally Wireless IP Camera (IPVideo Corp NomadHD) on Sep 21, 2016
Wireless battery powered cameras have been a surveillance pipe dream for years, limited by camera power consumption, battery technology, and...
Axis Launches IP Speakers on Sep 21, 2016
First, Axis introduced an IP horn, then it was video intercoms, and now it is Networked Speakers? While IP-based Public Address systems are not...
Tagged RFID Object Search Recorded Video on Sep 20, 2016
Video analytics has gotten fairly good at tagging people in video, but it does not solve the problem of finding items like specific merchandise or...
FLIR and Geovision Join the Hikvision Price Cut Race on Sep 20, 2016
Hikvision's price cuts are clearly a trend setter. After numerous and increasingly large cuts, the destructive cycle is accelerating. Last month,...

The world's leading video surveillance information source, IPVM provides the best reporting, testing and training for 10,000+ members globally. Dedicated to independent and objective information, we uniquely refuse any and all advertisements, sponsorship and consulting from manufacturers.

About | FAQ | Contact