What Does 3VR Face Recognition Test Results Mean?by John Honovich, IPVM posted on May 07, 2009 About John Contact John
Recently, results from a facial recognition test for the Korean National Police were reported widely by 3VR of obtaining: "85-92 percent accuracy in recognizing and matching faces in a few crowded, highly-trafficked public train stations in Seoul."
The manufacturer claims these results are the best ever in any public video facial recognition study (compared to only 60% historically). One of the security magazine editors suggested that this was reason to "change the standard analytics copy"
A Korean National Police spokesman praised the results, expressing interest and confidence in deploying facial recognition widely. It is always impressive to see end users expressing confidence in new surveillance technologies.
The Importance of Test Results
In the security industry, test results are key benchmarks for buyers and integrators in evaluating technologies. Improvements in test results can be important drivers for expanding the use of technologies. As such, accuracy and clarity in test results are essential in ensuring successful deployments.
In facial recognition test results, it is common that detailed reports are released and in-depth quantifications are offered. These tests are often administered by government agencies. (Read a facial recognition overview and examine the 2006 US government study for good examples and background.)
The Korean National Police Test
Three material differences should be considered when comparing the Korean National Police Test to other previous publicly released studies:
While the test was sponsored by the Korean government, the tester, SK Networks, is a corporation that sells technology products, not an independent testing agency.
The tester, SK Networks, is marketing 3VR products (see SK Networks' marketing website and a 20 page marketing proposal for 3VR products, both dated 2008 [Note: the website has since been taken down]). 3VR reports that they were not aware of these documents and that they have asked SK Networks to remove them from the Internet.
No details were publicly provided on false error rates, system setup, or testing methodology. Small variances in any of these elements can result in significantly different accuracy statistics. 3VR reports that a 170 page test result document is available in Korean (NIA pub number NIA 111-RER-07113/2007.12) which they are in the process of translating but have not been given permission to share publicly.
Concerns about The Test
3VR's claim that this test result is the best ever raises concerns. First, since we cannot read the test results, it cannot be assessed how comparable the implementation and design of this test is to leading global scientific studies. Secondly, unlike other facial recognition tests conducted by independent testing agencies, SK Networks sells technology products and has a general interest in promoting the purchase and use of such products.
Setting appropriate expectations for advanced technology projects is crucial, especially when statistics are being cited. This is why the community depends on scientific, independent studies that can be peer reviewed by outside experts.
It is promising to hear end users interested in using facial recognition and pleased with their test results. However, comparable statistical claims require far higher scrutiny and clarity than provided yet for the Korean NPA test results.
Most Recent Industry Reports
#1 IP Camera Benefit on May 20, 2013
Megapixel was, far and away, the number #1 cited benefit in going to IP cameras / video. We asked over 100 integrators to name the "3 biggest reasons" they found, allowing them to name whatever the...
Testing VSaaS / Dropcam HD on May 15, 2013
Dropcam is pretty clearly the strongest upstart in the VSaaS market. While Axis has pushed their hosted service offering for years, it is beset by poor ROI and weak performance. There are dozens of...
Testing Panoramic Cameras Outdoors on May 13, 2013
Outdoors, PTZs tend to be a favorite for covering large open areas but they only see whatever they are currently pointed at. Panoramics are an emerging alternative but how well do they perform and ...
Testing CarCam on May 08, 2013
Dash cams are getting a lot of attention for the amazing video they inadvertently capture, like a crashing 747 in Afghanistan, a meteor explosion, Russian crazy fights and accidents, etc. Moreov...
#1 IP Camera Problem on May 07, 2013
Cost was, far and away, the number #1 cited problem in going to IP cameras / video. We asked over 100 integrators to name the "3 biggest problems / barriers" they found, allowing them to name whate...
Testing Covert Cameras on May 02, 2013
Covert cameras are regularly referenced in TV shows and undercover reporting. We ordered two of the most frequently sold covert cameras on Amazon - a Car Key cam and a PenCam, as shown below: We...
Stop The Lying and Crazy Claims on Apr 29, 2013
The last few weeks have been an embarrassment for surveillance, with a small number of companies turning this into a despicable marketing campaign perpetuating lies and crazy myths about what surve...
Panoramic Camera Shootout on Apr 22, 2013
Panoramic cameras are one of the hottest growing markets within surveillance. The combination of multi-megapixel resolution and super wide FoVs claims to deliver greater coverage than conventional ...
Top Trends in Access Control on Apr 17, 2013
All of a sudden, Access Control has momentum? After a long period of apparent stagnation and staleness, new technologies and devoted marketing campaigns have breathed life into the segment. Indeed,...
Best and Worst ISC West 2013 on Apr 15, 2013
Great attendance with a limited number of new products marked ISC West 2013. Brian, Carlton, Ethan and myself attended, putting together the following list of highlights for you to review: Bosch...