UK Facial Recognition Essex Errors Report

•Published Jul 05, 2019 13:23 PM

Facial recognition trials in the UK have generated significant controversy and debate over the past few years. This week, it flared again when Sky News / Guardian trumpeted that 81% of 'suspects' flagged were false. However, their posts did not analyze the details nor share the original research report. IPVM has.

london MPS police face rec report

In this report, we share and examine the 128-page report including:

The new and enhanced methodology that Essex used to analyze these results from more recent facial recognition trials in 2018 and 2019.
Why police / human review of matches was so critical and so difficult
How the 'accuracy' statistics vary and what the tradeoffs in different approaches were
How logs recorded show logistical challenges
How UK / EU / GDPR rules for consent complicate the challenges involved
China / PRC facial recognition systems contrasted

Report ********

************* ** ***** ***** ****** ****** has ******** * ***-**** ******:

Observing ******

*** **** ********* ********** ** **** Essex *** **** ** ** ********** observe *** ****** ****** *********** *********** so **** **** ***** *** **** happened **** *** ****** **** ******* the ****** *********** *** ********** *******. The ***** **** ***** ****** ***** the * ****** **** ********:

Problems ********* **** *********** *******

*** ******* ******* *** ******* **** insightful ******* ** *** ****** ** that ********* *** ***** ********** ******** verifying *******, *.*., ** *** ****** on *** ****** *** **** ****** as *** ****** *** ****** *********** system **** ********?

** ***** **** *** * ******, the ****** **** *** **** ******* the ***** *** *** ****** ** the ****** ** **** ******* *** person, *** **** ** * ** those ** ***** *** *** ****** flagged ** *** ****** *** ****** person ** *** ******, ** *** chart ***** *****:

***** ********* **** *** ************* ** humans ********* *** ****** ** * key *******:

Sky **** **% *********

*** **% ******** ** *** **** includes ** ******* **** *** ****** were **** ** ******* ******* ** being ***** ******* (*.*., *** ****-**** surveillance ***** *** ******* **** ********* than *** ********* *****):

***********, *** **% ** ***** ** there **** **** ** ******* **** were **** ** ** ******** *** of *****, ** **** ******** ** being *****, *** * **** ** 78.9%. *** ********** ** **** * matches **** ****** ** ** ********. Either ***, *** **** ** '****'.

Debate **** **********

**** *********** * ****** ** *** to ******* *** *********** ** ***** systems.

************* ****** ******** **** *** *** total ******* ** ***** ******* ****** them ********** ** **% ** ******. The ****** ******** **** ********:

** **** ****, *** ****** ****** scanned ********* ** ***** ** ****** walking ** **** ***** * ****** and****** **** ****** ** ** *****. Expressing ** **** *** (*** ** false / **,*** ***** *******) ***** a ***** ******** **** ** **** 0.3% ** ** '********' ** **.*%).

***** ** *****? **** ***, ** some *****. *** ******** ******* ***** is **** ********.

Hard ** Visually ******

*** ******* ** **** ** ** hard ** ******** ****** ******* * match ** ***** ** *****. ****** look *****. *** *******, ** ********* *********** ****, **** *** ** hardcode '*************'******* *** ****** *** ********** *** many ******, *.*.:

*** **** *** ********* **** ******* still ******, *** ************ ***** ***** where ******, ********, ******** *********, ***. can **** ** **** **** ** know *** **** ** **** ******* at * ******.

*************, *** ***** *** *** ******* any ******** ** *** ******* ** show *** **** ** *** ** tell ******* *** ********* ***** *** the ****-**** *****.

Logs **** **********

*** *********** ******** * ******** *** of ************ *** *** ** *** test ***********. ** ******** *** ******* case ** ***** ** * ******* being ********:

** **** ******** ******** ********. ******* the **** *********** ** * ** them ********* **** * *******. ** the *****, *** ********* ******** ***** whether * ***** ** ******** *** the ****** *** ***** ***** *** ready ** ****** ***** **** **** not ******** ********** *** *****:

*** *********** ******* ** **** **** of ***** **** ********** *** ********** scores (** *.** *** *.**). *** report ***** **** *** **** *********** vendor (***) *** * ********* ** 0.55:

*** ******* ********** ***** *** ** adjusted, *** ****** ** **, *** more ****** *** ****** ******* ***** matches *** ****** **** ******* *** vice *****. ** **** *** ****** NEC's ****** *********** ** ** ****** opine ** *********** ******** *** ******* this ***** ***** ********* ****** ***** matches *** *** ******* "**%" ***** rate.

Technology ************

*** ****** ***** **** *** **** reconigition *** *** ******* **** ********:

** ******* **** ******** ***** **** cameras **** ****.

Positioning *** *******

*** ******* **** ********* ******* ********, up ****, ** *** ***** ***** from *** ****** *****:

*** *********** ******** ******* ** **** the *** ****** ** ******* ******* and ***** ********* **** ****** *********** was ***** *********, ** **** ******* notes:

** *** **** ****, **** ****** were ******** ***** ******** *** ****** recognition:

China / *** **********

*** ******'* ******** ** ***** (***) issues ** **** ****** ******* ** facial *********** *** **** ** **** any ******* ** *********** ******** ** it ** ************* ********** ** ******** the *********** ** ***** *******.

**** ** **** ************** ** *** well ***** *** ****** *********** ******* work (****** ***** **** **** ******* anecdotes ** *** ********, *.*.************************* ***********).

*** ******* **** *** *** ****** recognition ******* ** **** **** **** more ***** ***** ***** ** ****, more ******** ****, **** ********* ****** (i.e., ******** ******** **), *** **** (or **) ***** ************.

*** ******* **** ** **** ****** look *******, ********** **** *** *** dealing **** ***** ********** (*.*., ****** only *** * *** ******* ** a *** ********, ***** *** ******* easily ***** **** ********). ** ** inevitable **** ***** *** ***** ** be ******** ******** **** **** **** humans ** ******** ****** ** **** care ***** ******** ** ********.

**********, ***** **** **** *** ******* are ****** ***** ***** *** ******* performance, ** **** ** **** ** really **** *** **** ** *****.

UK ****** *********** ******?

***** ******* **** ********** ***** *** future ** ****** *********** ** *** UK. ***** *** **** ** *** this (*******, ********, *********, ***.) *** the **** ** ******* ******** *******, it ** **** ** *** **** as ***** * '**********' ** *********** expense. ***** *** ****** *** *** people ***** ******** *** *** **** and ********** ********.

**** ********* *** ****** ** *** technology ******** *** *** ********** ** visually ********* ****** *** ****** ** fundamentally ****** ***** *** **** ****** look ***** ********** ***** *** *********** of ***** ******* ** ********** *******, open ****** ***********.

Comments (3)

Slava H

•Jul 05, 2019

Interesting review, thank you. It's useful to get real life usage results.

Somewhat similar report about NEC have been published by California Department of Justice https://www.vice.com/en_us/article/kbvkg3/california-spent-nearly-dollar18-million-on-controversial-facial-recognition-software

In our tests that NN trained on common face datasets (CASIA, VGG and such..) performs poorly in real life scenario, where regular 2MP/4MP cameras are used and face position is not as perfect as in datasets.

In comparison NEC "claims" (we had no chance to test ourselves) it has more advanced face recognition technology and historically performed well in regular NIST tests.

Reactions:

(3)

Undisclosed Integrator #1

•Jul 08, 2019

Manufacturers prefer measures that use the total numbers of faces scanned giving them accuracies of 99% or higher....

In this case, the system easily scanned thousands of faces of people walking by over these 6 trials and only 30 were proved to be false. Expressing it this way (say 30 false / 10,000 faces scanned) gives a false positive rate of just 0.3% or an 'accuracy' of 99.7%).

Do the manufacturers in any way take false negatives into account in their statistics? The way this is explained here sounds really fishy.

Reactions:

John Honovich

•Jul 08, 2019

IPVM

In reply to Undisclosed Integrator #1

In marketing, manufacturers generally just give big vague numbers, e.g., here is a Hikvision explanation:

It 'can hit 99%' is pretty vague. When does it hit 99%? How much lower can it be, etc.?

There are certainly ways to take false negatives into account, as there is an inherent tradeoff between false positives and false negatives.

I believe ROC curves are the standard means to demonstrate this, e.g.:

Image result for roc curve facial recognition

However, in manufacturer marketing, they are rarely, if ever shown.

Reactions:

(2)