99.9997% Accurate Amazon Facial Rekognition Falsely Matches 28

Author: John Honovich, Published on Jul 27, 2018

A new facial recognition controversy has arisen, with the ACLU cleverly using Amazon's Rekognition service to falsely match 28 Congresspeople to mugshot photos.

However, the ACLU's results also show that the Amazon system was 99.9997% accurate and yet still generated those 28 false matches. While that may seem paradoxical, let us examine how that works.

99.9997% Accuracy

The ACLU test required Amazon Rekognition to conduct more than 13 million facial comparisons as it submitted the 535 members of Congress vs a 25,000 mugshot database. Amazon needs to compare all the combinations to determine whether any of them are a 'match', as our graphic below shows:

Out of those 13+ million comparisons, 28 of them were wrong, with Congresspeople being falsely identified as having been arrested [insert joke about Congresspeople being criminals, etc.]. The reality though is that in 99.9997% of the comparisons, Amazon made the correct decision, as the graphic below illustrates:

This is not a defense of Amazon; Rather it is to emphasize the fundamental challenges of doing facial recognition at a large scale. AI expert Andrew Ng explains this in the video embedded below:

Could Be Worse Actually

Get Video Surveillance News In Your Inbox
Get Video Surveillance News In Your Inbox

Indeed, the results of the ACLU test would have been worse if (1) the arrested people were of similar ages to the Congresspeople (i.e., the average Congressperson is ~60, the average arrested person's age is significantly less) and (2) surveillance video images were used (i.e., all images in this test were cooperative portrait style photos taken at direct angles, much easier to compare).

Amazon's Defense Ignores False Negatives

Amazon was quick to defend their facial recognition software, telling the Verge:

[Amazon] recommends at least a 95 percent threshold for law enforcement applications where a false ID might have more significant consequences. [emphasis added]

However, a major problem of raising it from the default 80% that the ACLU test used is that going to 95% radically increases the number of false negatives, that is the true matches that are not returned because the system is set at such a high threshold.

It is easy to demonstrate this with Amazon's Rekognition own demo, as the following examples we ran show how they would incorrectly not match at 95%:

At 95%, all of these valid true matches would be rejected by the system because their scores would be too low.

Fundamental Challenges Remain

At scale, false positives are going to be a significant risk, both for operationally managing them and for the risks to those people falsely matched. And eliminating those false positives, by raising the similarity threshold, will ensure many true positives are missed, i.e., false negatives, undermining the utility of the system. These challenges will remain and are at the core of increasing public debates about how this technology should be used.

1 report cite this report:

RealNetworks Free School Facial Recognition on Aug 03, 2018
The company that created RealPlayer is moving beyond media delivery and into the security space with a new facial recognition platform they have...
Comments (19) : PRO Members only. Login. or Join.

Related Reports on Facial Recognition

Massive Leak Of Chinese VMS Provider Exposes Xinjiang Surveillance on Feb 20, 2019
A subsidiary of China’s claimed largest VMS provider is tracking the precise location and ethnicity of millions in China’s Xinjiang region,...
Casino Surveillance Pro Interview: James Lathrop on Feb 15, 2019
James Lathrop has been working in casinos for almost 25 years. During that time, he says he has held "just about every job you can do in the...
Intersec 2019 Show Report on Jan 23, 2019
The 2019 Intersec show, held annually in Dubai, is now complete. IPVM attended for 3 days, interviewing numerous Chinese and Western video...
NYPD Refutes False SCMP Hikvision Story on Jan 18, 2019
The NYPD has refuted the SCMP Hikvision story, the Voice of America has reported. On January 11, 2018, the SCMP alleged that the NYPD was using...
The False SCMP Story on Hikvision NYC AI on Jan 14, 2019
In the past week, one of Asia's largest publications, the South China Morning Post (SCMP), posted an article about "Chinese [facial recognition]...
CES 2019 Show Report on Jan 10, 2019
IPVM was at CES 2019 for the second year (see our CES 2018 Show Final Report) and is reporting on announcements and interesting news from the show...
IPVM Best New Products 2019 Opened - 70+ Entrants on Jan 07, 2019
The inaugural IPVM Best New Product Awards has been opened - the industry's first and only program where the awards are not pay-to-play and the...
CyberExtruder Face Recognition Profile on Jan 04, 2019
CyberExtruder offers 3D modeling face recognition software that they say provides quicker and more accurate matches than other 2D face recognition...
Foolish Strategy: OEMing Facial Recognition on Dec 13, 2018
Almost as 'hot' as face recognition marketing right now is OEMing facial recognition. Last year, they were a who's who of company's with...
Akuvox Intercom Profile on Dec 06, 2018
Akuvox, a Chinese manufacturer of VoIP products, is expanding heavily into Video Intercom products with disruptive pricing targeted for commercial...

Most Recent Industry Reports

Outdoor Camera Mounting Hardware Guide on Feb 21, 2019
Mounting cameras outdoors can be challenging, requiring understanding different types of equipment and methods. In this guide, we teach this...
HID Favorability Results 2019 on Feb 21, 2019
HID favorability results were strong, in the 2019 IPVM integrator study of 200+ integrators, with a net +62% and low negativity as the table below...
First US State, Vermont, Bans Dahua and Hikvision on Feb 21, 2019
The first US state, Vermont, has issued a ban on a number of Chinese and Russian manufacturers including the world's 2 largest video surveillance...
ADI 'SAVE BIG' On FLIR And Hikvision Examined on Feb 20, 2019
One is a major US defense supplier. The other is owned by the Chinese government. But you can "SAVE BIG" on both at ADI. In this note, we...
BluB0x Company Profile on Feb 20, 2019
BluB0x has doubled in revenue every year since its founding in 2013, according to CEO Patrick Barry. We originally reported on them in 2015. At the...
Security Installation Tools Guide - 22 Tools Listed on Feb 19, 2019
In this guide, we cover 22 tools that security installers frequently use. This is one part of our upcoming Video Surveillance...
Sales Cuts At Rasilient on Feb 19, 2019
Over the past 2 years, video surveillance storage specialist Rasilient has expanded its workforce significantly, aiming to build its own branded...
Exacq Raises VMS Software Pricing Twice in Less Than a Year on Feb 18, 2019
Most VMSes regularly release new features, but rarely increase their prices. For the 3rd time in 4 years, and 2nd time in 8 months, since being...
Axis IR Multi Imager Camera Tested (P3717-PLE) on Feb 18, 2019
Axis has released their first IR multi imager, the P3717-PLE, a repositionable model listing 360° IR illumination and flexible positioning,...

The world's leading video surveillance information source, IPVM provides the best reporting, testing and training for 10,000+ members globally. Dedicated to independent and objective information, we uniquely refuse any and all advertisements, sponsorship and consulting from manufacturers.

About | FAQ | Contact