99.9997% Accurate Amazon Facial Rekognition Falsely Matches 28

By: John Honovich, Published on Jul 27, 2018

A new facial recognition controversy has arisen, with the ACLU cleverly using Amazon's Rekognition service to falsely match 28 Congresspeople to mugshot photos.

However, the ACLU's results also show that the Amazon system was 99.9997% accurate and yet still generated those 28 false matches. While that may seem paradoxical, let us examine how that works.

99.9997% Accuracy

The ACLU test required Amazon Rekognition to conduct more than 13 million facial comparisons as it submitted the 535 members of Congress vs a 25,000 mugshot database. Amazon needs to compare all the combinations to determine whether any of them are a 'match', as our graphic below shows:

Out of those 13+ million comparisons, 28 of them were wrong, with Congresspeople being falsely identified as having been arrested [insert joke about Congresspeople being criminals, etc.]. The reality though is that in 99.9997% of the comparisons, Amazon made the correct decision, as the graphic below illustrates:

This is not a defense of Amazon; Rather it is to emphasize the fundamental challenges of doing facial recognition at a large scale. AI expert Andrew Ng [link no longer available] explains this in the video embedded below:

Could Be Worse Actually

Get Notified of Video Surveillance Breaking News
Get Notified of Video Surveillance Breaking News

Indeed, the results of the ACLU test would have been worse if (1) the arrested people were of similar ages to the Congresspeople (i.e., the average Congressperson is ~60, the average arrested person's age is significantly less) and (2) surveillance video images were used (i.e., all images in this test were cooperative portrait style photos taken at direct angles, much easier to compare).

Amazon's Defense Ignores False Negatives

Amazon was quick to defend their facial recognition software, telling the Verge:

[Amazon] recommends at least a 95 percent threshold for law enforcement applications where a false ID might have more significant consequences. [emphasis added]

However, a major problem of raising it from the default 80% that the ACLU test used is that going to 95% radically increases the number of false negatives, that is the true matches that are not returned because the system is set at such a high threshold.

It is easy to demonstrate this with Amazon's Rekognition own demo, as the following examples we ran show how they would incorrectly not match at 95%:

At 95%, all of these valid true matches would be rejected by the system because their scores would be too low.

Fundamental Challenges Remain

At scale, false positives are going to be a significant risk, both for operationally managing them and for the risks to those people falsely matched. And eliminating those false positives, by raising the similarity threshold, will ensure many true positives are missed, i.e., false negatives, undermining the utility of the system. These challenges will remain and are at the core of increasing public debates about how this technology should be used.

1 report cite this report:

RealNetworks Free School Facial Recognition on Aug 03, 2018
The company that created RealPlayer is moving beyond media delivery and into...
Comments (19) : Members only. Login. or Join.

Related Reports

Bias In Facial Recognition Varies By Country, NIST Report Shows on Jul 15, 2020
While many argue that face recognition is inherently racist, results from one...
Face Shields Impact On Temperature Measurement And Mask Detection on Jul 27, 2020
First, the use of face masks, and now, plastic face shields are rising...
Dahua, Hikvision, ZKTeco Face Mask Detection Shootout on Jun 19, 2020
Temperature tablets with face mask detection are one of the hottest trends in...
Exacq "Facial Matching" Facial Recognition Tested on Sep 03, 2020
Exacq is claiming "Accurate Recall of Every Noteworthy Person of Interest"...
30 Million Criminal Face Database Tested (Captis Intelligence) on Apr 27, 2020
30 million criminal mugshots are now available for facial recognition...
The US Fight Over Facial Recognition Explained on Jul 08, 2020
The controversy around facial recognition has grown significantly in 2020,...
Anixter Runs Fake Coronavirus Marketing Using Shutterstock Watermarked Images on Jul 24, 2020
Coronavirus faked marketing is regrettably commonplace right now but Anixter...
Herta Facial Recognition Plus Masks Tested on Aug 19, 2020
Masks increase face recognition errors, but facial recognition developer...
Convergint Refuses To Fix Faked Fever Marketing, FTC Complaint Filed on Jun 19, 2020
Since Convergint has refused to fix their faked fever camera marketing, IPVM...
Panasonic i-PRO Hid Huawei, Does Damage Control on Aug 21, 2020
Panasonic i-PRO hid their usage of Huawei from the public, continues to...
Face Detection Shootout - Dahua, Hanwha, Hikvision, Uniview, Vivotek on Jul 30, 2020
Face detection analytics are available from a number of manufactures...
Faked Convergint Fever Camera 'Expert' Marketing on Jun 16, 2020
Convergint touts they are "THERMAL CAMERA SOLUTION EXPERTS" while faking...
WDR Cheat Sheet and Camera Tracking - 30 Manufacturers on Aug 26, 2020
Manufacturers are regularly cryptic about what WDR support they actually...
Face Masks Increase Face Recognition Errors Says NIST on Aug 04, 2020
COVID-19 has led to widespread facemask use, which as IPVM testing has shown...
Mobotix Thermal Detection Camera Tested on Jun 09, 2020
For years Mobotix has struggled but now sales are surging driven by Mobotix's...

Recent Reports

OnTech Smart Services Partners With Google and Amazon To Compete With Integrators on Sep 25, 2020
A pain point for many homeowners to use consumer security and surveillance is...
The Future of Metalens For Video Surveillance Cameras - MIT / UMass / Immervision on Sep 25, 2020
Panoramic cameras using 'fisheye' lens have become commonplace in video...
Hikvision Sues Over Brazilian Airport Loss on Sep 24, 2020
Hikvision was excluded from a Brazilian airport project because it is owned...
China General Chamber of Commerce Calls Out US Politics on Sep 24, 2020
While US-China relations are at an all-time low, optimism about relations...
Verkada Disruptive Embedded Live Help on Sep 24, 2020
Call up your integrator? Have someone come by the next day? Verkada is...
IP Networking Course Fall 2020 - Last Chance - Register Now on Sep 23, 2020
Today is the last chance to register for the only IP networking course...
Drain Wire For Access Control Reader Tutorial on Sep 23, 2020
An easy-to-miss cabling specification plays a key role in access control, yet...
Norway Council of Ethics Finds Hikvision Human Rights Abuses "Ongoing" on Sep 23, 2020
Hikvision's involvement in "serious human rights abuse" in Xinjiang is...
IPVM Camera Calculator User Manual / Guide on Sep 23, 2020
Learn how to use the IPVM Camera Calculator (updated for Version 3.1). The...
Installation Course Fall 2020 - Save $50 - Last Chance on Sep 22, 2020
This is a unique installation course in a market where little practical...
SimpliSafe Business Security Launched Examined on Sep 22, 2020
SimpliSafe has launched "SimpliSafe Business Security" that the company...
FLIR CEO: Many New Fever Entrants "Making Claims That The Science Just Won't Support" on Sep 22, 2020
FLIR's CEO joins a growing number calling out risks with fever / screening...
China Bems Temperature Measurement Terminal Tested on Sep 22, 2020
Guangzhou Bems (brand Benshi) is the manufacturer behind temperature...
Axis Exports To China Police Criticized By Amnesty International on Sep 21, 2020
Axis Communications and other EU surveillance providers are under fire from...
Milestone XProtect on AWS Tested on Sep 21, 2020
Milestone finally launched multiple cloud solutions in 2020, taking a...