99.9997% Accurate Amazon Facial Rekognition Falsely Matches 28

Author: John Honovich, Published on Jul 27, 2018

A new facial recognition controversy has arisen, with the ACLU cleverly using Amazon's Rekognition service to falsely match 28 Congresspeople to mugshot photos.

However, the ACLU's results also show that the Amazon system was 99.9997% accurate and yet still generated those 28 false matches. While that may seem paradoxical, let us examine how that works.

99.9997% Accuracy

The ACLU test required Amazon Rekognition to conduct more than 13 million facial comparisons as it submitted the 535 members of Congress vs a 25,000 mugshot database. Amazon needs to compare all the combinations to determine whether any of them are a 'match', as our graphic below shows:

Out of those 13+ million comparisons, 28 of them were wrong, with Congresspeople being falsely identified as having been arrested [insert joke about Congresspeople being criminals, etc.]. The reality though is that in 99.9997% of the comparisons, Amazon made the correct decision, as the graphic below illustrates:

This is not a defense of Amazon; Rather it is to emphasize the fundamental challenges of doing facial recognition at a large scale. AI expert Andrew Ng explains this in the video embedded below:

Could Be Worse Actually

Get Video Surveillance News In Your Inbox
Get Video Surveillance News In Your Inbox

Indeed, the results of the ACLU test would have been worse if (1) the arrested people were of similar ages to the Congresspeople (i.e., the average Congressperson is ~60, the average arrested person's age is significantly less) and (2) surveillance video images were used (i.e., all images in this test were cooperative portrait style photos taken at direct angles, much easier to compare).

Amazon's Defense Ignores False Negatives

Amazon was quick to defend their facial recognition software, telling the Verge:

[Amazon] recommends at least a 95 percent threshold for law enforcement applications where a false ID might have more significant consequences. [emphasis added]

However, a major problem of raising it from the default 80% that the ACLU test used is that going to 95% radically increases the number of false negatives, that is the true matches that are not returned because the system is set at such a high threshold.

It is easy to demonstrate this with Amazon's Rekognition own demo, as the following examples we ran show how they would incorrectly not match at 95%:

At 95%, all of these valid true matches would be rejected by the system because their scores would be too low.

Fundamental Challenges Remain

At scale, false positives are going to be a significant risk, both for operationally managing them and for the risks to those people falsely matched. And eliminating those false positives, by raising the similarity threshold, will ensure many true positives are missed, i.e., false negatives, undermining the utility of the system. These challenges will remain and are at the core of increasing public debates about how this technology should be used.

1 report cite this report:

RealNetworks Free School Facial Recognition on Aug 03, 2018
The company that created RealPlayer is moving beyond media delivery and into the security space with a new facial recognition platform they have...
Comments (19) : PRO Members only. Login. or Join.

Related Reports on Facial Recognition

Video Analytics Integration Guide on Aug 16, 2018
Video analytics is hot again (at least conceptually) but integrating video analytics with VMSes can be challenging. This is especially significant...
RealNetworks Free School Facial Recognition on Aug 03, 2018
The company that created RealPlayer is moving beyond media delivery and into the security space with a new facial recognition platform they have...
AI Startup Anyvision Raises $28 Million Led By Bosch on Jul 20, 2018
Anyvision is the most ambitious heavy-spending video surveillance startup in many years. And, now, the startup has raised $28 million led by...
FST Fails on Jul 17, 2018
FST was one of the hottest startups of the decade, selected as the best new product at ISC West 2011 and backed with tens of millions in...
Hikvision Wins Chinese Government Forced Facial Recognition Project Across 967 Mosques on Jul 16, 2018
Hikvision has won a Chinese government tender which requires that facial recognition cameras be set up at the entrance of every single mosque...
Belgium Bans Private Facial Surveillance on Jul 06, 2018
Belgium has effectively banned the use of facial recognition and other biometrics-based video analytics in surveillance cameras for private,...
GDPR / ICO Complaint Filed Against IFSEC Show Facial Recognition on Jun 20, 2018
IPVM has filed a complaint against IFSEC’s parent company UBM based on our concern that the conference violates core GDPR principles on...
IFSEC 2018 Final Show Report on Jun 20, 2018
IPVM attended the IFSEC show for the first time this year. The Chinese took over the show, centered on Hikvision, flanked by Dahua, Huawei and a...
China Public Video Surveillance Guide: From Skynet to Sharp Eyes on Jun 14, 2018
China is expanding its video surveillance network to achieve “100%” nationwide coverage by 2020, including facial recognition capabilities and a...

Most Recent Industry Reports

SNMP / Network Monitoring For Surveillance 2018 on Aug 21, 2018
Surveillance systems typically rely on the the VMS to report issues, but this most often just means knowing a camera is "down" with no warning or...
Axis First IR PTZ Tested (Q6125-LE) on Aug 21, 2018
Axis is very late in releasing IR PTZs. While competitors such as Hikvision and Dahua have offered them for years, Axis has just released their...
First US City Bans Hikvision on Aug 20, 2018
Just days after a ban of Hikvision was signed into law for the US federal government, the first US city has followed, banning Hikvision as...
Backup Power For Maglocks Guide on Aug 20, 2018
When the main power fails, many believe maglocks must leave doors unlocked. However, battery backed up maglocks are allowed according to IBC /...
Axis Strong Q2 2018 Results on Aug 20, 2018
Axis reported strong Q2 2018 financial results and, for the first time in many years, Axis grew faster than Hikvision. However, the company's...
2Gig Gun Lock / Motion Detector Tested on Aug 17, 2018
Safer guns for families and an opportunity for security dealers to sell more services? That is the aim of Nortek's 2GIG 'Gun Motion Detector'...
Video Analytics Integration Guide on Aug 16, 2018
Video analytics is hot again (at least conceptually) but integrating video analytics with VMSes can be challenging. This is especially significant...
Hikvision IP Camera Critical Vulnerability 2018 Disclosed on Aug 16, 2018
The same day that the US government passed a prohibition on Hikvision cameras, Hikvision disclosed a critical vulnerability for its IP...
ISS VMS / Video Analytics Company Profile on Aug 16, 2018
Who is ISS? In the past few months, they had one of the craziest ISC West promo items in years. Then, they hired industry veteran and ex-Dahua...
Chinese OEM Avycon Gets ADI Push on Aug 15, 2018
Who is Avycon? An American company? A Korean company? A couple of guys relabelling Chinese products? The latter is the best explanation. While...

The world's leading video surveillance information source, IPVM provides the best reporting, testing and training for 10,000+ members globally. Dedicated to independent and objective information, we uniquely refuse any and all advertisements, sponsorship and consulting from manufacturers.

About | FAQ | Contact