99.9997% Accurate Amazon Facial Rekognition Falsely Matches 28

Author: John Honovich, Published on Jul 27, 2018

A new facial recognition controversy has arisen, with the ACLU cleverly using Amazon's Rekognition service to falsely match 28 Congresspeople to mugshot photos.

However, the ACLU's results also show that the Amazon system was 99.9997% accurate and yet still generated those 28 false matches. While that may seem paradoxical, let us examine how that works.

99.9997% Accuracy

The ACLU test required Amazon Rekognition to conduct more than 13 million facial comparisons as it submitted the 535 members of Congress vs a 25,000 mugshot database. Amazon needs to compare all the combinations to determine whether any of them are a 'match', as our graphic below shows:

Out of those 13+ million comparisons, 28 of them were wrong, with Congresspeople being falsely identified as having been arrested [insert joke about Congresspeople being criminals, etc.]. The reality though is that in 99.9997% of the comparisons, Amazon made the correct decision, as the graphic below illustrates:

This is not a defense of Amazon; Rather it is to emphasize the fundamental challenges of doing facial recognition at a large scale. AI expert Andrew Ng explains this in the video embedded below:

Could Be Worse Actually

Get Video Surveillance News In Your Inbox
Get Video Surveillance News In Your Inbox

Indeed, the results of the ACLU test would have been worse if (1) the arrested people were of similar ages to the Congresspeople (i.e., the average Congressperson is ~60, the average arrested person's age is significantly less) and (2) surveillance video images were used (i.e., all images in this test were cooperative portrait style photos taken at direct angles, much easier to compare).

Amazon's Defense Ignores False Negatives

Amazon was quick to defend their facial recognition software, telling the Verge:

[Amazon] recommends at least a 95 percent threshold for law enforcement applications where a false ID might have more significant consequences. [emphasis added]

However, a major problem of raising it from the default 80% that the ACLU test used is that going to 95% radically increases the number of false negatives, that is the true matches that are not returned because the system is set at such a high threshold.

It is easy to demonstrate this with Amazon's Rekognition own demo, as the following examples we ran show how they would incorrectly not match at 95%:

At 95%, all of these valid true matches would be rejected by the system because their scores would be too low.

Fundamental Challenges Remain

At scale, false positives are going to be a significant risk, both for operationally managing them and for the risks to those people falsely matched. And eliminating those false positives, by raising the similarity threshold, will ensure many true positives are missed, i.e., false negatives, undermining the utility of the system. These challenges will remain and are at the core of increasing public debates about how this technology should be used.

1 report cite this report:

RealNetworks Free School Facial Recognition on Aug 03, 2018
The company that created RealPlayer is moving beyond media delivery and into the security space with a new facial recognition platform they have...
Comments (19) : PRO Members only. Login. or Join.

Related Reports on Facial Recognition

Foolish Strategy: OEMing Facial Recognition on Dec 13, 2018
Almost as 'hot' as face recognition marketing right now is OEMing facial recognition. Last year, they were a who's who of company's with...
Akuvox Intercom Profile on Dec 06, 2018
Akuvox, a Chinese manufacturer of VoIP products, is expanding heavily into Video Intercom products with disruptive pricing targeted for commercial...
No GDPR Penalties For UK Swann 'Spying Hack' on Nov 20, 2018
The UK’s data protection agency has closed its investigation into Infinova-owned Swann Security UK, the ICO confirmed to IPVM, deciding to take “no...
ISC East 2018 Mini-Show Final Report on Nov 16, 2018
This is our second (updated) and final show report from ISC East. ISC East, by its own admission, is not a national or international show, billed...
Facial Detection Tested on Nov 16, 2018
Facial detection and recognition are increasingly offered by video surveillance manufacturers. Facial detection detects faces in an image/video...
Huawei To Be “Top Three Global Position” in Video Surveillance on Oct 29, 2018
Huawei - China’s biggest networking gear and smartphone maker - plans to become one of the top three global video surveillance manufacturers in...
Axxon Face Search Tested on Oct 26, 2018
AxxonSoft has brought facial recognition to their Axxon Next VMS for free with the simply named Face Search, claiming to allow users to find...
When Your Salesperson Creates An International Incident (Amazon / ICE) on Oct 23, 2018
Having your salespeople be aggressive and go after big accounts is a virtue, mostly. On occasions, it can cause problems. A new case involves...
Dahua Face Recognition Camera Tested on Oct 15, 2018
Dahua has been one of the industry's most vocal proponents of the value that AI creates: As part of this, Dahua has released a facial...
Evidence Of Hikvision's Involvement With Xinjiang IJOP And Re-Education Camps on Oct 02, 2018
IPVM reveals as-yet unreported details about Hikvision’s activities in Xinjiang - a region of China where massive human rights abuses are being...

Most Recent Industry Reports

Imperial Capital Security Investor Conference 2018 Review - ADT, Resideo, Alarm.com, Arlo, Eagle Eye, ACRE, More on Dec 14, 2018
Imperial Capital Security Investor Conference is an event matching industry executives with financiers that frequently leads to future funding...
Cisco Meraki New Cameras and AI Analytics on Dec 14, 2018
Meraki has released their second generation of video surveillance with 3 new cameras, AI-based video analytics, and 2 cloud-based storage...
Foolish Strategy: OEMing Facial Recognition on Dec 13, 2018
Almost as 'hot' as face recognition marketing right now is OEMing facial recognition. Last year, they were a who's who of company's with...
DVR Examiner - Video Recovery from Recorder Hard Drives on Dec 13, 2018
Bypassing passwords and long download times on-site, DVR Examiner collects and organizes video evidence directly from a hard drive extracted from...
2019 Access Control Book Released on Dec 12, 2018
This is the best, most comprehensive access control book in the world, based on our unprecedented research and testing has been significantly...
Huawei Hisilicon Quietly Powering Tens of Millions of Western IoT Devices on Dec 12, 2018
Huawei Hisilicon chips are powering, at least, tens of millions of Western IoT devices, such as IP cameras and surveillance recorders, a fact that...
FLIR Launches Body Cameras Unified With VMS (TruWitness) on Dec 11, 2018
While FLIR is best known for their thermal cameras, now they have expanded into body cameras, launching TruWITNESS, a public safety focused body...
Startup Sunflower Labs' Autonomous Drone Security System on Dec 11, 2018
Startup Sunflower Labs is claiming a unique design on a home security system, combining autonomous drones and 'Sunflower' sensors. Imagine an...
The 2019 Video Surveillance Industry Guide on Dec 10, 2018
The 300 page, 2019 Video Surveillance Industry Guide, covers the key events and the future of the video surveillance market, is now available,...
Multi-Factor Access Control Authentication Guide on Dec 10, 2018
Can a stranger use your credentials? One of the oldest problems facing access control is making credentials as easy to use as keys, but restricting...

The world's leading video surveillance information source, IPVM provides the best reporting, testing and training for 10,000+ members globally. Dedicated to independent and objective information, we uniquely refuse any and all advertisements, sponsorship and consulting from manufacturers.

About | FAQ | Contact