99.9997% Accurate Amazon Facial Rekognition Falsely Matches 28

Author: John Honovich, Published on Jul 27, 2018

A new facial recognition controversy has arisen, with the ACLU cleverly using Amazon's Rekognition service to falsely match 28 Congresspeople to mugshot photos.

However, the ACLU's results also show that the Amazon system was 99.9997% accurate and yet still generated those 28 false matches. While that may seem paradoxical, let us examine how that works.

99.9997% Accuracy

The ACLU test required Amazon Rekognition to conduct more than 13 million facial comparisons as it submitted the 535 members of Congress vs a 25,000 mugshot database. Amazon needs to compare all the combinations to determine whether any of them are a 'match', as our graphic below shows:

Out of those 13+ million comparisons, 28 of them were wrong, with Congresspeople being falsely identified as having been arrested [insert joke about Congresspeople being criminals, etc.]. The reality though is that in 99.9997% of the comparisons, Amazon made the correct decision, as the graphic below illustrates:

This is not a defense of Amazon; Rather it is to emphasize the fundamental challenges of doing facial recognition at a large scale. AI expert Andrew Ng explains this in the video embedded below:

Could Be Worse Actually

Get Video Surveillance News In Your Inbox
Get Video Surveillance News In Your Inbox

Indeed, the results of the ACLU test would have been worse if (1) the arrested people were of similar ages to the Congresspeople (i.e., the average Congressperson is ~60, the average arrested person's age is significantly less) and (2) surveillance video images were used (i.e., all images in this test were cooperative portrait style photos taken at direct angles, much easier to compare).

Amazon's Defense Ignores False Negatives

Amazon was quick to defend their facial recognition software, telling the Verge:

[Amazon] recommends at least a 95 percent threshold for law enforcement applications where a false ID might have more significant consequences. [emphasis added]

However, a major problem of raising it from the default 80% that the ACLU test used is that going to 95% radically increases the number of false negatives, that is the true matches that are not returned because the system is set at such a high threshold.

It is easy to demonstrate this with Amazon's Rekognition own demo, as the following examples we ran show how they would incorrectly not match at 95%:

At 95%, all of these valid true matches would be rejected by the system because their scores would be too low.

Fundamental Challenges Remain

At scale, false positives are going to be a significant risk, both for operationally managing them and for the risks to those people falsely matched. And eliminating those false positives, by raising the similarity threshold, will ensure many true positives are missed, i.e., false negatives, undermining the utility of the system. These challenges will remain and are at the core of increasing public debates about how this technology should be used.

1 report cite this report:

RealNetworks Free School Facial Recognition on Aug 03, 2018
The company that created RealPlayer is moving beyond media delivery and into the security space with a new facial recognition platform they have...
Comments (19) : PRO Members only. Login. or Join.

Related Reports on Facial Recognition

Dahua Face Recognition Camera Tested on Oct 15, 2018
Dahua has been one of the industry's most vocal proponents of the value that AI creates: As part of this, Dahua has released a facial...
Evidence Of Hikvision's Involvement With Xinjiang IJOP And Re-Education Camps on Oct 02, 2018
IPVM reveals as-yet unreported details about Hikvision’s activities in Xinjiang - a region of China where massive human rights abuses are being...
ASIS GSX 2018 Show Report on Sep 25, 2018
In the first major US show since the US government ban of Dahua and Hikvision was passed into law, the mega Chinese companies were in retreat and...
Congressional Letter Urges Sanctions Against Dahua and Hikvision For Human Rights Abuses on Sep 04, 2018
17 US Congresspeople sent a letter to the Secretary of State and Treasury urging sanctions against Chinese officials plus Dahua and Hikvision,...
Video Analytics Integration Guide on Aug 16, 2018
Video analytics is hot again (at least conceptually) but integrating video analytics with VMSes can be challenging. This is especially significant...
RealNetworks Free School Facial Recognition on Aug 03, 2018
The company that created RealPlayer is moving beyond media delivery and into the security space with a new facial recognition platform they have...
AI Startup Anyvision Raises $28 Million Led By Bosch on Jul 20, 2018
Anyvision is the most ambitious heavy-spending video surveillance startup in many years. And, now, the startup has raised $28 million led by...
FST Fails on Jul 17, 2018
FST was one of the hottest startups of the decade, selected as the best new product at ISC West 2011 and backed with tens of millions in...
Hikvision Wins Chinese Government Forced Facial Recognition Project Across 967 Mosques on Jul 16, 2018
Hikvision has won a Chinese government tender which requires that facial recognition cameras be set up at the entrance of every single mosque...

Most Recent Industry Reports

Hikvision Growth Declines Q3 2018 on Oct 22, 2018
Hikvision's growth continues to decline in 2018 going from: Q1 - 33% Q2 - 22% Q3 - 14.6% In this note, we examine Hikvision's newest Q3...
Geutebruck Company Profile on Oct 22, 2018
Geutebrück has been in business for nearly 50 years, but they are not well known within the US surveillance market. In this report, we profile...
Chinese Government Blocks IPVM on Oct 22, 2018
IPVM has been blocked by the Chinese government without any notice or explanation. This means IPVM.com is no longer officially accessible anywhere...
Startup SafePass Profile on Oct 19, 2018
A major problem with visitor management is that the systems mostly require adhesive printed paper labels and paper logs, creating waste and an...
China Is Not A Security Megatrend, Says SIA on Oct 19, 2018
The US Security Industry Association has released its 10 "Security Megatrends" for 2019. SIA declares that these megatrends, such as "Advanced...
Hanwha Dual Imager Dome Camera Tested (PNM-7000VD) on Oct 18, 2018
Hanwha has introduced their first dual-imager model, the PNM-7000VD, a twin 1080p model featuring independently positionable sensors and a snap-in...
Camera Height / Blind Spot Added to IPVM Camera Calculator on Oct 18, 2018
IPVM has added camera height and blind spot estimation to the Camera Calculator. This is especially helpful for those who need to mount cameras up...
Axis Strong US Growth, Flat EMEA - Q3 2018 Financials on Oct 18, 2018
This spring, Axis had its best financials in many years (see Axis Strong Q2 2018 Results). However, over the summer, Axis had many products sold...
Best Alternatives to Banned Dahua and Hikvision on Oct 17, 2018
With the US government ban and a growing number of users banning Dahua and Hikvision, one key question is what to use for low cost? While Dahua and...
Video Quality / Compression Tutorial on Oct 17, 2018
While CODECs, like H.264, H.265, and MJPEG, get a lot of attention, a camera's 'quality' or compression setting has a big impact on overall...

The world's leading video surveillance information source, IPVM provides the best reporting, testing and training for 10,000+ members globally. Dedicated to independent and objective information, we uniquely refuse any and all advertisements, sponsorship and consulting from manufacturers.

About | FAQ | Contact