Facial Surveillance Used On US Politicians, Technical Challenges Remain

By: John Honovich, Published on Jun 12, 2018

Many fear that facial surveillance will be used against people by governments, such as the vast facial surveillance apparatus being built in China as well as the ACLU's complaints about US pilots.

Now, the New York Times has built their own facial surveillance system that they are using on US politicians as they seek to report on political issues in the US capital.

Inside this note, we examine the NY Times system, including problems they have faced, in particular with facial recognition matching issues.

Facial Recognition of US Congresspeople

As explained here, "How The New York Times Uses Software To Recognize Members of Congress", the NY Times built an application on top of Amazon's Rekognition service, which lets reporters take a picture of potential Congresspeople with their phones, text it to their service, let Amazon run facial recognition on it, returning potential face matches, as the demo image below shows:

The NY Times need was that, with 500+ Congresspeople, it is hard for every reporter to remember the face and name of every Congressperson, so that they could use this service to identify Congresspeople to help in pursuing stories. For example, they credited this application for their "Trump Groups Raised Millions, Then Paid It Out to Loyalists and a Trump Hotel" article.

Open Sourced

The NY Times has open sourced the application (see Github: Who The Hill: An MMS-based facial recognition service for members of Congress).

Amazon Rekognition Central Role

Get Notified of Video Surveillance Breaking News
Get Notified of Video Surveillance Breaking News

Like the ACLU case recently showed, Amazon's Rekognition service plays a central role in this application. Amazon makes it relatively cheap and easy to build your own 'DIY' facial surveillance system, as there are no expensive license fees up front, you pay as you go and it offers well-documented APIs for integration. 

Moreover, Amazon offers a specific API for recognizing celebrities. This means no work needs to be done by the user for this as tens of thousands of celebrities (actors, politicians, etc.) have been pre-indexed and trained by the Amazon system.

Indeed, Amazon offers a demo of the celebrity recognition service (for those with an AWS account). However, it is not always accurate as this IPVM experiment showed:

Doppelgangers / False Match Issues

Indeed, the NY Times faced a more serious false matching issue. They found certain false matches so significant that they had to hardcode their application with a 'doppelganger' feature to exclude known look-alikes. Below is the list with a sample of a few of the common false matches (Congressperson vs Doppelganger):

Here is another one, politician vs actor:

Problems With Facial Recognition

The problem is, fundamentally, lots of people look alike each other, especially when you get into larger populations of people (retail, public facilities, etc.) and not small offices.

Moreover, as the NY Times themselves learned in their application's usage:

our hit rate on photographs was very low because the halls of the Capitol are poorly lit and the photographs we took for testing were consistently marred by shadow and blur. [emphasis added]

That type of lamentation should be familiar to experienced video surveillance users where such conditions are commonplace.

On The Rise But Problems Too

On the plus side for facial recognition, this example of Amazon usage shows facial surveillance getting easier to do. On the negative side, these results also show the need to temper expectations about accuracy. For the NY Times use case where a reporter will only be periodically submitting requests and manually reviewing responses (looking just for Congresspeople), they should be able to fairly and quickly assess false matches. To the contrary, such results on a real-time video surveillance system would likely cause an overload of false alerts.

1 report cite this report:

UK Facial Recognition Essex Errors Report on Jul 05, 2019
Facial recognition trials in the UK have generated significant controversy...
Comments (2) : Members only. Login. or Join.

Related Reports

Exacq "Facial Matching" Facial Recognition Tested on Sep 03, 2020
Exacq is claiming "Accurate Recall of Every Noteworthy Person of Interest"...
Face Masks Increase Face Recognition Errors Says NIST on Aug 04, 2020
COVID-19 has led to widespread facemask use, which as IPVM testing has shown...
Bias In Facial Recognition Varies By Country, NIST Report Shows on Jul 15, 2020
While many argue that face recognition is inherently racist, results from one...
The US Fight Over Facial Recognition Explained on Jul 08, 2020
The controversy around facial recognition has grown significantly in 2020,...
U.S. Government Accountability Office Urges Facial Recognition Regulation on Aug 27, 2020
The US Government Accountability Office (GAO) is urging facial recognition...
Herta Facial Recognition Plus Masks Tested on Aug 19, 2020
Masks increase face recognition errors, but facial recognition developer...
30 Million Criminal Face Database Tested (Captis Intelligence) on Apr 27, 2020
30 million criminal mugshots are now available for facial recognition...
Face Shields Impact On Temperature Measurement And Mask Detection on Jul 27, 2020
First, the use of face masks, and now, plastic face shields are rising...
UK Court Rules Police Facial Recognition Needs Reform on Sep 01, 2020
A UK court has ruled that the South Wales Police use of facial recognition is...
FLIR Screen-EST Screening Software Tested on Jun 30, 2020
In our FLIR A Series Test, the cameras' biggest drawback was their lack of...
Clinton Public View Monitor (PVM) Mask Detection Tested on Jul 09, 2020
Face mask detection, or more specifically not wearing one, is expanding...
Face Detection Shootout - Dahua, Hanwha, Hikvision, Uniview, Vivotek on Jul 30, 2020
Face detection analytics are available from a number of manufactures...
Avigilon Face Mask Detection Tested on Jun 24, 2020
Face mask detection or, more specifically not wearing a face mask, is an...
Gait Recognition Examined on Sep 14, 2020
Facial recognition faces increasing ethical and political criticisms while...
Startup Solink $17 Million USD Fund Raise Expands To Mass Market on Jun 24, 2020
Solink has raised ~$17 million USD, a sizeable round for the company that...

Recent Reports

Hanwha AI Object Detection Tested on Sep 28, 2020
Hanwha has added detection and classification of people, cars, clothing...
Favorite Access Control Manufacturers 2020 on Sep 28, 2020
200+ Integrators told IPVM "What is your favorite access control management...
New Products Show Fall 2020 Starts Tomorrow! on Sep 27, 2020
Tomorrow, IPVM's sixth online show will feature New Products from over 25...
OnTech Smart Services Partners With Google and Amazon To Compete With Integrators on Sep 25, 2020
A pain point for many homeowners to use consumer security and surveillance is...
The Future of Metalens For Video Surveillance Cameras - MIT / UMass / Immervision on Sep 25, 2020
Panoramic cameras using 'fisheye' lens have become commonplace in video...
Hikvision Sues Over Brazilian Airport Loss on Sep 24, 2020
Hikvision was excluded from a Brazilian airport project because it is owned...
China General Chamber of Commerce Calls Out US Politics on Sep 24, 2020
While US-China relations are at an all-time low, optimism about relations...
Verkada Disruptive Embedded Live Help on Sep 24, 2020
Call up your integrator? Have someone come by the next day? Verkada is...
IP Networking Course Fall 2020 - Last Chance - Register Now on Sep 23, 2020
Today is the last chance to register for the only IP networking course...
Drain Wire For Access Control Reader Tutorial on Sep 23, 2020
An easy-to-miss cabling specification plays a key role in access control, yet...
Norway Council of Ethics Finds Hikvision Human Rights Abuses "Ongoing" on Sep 23, 2020
Hikvision's involvement in "serious human rights abuse" in Xinjiang is...
IPVM Camera Calculator User Manual / Guide on Sep 23, 2020
Learn how to use the IPVM Camera Calculator (updated for Version 3.1). The...
Installation Course Fall 2020 - Save $50 - Last Chance on Sep 22, 2020
This is a unique installation course in a market where little practical...
SimpliSafe Business Security Launched Examined on Sep 22, 2020
SimpliSafe has launched "SimpliSafe Business Security" that the company...
FLIR CEO: Many New Fever Entrants "Making Claims That The Science Just Won't Support" on Sep 22, 2020
FLIR's CEO joins a growing number calling out risks with fever / screening...