Bias In Facial Recognition Varies By Country, NIST Report Shows

By Zach Segal, Published Jul 15, 2020, 12:17pm EDT

While many argue that face recognition is inherently racist, results from one of the most extensive studies done on demographic bias in AI, the Facial Recognition Vendor Test (FRVT) Part 3 by the National Institute of Standards and Technolgy (NIST) analyzing over 100 algorithms, has shown that bias varied across the country of development.

IPVM Image

In particular, they observed that several algorithms developed by China groups performed better on East Asian faces than Eastern European faces while the vast majority of algorithms performed Eastern European faces than East Asians.

Executive *******

***** **** **% ** western ********* ********* ********** that ********* ****** ** Eastern ******** *** **** East ***** **** (***** 10-100 ***** ******), ~**% of ***** ********* ********* algorithms **** ********* ****** on **** ***** *** than ******* ******** ****.

**** ****** ******* ** all **** ***** *****, although *** ** ********, with ~**% ** ********** developed ** **** ***** groups ********** ****** ** Eastern ***** ***** **** European.

Training **** *****

**** ********* ** ***** of ****** ******** ****:

**** ******** **** ******** data, ** ******* **** other ****** ********* ** the ***********, *** ** effective ** ******** ********** false ******** *************. ****, the ******-**** ********** ***** be *** ********** ** investigate *** ******* ** more *******, ******** *******, training ****.

*******, **** *** *** an ********** ******** ** establish ***** *** **** did *** **** ** training ****. ***, **** result *** *** **** with ***** ***** ****** performance ** ***** ***** faces, ** ******* ****** may ******* *** *******.

****, ***** ** *** NIST ********* *******:

IPVM Image

Study ********

**** ****** **** *********** disparities ** ****** *********** algorithms ** ***** ****** facial *********** ****** ****, analyzing ********** **** **** 100 ******. *** ****** of ***** ****, ***** we **** ***** **, they ******** *** ********** in ***** ***** ***** (FMR) ** ****** ***** from ****-******* ** ***** Department *********** ******** (***/*** 39794-5 **************). ** ******* of * ***** ***** would ** ** * friend ******** **** ****** using ***** ****, ******* of *****. ** **** case, **** ******** ***,*** visa *********** ****** **** 441,517 ********* *********** ******, selecting ****** *** ******** to * ****** ** countries ****** *** *** historical ***********. *** **** researchers **** **** ***** of *** ****** ** false ******* **** ******** (all ******* **** ***** because *** **** **** disjoint).

**** ***** *********** ** false ***** *****, **** false ***** ***** ****** in ******* ********* (******, Ukraine, *** ******) *** highest **** **** ******** and **** ****** (*****, Japan, *****, ***********, ********, and *******). *******, *** study ***** ****

**** * ****** ** algorithms ********* ** ***** this ****** ** ********, with *** *****-******** ***** on **** ***** *****.

******** **** ******** **** or ********* *********** ***** be ****** * *** of *** ****** *** ethnic **** ******* ** most ** *** ********** they ******.

IPVM Image

How ** ********* *********** ***********

***** **-*** ***** **** accurate *** *. ********* than *. ******, ****** like * ***** **********, this *** ** **********. The *********** ** ********** can ** **** (**** algorithms *** ***** ***** of ~.***** *** *. Asians *** ~.****** *** E. *********), *** ** some *****, *, ** close ** * ****** would ** ********. ** an ********* ** **** for ************/*:* ********, **** as ***** ****** *********** to ****** **** ******, mistakes ***** ** ********** infrequent *** *** ********* would ** ****** ************. However, ** **** *** identification/1:n *********, **** ** scanning ** **** *** anyone **** ** *********** warrant, *** ***** **** increases ************* **** *, and *** ********* *** have * **** ****** (even **** **** ***) if * ** ***** enough.

**, ***** *** *********** differences *** ***, *** difference *** *** ** an ***** **** *:* verification *** *** ** a **** ***** (**** disparity ****** **** ***) if **** *** *:* identification.

Effect ** ********* ******

**** **% ** ***-****-***** made ********** ********* ****** (a ***** ***** ***** rate) ** ******* ******** faces **** ** **** Asian ****

***, ~**% ** ******* firms (********* *****, *********, and ******) *** ~**% of **** ***** ***** (expanded ** ******* ********* like ****** **** *** in **** **** *** not ** *** ***** of ********* ****** *** the **** *****) ********* better ** **** ***** faces **** ******* ********, as *** ***** ***** highlights:

IPVM Image

********

***** *** ******** ** widespread *********** **** ** a *******, *** **** that ********** ********* ** Chinese *** **** ***** teams ****** * ********* bias **** **** ** other *****, ******* **** much ** *** *********** disparity ** **** *********** accuracy *** ** ****** by *** *** ** diverse ******** ****. **** means **** ********** ****** ensure **** **** *** diverse ******** ****, ******* of **** ******* ** what ** ******* ** use. ** **** ***** that ********* ** ** teams *** **** ** better **********.

** ********, ***-***** ****** ask ***** *********** *********** in *** ********** **** are ******** ******* *** take ********* ****** **** consideration. ***** ** ********* may ******* ** ********** on *** **** ***********, it *** ************ ** the *********** ** ** used **.

***********

***** *** ***** *** rigorously *** **** ***, it *** *** ******* the ******** **** **** or **** ** ********* cause *** ******. **** means **** ***** ** can **** ******** ******* based ** *** *****'* findings, ** ****** ************ state ******** ** ****.

****, **** *****:

*** *** *** * corresponding *********** *** ***** Asian ***** *** *** few ********** [****] **** submitted ** ********** [** India *** **********].

*******, ** ** ***** possible **** ***** ***** Asian **** **** **** that *** *** ******* a ****** ********** ** South ***** **** ****** than ***-***** ***** ***** did. ** ** **** to **** ** ******* data **** ** ***** that *** ***** ****** to ****** ****** *********** neural ********, ** **** teams *** **** ****** on *** **** ****** accessible ********.

*** *** ** *** same ******** *** ******* why **** ***** **** clustered ******** *** *** bias *** ***** **** uniform, **** **** ********** performing ****** ** *** same ******** *** ****** groups.

Poll / ****

Comments (6)

John Honovich

IPVM | 07/15/20 04:19pm

****, **** ***!

***** *** **** ****** was **** * ****** ago, ****, **** ****, is ******* ** **** coverage ** ****** *********** technology. *** *********** *** topics ** *****, ****** share.

Avatar

Salvatore D'Agostino

IDmachines | 07/15/20 04:45pm

** **** ******* ****** out *** **** ****** did *** **** ** training ****. ** *** FR ******** ************ ******* on ******* ********, **** is ***** ****** ***** seem ** ** **** and **** ********. ******* a ****** **** **** training **** *** ******* a **** ** **** is *** ***** ** open ****** ****. **** will, ** ** *******, be ** *********** ***** to ****** **** ** not **** *** ****** to ***** **** ****** acquisition *** ******** **** becomes * *** ******** block ** ******** *** also *** ****** ** which **** ****** ******** sets. *** *********** ****** in **** ******** *** the ******* ************** ****** amassing * ***** **** set ** ******'* *****. This ***** ** *** question **, ** **** extent ***** * ********** training *** ** ******* that ***** ******* *** challenges ** *********** *** bias.

Anthony Hackett

In reply to Salvatore D'Agostino | 07/15/20 05:33pm

**** ***** ** *** question **, ** **** extent ***** * ********* training *** ** ******* that ***** ******* *** challenges ** *********** *** bias.

***** ***** *****, * can *** * ********* type ** ************* *** all ********** ************. **** would *** **** ***** to ** *** *** AI ** **** *** general ****** *** ******* entities *** ****** ** ethical *********** ** *** algorithms ***** ****.

Zach Segal

IPVM | In reply to Anthony Hackett | 07/15/20 06:54pm

* ***** ** ******* (Timnit *****, ***** ***********, Briana *********, ******** ******* Vaughan, ***** *******, *** Daumé ***, **** ********) wrote ** *********** ******* proposing ********* ***** ***** lines ********,********** *** ********. **** ***** **** all ******** ****** **** with ********** *** ***** use *** *********** ***** how *** **** *** collected *** **** ** contains.

** ***** ** *********** if *** ********** *** a "**** **** ***** dataset" ***** *** ****** accessible *********** ***** **** data.

***, *********, ********* **** is ** *********** ****, but ** ***** **** lead ** ********* *********** in ******** ******* * machine ******** ********* **** invariably **** *** ********* data ***** ** **** data. *******, ** ***** definitely ***** ******* ******.

Avatar

Salvatore D'Agostino

IDmachines | 07/15/20 07:05pm

** ****, ***** *********. I **** *** ****, its **** ******. ***** is * *** ***** on ***** *** ****** the ** *** *****. My ***** ** ***** will ** * *** more ** **** ** this *****. ******* **** is ******** ** ******* AI, *** **** * human *********** ***'* **** this ***** ** *** fore. ***** ** **** interesting ******* **** ** de-identification, *** ***** ******* related *******.

Zach Segal

IPVM | In reply to Salvatore D'Agostino | 07/15/20 07:14pm

***** *** *** * agree. ** ** * very ********* ***** *** only ******* **** ********.

Read this IPVM report for free.

This article is part of IPVM's 6,733 reports, 908 tests and is only available to members. To get a one-time preview of our work, enter your work email to access the full article.

Already a member? Login here | Join now
Loading Related Reports