Amazon Rekognition Facial Detection Tested

By IPVM Team, Published Dec 03, 2018, 08:48am EST

Amazon's Rekognition service has announced improvements, that if accurate, would be a significant advance over conventional facial detection.

amazon recognition

In particular, Amazon claims:

Amazon Rekognition can now detect 40 percent more faces – that would have been previously missed – in images that have some of the most challenging conditions described earlier... These aspects might include pose variations caused by head movement and/or camera movements, occlusion due to foreground or background objects (such as faces covered by hats, hair, or hands of another person in the foreground), illumination variations (such as low contrast and shadows), bright lighting that leads to washed out faces, low quality and resolution that leads to noisy and blurry faces, and distortion from cameras and lenses themselves. [emphasis added]

IPVM developed a dataset of 100+ 3-second video clips:

facial detection dataset 2

These clips are done across a number of common real-world scenarios:

  • Day light (200 lux)
  • Low light (1 lux)
  • Side tilt (from 0° to 90°)
  • Down tilt (from 0° to 30°)

We compiled various combinations (such as 200 lux, 15° side tilt, 22° down tilt or 1 lux, 75° side tilt, 30° down tilt, etc.).

We then ran tests of these 100+ clips across facial detection approaches:

  • HAAR
  • HOG
  • Openvino (specifically the face-detection-retail-0004 model on Intel i7 and Myriad X chips expanded upon from our NCS2 test)
  • Amazon Rekognition

Inside, we explain full results and key variances in performance.

Key ********

****** *********** ****** ********* accuracy *** *** ****** than *** ****** ****** including ****** ** **** of ********:

facial detection overall accuracy

*** ****** *** ********* quite '***' *** **** of *** ******** ** used **** ***** ****, including *** ********* *** bad ******.

*** *******, **** ** segment *** ****** ** key ********** ****, **** for ****** ***********, ***** differences ** *********** ******* are *******:

rekognition accuracy segmented

**** ******* *** *** severe ****** ******** ***, with '****' ***** *********** at ****** **% ***** low ***** *********** ** under **%.

***** *********** *** ********* in ****** ********* ********, it ***, *** *** away, *** *******, ** two ****** ** ********* versus ******** *** **** running ** ** **:

facial detection fps

*********** ** **** ****** than *** ***** ******** as *** *********** ***** service ******** *** ***** to ** **** ***** to *** ***** *** other ******** *** ** done ******* / ** the **** / ***** the ****** **. ** examine ***** *********** ******* in *** ***** ******* of **** ******.

********

*****, ** ******* ** 100% ******** *** ****** Rekognition, ***** ** ****** strong *********** *** **** is **° **** *** 45° **** **** **** the ******:

[video-to-gif output image]

****, ** ******* ** 59% ******** (******* ~* out ** ** ****** correctly ******* * ****) for ****** ***********, ****** a ***** ********* ** steep **** *** **** tilt **** *** *******'* head ****:

*** ***** ******, ** noted, **** **** **** challenging **** *** ***********. For *******, **** * out ** ***** * frames *** * **** accurately ******** ****** **** a ***** ***** *** the ****** / ********:

*******, *** ** *** few **** ************ *** Rekognition *** ** **** test ********, ****** *** light, *** ******* ****** straight ******* *** ****** but ***** **** *** not ******** ** ***:

Accuracy ***********

*** ******** ****** **** is ************** *** * simplification. *********** **** ********* evaluation **** * ****** called"**** ******* *********" (***), ***** ******** ***********-**-******* on ****** **** ********* datasets ************** ** ****** **** bounding-box ******* (********* ****** "Intersection **** *****" ** IoU) *** ******* **** box ********* ******** * face. ** ****'* ***** library, **** **** ****** has *** **** (***** frame *** * ****, and ***** ** **** one). ***** ****, ** can *** * ****** heuristic *** ********: ** how **** ****** *** a **** ******** (******** / *****-***** = ********). This ** *** *******: in **********, ** **** not ******* *** ***** positives (* **** ** detected ***********, **** ** the **** ** ** the *******'* *****). **** happens ****** - ** we ***** **** *** heuristic. ** ****** ** more ******* **** ********** (lux, *****, ***) ******* having ** ***** ******** boxes *** ***** ***** of *** *******.

FPS ***********

*** ******* **** ****** used *** *********** ***:

  1. ****** *** ***** ** Amazon **
  2. ***** *** *****
  3. ***** *********** (******** ** to *** ** ****, registering *****->***********)
  4. **** *** *****, ********* FPS

* **** ****** ******** for ************ ***** ** to ****** ****** ******** via*******->***********. **** ***** *** the ********** ******** ** a *****-**-***** *****, ***** our ***** ***** *** networking ******** **** *** FPS *********** ********. ** such, *** *** ***********, already **** ***, ***** be **** *****.

***** ** **** ************, this ** * ***** point ******* ***********, ** others ** *** ** a *******. *** *******, one *** *** ***-***** hardware *** ****** ******* performance. *** ***** ****** in **** ******* *** on ****** * ** Intel ** ******* (****-***) devices, ** **** *********** trade-off ** *********-*********. ************, Rekognition ** * ******* service, ***** *********** * model (**** ********'* ****-******-****) takes **** ************* *** setup. ** ***** ****-**** performance **** *********** ** unlikely (****: ** *** still ************* ** **** is ********), *** ***-****-**** applications, *********** ***** ** quite ******.

*******

*** * ****-**** ***** surveillance ***********,*********** ************ ** ***** **** but *** ***** ************ where **** ********** ***** processing ** ******, ** could ** *** **** affordable **** **-**** **********. For *******, *********** ******* for **-**** (*. ********) is $*.** *** * min ** **** ****** video ********. **** **** just **% ******** ** analyze (*.*., *,*** ******* in * *****), *** monthly **** ***** ** ~$50 (*,*** * **% x $*.**) **** *** the **** *********.

** *** ***** ****,****** *********** ****** ******* features********* ****** ***********, ****** analysis (***, ******, *******), object *** ********** *********, etc. **** **** ********* provider ****** ** *** have ** ***** ** likely ** ******** ** compete **** ***** ***********.

********

**** ** *** *** of ****'* *** ****** of **** ******** *****, starting ********** ********* ******,***** ****** ******* ***** 2 / ******** ** Test*** *** ****. ** are ******** * ******* of *** ***** ** the ******** ******. *** feedback, ******** ** *********, please *** ** *** comments.

Comments (8)

John Honovich

IPVM | 12/03/18 01:52pm

For those of you interested in the fundamentals, check out our new class next week:

deep learning webinar2

Tyler will be explaining fundamentals as part of our parallel effort to provide comprehensive training and education, in addition to our ongoing tests.

Undisclosed Manufacturer #1

12/03/18 02:50pm

Was any testing done running the algorithms with IR nightvision on? Most cameras would switch to IR nightvision (if they have the option) at 1 lux

John Honovich

IPVM | In reply to Undisclosed Manufacturer #1 | 12/03/18 09:02pm

We did not do any IR versions but we will try some in future tests.

Avatar

Matthew Del Salto

BLACKBOX USA
12/03/18 03:12pm

Just wait until they integrate it with Ring if they haven't already.

Undisclosed #2

12/03/18 07:45pm

Is there any way you can post all of your raw video used in the tests for IPVM users to download and run thru their own Analytics systems?

Undisclosed Manufacturer #3

In reply to Undisclosed #2 | 12/03/18 08:02pm

I think this would be a really great way to compare products.

John Honovich

IPVM | In reply to Undisclosed #2 | 12/03/18 09:03pm

We might release the dataset but have not decided yet.

Undisclosed #4

04/08/20 05:51am

Even with just 10% activity to analyze (i.e., 4,380 minutes in a month), the monthly cost would be ~$50 (4,380 x 10% x $0.12) just for the face analytics.

Aren't you considering the "10%" factor twice in this calculation? 4,380 minutes itself is 10% of total minutes in a month.

Read this IPVM report for free.

This article is part of IPVM's 6,803 reports, 913 tests and is only available to members. To get a one-time preview of our work, enter your work email to access the full article.

Already a member? Login here | Join now
Loading Related Reports