Masks Cause Major Facial Recognition Problems

By: Rob Kilpatrick, Published on Feb 24, 2020

Coronavirus is spurring an increase in the use of medical masks, which new IPVM test results show cause major problems for facial recognition systems.

IPVM Image

IPVM tested four facial recognition systems to see how they performed with masks:

The 1-minute video below overviews our findings:

Masks Drastically Reduce Recognition Performance

In our tests, facial recognition confidence dropped dramatically (over 50 points confidence) when wearing anti-viral face masks, with subjects not recognized at all in most cases, and no faces even detected much of the time.

Only Avigilon's Appearance Alerts was able to consistently recognize subjects, but this required confidence to be decreased from high to medium or low. While recognition was possible, it is certain that this will result in false identifications in large populations with hundreds or thousands of subjects.

Further Training Possible

It is possible that these manufacturers could improve masked face performance with further training of their algorithms. However, fundamentally, analytics will have much fewer points to base recognition on as so much of the face is covered, making recognition difficult, even with training.

Drastically Reduced Confidence

Shown below, when compared to the same subject not wearing a mask, confidence dropped by over 60 points when wearing a medical mask, from nearly 100% confidence to just over 30% in the Hikvision dual lens face rec camera.

IPVM Image

In order to recognize a face at these levels, minimum confidence needed to be reduced to ~25%. However, recognition remained inconsistent at this setting, with many subjects detected, but not recognized. This was especially common when hair covered the subject's forehead, making it more difficult to detect the face.

IPVM Image

IronYun performed similarly, with drastically reduced confidence and frequent total misses, even when using low similarity score requirements. For example, the face below matched with only 30-40 similarity, instead of 60-70 when the same subject was not wearing a mask.

IPVM Image

Avigilon Face Recognition Most Accurate But Only Medium/Low Confidence

In our tests, Avigilon's facial recognition had the best recognition rates, recognizing all subjects walking through the scene when using medium or low confidence settings (Avigilon does not show exact confidence scores).

IPVM Image

However, note that while recognition worked in our tests using a small population, users should not assume it would function similarly in larger installations where populations are more diverse. It is certain that false recognitions and missed recognitions will increase as the number of subjects increases because with more people this is more opportunity for people to look or appear to the system as looking similarily.

No Faces Detected In Masks: Verkada

In our tests, Verkada facial recognition completely missed faces of people walking through the scene when wearing masks. These subjects's faces were not detected at all and not searchable from Verkada's face list.

IPVM Image

All Systems Had Miss Detections

Our testing showed many miss facial detections - meaning that the system did not even recognize a face was present, which is necessary to even try to categorize what person's face it is:

  • Avigilon rarely missed faces walking through the scene.
  • Hikvision had the second best recognition rate but at the cost of a lower confidence threshold of 25. The default higher threshold had significant misses.
  • Ironyun missed people frequently and barely detected faces in the scene.
  • Verkada did not recognize anyone in the scene wearing a medical mask.

Improvements In Facial Detection With Masks Certain

Detecting a face with a mask is a solvable problem. The issue is most systems likely did not train their systems to recognize faces with masks. This can be solved by adding new training data of faces with masks on. As such, we would expect future firmware upgrades from many vendors to include that.

Harder Challenge To Recognize Faces With Masks

While detecting performance should go up, recognizing performance faces fundamental limits. Masks hide the chin, nose, mouth, lips, and jaw, essentially constraining systems to match on eyes and forehead.

IPVM Image

Trying to recognize faces can still try to be done and even sometimes or often be correct but fundamentally the accuracy, especially with larger scale, will be far reduced.

The best-case scenario is, structurally, building access control where limited populations, well-controlled lighting, and people presenting their faces increases the odds of accuracy. For example, China's Hanwang Technology is already claiming to do so, as their marketing video below demonstrates:

How well it actually works and how many mistakes are made is not explained. However, access control can also be tuned to allow more matches / access if the assumption is made that it is better to let people in quickly than block valid users from entering.

By contrast, facial recognition in video surveillance, e.g., the oft-cited smart city examples are going to face far harder examples - outdoors, poor weather, wide areas, etc., that combined with face masks are likely to make facial recognition mistakes the norm, even with optimizations.

6 reports cite this report:

Favorite Access Control Credentials 2020 on Sep 15, 2020
Credential choice is more debated than ever, with hacking risk for 125kHz and...
Face Masks Increase Face Recognition Errors Says NIST on Aug 04, 2020
COVID-19 has led to widespread facemask use, which as IPVM testing has shown...
Trueface Presents AI Face Recognition, Mask and Temperature Detection on Jun 10, 2020
Trueface presented its AI facial recognition, mask and temperature detection...
Anyvision Presents AI Facial Recognition and Mask Detection on Jun 08, 2020
AnyVision presented its AI facial recognition and mask detection at the May...
TVT / InVid Facial Recognition Tested on Mar 25, 2020
Facial recognition is frequently sold for thousands of dollars per channel...
Facial Recognition 101 on Mar 18, 2020
Facial recognition interest, use and fear is increasing. This guide aims to...

Comments (10)

Only IPVM Members may comment. Login or Join.

For example, the face below matched with only 30-40 similarity, instead of 60-70 when the same subject was not wearing a mask.

I am not sure I would recognize Ethan anymore with or without a mask :)

Surely the Verkada could recognize dinosaur faces with masks.

Are you seriously saying there was an expectation that facial recognition would work with face masks?

We are seriously saying that companies and pundits were claiming this so we did a real test to verify.

Now, there are people on that side who can still say they think it will work but we now have actual testing to show what happens.

I presume all tests were done with the same resolution?

Yes, they were all done with 1080p cameras (that was the only resolution available for Hikvison and Verkada), though we also used 4K for Avigilon and Ironyun, the only difference between 1080p and 4K with faces masks was the distance detected.

Basically, anything that ere-defines the jawline, (see picture) will subvert the facial recognition. This is particularly noted on a recent Insane Clown Posse tour. (Seriously)

Reports now say facial recognition can work even with masks on, not sure how true it is.

Thanks, #5, There were reports similar released before our cast was published, which we cited in the post above.

The reality is that accuracy is going to decline regardless of what optimization or attempted, as these results show.

So what those manufacturers need to prove is how well it works, at scale, in the wild, etc., not just that it can theoretically work.

Meanwhile, a Chinese company called Hanvon claims to have developed the technology to do 95% accurate face rec with masks. They even claim they can identify everyone in a crowd of up to 30 (masked) people “within a second”. Pretty ridiculous.

Related Reports

Herta Facial Recognition Plus Masks Tested on Aug 19, 2020
Masks increase face recognition errors, but facial recognition developer...
Face Masks Increase Face Recognition Errors Says NIST on Aug 04, 2020
COVID-19 has led to widespread facemask use, which as IPVM testing has shown...
Avigilon Face Mask Detection Tested on Jun 24, 2020
Face mask detection or, more specifically not wearing a face mask, is an...
Clinton Public View Monitor (PVM) Mask Detection Tested on Jul 09, 2020
Face mask detection, or more specifically not wearing one, is expanding...
Facial Recognition: Weak Sales, Anti Regulation, No Favorite, Says Security Integrators on Jul 07, 2020
While facial recognition has gained greater prominence, a new IPVM study of...
Dahua, Hikvision, ZKTeco Face Mask Detection Shootout on Jun 19, 2020
Temperature tablets with face mask detection are one of the hottest trends in...
The US Fight Over Facial Recognition Explained on Jul 08, 2020
The controversy around facial recognition has grown significantly in 2020,...
Anyvision Presents AI Facial Recognition and Mask Detection on Jun 08, 2020
AnyVision presented its AI facial recognition and mask detection at the May...
Anixter Runs Fake Coronavirus Marketing Using Shutterstock Watermarked Images on Jul 24, 2020
Coronavirus faked marketing is regrettably commonplace right now but Anixter...
Beware Rigged China Fever Cameras on Sep 08, 2020
Many China fever camera manufacturers have rigged algorithms dynamically...
Bias In Facial Recognition Varies By Country, NIST Report Shows on Jul 15, 2020
While many argue that face recognition is inherently racist, results from one...
FLIR Markets Windows Temperature Screening, Violates IEC And Causes Performance Problems on Jul 17, 2020
FLIR, one of the largest thermal screening manufacturers, is marketing...
Face Shields Impact On Temperature Measurement And Mask Detection on Jul 27, 2020
First, the use of face masks, and now, plastic face shields are rising...
Exacq "Facial Matching" Facial Recognition Tested on Sep 03, 2020
Exacq is claiming "Accurate Recall of Every Noteworthy Person of Interest"...
FLIR Screen-EST Screening Software Tested on Jun 30, 2020
In our FLIR A Series Test, the cameras' biggest drawback was their lack of...

Recent Reports

Temperature Tablet Shootout - Dahua, Hikvision, ZKTeco, TVT + 5 More on Sep 30, 2020
Temperature tablets, aka terminal or stations, have emerged as a 'low-cost...
New Products Show Fall 2020 Tomorrow Bosch, FLIR, Hanwha, Tyco, Avigilon More! on Sep 30, 2020
IPVM's sixth online show concludes tomorrow with our special temperature...
ButterflyMX Raises $35 Million on Sep 30, 2020
Startup ButterflyMX has raised $35 million for its smartphone based intercom...
Worst Access Control Manufacturers 2020 on Sep 30, 2020
200+ Integrators told IPVM "In the past year, what access control...
Access Control Levels and Schedules Tutorial on Sep 29, 2020
Configuring access levels and setting up schedules is central to maintaining...
Avigilon / Motorola VS Virtual ISC West on Sep 29, 2020
ISC West has historically been so dominant that no player would think of...
Dartmouth College Deploys K3 Temperature Screening on Sep 29, 2020
While Dartmouth College has a $6+ billion endowment, the College has bought...
Hanwha AI Object Detection Tested on Sep 28, 2020
Hanwha has added detection and classification of people, cars, clothing...
Favorite Access Control Manufacturers 2020 on Sep 28, 2020
200+ Integrators told IPVM "What is your favorite access control management...
OnTech Smart Services Partners With Google and Amazon To Compete With Integrators on Sep 25, 2020
A pain point for many homeowners to use consumer security and surveillance is...
The Future of Metalens For Video Surveillance Cameras - MIT / UMass / Immervision on Sep 25, 2020
Panoramic cameras using 'fisheye' lens have become commonplace in video...
Hikvision Sues Over Brazilian Airport Loss on Sep 24, 2020
Hikvision was excluded from a Brazilian airport project because it is owned...
China General Chamber of Commerce Calls Out US Politics on Sep 24, 2020
While US-China relations are at an all-time low, optimism about relations...
Verkada Disruptive Embedded Live Help on Sep 24, 2020
Call up your integrator? Have someone come by the next day? Verkada is...
IP Networking Course Fall 2020 - Last Chance - Register Now on Sep 23, 2020
Today is the last chance to register for the only IP networking course...