Masks Cause Major Facial Recognition Problems

By Rob Kilpatrick, Published Feb 24, 2020, 01:03am EST

Coronavirus is spurring an increase in the use of medical masks, which new IPVM test results show cause major problems for facial recognition systems.

IPVM Image

IPVM tested four facial recognition systems to see how they performed with masks:

The 1-minute video below overviews our findings:

Masks Drastically Reduce Recognition Performance

In our tests, facial recognition confidence dropped dramatically (over 50 points confidence) when wearing anti-viral face masks, with subjects not recognized at all in most cases, and no faces even detected much of the time.

Only Avigilon's Appearance Alerts was able to consistently recognize subjects, but this required confidence to be decreased from high to medium or low. While recognition was possible, it is certain that this will result in false identifications in large populations with hundreds or thousands of subjects.

Further Training Possible

It is possible that these manufacturers could improve masked face performance with further training of their algorithms. However, fundamentally, analytics will have much fewer points to base recognition on as so much of the face is covered, making recognition difficult, even with training.

Drastically Reduced Confidence

Shown below, when compared to the same subject not wearing a mask, confidence dropped by over 60 points when wearing a medical mask, from nearly 100% confidence to just over 30% in the Hikvision dual lens face rec camera.

IPVM Image

In order to recognize a face at these levels, minimum confidence needed to be reduced to ~25%. However, recognition remained inconsistent at this setting, with many subjects detected, but not recognized. This was especially common when hair covered the subject's forehead, making it more difficult to detect the face.

IPVM Image

IronYun performed similarly, with drastically reduced confidence and frequent total misses, even when using low similarity score requirements. For example, the face below matched with only 30-40 similarity, instead of 60-70 when the same subject was not wearing a mask.

IPVM Image

Avigilon Face Recognition Most Accurate But Only Medium/Low Confidence

In our tests, Avigilon's facial recognition had the best recognition rates, recognizing all subjects walking through the scene when using medium or low confidence settings (Avigilon does not show exact confidence scores).

IPVM Image

However, note that while recognition worked in our tests using a small population, users should not assume it would function similarly in larger installations where populations are more diverse. It is certain that false recognitions and missed recognitions will increase as the number of subjects increases because with more people this is more opportunity for people to look or appear to the system as looking similarily.

No Faces Detected In Masks: Verkada

In our tests, Verkada facial recognition completely missed faces of people walking through the scene when wearing masks. These subjects's faces were not detected at all and not searchable from Verkada's face list.

IPVM Image

All Systems Had Miss Detections

Our testing showed many miss facial detections - meaning that the system did not even recognize a face was present, which is necessary to even try to categorize what person's face it is:

  • Avigilon rarely missed faces walking through the scene.
  • Hikvision had the second best recognition rate but at the cost of a lower confidence threshold of 25. The default higher threshold had significant misses.
  • Ironyun missed people frequently and barely detected faces in the scene.
  • Verkada did not recognize anyone in the scene wearing a medical mask.

Improvements In Facial Detection With Masks Certain

Detecting a face with a mask is a solvable problem. The issue is most systems likely did not train their systems to recognize faces with masks. This can be solved by adding new training data of faces with masks on. As such, we would expect future firmware upgrades from many vendors to include that.

Harder Challenge To Recognize Faces With Masks

While detecting performance should go up, recognizing performance faces fundamental limits. Masks hide the chin, nose, mouth, lips, and jaw, essentially constraining systems to match on eyes and forehead.

IPVM Image

Trying to recognize faces can still try to be done and even sometimes or often be correct but fundamentally the accuracy, especially with larger scale, will be far reduced.

The best-case scenario is, structurally, building access control where limited populations, well-controlled lighting, and people presenting their faces increases the odds of accuracy. For example, China's Hanwang Technology is already claiming to do so, as their marketing video below demonstrates:

How well it actually works and how many mistakes are made is not explained. However, access control can also be tuned to allow more matches / access if the assumption is made that it is better to let people in quickly than block valid users from entering.

By contrast, facial recognition in video surveillance, e.g., the oft-cited smart city examples are going to face far harder examples - outdoors, poor weather, wide areas, etc., that combined with face masks are likely to make facial recognition mistakes the norm, even with optimizations.

6 reports cite this report:

Favorite Access Control Credentials 2020 on Sep 15, 2020
Credential choice is more debated than ever, with hacking risk for 125kHz and...
Face Masks Increase Face Recognition Errors Says NIST on Aug 04, 2020
COVID-19 has led to widespread facemask use, which as IPVM testing has shown...
Trueface Presents AI Face Recognition, Mask and Temperature Detection on Jun 10, 2020
Trueface presented its AI facial recognition, mask and temperature detection...
Anyvision Presents AI Facial Recognition and Mask Detection on Jun 08, 2020
AnyVision presented its AI facial recognition and mask detection at the May...
TVT / InVid Facial Recognition Tested on Mar 25, 2020
Facial recognition is frequently sold for thousands of dollars per channel...
Facial Recognition 101 on Mar 18, 2020
Facial recognition interest, use and fear is increasing. This guide aims to...

Comments (10)

Only IPVM Members may comment. Login or Join.

For example, the face below matched with only 30-40 similarity, instead of 60-70 when the same subject was not wearing a mask.

I am not sure I would recognize Ethan anymore with or without a mask :)

Surely the Verkada could recognize dinosaur faces with masks.

Are you seriously saying there was an expectation that facial recognition would work with face masks?

We are seriously saying that companies and pundits were claiming this so we did a real test to verify.

Now, there are people on that side who can still say they think it will work but we now have actual testing to show what happens.

I presume all tests were done with the same resolution?

Yes, they were all done with 1080p cameras (that was the only resolution available for Hikvison and Verkada), though we also used 4K for Avigilon and Ironyun, the only difference between 1080p and 4K with faces masks was the distance detected.

Basically, anything that ere-defines the jawline, (see picture) will subvert the facial recognition. This is particularly noted on a recent Insane Clown Posse tour. (Seriously)

Reports now say facial recognition can work even with masks on, not sure how true it is.

Thanks, #5, There were reports similar released before our cast was published, which we cited in the post above.

The reality is that accuracy is going to decline regardless of what optimization or attempted, as these results show.

So what those manufacturers need to prove is how well it works, at scale, in the wild, etc., not just that it can theoretically work.

Meanwhile, a Chinese company called Hanvon claims to have developed the technology to do 95% accurate face rec with masks. They even claim they can identify everyone in a crowd of up to 30 (masked) people “within a second”. Pretty ridiculous.

Loading Related Reports