Masks Cause Major Facial Recognition Problems

Coronavirus is spurring an increase in the use of medical masks, which new IPVM test results show cause major problems for facial recognition systems.

IPVM tested four facial recognition systems to see how they performed with masks:

• Avigilon Appearance Alerts
• Hikvision Dual Lens Face Rec Camera
• IronYun AI Analytics
• Verkada Face Rec

The 1-minute video below overviews our findings:

Masks Drastically Reduce Recognition Performance

In our tests, facial recognition confidence dropped dramatically (over 50 points confidence) when wearing anti-viral face masks, with subjects not recognized at all in most cases, and no faces even detected much of the time.

Only Avigilon's Appearance Alerts was able to consistently recognize subjects, but this required confidence to be decreased from high to medium or low. While recognition was possible, it is certain that this will result in false identifications in large populations with hundreds or thousands of subjects.

Further Training Possible

It is possible that these manufacturers could improve masked face performance with further training of their algorithms. However, fundamentally, analytics will have much fewer points to base recognition on as so much of the face is covered, making recognition difficult, even with training.

Drastically Reduced Confidence

Shown below, when compared to the same subject not wearing a mask, confidence dropped by over 60 points when wearing a medical mask, from nearly 100% confidence to just over 30% in the Hikvision dual lens face rec camera.

In order to recognize a face at these levels, minimum confidence needed to be reduced to ~25%. However, recognition remained inconsistent at this setting, with many subjects detected, but not recognized. This was especially common when hair covered the subject's forehead, making it more difficult to detect the face.

IronYun performed similarly, with drastically reduced confidence and frequent total misses, even when using low similarity score requirements. For example, the face below matched with only 30-40 similarity, instead of 60-70 when the same subject was not wearing a mask.

Avigilon Face Recognition Most Accurate But Only Medium/Low Confidence

In our tests, Avigilon's facial recognition had the best recognition rates, recognizing all subjects walking through the scene when using medium or low confidence settings (Avigilon does not show exact confidence scores).

However, note that while recognition worked in our tests using a small population, users should not assume it would function similarly in larger installations where populations are more diverse. It is certain that false recognitions and missed recognitions will increase as the number of subjects increases because with more people this is more opportunity for people to look or appear to the system as looking similarily.

No Faces Detected In Masks: Verkada

In our tests, Verkada facial recognition completely missed faces of people walking through the scene when wearing masks. These subjects's faces were not detected at all and not searchable from Verkada's face list.

All Systems Had Miss Detections

Our testing showed many miss facial detections - meaning that the system did not even recognize a face was present, which is necessary to even try to categorize what person's face it is:

• Avigilon rarely missed faces walking through the scene.
• Hikvision had the second best recognition rate but at the cost of a lower confidence threshold of 25. The default higher threshold had significant misses.
• Ironyun missed people frequently and barely detected faces in the scene.
• Verkada did not recognize anyone in the scene wearing a medical mask.

Improvements In Facial Detection With Masks Certain

Detecting a face with a mask is a solvable problem. The issue is most systems likely did not train their systems to recognize faces with masks. This can be solved by adding new training data of faces with masks on. As such, we would expect future firmware upgrades from many vendors to include that.

Harder Challenge To Recognize Faces With Masks

While detecting performance should go up, recognizing performance faces fundamental limits. Masks hide the chin, nose, mouth, lips, and jaw, essentially constraining systems to match on eyes and forehead.

Trying to recognize faces can still try to be done and even sometimes or often be correct but fundamentally the accuracy, especially with larger scale, will be far reduced.

The best-case scenario is, structurally, building access control where limited populations, well-controlled lighting, and people presenting their faces increases the odds of accuracy. For example, China's Hanwang Technology is already claiming to do so, as their marketing video below demonstrates:

How well it actually works and how many mistakes are made is not explained. However, access control can also be tuned to allow more matches / access if the assumption is made that it is better to let people in quickly than block valid users from entering.

By contrast, facial recognition in video surveillance, e.g., the oft-cited smart city examples are going to face far harder examples - outdoors, poor weather, wide areas, etc., that combined with face masks are likely to make facial recognition mistakes the norm, even with optimizations.