Evolv's Secret Metallic Sensitivity Setting And The Risk Of Missing Weapons

Evolv has a key setting called "Scan Sensitivity - Metallic" that greatly impacts the probability of Evolv missing weapons and falsely alerting on benign objects.

Evolv aims to hide this from the public, objecting to IPVM's investigation into this as "irresponsible and makes the public less safe." However, this is a key factor in how safe or unsafe the public truly is, contrary to Evolv's marketing.

Inside this note, we examine this setting, how Evolv recommends it be used, and the important tradeoffs involved in adjusting the detector.

Evolv Manuals Publicly Available

Evolv made Evolv Express Operator Manual and Evolv Insights User Manual publicly available on their website.

The 74-page Operator Manual contains a detailed overview of the system, including Setup, Threat Resolution, Alarm Overview, Alert Tagging, etc.

The 34-page Insights Manual contains an overview of different metrics for accessing Evolv Express performance, including Alarm Rate & Timeline Analysis, Detector Setting Analysis, Sensitivity Setting What-If analysis, etc.

6 Sensitivity Settings - A to F

The Operator Manual lists 6 different sensitivity settings (A to F) "from the least sensitive (Level A) to the most sensitive (Level F)."

Manuals Emphasize "Metallic"

Despite Evolv claiming that Evolv Express is not a metal detector, the word "Metallic" in the Sensitivity Setting shows that Evolv Express is looking for metal.

Further, Evolv made numerous claims that Evolv Express knows the "signatures" of weapons, therefore making the system a weapons detector. For example, in a July 2021 Fox Business interview, Evolv's CEO Peter George said:

We have the signatures for all the weapons in the world and we'll always find that metal barrel that's required to actually do damage with the gun. So we'll always find it in those cases. [emphasis added]

In an interview with TD Ameritrade, George made a similar claim:

And we've written the signatures for all the threats that are out there, for all the guns that exists....So all the guns, all the bombs, all the large tactical knives...So we've written the signatures for all those ... We can determine whether we see one of those signatures or not and most people aren't carrying a weapon when they come in. So they can flow in and we you know, we can find a needle in a hastage. [emphasiss addeed]

However, neither Operator Manual nor Insights User Manual has a single mention of weapon "signatures."

Sensitivity Setting Is The Only Adjustable Parameter

The only adjustable setting that can affect the performance of the system is the Sensitivity Setting, according to the Operator Manual.

The lack of control puts security and management personnel in the difficult position of having to choose between user experience and security: high false alarm rates and minimum amount of weapons missed or more weapons missed but low false alarm rates.

This correlates with the Evolv Insights Manual, which has a "Sensitivity Setting What-If Analysis" section, in which Evolv claims that the same objects can be undetected under different sensitivity settings:

Evolv Insights manual demonstrates the tradeoff in alarm rates, with a nearly 40% alarm rate at setting F (most sensitive) versus nearly 0% at A (least sensitive), as the excerpt below demonstrates:

While a near 0% false alarm rate at setting A is great for minimizing nuisance alarms, the risk is missing weapons.

Indeed, our analysis of 5 site videos showed the least sensitive setting used was setting D.

Real Life Sensitivity Settings

Setting "D" is the most common with Jacobi Medical Center, Mercedes-Benz, and Charlotte-Mecklenburg schools using this setting, as shown below.

Further, Urbana School District, which tested Evolv Express, also used setting "D", according to a November 2021 school board meeting.

During this board meeting, Evolv employee Kevin Rabinowitz also confirmed that Six Flags Amusement Park is using Evolv Express at sensitivity setting "F":

Yeah, the Six Flags is unique. I'll tell you why they run it on the most sensitive setting. They're looking for anything. And they just want the real reason to search a bag sometimes.

Setting "E" is also frequently used, as Omaha Performing Center and Saracen Casino center used this setting, as shown below:

IPVM found no instances where settings "A" - "C" were used. It raises the question of whether the low false alarm rates on settings "A" - "C" are worth the amount of missed weapons potentially getting into the facility.

Sensitivity Trade-Offs Inconveniencing Patrons

Putting Evolv Express at a higher sensitivity means that, in order to avoid false alarms, patrons will have to use workarounds to faster resolve alarms. An example of such a workaround is raising umbrellas above their heads, as an example from Mercedes Benz stadium shows:

Another frequent workaround concerns the laptops. Students at schools are asked to carry their laptops outside of their backpacks, according to the Urbana School meeting. The South Carolina school uses the same workaround as shown below:

Evolv has numerous installations in schools (1,2,3,4).

False Alarm Rate

Evolv's own documentation shows a false alarm rate in the range of 5 - 10%, though as Evolv's own example above shows, it can be even higher depending on the setting used and the conditions of the site (e.g., school children all carrying Chromebooks).

Evolv, however, stated that the data from the Evolv Insight Manual is "fictitious":

Evolv does not share actual customer operational data. The graphics in the solution brief are from fictitious data used to illustrate some of the analytics views and data elements representative of the Evolv InsightsTM dashboard. No conclusions should be drawn from these. Alert rates vary per customer. [emphasis added]

IPVM believes that the data from Evolv's documentation comes from either real life implementations or tests. Regardless, these numbers provide a benchmark to estimate the performance of the system.

Alarm fatigue is another issue with Evolv Express that diminishes the attractiveness of the system. An excerpt from the video of Evolv Express in the Charlotte-Mecklenburg at sensitivity setting "D" shows that the systems alert on laptops every time.

Alert Tagging: 4 Most Common False Alarm Objects

False alarms with umbrellas, eyeglass cases, strollers, and laptops are so commonplace for Evolv that Evolv has buttons specifically for each of those categories allowing operators to tag those false alarm types, as shown below from Evolv's client interface:

Evolv Objects to IPVM's Reporting, Hides Manuals

IPVM contacted Evolv for comment on the manuals, and Evolv's CMO Dana Loof responded:

IPVM’s desire to publicly share sensitive security information puts the general public at greater risk. Our company’s focus (and our customer's) is to make the public safe in the places they gather. Evolv Express was designed to do that. As is common with all security screening technologies, there are multiple sensitivity settings and modes of operation that enable each user to optimize the screening process to their specific environment. It is well understood that the security community closely guards information related to the specific performance of technology related to threat detection. Publishing a blueprint of any security screening technology is irresponsible and makes the public less safe. This is why we do not to support IPVM's effort to disclose information on Evolv Express (or other security detection systems), and why will not work with IPVM.

Shortly after this email, the web page where IPVM found both manuals was no longer publicly available and required a password for access.

IPVM, however, was able to archive the page.

Public Risk

The public (including children) who depend on these systems for "weapon detection," which Evolv publicly markets, should know how well (or not) the system works. This Metallic Sensitivity Setting underscores the challenges that Evolv has in actually identifying weapons.