BBC Exposes Evolv with IPVM Research

A BBC special report based on 1,000+ pages of documents obtained by IPVM has exposed security screening manufacturer Evolv for deceptive marketing and colluding with NCS⁴, a public entity, to hide test results showing failures at weapons screening. The one-minute clip from the BBC below overviews the findings:

These documents prove Evolv has deceived the public, putting at risk countless individuals at Evolv customers such as schools, museums, theme parks, and venues across the US and abroad, including at least 16 NFL, MLB, and MLS stadiums, the Metropolitan Museum of Art, Lincoln Center, Six Flags, Dollywood, Kennywood (where a recent mass shooting occurred despite having Evolv) and more.

Inside this report, IPVM examines the most important of these documents: the unredacted & unpublished NCS4 testing report that was, until now, controlled only by Evolv. IPVM will publish additional reporting on the full trove of documents in the coming days.

Evolv Technologies Overview

Evolv Technologies manufactures Evolv Express, which it markets as a revolution in security screening, claiming these detectors use "AI" to seamlessly distinguish between innocuous items (phones, laptops, keys, etc.) and real threats (guns, bombs, knives, etc.). With investors including Bill Gates, Joe Torre, Payton Manning, and Jeb Bush, Evolv went public in 2021, raising $470 M at a $1.25 B valuation.

This video from earlier this year highlights some of our previous investigations:

NCS⁴ Security Testing Overview

The National Center for Spectator Sports Safety and Security (NCS⁴), part of the University of Southern Mississippi, offers an "operational exercise" program "for the purposes of demonstrating advertised capabilities, industry best practices, and operational capacity to address gaps in sports safety and security." These occur at real public events, with experts invited to act as "independent evaluators."

Results are then published in "a white paper that will be distributed to venue managers and operators for the purposes of education and as an aid in the procurement decision-making process."

For background on NCS⁴ operational exercises see IPVM's previous coverage: Evolv And PatriotOne Compete As NCS⁴ Says "Transparency Is a Priority", Axon Body Camera NCS⁴ Report Obtained And Examined, and Evolv Hides Critical Details in NCS⁴ Results.

NCS⁴ Buried Results At Evolv's Request

While NCS⁴'s public report on Evolv Express shows a seemingly-strong score of 2.84/3.00, the hidden private version (that IPVM obtained) tells a much different story including several failures that conflict with Evolv's public marketing. NCS⁴'s heavily-redacted public report withheld detailed test results, evaluator comments, and other information, only providing overall averages that are favorable to Evolv.

Evolv publicized the report as "validation" from a "fully independent third party," but emails show Evolv forced NCS⁴ to redact the public report, and that Evolv executives heavily influenced the design of testing criteria and directly edited the report in numerous rounds of drafts, even asking for certain results to be removed.

Below is one of more than a dozen draft versions of the report including numerous changes by Richard "Rick" Abraham, VP of Technical Sales & Solutions at Evolv.

92% Detection Rate On Micro-Compact Pistols

The full, unredacted NCS⁴ white paper reveals that Evolv failed to meet testing criteria for detection of micro-compact pistols, with only a 92% detection rate, whereas a conventional metal detector would alert on this pistol virtually 100% of the time. Such firearms are increasingly popular according to industry data, representing 25% of 9mm handgun sales.

The evaluator noted without further explanation that "The system had a 100% detection rate prior to adding additional rigors," indicating the actual detection rate with "additional rigors" was lower than 92%.

All the results for micro-compact pistols were deleted from the public report.

Knives "Are Not Consistently Detected", "Full Transparency" Recommended

Evolv struggled with knives the most, prompting concern from evaluators. Again, these results and comments were deleted from the public report.

Testing revealed that "the [Evolv Express] system was incapable of detecting every knife," despite conducting tests at one of the highest sensitivity settings for the device. With a score of 1.3 out of 3.0 for this category, Evolv detected some knives at a rate of 0% with a 53% rate overall.

Results were so poor that evaluators said they "Recommend full-transparency to potential customers based on data collected." In a separate section of the report dedicated to evaluator feedback, multiple evaluators called knife detection "not consistent" and "unreliable".

While the requirements for knife testing in the final version only mention blades longer than 5", test images show knives with varying blades including some that seem shorter than 5":

In earlier drafts, specifications for knife testing read "knives with blades that ranged from under four inches to greater than six inches." After testing, this was changed to simply read "knives", as shown below:

The wording of this requirement raises questions, as it differs from past NCS⁴ language. While NCS⁴ requirements for object detection are typically quantitative (e.g. the system will detect X at a rate of Y% or greater) the language for Evolv has no quantitative requirement:

Evolv Express is capable of detecting knives made of ferrous metals and exceeding 5” in length (no detection rate provided).

By comparison, NCS⁴'s Patriot One testing had far more extensive testing requirements for knives:

Evolv "Unable to Detect Any Non-Ferrous Metals"

In NCS⁴'s original white paper draft an evaluator observed - before Evolv successfully pushed for the comment's deletion according to emails and tracked changes - that "Evolv is unable to detect any non-ferrous metals," such as aluminum and lead, as well as non-metallic materials like plastics.

(Similarly, IPVM found Evolv does not detect non-ferrous materials in informal tests performed at an Evolv customer venue.)

This means Evolv Express would not detect plastic explosives, 3D-printed guns, and many types of improvised explosive devices (IEDs) such as lead pipe bombs. Additionally, though uncommon, non-ferrous guns and knives do exist and would not be detected.

For these reasons, non-ferrous materials feature in the widely-used National Institute of Justice's (NIJ) walk-through security screening standards, NIJ-0601-02, which include non-ferrous aluminum and zinc gun and knife test objects. Many conventional metal detectors can detect non-ferrous materials.

In fact, NCS⁴'s own recommendations to venue managers include non-ferrous detection as a criterion to consider:

Aside from how security managers at venues consider non-ferrous detection in risk assessments, the issue here is disclosure and false marketing. Evolv claims it can detect all weapons, but its inability to detect non-ferrous materials clearly shows this is false. Venues and the public ought to be aware of this, but Evolv and NCS⁴ elected to cover it up.

Deceptive Design of Testing Criteria and Results Reporting

When compared with the full testing results, the Evolv Express white paper presents results in a grossly-misleading manner. NCS⁴, considering its staff's extensive expertise as University researchers, should have known better.

For example, in the full results the "Ferrous Metal Detection" category includes 8 different assessments covering different categories of weapons:

However, in the public report, all eight areas were grouped into a single category with only the overall average score:

This average score of 2.7 gives readers no indication of Evolv's failures with knives or micro-compact pistols, not to mention qualitative comments from evaluators, which were deleted.

For instance, Evolv Express's ability to detect knives is weighted equally to such categories as "Visual Indicator" and "Function", both of which are essentially satisfied (with perfect scores) as long as red/green LED lights and basic code are functional.

Put differently, knives or micro-compact pistol tests had half as much weight on the overall average score given to the public as these two functionally-identical categories:

Evolv fulfilled numerous other comparatively frivolous categories with perfect scores, artificially increasing its overall average score of 2.84 out of 3. For example, this included:

• "Settings Visibility," the ability to see settings;
• "Post-Event Summary and Analysis," the successful receipt of an email after a weapon is detected;
• both "Statistics" and "System Statistics," essentially identical categories requiring that users can see system statistics.

The final testing requirement, perhaps the most frivolous item, was "Connectivity" or "The tablets will display a red icon to indicate a connectivity issue," an utterly pedestrian feature in modern devices that was weighted the same in Evolv's overall score as its ability to detect micro-compact pistols or knives. Unsurprisingly, Evolv Express earned perfect scores for "Connectivity."

Large Items Not Tested - Laptops, Tumblers, Umbrellas, Etc.

Evolv has well-known, and well-publicized, problems with alerting on large innocuous items.

Despite this, NCS⁴ did not test large innocuous items - laptops, tumblers, umbrellas, etc. - unlike smaller objects.

The published report addresses the lack of testing on such objects, noting they "may produce false alerts." In earlier drafts, NCS⁴ said these objects "will produce false alerts," wording that was changed by the VP of Technical Sales & Solutions at Evolv, as shown below:

In particular, the value of the report is limited by not testing laptops, as schools - a significant portion of Evolv's customers - struggle with false alarms on these objects.

Evolv Response

Evolv's Chief Marketing Officer Dana Loof provided the following statement in response to this report.

Evolv works with and for our customers with the goal of keeping people safe. Respectfully, reporting on security screening vs video surveillance technologies must be different, given the nature of what they do. Long time experts who have been securing venues, removing weapons and keeping people safe from harm should be consulted and their feedback considered when reporting on the technology, people and processes they employ. We have done just that.As you know, Evolv has a fundamental disagreement with IPVM regarding the disclosure of sensitive security information in screening technologies. IPVM has communicated to Evolv, that the general public should know everything there is to know about these technologies, similar to the way IPVM has been reporting on video surveillance technologies. Experienced physical security experts who deal with threats on a daily basis, Evolv, our advisors and our customers disagree. We urge you not to make sensitive security information available to the general public. If there are any prospects, customers or partners that have not already viewed the private report under NDA, they can sign up HERE.

Regarding your accusations, as per standard practice in the operational exercise process, commentary was provided by Evolv, and questions were asked for clarification, no requests for changes were ever made to the NCS⁴ scores. NCS⁴ maintains the source document and reviews the merit of each comment. There were no alterations to scores, and any changes to evaluator feedback were communicated with and approved by individual evaluators. The only difference in the process followed was to provide a public and a private version of the report, with all of the information the evaluators felt necessary to include. Again, we feel providing a blueprint of how to get around the security screening process and technology will make the public and the venues our customers secure less safe.

NCS⁴/USM Response

NCS⁴'s Executive Director Stacey Hall provided the following statement in response to this report:

An Operational Exercise follows exercise principles espoused by the response community and Homeland Security Exercise and Evaluation Program (HSEEP). The NCS⁴ stands by its process, which has proven effective in informing and educating solution providers and practitioners. By design, an exercise provides a low-risk environment to observe capabilities, identify strengths, and document areas for improvement. Due to the lack of controls for validation, an operational exercise white paper serves as a report of observation. The reports are not research output or a validation of capabilities.

Major points:

• Evolv did not make direct edits to the report. The “track changes” feature was used to collect feedback only. The NCS⁴ maintains and approves original documents.
• The NCS⁴ did not edit individual ratings, including those Evolv identified as low.
• Item 18.1 (Sensitivity Setting What-if Analysis) was removed from rated criteria due to the absence of exercise evaluator feedback. The NCS⁴, in consultation with independent evaluators, determined that this functional area was not adequately observed during the exercise.
• Exercise evaluator feedback is provided and approved by evaluators. Exercise evaluators, not the NCS⁴, approve any changes to evaluator comments.
• As outlined on the website and in the white paper, the NCS⁴ does not endorse any products or services.

Offered as a service to solution providers, the Operational Exercise program provides a means of third-party recording and reporting on demonstration capabilities. The NCS⁴ facilitates the process while practitioners and industry experts observe and rate exercise criteria approved by the solution provider. Solution providers are an active participant in the process.

Acknowledgment: This report is published with thanks to Brett Max Kaufman and Jay Stanley at the American Civil Liberties Union, which defended IPVM from legal threats over this reporting.