NCS4's Secretly Paid-For, Manipulated Research Results

While NCS⁴ is a government entity - part of the University of Southern Mississippi (USM) - and bills itself as an academic center, NCS⁴ secretly signed and was paid for 'sponsorship' deals with organizations for whom it performed testing.

It allowed one company, Evolv, to set evaluation criteria, extensively redline the final released report, and cover up testing failures in its weapons detection system used in schools, museums, theme parks, stadiums, and other venues across the US.

Inside this report, IPVM examines 1,000+ pages of documents showing how this reputable, national authority on public safety failed the public. Specifically, these documents show:

• Evolv paid NCS⁴ for the testing, which neither party disclosed and Evolv falsely called "fully independent."
• NCS⁴ let Evolv devise its own testing process and criteria, resulting in a fundamentally-flawed scoring system.
• Evolv directly edited the NCS⁴ report, going through at least 14 drafts - which IPVM obtained - including tracked changes naming Evolv executives that repeatedly deleted important material.
• NCS⁴'s testing found Evolv did not reliably detect knives, with 0% detection for some knives and 53% overall for all knives tested. This was deleted from the public report.
• Evolv did not detect non-ferrous metals, meaning it would not detect plastic explosives, or many types of improvised explosive devices (e.g. lead pipe bombs). This was deleted from the report.
• Evaluators recommended that Evolv be transparent about these poor results, but these evaluator comments were deleted from the report.
• NCS⁴ collaborated with Evolv on marketing and PR for the report.
• NCS⁴ received additional tens of thousands in sponsorship payments from Evolv, including $10,000 luncheons.

Evolv Claimed "Top Scores", but NCS⁴ Hid True Results

In March 2022, when the National Center for Spectator Sports Safety and Security (NCS⁴) released its report on Evolv Express, it quickly became a prominent Evolv talking point. Evolv celebrated the results, with a seemingly-strong overall score of 2.84 / 3.00, as "validation" by a "trusted, fully independent third party" that it did not disclose was paid by the company:

We at Evolv Technology feel confident that these NCS⁴ results provide third party validation of what hundreds of customers and over 200 million visitors already know from their personal experience: that Express offers an unmatched combination of high-performance weapons detection, low false alarm rates, high throughput, unique operational insight, and an awesome visitor experience...Having a trusted, fully independent third party available to stress test our product in real-world environment is an incredibly valuable asset. [emphasis added]

A company press release lauded Evolv's "top scores" from the "rigorous exercise." CEO Peter George even emphasized NCS⁴'s validation in a Q1 2022 earnings call:

We were also recently recognized by the National Center for Spectator Sports Safety and Security or NCS⁴, which tested the effectiveness of the Evolv Express platform.

The results of that testing were very positive, as we earned a score of 2.84 on a scale of zero to three, the highest among the high-throughput touchless weapons detection screening systems. The DHS Safety Act designation and positive NCS⁴ testing results are important acknowledgments for the company, our customers and our partners and provide further validation of our technological and market lead. [emphasis added]

Documents uncovered by IPVM tell a much different story. As recently reported by IPVM and the BBC, the full, unpublished NCS⁴ results showed Evolv failed to regularly detect knives, certain guns, and bomb components, despite Evolv's public claims of recognizing "all weapons."

NCS⁴ & Operational Exercises Background

NCS⁴'s stated vision is "to create and deliver critical resources for enhancing safety and security." Among other things, it publishes white papers based on testing of security products through its "operational exercise" program to assess solutions' "advertised capabilities", "address gaps in sports safety", and help venues across the US make buying decisions:

The exercise program consists of field testing/observation for the purposes of demonstrating advertised capabilities, industry best practices, and operational capacity to address gaps in sports safety and security. As an output, the NCS⁴ will publish a white paper that will be distributed to venue managers and operators for the purposes of education and as an aid in the procurement decision-making process. [emphasis added]

On paper, operational exercises are about serving the needs of security managers or end-users - and, by extension, the public. NCS⁴ says it established the program to "enable venue operators and security personnel to make informed decisions related to the selection and procurement of solutions."

Though part of the state-level University of Southern Mississippi, NCS⁴ has a national impact; it claims to be "the nation’s only academic center devoted to the study and practice of spectator sports safety and security," and receives funding from the Department of Homeland Security (DHS) and Federal Emergency Management Agency (FEMA).

Evolv Paid NCS⁴ as "Sponsor"

According to a draft service agreement IPVM obtained, Evolv paid NCS⁴ for the operational exercise. The agreement names Evolv as "SPONSOR," and describes a "fixed price agreement."

NCS⁴ staff identified this as a "template" service agreement, and emails addressing late payment by Evolv to NCS⁴ discuss when payment is "typically" due, indicating previous reports were also funded by private corporations.

A specific amount is never disclosed in emails and documents. However, IPVM spoke to two sources familiar with the costs, both estimating ~$20,000 USD.

NCS⁴ Pushed for Additional Sponsorship Payments

At the urging of NCS⁴ staff, Evolv made additional payments to NCS⁴. While we did not ascertain the total amount paid to NCS⁴, it included at least two luncheons sponsored by Evolv at a listed price of $10,000 each on top of the operational exercise.

In May 2021, Daniel Ward - an NCS⁴ director who was in charge of the operational exercise - contacted Evolv executives to "give [Evolv] the opportunity to secure you [sic] spot" as a conference sponsor.

NCS⁴ and Evolv discussed sponsorship payments on multiple other occasions - discussions of which Daniel Ward remained a part - while simultaneously going through the operational exercise process, including on March 7, 2022, just 3 weeks prior to the white paper's publication.

NCS⁴ Failed to Disclose Financial Relationship, Evolv Lied

The Evolv Express NCS⁴ white paper does not disclose, or even allude to, NCS⁴'s financial relationship with Evolv.

Only recently, after IPVM questioned NCS⁴ did the organization add any reference to this being a "service offered to solution provider". For example, a July 12, 2022, archive does not mention any agreement just a generic reference to costs:

However, NCS⁴'s current page on operational exercises added a new first bullet point emphasizing that this is a "service offered to solution providers":

Disclosure of relevant financial relationships is a nearly-universal expectation for any published work, in particular, research; NCS⁴ staff would be well aware of this given their academic backgrounds.

Transparent disclosure is also required by the FTC, although this applies mainly to Evolv's false statements about NCS⁴ rather than NCS⁴ itself. For instance, an FTC guide gives the example of a strikingly-similar violating situation in which a company commissions research, and advertises the results with no disclosure of its role as sponsor:

But Evolv not only failed to disclose its true relationship with NCS⁴ across the board, Evolv VP Rick Abraham lied about it on the company's site. On the day of the report's release, Abraham - who managed the operational exercise process for Evolv - published an article "NCS⁴ and Evolv Express" lauding the report for providing "third party validation...from a trusted, fully independent third party..."

Evolv Controlled Testing Criteria

Evolv began influencing the process early on as NCS⁴ developed testing criteria, the "operational matrix," which determined how evaluators scored Evolv Express. NCS⁴ has claimed to design these matrices itself, but emails clearly show Evolv took charge of designing the matrix and decided what was included.

On July 12, 2021, NCS⁴ emailed Evolv an operational matrix template. Three weeks later, Evolv delivered its first draft:

Evolv went through several additional drafts, and worked on the matrix practically up to the last minute, delivering a final version to NCS⁴ one day before the real-world tests took place on October 27, 2022, in Columbus, OH, in an almost-entirely redacted email:

Importantly, the testing criteria did not come close to assessing "advertised capabilities," as NCS⁴ claims of its operational exercises. One would have no way of knowing this since the matrix was largely redacted in the published report, including the sections on weapons testing. Evolv advertises it can recognize signatures for "all weapons", a vastly greater capability than the operational matrix assessed; until recently, Evolv claimed its detectors create "weapons-free zones."

In fact, NCS⁴ specifically allowed Evolv to omit "advertised capabilities." In one example, NCS⁴ asked Evolv, "Do we want to incorporate 'Evolv TempCheck," a feature on Evolv devices that it claims can check for fevers as individuals pass through the machine. Evolv said, "Not at this time."

In the 1000+ pages of emails, IPVM did not find a single instance in which NCS⁴ pushed back on Evolv's design of the testing criteria. To the contrary, when NCS⁴ did provide feedback, it encouraged Evolv to look at this as a marketing exercise. On August 24, 2021, NCS⁴'s Director of Training & Exercise Daniel Ward said:

Currently, the matrix is a bit on the short side. We would like to show some of the additional capabilities in order to bolster it a bit. This is a great time to showcase anything you find yourself frequenting during demos. [emphasis added]

The final product was a highly questionable set of criteria placing far more weight on tangential functionalities - like working LED lights or an icon for connectivity status - than actual weapons detection, as examined further in IPVM's report on the full unreleased white paper. (See: BBC Exposes Evolv with IPVM Research)

Evolv: "Results appear lower than expected."

Despite its role in designing the testing criteria, Evolv was dissatisfied with the October testing results. While Evolv predictably earned perfect scores on several easily-fulfilled requirements, its performance on actual weapons detection came up short in several categories.

When NCS⁴ distributed its white paper draft in December, Evolv executive Richard Abraham quickly expressed dissatisfaction, saying "I'd like to set something up to discuss...the results appear lower than expected."

Ultimately, Evolv and NCS⁴ worked together for months to create two drafts of the report. One was a private report, controlled only by Evolv, containing the full results and evaluator comments. The other is what NCS⁴ actually published, which only provided an overall average of Evolv's weapons detection scores and deleted evaluator comments calling for transparency.

Evolv's Extensive Edits to NCS⁴ White Paper

While NCS⁴ has disclosed that it allows manufacturers to see white papers before publication, this is only "to review for accuracy," and Evolv's role went far beyond this. Evolv engaged in a months-long process of edits and review by its executives including at least 14 different draft versions.

Ultimately, Evolv deleted so much material from NCS⁴'s original draft that its page count was reduced by more than half. This included test results and evaluator comments showing Evolv Express' poor performance with knives, certain guns, bomb components, and other materials - again, this is examined further in IPVM's report on the full unreleased white paper. (See: BBC Exposes Evolv with IPVM Research).

On January 18, 2022, Evolv sent its first round of edits in an excel document. It identified numerous scores as "a bit low," arguing that Evolv "fully met" the requirements:

The next day, Evolv shifted to directly editing NCS⁴'s white paper with tracked changes, submitting a heavily revised version, noting "This will need further review within Evolv as well."

On January 24, Evolv returned with two new drafts, one for publication and one private. In further drafts on February 8, Rick Abraham's changed the statement that the umbrellas and other innocuous items "will produce false alerts" to "may product false alerts."

And another "Public Facing" draft on February 16:

Evolv continued its review, which involved "several folks", sending "v8" on February 21 and noting that Evolv was "lining up" marketing materials:

2 weeks later, Evolv was still "finalizing activities around the documents" and noted even more changes:

NCS⁴ says on its operational exercises page that "Exercise team members, solution providers, and the NCS⁴ staff are reviewers of the draft." However, this disclosure was only added recently. The page previously said:

A draft report is first sent to the operational exercise team for review. Upon revision by the team the report is then reviewed internally by the NCS⁴ before being sent to the solution provider to review for accuracy. [emphasis added]

Evolv's role went far beyond a "review for accuracy," as the above drafts and correspondence show. For that matter, it went far beyond the level of influence even a 'reviewer' of a research publication might have.

Evolv Marketing Chief Asked for Score to be Deleted

Even a day before the white paper's publication, Evolv sought to materially alter NCS⁴'s findings. Chief Marketing Officer Dana Loof asked, "Would it be possible to delete the Audible score[?]"

Evolv performed poorly in this category with a score of 2.3:

To NCS⁴'s credit, they did not delete the Audible Indicator score, though another score identified by Evolv as "low" in January was removed and designated "N/A" with little explanation. Evolv originally received 2.0 from an evaluator for "Sensitivity Setting What-If Analysis," and told NCS⁴ on January 18 "We fully met this requirement...the 2.0 is a bit low."

On January 19, Rick Abraham deleted this from the report:

In the final version, the score was simply marked "N/A" saying "This requirement was observed but unrated."

In a statement, NCS⁴ said this score "was removed from rated criteria due to the absence of exercise evaluator feedback. The NCS⁴, in consultation with independent evaluators, determined that this functional area was not adequately observed during the exercise."

This conflicts with Evolv's statement that it "fully met this requirement" during the tests, as well as NCS⁴'s original draft, which said "This requirement was observed during the exercise and event," and listed ratings from all 3 evaluators. Although the "Evaluator Feedback" section said "No additional feedback," numerous other categories - which were not removed - also had "No additional feedback."

NCS⁴ to Evolv: "Why is the NCS⁴ allowing Evolv to withhold information from the public?"

As the publication date approached, with CMO Dana Loof now involved, NCS⁴ and Evolv began coordinating on marketing the white paper, including NCS⁴'s public explanation of why it is "allowing Evolv to withhold information from the public":

NCS⁴ provided an almost-identical statement to IPVM when we asked them that question the next day:

NCS⁴ also cleared items like social media posts with Evolv:

NCS⁴ Report is Marketing Pretending to be Research

NCS⁴ claims to be a research organization, but its Evolv Express operational exercise violated fundamental principles of research while favoring the interests of a private corporation. More accurately, the report is marketing disguised as research.

Avoiding conflicts of interest, or if impossible transparently disclosing them, is a cornerstone of legitimate research. NCS⁴ not only failed to disclose conflicts, it embraced them: NCS⁴ let its corporate sponsor rig its own testing requirements, write its own research report, hide negative findings, and use the NCS⁴ brand and "validation" to backstop its long history of false marketing claims.

This raises several serious questions for NCS⁴, including about the validity of NCS⁴'s past operational exercises, its other research work, the ethics of its team, and whether NCS⁴ is fit to receive Federal funding. It also raises serious public safety questions: if venues are relying on NCS⁴ research as objective - whether about Evolv or anything else - have they made poorly-informed decisions that are currently putting people at risk?

NCS⁴ Failed the Public

While Evolv's actions are condemnable, this is far from the first time a company has relied on unethical behavior to gain competitive advantage. However, we should expect more from NCS⁴ and the University of Southern Mississippi. These public institutions do not exist so that their researchers can serve the marketing interests of private corporations instead of the public or worse, as in this case, at the expense of the public.

For NCS⁴, this situation serves as a litmus test, which it has categorically failed. There is little question that the public is better-served by a report which tells the truth about Evolv's weapons detection capabilities, than by a report which gives a false impression of strong performance. NCS⁴ faced a choice between the financial interests of a private corporation and the safety interests of the public, and it chose the latter.

Recently, a student in Utica, NY was stabbed after a 7-inch knife passed through an Evolv scanner undetected (an unsurprising development given NCS⁴'s hidden testing results on knives). In September, 3 people were shot - including 2 teenagers - after guns entered a Pittsburgh theme park protected by Evolv scanners, although local police have not confirmed how the guns entered the park.

When it comes to weapons detection systems like Evolv, marketing disguised as research misleads decision makers and the public, and people can get hurt or killed. NCS⁴ should consider retracting its research on Evolv, as well as any other research that was inappropriately influenced by corporate interests. The University of Southern Mississippi should also consider whether NCS⁴'s corrupt practices have a home at the institution.

Evolv Response

Evolv's Chief Marketing Officer Dana Loof provided the following statement in response to our reporting. Notably, although she said in this statement that "no requests for changes were ever made to the NCS⁴ scores," Ms. Loof herself asked to delete a score as examined above.

Evolv works with and for our customers with the goal of keeping people safe. Respectfully, reporting on security screening vs video surveillance technologies must be different, given the nature of what they do. Long time experts who have been securing venues, removing weapons and keeping people safe from harm should be consulted and their feedback considered when reporting on the technology, people and processes they employ. We have done just that.

As you know, Evolv has a fundamental disagreement with IPVM regarding the disclosure of sensitive security information in screening technologies. IPVM has communicated to Evolv, that the general public should know everything there is to know about these technologies, similar to the way IPVM has been reporting on video surveillance technologies. Experienced physical security experts who deal with threats on a daily basis, Evolv, our advisors and our customers disagree. We urge you not to make sensitive security information available to the general public. If there are any prospects, customers or partners that have not already viewed the private report under NDA, they can sign up HERE.

Regarding your accusations, as per standard practice in the operational exercise process, commentary was provided by Evolv, and questions were asked for clarification, no requests for changes were ever made to the NCS⁴ scores. NCS⁴ maintains the source document and reviews the merit of each comment. There were no alterations to scores, and any changes to evaluator feedback were communicated with and approved by individual evaluators. The only difference in the process followed was to provide a public and a private version of the report, with all of the information the evaluators felt necessary to include. Again, we feel providing a blueprint of how to get around the security screening process and technology will make the public and the venues our customers secure less safe.

NCS⁴/USM Response

NCS⁴'s Executive Director Stacey Hall provided the following statement in response to our reporting:

An Operational Exercise follows exercise principles espoused by the response community and Homeland Security Exercise and Evaluation Program (HSEEP). The NCS⁴ stands by its process, which has proven effective in informing and educating solution providers and practitioners. By design, an exercise provides a low-risk environment to observe capabilities, identify strengths, and document areas for improvement. Due to the lack of controls for validation, an operational exercise white paper serves as a report of observation. The reports are not research output or a validation of capabilities.

Major points:

• Evolv did not make direct edits to the report. The “track changes” feature was used to collect feedback only. The NCS⁴ maintains and approves original documents.
• The NCS⁴ did not edit individual ratings, including those Evolv identified as low.
• Item 18.1 (Sensitivity Setting What-if Analysis) was removed from rated criteria due to the absence of exercise evaluator feedback. The NCS⁴, in consultation with independent evaluators, determined that this functional area was not adequately observed during the exercise.
• Exercise evaluator feedback is provided and approved by evaluators. Exercise evaluators, not the NCS⁴, approve any changes to evaluator comments.
• As outlined on the website and in the white paper, the NCS⁴ does not endorse any products or services.

Offered as a service to solution providers, the Operational Exercise program provides a means of third-party recording and reporting on demonstration capabilities. The NCS⁴ facilitates the process while practitioners and industry experts observe and rate exercise criteria approved by the solution provider. Solution providers are an active participant in the process.