VMS Testing Plan 2013/2014

By: John Honovich, Published on Jul 23, 2013

We are embarking on a major new VMS testing program where we will perform in-depth tests both on features and products never done anywhere before. To lead that, Sarit Williams has joined IPVM. She previously led QA for a VMS developer. Now, she will be solely focused on testing and researching VMSes.

This note explains our testing approach and timetable for the next year.

Previously on IPVM

The last major round of VMS testing we did finished in 2010 - 2011, including more than a dozen VMS test reports and a VMS competitive comparison.

Step 1 - Feature Comparisons / Shootouts

This time around, we are starting with feature comparisons / shootouts where we look at specific VMS areas and break them down. For instance, we have already done mutli-streaming and exporting, where we take 6 VMSes, examining the low level details of how they implement these features.

The purpose of this is to deliver fundamental research. No one publicly has ever closely compared and contrasted how VMSes work on a feature by feature basis. For instance, people talk about exporting but what does it really consist of? What makes it better or worse? What options exist? This initial research is to develop a foundation to understand these specifics.

We anticipate about a dozen similar feature comparisons / shootouts. Update November 2013: already done:

VMSes Tested

Ultimately, we plan to test up to 30 VMSes. However, for the feature / comparisons, we are limiting to 6 to keep it manageable. (The more VMSes we test, the longer each report takes, the fewer ones will be released). We believe 6 is enough to show patterns while making the tests manageable to execute.

Get Notified of Video Surveillance Breaking News
Get Notified of Video Surveillance Breaking News

There are 4 we consider a priority simply based on usage and member interest:

  • Genetec and Milestone (the 'Old Guard')
  • Avigilon and Exacq (the "Up and Comers')

This does not mean they are the 'best'. Indeed, many aspects are surprisingly lacking. However, it does cover companies that have broad appeal (whether it is to use or to beat them).

Beyond that, we will rotate 2 others in (so far we have done Network Optix and VideoInsight but we might also use OnSSI Ocularis and a few others over time).

Step 2 - Individual VMS Tests

After we complete the (initial round of) feature tests - likely some time in November, we will switch to individual VMS tests, where the report will focus on just a single manufacturer. We will use the feature tests as a baseline for comparison (e.g., if when we test Luxriot, we can say, "Luxriot's exporting is better/worse than competitors because they have/lack specific elements X, Y and Z previously tested in the exporting comparison, etc.).

There are clearly dozens of VMSes worth testing and we will plan to do so, far more than the 6 we do for the feature tests.

Step 3 - First Half 2014

Before ISC West, we plan to release an all new IPVM VMS Competitive Comparison that breaks down feature by feature how VMSes match up, letting specifiers and users know where the unstated weaknesses are and what VMSes perform better than their marketing / sales hype.

Going Forward

Obviously, that is just the beginning. After Step 3, we will continue to expand, with more advanced testing as well as new VMS testing. This, though, should give an overview of our general approach for the next year. Let us know what questions or comments you have.

Comments (13) : PRO Members only. Login. or Join.

Related Reports

Axis Perimeter Defender Improves, Yet Worse Than Dahua and Wyze on Sep 19, 2019
While Axis Perimeter Defender analytics improved from our 2018 testing, the market has improved much faster, with much less expensive offerings...
Directory of 68 Video Surveillance Startups on Sep 18, 2019
This directory provides a list of video surveillance startups to help you see and research what companies are new or not yet broadly known. 2019...
Vivotek "Neural Network-Powered Detection Engine" Analytics Tested on Sep 17, 2019
Vivotek has released "a neural network-powered detection engine", named Smart Motion Detection, claiming that "swaying vegetation, vehicles passing...
How Cobalt Robotics May Disrupt Security on Sep 13, 2019
While security robots have largely become a joke over the last few years, one organization, Cobalt Robotics, has raised $50+ million from top US...
Panasonic 4K Camera Tested (WV-S2570L) on Sep 13, 2019
Panasonic has released their latest generation 4K dome, the WV-S2570L, claiming "Extreme image quality allows evidence to be captured even under...
Genetec Stratocast VSaaS Tested on Sep 05, 2019
The VSaaS market is rapidly expanding in 2019, with Verkada, Meraki, Eagle Eye, Avigilon and numerous startups growing their market share. When we...
Scylla AI Video Analytics Company Profile on Aug 29, 2019
Scylla, an AI analytics startup, says they are targeting 1 Billion dollar valuation in 5 years and it "is not rocket science" to detect weapons and...
Uniview Pro 4K Camera Tested (IPC3238ER3-DVZ) on Aug 29, 2019
In our 8MP / 4K Fixed Lens Camera Shootout, Uniview's low end 8MP was firmly in the middle of the pack. Does their higher end Pro Series 4K camera...
Warning: Windows 7 Update Crashing NVRs on Aug 26, 2019
Windows 7 updates are causing VMS servers to fail to boot. After running the update, impacted systems do not boot as normal, instead display this...
Security Integrators Outlook On Remaining Integrators In 2025 on Aug 22, 2019
The industry has changed substantially in the last decade, with the rise of IP cameras and the race to the bottom. Indeed, more changes may be...

Most Recent Industry Reports

ONVIF Suspends Huawei on Sep 20, 2019
Huawei has been 'suspended', and effectively expelled, from ONVIF so long as US sanctions remain on the mega Chinese manufacturer. Inside this...
Open Access Controller Guide (Axis, HID, Isonas, Mercury) on Sep 19, 2019
In the access control market, there are many software platforms, but only a few companies that make non-proprietary door controllers. Recently,...
Axis Perimeter Defender Improves, Yet Worse Than Dahua and Wyze on Sep 19, 2019
While Axis Perimeter Defender analytics improved from our 2018 testing, the market has improved much faster, with much less expensive offerings...
Directory of 68 Video Surveillance Startups on Sep 18, 2019
This directory provides a list of video surveillance startups to help you see and research what companies are new or not yet broadly known. 2019...
Uniview Prime Series 4K Camera Tested on Sep 18, 2019
Is the new Uniview 'Prime' better than the more expensive existing Uniview 'Pro'? In August, IPVM tested Uniview 4K 'Pro' but members advocated...
US Army Base To Buy Banned Honeywell Surveillance on Sep 17, 2019
The U.S. Army's Fort Gordon, home to their Cyber Center of Excellence, has issued a solicitation to purchase Honeywell products that are US...
Vivotek "Neural Network-Powered Detection Engine" Analytics Tested on Sep 17, 2019
Vivotek has released "a neural network-powered detection engine", named Smart Motion Detection, claiming that "swaying vegetation, vehicles passing...
Schmode is Back, Aims To Turn Boulder AI Into Giant on Sep 16, 2019
One of the most influential and controversial executives in the past decade is back. Bryan Schmode ascended and drove the hypergrowth of Avigilon...
Manufacturers Unhappy With Weak ASIS GSX 2019 And 2020 Shift on Sep 16, 2019
Manufacturers were generally unhappy with ASIS GSX, both for weak 2019 booth traffic and a scheduling shift for the 2020 show, according to a new...