VMS Testing Plan 2013/2014

By: John Honovich, Published on Jul 23, 2013

We are embarking on a major new VMS testing program where we will perform in-depth tests both on features and products never done anywhere before. To lead that, Sarit Williams has joined IPVM. She previously led QA for a VMS developer. Now, she will be solely focused on testing and researching VMSes.

This note explains our testing approach and timetable for the next year.

Previously on IPVM

The last major round of VMS testing we did finished in 2010 - 2011, including more than a dozen VMS test reports and a VMS competitive comparison.

Step 1 - Feature Comparisons / Shootouts

This time around, we are starting with feature comparisons / shootouts where we look at specific VMS areas and break them down. For instance, we have already done mutli-streaming and exporting, where we take 6 VMSes, examining the low level details of how they implement these features.

The purpose of this is to deliver fundamental research. No one publicly has ever closely compared and contrasted how VMSes work on a feature by feature basis. For instance, people talk about exporting but what does it really consist of? What makes it better or worse? What options exist? This initial research is to develop a foundation to understand these specifics.

We anticipate about a dozen similar feature comparisons / shootouts. Update November 2013: already done:

VMSes Tested

Ultimately, we plan to test up to 30 VMSes. However, for the feature / comparisons, we are limiting to 6 to keep it manageable. (The more VMSes we test, the longer each report takes, the fewer ones will be released). We believe 6 is enough to show patterns while making the tests manageable to execute.

Get Notified of Video Surveillance Breaking News
Get Notified of Video Surveillance Breaking News

There are 4 we consider a priority simply based on usage and member interest:

  • Genetec and Milestone (the 'Old Guard')
  • Avigilon and Exacq (the "Up and Comers')

This does not mean they are the 'best'. Indeed, many aspects are surprisingly lacking. However, it does cover companies that have broad appeal (whether it is to use or to beat them).

Beyond that, we will rotate 2 others in (so far we have done Network Optix and VideoInsight but we might also use OnSSI Ocularis and a few others over time).

Step 2 - Individual VMS Tests

After we complete the (initial round of) feature tests - likely some time in November, we will switch to individual VMS tests, where the report will focus on just a single manufacturer. We will use the feature tests as a baseline for comparison (e.g., if when we test Luxriot, we can say, "Luxriot's exporting is better/worse than competitors because they have/lack specific elements X, Y and Z previously tested in the exporting comparison, etc.).

There are clearly dozens of VMSes worth testing and we will plan to do so, far more than the 6 we do for the feature tests.

Step 3 - First Half 2014

Before ISC West, we plan to release an all new IPVM VMS Competitive Comparison that breaks down feature by feature how VMSes match up, letting specifiers and users know where the unstated weaknesses are and what VMSes perform better than their marketing / sales hype.

Going Forward

Obviously, that is just the beginning. After Step 3, we will continue to expand, with more advanced testing as well as new VMS testing. This, though, should give an overview of our general approach for the next year. Let us know what questions or comments you have.

Comments (13) : Members only. Login. or Join.

Related Reports

Milestone Launches Multiple Cloud Solutions on Feb 18, 2020
Milestone is going to the cloud, becoming one of the last prominent VMSes to do so. Milestone is clearly late but how competitive do these new...
Genetec Security Center 5.9 Release Examined on Feb 06, 2020
Genetec released the next major version of Security Center, less than a year after version 5.8. This also follows no Genetec major releases in...
2020 Camera Book Released on Jan 10, 2020
This is the best, most comprehensive security camera training in the world, based on our unprecedented testing. Now, all IPVM PRO Members can get...
Milestone Has Problems on Oct 01, 2019
Milestone has problems. While the company previously excelled in the shift to IP cameras, as IP has matured and competitive differentiation has...
Avigilon ACC7 VMS Tested on Jul 22, 2019
Avigilon's Control Center 7 boldly claims it will "transform live video monitoring" with the new Focus of Attention "AI-enabled" interface. We...
Manufacturer Favorability Guide 2019 on Jun 12, 2019
The 259 page PDF guide may be downloaded inside by all IPVM members. It includes our manufacturer favorability rankings and individual...
Milestone XProtect 2019 R1 Tested on May 15, 2019
For the past few years, Milestone has released quarterly software updates XProtect VMS platform. What is new and how much impact do the updates...
Bluecherry Open Sources Entire VMS on May 13, 2019
Bluecherry announced they have "released the entire Bluecherry software application open source with a GPL license". We spoke to Bluecherry's...
Verkada False Allegations Against Avigilon Exposed on May 08, 2019
Verkada has leveled false allegations against Avigilon, as part of their aggressive marketing tactics against the 'dinosaurs' in the 'ancient'...
Milestone Alleges [Now Removed], Qognify Denies VMS "Running Out Of Support" on May 01, 2019
Milestone is running a marketing campaign where they allege that OnSSI and Seetec VMSes are 'running out of support'. However, Qognify's CEO flatly...

Most Recent Industry Reports

ISC West 2020 Cancelled Again, Rescheduled For October 2020 on Apr 09, 2020
ISC West March 2020 was canceled. Now ISC West July 2020 has been canceled. SIA and Reed hope to run it in October 2020. This is a masterclass...
Sunell "Panda Cam" Thermal Body Temperature System Examined on Apr 09, 2020
Sunell's Panda Cam is the most visually recognizable coronavirus camera, sold by numerous OEMs around the world. But who is Sunell? This company is...
Use Access Control Logs To Constrain Coronavirus on Apr 09, 2020
Access control users have included capabilities that are not commonly used that can help zero-in and discover potential Coronavirus hotspots in a...
Cisco Acquiring Fluidmesh on Apr 09, 2020
Cisco announced it is acquiring niche wireless manufacturer Fluidmesh. Both Cisco and Fluidmesh have been offering video surveillance solutions...
Junk Debt Laden Convergint Facing Coronavirus Crisis on Apr 09, 2020
Convergint has $1+ billion in junk debt putting significant pressure on the company. While the rapid debt buildup enabled the company to go on an...
Embedded Logix Thermal Temperature Detection System Examined on Apr 08, 2020
Embedded Logix has been producing thermal temperature measurement systems for industry and fire detection for over 10 years. Now, they are entering...
Micron 1 TB SD Cards Aim To Eliminate NVRs on Apr 08, 2020
Micron has boldly proclaimed their latest 1TB microSD "eliminates the need for network video recorders", targeting the growing market of...
US DoD Declares "Can No Longer Do Business" With Contractors Using Dahua, Hikvision, Huawei on Apr 08, 2020
The US Department of Defense has confirmed to IPVM that they fully support and intend to proceed with the NDAA 'blacklist clause' covering Dahua,...
IPVM's 12th Anniversary - Thank You! on Apr 07, 2020
IPVM is proud to celebrate it's 12 anniversary expanding our commitment to providing the industry independent and objective information on video...
Mobotix Thermal Body Temperature Detection Examined on Apr 07, 2020
Mobotix has jumped into the Coronavirus temperature detection market, but how do they compare to thermal incumbents like FLIR or ICI who have been...