VMS Testing Plan 2013/2014

By John Honovich, Published on Jul 23, 2013

We are embarking on a major new VMS testing program where we will perform in-depth tests both on features and products never done anywhere before. To lead that, Sarit Williams has joined IPVM. She previously led QA for a VMS developer. Now, she will be solely focused on testing and researching VMSes.

This note explains our testing approach and timetable for the next year.

Previously on IPVM

The last major round of VMS testing we did finished in 2010 - 2011, including more than a dozen VMS test reports and a VMS competitive comparison.

Step 1 - Feature Comparisons / Shootouts

This time around, we are starting with feature comparisons / shootouts where we look at specific VMS areas and break them down. For instance, we have already done mutli-streaming and exporting, where we take 6 VMSes, examining the low level details of how they implement these features.

The purpose of this is to deliver fundamental research. No one publicly has ever closely compared and contrasted how VMSes work on a feature by feature basis. For instance, people talk about exporting but what does it really consist of? What makes it better or worse? What options exist? This initial research is to develop a foundation to understand these specifics.

We anticipate about a dozen similar feature comparisons / shootouts. Update November 2013: already done:

VMSes Tested

Ultimately, we plan to test up to 30 VMSes. However, for the feature / comparisons, we are limiting to 6 to keep it manageable. (The more VMSes we test, the longer each report takes, the fewer ones will be released). We believe 6 is enough to show patterns while making the tests manageable to execute.

Get Notified of Video Surveillance Breaking News
Get Notified of Video Surveillance Breaking News

There are 4 we consider a priority simply based on usage and member interest:

  • Genetec and Milestone (the 'Old Guard')
  • Avigilon and Exacq (the "Up and Comers')

This does not mean they are the 'best'. Indeed, many aspects are surprisingly lacking. However, it does cover companies that have broad appeal (whether it is to use or to beat them).

Beyond that, we will rotate 2 others in (so far we have done Network Optix and VideoInsight but we might also use OnSSI Ocularis and a few others over time).

Step 2 - Individual VMS Tests

After we complete the (initial round of) feature tests - likely some time in November, we will switch to individual VMS tests, where the report will focus on just a single manufacturer. We will use the feature tests as a baseline for comparison (e.g., if when we test Luxriot, we can say, "Luxriot's exporting is better/worse than competitors because they have/lack specific elements X, Y and Z previously tested in the exporting comparison, etc.).

There are clearly dozens of VMSes worth testing and we will plan to do so, far more than the 6 we do for the feature tests.

Step 3 - First Half 2014

Before ISC West, we plan to release an all new IPVM VMS Competitive Comparison that breaks down feature by feature how VMSes match up, letting specifiers and users know where the unstated weaknesses are and what VMSes perform better than their marketing / sales hype.

Going Forward

Obviously, that is just the beginning. After Step 3, we will continue to expand, with more advanced testing as well as new VMS testing. This, though, should give an overview of our general approach for the next year. Let us know what questions or comments you have.

Comments (13) : Members only. Login. or Join.

Related Reports

Digifort Presents VMS 7.3 with AI Analytics on May 07, 2020
Digifort presented its VMS 7.3 with AI analytics at the April 2020 IPVM New...
Hanwha Presents AI Analytics and First 8K Camera on May 11, 2020
Hanwha presented their AI Analytics and the "Industry's First 8K" camera at...
VMS 101 on Mar 03, 2020
This guide teaches the fundamentals about video management...
IPVM Opens 12,000 Sqft Testing Facility on Dec 16, 2019
IPVM is proud to announce the opening of the world's first video surveillance...
2020 Camera Book Released on Jan 10, 2020
This is the best, most comprehensive security camera training in the world,...
Camera Course Summer 2020 - Last Chance on Jul 18, 2020
This is your last chance to register for the Summer 2020 Camera Course. This...
Access Control Course Fall 2020 - Register Now - Last Chance on Sep 09, 2020
Starts next week! IPVM offers the most comprehensive access control course in...
ISS Presents Face As A Credential and UVSS on Apr 30, 2020
ISS presented its security platform, including access control integration,...
Seek Scan Thermal Temperature Screening System Tested on May 28, 2020
Now that IPVM has tested Dahua, Hikvision, and Sunell, we are returning to...
Video Surveillance 101 Book Released on Jul 07, 2020
IPVM's unique introduction to video surveillance series is now available as a...
Genetec Security Center 5.9 Release Examined on Feb 06, 2020
Genetec released the next major version of Security Center, less than a year...
Euklis Presents AI Analytics on May 05, 2020
Euklis presented its AI facial recognition, LPR, and object recognition...
Vivotek Presents AI Analytics and LPR on May 19, 2020
Vivotek presented its AI Analytics and LPR at the April 2020 IPVM New...
Avigilon Presents Cloud Connected ACC on May 13, 2020
Avigilon presented Avigilon Cloud Services, which supports cloud-connected...
2020 IP Networking Book Released on Jan 06, 2020
The new IP Networking Book 2020 is a 280 page in-depth guide that teaches you...

Recent Reports

Panasonic Presents i-PRO Cameras and Video Analytics on Oct 19, 2020
Panasonic presented its i-PRO X-Series cameras and AI video analytics at the...
Augmented Reality (AR) Cameras From Hikvision and Dahua Examined on Oct 19, 2020
Hikvision, Dahua, and other China companies are marketing augmented reality...
18 TB Video Surveillance Drives (WD and Seagate) on Oct 19, 2020
Both Seagate and Western Digital recently announced 18TB hard drives...
Watrix Gait Recognition Profile on Oct 16, 2020
Watrix is the world's only gait recognition surveillance provider IPVM has...
Intel Presents Edge-to-Cloud Ecosystem for Video Analytics on Oct 16, 2020
Intel presented its processors and software toolkit for computer vision at...
Best Manufacturer Technical Support 2020 on Oct 16, 2020
5 manufacturers stood out as providing the best technical support to ~200...
Microsoft Azure Presents Live Video Analytics on Oct 15, 2020
Microsoft Azure presented its Live Video Analytics offering at the September...
Worst Manufacturer Technical Support 2020 on Oct 15, 2020
4 manufacturers stood out as providing the worst technical support to ~200...
Clorox Announces, Then Pulls, Fever Camera on Oct 15, 2020
For almost one week, Clorox was marketing fever cameras. The booming...
Faulty Hikvision Fever Cam Setup at Mexico City Basilica and Cathedral on Oct 14, 2020
Donated Hikvision fever cameras (claiming screening of 1,800 people/min. with...
Directory of 209 "Fever" Camera Suppliers on Oct 14, 2020
This directory provides a list of "Fever" scanning thermal camera providers...
Avigilon UMD / UAD Tested on Oct 14, 2020
Avigilon's Unusual Activity Detection and Unusual Motion Detection claim to...
Longse Promoting Hikvision Partner Fullhan Chip Based Cameras on Oct 14, 2020
With Huawei HiSilicon production being shut down at TSMC, camera...
Meridian & Goodview (BEMS Relabeller) Temperature Screening Tested on Oct 13, 2020
A lot of temperature tablets look exactly alike and that is because they use...
Monitoring Alarm Systems From Home - Innovation or Danger? on Oct 13, 2020
Remote monitoring by alarm companies since COVID-19 is bringing cost savings...