VMS Testing Plan 2013/2014

By: John Honovich, Published on Jul 23, 2013

We are embarking on a major new VMS testing program where we will perform in-depth tests both on features and products never done anywhere before. To lead that, Sarit Williams has joined IPVM. She previously led QA for a VMS developer. Now, she will be solely focused on testing and researching VMSes.

This note explains our testing approach and timetable for the next year.

Previously on IPVM

The last major round of VMS testing we did finished in 2010 - 2011, including more than a dozen VMS test reports and a VMS competitive comparison.

Step 1 - Feature Comparisons / Shootouts

This time around, we are starting with feature comparisons / shootouts where we look at specific VMS areas and break them down. For instance, we have already done mutli-streaming and exporting, where we take 6 VMSes, examining the low level details of how they implement these features.

The purpose of this is to deliver fundamental research. No one publicly has ever closely compared and contrasted how VMSes work on a feature by feature basis. For instance, people talk about exporting but what does it really consist of? What makes it better or worse? What options exist? This initial research is to develop a foundation to understand these specifics.

We anticipate about a dozen similar feature comparisons / shootouts. Update November 2013: already done:

VMSes Tested

Ultimately, we plan to test up to 30 VMSes. However, for the feature / comparisons, we are limiting to 6 to keep it manageable. (The more VMSes we test, the longer each report takes, the fewer ones will be released). We believe 6 is enough to show patterns while making the tests manageable to execute.

Get Notified of Video Surveillance Breaking News
Get Notified of Video Surveillance Breaking News

There are 4 we consider a priority simply based on usage and member interest:

  • Genetec and Milestone (the 'Old Guard')
  • Avigilon and Exacq (the "Up and Comers')

This does not mean they are the 'best'. Indeed, many aspects are surprisingly lacking. However, it does cover companies that have broad appeal (whether it is to use or to beat them).

Beyond that, we will rotate 2 others in (so far we have done Network Optix and VideoInsight but we might also use OnSSI Ocularis and a few others over time).

Step 2 - Individual VMS Tests

After we complete the (initial round of) feature tests - likely some time in November, we will switch to individual VMS tests, where the report will focus on just a single manufacturer. We will use the feature tests as a baseline for comparison (e.g., if when we test Luxriot, we can say, "Luxriot's exporting is better/worse than competitors because they have/lack specific elements X, Y and Z previously tested in the exporting comparison, etc.).

There are clearly dozens of VMSes worth testing and we will plan to do so, far more than the 6 we do for the feature tests.

Step 3 - First Half 2014

Before ISC West, we plan to release an all new IPVM VMS Competitive Comparison that breaks down feature by feature how VMSes match up, letting specifiers and users know where the unstated weaknesses are and what VMSes perform better than their marketing / sales hype.

Going Forward

Obviously, that is just the beginning. After Step 3, we will continue to expand, with more advanced testing as well as new VMS testing. This, though, should give an overview of our general approach for the next year. Let us know what questions or comments you have.

Comments (13) : PRO Members only. Login. or Join.

Related Reports

Vivotek Trend Micro Cyber Security Camera App Tested on Jul 22, 2019
Vivotek and Trend Micro are claiming five million blocked attacks on IP cameras, with their jointly developed app for Vivotek cameras. This new...
Avigilon ACC7 VMS Tested on Jul 22, 2019
Avigilon's Control Center 7 boldly claims it will "transform live video monitoring" with the new Focus of Attention "AI-enabled" interface. We...
History of Video Surveillance on Jul 19, 2019
The video surveillance market has changed significantly since 2000, going from VCRs to ab emerging AI cloud era.  The goal of this history is to...
ZeroEyes Gun Detection Startup on Jul 16, 2019
A gun detection video analytics startup, ZeroEyes, is being led by a group of 6 former Navy SEALs, aiming to "save lives" by using AI to assist...
Axis ARTPEC-7 P1375-E Camera Tested on Jul 12, 2019
Axis claims the new P1375-E box camera with ARTPEC-7 chip delivers "clear, sharp images in any lighting condition." But how well does it do? We...
Calipsa - UK AI Startup Profile on Jul 10, 2019
Analytic startups are a major industry trend. One UK company, Calipsa is aiming to use AI to filter out false positive alarms for live video...
Network Optix / Hanwha Cloud Access Tested on Jul 02, 2019
Remote cloud access is becoming a bigger differentiator, as cybersecurity issues underscore the problems of port forwarding and many integrators...
FLIR Saros Visible / Thermal Analytic Camera Tested on Jun 26, 2019
FLIR's Saros claims "accurate, actionable alerts" with a combination of 1080p visible and dual thermal sensors along with IR and white light...
Axis Live Privacy Shield Analytics Tested on Jun 25, 2019
Privacy is becoming a bigger factor in video surveillance, driven both by increased public awareness and by GDPR. Now, Axis has released Live...
Directory of 60 Video Surveillance Startups on Jun 25, 2019
This directory provides a list of video surveillance startups to help you see and research what companies are new or not yet broadly known. 2019...

Most Recent Industry Reports

How To Quickly Assess An Unknown Company on Jul 23, 2019
There are hundreds of companies in the physical security markets, ranging from a flood of new startups to small companies that may or may not be...
Bosch Starlight 8000i Cameras Tested on Jul 23, 2019
Bosch has released their new Flexidome IP Starlight 8000i cameras, claiming "exceptional detail even in extreme low-light situations." To see...
Responsibility Split Selecting Locks - Statistics on Jul 22, 2019
A heated access debate surrounds who should pick and install the locks. While responsible for selecting the control systems, integrators often...
Vivotek Trend Micro Cyber Security Camera App Tested on Jul 22, 2019
Vivotek and Trend Micro are claiming five million blocked attacks on IP cameras, with their jointly developed app for Vivotek cameras. This new...
Avigilon ACC7 VMS Tested on Jul 22, 2019
Avigilon's Control Center 7 boldly claims it will "transform live video monitoring" with the new Focus of Attention "AI-enabled" interface. We...
History of Video Surveillance on Jul 19, 2019
The video surveillance market has changed significantly since 2000, going from VCRs to emerging into an AI cloud era.  The goal of this history...
Mobile Access Usage Statistics 2019 on Jul 18, 2019
The ability to use mobile phones as access credentials is one of the biggest trends in a market that historically has been slow in adopting new...
New GDPR Guidelines for Video Surveillance Examined on Jul 18, 2019
The highest-level EU data protection authority has issued a new series of provisional video surveillance guidelines. While GDPR has been in...
Wyze AI Analytics Tested - Beats Axis and Hikvision on Jul 17, 2019
$20 camera disruptor Wyze has released free person detection deep learning analytics to all of their users, claiming users will "Only get notified...
Anyvision Aims For 2022 Revenue of $1 Billion on Jul 17, 2019
Only 3 video surveillance manufacturers do a billion dollars or more in annual revenue - Hikvision, Dahua, and Axis. Now, Anyvision plans to join...

The world's leading video surveillance information source, IPVM provides the best reporting, testing and training for 10,000+ members globally. Dedicated to independent and objective information, we uniquely refuse any and all advertisements, sponsorship and consulting from manufacturers.

About | FAQ | Contact