VMS Testing Plan 2013/2014

By: John Honovich, Published on Jul 23, 2013

We are embarking on a major new VMS testing program where we will perform in-depth tests both on features and products never done anywhere before. To lead that, Sarit Williams has joined IPVM. She previously led QA for a VMS developer. Now, she will be solely focused on testing and researching VMSes.

This note explains our testing approach and timetable for the next year.

Previously on IPVM

The last major round of VMS testing we did finished in 2010 - 2011, including more than a dozen VMS test reports and a VMS competitive comparison.

Step 1 - Feature Comparisons / Shootouts

This time around, we are starting with feature comparisons / shootouts where we look at specific VMS areas and break them down. For instance, we have already done mutli-streaming and exporting, where we take 6 VMSes, examining the low level details of how they implement these features.

The purpose of this is to deliver fundamental research. No one publicly has ever closely compared and contrasted how VMSes work on a feature by feature basis. For instance, people talk about exporting but what does it really consist of? What makes it better or worse? What options exist? This initial research is to develop a foundation to understand these specifics.

We anticipate about a dozen similar feature comparisons / shootouts. Update November 2013: already done:

VMSes Tested

Ultimately, we plan to test up to 30 VMSes. However, for the feature / comparisons, we are limiting to 6 to keep it manageable. (The more VMSes we test, the longer each report takes, the fewer ones will be released). We believe 6 is enough to show patterns while making the tests manageable to execute.

Get Notified of Video Surveillance Breaking News
Get Notified of Video Surveillance Breaking News

There are 4 we consider a priority simply based on usage and member interest:

  • Genetec and Milestone (the 'Old Guard')
  • Avigilon and Exacq (the "Up and Comers')

This does not mean they are the 'best'. Indeed, many aspects are surprisingly lacking. However, it does cover companies that have broad appeal (whether it is to use or to beat them).

Beyond that, we will rotate 2 others in (so far we have done Network Optix and VideoInsight but we might also use OnSSI Ocularis and a few others over time).

Step 2 - Individual VMS Tests

After we complete the (initial round of) feature tests - likely some time in November, we will switch to individual VMS tests, where the report will focus on just a single manufacturer. We will use the feature tests as a baseline for comparison (e.g., if when we test Luxriot, we can say, "Luxriot's exporting is better/worse than competitors because they have/lack specific elements X, Y and Z previously tested in the exporting comparison, etc.).

There are clearly dozens of VMSes worth testing and we will plan to do so, far more than the 6 we do for the feature tests.

Step 3 - First Half 2014

Before ISC West, we plan to release an all new IPVM VMS Competitive Comparison that breaks down feature by feature how VMSes match up, letting specifiers and users know where the unstated weaknesses are and what VMSes perform better than their marketing / sales hype.

Going Forward

Obviously, that is just the beginning. After Step 3, we will continue to expand, with more advanced testing as well as new VMS testing. This, though, should give an overview of our general approach for the next year. Let us know what questions or comments you have.

Comments (13) : PRO Members only. Login. or Join.

Related Reports

2020 Camera Book Released on Jan 10, 2020
This is the best, most comprehensive security camera training in the world, based on our unprecedented testing. Now, all IPVM PRO Members can get...
No Genetec Major Releases In Over A Year on Feb 06, 2019
Annual VMS licenses are a controversial practice in the video surveillance industry, with many questioning their need or value. However, enterprise...
Exacq Raises VMS Software Pricing Twice in Less Than a Year on Feb 18, 2019
Most VMSes regularly release new features, but rarely increase their prices. For the 3rd time in 4 years, and 2nd time in 8 months, since being...
Prysm PSIM Profile on Mar 05, 2019
A decade ago, PSIM promised significant potential but has always suffered from significant problems. Now, a number of PSIMs have either gone out of...
Milestone Launches Marketplace Where Nothing Is For Sale on Feb 26, 2019
The central announcement at Milestone's annual conference MIPS is Marketplace, which Milestone compared to Uber and Match.com for its ability to...
Genetec Security Center 5.8 Tested on Mar 19, 2019
Genetec has released Version 5.8. This comes after a wait of more than a year that caused frustrations for many Genetec partners. Our previous...
Airship VMS Profile on Apr 03, 2019
Airship has been developing VMS software for over 10 years, however, with no outside investment, and minimal marketing, the company is not well...
Manufacturer Favorability Guide 2019 on Jun 12, 2019
The 259 page PDF guide may be downloaded inside by all IPVM members. It includes our manufacturer favorability rankings and individual...
Bluecherry Open Sources Entire VMS on May 13, 2019
Bluecherry announced they have "released the entire Bluecherry software application open source with a GPL license". We spoke to Bluecherry's...
Ionodes Company Profile on Jul 31, 2019
What happened to Ionodes? With seasoned leadership from Jean-Paul Saindon founder of SmartSight and Dr. Michael Gilge, founder of VCS, back in...

Most Recent Industry Reports

Verkada Paying $100 For Referrals Just To Demo on Jan 22, 2020
Some companies pay for referrals when the referral becomes a customer. Verkada is taking it to the next level - paying $100 referrals fees simply...
Camera Analytics Shootout 2020 - Avigilon, Axis, Bosch, Dahua, Hanwha, Hikvision, Uniview, Vivotek on Jan 22, 2020
Analytics are hot again, thanks to a slew of AI-powered cameras, but whose analytics really work? And how do these new smart cameras compare to top...
Intersec 2020 Final Show Report on Jan 21, 2020
IPVM spent all 3 days at the Intersec 2020 show interviewing various companies and finding key trends. We cover: Middle East Enterprise...
Vehicle & Long Range Access Reader Tutorial on Jan 21, 2020
One of the classic challenges for access control are parking lots and garages, where the user's credential is far from the reader. With modern...
Clearview AI Alarm - NY Times Report Says "Might End Privacy" on Jan 20, 2020
Over the weekend, the NY Times released a report titled "The Secretive Company That Might End Privacy as We Know It" about a company named...
Favorite Camera Manufacturers 2020 on Jan 20, 2020
The past 2 years of US bans and sanctions have shaken the video surveillance industry but what impact would this have on integrators' favorite...
"Severely Impacted" Mercury Security 2020 Leap Year Firmware Issue on Jan 17, 2020
One of the largest access controller manufacturers has a big problem: February 29th. Mercury Security, owned by HID, is alerting partners of the...
Apple Acquires XNOR.ai, Loss For The Industry on Jan 16, 2020
Apple has acquired XNOR.ai for $200 million, reports GeekWire. This is a loss for the video surveillance industry. XNOR.ai stunned the industry...
Installation Course January 2020 - Last Chance on Jan 16, 2020
Thursday, January 16th is your last chance to register for the Winter 2020 Video Surveillance Installation Course. This is a unique installation...
Halo Smart Vape Detector Tested on Jan 16, 2020
The Halo Smart Sensor claims to detect vaping, including popular brand Juul and even THC vapes. But how well does it work in real world...