VMS Testing Plan 2013/2014

Author: John Honovich, Published on Jul 23, 2013

We are embarking on a major new VMS testing program where we will perform in-depth tests both on features and products never done anywhere before. To lead that, Sarit Williams has joined IPVM. She previously led QA for a VMS developer. Now, she will be solely focused on testing and researching VMSes.

This note explains our testing approach and timetable for the next year.

Previously ** ****

*** **** ***** ***** ** *** ******* ** ************* **** - ****, ********* **** **** * ***** *** test ******* *** * *** *********** **********.

Step * - ******* *********** / *********

**** **** ******, ** *** ******** **** ******* *********** / shootouts ***** ** **** ** ******** *** ***** *** ***** them ****. *** ********, ** **** ******* *********-*********************, ***** ** **** * *****, ********* *** *** ***** details ** *** **** ********* ***** ********.

*** ******* ** **** ** ** ******* *********** ********. ** one ******** *** **** ******* ******** *** ********** *** ***** work ** * ******* ** ******* *****. *** ********, ****** talk ***** ********* *** **** **** ** ****** ******* **? What ***** ** ****** ** *****? **** ******* *****? **** initial ******** ** ** ******* * ********** ** ********** ***** specifics.

** ********** ***** * ***** ******* ******* *********** / *********. Update ******** ****: ******* ****:

VMSes ******

**********, ** **** ** **** ** ** ** *****. *******, for *** ******* / ***********, ** *** ******** ** * to **** ** **********. (*** **** ***** ** ****, *** longer **** ****** *****, *** ***** **** **** ** ********). We ******* * ** ****** ** **** ******** ***** ****** the ***** ********** ** *******.

***** *** * ** ******** * ******** ****** ***** ** usage *** ****** ********:

  • ******* *** ********* (*** '*** *****')
  • ******** *** ***** (*** "** *** ******')

**** **** *** **** **** *** *** '****'. ******, **** aspects *** ************ *******. *******, ** **** ***** ********* **** have ***** ****** (******* ** ** ** *** ** ** beat ****).

****** ****, ** **** ****** * ****** ** (** *** we **** **** ******* ***** *** ************ *** ** ***** also *** ***** ******** *** * *** ****** **** ****).

Step * - ********** *** *****

***** ** ******** *** (******* ***** **) ******* ***** - likely **** **** ** ********, ** **** ****** ** ********** VMS *****, ***** *** ****** **** ***** ** **** * single ************. ** **** *** *** ******* ***** ** * baseline *** ********** (*.*., ** **** ** **** *******, ** can ***, "*******'* ********* ** ******/***** **** *********** ******* **** have/lack ******** ******** *, * *** * ********** ****** ** the ********* **********, ***.).

***** *** ******* ****** ** ***** ***** ******* *** ** will **** ** ** **, *** **** ******* * ** ** *** *** ******* *****.

Step * - ***** **** ****

****** *** ****, ** **** ** ******* ** *** *** IPVM *** *********** ********** **** ****** **** ******* ** ******* how ***** ***** **, ******* ********** *** ***** **** ***** the ******** ********** *** *** **** ***** ******* ****** **** their ********* / ***** ****.

Going *******

*********, **** ** **** *** *********. ***** **** *, ** will ******** ** ******, **** **** ******** ******* ** **** as *** *** *******. ****, ******, ****** **** ** ******** of *** ******* ******** *** *** **** ****. *** ** know **** ********* ** ******** *** ****.

Comments (13)

Will relative costs to the end user, including SMA's and license renewals, be included? Total affordability, over a certain number of years, should be a major benchmark item, not just product feature sets. If you were doing automobiles, Rolls Royce or Ferrari might have the highest score, but be totally inappropriate for the majority of buyers due to cost factors.

Hi Ted,

Yes, we absolutely will cover total costs and compare to overall features/value delivered. The focus to start is understanding and documenting how the VMSes work, since this takes the most time. Once we do that, we can start comparing to the costs to better understand the relative value.

Welcome Sarit, I am looking forward to the reports! I am curious as to the level of “training” that these reviews will contain in the future? I don’t mean full how to type set up guides for each VMS or anything that in depth. Just some basics and maybe some tips/tricks discovered through using the VMS.

Thank you Ross! I will certainly continue to include training tips/tricks discovered in each VMS (like in the multistream and exporting reports). Additionally, we will be preparing new VMS 101 training to help train members on fundamentals and main differentiators; I hope to see you there.

What happens as feature sets change and evolve? Most certainly all vendors will have changes and upgrades in time for ISC. And where does Avigilon stand with IPVM continuing research on their products? Does IPVM foresee issues with obtaining new cameras or software updates from them considering the recent and ongoing problems with them?

I'll leave the first couple questions to Sarit, but as far as Avigilon: we have sources for Avigilon product and support has actually been answering questions for us. We also have sent questions and copies of posts their way, as we do with all manufacturers when doing tests or updates, but they have chosen not to respond.

So in short, we don't see it as a problem, and we welcome their feedback on technical issues, as always, if they choose to give it.

Would be interesting to also redo the integrator surveys again that you performed a couple years back.

Hi Thomas, We are going to redo them. However, it has not even been 2 years yet (e.g., favorite VMSes) so we are waiting to give some time for things to actually change. This fall would looks like an appropriate time.

Great Job ,

Can you include support issues relevant to the vms?

Can we send in suggestions on some weve used in past?

look forward to this report .

Christopher,

Thanks so much. Please feel free to email any info to sarit@ipvm.com; any information you have would be greatly appreciated. I can tell you that as I'm setting up these environments and clients and going through this process as an end user would I have lots to write about (errors during install, crashes, activations, etc) and those subjects are certainly on the list.

This is great and I look forward to the analysis. Would it be possible to see table listing all features in a side-by-side comparison? Each feature set would also need to be explained as well. I understand that such a schedule may be difficult, but it would be pretty easy to follow.

Sarit, John,

This is exactly what we think is required for BMS benchmarking. As some time back I raised this in this forum seeking the stress testing approaches as well as quantitatively measuring the features/performance.

I am looking forward the step1 and also will be happy to contribute in the process.

Thanks, Krish

Thank you Krish and Jim for the suggestions- duly noted! We will continue to jot down suggestions and recommendations as we expand our baseline tests.

Login to read this IPVM report.
Why do I need to log in?
IPVM conducts unique testing and research funded by member's payments enabling us to offer the most independent, accurate and in-depth information.

Related Reports

Directory of Video Surveillance Startups on Jul 18, 2018
This directory provides a list of video surveillance startups to help you see and research what companies are new or not yet broadly known entity...
Axis ~$100 Camera Tested on Jul 17, 2018
Axis has released their lowest cost camera ever, the Companion Eye Mini L, setting their sights on a market dominated by Hikvision and Dahua. Can...
Amazon Ring Alarm System Tested on Jul 16, 2018
Amazon Ring is going to hurt traditional dealers, and especially ADT, new IPVM test results of Ring's Alarm system underscore. IPVM found that...
Axis Perimeter Defender Video Analytics Tested on Jul 12, 2018
Axis 'high security' video analytics offering is Perimeter Defender, OEMed / developed with Digital Barriers. But how good is Perimeter Defender?...
Amazon Ring Partners With Rapid Response For $10 Monitoring on Jul 10, 2018
Amazon's Ring alarm system is using Rapid Response for monitoring, IPVM has confirmed in our testing. Amazon is arguably the most feared new...
Drops Dahua, Fenner Becomes ISS CMO on Jul 09, 2018
Hired to improve Dahua's miserable marketing just last year, Janet Fenner has quit Dahua, joining VMS manufacturer ISS as Chief Marketing...
UK VSaaS Startup Ocucon on Jul 03, 2018
Decreasing exposure to fraudulent slip-and-fall insurance claims and lawsuits is one of the oldest selling points of video surveillance for retail....
Digital Watchdog Low Cost 4MP Camera Tested on Jul 02, 2018
Based on member 4MP testing requests, we bought and tested Digital Watchdog's low-cost 4MP DWC-MTT4Wi to see how it performs in real world scenes,...
Simplisafe Acquired At $1 Billion Valuation on Jun 29, 2018
Simplisafe, the competitor alarm monitoring companies historically most love to hate, has been acquired by a mega private equity firm for a $1...
Panoramic Fisheye Camera Shootout - Avigilon, Axis, Dahua, Hanwha, Hikvision, Oncam And Vivotek on Jun 27, 2018
IPVM tested Avigilon, Axis, Dahua, Hanwha, Hikvision, Oncam And Vivotek 12MP panoramic fisheye cameras head to head, as shown in the test setup...

Most Recent Industry Reports

AI Startup Anyvision Raises $28 Million Led By Bosch on Jul 20, 2018
Anyvision is the most ambitious heavy-spending video surveillance startup in many years. And, now, the startup has raised $28 million led by...
Fail: Dahua "Didn't Check The Lux Levels but It Was Dark" on Jul 20, 2018
Dahua UK has been promoting their camera quality on LinkedIn: I, and others, asked what the lux level of the scene was. (background: Lux Rating...
Free 100+ Manufacturer-Customized Camera Calculator Released on Jul 19, 2018
Now, any manufacturer has a customized IPVM Camera Calculator, free. The goal is to make it easier for companies to help their customers better...
Improved Security And Surveillance Bidding - 2018 MasterFormat Divisions Examined) on Jul 19, 2018
Navigating the world of system specifications and bidding work can be complex and confusing, but a standard format exists, and understanding it...
Last Chance - Security Sales Course Summer 2018 on Jul 19, 2018
Today is the last day to register. Based on member's interest, IPVM is offering a security sales course this summer. Register Now - IPVM Security...
Directory of Video Surveillance Startups on Jul 18, 2018
This directory provides a list of video surveillance startups to help you see and research what companies are new or not yet broadly known entity...
Ladder Lockdown and Ladder Levelizer Tested on Jul 18, 2018
Ladders are a daily necessity for surveillance and security installers, but working on an unstable surface can be extremely dangerous. In addition...
FST Fails on Jul 17, 2018
FST was one of the hottest startups of the decade, selected as the best new product at ISC West 2011 and backed with tens of millions in...
Axis ~$100 Camera Tested on Jul 17, 2018
Axis has released their lowest cost camera ever, the Companion Eye Mini L, setting their sights on a market dominated by Hikvision and Dahua. Can...
Amazon Ring Alarm System Tested on Jul 16, 2018
Amazon Ring is going to hurt traditional dealers, and especially ADT, new IPVM test results of Ring's Alarm system underscore. IPVM found that...

The world's leading video surveillance information source, IPVM provides the best reporting, testing and training for 10,000+ members globally. Dedicated to independent and objective information, we uniquely refuse any and all advertisements, sponsorship and consulting from manufacturers.

About | FAQ | Contact