VMS Testing Plan 2013/2014

By: John Honovich, Published on Jul 23, 2013

We are embarking on a major new VMS testing program where we will perform in-depth tests both on features and products never done anywhere before. To lead that, Sarit Williams has joined IPVM. She previously led QA for a VMS developer. Now, she will be solely focused on testing and researching VMSes.

This note explains our testing approach and timetable for the next year.

Previously ** ****

*** **** ***** ***** of *** ******* ** did********** **** - ****, including **** **** * dozen *** **** ******* and * *** *********** comparison.

Step * - ******* *********** / *********

**** **** ******, ** are ******** **** ******* comparisons / ********* ***** we **** ** ******** VMS ***** *** ***** them ****. *** ********, we **** ******* *********-*********************, ***** ** **** 6 *****, ********* *** low ***** ******* ** how **** ********* ***** features.

*** ******* ** **** is ** ******* *********** research. ** *** ******** has **** ******* ******** and ********** *** ***** work ** * ******* by ******* *****. *** instance, ****** **** ***** exporting *** **** **** it ****** ******* **? What ***** ** ****** or *****? **** ******* exist? **** ******* ******** is ** ******* * foundation ** ********** ***** specifics.

** ********** ***** * dozen ******* ******* *********** / *********. ****** ******** 2013: ******* ****:

VMSes ******

**********, ** **** ** test ** ** ** VMSes. *******, *** *** feature / ***********, ** are ******** ** * to **** ** **********. (The **** ***** ** test, *** ****** **** report *****, *** ***** ones **** ** ********). We ******* * ** enough ** **** ******** while ****** *** ***** manageable ** *******.

***** *** * ** consider * ******** ****** based ** ***** *** member ********:

  • ******* *** ********* (*** 'Old *****')
  • ******** *** ***** (*** "Up *** ******')

**** **** *** **** they *** *** '****'. Indeed, **** ******* *** surprisingly *******. *******, ** does ***** ********* **** have ***** ****** (******* it ** ** *** or ** **** ****).

****** ****, ** **** rotate * ****** ** (so *** ** **** done ******* ***** *** VideoInsight *** ** ***** also *** ***** ******** and * *** ****** over ****).

Step * - ********** *** *****

***** ** ******** *** (initial ***** **) ******* tests - ****** **** time ** ********, ** will ****** ** ********** VMS *****, ***** *** report **** ***** ** just * ****** ************. We **** *** *** feature ***** ** * baseline *** ********** (*.*., if **** ** **** Luxriot, ** *** ***, "Luxriot's ********* ** ******/***** than *********** ******* **** have/lack ******** ******** *, Y *** * ********** tested ** *** ********* comparison, ***.).

***** *** ******* ****** of ***** ***** ******* and ** **** **** to ** **, *** more ******* * ** ** for *** ******* *****.

Step * - ***** **** ****

****** *** ****, ** plan ** ******* ** all *** **** *** Competitive ********** **** ****** down ******* ** ******* how ***** ***** **, letting ********** *** ***** know ***** *** ******** weaknesses *** *** **** VMSes ******* ****** **** their ********* / ***** hype.

Going *******

*********, **** ** **** the *********. ***** **** 3, ** **** ******** to ******, **** **** advanced ******* ** **** as *** *** *******. This, ******, ****** **** an ******** ** *** general ******** *** *** next ****. *** ** know **** ********* ** comments *** ****.

Comments (13)

Will relative costs to the end user, including SMA's and license renewals, be included? Total affordability, over a certain number of years, should be a major benchmark item, not just product feature sets. If you were doing automobiles, Rolls Royce or Ferrari might have the highest score, but be totally inappropriate for the majority of buyers due to cost factors.

Hi Ted,

Yes, we absolutely will cover total costs and compare to overall features/value delivered. The focus to start is understanding and documenting how the VMSes work, since this takes the most time. Once we do that, we can start comparing to the costs to better understand the relative value.

Welcome Sarit, I am looking forward to the reports! I am curious as to the level of “training” that these reviews will contain in the future? I don’t mean full how to type set up guides for each VMS or anything that in depth. Just some basics and maybe some tips/tricks discovered through using the VMS.

Thank you Ross! I will certainly continue to include training tips/tricks discovered in each VMS (like in the multistream and exporting reports). Additionally, we will be preparing new VMS 101 training to help train members on fundamentals and main differentiators; I hope to see you there.

What happens as feature sets change and evolve? Most certainly all vendors will have changes and upgrades in time for ISC. And where does Avigilon stand with IPVM continuing research on their products? Does IPVM foresee issues with obtaining new cameras or software updates from them considering the recent and ongoing problems with them?

I'll leave the first couple questions to Sarit, but as far as Avigilon: we have sources for Avigilon product and support has actually been answering questions for us. We also have sent questions and copies of posts their way, as we do with all manufacturers when doing tests or updates, but they have chosen not to respond.

So in short, we don't see it as a problem, and we welcome their feedback on technical issues, as always, if they choose to give it.

Would be interesting to also redo the integrator surveys again that you performed a couple years back.

Hi Thomas, We are going to redo them. However, it has not even been 2 years yet (e.g., favorite VMSes) so we are waiting to give some time for things to actually change. This fall would looks like an appropriate time.

Great Job ,

Can you include support issues relevant to the vms?

Can we send in suggestions on some weve used in past?

look forward to this report .

Christopher,

Thanks so much. Please feel free to email any info to sarit@ipvm.com; any information you have would be greatly appreciated. I can tell you that as I'm setting up these environments and clients and going through this process as an end user would I have lots to write about (errors during install, crashes, activations, etc) and those subjects are certainly on the list.

This is great and I look forward to the analysis. Would it be possible to see table listing all features in a side-by-side comparison? Each feature set would also need to be explained as well. I understand that such a schedule may be difficult, but it would be pretty easy to follow.

Sarit, John,

This is exactly what we think is required for BMS benchmarking. As some time back I raised this in this forum seeking the stress testing approaches as well as quantitatively measuring the features/performance.

I am looking forward the step1 and also will be happy to contribute in the process.

Thanks, Krish

Thank you Krish and Jim for the suggestions- duly noted! We will continue to jot down suggestions and recommendations as we expand our baseline tests.

Login to read this IPVM report.
Why do I need to log in?
IPVM conducts unique testing and research funded by member's payments enabling us to offer the most independent, accurate and in-depth information.

Related Reports

Panasonic 32MP Multi Imager Camera Tested (WV-X8570N) on May 16, 2019
Panasonic has released their first multi imager models including the 32MP (4x4K) WV-X8570N, claiming "Extreme image quality for evidence capturing...
Bank Security Manager Interview on May 15, 2019
Bank security contends with many significant threats - from fraudsters to robbers and more. In this interview, IPVM spoke with bank security...
Milestone XProtect 2019 R1 Tested on May 15, 2019
For the past few years, Milestone has released quarterly software updates XProtect VMS platform. What is new and how much impact do the updates...
China PRC Government New National Video Surveillance Standards on May 14, 2019
The People's Republic of China (PRC) government has released a new set of overarching standards for authorities to follow when they install video...
Bluecherry Open Sources Entire VMS on May 13, 2019
Bluecherry announced they have "released the entire Bluecherry software application open source with a GPL license". We spoke to Bluecherry's...
Mining Company Security Manager Interview on May 10, 2019
First Quantum Minerals Limited (FQML) is a global enterprise with offices on 4 continents and operations in 7 countries with exploratory operations...
Aegis AI Gun Detection Video Analytics Startup on May 07, 2019
Gun detection analytic startups are increasing as the promise of AI and the threats of active shooters grow.  One company, Aegis AI, is being led...
Restaurant Security Manager Interview on May 06, 2019
Wright’s Gourmet House in Tampa, Florida has been around for over 50 years. During most of that time, there were no security measures in place. Now...
Ranking Manufacturer Favorability 2019 on May 06, 2019
24 manufacturer's favorability was ranked based on 170+ integrators feedback. Voting plus in-depth comments revealed insights on which brands were...
Verkada Cloud VMS/Cameras Tested on May 02, 2019
Verkada is arguably the most ambitious video surveillance startup in many years. The company is developing their own cameras, their own VMS, their...

Most Recent Industry Reports

Amazon Ring Public Subsidy Program Aims To Dominate Residential Security on May 20, 2019
Amazon dominates market after market. Quitely, but increasingly, they are doing so in residential security, through a combination of significant...
LifeSafety Power NetLink Vulnerabilities And Problematic Response on May 20, 2019
'Power supplies' are not devices that many think about when considering vulnerabilities but as more and more devices go 'online', the risks for...
Facial Recognition Systems Fail Simple Liveness Detection Test on May 17, 2019
Facial recognition is being widely promoted as a solution to physical access control but we were able to simply spoof 3 systems because they had no...
Inside Look Into Scam Market Research on May 17, 2019
Scam market research has exploded over the last few years becoming the most commonly cited 'statistics' for most industries, despite there clearly...
Maglock Selection Guide on May 16, 2019
One of the most misunderstood yet valuable pieces of electrified hardware is the maglock. Few locks are stronger, but myths and confusion surround...
Panasonic 32MP Multi Imager Camera Tested (WV-X8570N) on May 16, 2019
Panasonic has released their first multi imager models including the 32MP (4x4K) WV-X8570N, claiming "Extreme image quality for evidence capturing...
Trump Signs 'Huawei Ban' - Executive Order Targeting Foreign Adversary Technology on May 16, 2019
US President Donald Trump has signed an executive order targeting technology provided by 'foreign adversaries', in what is widely being called a...
Bank Security Manager Interview on May 15, 2019
Bank security contends with many significant threats - from fraudsters to robbers and more. In this interview, IPVM spoke with bank security...
Milestone XProtect 2019 R1 Tested on May 15, 2019
For the past few years, Milestone has released quarterly software updates XProtect VMS platform. What is new and how much impact do the updates...
San Francisco Face Recognition Ban And Surveillance Regulation Details Examined on May 14, 2019
San Francisco passed the legislation 8-1 today. While the face recognition 'ban' has already received significant attention over the past few...

The world's leading video surveillance information source, IPVM provides the best reporting, testing and training for 10,000+ members globally. Dedicated to independent and objective information, we uniquely refuse any and all advertisements, sponsorship and consulting from manufacturers.

About | FAQ | Contact