How Should Video Management Software be Evaluated?

By: John Honovich, Published on Dec 27, 2009

In this report, we propose criteria to evaluate Video Management Software (VMS). Those selecting VMS software may find this useful. Moreover we plan to use this criteria for our upcoming testing series on VMS software (expands our ongoing IP Video Surveillance product testing program) .

We encourage comments, feedback and discussion on these criteria.

Goals of Our Evaluation

We believe VMS software evaluations should:

  • Focus on users rather than features: The criteria should examine how well the VMS software meets the goals of real users (conduct an investigation, review an alert) rather than simply list technical features.
  • Educate rather than rank: The evaluation should help users better understand how the software works rather than simply providing scores that obscure details and differences.
  • Show Usability: What is most hard to judge about VMS software from marketing material is their usability (see the widespread unhappiness in our recent discussion on VMS usability). All VMS software have dozens of features (or more) but which are the most complete, are the easiest to use, etc.
  • General rather than specific to an application: Since our results will be read by thousands of people around the world, we need to make it general enough so that many readers can make use of it. For a specific deployment, we would certainly supplement this with specific questions for the user's application.

The common 'trick' in evaluations designed by vendors is to take their data sheet and create a checklist that matches the VMS's proprietary implementation and to judge other systems by that criteria. By contrast it is critical that our evaluation is independent and user focused.

Criteria/Questions

We propose the following questions be used (with some notes on issues involved):

Basic Use (Live and Recorded Video)
  • How do I watch live video? (There are surprising variances in the complexity and options even in watching 1 or a few cameras)
  • What ways can I monitor many video feeds simultaneously?
  • How can I be notified and review alarms?
  • How do I search for recorded video? (The variances are tremendous amongst systems)
  • What ways can I search multiple video feeds?
  • How do you share images from video?
  • How do you share video clips from recorded video? 
  • How do I know if I am in live or recorded video mode? (This is often difficult to do in VMS software which is both sad and confusing)
  • How do you switch between live video, investigations and configuration? (Some systems require multiple applications, others are integrated into a single client, etc.)

Administration
  • How do I add/setup a camera?
  • How do you set up alarms to be displayed? (Some systems are very complex in setting up alarms)
  • How do you restrict access to the system? 
  • How do you restrict access to certain features or functionalities?
  • How can you setup the VMS to handle failures to servers/appliances?
  • How can you access reports or alerts on the health status of the servers/appliances/applications?
  • How can you upgrade the VMS software?

Usability
  • How does the VMS handle the higher resolution of megapixel cameras?
  • How does the VMS perform with low bandwidth connection (256 kb/s)? (Many VMS users are remote)
  • What, if any, VMS functionalities regulary take more than 10 seconds to load? (When software is unresponsive, users are less likely to use it and be more unsatisfied)
  • What happens when the VMS client cannot connect to a camera? (Since IP cameras can drop frequently, this can become a major source of frustration depending on how the VMS system handles it)
  • What options does the VMS provide to reduce storage consumption?
  • What help functionality does the VMS provide while using the client?
  • How similiar is the use of the VMS client to a DVR? (This is important to most security operators and guards who have limited technical training and prefer simple systems)
  • How similar is the configuration of the VMS server to a DVR? (This is important to traditional security integrators who often have limited IT expertise)
  • How simple is it for an untrained operator? (Many systems have impressive feature offerings but are extremely difficult for untrained operators to understand without formal training)
  • How powerful is it for the expert operator? (Some systems are quite limited in advanced functionality)

Scalability
  • What ways can I add/setup many cameras at the same time?
  • How do you do restrict access when using multiple servers/appliances? (This is a major differentiator amongst systems and a core requirement for larger, multi-site deployments)
  • How can I watch live video feeds from different appliances/servers?
  • How can I conduct searches from video feeds stored in different appliances/servers?

In this criteria, we omitted 3rd party support for cameras, security, business and storage systems. As a practical matter, we will unlikely be able to test these comprehensively. However, we will factor all of those into the overall report. Finally, pricing will certainly be included, allowing us to evaluate overall value comparing performance to price.

Our Tests vs Your Own Tests

We expect our tests to provide a low cost way to understand key comparative strengths and benefits of various VMS systems. Testing numerous VMS systems oneself is impractical for most, simply because of the cost and time involved (which can easily be thousands of dollars). We can provide this for a fraction of the cost. Moreover, because we focus solely on video surveillance, we can identify key issues and comparative concerns easily.

We do recommend that end users and integrators tests the systems themselves that they plan to roll out. Our tests can help you determine what the 'short list' of vendors should be and provide basis for your own testing for your finalists. 

Feedback

1 report cite this report:

Readers Respond: 134 Votes, 25 Comments Examined on Feb 14, 2010
Over 100 people voted and more than 25 left detailed comments to our recent question/discussion. In this post, we review and respond to the...

Related Reports

Phone Camera Calculator Released on Mar 10, 2020
IPVM has released the first-ever Phone Calculator, video surveillance design software that you can use on your phone, without installing an...
VMS 101 on Mar 03, 2020
This guide teaches the fundamentals about video management software. Inside, we cover: NVR vs VMS Viewing Video - What are common client...
Video Surveillance 101 Course - Last Chance on Feb 20, 2020
This is the last chance to join IPVM's first Video Surveillance 101 course, designed to help those new to the industry to quickly understand the...
Video Surveillance Architecture 101 on Feb 18, 2020
Video surveillance can be designed and deployed in a number of ways. This 101 examines the most common options and architectures used in...
2020 IP Networking Book Released on Jan 06, 2020
The new IP Networking Book 2020 is a 280 page in-depth guide that teaches you how IT and telecom technologies impact modern security systems,...
Milestone XProtect 2019 R3 'Centralized Search' Tested on Oct 30, 2019
Milestone has had problems over the last few years releasing significant new software. Now, in XProtect 2019 R3, Milestone is touting "one search...
Camera Calculator V3.1 Release Improves User Experience on Oct 17, 2019
IPVM has released a new version of our Camera Calculator, V3.1, with significant user experience improvements, a new development plan, and an...
Avigilon Blue VSaaS Tested on Aug 05, 2019
Avigilon says Blue is a "powerful integrator cloud service platform", easy to set up and configure, quickly scale business, by leveraging cloud...
IndigoVision Control Center VMS Tested on May 30, 2019
IPVM's last test of IndigoVision's VMS was in 2010, which found enterprise VMS features and a simple client interface. but no 3rd party camera...
Milestone XProtect 2019 R1 Tested on May 15, 2019
For the past few years, Milestone has released quarterly software updates XProtect VMS platform. What is new and how much impact do the updates...

Most Recent Industry Reports

Wyze Raises $10 Million And Seeks Services Expansion on May 27, 2020
Wyze has raised $10 million, the company's first disclosed raise since the $20 million announced at the beginning of 2019. Inside this note,...
Startup Videoloft Presents Cloud Storage on May 27, 2020
Videoloft presented offsite cloud storage at the May 2020 IPVM Startups show. A 30-minute video from Videoloft including IPVM...
Directory of 250+ Fever Camera News Reports Globally on May 27, 2020
This global directory tracks 250+ articles about thermal cameras used to detect fevers in response to the coronavirus pandemic. Articles are...
Integrators Rising Against Coronavirus on May 27, 2020
IPVM integrator statistics make it clear - Coronavirus's impact on business is lessening and many are anticipating even better news in weeks...
Netposa Stock Surges 46% After US Human Rights Abuse Sanctions on May 27, 2020
Last Friday, the US government announced it would sanction PRC video management provider NetPosa for being "complicit in human rights violations...
LILIN Presents NDAA-Compliant P2 Cameras on May 26, 2020
Merit LILIN presented its NDAA-compliant P2 camera series at the April 2020 IPVM New Products show. Inside this report: A 30-minute video...
ZKTeco Body Temperature and Mask Detection Reader Tested on May 26, 2020
While dedicated fever cameras emerged first, now tablet/kiosk fever detectors are ramping up. China's ZKTeco has been aggressively promoting such...
IDIS Presents 12MP IR Panoramic Fisheye on May 26, 2020
IDIS presented its 12MP IR panoramic fisheye camera at the April 2020 IPVM New Products show. Inside this report: A 30-minute video from...
FDA Defines Correct Operation of "Fever Cameras" on May 26, 2020
The US FDA has now defined the correct operation of "Thermal Imaging Systems", colloquially known as "fever cameras". Many in video...
OnSSI Founders Return, Start Corsight on May 25, 2020
The OnSSI founders are back, less than 2 years after selling OnSSI to Qognify, they have returned to Corsight, a spin-out of an Israeli AI...