IPVM Camera Testing Procedures

Author: John Honovich, Published on May 08, 2013

The foundation of IPVM's camera testing procedure is simultaneous, head to head comparisons, like so:

This is unique among surveillance industry publications.

Each camera is placed side by side, with the same FoV, connected to the same VMS and recorded at the same time and place. The setup looks like so:

We perform a few adjustments:

  • Firmware is upgraded to the most current version.
  • Minimum shutter speed is set to 1/30s to eliminate any slow shutter tricks.
  • Cameras are re-focused prior to testing.

However, unlike manufacturer demos, we do not 'tweak' cameras to optimize for individual scenes. Rather, we keep the same settings for all tests, to keep it fair, consistent and true to the most common real world uses.

We conduct tests in 3 fundamental areas:

  • Even light - this is the 'ideal' test when all cameras do the best.
  • Low light - measured at 0.2 lux
  • WDR - in an open door way back light

They look like so:

Get Video Surveillance News In Your Inbox
Get Video Surveillance News In Your Inbox

We typically run a series of pre-tests to get a feel for performance and identify any abnormalities. If the image quality looks excessively poor, we will re-check settings and/or request feedback from a manufacturer engineer. The multiple pre-test runs works out issues and increases the level of confidence that the test results accurately and fairly reflect performance.

Finally, we release test reports that are centered on a series of 5 - 15 image comparisons of multiple cameras head to head; again, like so:

The best way to understand relative performance is to properly setup and test simultaneously. Unlike looking at cameras individually, where it is easy to overestimate performance, the IPVM approach makes it clear which cameras have advantages in specific scenarios.

Bandwidth / Quantization / Compression

We typically measure bandwidth levels for each camera in each scenario. Some camera manufacturers attempt to game quantization / compression levels. We will measure those levels using a DRF analysis tool that allows us to understand if there is significant variances. This is especially important for manufacturers claiming to deliver super low bandwidth who typically do so by increasing compression levels but failing to disclose. We will report this and make adjustments in testing as appropriate.

Test Reports

See our directory of 160+ surveillance test reports. Recent favorites include our WDR Shootout, Panoramic Shootout and Axis vs Bosch Low Light Shootout.

Feel free to ask questions and we are happy to expand on this reference document.

Comments (11): PRO Members only. Login. or Join.

Related Reports

Last Day IP Networking Course January 2017 on Jan 12, 2017
This is the only networking course designed specifically for video surveillance professionals plus it includes live training, personal help and...
Camera Course Winter 2017 on Jan 05, 2017
Learn video surveillance and get certified. IPVM provides live online classes, recorded videos, personal help, cutting edge education and...
Avigilon Pro 4K Camera Tested on Jan 03, 2017
Avigilon is best known for their large sensor, high megapixel cameras. But with new offerings from Sony and soon Axis, the questions is how well...
IP Networking 2017 Book Released on Jan 02, 2017
The new IP Networking Book 2017 is a 197 page in-depth guide that teaches you how IT and telecom technologies impact modern security systems. This...
Hikvision No on H.265 But Pro On H.265+ on Jan 02, 2017
Hikvision said H.265 was a 'solution in search of a problem' a half year ago but now they are getting bullish, embracing H.265+. In this report,...
Surveillance Cameras 2017 Review on Jan 02, 2017
This report concisely explains the developments and most common options for surveillance cameras offered in 2017, including resolution, H.265, HD...
Calculating Video Surveillance Storage / Bandwidth on Dec 29, 2016
Calculating surveillance bandwidth is complex, and inexperienced users can easily underestimate bandwidth, leading to reduced storage durations...
Lux Rating / Minimum Illumination Guide 2017 on Dec 23, 2016
Lux ratings are one of the poorest specifications to use in selecting cameras. Now, with the rise of integrated IR, they are increasingly...
The PPF / PPM Video Surveillance Guide on Dec 23, 2016
Pixels per foot / Pixels per meter is the most fundamental and valuable, though imperfect, metric for specifying video surveillance image...
Surveillance Camera Imager Tutorial on Dec 23, 2016
Imagers - CCD, CMOS, 1/2", 1/4", big pixels, small pixels, etc. In this tutorial, we explain the fundamental issues and drivers in surveillance...

Most Recent Industry Reports

Canon 250MP Prototype Targeted At Surveillance on Jan 17, 2017
At one point Axis declared the megapixel race was over, but now parent company Canon is showing an imager that would be a rocket ship in a...
Cut in Half, Everfocus Shifts Strategies on Jan 17, 2017
The race to the bottom impact continues. Now, Everfocus, who used to be one of the larger budget providers, is shifting strategies after years of...
Top ISC West 2017 Booth Trends on Jan 16, 2017
There are significant moves among many major manufacturers on the ISC West show floor, and very few prime spaces left unoccupied. In this report we...
Genetec Favorability Results on Jan 16, 2017
In the race to the bottom and flight to 'solutions', Genetec has taken a contrary path. The company remains independent, focusing up market,...
Free VMS Software Directory on Jan 13, 2017
Many Video Management Software (VMS) providers offer free versions, either open source, for a limited number of cameras or for a limited amount of...
Milestone Essential Subscription Removed on Jan 12, 2017
Subscriptions may be the future for Milestone and VMSes but not right now. Responding to negative feedback, Milestone has removed subscriptions...
Alarm Reporting Formats Overview on Jan 12, 2017
Alarm reporting formats are methods for communicating complicated information quickly, using plain old telephone service. It is still the main...
Amazon Sales of Hikvision China Cameras on Jan 12, 2017
Hikvision has become widely available in the US, including on popular retail outlets like Amazon, with over 2,600 results: 4MP Hikvision outdoor...
Last Day IP Networking Course January 2017 on Jan 12, 2017
This is the only networking course designed specifically for video surveillance professionals plus it includes live training, personal help and...
ONVIF Favorability Results on Jan 11, 2017
ONVIF has been one of the most debated aspects of the video surveillance industry. On the one hand, its aim to increase interoperability has been...

The world's leading video surveillance information source, IPVM provides the best reporting, testing and training for 10,000+ members globally. Dedicated to independent and objective information, we uniquely refuse any and all advertisements, sponsorship and consulting from manufacturers.

About | FAQ | Contact