IPVM Camera Testing Procedures

Author: John Honovich, Published on May 08, 2013

The foundation of IPVM's camera testing procedure is simultaneous, head to head comparisons, like so:

This is unique among surveillance industry publications.

Each camera is placed side by side, with the same FoV, connected to the same VMS and recorded at the same time and place. The setup looks like so:

We perform a few adjustments:

  • Firmware is upgraded to the most current version.
  • Minimum shutter speed is set to 1/30s to eliminate any slow shutter tricks.
  • Cameras are re-focused prior to testing.

However, unlike manufacturer demos, we do not 'tweak' cameras to optimize for individual scenes. Rather, we keep the same settings for all tests, to keep it fair, consistent and true to the most common real world uses.

We conduct tests in 3 fundamental areas:

  • Even light - this is the 'ideal' test when all cameras do the best.
  • Low light - measured at 0.2 lux
  • WDR - in an open door way back light

They look like so:

Get Video Surveillance News In Your Inbox
Get Video Surveillance News In Your Inbox

We typically run a series of pre-tests to get a feel for performance and identify any abnormalities. If the image quality looks excessively poor, we will re-check settings and/or request feedback from a manufacturer engineer. The multiple pre-test runs works out issues and increases the level of confidence that the test results accurately and fairly reflect performance.

Finally, we release test reports that are centered on a series of 5 - 15 image comparisons of multiple cameras head to head; again, like so:

The best way to understand relative performance is to properly setup and test simultaneously. Unlike looking at cameras individually, where it is easy to overestimate performance, the IPVM approach makes it clear which cameras have advantages in specific scenarios.

Bandwidth / Quantization / Compression

We typically measure bandwidth levels for each camera in each scenario. Some camera manufacturers attempt to game quantization / compression levels. We will measure those levels using a DRF analysis tool that allows us to understand if there is significant variances. This is especially important for manufacturers claiming to deliver super low bandwidth who typically do so by increasing compression levels but failing to disclose. We will report this and make adjustments in testing as appropriate.

Test Reports

See our directory of 160+ surveillance test reports. Recent favorites include our WDR Shootout, Panoramic Shootout and Axis vs Bosch Low Light Shootout.

Feel free to ask questions and we are happy to expand on this reference document.

Comments (11): PRO Members only. Login. or Join.

Related Reports

Uniview (UNV) IP Cameras Tested on Feb 22, 2017
"We're #3," in China says Uniview (UNV). While the company significantly trails Hikvision and Dahua in total sales, one notable difference is that...
Glass Doors and Access Control Tutorial on Feb 22, 2017
The biggest challenge for many access control systems are glass doors. Here's what happens when a maglock is improperly installed to an existing...
Arecont Vision MicroDome Duo Tested on Feb 14, 2017
Arecont Vision is back with another multi-head camera, this time thinking smaller with the MicroDome Duo, a two-imager model, with the tagline...
Alarm Panels - Canned vs Integrated on Feb 07, 2017
Alarm control panels generally come in one of two types: Integrated alarm controllers, with controls and the keypad in the same...
LAST DAY - Camera Course Winter 2017 on Feb 02, 2017
[NOTE: Today, Thursday is the last day to register.] Learn video surveillance and get certified. IPVM provides live online classes, recorded...
Panasonic Extreme H.265 Cameras Tested on Jan 26, 2017
Panasonic has released their latest generation, the i-Pro Extreme series, with enhanced compression (smart H.265) enhanced WDR and secure...
Arecont Lowest Favorability Results on Jan 25, 2017
Given Arecont is 3 time defending integrator choice for worst camera manufacturer, it should be little surprise they had the lowest score in the...
WDR Shootout 2017: Low vs High Cost Cameras on Jan 24, 2017
Multi-exposure WDR was once only available in expensive cameras but in the past few years, this capability has become available even in sub $200...
Goodbye Samsung, Hello Wisenet X on Jan 23, 2017
Samsung is gone but Hanwha is back. Their latest generation Wisenet X, touts a slew of new high end features including H.265, WiseStream II,...
IP Networking Course January 2017 on Jan 12, 2017
This is the only networking course designed specifically for video surveillance professionals plus it includes live training, personal help and...

Most Recent Industry Reports

Honeywell Sues Alarm.com For Violating Anti-Trust Laws on Feb 24, 2017
Is Alarm.com about to dominate the smart home software market? That is what Honeywell alleges in its new lawsuit, first reported by...
Axis: "Everything is IP" - False on Feb 24, 2017
Axis is congratulating itself, with executive Fredrick Nilsson declaring: "Now the conversion is all done and everything is IP and analog is...
Advertising Like Avigilon at the ISC West Airport on Feb 24, 2017
Avigilon has grabbed a lot of attention over the last few years advertising at the Las Vegas airport when attendees fly in. But how does that...
Artificial Intelligence Robot Assistant (ACTi) on Feb 23, 2017
Has artificial intelligence come to the video surveillance industry? ACTi has released 'SARA' which it bills as an 'AI assistant that brings...
Cutting Costs 70% Using Milestone With HD Analog on Feb 23, 2017
HD analog and enterprise VMSes are often thought of as being on opposite sides of the spectrum, with HD analog best for small jobs due to its low...
Dahua 4K HD Analog Cameras Announced on Feb 23, 2017
HD analog has been gaining popularity (even if Axis hopes otherwise). Last year, HD analog's max resolution doubled from 1080p to 4MP (see our 4MP...
Uniview (UNV) IP Cameras Tested on Feb 22, 2017
"We're #3," in China says Uniview (UNV). While the company significantly trails Hikvision and Dahua in total sales, one notable difference is that...
Glass Doors and Access Control Tutorial on Feb 22, 2017
The biggest challenge for many access control systems are glass doors. Here's what happens when a maglock is improperly installed to an existing...
Exacq Favorability Results on Feb 22, 2017
For years, Exacq has been one of the most frequently favored VMSes in IPVM integrator statistics (e.g., see Favorite VMS Manufacturers...
The Hot RMR Company - Electric Guard Dog on Feb 22, 2017
The financiers at the Barnes Buchanan conference praised a company named 'Electric Guard Dog'. While the name sounds fairly low tech, the money and...

The world's leading video surveillance information source, IPVM provides the best reporting, testing and training for 10,000+ members globally. Dedicated to independent and objective information, we uniquely refuse any and all advertisements, sponsorship and consulting from manufacturers.

About | FAQ | Contact