IPVM Camera Testing Procedures

Author: John Honovich, Published on May 08, 2013

The foundation of IPVM's camera testing procedure is simultaneous, head to head comparisons, like so:

This is unique among surveillance industry publications.

Each camera is placed side by side, with the same FoV, connected to the same VMS and recorded at the same time and place. The setup looks like so:

We perform a few adjustments:

  • Firmware is upgraded to the most current version.
  • Minimum shutter speed is set to 1/30s to eliminate any slow shutter tricks.
  • Cameras are re-focused prior to testing.

However, unlike manufacturer demos, we do not 'tweak' cameras to optimize for individual scenes. Rather, we keep the same settings for all tests, to keep it fair, consistent and true to the most common real world uses.

We conduct tests in 3 fundamental areas:

  • Even light - this is the 'ideal' test when all cameras do the best.
  • Low light - measured at 0.2 lux
  • WDR - in an open door way back light

They look like so:

Get Video Surveillance News In Your Inbox
Get Video Surveillance News In Your Inbox

We typically run a series of pre-tests to get a feel for performance and identify any abnormalities. If the image quality looks excessively poor, we will re-check settings and/or request feedback from a manufacturer engineer. The multiple pre-test runs works out issues and increases the level of confidence that the test results accurately and fairly reflect performance.

Finally, we release test reports that are centered on a series of 5 - 15 image comparisons of multiple cameras head to head; again, like so:

The best way to understand relative performance is to properly setup and test simultaneously. Unlike looking at cameras individually, where it is easy to overestimate performance, the IPVM approach makes it clear which cameras have advantages in specific scenarios.

Bandwidth / Quantization / Compression

We typically measure bandwidth levels for each camera in each scenario. Some camera manufacturers attempt to game quantization / compression levels. We will measure those levels using a DRF analysis tool that allows us to understand if there is significant variances. This is especially important for manufacturers claiming to deliver super low bandwidth who typically do so by increasing compression levels but failing to disclose. We will report this and make adjustments in testing as appropriate.

Test Reports

See our directory of 160+ surveillance test reports. Recent favorites include our WDR Shootout, Panoramic Shootout and Axis vs Bosch Low Light Shootout.

Feel free to ask questions and we are happy to expand on this reference document.

Comments (11): PRO Members only. Login. or Join.

Related Reports

Dahua 4K IR PTZ Tested on Aug 21, 2017
4K has made its way to IR PTZs. In this report, we examine the Dahua 6AE830VNI, a 4K PTZ with 30x optical zoom, 200m (~650') integrated IR, and...
Sony Next Gen HD Dome Camera Tested (SNC-EM642R) on Aug 18, 2017
Sony has released their latest generation, claiming improved WDR and low light, increased IR range, and more. We tested the SNC-EM642R outdoor IR...
IP Networking Course September 2017 on Aug 17, 2017
This is the only networking course designed specifically for video surveillance professionals plus it includes live training, personal help and...
Cat 5e vs Cat 6 vs Cat 6a Network Cable Usage Statistics on Aug 16, 2017
Cat 5e? Cat 6? Cat 6a? What do integrators use in practice, today? 140+ integrators told IPVM. Here are the results: For those who want to...
IP Camera Specification / RFP Guide 2017 on Aug 14, 2017
RFPs are hard. Do them 'right' and it takes a lot of knowledge and time. Do them 'wrong' and you can be (a) unwittingly locked into a specific...
Dahua 4K Turret Tested Vs Hikvision (N84BG44 ) on Aug 02, 2017
Dahua has released their latest low cost 4K model, the N84BG44, claiming Smart H.265+ and 50m IR range. We bought and tested this new model against...
Super Low Light Surveillance Guide on Aug 01, 2017
At the beginning of this decade, low light performance of megapixel cameras was generally terrible. This was the era of super slow shutter where...
IR Surveillance Guide on Jul 31, 2017
Infrared (IR) has become an increasing core component to video surveillance systems. In particular, the expansion of integrated IR cameras that...
Hikvision H.265+ Bullet Tested (2035) on Jul 24, 2017
Continuing our tests of Hikvision's new low cost Value Plus line, we bought and tested the 3MP DS-2CD2035FWD-I, now including H.265+. We shot the...
Axis Door Station Tested (A8105-E) on Jul 19, 2017
Axis continues their push into niche markets, especially audio, with network speakers, an IP horn, and video door stations. We bought and tested...

Most Recent Industry Reports

Dahua 4K IR PTZ Tested on Aug 21, 2017
4K has made its way to IR PTZs. In this report, we examine the Dahua 6AE830VNI, a 4K PTZ with 30x optical zoom, 200m (~650') integrated IR, and...
Top Used License Plate Capture Cameras on Aug 21, 2017
Capturing license plates is a common video surveillance application. But what cameras do integrators mostly commonly used? Special purpose LPC...
VLAN For Video Surveillance Usage Statistics on Aug 21, 2017
VLANs (see our tutorial) are an option for networks using video surveillance, but how often are they actually used? 125+ integrators told us how...
Avigilon CEO Attacks Asian Companies Cyber Insecurity on Aug 18, 2017
Avigilon CEO is taking aim at their Asian competitors. And he is going directly after these company's cyber security issues. In this note, we...
Sony Next Gen HD Dome Camera Tested (SNC-EM642R) on Aug 18, 2017
Sony has released their latest generation, claiming improved WDR and low light, increased IR range, and more. We tested the SNC-EM642R outdoor IR...
IP Networking Course September 2017 on Aug 17, 2017
This is the only networking course designed specifically for video surveillance professionals plus it includes live training, personal help and...
Knightscope Raises $10 Million With $3,320 Average Per Investor on Aug 17, 2017
Congrats to Knightscope. And condolences to their legion of little investors. Knightscope has disclosed they have raised $10+ million from their...
Axis and Arecont Legal Conflict Over Multi-Imager Cameras on Aug 17, 2017
Arecont threatened Axis. Axis has responded by moving to invalidate an Arecont patent. It is an important contest. Multi-imagers are Arecont's...
Directory Of Consumer Security Cameras on Aug 16, 2017
The consumer camera segment continues to grow, with new startups and models from existing players released seemingly every month. In this report we...
Cat 5e vs Cat 6 vs Cat 6a Network Cable Usage Statistics on Aug 16, 2017
Cat 5e? Cat 6? Cat 6a? What do integrators use in practice, today? 140+ integrators told IPVM. Here are the results: For those who want to...

The world's leading video surveillance information source, IPVM provides the best reporting, testing and training for 10,000+ members globally. Dedicated to independent and objective information, we uniquely refuse any and all advertisements, sponsorship and consulting from manufacturers.

About | FAQ | Contact