IPVM Camera Testing Procedures

Author: John Honovich, Published on May 08, 2013

The foundation of IPVM's camera testing procedure is simultaneous, head to head comparisons, like so:

This is unique among surveillance industry publications.

Each camera is placed side by side, with the same FoV, connected to the same VMS and recorded at the same time and place. The setup looks like so:

We perform a few adjustments:

  • Firmware is upgraded to the most current version.
  • Minimum shutter speed is set to 1/30s to eliminate any slow shutter tricks.
  • Cameras are re-focused prior to testing.

However, unlike manufacturer demos, we do not 'tweak' cameras to optimize for individual scenes. Rather, we keep the same settings for all tests, to keep it fair, consistent and true to the most common real world uses.

We conduct tests in 3 fundamental areas:

  • Even light - this is the 'ideal' test when all cameras do the best.
  • Low light - measured at 0.2 lux
  • WDR - in an open door way back light

They look like so:

Get Video Surveillance News In Your Inbox
Get Video Surveillance News In Your Inbox

We typically run a series of pre-tests to get a feel for performance and identify any abnormalities. If the image quality looks excessively poor, we will re-check settings and/or request feedback from a manufacturer engineer. The multiple pre-test runs works out issues and increases the level of confidence that the test results accurately and fairly reflect performance.

Finally, we release test reports that are centered on a series of 5 - 15 image comparisons of multiple cameras head to head; again, like so:

The best way to understand relative performance is to properly setup and test simultaneously. Unlike looking at cameras individually, where it is easy to overestimate performance, the IPVM approach makes it clear which cameras have advantages in specific scenarios.

Bandwidth / Quantization / Compression

We typically measure bandwidth levels for each camera in each scenario. Some camera manufacturers attempt to game quantization / compression levels. We will measure those levels using a DRF analysis tool that allows us to understand if there is significant variances. This is especially important for manufacturers claiming to deliver super low bandwidth who typically do so by increasing compression levels but failing to disclose. We will report this and make adjustments in testing as appropriate.

Test Reports

See our directory of 160+ surveillance test reports. Recent favorites include our WDR Shootout, Panoramic Shootout and Axis vs Bosch Low Light Shootout.

Feel free to ask questions and we are happy to expand on this reference document.

Comments (11): PRO Members only. Login. or Join.

Related Reports

Access Controller Software Guide on Dec 11, 2017
Properly configuring access controllers software is key to a professional access system. These devices have fundamental settings that must be...
2018 Video Surveillance Cameras Overview on Dec 11, 2017
This report concisely explains the developments for surveillance cameras offered in 2017 and the state of offerings going into 2018, including...
D-Link ONVIF Switch Tested on Dec 04, 2017
D-Link's surveillance switches claim to "enhance ease of use and streamline management" for network administrators, with simplified UIs and...
Hikvision Door Station Tested on Nov 30, 2017
Hikvision has entered the video intercom market, aiming to bring the race to the bottom to a whole new audience. To see how it stacks up, we...
IP Camera Manufacturer Compression Comparison on Nov 27, 2017
Despite the use of standards-based video compression (H.264/H.265), our tests show that default image quality settings for different manufacturers...
Hazardous & Explosion Proof Access Control Tutorial on Nov 27, 2017
Controlling access to hazardous environments require equipment meeting specific ratings that certify they will not start fires. Understanding those...
Camera Multi-Streaming Usage on Nov 22, 2017
IP cameras typically support multiple streams, allowing a single camera to transmit multiple streams at different resolutions, frame rates and even...
CBR vs VBR vs MBR - Surveillance Streaming on Nov 21, 2017
How you stream video has a major impact on quality and bandwidth. And it is not simply CODEC choice (e.g., H.264 vs H.265). Regardless of the...
Hikvision 4K IR PTZ Tested (DS-2DF8836IV-AELW) on Nov 10, 2017
The IR PTZ trend continues, now with 4K resolution, with Hikvision releasing its DS-2DF8836IVAEL-W camera, a 4K 1/1.9" PTZ model specifying 200m IR...
ZoneMinder Open Source Profile on Nov 06, 2017
The first, oldest and probably the best known open source VMS is ZoneMinder. But where has it gone and is it still relevant today in a world of...

Most Recent Industry Reports

Robot Vandalism on Dec 11, 2017
Vandalism of security systems is a common concern. It is so common that camera vandalism statistics show that designers routinely sacrifice camera...
Access Controller Software Guide on Dec 11, 2017
Properly configuring access controllers software is key to a professional access system. These devices have fundamental settings that must be...
2018 Video Surveillance Cameras Overview on Dec 11, 2017
This report concisely explains the developments for surveillance cameras offered in 2017 and the state of offerings going into 2018, including...
Imperial Capital Security Investor Conference Review on Dec 08, 2017
Investment bank Imperial Capital holds an annual Security Investor Conference where 60+ companies present, including this year: IPVM bought a...
Integrator GPS Vehicle Tracking Statistics and Success Examined on Dec 08, 2017
GPS vehicle tracking is a growing but somewhat controversial topic. On the plus side, tracking may increases productivity by providing greater...
Hikvision NA Biggest Sale of 2017 on Dec 07, 2017
Hikvision North America has been relatively disciplined the past 5 months, reducing the number of sales and the breadth of what is on sale. No...
Security Integrator IT Expertise Statistics on Dec 07, 2017
20 years ago, putting physical security systems on IP networks was just emerging. Today, almost every system is networked in some way, IP cameras...
Lighthouse Deep Learning Camera Tested on Dec 07, 2017
A Silicon Valley startup, Lighthouse, with a Stanford PhD CTO, has released a deep learning AI camera with 3D sensors for just $300. The company...
Access Control Course Winter 2018 on Dec 07, 2017
Learn more below about the Winter 2018 IPVM Access Control Course. Register here. IPVM offers the most comprehensive access control course in...
Broken Hikvision App Exposes Hypocrisy on Dec 06, 2017
While Hikvision talks about a commitment to cybersecurity, their broken app and their insecure 'solution' exposes not only their engineering...

The world's leading video surveillance information source, IPVM provides the best reporting, testing and training for 10,000+ members globally. Dedicated to independent and objective information, we uniquely refuse any and all advertisements, sponsorship and consulting from manufacturers.

About | FAQ | Contact