IPVM Camera Testing ProceduresBy: John Honovich, Published on May 08, 2013
The foundation of IPVM's camera testing procedure is simultaneous, head to head comparisons, like so:
This is unique among surveillance industry publications.
Each camera is placed side by side, with the same FoV, connected to the same VMS and recorded at the same time and place. The setup looks like so:
We perform a few adjustments:
- Firmware is upgraded to the most current version.
- Minimum shutter speed is set to 1/30s to eliminate any slow shutter tricks.
- Cameras are re-focused prior to testing.
However, unlike manufacturer demos, we do not 'tweak' cameras to optimize for individual scenes. Rather, we keep the same settings for all tests, to keep it fair, consistent and true to the most common real world uses.
We conduct tests in 3 fundamental areas:
- Even light - this is the 'ideal' test when all cameras do the best.
- Low light - measured at 0.2 lux
- WDR - in an open door way back light
They look like so:
We typically run a series of pre-tests to get a feel for performance and identify any abnormalities. If the image quality looks excessively poor, we will re-check settings and/or request feedback from a manufacturer engineer. The multiple pre-test runs works out issues and increases the level of confidence that the test results accurately and fairly reflect performance.
Finally, we release test reports that are centered on a series of 5 - 15 image comparisons of multiple cameras head to head; again, like so:
The best way to understand relative performance is to properly setup and test simultaneously. Unlike looking at cameras individually, where it is easy to overestimate performance, the IPVM approach makes it clear which cameras have advantages in specific scenarios.
Bandwidth / Quantization / Compression
We typically measure bandwidth levels for each camera in each scenario. Some camera manufacturers attempt to game quantization / compression levels. We will measure those levels using a DRF analysis tool that allows us to understand if there is significant variances. This is especially important for manufacturers claiming to deliver super low bandwidth who typically do so by increasing compression levels but failing to disclose. We will report this and make adjustments in testing as appropriate.
Feel free to ask questions and we are happy to expand on this reference document.