IPVM Camera Testing Procedures

By: John Honovich, Published on May 08, 2013

The foundation of IPVM's camera testing procedure is simultaneous, head to head comparisons, like so:

This is unique among surveillance industry publications.

Each camera is placed side by side, with the same FoV, connected to the same VMS and recorded at the same time and place. The setup looks like so:

We perform a few adjustments:

  • Firmware is upgraded to the most current version.
  • Minimum shutter speed is set to 1/30s to eliminate any slow shutter tricks.
  • Cameras are re-focused prior to testing.

However, unlike manufacturer demos, we do not 'tweak' cameras to optimize for individual scenes. Rather, we keep the same settings for all tests, to keep it fair, consistent and true to the most common real world uses.

We conduct tests in 3 fundamental areas:

  • Even light - this is the 'ideal' test when all cameras do the best.
  • Low light - measured at 0.2 lux
  • WDR - in an open door way back light

They look like so:

We typically run a series of pre-tests to get a feel for performance and identify any abnormalities. If the image quality looks excessively poor, we will re-check settings and/or request feedback from a manufacturer engineer. The multiple pre-test runs works out issues and increases the level of confidence that the test results accurately and fairly reflect performance.

Get Notified of Video Surveillance Breaking News
Get Notified of Video Surveillance Breaking News

Finally, we release test reports that are centered on a series of 5 - 15 image comparisons of multiple cameras head to head; again, like so:

The best way to understand relative performance is to properly setup and test simultaneously. Unlike looking at cameras individually, where it is easy to overestimate performance, the IPVM approach makes it clear which cameras have advantages in specific scenarios.

Bandwidth / Quantization / Compression

We typically measure bandwidth levels for each camera in each scenario. Some camera manufacturers attempt to game quantization / compression levels. We will measure those levels using a DRF analysis tool that allows us to understand if there is significant variances. This is especially important for manufacturers claiming to deliver super low bandwidth who typically do so by increasing compression levels but failing to disclose. We will report this and make adjustments in testing as appropriate.

Test Reports

See our directory of 160+ surveillance test reports. Recent favorites include our WDR Shootout, Panoramic Shootout and Axis vs Bosch Low Light Shootout.

Feel free to ask questions and we are happy to expand on this reference document.

Comments (10) : Members only. Login. or Join.

Related Reports

Camera Course Summer 2020 - Last Chance on Jul 18, 2020
This is your last chance to register for the Summer 2020 Camera Course. This...
New Axis M30 Cameras Tested on Mar 26, 2020
Axis has released a new generation of, for them, relatively low cost M30...
Video Analytics 101 on Mar 16, 2020
This guide teaches the fundamentals of video surveillance...
Video Surveillance Cameras 101 on Feb 25, 2020
Cameras come in many shapes, sizes and specifications. This 101 examines the...
Seek Scan Thermal Temperature Screening System Tested on May 28, 2020
Now that IPVM has tested Dahua, Hikvision, and Sunell, we are returning to...
Video Surveillance 101 Book Released on Jul 07, 2020
IPVM's unique introduction to video surveillance series is now available as a...
"Fever Camera" Online Show June 2020 - On-Demand Recordings on Jun 03, 2020
IPVM has successfully completed the world's first "Fever Camera" show....
FDA Defines Correct Operation of "Fever Cameras" on May 26, 2020
The US FDA has now defined the correct operation of "Thermal Imaging...
Multi-Imagers Added to IPVM Camera Calculator on Jun 15, 2020
IPVM has added multi-imager cameras to the Camera Calculator, supporting this...
Hanwha Presents AI Analytics and First 8K Camera on May 11, 2020
Hanwha presented their AI Analytics and the "Industry's First 8K" camera at...
ISS Presents Face As A Credential and UVSS on Apr 30, 2020
ISS presented its security platform, including access control integration,...
Vivotek Presents AI Analytics and LPR on May 19, 2020
Vivotek presented its AI Analytics and LPR at the April 2020 IPVM New...
Trueface Presents AI Face Recognition, Mask and Temperature Detection on Jun 10, 2020
Trueface presented its AI facial recognition, mask and temperature detection...
Video Surveillance History on May 06, 2020
The video surveillance market has changed significantly since 2000, going...
Phone Camera Calculator Released on Mar 10, 2020
IPVM has released the first-ever Phone Calculator, video surveillance design...

Recent Reports

Dangerous Hikvision Fever Camera Showcased by Chilean City on Aug 07, 2020
Deploying a fever camera outdoors, in the rain, with no black body, is...
"Grand Slam" For Pelco's PE Firm, A Risk For Motorola on Aug 07, 2020
The word "Pelco" and "grand slam" have not been said together for many years....
FLIR Stock Falls, Admits 'Decelerating' Demand For Temperature Screening on Aug 07, 2020
Is the boom going to bust for temperature screening? FLIR disappointed...
VSaaS Will Hurt Integrators on Aug 06, 2020
VSaaS will hurt integrators, there is no question about that. How much...
Dogs For Coronavirus Screening Examined on Aug 06, 2020
While thermal temperature screening is the surveillance industry's most...
ADT Slides Back, Disappointing Results, Poor Commercial Performance on Aug 06, 2020
While ADT had an incredible start to the week, driven by the Google...
AHJ / Authority Having Jurisdiction Tutorial on Aug 06, 2020
One of the most powerful yet often underappreciated characters in all of the...
SIA Coaches Sellers on NDAA 889B Blacklist Workarounds on Aug 05, 2020
Last month SIA demanded that NDAA 899B "must be delayed". Now that they have...
ADI Returns To Growth, Back To 'Pre-COVID Levels' on Aug 05, 2020
While ADI was hit hard in April, with revenue declining 21%, the company's...
Exposing Fever Tablet Suppliers and 40+ Relabelers on Aug 05, 2020
IPVM has found 40+ USA and EU companies relabeling fever tablets designed,...
Indian Government Restricts PRC Manufacturers From Public Projects on Aug 04, 2020
In a move that mirrors the U.S. government’s ban on Dahua and Hikvision...
Directory of 201 "Fever" Camera Suppliers on Aug 04, 2020
This directory provides a list of "Fever" scanning thermal camera providers...
Face Masks Increase Face Recognition Errors Says NIST on Aug 04, 2020
COVID-19 has led to widespread facemask use, which as IPVM testing has shown...
Dahua Loses Australian Medical Device Approval on Aug 04, 2020
Dahua has cancelled its medical device registration after "discussions" with...
Google Invests in ADT, ADT Stock Soars on Aug 03, 2020
Google has announced a $450 million investment in the Florida-based security...