IPVM Camera Testing Procedures

By John Honovich, Published on May 08, 2013

The foundation of IPVM's camera testing procedure is simultaneous, head to head comparisons, like so:

This is unique among surveillance industry publications.

Each camera is placed side by side, with the same FoV, connected to the same VMS and recorded at the same time and place. The setup looks like so:

We perform a few adjustments:

  • Firmware is upgraded to the most current version.
  • Minimum shutter speed is set to 1/30s to eliminate any slow shutter tricks.
  • Cameras are re-focused prior to testing.

However, unlike manufacturer demos, we do not 'tweak' cameras to optimize for individual scenes. Rather, we keep the same settings for all tests, to keep it fair, consistent and true to the most common real world uses.

We conduct tests in 3 fundamental areas:

  • Even light - this is the 'ideal' test when all cameras do the best.
  • Low light - measured at 0.2 lux
  • WDR - in an open door way back light

They look like so:

We typically run a series of pre-tests to get a feel for performance and identify any abnormalities. If the image quality looks excessively poor, we will re-check settings and/or request feedback from a manufacturer engineer. The multiple pre-test runs works out issues and increases the level of confidence that the test results accurately and fairly reflect performance.

Get Notified of Video Surveillance Breaking News
Get Notified of Video Surveillance Breaking News

Finally, we release test reports that are centered on a series of 5 - 15 image comparisons of multiple cameras head to head; again, like so:

The best way to understand relative performance is to properly setup and test simultaneously. Unlike looking at cameras individually, where it is easy to overestimate performance, the IPVM approach makes it clear which cameras have advantages in specific scenarios.

Bandwidth / Quantization / Compression

We typically measure bandwidth levels for each camera in each scenario. Some camera manufacturers attempt to game quantization / compression levels. We will measure those levels using a DRF analysis tool that allows us to understand if there is significant variances. This is especially important for manufacturers claiming to deliver super low bandwidth who typically do so by increasing compression levels but failing to disclose. We will report this and make adjustments in testing as appropriate.

Test Reports

See our directory of 160+ surveillance test reports. Recent favorites include our WDR Shootout, Panoramic Shootout and Axis vs Bosch Low Light Shootout.

Feel free to ask questions and we are happy to expand on this reference document.

Comments (10) : Members only. Login. or Join.

Related Reports

Hanwha Presents AI Analytics and First 8K Camera on May 11, 2020
Hanwha presented their AI Analytics and the "Industry's First 8K" camera at...
Multi-Imagers Added to IPVM Camera Calculator on Jun 15, 2020
IPVM has added multi-imager cameras to the Camera Calculator, supporting this...
Camera Course Summer 2020 - Last Chance on Jul 18, 2020
This is your last chance to register for the Summer 2020 Camera Course. This...
"Fever Camera" Online Show June 2020 - On-Demand Recordings on Jun 03, 2020
IPVM has successfully completed the world's first "Fever Camera" show....
New Axis M30 Cameras Tested on Mar 26, 2020
Axis has released a new generation of, for them, relatively low cost M30...
AndroVideo Presents Edge AI Face Recognition Cameras on Jun 26, 2020
AndroVideo presented its AI at the edge face recognition cameras at the May...
Video Surveillance Cameras 101 on Feb 25, 2020
Cameras come in many shapes, sizes and specifications. This 101 examines the...
Seek Scan Thermal Temperature Screening System Tested on May 28, 2020
Now that IPVM has tested Dahua, Hikvision, and Sunell, we are returning to...
2020 Camera Book Released on Jan 10, 2020
This is the best, most comprehensive security camera training in the world,...
IPVM On-Demand Courses on Mar 24, 2020
For nearly a decade, IPVM has been a leader in online live courses. Now, we...
Video Surveillance 101 Book Released on Jul 07, 2020
IPVM's unique introduction to video surveillance series is now available as a...
Video Surveillance 101 November Course - Last Chance on Sep 16, 2020
Today is the last chance to sign up for the November Video Surveillance 101...
FLIR Presents Dual Spectrum High Security Focused PTZ on May 01, 2020
FLIR presented its Elara DX-Series bispectral, visible and thermal, PTZ...
FLIR Screen-EST Screening Software Tested on Jun 30, 2020
In our FLIR A Series Test, the cameras' biggest drawback was their lack of...
FaceFirst Problems And Layoff on Oct 01, 2020
FaceFirst, a US company and one of the oldest ongoing facial recognition...

Recent Reports

Recruiters Online Show LIVE Today! on Oct 29, 2020
IPVM's 7th online show resumes today with 12 recruiters presenting themselves...
Hikvision AcuSense G2 Camera Test on Oct 29, 2020
Hikvision has released their next generation of AcuSense analytic cameras...
Biggest Problems Selling Access Control 2020 on Oct 29, 2020
Access control can cause integrators big headaches. What practical issues do...
Taiwan Geovision AI Analytics and NDAA Examined on Oct 29, 2020
Taiwan manufacturer Geovision's revenue has been falling for years. However,...
Bedside Cough and Sneeze Detector (Sound Intelligence and CLB) on Oct 28, 2020
Coronavirus has increased interest in detecting symptoms such as fever and...
Fever Tablet Thermal Sensors Examined (Melexis) on Oct 28, 2020
Fever tablet suppliers heavily rely on the accuracy and specs of...
Verkada Fires 3 on Oct 28, 2020
Verkada has fired three employees over an incident where female colleagues...
Eagle Eye Networks Raises $40 Million on Oct 27, 2020
Eagle Eye has raised $40 million aiming to "reinvent video...
Hikvision Q3 2020 Global Revenue Rises, US Revenue Falls on Oct 27, 2020
While Hikvision's global revenue rises driven by domestic recovery, its US...
VICE Investigates Verkada's Harassing "RawVerkadawgz" on Oct 26, 2020
This month, IPVM investigated Verkada's sexism, discrimination, and cultural...
Six Flags' FDA Violating Outdoor Dahua Fever Cameras on Oct 26, 2020
As Six Flags scrambled to reopen parks amid plummeting revenues caused by the...
ISC Brasil Digital Experience 2020 Report on Oct 23, 2020
ISC Brasil 2020 rebranded itself to ISC Digital Experience and, like its...
Top Video Surveillance Service Call Problems 2020 on Oct 23, 2020
3 primary and 4 secondary issues stood out as causing the most problems when...
GDPR Impact On Temperature / Fever Screening Explained on Oct 22, 2020
What impact does GDPR have on temperature screening? Do you risk a GDPR fine...
Security And Safety Things (S&ST) Tested on Oct 22, 2020
S&ST, a Bosch spinout, is spending tens of millions of dollars aiming to...