IPVM Camera Testing Procedures

By: John Honovich, Published on May 08, 2013

The foundation of IPVM's camera testing procedure is simultaneous, head to head comparisons, like so:

This is unique among surveillance industry publications.

Each camera is placed side by side, with the same FoV, connected to the same VMS and recorded at the same time and place. The setup looks like so:

We perform a few adjustments:

  • Firmware is upgraded to the most current version.
  • Minimum shutter speed is set to 1/30s to eliminate any slow shutter tricks.
  • Cameras are re-focused prior to testing.

However, unlike manufacturer demos, we do not 'tweak' cameras to optimize for individual scenes. Rather, we keep the same settings for all tests, to keep it fair, consistent and true to the most common real world uses.

We conduct tests in 3 fundamental areas:

  • Even light - this is the 'ideal' test when all cameras do the best.
  • Low light - measured at 0.2 lux
  • WDR - in an open door way back light

They look like so:

Get Notified of Video Surveillance Breaking News
Get Notified of Video Surveillance Breaking News

We typically run a series of pre-tests to get a feel for performance and identify any abnormalities. If the image quality looks excessively poor, we will re-check settings and/or request feedback from a manufacturer engineer. The multiple pre-test runs works out issues and increases the level of confidence that the test results accurately and fairly reflect performance.

Finally, we release test reports that are centered on a series of 5 - 15 image comparisons of multiple cameras head to head; again, like so:

The best way to understand relative performance is to properly setup and test simultaneously. Unlike looking at cameras individually, where it is easy to overestimate performance, the IPVM approach makes it clear which cameras have advantages in specific scenarios.

Bandwidth / Quantization / Compression

We typically measure bandwidth levels for each camera in each scenario. Some camera manufacturers attempt to game quantization / compression levels. We will measure those levels using a DRF analysis tool that allows us to understand if there is significant variances. This is especially important for manufacturers claiming to deliver super low bandwidth who typically do so by increasing compression levels but failing to disclose. We will report this and make adjustments in testing as appropriate.

Test Reports

See our directory of 160+ surveillance test reports. Recent favorites include our WDR Shootout, Panoramic Shootout and Axis vs Bosch Low Light Shootout.

Feel free to ask questions and we are happy to expand on this reference document.

Comments (10) : PRO Members only. Login. or Join.

Related Reports

Anyvision Facial Recognition Tested on Aug 21, 2019
Anyvision is aiming for $1 billion in revenue by 2022, backed by $74 million in funding. But does their performance live up to the hype they have...
Dahua 4K Camera Shootout on Aug 20, 2019
Dahua's new Pro Series 4K N85CL5Z claims to "deliver superior images in all lighting and environmental conditions", but how does this compare to...
Axis 4K Camera Shootout 2019 on Aug 14, 2019
Axis' 4K Q3518-LVE claims the "best video quality possible", with Lightfinder super low light performance, Axis' high end Forensic WDR, and...
Proactive CCTV "Only Affordable Video Archiving Solution" Profile on Aug 12, 2019
Proactive CCTV is claiming to offer "the only affordable video archiving solution on the market", reducing the storage typically required for H.265...
Axis Door Station A8207-VE Tested on Aug 07, 2019
Axis newest door station, the A8207-VE, claims to deliver "video surveillance, two-way communication, and access control" in a single device. But...
Hikvision 4K Camera Shootout on Aug 02, 2019
With their latest Smart Series 5 cameras, Hikvision is claiming cameras "fully loaded" with "state-of-the-art technology for high performance and...
Hanwha 4K Wisenet P Camera Tested on Jul 30, 2019
Hanwha claims "premium performance" and to "deliver the best in high resolution performance and efficiency" with their Wisenet P series 4K models,...
Bosch Starlight 8000i Cameras Tested on Jul 23, 2019
Bosch has released their new Flexidome IP Starlight 8000i cameras, claiming "exceptional detail even in extreme low-light situations." To see...
Avigilon ACC7 VMS Tested on Jul 22, 2019
Avigilon's Control Center 7 boldly claims it will "transform live video monitoring" with the new Focus of Attention "AI-enabled" interface. We...
History of Video Surveillance on Jul 19, 2019
The video surveillance market has changed significantly since 2000, going from VCRs to ab emerging AI cloud era.  The goal of this history is to...

Most Recent Industry Reports

TMA Apologizes to Amazon / Ring on Aug 23, 2019
Not only is Amazon / Ring making major incursions into the residential security market, the organization representing the biggest incumbents, The...
China Dahua Replaces Their Software With US Pepper on Aug 22, 2019
What does a US government banned company do to improve its security positioning in the US? Well, Dahua is unveiling a novel solution, partnering...
Security Integrators Outlook On Remaining Integrators In 2025 on Aug 22, 2019
The industry has changed substantially in the last decade, with the rise of IP cameras and the race to the bottom. Indeed, more changes may be...
First GDPR Facial Recognition Fine For Sweden School on Aug 22, 2019
A school in Sweden has been fined $20,000 for using facial recognition to keep attendance in what is Sweden's first GDPR fine. Notably, the fine is...
Anyvision Facial Recognition Tested on Aug 21, 2019
Anyvision is aiming for $1 billion in revenue by 2022, backed by $74 million in funding. But does their performance live up to the hype they have...
JCI Sues Wyze on Aug 21, 2019
The mega manufacturer / integrator JCI has sued the fast-growing $20 camera Seattle startup Wyze. Inside this note: Share the court...
Dahua 4K Camera Shootout on Aug 20, 2019
Dahua's new Pro Series 4K N85CL5Z claims to "deliver superior images in all lighting and environmental conditions", but how does this compare to...
ZK Teco Atlas Access Control Tested on Aug 20, 2019
Who needs access specialists? China-based ZKTeco claims its newest access panel 'makes it very easy for anyone to learn and install access control...
Uniview Beats Intel In Trademark Lawsuit on Aug 19, 2019
Uniview has won a long-running trademark lawsuit brought by Intel, with Beijing's highest court reversing an earlier Intel win, centered on...
Suprema Biometric Mass Leak Examined on Aug 19, 2019
While Suprema is rarely discussed even within the physical security market, the South Korean biometrics manufacturer made global news this past...

The world's leading video surveillance information source, IPVM provides the best reporting, testing and training for 10,000+ members globally. Dedicated to independent and objective information, we uniquely refuse any and all advertisements, sponsorship and consulting from manufacturers.

About | FAQ | Contact