IPVM Camera Testing Procedures

Author: John Honovich, Published on May 08, 2013

The foundation of IPVM's camera testing procedure is simultaneous, head to head comparisons, like so:

This is unique among surveillance industry publications.

Each camera is placed side by side, with the same FoV, connected to the same VMS and recorded at the same time and place. The setup looks like so:

We perform a few adjustments:

  • Firmware is upgraded to the most current version.
  • Minimum shutter speed is set to 1/30s to eliminate any slow shutter tricks.
  • Cameras are re-focused prior to testing.

However, unlike manufacturer demos, we do not 'tweak' cameras to optimize for individual scenes. Rather, we keep the same settings for all tests, to keep it fair, consistent and true to the most common real world uses.

We conduct tests in 3 fundamental areas:

  • Even light - this is the 'ideal' test when all cameras do the best.
  • Low light - measured at 0.2 lux
  • WDR - in an open door way back light

They look like so:

Get Video Surveillance News In Your Inbox
Get Video Surveillance News In Your Inbox

We typically run a series of pre-tests to get a feel for performance and identify any abnormalities. If the image quality looks excessively poor, we will re-check settings and/or request feedback from a manufacturer engineer. The multiple pre-test runs works out issues and increases the level of confidence that the test results accurately and fairly reflect performance.

Finally, we release test reports that are centered on a series of 5 - 15 image comparisons of multiple cameras head to head; again, like so:

The best way to understand relative performance is to properly setup and test simultaneously. Unlike looking at cameras individually, where it is easy to overestimate performance, the IPVM approach makes it clear which cameras have advantages in specific scenarios.

Bandwidth / Quantization / Compression

We typically measure bandwidth levels for each camera in each scenario. Some camera manufacturers attempt to game quantization / compression levels. We will measure those levels using a DRF analysis tool that allows us to understand if there is significant variances. This is especially important for manufacturers claiming to deliver super low bandwidth who typically do so by increasing compression levels but failing to disclose. We will report this and make adjustments in testing as appropriate.

Test Reports

See our directory of 160+ surveillance test reports. Recent favorites include our WDR Shootout, Panoramic Shootout and Axis vs Bosch Low Light Shootout.

Feel free to ask questions and we are happy to expand on this reference document.

Comments (11): PRO Members only. Login. or Join.

Related Reports

Winter 2018 Camera Course on Jan 18, 2018
Learn video surveillance and get certified. Register now. Save $50 on the course, ending this Thursday the 18th, plus get access to 2 class times...
VSaaS Usage Statistics 2018 on Jan 18, 2018
VSaaS has been a 'next big thing' for more than a decade. The prospect of managing, storing and streaming video from the cloud rather than...
Multicasting Surveillance Tutorial on Jan 04, 2018
Network bandwidth can be a concern for some surveillance systems. While improvements in video codecs, such as smart codecs for H.264 and H.265,...
Cabling Best Practices Guide on Jan 03, 2018
Surveillance cabling can be a major problem. Poorly installed and maintained networks are often costly, lengthy, frustrating ordeals to...
2018 IP Networking Book Released on Jan 03, 2018
The new IP Networking Book 2018 is a 228-page in-depth guide that teaches you how IT and telecom technologies impact modern security...
QoS for Video Surveillance on Dec 29, 2017
Along with VLANs, QoS is one of the most misunderstood topics in IP surveillance networks. Many purported "experts" claim it is required in any...
Resolution Tutorial on Dec 28, 2017
Understanding video surveillance resolution can be surprisingly difficult and complex. While the word 'resolution' seems self-explanatory, its use...
Repositionable Multi-Imager Camera Guide on Dec 22, 2017
Multi-imager usage has been growing strongly over the past few years. Now, a new segment of multi-imagers has taken off, with many manufacturers...
STP vs UTP for Video Surveillance Tutorial on Dec 20, 2017
For many video system designers, deciding which ethernet cabling to use is a quick decision: go with the cheapest.  However, this overlooks the...
Lock Status Monitoring Tutorial on Dec 18, 2017
Just because your doors look secure does not mean they are. Unless access systems are using lock status monitoring, the doors and areas they...

Most Recent Industry Reports

PoE Powered Access Control Tutorial on Jan 19, 2018
Powering access control with Power over Ethernet is becoming increasingly common.  However, access requires more power than cameras, and the...
If You Have 4 Cameras, You Can Throw Them Away, If You Have 400, They Throw You Away on Jan 19, 2018
Do users care about anything but price? Do user care about cybersecurity? Do users care about trusting their supplier? These have become...
Chinese Government Hikvision Surveillance System On US Government Network on Jan 18, 2018
Hikvision, the Chinese government-owned manufacturer, has publicly claimed that their products are running on a US government network. Moreover,...
Winter 2018 Camera Course on Jan 18, 2018
Learn video surveillance and get certified. Register now. Save $50 on the course, ending this Thursday the 18th, plus get access to 2 class times...
VSaaS Usage Statistics 2018 on Jan 18, 2018
VSaaS has been a 'next big thing' for more than a decade. The prospect of managing, storing and streaming video from the cloud rather than...
Vivint Streety Video Strengthens Door Knocking on Jan 17, 2018
Vivint is famous (or infamous depending on your perspective) for mastering large scale door to door selling. The company has skyrocketed from a...
Axis: "It’s A Question Of Trust And Who You Want To Be Associated With" on Jan 17, 2018
Who do you trust? Who do you want to be associated with? Axis is raising hard questions to start 2018. In this note, we examine these questions,...
Software House Vulnerability Allows Inside Attacker To Open Doors on Jan 17, 2018
A vulnerability in Software House IP-ACM modules allows an attacker to potentially unlock doors, or perform other actions, on affected systems....
'Defiant' Hikvision 'Strikes Back' At WSJ And US on Jan 16, 2018
The fight is on. Hikvision and their owner, the Chinese government, 'strikes back' against the Wall Street Journal and US politicians raising...
The 2018 Surveillance Industry Guide on Jan 16, 2018
The 300 page, 2018 Video Surveillance Industry Guide, covering the key events and the future of the video surveillance market, is now available,...

The world's leading video surveillance information source, IPVM provides the best reporting, testing and training for 10,000+ members globally. Dedicated to independent and objective information, we uniquely refuse any and all advertisements, sponsorship and consulting from manufacturers.

About | FAQ | Contact