IPVM Camera Testing Procedures

Author: John Honovich, Published on May 08, 2013

The foundation of IPVM's camera testing procedure is simultaneous, head to head comparisons, like so:

This is unique among surveillance industry publications.

Each camera is placed side by side, with the same FoV, connected to the same VMS and recorded at the same time and place. The setup looks like so:

We perform a few adjustments:

  • Firmware is upgraded to the most current version.
  • Minimum shutter speed is set to 1/30s to eliminate any slow shutter tricks.
  • Cameras are re-focused prior to testing.

However, unlike manufacturer demos, we do not 'tweak' cameras to optimize for individual scenes. Rather, we keep the same settings for all tests, to keep it fair, consistent and true to the most common real world uses.

We conduct tests in 3 fundamental areas:

  • Even light - this is the 'ideal' test when all cameras do the best.
  • Low light - measured at 0.2 lux
  • WDR - in an open door way back light

They look like so:

Get Video Surveillance News In Your Inbox
Get Video Surveillance News In Your Inbox

We typically run a series of pre-tests to get a feel for performance and identify any abnormalities. If the image quality looks excessively poor, we will re-check settings and/or request feedback from a manufacturer engineer. The multiple pre-test runs works out issues and increases the level of confidence that the test results accurately and fairly reflect performance.

Finally, we release test reports that are centered on a series of 5 - 15 image comparisons of multiple cameras head to head; again, like so:

The best way to understand relative performance is to properly setup and test simultaneously. Unlike looking at cameras individually, where it is easy to overestimate performance, the IPVM approach makes it clear which cameras have advantages in specific scenarios.

Bandwidth / Quantization / Compression

We typically measure bandwidth levels for each camera in each scenario. Some camera manufacturers attempt to game quantization / compression levels. We will measure those levels using a DRF analysis tool that allows us to understand if there is significant variances. This is especially important for manufacturers claiming to deliver super low bandwidth who typically do so by increasing compression levels but failing to disclose. We will report this and make adjustments in testing as appropriate.

Test Reports

See our directory of 160+ surveillance test reports. Recent favorites include our WDR Shootout, Panoramic Shootout and Axis vs Bosch Low Light Shootout.

Feel free to ask questions and we are happy to expand on this reference document.

Comments (11): PRO Members only. Login. or Join.

Related Reports

Last Chance February 2018 Camera Course on Feb 15, 2018
This is the last chance to get into the Winter camera course, starts next Tuesday. Register now. IPVM provides the best education, live online...
Hanwha Wisenet X 5MP Camera Tested (XNV-8080R) on Feb 13, 2018
Wisenet X is Hanwha's high-end camera line. We tested their Wisenet X 1080p camera last year. Now Hanwha is offering 5MP cameras listing super low...
Axis 'Best' Q3517-LVE 5MP Camera Tested on Feb 05, 2018
Axis boasts of its Q3517-LVE: Simply put, AXIS Q3517-LVE delivers the best video quality possible. But is it really the 'best'? We bought...
Dahua 5MP Starlight Camera Tested (N52BM3Z) on Jan 30, 2018
Is 5MP the new 1080p? According to our recent statistics, average resolution continues to trend upwards. And now, manufacturers releasing new...
Hanwha Wave VMS Tested on Jan 22, 2018
Hanwha has released their first open platform VMS, Wisenet Wave, an Network Optix OEM (see test results) enhanced with integrations and...
Resolution Usage Statistics 2018 - Moving Up From 1080p on Jan 22, 2018
In 2016, IPVM statistics showed the most common camera resolution used was 1080p, rising from 2014's 720p. Now, new IPVM statistics of 200+...
VSaaS Usage Statistics 2018 on Jan 18, 2018
VSaaS has been a 'next big thing' for more than a decade. The prospect of managing, storing and streaming video from the cloud rather than...
Multicasting Surveillance Tutorial on Jan 04, 2018
Network bandwidth can be a concern for some surveillance systems. While improvements in video codecs, such as smart codecs for H.264 and H.265,...
Cabling Best Practices Guide on Jan 03, 2018
Surveillance cabling can be a major problem. Poorly installed and maintained networks are often costly, lengthy, frustrating ordeals to...
2018 IP Networking Book Released on Jan 03, 2018
The new IP Networking Book 2018 is a 228-page in-depth guide that teaches you how IT and telecom technologies impact modern security...

Most Recent Industry Reports

Visio For Video Surveillance Design on Feb 20, 2018
Many integrators have standardized on AutoCAD for camera layouts but new users may be overwhelmed by its learning curve. Microsoft's Visio...
Health Care Insurance Integrator Benefits Statistics on Feb 20, 2018
How common and how much healthcare coverage is typically provided by security companies? 150+ integrators explained how their companies provide the...
Hikvision Deletes Genetec Support on Feb 20, 2018
There will be no peace between Hikvision and Genetec. A year after Genetec expelled Hikvision (and Huawei, citing Chinese government control...
Change Orders - Sometimes Necessary, Sometimes Unethical on Feb 19, 2018
Change orders are a common element in project sales. Sometimes they are a necessity and appropriate ways to deal with arising issues, but sometimes...
Bosch Merges Video, Intrusion and Access Businesses on Feb 19, 2018
Bosch is merging their "video systems, intrusion detection, as well as its access control and management software business units to form a single...
Why 3VR Failed on Feb 16, 2018
3VR destroyed transformed ~$65 million in VC funding into a $6.9 million exit. The reason they failed is simple. They bet on analytics. They...
"Fear Mongering": Hikvision USA Cybersecurity Director Dismisses Chinese Government Ownership Concerns on Feb 16, 2018
The facts are: The Chinese government created Hikvision and is Hikvision's controlling shareholder. Hikvision's Chairman, a Communist Party...
16:9 vs 4:3 Video Aspect Ratio Statistics on Feb 16, 2018
What aspect ratio do security integrators prefer? The 'standard' 4:3 or the 'wide' 16:9 one? 100+ integrators told us what they preferred, with...
Mercury Releases New Series 3 Redboard Access Panels on Feb 15, 2018
Mercury Security has their first major product release post-HID buyout, and things literally look different. The Series 3 SIO boards now are red...
Last Chance February 2018 Camera Course on Feb 15, 2018
This is the last chance to get into the Winter camera course, starts next Tuesday. Register now. IPVM provides the best education, live online...

The world's leading video surveillance information source, IPVM provides the best reporting, testing and training for 10,000+ members globally. Dedicated to independent and objective information, we uniquely refuse any and all advertisements, sponsorship and consulting from manufacturers.

About | FAQ | Contact