IPVM Camera Testing Procedures

Author: John Honovich, Published on May 08, 2013

The foundation of IPVM's camera testing procedure is simultaneous, head to head comparisons, like so:

This is unique among surveillance industry publications.

Each camera is placed side by side, with the same FoV, connected to the same VMS and recorded at the same time and place. The setup looks like so:

We perform a few adjustments:

  • Firmware is upgraded to the most current version.
  • Minimum shutter speed is set to 1/30s to eliminate any slow shutter tricks.
  • Cameras are re-focused prior to testing.

However, unlike manufacturer demos, we do not 'tweak' cameras to optimize for individual scenes. Rather, we keep the same settings for all tests, to keep it fair, consistent and true to the most common real world uses.

We conduct tests in 3 fundamental areas:

  • Even light - this is the 'ideal' test when all cameras do the best.
  • Low light - measured at 0.2 lux
  • WDR - in an open door way back light

They look like so:

Get Video Surveillance News In Your Inbox
Get Video Surveillance News In Your Inbox

We typically run a series of pre-tests to get a feel for performance and identify any abnormalities. If the image quality looks excessively poor, we will re-check settings and/or request feedback from a manufacturer engineer. The multiple pre-test runs works out issues and increases the level of confidence that the test results accurately and fairly reflect performance.

Finally, we release test reports that are centered on a series of 5 - 15 image comparisons of multiple cameras head to head; again, like so:

The best way to understand relative performance is to properly setup and test simultaneously. Unlike looking at cameras individually, where it is easy to overestimate performance, the IPVM approach makes it clear which cameras have advantages in specific scenarios.

Bandwidth / Quantization / Compression

We typically measure bandwidth levels for each camera in each scenario. Some camera manufacturers attempt to game quantization / compression levels. We will measure those levels using a DRF analysis tool that allows us to understand if there is significant variances. This is especially important for manufacturers claiming to deliver super low bandwidth who typically do so by increasing compression levels but failing to disclose. We will report this and make adjustments in testing as appropriate.

Test Reports

See our directory of 160+ surveillance test reports. Recent favorites include our WDR Shootout, Panoramic Shootout and Axis vs Bosch Low Light Shootout.

Feel free to ask questions and we are happy to expand on this reference document.

Comments (11): PRO Members only. Login. or Join.

Related Reports

VLANs for Video Surveillance on Sep 26, 2016
Many people confidently say to 'use VLANs' as an answer to IP video networking problems and as a way to signal expertise. But how should VLANs be...
History of Video Surveillance on Sep 22, 2016
This is a concise history of video surveillance covering the past decade.  The goal is to help professionals newer to the industry understand...
Door Fundamentals For Electronic Access Control on Sep 20, 2016
Assuming every door can be secured with either a maglock or an electric strike can be a painful assumption in the field. While those items can be...
How to Measure Video Quality / Compression Levels on Sep 16, 2016
Two cameras have the same resolution, frame rate and scene monitored but camera A consumes half the bandwidth than camera B. Is Camera A better?...
Camera Course September 2016 on Sep 15, 2016
This is the only independent surveillance camera course, based on in-depth product and technology testing. Lots of manufacturer training exists...
PTZ Camera Guide 2016 on Sep 15, 2016
Pan/Tilt/Zoom cameras remain critical parts of many surveillance systems, especially in large security operations. Because of this it is important...
Milestone VMS Adds H.265, SVQR, RAM Video Optimization on Sep 09, 2016
Milestone is rolling out enhancements to XProtect to support H.265, enhanced edge recording functionality, and potentially allow users to reduce...
Hikvision 4K Camera Tested on Sep 09, 2016
Hikvision is the most common choice for low price entry level products but they are also competing with low light models, smart CODECs, WDR...
Pelco Optera 270° Camera Tested on Sep 06, 2016
Multi-imager cameras are typically 180° or 360°. Pelco has released a fixed 270° versions of their Optera intended to cover exterior building...
Panoramic Camera (Fisheye / Multi-Imager) Guide on Aug 31, 2016
Panoramic cameras, including fisheyes and multi-imagers, have become increasingly widespread, with most manufacturers offering fisheyes and an...

Most Recent Industry Reports

Axis Secretly Paid Anixter Sales People To Push Axis NVRs on Sep 26, 2016
Internal Axis communication shows how Axis paid Anixter and Tri-Ed sales people with secret bonuses to push Axis NVRs. In this report, we examine...
VLANs for Video Surveillance on Sep 26, 2016
Many people confidently say to 'use VLANs' as an answer to IP video networking problems and as a way to signal expertise. But how should VLANs be...
Ambarella CEO Admits H.265 and 4K Not Popular on Sep 26, 2016
Ambarella is the main chip provider for high-end surveillance cameras driving higher resolution and new CODECs. While Ambarella has been pushing...
Nest Cam Outdoor Tested on Sep 23, 2016
After years of claiming an outdoor model was "coming", addressing their biggest user demand, Nest has finally released their Outdoor Camera, an...
ACTi Refuses Race To The Bottom, Shifts To Solutions on Sep 23, 2016
The original low cost IP camera disruptor was ACTi. Back in the 2008 - 2010 time frame, Taiwanese manufacturer ACTi challenged the Western and...
You Get Robbed, Canary Will Pay You Up To $1,000 on Sep 22, 2016
Canary is trying to break the status quo in DIY security, first by raising over $40 million, and now a revamp of their monthly services package...
Milestone Ends Development of "Enterprise" VMS on Sep 22, 2016
Milestone 'Enterprise' was one of the first enterprise video management software offerings, selected by many early adopters of IP video. However,...
History of Video Surveillance on Sep 22, 2016
This is a concise history of video surveillance covering the past decade.  The goal is to help professionals newer to the industry understand...
Access Control Course Fall 2016 on Sep 22, 2016
IPVM offers the most comprehensive access control course in the industry. Unlike manufacturer training that focuses only on a small part of the...
Totally Wireless IP Camera (IPVideo Corp NomadHD) on Sep 21, 2016
Wireless battery powered cameras have been a surveillance pipe dream for years, limited by camera power consumption, battery technology, and...

The world's leading video surveillance information source, IPVM provides the best reporting, testing and training for 10,000+ members globally. Dedicated to independent and objective information, we uniquely refuse any and all advertisements, sponsorship and consulting from manufacturers.

About | FAQ | Contact