IPVM Camera Testing Procedures

Author: John Honovich, Published on May 08, 2013

The foundation of IPVM's camera testing procedure is simultaneous, head to head comparisons, like so:

This is unique among surveillance industry publications.

Each camera is placed side by side, with the same FoV, connected to the same VMS and recorded at the same time and place. The setup looks like so:

We perform a few adjustments:

  • Firmware is upgraded to the most current version.
  • Minimum shutter speed is set to 1/30s to eliminate any slow shutter tricks.
  • Cameras are re-focused prior to testing.

However, unlike manufacturer demos, we do not 'tweak' cameras to optimize for individual scenes. Rather, we keep the same settings for all tests, to keep it fair, consistent and true to the most common real world uses.

We conduct tests in 3 fundamental areas:

  • Even light - this is the 'ideal' test when all cameras do the best.
  • Low light - measured at 0.2 lux
  • WDR - in an open door way back light

They look like so:

Get Video Surveillance News In Your Inbox
Get Video Surveillance News In Your Inbox

We typically run a series of pre-tests to get a feel for performance and identify any abnormalities. If the image quality looks excessively poor, we will re-check settings and/or request feedback from a manufacturer engineer. The multiple pre-test runs works out issues and increases the level of confidence that the test results accurately and fairly reflect performance.

Finally, we release test reports that are centered on a series of 5 - 15 image comparisons of multiple cameras head to head; again, like so:

The best way to understand relative performance is to properly setup and test simultaneously. Unlike looking at cameras individually, where it is easy to overestimate performance, the IPVM approach makes it clear which cameras have advantages in specific scenarios.

Bandwidth / Quantization / Compression

We typically measure bandwidth levels for each camera in each scenario. Some camera manufacturers attempt to game quantization / compression levels. We will measure those levels using a DRF analysis tool that allows us to understand if there is significant variances. This is especially important for manufacturers claiming to deliver super low bandwidth who typically do so by increasing compression levels but failing to disclose. We will report this and make adjustments in testing as appropriate.

Test Reports

See our directory of 160+ surveillance test reports. Recent favorites include our WDR Shootout, Panoramic Shootout and Axis vs Bosch Low Light Shootout.

Feel free to ask questions and we are happy to expand on this reference document.

Comments (10) : PRO Members only. Login. or Join.

Related Reports

Camera Course Winter 2019 - Last Chance on Jan 24, 2019
Today, Thursday Jan 24th is the last chance to register for the Winter 2019 Camera Course. This is the only independent surveillance camera...
Testing Bandwidth vs. Frame Rate on Jan 23, 2019
Selecting frame rate has a major impact on surveillance bandwidth and storage consumption. But with smart codecs now common and cameras more...
Milesight Company Profile on Jan 22, 2019
Milesight Technology, a Chinese company building an International branded business, says they are slowly building their presence through a series...
Cable Trenching for Surveillance on Jan 21, 2019
Trenching cable for surveillance is surprisingly complex. While using shovels, picks, and hoes is not advanced technology, the proper planning,...
Testing Bandwidth Vs. Low Light on Jan 16, 2019
Nighttime bandwidth spikes are a major concern in video surveillance. Many calculate bandwidth as a single 24/7 number, but bit rates vary...
WDR Tutorial on Jan 11, 2019
Understanding wide dynamic range (WDR) is critical to capturing high quality images in demanding conditions. However, with no real standards, any...
Winter 2019 IP Networking Course on Jan 10, 2019
Today is the last day to register for the Winter 2019 IP Networking course. This is the only networking course designed specifically for video...
Worst Products Tested In Past Year on Jan 09, 2019
IPVM has done over 100 tests in the past year. But which products performed the worst? Which ones should users be most aware of? In this report,...
Managed Video Services UL 827B Examined on Jan 09, 2019
Historically, UL listings for central stations have been important, with UL 827 having widespread support. However, few central stations have...
H.265 / HEVC Codec Tutorial on Jan 08, 2019
H.265 support improved significantly in 2018, with H.265 camera/VMS compatibility increased compared to only a year ago, and most manufacturers...

Most Recent Industry Reports

Testing Bandwidth vs. Frame Rate on Jan 23, 2019
Selecting frame rate has a major impact on surveillance bandwidth and storage consumption. But with smart codecs now common and cameras more...
Camera Course January 2019 on Jan 23, 2019
This is the only independent surveillance camera course, based on in-depth product and technology testing. Lots of manufacturer training exists...
Bosch Favorability Results 2019 on Jan 23, 2019
Bosch's favorability moderately strengthed, in new IPVM integrator statistics over their results from 2017, with 2019 results showing strong net...
Intersec 2019 Show Report on Jan 23, 2019
The 2019 Intersec show, held annually in Dubai, is now complete. IPVM attended for 3 days, interviewing numerous Chinese and Western video...
2019 Camera Book Released on Jan 22, 2019
This is the best, most comprehensive security camera training in the world, based on our unprecedented testing. Now, all IPVM PRO Members can get...
Milesight Company Profile on Jan 22, 2019
Milesight Technology, a Chinese company building an International branded business, says they are slowly building their presence through a series...
Cable Trenching for Surveillance on Jan 21, 2019
Trenching cable for surveillance is surprisingly complex. While using shovels, picks, and hoes is not advanced technology, the proper planning,...
Milestone Favorability Results 2019 on Jan 21, 2019
Milestone's favorability moderately strengthed, in new IPVM integrator statistics over their results from 2016. While the industry has been...
The IP Camera Lock-In Trend: Meraki and Verkada on Jan 18, 2019
Open systems and interoperability have become core features of video surveillance systems, as virtually all professional IP cameras integrate with...
NYPD Refutes False SCMP Hikvision Story on Jan 18, 2019
The NYPD has refuted the SCMP Hikvision story, the Voice of America has reported. On January 11, 2018, the SCMP alleged that the NYPD was using...

The world's leading video surveillance information source, IPVM provides the best reporting, testing and training for 10,000+ members globally. Dedicated to independent and objective information, we uniquely refuse any and all advertisements, sponsorship and consulting from manufacturers.

About | FAQ | Contact