IPVM Camera Testing Procedures

Author: John Honovich, Published on May 08, 2013

The foundation of IPVM's camera testing procedure is simultaneous, head to head comparisons, like so:

This is unique among surveillance industry publications.

Each camera is placed side by side, with the same FoV, connected to the same VMS and recorded at the same time and place. The setup looks like so:

We perform a few adjustments:

  • Firmware is upgraded to the most current version.
  • Minimum shutter speed is set to 1/30s to eliminate any slow shutter tricks.
  • Cameras are re-focused prior to testing.

However, unlike manufacturer demos, we do not 'tweak' cameras to optimize for individual scenes. Rather, we keep the same settings for all tests, to keep it fair, consistent and true to the most common real world uses.

We conduct tests in 3 fundamental areas:

  • Even light - this is the 'ideal' test when all cameras do the best.
  • Low light - measured at 0.2 lux
  • WDR - in an open door way back light

They look like so:

Get Video Surveillance News In Your Inbox
Get Video Surveillance News In Your Inbox

We typically run a series of pre-tests to get a feel for performance and identify any abnormalities. If the image quality looks excessively poor, we will re-check settings and/or request feedback from a manufacturer engineer. The multiple pre-test runs works out issues and increases the level of confidence that the test results accurately and fairly reflect performance.

Finally, we release test reports that are centered on a series of 5 - 15 image comparisons of multiple cameras head to head; again, like so:

The best way to understand relative performance is to properly setup and test simultaneously. Unlike looking at cameras individually, where it is easy to overestimate performance, the IPVM approach makes it clear which cameras have advantages in specific scenarios.

Bandwidth / Quantization / Compression

We typically measure bandwidth levels for each camera in each scenario. Some camera manufacturers attempt to game quantization / compression levels. We will measure those levels using a DRF analysis tool that allows us to understand if there is significant variances. This is especially important for manufacturers claiming to deliver super low bandwidth who typically do so by increasing compression levels but failing to disclose. We will report this and make adjustments in testing as appropriate.

Test Reports

See our directory of 160+ surveillance test reports. Recent favorites include our WDR Shootout, Panoramic Shootout and Axis vs Bosch Low Light Shootout.

Feel free to ask questions and we are happy to expand on this reference document.

Comments (11): PRO Members only. Login. or Join.

Related Reports

IP Networking Course May 2017 on Apr 26, 2017
NOTE: Registration ends tomorrow, Thursday the 27th. This is the only networking course designed specifically for video surveillance professionals...
Axis Lowest Cost Outdoor IR Camera M2025-LE Tested on Apr 24, 2017
Axis has lagged offering low cost IR cameras while their Asian competitors have made IR standard even in their most entry level cameras. Recently,...
Hanwha Lowest Cost WiseNet X Camera (XND-6010) Tested on Apr 13, 2017
IPVM bought and tested the lowest-cost WiseNet X model, the indoor 2MP fixed focal XND-6010 dome, continuing our testing of Hanwha's new WiseNet X...
Pelco Surevision 3 Tested on Apr 12, 2017
Pelco has released generation 3 of its Surevision cameras, claiming improved performance in both super low light and WDR performance. We bought...
Burglar Alarm Strobes Guide on Mar 31, 2017
Strobes provide visual notification of alarm incidents, as sirens are used to give audible notification. Using a strobe gives alarm users and alarm...
Hanwha Wisenet X Tested on Mar 27, 2017
Hanwha has released their latest generation, the Wisenet X series, powered by their new Wisenet 5 processor. This new series claims improvements...
Burglar Alarm Sirens Guide on Mar 27, 2017
Sirens are used to alert users to an alarm condition. In this note, we examine how to choose, locate, and install alarm sirens, including Siren...
Glass Break Sensor Tutorial on Mar 21, 2017
Burglars often break glass windows to get into a house. Using glass break detectors in conjunction with alarm contacts is a good way to protect the...
Alarm Panic Switches Tutorial on Mar 16, 2017
Panic switches allow silently triggering an alarm system when it is otherwise disarmed. In this tutorial we explain and contrast the 7 most common...
Environmental Sensors For Burglar Alarm Tutorials on Mar 15, 2017
Intrusion detection systems can be used to alert users to environmental issues. By connecting sensors that monitor the heat, humidity, and...

Most Recent Industry Reports

IP Networking Course May 2017 on Apr 26, 2017
NOTE: Registration ends tomorrow, Thursday the 27th. This is the only networking course designed specifically for video surveillance professionals...
Tri-Ed Favorability Results on Apr 25, 2017
Tri-Ed, owned by Anixter, far outranked Anixter, the lowest ranked company in our distributor favorability series. Still, Anixter's ownership did...
Chinese 'Attacking Us From Every Direction', Says US FBI on Apr 25, 2017
"Chinese eating our lunch. Attacking us from every direction" said the US FBI's Deputy Director Andrew McCabe at the ASIS 2017 CSO Summit. .@FBI...
Eagle Eye Exec On Mountain Of Servers - VSaaS Growth Analysis on Apr 25, 2017
Eagle Eye VP of Operations, Hans Kahler, posted a picture of himself sitting on top of a shipment of new servers, as a testament to the companies...
Axis Posts Strong Q1 2017 Financial Results on Apr 24, 2017
Axis posted strong numbers for Q1 2017, after having some challenges in 2016 (Q1 2016, Q3 2016). Inventory levels and overall spending show...
Axis Lowest Cost Outdoor IR Camera M2025-LE Tested on Apr 24, 2017
Axis has lagged offering low cost IR cameras while their Asian competitors have made IR standard even in their most entry level cameras. Recently,...
IPVM First Dean's List W2017 - Thomas Atkinson, Matt Hurly and Fredrik Lundqvist on Apr 24, 2017
IPVM is happy to congratulate and celebrate our first "Dean's List", the top students in our courses. For the Winter 2017 IP Networking course...
Splicing Alarm Circuits Guide on Apr 24, 2017
Alarm installers commonly connect multiple sensors to a single zone. They do this by splicing the wires together. In this report, we will explain...
PureTech Video Analytics Examined on Apr 21, 2017
PureTech's analytics were chosen for a US border protection system (see related post), which the company claims no other analytics vendor was able...
US Border RVSS / Video Analytics System Examined on Apr 21, 2017
US Customs and Border Protection has been rolling out a video analytics-based detection system along the US/Mexico border, with detection ranges...

The world's leading video surveillance information source, IPVM provides the best reporting, testing and training for 10,000+ members globally. Dedicated to independent and objective information, we uniquely refuse any and all advertisements, sponsorship and consulting from manufacturers.

About | FAQ | Contact