IPVM Camera Testing Procedures

Author: John Honovich, Published on May 08, 2013

The foundation of IPVM's camera testing procedure is simultaneous, head to head comparisons, like so:

This is unique among surveillance industry publications.

Each camera is placed side by side, with the same FoV, connected to the same VMS and recorded at the same time and place. The setup looks like so:

We perform a few adjustments:

  • Firmware is upgraded to the most current version.
  • Minimum shutter speed is set to 1/30s to eliminate any slow shutter tricks.
  • Cameras are re-focused prior to testing.

However, unlike manufacturer demos, we do not 'tweak' cameras to optimize for individual scenes. Rather, we keep the same settings for all tests, to keep it fair, consistent and true to the most common real world uses.

We conduct tests in 3 fundamental areas:

  • Even light - this is the 'ideal' test when all cameras do the best.
  • Low light - measured at 0.2 lux
  • WDR - in an open door way back light

They look like so:

Get Video Surveillance News In Your Inbox
Get Video Surveillance News In Your Inbox

We typically run a series of pre-tests to get a feel for performance and identify any abnormalities. If the image quality looks excessively poor, we will re-check settings and/or request feedback from a manufacturer engineer. The multiple pre-test runs works out issues and increases the level of confidence that the test results accurately and fairly reflect performance.

Finally, we release test reports that are centered on a series of 5 - 15 image comparisons of multiple cameras head to head; again, like so:

The best way to understand relative performance is to properly setup and test simultaneously. Unlike looking at cameras individually, where it is easy to overestimate performance, the IPVM approach makes it clear which cameras have advantages in specific scenarios.

Bandwidth / Quantization / Compression

We typically measure bandwidth levels for each camera in each scenario. Some camera manufacturers attempt to game quantization / compression levels. We will measure those levels using a DRF analysis tool that allows us to understand if there is significant variances. This is especially important for manufacturers claiming to deliver super low bandwidth who typically do so by increasing compression levels but failing to disclose. We will report this and make adjustments in testing as appropriate.

Test Reports

See our directory of 160+ surveillance test reports. Recent favorites include our WDR Shootout, Panoramic Shootout and Axis vs Bosch Low Light Shootout.

Feel free to ask questions and we are happy to expand on this reference document.

Comments (11): PRO Members only. Login. or Join.

Related Reports

IP Camera - 15 Year Shootout on May 22, 2017
How far have IP cameras come? We bought and tested 4 cameras across the past 15 years to understand how much and where performance has...
Selling and Valuing Security Integrators on May 12, 2017
This ia a tutorial in how to (1) determine your security integrator's value and (2) to sell your security integrator. If you own an integrator,...
48MP 180 Camera (Digital Watchdog) Test on May 10, 2017
Camera resolution continues to advance, with Digital Watchdog offering the MegaPIX PANO 48MP 180° camera, the highest resolution mainstream camera...
Hanwha 9MP Fisheye Camera Tested (PNF-9010R) on May 09, 2017
12MP sensor fisheye panoramic cameras are becoming increasingly common. We have tested Hikvision's DS-2CD63C2F as well as Panasonic's SFV481 4K...
Duress Codes For Alarms Systems on May 02, 2017
An alarm system can call for help in the event of an attempted break in, but only if it is armed. If an adversary forces an authorized user to...
Last Day - IP Networking Course May 2017 on Apr 26, 2017
Today is the last day to register for the May IP Networking Course. This is the only networking course designed specifically for video...
Axis Lowest Cost Outdoor IR Camera M2025-LE Tested on Apr 24, 2017
Axis has lagged offering low cost IR cameras while their Asian competitors have made IR standard even in their most entry level cameras. Recently,...
Hanwha Lowest Cost WiseNet X Camera (XND-6010) Tested on Apr 13, 2017
IPVM bought and tested the lowest-cost WiseNet X model, the indoor 2MP fixed focal XND-6010 dome, continuing our testing of Hanwha's new WiseNet X...
Pelco Surevision 3 Tested on Apr 12, 2017
Pelco has released generation 3 of its Surevision cameras, claiming improved performance in both super low light and WDR performance. We bought...
Burglar Alarm Strobes Guide on Mar 31, 2017
Strobes provide visual notification of alarm incidents, as sirens are used to give audible notification. Using a strobe gives alarm users and alarm...

Most Recent Industry Reports

Axis Criticizes OEMs: "When You Buy An Axis Camera, An Axis Camera Is What You Get!" on May 19, 2017
When you buy a Honeywell camera, you likely get a Hikvision, Dahua or some other company's product. The same goes for easily 100 different...
Hackable 125kHz Access Control Migration Guide on May 19, 2017
Despite being one of the most popular credentials, 125 kHz credentials are easily copied and insecure as we showed in our test results, video...
Forget The Backdoor, "ALL HIKVISION PRODUCTS" On Sale on May 18, 2017
Less than 2 weeks after the Hikvision Backdoor was confirmed, Hikvision has launched a sale "ON ALL HIKVISION PRODUCTS". In this note, we examine...
Amazon Techs Installing IP Cameras Tested on May 18, 2017
In 2015, Amazon started offering video surveillance installation. Now, Amazon has made it a lot easier, with automatic add-on options and...
Hanwha Recorder Vulnerability Analyzed on May 18, 2017
ICS-CERT has released a vulnerability notice for Hanwha SRN-4000 recorders.  Hanwha provided additional information to IPVM about this issue,...
ShotSpotter To IPO, Facing Low Revenue and Losses on May 17, 2017
A rare event for North American security manufacturers is upcoming. ShotSpotter is planning to IPO on the NASDAQ, aiming to raise $34.5...
DMP Video Doorbell / Access Reader Examined on May 17, 2017
Consumers increasingly demand video doorbells, with "doorbells selling like hotcakes, everyone wants a doorbell", according to ADT's CEO. At ISC...
FLIR Is Giving Away $3,000 Demo Kits on May 17, 2017
Everybody likes free stuff, and FLIR is using that concept to attract dealers by giving them $3,000 worth of demo gear for attending a...
Shark Tank Startup Guard Llama Mobile Panic Tested on May 16, 2017
A Shark Tank TV show appearance has attracted big attention for Guard Llama, a security startup touting a 'panic button' paired with mobile app...
Axis Beats Avigilon Growth Rate (AVO Q1 2017) on May 16, 2017
In what is likely the first time ever, Axis beat Avigilon's revenue growth for Q1 2017. Inside this note, we examine Avigilon's Q1 2017 financial...

The world's leading video surveillance information source, IPVM provides the best reporting, testing and training for 10,000+ members globally. Dedicated to independent and objective information, we uniquely refuse any and all advertisements, sponsorship and consulting from manufacturers.

About | FAQ | Contact