IPVM Camera Testing Procedures

Author: John Honovich, Published on May 08, 2013

The foundation of IPVM's camera testing procedure is simultaneous, head to head comparisons, like so:

This is unique among surveillance industry publications.

Each camera is placed side by side, with the same FoV, connected to the same VMS and recorded at the same time and place. The setup looks like so:

We perform a few adjustments:

  • Firmware is upgraded to the most current version.
  • Minimum shutter speed is set to 1/30s to eliminate any slow shutter tricks.
  • Cameras are re-focused prior to testing.

However, unlike manufacturer demos, we do not 'tweak' cameras to optimize for individual scenes. Rather, we keep the same settings for all tests, to keep it fair, consistent and true to the most common real world uses.

We conduct tests in 3 fundamental areas:

  • Even light - this is the 'ideal' test when all cameras do the best.
  • Low light - measured at 0.2 lux
  • WDR - in an open door way back light

They look like so:

Get Video Surveillance News In Your Inbox
Get Video Surveillance News In Your Inbox

We typically run a series of pre-tests to get a feel for performance and identify any abnormalities. If the image quality looks excessively poor, we will re-check settings and/or request feedback from a manufacturer engineer. The multiple pre-test runs works out issues and increases the level of confidence that the test results accurately and fairly reflect performance.

Finally, we release test reports that are centered on a series of 5 - 15 image comparisons of multiple cameras head to head; again, like so:

The best way to understand relative performance is to properly setup and test simultaneously. Unlike looking at cameras individually, where it is easy to overestimate performance, the IPVM approach makes it clear which cameras have advantages in specific scenarios.

Bandwidth / Quantization / Compression

We typically measure bandwidth levels for each camera in each scenario. Some camera manufacturers attempt to game quantization / compression levels. We will measure those levels using a DRF analysis tool that allows us to understand if there is significant variances. This is especially important for manufacturers claiming to deliver super low bandwidth who typically do so by increasing compression levels but failing to disclose. We will report this and make adjustments in testing as appropriate.

Test Reports

See our directory of 160+ surveillance test reports. Recent favorites include our WDR Shootout, Panoramic Shootout and Axis vs Bosch Low Light Shootout.

Feel free to ask questions and we are happy to expand on this reference document.

Comments (10) : PRO Members only. Login. or Join.

Related Reports

Door Operators Access Control Tutorial on Apr 17, 2019
Doors equipped with door operators, specialty devices that automate opening and closing, tend to be quite complex. The mechanisms needed to...
Dahua Repositionable IR Multi-Imager Camera Tested on Apr 16, 2019
Dahua has released their first repositionable multi-imager camera, the Multi-Flex 4x2MP, claiming integrated IR, true WDR, and flexible...
Pole Mount Camera Installation Guide on Apr 11, 2019
Poles are a popular but challenging choice for deploying surveillance cameras outdoors. Poles are indispensable for putting cameras at the right...
Axis ARTPEC-7 Chip Release Examined on Apr 08, 2019
For years, Axis essentially de-promoted their own chips. Now, they are reversing course. Axis has announced ARTPEC-7, their latest chip, which...
Spring 2019 IP Networking Course- Register Now on Apr 04, 2019
Register now for the Spring 2019 IP Networking course here. Just $299 for the course. This is the only networking course designed specifically...
Axis 5MP P Vs Q Camera Shootout (P3227-LVE vs Q3517-LVE) on Apr 02, 2019
While Axis' Q3517-LVE tested well, it is one of Axis' most expensive dome cameras, over $1,000 USD. Because of this, members asked us how it...
Casino Security Consultant Carl Lindgren Interview on Mar 26, 2019
For more than 20 years, Carl Lindgren worked as a casino surveillance pro, while being active (and sometimes outspoken) on various online video...
Covert IP Camera Shootout - Axis, Hanwha, Hikvision, March, Vivotek on Mar 20, 2019
Covert cameras were one of the last holdout areas for analog cameras. However, in the past few years, IP / HD covert cameras have become...
Pelco GFC 4K Dome Camera Tested (IMP831-1ERS) on Mar 12, 2019
Pelco has finally released their first 4K IP camera, after years of competitors' releases. Is this move too late? Or is their new GFC Professional...
Bandwidth vs Low Light Shootout - Avigilon, Axis, Bosch, Dahua, Geovision, Hanwha, Hikvision, Uniview, Vivotek on Feb 08, 2019
Nighttime bandwidth spikes are a major concern in video surveillance, but do all manufacturers' cameras perform the same? Are some more consistent...

Most Recent Industry Reports

Access Control Course Spring 2019 - Last Chance on Apr 19, 2019
This is the last chance to register for the Spring Access Control Course. IPVM offers the most comprehensive access control course in the...
Riser vs Plenum Cabling Explained on Apr 18, 2019
You could be spending twice as much for cable as you need. The difference between 'plenum' rated cable and 'riser' rated cable is subtle, but the...
Verint Victimized By Ransomware on Apr 18, 2019
Verint, which is best known in the physical security industry for video surveillance but has built a sizeable cybersecurity business as well, was...
Milestone Drops IFSEC on Apr 18, 2019
Milestone has dropped out of Europe's largest annual security trade show (IFSEC 2019), telling IPVM that they "have found that IFSEC in EMEA no...
The Fastest Growing Video Surveillance Sales Organization Ever - Verkada on Apr 17, 2019
Verkada has the fastest growing video surveillance sales organization ever. In less than 2 years, they already have more salespeople in the US...
Door Operators Access Control Tutorial on Apr 17, 2019
Doors equipped with door operators, specialty devices that automate opening and closing, tend to be quite complex. The mechanisms needed to...
Securadyne CEO: IPVM 'Entertaining For An Ignorant Few' on Apr 16, 2019
Securadyne's CEO Carey Boethel is unhappy with IPVM's report - Failed Integrator Rollup, Securadyne Sells to Guard Giant Allied. Indeed, he...
Dahua Repositionable IR Multi-Imager Camera Tested on Apr 16, 2019
Dahua has released their first repositionable multi-imager camera, the Multi-Flex 4x2MP, claiming integrated IR, true WDR, and flexible...
Strong ISC West 2019 For Manufacturers But Concerns For 2020 March Move on Apr 16, 2019
ISC West 2019 was strong for manufacturers, according to new IPVM survey results of 100+ manufacturers, consistent with 2018 results. However,...
Axis Supports HD Analog on Apr 15, 2019
In 2017, Axis declared 'Everything is IP': Now, in 2019, Axis has released support for HD analog, with their new encoders.  Why the change?...

The world's leading video surveillance information source, IPVM provides the best reporting, testing and training for 10,000+ members globally. Dedicated to independent and objective information, we uniquely refuse any and all advertisements, sponsorship and consulting from manufacturers.

About | FAQ | Contact