IPVM Camera Testing Procedures

Author: John Honovich, Published on May 08, 2013

The foundation of IPVM's camera testing procedure is simultaneous, head to head comparisons, like so:

This is unique among surveillance industry publications.

Each camera is placed side by side, with the same FoV, connected to the same VMS and recorded at the same time and place. The setup looks like so:

We perform a few adjustments:

  • Firmware is upgraded to the most current version.
  • Minimum shutter speed is set to 1/30s to eliminate any slow shutter tricks.
  • Cameras are re-focused prior to testing.

However, unlike manufacturer demos, we do not 'tweak' cameras to optimize for individual scenes. Rather, we keep the same settings for all tests, to keep it fair, consistent and true to the most common real world uses.

We conduct tests in 3 fundamental areas:

  • Even light - this is the 'ideal' test when all cameras do the best.
  • Low light - measured at 0.2 lux
  • WDR - in an open door way back light

They look like so:

Get Video Surveillance News In Your Inbox
Get Video Surveillance News In Your Inbox

We typically run a series of pre-tests to get a feel for performance and identify any abnormalities. If the image quality looks excessively poor, we will re-check settings and/or request feedback from a manufacturer engineer. The multiple pre-test runs works out issues and increases the level of confidence that the test results accurately and fairly reflect performance.

Finally, we release test reports that are centered on a series of 5 - 15 image comparisons of multiple cameras head to head; again, like so:

The best way to understand relative performance is to properly setup and test simultaneously. Unlike looking at cameras individually, where it is easy to overestimate performance, the IPVM approach makes it clear which cameras have advantages in specific scenarios.

Bandwidth / Quantization / Compression

We typically measure bandwidth levels for each camera in each scenario. Some camera manufacturers attempt to game quantization / compression levels. We will measure those levels using a DRF analysis tool that allows us to understand if there is significant variances. This is especially important for manufacturers claiming to deliver super low bandwidth who typically do so by increasing compression levels but failing to disclose. We will report this and make adjustments in testing as appropriate.

Test Reports

See our directory of 160+ surveillance test reports. Recent favorites include our WDR Shootout, Panoramic Shootout and Axis vs Bosch Low Light Shootout.

Feel free to ask questions and we are happy to expand on this reference document.

Comments (11): PRO Members only. Login. or Join.

Related Reports

Axis 20MP Q1659 Camera Tested on Jun 13, 2017
Axis has joined the super high resolution camera trend with their Q1659, a 20MP model equipped with Canon's APS-C sensor. We tested the Q1659...
Camera Course Summer 2017 on Jun 08, 2017
Learn video surveillance and get certified. IPVM provides live online classes, recorded videos, personal help, cutting edge education and...
Avigilon Announces New Lowest Cost Camera (Mini Domes) on Jun 06, 2017
Avigilon's first Micro Dome (see IPVM's test) is now 4 years old. Since that time, low-cost fixed focal Chinese manufacturer dome usage has...
Samsung SmartCam A1 Totally Wireless System on Jun 06, 2017
Hanwha is keeping the Samsung brand alive in consumer cameras with the SmartCam A1, a new offering combining an autotracking 1080p camera with base...
IP Camera - 15 Year Shootout on May 22, 2017
How far have IP cameras come? We bought and tested 4 cameras across the past 15 years to understand how much and where performance has...
Selling and Valuing Security Integrators on May 12, 2017
This ia a tutorial in how to (1) determine your security integrator's value and (2) to sell your security integrator. If you own an integrator,...
48MP 180 Camera (Digital Watchdog) Test on May 10, 2017
Camera resolution continues to advance, with Digital Watchdog offering the MegaPIX PANO 48MP 180° camera, the highest resolution mainstream camera...
Hanwha 9MP Fisheye Camera Tested (PNF-9010R) on May 09, 2017
12MP sensor fisheye panoramic cameras are becoming increasingly common. We have tested Hikvision's DS-2CD63C2F as well as Panasonic's SFV481 4K...
Duress Codes For Alarms Systems on May 02, 2017
An alarm system can call for help in the event of an attempted break in, but only if it is armed. If an adversary forces an authorized user to...
Last Day - IP Networking Course May 2017 on Apr 26, 2017
Today is the last day to register for the May IP Networking Course. This is the only networking course designed specifically for video...

Most Recent Industry Reports

Importance of Sales To Integrators - Statistics on Jun 23, 2017
One of the top trends in the industry over the past few years has been the rise of across-the-board sales (e.g.: Hikvision Sales, Dahua Sale,...
Deep Learning Surveillance Startups Deep Problem on Jun 23, 2017
The undeniably good news for the video surveillance market is that we are seeing the rise of more startups than in many years. The cause of this...
Avigilon Announces RADAR-Based Presence Detector on Jun 22, 2017
RADAR is gaining momentum within physical security. Two months after Axis announced a network radar detector, Avigilon has announced a RADAR-Based...
Covert Cloud Camera Service Launching (KJB) on Jun 22, 2017
Cloud IP cameras, for consumers, has become increasingly commonplace. However, covert cameras, lag there, with few options. Now, North America's...
Manufacturers Shipping Unlicensed H.265 Products on Jun 22, 2017
While H.265 support in video surveillance is growing, IPVM's research shows that most surveillance manufacturers are shipping H.265 products with...
Uniview Low-Cost Bullet PTZ Tested on Jun 21, 2017
Uniview is offering a HD zoom bullet camera, the IPC742SR9-PZ30-32G, with an integrated pan / tilt positioner, for the price of a low-cost...
QSR Video Surveillance Best Practices on Jun 21, 2017
Fast food restaurants or QSRs (quick service restaurants), are frequent victims of crime and fraud. Because they are open late, deal with cash, and...
45 Drives 'Lowest Cost' Enterprise Storage Company Profile on Jun 21, 2017
45 Drives claims the "lowest cost per Hard Drive Slot in the industry." But who or what is '45 Drives'? What started as a product design to...
No Hack, Still Liable, Court Finds ADT on Jun 20, 2017
Recently, ADT has been in the news for a $16 million settlement for a cyber security vulnerability class action suit. One of the most important...
Resolver / PPM 2000 Incident Management Platform Profile on Jun 20, 2017
You might have seen the company whose employees wear hockey jerseys at trade shows and wondered "what do they do?" PPM 2000 has been active in...

The world's leading video surveillance information source, IPVM provides the best reporting, testing and training for 10,000+ members globally. Dedicated to independent and objective information, we uniquely refuse any and all advertisements, sponsorship and consulting from manufacturers.

About | FAQ | Contact