IPVM Camera Testing Procedures

Author: John Honovich, Published on May 08, 2013

The foundation of IPVM's camera testing procedure is simultaneous, head to head comparisons, like so:

This is unique among surveillance industry publications.

Each camera is placed side by side, with the same FoV, connected to the same VMS and recorded at the same time and place. The setup looks like so:

We perform a few adjustments:

  • Firmware is upgraded to the most current version.
  • Minimum shutter speed is set to 1/30s to eliminate any slow shutter tricks.
  • Cameras are re-focused prior to testing.

However, unlike manufacturer demos, we do not 'tweak' cameras to optimize for individual scenes. Rather, we keep the same settings for all tests, to keep it fair, consistent and true to the most common real world uses.

We conduct tests in 3 fundamental areas:

  • Even light - this is the 'ideal' test when all cameras do the best.
  • Low light - measured at 0.2 lux
  • WDR - in an open door way back light

They look like so:

Get Video Surveillance News In Your Inbox
Get Video Surveillance News In Your Inbox

We typically run a series of pre-tests to get a feel for performance and identify any abnormalities. If the image quality looks excessively poor, we will re-check settings and/or request feedback from a manufacturer engineer. The multiple pre-test runs works out issues and increases the level of confidence that the test results accurately and fairly reflect performance.

Finally, we release test reports that are centered on a series of 5 - 15 image comparisons of multiple cameras head to head; again, like so:

The best way to understand relative performance is to properly setup and test simultaneously. Unlike looking at cameras individually, where it is easy to overestimate performance, the IPVM approach makes it clear which cameras have advantages in specific scenarios.

Bandwidth / Quantization / Compression

We typically measure bandwidth levels for each camera in each scenario. Some camera manufacturers attempt to game quantization / compression levels. We will measure those levels using a DRF analysis tool that allows us to understand if there is significant variances. This is especially important for manufacturers claiming to deliver super low bandwidth who typically do so by increasing compression levels but failing to disclose. We will report this and make adjustments in testing as appropriate.

Test Reports

See our directory of 160+ surveillance test reports. Recent favorites include our WDR Shootout, Panoramic Shootout and Axis vs Bosch Low Light Shootout.

Feel free to ask questions and we are happy to expand on this reference document.

Comments (11): PRO Members only. Login. or Join.

Related Reports

Dahua Discontinuing H.264 Only Products on Dec 08, 2016
Dahua has taken a stand for H.265 and is discontinuing its H.264 only products. We examine the shakeup inside this...
IP Networking Course January 2017 on Dec 08, 2016
This is the only networking course designed specifically for video surveillance professionals plus it includes live training, personal help and...
Power Drill Selection Guide on Dec 06, 2016
Boring holes is a basic part of running cables for most security system projects. To do so, you will need to choose the right drills for various...
CODEC Guide 2016 on Nov 30, 2016
CODECs are core to surveillance, with names like H.264, H.265, and MJPEG commonly cited. How do they work? Why should you use them? What issues may...
Solutions to 5 Common Access Control Problems on Nov 21, 2016
Effective security is more than just good equipment. In fact, thousands of dollars in electronic access control can be wasted if they are not used...
Longse vs Dahua and Hikvision Tested on Nov 16, 2016
For many, even $100 cameras are too expensive. That is where spam king Longse comes in with their relentless offer of ~$20 cameras. In our past...
Dahua 4MP HD Analog Tested on Nov 10, 2016
A few years ago, analog was SD only. In the past few years, HD analog, up to 1080p, has become commonplace. Now, HD analog is doubling its max...
Major Manufacturers Dropping MJPEG on Nov 04, 2016
MJPEG has hung on for years, still in use in some systems despite H.264's dominance for the better part of the past decade. However, now some...
Samsung Wisenet Q 4MP Tested on Nov 03, 2016
The 4MP trend continues, now with Hanwha Techwin / Samsung in their latest Wisenet Q series. We bought one of these models, the QNV-7080R, to see...
Sony 35mm Super Low Light Camera Tested on Oct 31, 2016
Sony is making a big push for their SNCVB770 camera, touting its low light capabilities that they claim can deliver true color in pitch black...

Most Recent Industry Reports

Knightscope - $122,509 Revenue, $2.5 Million Loss Seeks $20 Million Investment on Dec 09, 2016
The robot that ran over a child, Knightscope, wants money and they need it. Investors can invest as little as $1,000 to participate and...
The Russian SMP Security Robot on Dec 08, 2016
A Russian manufacturer, SMP, has a commercially available outdoor security robot, at a lower price and with much less marketing than their main...
How Hikvision Beats Its OEMs on Dec 08, 2016
Hikvision GM declared that they are not aggressive with their competitors. But some of their own OEM partners disagree. Inside, we reveal a key...
Dahua Discontinuing H.264 Only Products on Dec 08, 2016
Dahua has taken a stand for H.265 and is discontinuing its H.264 only products. We examine the shakeup inside this...
IP Networking Course January 2017 on Dec 08, 2016
This is the only networking course designed specifically for video surveillance professionals plus it includes live training, personal help and...
Hikvision vs Dahua Mobile Apps Tested on Dec 07, 2016
With smartphone use and low-cost video recorders surging, many user's main interface to their surveillance system is their phone. With mobile video...
Paxton Drops US Reps, Plans Major Expansion on Dec 07, 2016
Paxton is gearing up to make a big run at  US access control success. The first step they have made is to cut all US Rep Firms, in anticipation of...
Axis Partner Elder Care Video Analytics (Smartervision) on Dec 07, 2016
Can video analytics be used to improve the care of the elderly? Axis and a video analytics startup, Smartervision, are working together to do so....
Sony IP Camera Backdoor Uncovered on Dec 06, 2016
A backdoor has been uncovered in ~80 Sony IP camera models, attackers can remotely enable telnet on the camera, and then potentially login as root,...

The world's leading video surveillance information source, IPVM provides the best reporting, testing and training for 10,000+ members globally. Dedicated to independent and objective information, we uniquely refuse any and all advertisements, sponsorship and consulting from manufacturers.

About | FAQ | Contact