IPVM Camera Testing Procedures

Author: John Honovich, Published on May 08, 2013

The foundation of IPVM's camera testing procedure is simultaneous, head to head comparisons, like so:

This is unique among surveillance industry publications.

Each camera is placed side by side, with the same FoV, connected to the same VMS and recorded at the same time and place. The setup looks like so:

We perform a few adjustments:

  • Firmware is upgraded to the most current version.
  • Minimum shutter speed is set to 1/30s to eliminate any slow shutter tricks.
  • Cameras are re-focused prior to testing.

However, unlike manufacturer demos, we do not 'tweak' cameras to optimize for individual scenes. Rather, we keep the same settings for all tests, to keep it fair, consistent and true to the most common real world uses.

We conduct tests in 3 fundamental areas:

  • Even light - this is the 'ideal' test when all cameras do the best.
  • Low light - measured at 0.2 lux
  • WDR - in an open door way back light

They look like so:

Get Video Surveillance News In Your Inbox
Get Video Surveillance News In Your Inbox

We typically run a series of pre-tests to get a feel for performance and identify any abnormalities. If the image quality looks excessively poor, we will re-check settings and/or request feedback from a manufacturer engineer. The multiple pre-test runs works out issues and increases the level of confidence that the test results accurately and fairly reflect performance.

Finally, we release test reports that are centered on a series of 5 - 15 image comparisons of multiple cameras head to head; again, like so:

The best way to understand relative performance is to properly setup and test simultaneously. Unlike looking at cameras individually, where it is easy to overestimate performance, the IPVM approach makes it clear which cameras have advantages in specific scenarios.

Bandwidth / Quantization / Compression

We typically measure bandwidth levels for each camera in each scenario. Some camera manufacturers attempt to game quantization / compression levels. We will measure those levels using a DRF analysis tool that allows us to understand if there is significant variances. This is especially important for manufacturers claiming to deliver super low bandwidth who typically do so by increasing compression levels but failing to disclose. We will report this and make adjustments in testing as appropriate.

Test Reports

See our directory of 160+ surveillance test reports. Recent favorites include our WDR Shootout, Panoramic Shootout and Axis vs Bosch Low Light Shootout.

Feel free to ask questions and we are happy to expand on this reference document.

Comments (11): PRO Members only. Login. or Join.

Related Reports

Bosch First Starlight 1080p Camera Tested on Oct 13, 2016
Bosch has released their first 'Starlight' 1080p camera, the Flexidome IP starlight 7000 VR. We tested this new dome against competitive super low...
4 Biggest Low Light Problems on Oct 10, 2016
100 integrators told IPVM what their biggest problems were with low light images. The most commonly cited themes were: Limited IR Range Uneven...
Samsung 'Q' Low-Cost 1080p Dome Tested on Oct 10, 2016
The most competitive part of the market is for low-cost cameras. Hanwha Techwin / Samsung has released its new low-cost Q series to go after this...
Intrusion Motion Sensors Tutorial on Oct 04, 2016
Motion sensors are one of the most common, and most useful alarm sensors, but they also often cause problems. Many motion sensors types exist and...
Axis Zipstream 2 Tested on Oct 03, 2016
Smart codecs are one of the big trends in the industry now. Axis was the first manufacturer to launch a smart codec, releasing Zipstream in Spring...
Arecont To Release Industries' Best H.265 Support on Sep 30, 2016
Go big, or go home. That seems to be Arecont's philosophy for H.265, they say they are working on the "best" H.265 implementation in the...
Axis 4MP Camera Tested (M3046-V) on Sep 28, 2016
Axis has brought 4MP to its camera line in the new M3046-V, the highest resolution model in their revamped M30 series. We bought and tested this...
Access Door Controller Configuration Guide on Sep 27, 2016
Properly configuring access control door controllers is key to a professional system. These devices have fundamental settings that must be...
VLANs for Video Surveillance Tutorial on Sep 26, 2016
Many people confidently say to 'use VLANs' as an answer to IP video networking problems and as a way to signal expertise. But how should VLANs be...
History of Video Surveillance on Sep 22, 2016
This is a concise history of video surveillance covering the past decade.  The goal is to help professionals newer to the industry understand...

Most Recent Industry Reports

Axis Video Revenue Down on Oct 20, 2016
An important milestone. Axis revenue for video products is down year over year. But Axis is now focusing on 'diversification'. In this report, we...
Worst Access Control 2016 on Oct 19, 2016
Two access control providers stood out as being the worst to work with for integrators. In this report, we analyze the answers to: "In the past...
Hacked DVRs Surge To 400,000 on Oct 19, 2016
The global internet is under attack from record breaking botnets. And it is getting worse, Mirai doubled in size in the last month. Shamefully,...
Longse Rips Off Hikvision on Oct 19, 2016
Longse is on the attack, and now they are targeting Hikvision. Evidently not content just to rip off Milestone and Video Insight, Longse has now...
China "Unswerving Leadership Over State-Owned Enterprises" Like Hikvision on Oct 18, 2016
The PR agency of the Chinese government declared: President Xi Jinping stressed the Communist Party of China's (CPC) unswerving leadership over...
Move Aside Cisco, Axis Has A Network Switch For Integrators on Oct 18, 2016
Cisco is a common choice for network switches, including in our Favorite Network Switches survey, but now Axis is releasing a 16 port PoE+ switch...
IHS Sells Credibility To Manufacturer Sponsorships on Oct 18, 2016
A valuable manufacturer opportunity or a conflict of interest? Trade magazines have long offered promotional whitepapers and webinars to...
The Last Decade's Big Shift In One Image on Oct 17, 2016
Look at the two cameras side by side in the image below: This is inside an urban Shell gas station. The two cameras are Sensormatic and Lorex,...
No Marketing Gimmick, Sharp Security Robot Targets Professional Market on Oct 17, 2016
Security robots are getting a lot of attention. Unfortunately, this is often for wrong reasons, most spectacularly highlighted by the Knightscope...

The world's leading video surveillance information source, IPVM provides the best reporting, testing and training for 10,000+ members globally. Dedicated to independent and objective information, we uniquely refuse any and all advertisements, sponsorship and consulting from manufacturers.

About | FAQ | Contact