Guidelines for Testing IP Cameras

By: John Honovich, Published on May 25, 2009

What is the best way to test an IP cameras? We have established guidelines for our tests. These guidelines are listed below for your review and comment:

Objective

The goal of our tests is to provide recommendations on the real world capabilities of cameras and the best way to use those cameras in the field. The target audience of these reviews is the integrator or end user respondible for deploying or using the cameras.

No Payment from Manufacturers

We will not accept payment from manufacturers for these tests. When manufacturers pay for tests, they generally want control over the structure of tests and the content of the results.

Manufacturers may loan products and provide training on the use of their products. Prior to publication, key findings will be previewed with manufacturers, allowing them to provide feedback.

The tests will be funded by subscribers to the Video Surveillance Professional Service. While the public will be provided with a summary of key findings, detailed test results, screencasts and video samples will be exclusive to subscription members.

Include Training on Use of Product

Each test will include screencasts explaining how to setup and optimize the use of the camera. The screencasts will review physical system setup and software configuration. The training shall help viewers to learn about product details and help them in future deployments.

Test results are generally provided without training. However, training makes test results more valuable and easier to appreciate.

Get Notified of Video Surveillance Breaking News
Get Notified of Video Surveillance Breaking News

Providing AVIs - not just Images

The results report will include multiple video clips from each test conducted. Those video clips may be downloaded, allowing the viewer to watch all of the video clips and see for themselves.

Traditionally, tests only included sample images. However, sample images do not show motion nor frame rate. Also, sample images can be selected to give an ideal snapshot that is unrepresentative of general conditions.

Multiple Testing Scenarios/Locations

Each test will include at least 3 testing scenarios:

  • Street Intersection will demonstrate significant motion of people and cars.
  • Park/Parking lot will demonstrate wide area coverage with minimal movement
  • Indoors/office will demonstrate moderate motion and controlled lighting

All tests will be conducted both day and night to demonstrate how changes in lighting conditions impact performance.

Multiple scenes day and night testing is critical to truly getting a sense of all around deployment. A single image can be significantly misleading.

Defaults Used Except for Exposure

For each camera, we use the manufacturer's factory defaults to ensure that no 'tricks' are used to make one camera better than another. One setting we have to normalize is the maximum exposure setting which some manufacturers use 'trick' default settings (review our exposure trick report).

For our low light tests, we conduct multiple tests using both 1/30s and 1/8s maximum exposure settings.

Low Light Testing

In a controlled indoor environment, we will conduct and record sample videos at both .3 and 1 lux. Lux measurements will be done in the middle of the scene at the point where the subject stops and looks directly at the camera. For background, review our lux meter/lux measurement tutorial.

Wide Dynamic Range Testing

Starting January 2010, we have added wide dynamic range testing. We use a scene with an entrance facing direct sunlight. We measure both the sunlight area and the adjacent darker areas to the left and right. We are able to reproduce scenarios with a 12x and 25x variance in lighting (e.g., 2000 lux in the sunlight area and either 160 or 80 lux in the shadow area).

Recording Bandwidth Consumption

For each scene, bandwidth consumption shall be recorded. For cameras supporting variable bit rate encoding, this rate can vary significantly based on the complexity of the scene and the configurations of each camera. 

Verify Claims to Unique/Special Features

Some cameras will make claims to special features that only they or very few manufacturers possess. When a camera is tested with such features, a special test will be done to verify the claims made. For an example, see our test of the plug n play capability of the Starvedia 202.

No Rating or Approvals

The tests will not provide ratings (e.g., 7 out of 10), grades or signs of approval. Our philosophy is to provide information and recommendations on how to use the camera and what issues to consider. With that, readers may make their own decisions.

Ratings and approvals are primarily for the benefit of manufacturers to help validate the sales of their products (e.g., this camera is top ranked or has a seal of approval). They also needlessly create political issues (why did this vendor get a 8.1 and the other one get an 8.2). We believe in providing good information and let our knowledgeable readers decide what is best for their own particular circumstances.

Feedback?

3 reports cite this report:

Readers Respond: 134 Votes, 25 Comments Examined on Feb 14, 2010
Over 100 people voted and more than 25 left detailed comments to our recent...
IP Camera 2009 Mid Year Review on Jul 26, 2009
Megapixel was the most dominant trend in IP cameras over the first half of...
How Well do IP Cameras Work in Low Light? on Jun 06, 2009
[2016 Update: Camera technology has changed substantially since 2009. The...

Related Reports

School District Admits Not Following FDA Guidelines With 144, No Blackbody, Hikvision Fever Cameras on Aug 21, 2020
The Baldwin County School District has admitted it is not following FDA...
Dynamic vs Static IP Addresses Tutorial on Apr 16, 2020
While many cameras default to DHCP out of the box, that does not mean you...
How To See If Your Camera Uses Huawei Hisilicon Chips on Aug 30, 2019
Rarely do manufacturers disclose what SoCs (System on a Chip) they use, even...
Avigilon Social Distancing Analytics Tested on Aug 26, 2020
Avigilon released its social distancing analytics in response to the...
Vivotek LPR Camera Tested on Apr 15, 2020
Vivotek has historically sold license plate capture cameras but not LPR. Now,...
Video Analytics 101 on Mar 16, 2020
This guide teaches the fundamentals of video surveillance...
Disruptive Free Lead Generation Added To IPVM on May 15, 2020
IPVM has added lead generation for sellers, for free, disrupting the...
Face Masks Increase Face Recognition Errors Says NIST on Aug 04, 2020
COVID-19 has led to widespread facemask use, which as IPVM testing has shown...
Camera Focusing Tutorial on Oct 14, 2019
Camera focus is fundamental to quality imaging. Mistakes can significantly...
Hidden Camera Detectors Tested on Nov 18, 2019
Hidden cameras are a growing problem as cameras become smaller, cheaper and...
IPVM Camera Calculator User Manual / Guide on Sep 23, 2020
Learn how to use the IPVM Camera Calculator (updated for Version 3.1). The...
IP Networking Course Fall 2020 - Last Chance - Register Now on Sep 23, 2020
Today is the last chance to register for the only IP networking course...
CheckMySystems Company Profile on Aug 14, 2019
CheckMySystems says that too many users respond, "I get an email when...
Use Access Control Logs To Constrain Coronavirus on Apr 09, 2020
Access control users have included capabilities that are not commonly used...
Hikvision OEM Directory on Aug 13, 2019
The Chinese government-owned and US-government banned Hikvision has become...

Recent Reports

New Products Show Fall 2020 Starts Tomorrow! on Sep 27, 2020
Tomorrow, IPVM's sixth online show will feature New Products from over 25...
OnTech Smart Services Partners With Google and Amazon To Compete With Integrators on Sep 25, 2020
A pain point for many homeowners to use consumer security and surveillance is...
The Future of Metalens For Video Surveillance Cameras - MIT / UMass / Immervision on Sep 25, 2020
Panoramic cameras using 'fisheye' lens have become commonplace in video...
Hikvision Sues Over Brazilian Airport Loss on Sep 24, 2020
Hikvision was excluded from a Brazilian airport project because it is owned...
China General Chamber of Commerce Calls Out US Politics on Sep 24, 2020
While US-China relations are at an all-time low, optimism about relations...
Verkada Disruptive Embedded Live Help on Sep 24, 2020
Call up your integrator? Have someone come by the next day? Verkada is...
IP Networking Course Fall 2020 - Last Chance - Register Now on Sep 23, 2020
Today is the last chance to register for the only IP networking course...
Drain Wire For Access Control Reader Tutorial on Sep 23, 2020
An easy-to-miss cabling specification plays a key role in access control, yet...
Norway Council of Ethics Finds Hikvision Human Rights Abuses "Ongoing" on Sep 23, 2020
Hikvision's involvement in "serious human rights abuse" in Xinjiang is...
IPVM Camera Calculator User Manual / Guide on Sep 23, 2020
Learn how to use the IPVM Camera Calculator (updated for Version 3.1). The...
Installation Course Fall 2020 - Save $50 - Last Chance on Sep 22, 2020
This is a unique installation course in a market where little practical...
SimpliSafe Business Security Launched Examined on Sep 22, 2020
SimpliSafe has launched "SimpliSafe Business Security" that the company...
FLIR CEO: Many New Fever Entrants "Making Claims That The Science Just Won't Support" on Sep 22, 2020
FLIR's CEO joins a growing number calling out risks with fever / screening...
China Bems Temperature Measurement Terminal Tested on Sep 22, 2020
Guangzhou Bems (brand Benshi) is the manufacturer behind temperature...