Guidelines for Testing IP Cameras

By: John Honovich, Published on May 25, 2009

What is the best way to test an IP cameras? We have established guidelines for our tests. These guidelines are listed below for your review and comment:

Objective

The goal of our tests is to provide recommendations on the real world capabilities of cameras and the best way to use those cameras in the field. The target audience of these reviews is the integrator or end user respondible for deploying or using the cameras.

No Payment from Manufacturers

We will not accept payment from manufacturers for these tests. When manufacturers pay for tests, they generally want control over the structure of tests and the content of the results.

Manufacturers may loan products and provide training on the use of their products. Prior to publication, key findings will be previewed with manufacturers, allowing them to provide feedback.

The tests will be funded by subscribers to the Video Surveillance Professional Service. While the public will be provided with a summary of key findings, detailed test results, screencasts and video samples will be exclusive to subscription members.

Include Training on Use of Product

Each test will include screencasts explaining how to setup and optimize the use of the camera. The screencasts will review physical system setup and software configuration. The training shall help viewers to learn about product details and help them in future deployments.

Test results are generally provided without training. However, training makes test results more valuable and easier to appreciate.

Get Notified of Video Surveillance Breaking News
Get Notified of Video Surveillance Breaking News

Providing AVIs - not just Images

The results report will include multiple video clips from each test conducted. Those video clips may be downloaded, allowing the viewer to watch all of the video clips and see for themselves.

Traditionally, tests only included sample images. However, sample images do not show motion nor frame rate. Also, sample images can be selected to give an ideal snapshot that is unrepresentative of general conditions.

Multiple Testing Scenarios/Locations

Each test will include at least 3 testing scenarios:

  • Street Intersection will demonstrate significant motion of people and cars.
  • Park/Parking lot will demonstrate wide area coverage with minimal movement
  • Indoors/office will demonstrate moderate motion and controlled lighting

All tests will be conducted both day and night to demonstrate how changes in lighting conditions impact performance.

Multiple scenes day and night testing is critical to truly getting a sense of all around deployment. A single image can be significantly misleading.

Defaults Used Except for Exposure

For each camera, we use the manufacturer's factory defaults to ensure that no 'tricks' are used to make one camera better than another. One setting we have to normalize is the maximum exposure setting which some manufacturers use 'trick' default settings (review our exposure trick report).

For our low light tests, we conduct multiple tests using both 1/30s and 1/8s maximum exposure settings.

Low Light Testing

In a controlled indoor environment, we will conduct and record sample videos at both .3 and 1 lux. Lux measurements will be done in the middle of the scene at the point where the subject stops and looks directly at the camera. For background, review our lux meter/lux measurement tutorial.

Wide Dynamic Range Testing

Starting January 2010, we have added wide dynamic range testing. We use a scene with an entrance facing direct sunlight. We measure both the sunlight area and the adjacent darker areas to the left and right. We are able to reproduce scenarios with a 12x and 25x variance in lighting (e.g., 2000 lux in the sunlight area and either 160 or 80 lux in the shadow area).

Recording Bandwidth Consumption

For each scene, bandwidth consumption shall be recorded. For cameras supporting variable bit rate encoding, this rate can vary significantly based on the complexity of the scene and the configurations of each camera. 

Verify Claims to Unique/Special Features

Some cameras will make claims to special features that only they or very few manufacturers possess. When a camera is tested with such features, a special test will be done to verify the claims made. For an example, see our test of the plug n play capability of the Starvedia 202.

No Rating or Approvals

The tests will not provide ratings (e.g., 7 out of 10), grades or signs of approval. Our philosophy is to provide information and recommendations on how to use the camera and what issues to consider. With that, readers may make their own decisions.

Ratings and approvals are primarily for the benefit of manufacturers to help validate the sales of their products (e.g., this camera is top ranked or has a seal of approval). They also needlessly create political issues (why did this vendor get a 8.1 and the other one get an 8.2). We believe in providing good information and let our knowledgeable readers decide what is best for their own particular circumstances.

Feedback?

3 reports cite this report:

Readers Respond: 134 Votes, 25 Comments Examined on Feb 14, 2010
Over 100 people voted and more than 25 left detailed comments to our recent question/discussion. In this post, we review and respond to the...
IP Camera 2009 Mid Year Review on Jul 26, 2009
Megapixel was the most dominant trend in IP cameras over the first half of 2009. So dominant was megapixel, that not only did most manufacturers...
How Well do IP Cameras Work in Low Light? on Jun 06, 2009
[2016 Update: Camera technology has changed substantially since 2009. The same pattern occurs but cameras are generally much better in low light....

Related Reports

Cheap Camera Problems at Night on Feb 19, 2020
Cheap cameras generally have problems at night, despite the common perception that integrated IR makes cameras mostly the same, according to new...
IR Surveillance Camera Guide on Feb 06, 2020
Integrated infrared (IR) cameras are everywhere in 2020, but not all IR is created equal, with many differences in coverage, overexposure,...
IPVM Opens 12,000 Sqft Testing Facility on Dec 16, 2019
IPVM is proud to announce the opening of the world's first video surveillance testing facility that will allow us to significantly expand our...
Budget Covert Cameras Tested on Nov 26, 2019
Covert cameras under $100 are widely available online but are they any good? To see how these models really work in the real world, we bought...
Biggest Low Light Problems 2019 on Nov 08, 2019
Over 150 integrators responded to our survey question: "What are the biggest problems you face getting good low-light images?" Inside, we share...
Hikvision ColorVu is Smart Marketing on Oct 03, 2019
Hikvision ColorVu (see IPVM test results) is smart marketing, a lesson to be learned by competitors and a rising trend. Inside this note, we...
Wide Dynamic Range (WDR) Guide on Oct 01, 2019
Understanding wide dynamic range (WDR) is critical to capturing high quality images in demanding conditions. However, with no real standards, any...
False Verkada 'Unrivaled' Low Light Performance Claim Removed on Jun 12, 2019
Verkada falsely claimed that it delivered 'UNRIVALED LOW LIGHT PERFORMANCE' until IPVM questioned. In fact, Verkada's low light performance is...
Verkada Video Quality Problems Tested on May 23, 2019
Verkada suffers from numerous video quality problems, not found in commercial IP cameras, new IPVM testing of Verkada vs Axis and Hikvision...
Speco Ultra Intensifier Tested on Apr 22, 2019
While ISC West 2019 named Speco's Ultra Intensifier the best new "Video Surveillance Cameras IP", IPVM testing shows the camera suffers from...

Most Recent Industry Reports

Pivot3 Mass Layoffs on Mar 27, 2020
Pivot3 has conducted mass layoffs, the culmination of grand hopes, a quarter of a billion dollars in VC funding, and multiple failures to gain...
Athena CEO Criticizes 'Deplorable' 'Nitpicking', IPVM Refutes on Mar 27, 2020
Athena Security's CEO Lisa Falzone has strongly objected to IPVM's reporting on Athena, calling it 'deplorable' and repeatedly criticizing IPVM's...
Hikvision Admits Sanctions Harming Its Financial Performance on Mar 27, 2020
While Hikvision initially downplayed being sanctioned for human rights abuses, the company is now admitting a significant impact in a new PRC...
New Axis M30 Cameras Tested on Mar 26, 2020
Axis has released a new generation of, for them, relatively low cost M30 series cameras, claiming to deliver "sharp video quality even in poor...
Coronavirus Shuts Down ADT Door Knockers on Mar 26, 2020
Coronavirus has another victim - this time, alarm giant ADT has stopped all door to door sales. Door knockers are a critical but controversial...
Access Control Course Spring 2020 - Save $50 Last Day on Mar 26, 2020
Register Now - Spring 2020 Access Control Course. Today, March 26th is the last day to save $50. IPVM offers the most comprehensive access...
Convergint Coronavirus Cuts on Mar 25, 2020
One of the world's largest security integrators, Convergint, has made a major move to handle the impact of coronavirus, with cuts across the...
VSaaS 101 on Mar 25, 2020
Video Surveillance as a Service (VSaaS) is the common industry term for cloud video. But what does it mean? How does it all work? Inside this...
TVT / InVid Facial Recognition Tested on Mar 25, 2020
Facial recognition is frequently sold for thousands of dollars per channel but some China manufacturers are offering full facial recognition...
IPVM Launches On-Demand Courses on Mar 24, 2020
For nearly a decade, IPVM has been a leader in online live courses. Now, we have added on-demand versions for all courses. The same course...