Guidelines for Testing IP Camerasby John Honovich, IPVM posted on May 26, 2009 About John Contact John
What is the best way to test an IP cameras? We have established guidelines for our tests. These guidelines are listed below for your review and comment:
The goal of our tests is to provide recommendations on the real world capabilities of cameras and the best way to use those cameras in the field. The target audience of these reviews is the integrator or end user respondible for deploying or using the cameras.
No Payment from Manufacturers
We will not accept payment from manufacturers for these tests. When manufacturers pay for tests, they generally want control over the structure of tests and the content of the results.
Manufacturers may loan products and provide training on the use of their products. Prior to publication, key findings will be previewed with manufacturers, allowing them to provide feedback.
The tests will be funded by subscribers to the Video Surveillance Professional Service. While the public will be provided with a summary of key findings, detailed test results, screencasts and video samples will be exclusive to subscription members.
Include Training on Use of Product
Each test will include screencasts explaining how to setup and optimize the use of the camera. The screencasts will review physical system setup and software configuration. The training shall help viewers to learn about product details and help them in future deployments.
Test results are generally provided without training. However, training makes test results more valuable and easier to appreciate.
Providing AVIs - not just Images
The results report will include multiple video clips from each test conducted. Those video clips may be downloaded, allowing the viewer to watch all of the video clips and see for themselves.
Traditionally, tests only included sample images. However, sample images do not show motion nor frame rate. Also, sample images can be selected to give an ideal snapshot that is unrepresentative of general conditions.
Multiple Testing Scenarios/Locations
Each test will include at least 3 testing scenarios:
- Street Intersection will demonstrate significant motion of people and cars.
- Park/Parking lot will demonstrate wide area coverage with minimal movement
- Indoors/office will demonstrate moderate motion and controlled lighting
Multiple scenes day and night testing is critical to truly getting a sense of all around deployment. A single image can be significantly misleading.
Defaults Used Except for Exposure
For each camera, we use the manufacturer's factory defaults to ensure that no 'tricks' are used to make one camera better than another. One setting we have to normalize is the maximum exposure setting which some manufacturers use 'trick' default settings (review our exposure trick report).
For our low light tests, we conduct multiple tests using both 1/30s and 1/8s maximum exposure settings.
Low Light Testing
In a controlled indoor environment, we will conduct and record sample videos at both .3 and 1 lux. Lux measurements will be done in the middle of the scene at the point where the subject stops and looks directly at the camera. For background, review our lux meter/lux measurement tutorial.
Wide Dynamic Range Testing
Starting January 2010, we have added wide dynamic range testing. We use a scene with an entrance facing direct sunlight. We measure both the sunlight area and the adjacent darker areas to the left and right. We are able to reproduce scenarios with a 12x and 25x variance in lighting (e.g., 2000 lux in the sunlight area and either 160 or 80 lux in the shadow area).
Recording Bandwidth Consumption
For each scene, bandwidth consumption shall be recorded. For cameras supporting variable bit rate encoding, this rate can vary significantly based on the complexity of the scene and the configurations of each camera.
Verify Claims to Unique/Special Features
Some cameras will make claims to special features that only they or very few manufacturers possess. When a camera is tested with such features, a special test will be done to verify the claims made. For an example, see our test of the plug n play capability of the Starvedia 202.
No Rating or Approvals
The tests will not provide ratings (e.g., 7 out of 10), grades or signs of approval. Our philosophy is to provide information and recommendations on how to use the camera and what issues to consider. With that, readers may make their own decisions.
Ratings and approvals are primarily for the benefit of manufacturers to help validate the sales of their products (e.g., this camera is top ranked or has a seal of approval). They also needlessly create political issues (why did this vendor get a 8.1 and the other one get an 8.2). We believe in providing good information and let our knowledgeable readers decide what is best for their own particular circumstances.
What do you think I should change or add? I expect these guidelines to evolve and expand as the tests continue and hear from you about what you recommend.
Most Recent Industry Reports
30 vs 60 FPS Shootout on May 22, 2013
30 frames per second used to be the maximum practical limit for surveillance cameras. Now, it is becoming increasingly common for IP cameras to support double that, 60 frames per second. (For backg...
#1 IP Camera Benefit on May 20, 2013
Megapixel was, far and away, the number #1 cited benefit in going to IP cameras / video. We asked over 100 integrators to name the "3 biggest reasons" they found, allowing them to name whatever the...
Testing VSaaS / Dropcam HD on May 15, 2013
Dropcam is pretty clearly the strongest upstart in the VSaaS market. While Axis has pushed their hosted service offering for years, it is beset by poor ROI and weak performance. There are dozens of...
Testing Panoramic Cameras Outdoors on May 13, 2013
Outdoors, PTZs tend to be a favorite for covering large open areas but they only see whatever they are currently pointed at. Panoramics are an emerging alternative but how well do they perform and ...
Testing CarCam on May 08, 2013
Dash cams are getting a lot of attention for the amazing video they inadvertently capture, like a crashing 747 in Afghanistan, a meteor explosion, Russian crazy fights and accidents, etc. Moreov...
#1 IP Camera Problem on May 07, 2013
Cost was, far and away, the number #1 cited problem in going to IP cameras / video. We asked over 100 integrators to name the "3 biggest problems / barriers" they found, allowing them to name whate...
Testing Covert Cameras on May 02, 2013
Covert cameras are regularly referenced in TV shows and undercover reporting. We ordered two of the most frequently sold covert cameras on Amazon - a Car Key cam and a PenCam, as shown below: We...
Stop The Lying and Crazy Claims on Apr 29, 2013
The last few weeks have been an embarrassment for surveillance, with a small number of companies turning this into a despicable marketing campaign perpetuating lies and crazy myths about what surve...
Panoramic Camera Shootout on Apr 22, 2013
Panoramic cameras are one of the hottest growing markets within surveillance. The combination of multi-megapixel resolution and super wide FoVs claims to deliver greater coverage than conventional ...
Top Trends in Access Control on Apr 17, 2013
All of a sudden, Access Control has momentum? After a long period of apparent stagnation and staleness, new technologies and devoted marketing campaigns have breathed life into the segment. Indeed,...