The Indoor 7 Camera Shootout

Indoors, surveillance cameras are used all over. Given the generally small areas monitored and their shelter from the weather, getting good surveillance video is easier than outdoors. It raises the question: Does it even matter what type of surveillance cameras you use indoors? What type of practical differences or benefits does one camera provide over another?

We wanted to test this rigorously so we set up a 7 camera test kit to do a simultaneous shootout. In our kit we included cameras ranging from SD through 5MP from Arecont Vision, Avigilon, Axis, Bosch and Sony to see if more resolution helps, if D/N versus color only matters and if different manufacturers are 'better', etc.

We mounted our test kit in a corner at ceiling height as is common in real world installations.

We first tested during the day with even artifical lighting conditions to get a baseline test of an 'easy' scenario:

We tested with the shades open to see the impact of outdoor lighting through a window. Often businesses (like retailers) like to have the windows open to bring in natural light and to let prospective customers see in. We wanted to see what negative impact this might have on each camera:

Finally, we tested in the dark to see how cameras would perform after hours in case an intruder broke in. Since manufacturers use varying tricks at night, we did 2 tests here - one with manufacturer defaults and one with exposure normalized.

The overhead image and map below shows the coverage, areas, distances and pixels per foot at different testing positions. The human figure indicates the 3 spots where we performed analysis of image quality:

Analyzing Video Quality

We analyzed a series of scenes to see how performance varied across the cameras. We digitally zoomed each camera to the same level to show differences. See an example scene comparison from our parking lot test.

We conducted our analysis across 12 scenes. They are:

• Even Artificial Lighting (Doorway)
• Even Artificial Lighting (Far)
• Even Artificial Lighting (Near)
• Backlit/WDR via Open Window (Doorway)
• Backlit/WDR via Open Window (Far)
• Backlit/WDR via Open Window (Near)
• Front-lit via Open Window (Doorway)
• Front-lit via Open Window (Far)
• Front-lit via Open Window (Near)
• Low-Light (Default Settings)
• Low-Light (Normalized Settings)
• Low-Light Default vs Low-Light Normalized

Analyzing Bandwidth Consumption - NEW

Not only do we examine which cameras provide the best quality in each scene, we now break down the bandwidth consumption, showing who's the best and worse for the amount of network bandwidth and storage consumed.

Our analysis covers 7 metrics:

• Even Artificial Lighting Bandwidth Consumption Per Camera
• Backlight / WDR Bandwidth Consumption Per Camera
• Front-Lit Bandwidth Consumption Per Camera
• Low-Light (Default Settings) Bandwidth Consumption Per Camera
• Low-Light (Normalized Settings) Bandwidth Consumption Per Camera
• Average Bandwidth Consumption Per Scene
• Average Bandwidth Consumption Per Camera Overall

• In practically every scene tested, 720p appeared to provide a distinct edge over SD. At the wider FoVs tested, e.g. 31ft, it typically meant the difference between near absolute identification (720p) versus only a slight to moderate certainty of identification (SD).
• Contrary to expectations, higher MP (e.g. 5MP) did not provide real material benefits over 720p cameras under test conditions. Often, even at the widest FoVs tested, the two categories rendered similar if not equal results.
• SD approached near absolute certainties of subject identification only at more narrow FoVs and under more ideal lighting conditions (e.g., front-lit at 16ft or less FoVs).
• Low-light tests revealed only a minority of cameras can be deployed with default settings to effectively monitor in ~0 lux environments. Most cameras will require exposure settings be optimized for a particular low-light application. When shutter speeds were normalized to 1/30s, SD D/N appeared to exhibit the best performance. 

In light of these findings we recommend the following:

• Consider megapixel between 1MP to 2MP resolution. Higher megapixel may present only marginal gains in image quality, a diminished low-light sensitivity and a lower frame-rate.
• SD may be considered for narrower FoV applications (e.g. ~16ft or less). SD will tend to provide slightly better low-light and bandwidth performance.
• Consider positioning cameras 'downstream' from bright windows or light sources. If monitoring activity at a window is desired, however, consider a camera with capable WDR performance

Even Artificial Lighting (Doorway)

In this scenario cameras are digitally zoomed into a human subject ~28ft out, corresponding to a HFoV of ~26ft. Lighting is even and not particularly challenging (~200 lux).

Below is the Composite Snapshot (Download Full Video Clip [link no longer available]):

The majority of MP cameras in the composite (i.e. 5MP, 720p Avigilon/Sony) provide unequivocally greater detail levels on the human subject than the SD cameras. In these cameras' images identification is highly certain. Still, SD provides some key investigative details such as hair-style, build, complexion, and gender. Note that both the 720p M1114 and 1.3MP AV1315DN provide lesser image qualities than the other two 720p cameras, despite having equal horizontal resolutions.

Even Artificial Lighting (Far)

In this scenario cameras are digitally zoomed into a human subject ~33ft out, corresponding to a HFoV of ~31ft. Lighting is even and not particularly challenging (~200 lux).

Below is the Composite Snapshot (Download Full Video Clip [link no longer available]):

Results are similar to the 'doorway' counterpart, where three of five MPs (i.e. 5MP, 720p Avigilon, 720p Sony CH140) provide strong detail levels and strong odds of identification. SDs are considerably less detailed but still offer clues as to the subject's hair-style, build, complexion and gender. Note that in the SDs and two of the MPs  (1.3MP AV1315 and 720p Axis M1114) the presence of eye-glasses are not readily detectable.

Even Artificial Lighting (Near)

In this scenario cameras are digitally zoomed into a human subject ~17ft out, corresponding to a HFoV of ~16ft. Lighting is even and not particularly challenging (~200 lux).

Below is the Composite Snapshot (Download Full Video Clip [link no longer available]):

In this 'near' composite all cameras provide fair to high levels of detail, and subsequent moderate to high certainty levels of identification. The same three MPs (5MP, 720p Avigilon and 720 Sony CH140) provide the greatest level of details once again. Note that the 720p M1114 provides a lesser image quality than the other two 720p cameras, despite having equal horizontal resolutions.

Back-lit/WDR via Open Window (Doorway)

In this scenario cameras are digitally zoomed into a human subject ~28ft out, corresponding to a HFoV of ~26ft. The open window creates a lighting challenge with moderate WDR properties. Roughly ~1000 lux is measured at the window, while ~300 lux is measured at center room.

Below is the Composite Snapshot (Download Full Video Clip [link no longer available]):

In this backlit scene all cameras render the human subject's face somewhat under-exposed, leading to loss of contrast and overall clarity. The 720p Sony CH140 is the one exception which exposes the subject the best. Despite diminished picture qualities relative to even lit scenarios, the MPs (i.e. 5MP, 720p Avigilon and 720p Sony CH140) still provide moderate to strong odds of identification. Note that both the 720p M1114 and 1.3MP AV1315DN, despite having nearly twice the ppf, appear to perform only similarly to the SD Bosch NBN-498.

Back-lit/WDR via Open Window (Far)

In this scenario cameras are digitally zoomed into a human subject ~33ft out, corresponding to a HFoV of ~31ft. The open window creates a lighting challenge with moderate WDR properties. Roughly ~1000 lux is measured at the window, while ~300 lux is measured at center room.

Below is the Composite Snapshot (Download Full Video Clip [link no longer available]):

The challenges of back-lighting and WDR are most evident in this composite. All cameras show severely diminished levels of detail compared to the same 'far' scene of the even lit scenario. The 5MP and 720p Avigilon/Sony provide the best level of details, but likelihood of identification has degraded to fair/moderate levels. A majority of the MPs however, still provide other key investigative details regarding clothes, gender, build, complexion and eye-glasses. Note the 720p (Sony CH140) and SD Bosch NBN-498's ability to provide more appropriate exposure levels on the external side of the window. This could be a potential benefit in the detection of suspicious activity occuring outside (e.g. potential assault, vandalism or break-in).

Back-lit/WDR via Open Window (Near)

In this scenario cameras are digitally zoomed into a human subject ~17ft out, corresponding to a HFoV of ~16ft. The open window creates a lighting challenge with moderate WDR properties. Roughly ~1000 lux is measured at the window, while ~300 lux is measured at center room.

Below is the Composite Snapshot (Download Full Video Clip [link no longer available]):

Three of five MPs (i.e. 5MP, 720p Avigilon, and 720 Sony) provide strong enough detail levels for high certainty identification. The remaining MPs and both SDs are considerably 'blurry' and somewhat under-exposed.

In this scenario cameras are digitally zoomed into a human subject ~28ft out, corresponding to a HFoV of ~26ft. The room is illuminated with natural light provided by the open window. Note that the open window has front-lit the room due to positioning cameras at the opposite end - 'down-stream' of light source. As a result lighting is relatively even (~300 lux) and does not present a challenge.

Below is the Composite Snapshot (Download Full Video Clip [link no longer available]):

The 5MP, 720p Avigilon and 720p Sony CH140 provide the greatest level of details and fairly strong certainties of identification. The 1.3MP, 720p Axis M1114 and SDs provide similar levels of detail, which are not sufficient for strong odds of identification.

Front-lit via Open Window (Far)

In this scenario cameras are digitally zoomed into a human subject ~33ft out, corresponding to a HFoV of ~31ft. The room is illuminated with natural light provided by the open window. Note that the open window has front-lit the room due to positioning cameras at the opposite end - 'down-stream' of light source. As a result lighting is relatively even (~300 lux) and does not present a challenge.

Below is the Composite Snapshot (Download Full Video Clip [link no longer available]):

None of the cameras appear to provide an exceptional level of detail in this composite. The 5MP, 720p Avigilon and 720p Sony provide the best level of details, but do not result in near or absolute certainty of identification.

Front-lit via Open Window (Near)

In this scenario cameras are digitally zoomed into a human subject ~17ft out, corresponding to a HFoV of ~16ft. The room is illuminated with natural light provided by the open window. Note that the open window has front-lit the room due to positioning cameras at the opposite end - 'down-stream' of light source. As a result lighting is relatively even (~300 lux) and does not present a challenge.

Below is the Composite Snapshot (Download Full Video Clip [link no longer available]):

Strong detail levels are provided by all MP cameras (both higher MP and 720p), making identification highly likely. The one exception is the 720p Axis M1114 which is more comparable to the SD cameras in this composite. Despite a somewhat weaker picture quality, they still provide a reasonably good level of detail and chance at identification.

Low-light (Default Settings)

In this composite, cameras are left at their full field of view (no digital zoom). Light levels are measured ~0.1 lux in this scene. Note that all cameras are set to default settings.

Below is the Composite Snapshot (Download Full Video Clip [link no longer available]):

With default settings applied, most cameras in the composite provide some measure of utility after low-light adaptation. The SD Bosch NBN-498 clearly provides the best overall picture quality amongst them. The exposure maximums of the 720p Sony CH140 and 720p Avigilon are among the lowest (1/30s), and, not surprisingly, exhibit considerably under-exposed images.

Low-light (Normalized)

In this composite, cameras are left at their full field of view (no digital zoom). Light levels are measured ~0.1 lux in this scene. Note that all cameras have been 'normalized' to a 1/30s max exposure time.

Below is the Composite Snapshot (Download Full Video Clip [link no longer available]):

Interestingly, cameras on the top row (5MP Axis P1347, 720p Axis M1114, and SD Axis M1011), which all featured 1/6s max shutter speeds, have all degraded into essentially 'black' images, since normalizing to 1/30s. The SD Bosch NBN-498 continues to provide the best apparent illumination even after the normalization process. Note that the 720p Avigilon and 720p Sony CH140 have default max shutters of 1/30s, and as such underwent no changes in the normalization scenario.

Low-light Default vs. Normalized

In this comparison images from each of the seven (7) cameras are arranged side by side with default settings (left) vs. normalized settings (right). The light levels are the same for either side (~0.1 lux).

Below is the Composite Snapshot:

Bandwidth Analysis

In this section, we examine the comparative bandwidth consumption of each camera, explaining variances in results and key issues.

In the even lighting scene, bandwidth consumption is fairly similar as the scene has minimal motion and no lighting challenges. Keep this in mind when we contrast this to other scenes.

In the front-lit scene, we have the lowest overall bandwidth consumption of the all the scenes. Our hypothesis is that camera bandwidth consumption is minimized when the lighting is quite level. To the contrary, note the jump in Arecont's bandwidth. This will be a consistent (and increasing problem) for the Arecont.

In the backlight / WDR scene, all of the camera's bandwidth consumption remains relatively the same except for the Arecont which jumps even higher.

Here in the low light scene is where things get really interesting. The Arecont, Axis and Avigilon cameras all show steep jumps in bandwidth consumption (despite the fact that Arecont and Axis frame rate drops significantly). Key driving factor is the hard to compress visual noise from aggressive gain control. Since all of these cameras are VBR, the bitrate surges. While the Sony also displays significant visual noise, as it uses constant bit rate encoding, no change occurs. While the Bosch's rate control allows the bandwith to rise it does not because of strong low light performance with minimal noise.

In this final scene, we kept the low light levels the same but normalized all the cameras to a 1/30s exposure (note: the Avigilon and Sony default to 1/30s, all others were changed). The Arecont bandwidth consumption increases even higher (to scary levels) as well as the Axis SD (in color mode with high visible noise). Interestingly, though, the Axis MP cameras bandwidth drops to nearly zero as the scene is pitch black.

Now we can look at the average bandwidth levels across scenes. The pattern is obvious - low light bandwidth consumption is nearly triple that of daytime bandwidth consumption.

Finally, we can see the average bandwidth consumption by camera:

 

• Arecont consumes the most - driven by its challenges in handling lighting variations.
• Avigilon is the next highest - though the image quality was comparatively strong throughout
• The SD Axis came in 3rd highest mainly do to challenges in handling challenging light conditions
• Using the default 2Mb/s bandwidth level, Sony delivered very consistent bandwidth consumption.
• The 5MP Axis was the second lowest. However, its max frame rate was only 40% of the other cameras (12 vs 30fps) so direct comparison is misleading.
• The MP Axis camera's bandwidth consumption was helped by dropping their bandwidth consumption to near zero in the normalized test. On the other hand, the Avigilon and the Sony consumed more bandwidth but provided at least some details in its video (making them more usable than the MP Axis cameras).

 

Methodology

The following are the seven (7) cameras used in the 'Office Shootout':

• Arecont AV1315DN (online $460) - 1.3MP D/N; 1/2.7" CMOS; F1.6; 0.1 Lux (BW)
• Avigilon 1.0MP-HD-H264-B1 (MSRP $645) -720p D/N; 1/2.7" CMOS; F1.6; 0.4/0.04 Lux (Color/BW)
• Axis M1011 (online $160) -  4CIF Color-only; 1/4" CMOS; Integrated F2.0; 1.0 Lux (Color)
• Axis M1114 (online $475) - 720p D/N; 1/4" CMOS;
• Axis P1347 (online $1350) - 5MP D/N; 1/2.5" CMOS; F1.6 Kowa;0.08 Lux (BW)
• Bosch NBN-498 [link no longer available] (online $700) - SD D/N; 1/3" CCD; F1.3 Bosch; 0.06 Lux (BW)
• Sony CH140 (online $800) - 720p D/N; 1/3" CMOS; F1.2 Fujinon; 0.1 Lux (BW)

Each of the seven (7) cameras were simultaneously recorded to an Exacqvision VMS during five (5) key scenarios. All cameras were set to their defaults during testing, except during the low-light tests. During low-light tests D/N cameras were 'forced' into B/W to ensure consistency of comparison. Furthermore, cameras were normalized to 1/30s shutter speed for one of the tests. All camera lenses were adjusted to a uniform lens angle (~50 degrees) and recorded/analyzed at their - VMS allowed - maximum resolutions.

Here are the key default settings and firmware versions for each camera:

• Arecont 1315DN - Automatic Exposure 'Balanced'; fw 65197
• Avigilon 1.0MP - 1/30s maximum exposure; fw 1.0.4.4(3305)
• Axis M1011 - 1/6s max exposure w/ AGC; fw 5.00.1
• Axis M1114 - 1/6s max exposure w/ AGC; fw - 5.09
• Axis P1347 - 1/6s max exposure w/ AGC; fw 5.20
• Bosch NBN-498 -  1/60s shutter; SensUp 4x; AGC on '15'; fw 01500411
• Sony CH140 - 1/30s shutter; View-DR/VE On; AGC 'Middle'; fw 1.26.01

Here are the five (5) key scenarios examined:

• Even Artificial Lighting
• Back-lit/WDR via Open Window
• Front-lit via Open Window
• Low-Light w/ Default Settings
• Low-Light w/ Normalized Settings

All five (5) scenarios, employed the same choreagraphed activity. The activity involves, at the start, one human subject entering the room from a doorway (center room) and another subject positioned at the far corner of the room. Both subjects progress toward the center of the room and then toward nearer field of views until finally exiting scene.