Optimized Camera Shootout

If you optimize a camera's settings, how much can you improve its performance? An interesting and important question, this report provides concrete evidence based on our systematic experimentation.

Over the last two months, our camera shootouts have become our most popular series ever. In them, we use the camera's default settings. The advantage of this includes: (1) replicating an 'out of the box' setup, (2) reflecting common usage patterns (i.e., most users do minimal 'tweaking'), (3) easy for others to replicate and (4) ensures no 'tricks' might unfairly help or hinder manufacturers.

On the other hand, a number of people have protested our use of camera defaults. They assured us that if we 'tweaked' their cameras that the poor quality images that our tests documented would be rectified.

We were quite interested to see how much of a difference optimizations would make. We contacted all the manufacturers of the cameras used in our parking garage shootout (Axis, Arecont, Bosch, IQinVision, and Sony). They each provided recommendations to optimize their particular camera for a parking garage environment (specifically the parking garage in our test). This report compares the manufacturer recommended settings to camera defaults.

Below is an image from our testing scene showing the presence of an outdoor opening and artificial lighting. The cameras were mounted at ceiling level aimed directly down the middle of the driving lane to capture people and cars moving through the garage.

This is the same scene as in the original report. An important point to emphasize is that is a fairly wide scene - more relevant to real world surveillance, unlike the tiny FoVs common in manufacturer marketing images. Below is the overhead map depicting distances, the various subject locations, and pixels/foot values at several key locations:

To systematically spot differences, we examined performance differences across 15 different scenarios / comparisons:

• Midday Back Lighting (Far Vehicle) - All Cameras, Default vs Optimized
• Midday Back Lighting (Near Vehicle) - All Cameras, Default vs Optimized
• Midday Even Lighting (Far Human) - All Cameras, Default vs Optimized
• Midday Even Lighting (Near Human) - All Cameras, Default vs Optimized
• Nighttime Color (Far Vehicle) - All Cameras, Default vs Optimized
• Nighttime Color (Near Vehicle) - All Cameras, Default vs Optimized
• Nighttime Color (Far Human) - All Cameras, Default vs Optimized
• Nighttime Color (Near Human) - All Cameras, Default vs Optimized
• Arecont Default vs Optimized
• Axis Default vs Optimized
• Bosch Default vs Optimized
• Sony Default vs Optimized
• Exposure Region Impact (Far Field)
• Exposure Region Impact (Near Field)
• Setting Experiments

We prepared 15 comparisons slides like the one below to better understand and document the level of change from a camera with its default settings to an optimized one using the manufacturer's recommendations.

The following are our key findings:

• Majority of optimizations produce very subtle/superficial changes to image quality, and may have little to no impact on investigations (specifically, revealing concrete details such as facial features or license plates).
• From an aesthetic or subjective viewpoint, differences can be significant.
• Subtle differences, due to optimizations, tend to have a more material impact as field of view narrows.
• Among many types of optimizations those involving exposure compensation (e.g. SmartBLC, Exposure Zones etc.) tend to have the most pronounced affect on image quality and investigative outcomes.
• Optimizations involving exposure compensation may tend to limit the overall area over which the camera is 'effective' in scenes with wide lighting variation (e.g. a WDR scene), and may also become more of a disadvantage as scenes transition from day to night or vice versa.

In light of these findings we recommend the following:

• If you have a scene where you cannot make out a license plate or facial features (such as was common in our parking garage scene), optimizing camera software settings is unlikely to make a material difference (i.e., 'reveal' those details).
• "Tweaking" settings will not overcome fundamental challenges in the scene. Physical changes will almost always be necessary. Consider (1) moving the cameras' position to avoid light, (2) narrowing the Field of View to decrease coverage area or (3) adding in lighting or shading to even out conditions.
• If you do not like the 'look' of the image, you should consider changing settings. We think this will be primarily a subjective judgment, unlikely to impact prosecution or system use. However, if the decision user prefers the video to look a certain way, optimizations can help.
• Most 'tweaks' will make things worse - either right away or at a different time of day (i.e., the tweak that makes the video better during the day makes it significantly worse during the night). Be careful how you tweak and check the impact over time. If you make things worse, stop tweaking and factory default the camera's settings.

Midday Back Lighting Default vs. Optimized (Far Vehicle)

In this scenario, cameras are digitally zoomed into to a vehicle ~60ft out, corresponding to a HFoV of ~56ft. There is ~40,000 lux of back lighting and ~200 lux of illumination in the darker foreground area. As a result lighting conditions are considerably challenging.

Below is the Composite Snapshot (Download Full Video Clip [link no longer available]):

Most noticeable improvements after 'optimizations' are seen in the 1.3MP D/N Arecont AV1315DN, the 720p D/N Bosch NBN-921, and the SD Bosch NBN-498. For example, the vehicle in the AV1315DN is quite underexposed in the default image, but appears better exposed in the 'optimized' image. However, this improvement is mostly superficial, and provides no real benefit. The 720p and SD Bosch's post 'optimization' images display more contrast and overall better exposure of the vehicle. But again, the improvement provides little to no real benefit. It is interesting to note the slight artificiality or unnaturalness of the 'optimized' Bosch images.

Midday Back Lighting Default vs. Optimized (Near Vehicle)

In this scenario, cameras are digitally zoomed into to a vehicle ~20ft out, corresponding to a HFoV of ~19ft. There is ~40,000 lux of back lighting and ~200 lux of illumination in the darker foreground area. As a result lighting conditions pose a considerable challenge.

Below is the Composite Snapshot (Download Full Video Clip [link no longer available]):

The same cameras showing the most post 'optimization' improvements in the 'far' scenario, show the most improvements in this 'near' scenario. However, in the 'far' scenario improvements were mostly superficial, while in this 'near' scenario improvements provide a slight to moderate advantage. The AV1315DN default vs. optimized comparison shows more appropriate exposure levels on the optimized image. As such, the vehicle license plate can be captured with a fair degree of certainty. The Bosch's (both 720p and SD) demonstrate similar post 'optimization' benefits. Note that some cameras (i.e. 5MP Axis and SD Axis) actually declined in effectiveness - becoming underexposed post optimizations.

Midday Back Lighting Default vs. Optimized (Far Human)

In this scenario, cameras are digitally zoomed into to a human subject ~100ft out, corresponding to a HFoV of ~93ft. There is ~40,000 lux of back-light and ~200 lux of illumination in the darker foreground area. As a result lighting conditions pose a considerable challenge.

Below is the Composite Snapshot (Download Full Video Clip [link no longer available]):

For most cameras in this scenario, post optimization does not materially affect the utility of the images - either positively or negatively. However, one camera in particular, the AV1315DN, reveals a major image quality issue after optimizations. The AV1315DN's default image detects the human subject and provides a modicum of details, whereas the optimized image produces practically no indication of the human subject whatsoever. As somewhat expected, the AV1315DN's optimization (exposure region in darker near field area), which may have proved beneficial when the subject was in a dark area, adversely affects image quality when the subject is positioned in a brighter area.

Midday Back Lighting Default vs. Optimized (Near Human)

In this scenario, cameras are digitally zoomed into to a human subject ~20ft out, corresponding to a HFoV of ~19ft. There is ~40,000 lux of back-light and ~200 lux of illumination in the darker foreground area. As a result lighting conditions pose a considerable challenge.

Below is the Composite Snapshot (Download Full Video Clip [link no longer available]):

As evidenced in the composite, optimizations did not provide any meaningful improvements to image quality. Conversely, they did not provide any real deteriorations either. Perhaps, it can be argued that the AV1315DN did provide some level of improvement post optimization. Whereas, the default provided no color contrast, the optimized provides a level of color contrast and subsequent indication of complexion.

Nighttime Color Default vs. Optimized (Far Vehicle)

In this scenario, cameras are digitally zoomed into to a vehicle ~60ft out, corresponding to a HFoV of ~56ft. Note that illumination at the uncovered portion of the parking garage is ~3 lux, and that interior portions are illuminated by artificial lights (~10 to ~50 lux). Note that D/N cameras have been 'forced' to color mode.

Below is the Composite Snapshot (Download Full Video Clip [link no longer available]):

While subtle qualitative differences can be detected across all cameras post optimization, none of these differences provide meaningful advantages for investigation purposes.

Nighttime Color Default vs. Optimized (Near Vehicle)

In this scenario, cameras are digitally zoomed into to a vehicle ~20ft out, corresponding to a HFoV of ~19ft. Note that illumination at the uncovered portion of the parking garage is ~3 lux, and that interior portions are illuminated by artificial lights (~10 to ~50 lux). Note that D/N cameras have been 'forced' to color mode.

Below is the Composite Snapshot (Download Full Video Clip [link no longer available]):

Subtle differences which proved non-substantive, in the 'far' scene, begin to evidence meaningful benefits here in this 'near' scenario. The three (3) post optimization 'gainers' are the 2MP IQ042SI, 720p NBN-921, and 720p CH140. All three increase clarity on the license plate with a subsequent increased likelihood of capture. Interestingly, while the 720p Bosch shows improvement, its SD Bosch counterpart shows a slight loss of license plate capture ability. Other 'decliners' include the 3MP AV3135, and 1.3MP AV1315DN, which are both highly under-exposed post optimization. The rest of the field exhibit no meaningful changes to image quality one way or another (i.e. 5MP P1347 and SD M1011).

Nighttime Color Default vs. Optimized (Far Human)

In this scenario, cameras are digitally zoomed into to a human subject ~100ft out, corresponding to a HFoV of ~93ft. Note that illumination at the uncovered portion of the parking garage is ~3 lux, and that interior portions are illuminated by artificial lights (~10 to ~50 lux). Note that D/N cameras have been 'forced' to color mode.

Below is the Composite Snapshot (Download Full Video Clip [link no longer available]):

Optimizations provided no material security benefits in any of the cameras. In the case of the 3MP AV3135, and 1.3MP AV1315DN their post optimized images are nearly black (extreme under-exposure). This may suggest an overall deterioration, but the default images are equally 'useless' as well.

Nighttime Color Default vs. Optimized (Near Human)

In this scenario, cameras are digitally zoomed into to a human subject ~20ft out, corresponding to a HFoV of ~19ft. Note that illumination at the uncovered portion of the parking garage is ~3 lux, and that interior portions are illuminated by artificial lights (~10 to ~50 lux). Note that D/N cameras have been 'forced' to color mode.

Below is the Composite Snapshot (Download Full Video Clip [link no longer available]):

Overall the field is evenly split between those providing no change and those changing detrimentally post optimization. For the AV3135 and AV1315DN the deterioration is an exposure issue, wherein both cameras render the subject too dark post optimization. For the 2MP IQ042SI and 720p Bosch the deterioration is marked by an overall loss of clarity and contrast in the subject's facial details.

5MP D/N Axis P1347 Default vs. Optimized

In this composite, the Axis P1347 is examined individually. For each of eight (8) scenarios default and 'optimized' images are arranged/analyzed side by side. Optimizations include an increase in sharpness to +20, and a slight closure of the iris to increase depth of field.

Below is the Composite Snapshot:

Throughout most of the scenarios, no significant changes were discovered post-optimization. Indeed, the majority of comparisons between default vs. optimized can be classified as 'non-changers'. The most significant changes are seen in the daytime and nighttime 'near' vehicle scenarios. In the daytime the defaults provided strong license plate details with a corresponding high certainty of capture. After optimizations, however, the license plate becomes somewhat underexposed and certainty of plate capture drops significantly. Strangely, the trend somewhat reverses in the nighttime 'near' vehicle scene, wherein the optimized image of the license plate appears slightly clearer in comparison to the default.

1.3MP D/N Arecont AV1315DN Default vs. Optimized

In this composite, the Arecont AV1315DN is examined individually. For each of eight (8) scenarios default and 'optimized' images are arranged/analyzed side by side. Optimizations include a decrease in gamma, an increase in saturation, and an exposure region in the darker near field area of the scene.

Below is the Composite Snapshot:

Half of the scenes (4 of 8) exhibit no material differences in terms of security purposes. Two instances can be classified as providing a material benefit (daytime 'far' and 'near' vehicle) post optimization, and two instances as having a negative impact post optimization (daytime 'far' human and nighttime 'near' vehicle). For example, the default daytime 'near' vehicle is under-exposed and no ability for plate capture exists, whereas, the optimized version is more appropriately exposed and as such, offers a fair degree of plate capture ability. Interestingly, the same 'near' vehicle scenario performed at night tends to reverse the optimization benefit. In the nighttime 'near' vehicle the default is more appropriately exposed, while the optimized is under-exposed.

720p D/N Bosch NBN-921 Default vs. Optimized

In this composite, the Bosch NBN-921 is examined individually. For each of eight (8) scenarios default and 'optimized' images are arranged/analyzed side by side. Optimizations include a SmartBLC preset, ALC Peak Average increase to +4, ALC Sharpness increase to +5, and ALC decrease to -2.

Below is the Composite Snapshot:

What makes this composite interesting is that in our judgement, there are no scenarios in which optimizations produced material or significant levels of deterioration, while still providing two cases where functionality improved. Indeed, most post optimization scenarios appear rather neutral in their effects. Two scenarios, however, do exhibit detectable post optimization improvements. The daytime 'near' vehicle is better exposed specifically on the license plate post optimization, and yields better odds of plate capture. In the nighttime 'near' vehicle it is not better exposure characteristics that results in improvement, but rather better contrast and sharpness on the license plate.

720p D/N Sony CH140 Default vs. Optimized

In this composite, the Sony CH140 is examined individually. For each of eight (8) scenarios default and 'optimized' images are arranged/analyzed side by side. Only one optimization of increasing the bit-rate from 2mbps to 4mbps is involved.

Below is the Composite Snapshot:

Overall, optimizations provided neither real material benefits nor significant deteriorations. Only one scenario depicts some level of improvement, albeit not significant enough to influence investigative outcomes. This is the daytime 'far' human scenario in which it can be seen that the subject's overall appearance is 'smoother' or less 'blocky' in the optimized image than in the default image.

 

5MP D/N Axis P1347  Default vs. Exposure Zone on Car (Far)

In this scenario a default image is compared to an optimized image (exposure zone on car). The vehicle of interest is positioned ~60ft out, corresponding to a ~56ft HFoV.

Below is the composite snapshot:

The exposure zone on the vehicle results in an image that is considerably less exposed in comparison with the default image. It is difficult to declare a clear 'winner' in this case as both exhibit pros and cons. For example, the default is quite washed out in general, but manages to deliver a bit more detail of the front apron. On the downside, the default tends to diminish the vehicle body contours (over-exposure or blooming around the edges of the vehicle). The optimized image clearly provides the body contours of the vehicle but tends to diminish front apron details. Note that the optimized image provides much clearer details of the background beyond the vehicle (i.e. vegetation and retaining wall).

5MP D/N Axis P1347  Default vs. Exposure Zone on Car (Near)

In this scenario a default image is compared to an optimized image (exposure zone on car). The vehicle of interest is positioned ~20ft out, corresponding to a ~19ft HFoV.

Below is the composite snapshot:

The exposure zone configured on the vehicle effectively provides a 'darker' reference than the overall scene referenced by the default settings. As such the vehicle appears a bit under-exposed in comparison to the default. It appears also that gain and subsequent noise is introduced in the optimized image. These factors result in overall less clarity. Looking at the readability of the license plate between the two images demonstrates this quite well. The default provides a near absolute certainty of plate capture, while the optimized version only a slight level of certainty.

1.3MP D/N Arecont AV1315 - Various Settings Applied (Far)

In this scenario various settings are tested on a 'far' vehicle and human subject positioned ~60ft out, corresponding to a ~56ft HFoV. The particular setting applied to the camera are noted as an overlay text within the image.

Below is the composite snapshot:

None of the optimizations provide material benefits when compared to the first two default images (bw and color default). Of the two defaults, the color tends to provide a slight advantage, as it gives an indication of the subject's complexion. Improved overall contrast reveals body contours more clearly as well. Among all the optimizations performed, the exposure zone on the car results in the most pronounced change. This manifests in more apparent illumination, e.g. revealing slightly more details of the human subject and the retaining wall. However, its impact on investigative outcomes is arguably quite minimal.

Methodology

The key difference between this 'shootout' and previous ones is the additional execution of the choreographed activity for each scenario. The first 'run' is to capture video from cameras in default configurations and the additional 'run' is to capture video from 'optimized' cameras.

The following are the eight (8) cameras used in the 'Optimized Parking Garage Shootout':

• Arecont AV1315DN (online $460) - 1.3MP D/N; 1/2.7" CMOS; F1.6; 0.1 Lux (BW)
• Arecont AV3135 (online $883) - 3MP/1.3MP D/N; 1/2" CMOS; Tamron F1.6; 0.0 Lux (BW)
• Axis M1011 (online $160) -  4CIF Color-only; 1/4" CMOS; Integrated F2.0; 1.0 Lux (Color)
• Axis P1347 (online $1350) - 5MP D/N; 1/2.5" CMOS; F1.6 Kowa;0.08 Lux (BW)
• Bosch NBN-921 [link no longer available] (online $775) - 720p D/N; 1/3" CCD; F1.3 Bosch; 0.4 Lux (BW)
• Bosch NBN-498 [link no longer available] (online $700) - SD D/N; 1/3" CCD; F1.3 Bosch; 0.06 Lux (BW)
• IQinVision IQ042SI (online $400) - 2MP Color-only; 1/3" CMOS; F1.6 IQinVision; 0.2 Lux
• Sony CH140 (online $800) - 720p D/N; 1/3" CMOS; F1.2 Fujinon; 0.1 Lux (BW)

Each of the eight cameras were simultaneously recorded to an Exacqvision VMS during two (2) key scenarios. For each scenario cameras are initially set to default configurations while the choreographed activity is performed and recorded. Cameras are then 'optimized' according to manufacturer input and the activity is repeated. This provides 'equal' points of comparison for review and analysis.  Note that all camera lenses were adjusted to a uniform lens angle (~50 degrees) and recorded/analyzed at their maximum resolutions.

Here are the key default settings for each camera:

• Arecont AV1315DN - Automatic Exposure 'Balanced' (1/12.5s)
• Arecont AV3135 - Automatic Exposure 'Balanced' (1/12.5s)
• Axis M1011 - 1/6s Max Shutter
• Axis P1347 - 1/6s Max Shutter
• Bosch NBN-921 - 1/60s shutter; SensUp 4x; AGC on '15'
• Bosch NBN-498 -  1/60s shutter; SensUp 4x; AGC on '15'
• IQinVision IQ042SI -1/30 shutter; Color-only
• Sony CH140 - 1/30s shutter; View-DR/VE On; AGC 'Middle'

Here are the Two (2) key scenarios we examined:

• Midday Back Lighting (Daytime) - ~40,000 lux (uncovered area); ~200 lux (covered area)
• Nighttime Color Mode (Nighttime) - ~3 lux (uncovered area); ~ 10 to 50 lux (covered area)

Each of the two (2) key scenarios feature the same choreographed activity. The activity features a human subject walking towards the camera rig from a 'far' location (~100ft) to  a 'near' location (~20ft). Next, a vehicle subject begins driving in towards the camera rig from a 'far' location (~100ft) to a 'near' location' (~20ft). The scenarios are performed/recorded twice (once for defaults and once for 'optimized').