Optimized Camera Shootout

Author: Benros Emata, Published on Jun 05, 2011

If you optimize a camera's settings, how much can you improve its performance? An interesting and important question, this report provides concrete evidence based on our systematic experimentation.

Over the last two months, our camera shootouts have become our most popular series ever. In them, we use the camera's default settings. The advantage of this includes: (1) replicating an 'out of the box' setup, (2) reflecting common usage patterns (i.e., most users do minimal 'tweaking'), (3) easy for others to replicate and (4) ensures no 'tricks' might unfairly help or hinder manufacturers.

On the other hand, a number of people have protested our use of camera defaults. They assured us that if we 'tweaked' their cameras that the poor quality images that our tests documented would be rectified.

We were quite interested to see how much of a difference optimizations would make. We contacted all the manufacturers of the cameras used in our parking garage shootout (Axis, Arecont, Bosch, IQinVision, and Sony). They each provided recommendations to optimize their particular camera for a parking garage environment (specifically the parking garage in our test). This report compares the manufacturer recommended settings to camera defaults.

Below is an image from our testing scene showing the presence of an outdoor opening and artificial lighting. The cameras were mounted at ceiling level aimed directly down the middle of the driving lane to capture people and cars moving through the garage.

This is the same scene as in the original report. An important point to emphasize is that is a fairly wide scene - more relevant to real world surveillance, unlike the tiny FoVs common in manufacturer marketing images. Below is the overhead map depicting distances, the various subject locations, and pixels/foot values at several key locations:

To systematically spot differences, we examined performance differences across 15 different scenarios / comparisons:

  • Midday Back Lighting (Far Vehicle) - All Cameras, Default vs Optimized
  • Midday Back Lighting (Near Vehicle) - All Cameras, Default vs Optimized
  • Midday Even Lighting (Far Human) - All Cameras, Default vs Optimized
  • Midday Even Lighting (Near Human) - All Cameras, Default vs Optimized
  • Nighttime Color (Far Vehicle) - All Cameras, Default vs Optimized
  • Nighttime Color (Near Vehicle) - All Cameras, Default vs Optimized
  • Nighttime Color (Far Human) - All Cameras, Default vs Optimized
  • Nighttime Color (Near Human) - All Cameras, Default vs Optimized
  • Arecont Default vs Optimized
  • Axis Default vs Optimized
  • Bosch Default vs Optimized
  • Sony Default vs Optimized
  • Exposure Region Impact (Far Field)
  • Exposure Region Impact (Near Field)
  • Setting Experiments

We prepared 15 comparisons slides like the one below to better understand and document the level of change from a camera with its default settings to an optimized one using the manufacturer's recommendations.

Even for ourselves, seeing and documenting differences across multiple cameras and multiple times of day showed the potential and limits of camera optimizations. Inside, the Pro section we break down the specifics.

Key Findings and Recommendations

Get Video Surveillance News In Your Inbox
Get Video Surveillance News In Your Inbox

The following are our key findings:

  • Majority of optimizations produce very subtle/superficial changes to image quality, and may have little to no impact on investigations (specifically, revealing concrete details such as facial features or license plates).
  • From an aesthetic or subjective viewpoint, differences can be significant.
  • Subtle differences, due to optimizations, tend to have a more material impact as field of view narrows.
  • Among many types of optimizations those involving exposure compensation (e.g. SmartBLC, Exposure Zones etc.) tend to have the most pronounced affect on image quality and investigative outcomes.
  • Optimizations involving exposure compensation may tend to limit the overall area over which the camera is 'effective' in scenes with wide lighting variation (e.g. a WDR scene), and may also become more of a disadvantage as scenes transition from day to night or vice versa.

In light of these findings we recommend the following:

  • If you have a scene where you cannot make out a license plate or facial features (such as was common in our parking garage scene), optimizing camera software settings is unlikely to make a material difference (i.e., 'reveal' those details).
  • "Tweaking" settings will not overcome fundamental challenges in the scene. Physical changes will almost always be necessary. Consider (1) moving the cameras' position to avoid light, (2) narrowing the Field of View to decrease coverage area or (3) adding in lighting or shading to even out conditions.
  • If you do not like the 'look' of the image, you should consider changing settings. We think this will be primarily a subjective judgment, unlikely to impact prosecution or system use. However, if the decision user prefers the video to look a certain way, optimizations can help.
  • Most 'tweaks' will make things worse - either right away or at a different time of day (i.e., the tweak that makes the video better during the day makes it significantly worse during the night). Be careful how you tweak and check the impact over time. If you make things worse, stop tweaking and factory default the camera's settings.

Midday Back Lighting Default vs. Optimized (Far Vehicle)

In this scenario, cameras are digitally zoomed into to a vehicle ~60ft out, corresponding to a HFoV of ~56ft. There is ~40,000 lux of back lighting and ~200 lux of illumination in the darker foreground area. As a result lighting conditions are considerably challenging.

Below is the Composite Snapshot (Download Full Video Clip):

Most noticeable improvements after 'optimizations' are seen in the 1.3MP D/N Arecont AV1315DN, the 720p D/N Bosch NBN-921, and the SD Bosch NBN-498. For example, the vehicle in the AV1315DN is quite underexposed in the default image, but appears better exposed in the 'optimized' image. However, this improvement is mostly superficial, and provides no real benefit. The 720p and SD Bosch's post 'optimization' images display more contrast and overall better exposure of the vehicle. But again, the improvement provides little to no real benefit. It is interesting to note the slight artificiality or unnaturalness of the 'optimized' Bosch images.

Midday Back Lighting Default vs. Optimized (Near Vehicle)

In this scenario, cameras are digitally zoomed into to a vehicle ~20ft out, corresponding to a HFoV of ~19ft. There is ~40,000 lux of back lighting and ~200 lux of illumination in the darker foreground area. As a result lighting conditions pose a considerable challenge.

Below is the Composite Snapshot (Download Full Video Clip):

The same cameras showing the most post 'optimization' improvements in the 'far' scenario, show the most improvements in this 'near' scenario. However, in the 'far' scenario improvements were mostly superficial, while in this 'near' scenario improvements provide a slight to moderate advantage. The AV1315DN default vs. optimized comparison shows more appropriate exposure levels on the optimized image. As such, the vehicle license plate can be captured with a fair degree of certainty. The Bosch's (both 720p and SD) demonstrate similar post 'optimization' benefits. Note that some cameras (i.e. 5MP Axis and SD Axis) actually declined in effectiveness - becoming underexposed post optimizations.

Midday Back Lighting Default vs. Optimized (Far Human)

In this scenario, cameras are digitally zoomed into to a human subject ~100ft out, corresponding to a HFoV of ~93ft. There is ~40,000 lux of back-light and ~200 lux of illumination in the darker foreground area. As a result lighting conditions pose a considerable challenge.

Below is the Composite Snapshot (Download Full Video Clip):

For most cameras in this scenario, post optimization does not materially affect the utility of the images - either positively or negatively. However, one camera in particular, the AV1315DN, reveals a major image quality issue after optimizations. The AV1315DN's default image detects the human subject and provides a modicum of details, whereas the optimized image produces practically no indication of the human subject whatsoever. As somewhat expected, the AV1315DN's optimization (exposure region in darker near field area), which may have proved beneficial when the subject was in a dark area, adversely affects image quality when the subject is positioned in a brighter area.

Midday Back Lighting Default vs. Optimized (Near Human)

In this scenario, cameras are digitally zoomed into to a human subject ~20ft out, corresponding to a HFoV of ~19ft. There is ~40,000 lux of back-light and ~200 lux of illumination in the darker foreground area. As a result lighting conditions pose a considerable challenge.

Below is the Composite Snapshot (Download Full Video Clip):

As evidenced in the composite, optimizations did not provide any meaningful improvements to image quality. Conversely, they did not provide any real deteriorations either. Perhaps, it can be argued that the AV1315DN did provide some level of improvement post optimization. Whereas, the default provided no color contrast, the optimized provides a level of color contrast and subsequent indication of complexion.

Nighttime Color Default vs. Optimized (Far Vehicle)

In this scenario, cameras are digitally zoomed into to a vehicle ~60ft out, corresponding to a HFoV of ~56ft. Note that illumination at the uncovered portion of the parking garage is ~3 lux, and that interior portions are illuminated by artificial lights (~10 to ~50 lux). Note that D/N cameras have been 'forced' to color mode.

Below is the Composite Snapshot (Download Full Video Clip):

While subtle qualitative differences can be detected across all cameras post optimization, none of these differences provide meaningful advantages for investigation purposes.

Nighttime Color Default vs. Optimized (Near Vehicle)

In this scenario, cameras are digitally zoomed into to a vehicle ~20ft out, corresponding to a HFoV of ~19ft. Note that illumination at the uncovered portion of the parking garage is ~3 lux, and that interior portions are illuminated by artificial lights (~10 to ~50 lux). Note that D/N cameras have been 'forced' to color mode.

Below is the Composite Snapshot (Download Full Video Clip):

Subtle differences which proved non-substantive, in the 'far' scene, begin to evidence meaningful benefits here in this 'near' scenario. The three (3) post optimization 'gainers' are the 2MP IQ042SI, 720p NBN-921, and 720p CH140. All three increase clarity on the license plate with a subsequent increased likelihood of capture. Interestingly, while the 720p Bosch shows improvement, its SD Bosch counterpart shows a slight loss of license plate capture ability. Other 'decliners' include the 3MP AV3135, and 1.3MP AV1315DN, which are both highly under-exposed post optimization. The rest of the field exhibit no meaningful changes to image quality one way or another (i.e. 5MP P1347 and SD M1011).

Nighttime Color Default vs. Optimized (Far Human)

In this scenario, cameras are digitally zoomed into to a human subject ~100ft out, corresponding to a HFoV of ~93ft. Note that illumination at the uncovered portion of the parking garage is ~3 lux, and that interior portions are illuminated by artificial lights (~10 to ~50 lux). Note that D/N cameras have been 'forced' to color mode.

Below is the Composite Snapshot (Download Full Video Clip):

Optimizations provided no material security benefits in any of the cameras. In the case of the 3MP AV3135, and 1.3MP AV1315DN their post optimized images are nearly black (extreme under-exposure). This may suggest an overall deterioration, but the default images are equally 'useless' as well.

Nighttime Color Default vs. Optimized (Near Human)

In this scenario, cameras are digitally zoomed into to a human subject ~20ft out, corresponding to a HFoV of ~19ft. Note that illumination at the uncovered portion of the parking garage is ~3 lux, and that interior portions are illuminated by artificial lights (~10 to ~50 lux). Note that D/N cameras have been 'forced' to color mode.

Below is the Composite Snapshot (Download Full Video Clip):

Overall the field is evenly split between those providing no change and those changing detrimentally post optimization. For the AV3135 and AV1315DN the deterioration is an exposure issue, wherein both cameras render the subject too dark post optimization. For the 2MP IQ042SI and 720p Bosch the deterioration is marked by an overall loss of clarity and contrast in the subject's facial details.

5MP D/N Axis P1347 Default vs. Optimized

In this composite, the Axis P1347 is examined individually. For each of eight (8) scenarios default and 'optimized' images are arranged/analyzed side by side. Optimizations include an increase in sharpness to +20, and a slight closure of the iris to increase depth of field.

Below is the Composite Snapshot:

Throughout most of the scenarios, no significant changes were discovered post-optimization. Indeed, the majority of comparisons between default vs. optimized can be classified as 'non-changers'. The most significant changes are seen in the daytime and nighttime 'near' vehicle scenarios. In the daytime the defaults provided strong license plate details with a corresponding high certainty of capture. After optimizations, however, the license plate becomes somewhat underexposed and certainty of plate capture drops significantly. Strangely, the trend somewhat reverses in the nighttime 'near' vehicle scene, wherein the optimized image of the license plate appears slightly clearer in comparison to the default.

1.3MP D/N Arecont AV1315DN Default vs. Optimized

In this composite, the Arecont AV1315DN is examined individually. For each of eight (8) scenarios default and 'optimized' images are arranged/analyzed side by side. Optimizations include a decrease in gamma, an increase in saturation, and an exposure region in the darker near field area of the scene.

Below is the Composite Snapshot:

Half of the scenes (4 of 8) exhibit no material differences in terms of security purposes. Two instances can be classified as providing a material benefit (daytime 'far' and 'near' vehicle) post optimization, and two instances as having a negative impact post optimization (daytime 'far' human and nighttime 'near' vehicle). For example, the default daytime 'near' vehicle is under-exposed and no ability for plate capture exists, whereas, the optimized version is more appropriately exposed and as such, offers a fair degree of plate capture ability. Interestingly, the same 'near' vehicle scenario performed at night tends to reverse the optimization benefit. In the nighttime 'near' vehicle the default is more appropriately exposed, while the optimized is under-exposed.

720p D/N Bosch NBN-921 Default vs. Optimized

In this composite, the Bosch NBN-921 is examined individually. For each of eight (8) scenarios default and 'optimized' images are arranged/analyzed side by side. Optimizations include a SmartBLC preset, ALC Peak Average increase to +4, ALC Sharpness increase to +5, and ALC decrease to -2.

Below is the Composite Snapshot:

What makes this composite interesting is that in our judgement, there are no scenarios in which optimizations produced material or significant levels of deterioration, while still providing two cases where functionality improved. Indeed, most post optimization scenarios appear rather neutral in their effects. Two scenarios, however, do exhibit detectable post optimization improvements. The daytime 'near' vehicle is better exposed specifically on the license plate post optimization, and yields better odds of plate capture. In the nighttime 'near' vehicle it is not better exposure characteristics that results in improvement, but rather better contrast and sharpness on the license plate.

720p D/N Sony CH140 Default vs. Optimized

In this composite, the Sony CH140 is examined individually. For each of eight (8) scenarios default and 'optimized' images are arranged/analyzed side by side. Only one optimization of increasing the bit-rate from 2mbps to 4mbps is involved.

Below is the Composite Snapshot:

Overall, optimizations provided neither real material benefits nor significant deteriorations. Only one scenario depicts some level of improvement, albeit not significant enough to influence investigative outcomes. This is the daytime 'far' human scenario in which it can be seen that the subject's overall appearance is 'smoother' or less 'blocky' in the optimized image than in the default image.

5MP D/N Axis P1347 Default vs. Exposure Zone on Car (Far)

In this scenario a default image is compared to an optimized image (exposure zone on car). The vehicle of interest is positioned ~60ft out, corresponding to a ~56ft HFoV.

Below is the composite snapshot:

The exposure zone on the vehicle results in an image that is considerably less exposed in comparison with the default image. It is difficult to declare a clear 'winner' in this case as both exhibit pros and cons. For example, the default is quite washed out in general, but manages to deliver a bit more detail of the front apron. On the downside, the default tends to diminish the vehicle body contours (over-exposure or blooming around the edges of the vehicle). The optimized image clearly provides the body contours of the vehicle but tends to diminish front apron details. Note that the optimized image provides much clearer details of the background beyond the vehicle (i.e. vegetation and retaining wall).

5MP D/N Axis P1347 Default vs. Exposure Zone on Car (Near)

In this scenario a default image is compared to an optimized image (exposure zone on car). The vehicle of interest is positioned ~20ft out, corresponding to a ~19ft HFoV.

Below is the composite snapshot:

The exposure zone configured on the vehicle effectively provides a 'darker' reference than the overall scene referenced by the default settings. As such the vehicle appears a bit under-exposed in comparison to the default. It appears also that gain and subsequent noise is introduced in the optimized image. These factors result in overall less clarity. Looking at the readability of the license plate between the two images demonstrates this quite well. The default provides a near absolute certainty of plate capture, while the optimized version only a slight level of certainty.

1.3MP D/N Arecont AV1315 - Various Settings Applied (Far)

In this scenario various settings are tested on a 'far' vehicle and human subject positioned ~60ft out, corresponding to a ~56ft HFoV. The particular setting applied to the camera are noted as an overlay text within the image.

Below is the composite snapshot:

None of the optimizations provide material benefits when compared to the first two default images (bw and color default). Of the two defaults, the color tends to provide a slight advantage, as it gives an indication of the subject's complexion. Improved overall contrast reveals body contours more clearly as well. Among all the optimizations performed, the exposure zone on the car results in the most pronounced change. This manifests in more apparent illumination, e.g. revealing slightly more details of the human subject and the retaining wall. However, its impact on investigative outcomes is arguably quite minimal.

Methodology

The key difference between this 'shootout' and previous ones is the additional execution of the choreographed activity for each scenario. The first 'run' is to capture video from cameras in default configurations and the additional 'run' is to capture video from 'optimized' cameras.

The following are the eight (8) cameras used in the 'Optimized Parking Garage Shootout':

  • Arecont AV1315DN (online $460) - 1.3MP D/N; 1/2.7" CMOS; F1.6; 0.1 Lux (BW)
  • Arecont AV3135 (online $883) - 3MP/1.3MP D/N; 1/2" CMOS; Tamron F1.6; 0.0 Lux (BW)
  • Axis M1011 (online $160) - 4CIF Color-only; 1/4" CMOS; Integrated F2.0; 1.0 Lux (Color)
  • Axis P1347 (online $1350) - 5MP D/N; 1/2.5" CMOS; F1.6 Kowa;0.08 Lux (BW)
  • Bosch NBN-921 (online $775) - 720p D/N; 1/3" CCD; F1.3 Bosch; 0.4 Lux (BW)
  • Bosch NBN-498 (online $700) - SD D/N; 1/3" CCD; F1.3 Bosch; 0.06 Lux (BW)
  • IQinVision IQ042SI (online $400) - 2MP Color-only; 1/3" CMOS; F1.6 IQinVision; 0.2 Lux
  • Sony CH140 (online $800) - 720p D/N; 1/3" CMOS; F1.2 Fujinon; 0.1 Lux (BW)

Each of the eight cameras were simultaneously recorded to an Exacqvision VMS during two (2) key scenarios. For each scenario cameras are initially set to default configurations while the choreographed activity is performed and recorded. Cameras are then 'optimized' according to manufacturer input and the activity is repeated. This provides 'equal' points of comparison for review and analysis. Note that all camera lenses were adjusted to a uniform lens angle (~50 degrees) and recorded/analyzed at their maximum resolutions.

Here are the key default settings for each camera:

  • Arecont AV1315DN - Automatic Exposure 'Balanced' (1/12.5s)
  • Arecont AV3135 - Automatic Exposure 'Balanced' (1/12.5s)
  • Axis M1011 - 1/6s Max Shutter
  • Axis P1347 - 1/6s Max Shutter
  • Bosch NBN-921 - 1/60s shutter; SensUp 4x; AGC on '15'
  • Bosch NBN-498 - 1/60s shutter; SensUp 4x; AGC on '15'
  • IQinVision IQ042SI -1/30 shutter; Color-only
  • Sony CH140 - 1/30s shutter; View-DR/VE On; AGC 'Middle'

Here are the Two (2) key scenarios we examined:

  • Midday Back Lighting (Daytime) - ~40,000 lux (uncovered area); ~200 lux (covered area)
  • Nighttime Color Mode (Nighttime) - ~3 lux (uncovered area); ~ 10 to 50 lux (covered area)

Each of the two (2) key scenarios feature the same choreographed activity. The activity features a human subject walking towards the camera rig from a 'far' location (~100ft) to a 'near' location (~20ft). Next, a vehicle subject begins driving in towards the camera rig from a 'far' location (~100ft) to a 'near' location' (~20ft). The scenarios are performed/recorded twice (once for defaults and once for 'optimized').

2 reports cite this report:

Directory of Camera Shootout Series on Dec 27, 2011
The following directory lists all of our camera shootouts. These shootouts pit 4 - 8 different surveillance cameras in simultaneous tests on real...
Optimized Camera Shootout - Upcoming and Released on May 16, 2011
The response to our camera shootout series has been phenomenal with more than 10,000 reads in the last month and growing.The main question or...
Comments : PRO Members only. Login. or Join.

Most Recent Industry Reports

Alexa Guard Expands Amazon's Security Offerings, Boosts ADT's Stock on Sep 21, 2018
Amazon is expanding their security offerings yet again, this time with Alexa Guard that delivers security audio analytics and a virtual "Fake...
UTC, Owner of Lenel, Acquires S2 on Sep 20, 2018
UTC now owns two of the biggest access control providers, one of integrator's most hated access control platforms, Lenel, and one of their...
BluePoint Aims To Bring Life-Safety Mind-Set To Police Pull Stations on Sep 20, 2018
Fire alarm pull stations are commonplace but police ones are not. A self-funded startup, BluePoint Alert Solutions is aiming to make police pull...
SIA Plays Dumb On OEMs And Hikua Ban on Sep 20, 2018
OEMs widely pretend to be 'manufacturers', deceiving their customers and putting them at risk for cybersecurity attacks and, soon, violation of US...
Axis Vs. Hikvision IR PTZ Shootout on Sep 20, 2018
Hikvision has their high-end dual-sensor DarkfighterX. Axis has their high-end concealed IR Q6125-LE. Which is better? We bought both and tested...
Avigilon Announces AI-Powered H5 Camera Development on Sep 19, 2018
Avigilon will be showcasing "next-generation AI" at next week's ASIS GSX. In an atypical move, the company is not actually releasing these...
Favorite Request-to-Exit (RTE) Manufacturers 2018 on Sep 19, 2018
Request To Exit devices like motion sensors and lock releasing push-buttons are a part of almost every access install, but who makes the equipment...
25% China Tariffs Finalized For 2019, 10% Start Now, Includes Select Video Surveillance on Sep 18, 2018
A surprise move: In July, when the most recent tariff round was first announced, the tariffs were only scheduled for 10%. However, now, the US...
Central Stations Face Off Against NFPA On Fire Monitoring on Sep 18, 2018
Central stations are facing off against the NFPA over what they call anti-competitive language in NFPA 72, the standard that covers fire alarms....
Hikvision USA Starts Layoffs on Sep 18, 2018
Hikvision USA has started layoffs, just weeks after the US government ban was passed into law. Inside this note, we examine: The important...

The world's leading video surveillance information source, IPVM provides the best reporting, testing and training for 10,000+ members globally. Dedicated to independent and objective information, we uniquely refuse any and all advertisements, sponsorship and consulting from manufacturers.

About | FAQ | Contact