Bandwidth vs. Image Quality Shootout 2011

When thinking about maximizing image quality, resolution is usually the first thing that comes to mind. However, resolution is not the only factor that impacts quality. The amount of bandwidth available and used can have a dramatic impact on image quality. In this report, we examine bandwidth and the effect that it has on quality across numerous cameras.

Which Image Quality is Better?

To better understand image quality, let's start by examining two samples of the same scene side by side:

Consider two questions:

• Which camera has higher resolution? A or B?
• Which camera is better? A or B?

It is pretty obvious that the image from Camera B is better so this should be a simple case.

The reality is that those images are from the same camera at the same resolution and frame rate (720p/30). All that was done to the camera was changing the Constant Bit Rate target from 512 Kb/s to 8 Mb/s.

Factors Impacting Quality

Even with the same resolution, two common settings impact quality:

• Bit Rate: Most cameras can have their bit rate adjusted to specific levels (e.g., 512 Kb/s, 2 Mb/s, 8Mb/s, etc.)
• Quantization Level: Most cameras can have the level of compression adjusted (often called a quality or compression setting with options from 1-10 or 0-100)

Typically, these are mutually exclusive. If you lock in bit rate, the camera will automatically adjust the quantization level to not exceed the bandwidth set. Vice versa, if you set the quantization level, the camera will automatically change the bandwidth consumed to make sure the quality / compression always stays at the same level.

Our Test Process

We wanted to better understand how changes in these two factors impact video quality. To do so, we did a series of tests with three HD cameras: the Axis P1344, the Sony CH140 and the Bosch NBN-921.

For the bandwidth tests, we tested each camera at the following levels:

• 512 Kb/s
• 1 Mb/s
• 2 Mb/s
• 4 Mb/s
• 8 Mb/s

We did this across a series of scenes to see how quality would vary in different conditions:

• Daytime Indoors (300 lux)
• Nighttime Indoors (.5 lux)
• Daytime Intersection

Finally, we did a similar series of tests varying the quality level of a VBR camera (the Axis across 0, 30, 60 and 100 levels) to better understand changes in quality and bandwidth consumption.

Questions We Considered

Here are the questions we considered and answered inside the report:

• When changing bit rates, how much are image details impacted?
• How much quality differences exist among manufacturers at the same bit rates?
• How do manufacturer streaming methods differ? How does that impact quality and performance?
• What is the 'optimal' bit rate to use?
• When should you consider using higher bit rates?
• When adjusting compression levels, how much does bandwidth consumption and image quality vary?
• What is the 'right' compression level to use?

Key Findings and Recommendations

• Major, obvious, quality differences exist when increasing bandwidth from 512 kb/s to 2Mb/s across all 720p/30 cameras. Minor improvements occured in scenes between 2Mb/s and 8 Mb/s.
• Comparing image quality across cameras at the same bit rate setting is not feasible as manufacturers differ in how they adjust bit rate, frame rate and compression levels.
• Bandwidth variations exhibit less of an image quality impact in low-light scenes, where image details are already reduced by limited light.
• Bandwidth tends to have a greater impact in wider FoVs (i.e. where object size is small relative to the FoV)
• Bandwidth exhibits a greater material impact on moving objects versus static ones.
• In wider FoVs increasing bit-rates may tend to improve image details, but the level required to do so is quite high (e.g. going from 2mbps to 6mpbs). Done on a large scale this could significantly increase system cost.
• Implementation of CBR, and their practical effects, varied across the three manufacturers. Axis CBR streaming has no means to 'lock' in frame-rate. Sony's CBR was conventional, allowing frame rate to be locked in. Bosch's CBR allowed actual bit-rates to vary modestly from the user defined bit-rate. Frame rates could, effectively, be locked in.
• For Axis VBR compression levels, the default 30 provided a good balance between bandwidth consumption and image quality. Lower compression levels (e.g., 0) exploded bandwidth consumption.

In light of these findings we recommend the following:

• For 720p/30 streams, 2 Mb/s looks to be the 'sweet spot' bandwidth setting, providing the most details per bandwidth spent across all 3 cameras.
• Increasing bandwidth consumption from 2 Mb/s to 8 Mb/s will capture slightly more details but will increase bandwidth consumption by 2TBs per month for a continuously recording schedule. For most users and scenes, this is likely not worth the cost.
• If a high minimum frame-rate is desired, consider cameras that support a more 'conventional' CBR implementation (e.g. the Sony CH140 or Bosch NBN-921). The Axis P1344's CBR implementation, e.g., can drop frames.
• If consistent image quality rather than fps is more the concern consider VBR
• Consider CBR in scenes involving low-light to prevent unwanted bandwidth spikes
• If using VBR in scenes involving low-light or 24 hour surveillance (both low-light and well-lit times of day), consider a slight to moderate increase in compression levels to mitigate the degree of bandwidth spikes. Also consider reducing max gain and/or increasing exposure.

Why Does Bandwidth Impact Quality?

Please read this important preface:

Just because you lock the resolution and frame rate does not mean that the bandwidth or quality will remain constant. These can vary greatly because the camera can and will automatically adjust the amount of compression (or detail removal).

Principle #1:

The more complex the scene, the more bandwidth it takes to encode the scene. If you do not have enough bandwidth, quality will suffer - always. Only question is how visible the quality reduction will be.

Principle #2:

The more objects and movement in the scene, the more complex the scene is. This could be from people walking through the scene, leaves swaying in the wind, cars driving or even visible noise in low light. The simplest scene is pointing a camera at a white wall. Add in any of these other factors and complexity goes up. More complexity, more bandwidth is needed.

Principle #3:

Even with the resolution and frame rate set, the camera must set a specific compression level. This might sound counter intuitive but it is the case. Technically, this is usually called quantization. Sometimes, it is referred to as the detail removal factor. This scale sets how 'heavy' the compression. This occurs in various CODECs (MJPEG, MPEG-4, H264, etc.). H.264 has a scale ranging from 0 to 51 with 0 being minimum compression and 51 being maximum compression (details: see quantization low level tech paper [link no longer available]).

Principle #4:

The less bandwidth available and the more complex the scene, the more details need to be removed (i.e., the higher the compression level). Let's look at this in action:

When looking at this image, notice the following:

• DRF is Detail Removal Factor, the amount of compression / quantization occuring. The higher the number, the more details being removed. 
• DRF drops as bandwidth increases (look down each row).
• At any bandwidth level, the DRF is higher for the intersection than the indoor wall because the intersection is a more complex scene (compare each row right to left).
• As bandwidth increases, DRF drops and visual details increase (look carefully at the sample images provided).

You can actually run these numbers with your own cameras. Export AVIs from your VMS and try opening them in Avinaptics. It does not always work but it can be help you understand what is going on and how compressed the image is.

Optimizing CBR and VBR in Axis, Bosch, and Sony

In this video we examine how to optimize CBR and VBR settings within the web interfaces of the three cameras used in our study. The P1344 is unique among the group, because it supports both a VBR and CBR method. One key thing to note is that the P1344's CBR implementation does not allow frame-rates to be fixed or locked in. In contrast both the NBN-921 and CH140 CBR methods will lock in a specified frame-rate. In the last portion of the video we examine the three cameras in action to demonstrate the practical implications of their manufacturer/model specific CBR implementations.

CBR Image Quality Analysis (Axis P1344)

In this composite we examine the Axis P1344's image quality at five different CBR bit-rates (0.5, 1, 2, 4, and 8 mbps) across three different scenes (indoor low-light, indoor daytime, and busy intersection). Images are digitally zoomed to better reveal differences in detail. The reader is encouraged to click on the image to enlarge the composite for closer examination.

The P1344's CBR uses two main strategies to maintain bit-rates close to the user defined bit-rates: 1) drops frames; 2) increase compression. The result is that the majority of scenes operate below full motion video of 30fps. But more importantly, image quality will tend to vary less across the spectrum of user defined CBR caps compared to other CBR implementations (e.g. Sony CH140 and Bosch NBN-921), because the P1344 allows for the discarding of frames in lieu of increasing the compression level.

In the low-light scenario compression artifacts are quite noticeable at 512kbps and 1mbps and begin to clear up at 2mbps and beyond. However, the practical impact of increasing bit-rates is quite minimal as the scene's sub-optimal lighting imposes a relatively low upper limit on potential image quality.

In the indoor daytime scenario, compression artifacts steadily decrease as the bit-rate increases. However, in this well-lit and relatively narrow FoV the additional details are of little practical consequence as the artifacts clear up. Interstingly, a plateau in measured bit-rate and fps is reached at the 2mbps level, where subsequent increases in user defined CBR caps, result in no change in frame-rate, and little changes to the actual bit-rate and image quality.

In the higher complexity intersection scene, the same general pattern is observed of decreasing artifacts as user defined CBR caps are increased. However, a distinct plateau of measured bit-rate and resulting image quality found in the indoor less complex scene is absent here in this more complex one. Indeed, a fairly noticeable improvement still occurs from the 2mbps to 4mbps mark.

CBR Image Quality Analysis (Bosch NBN-921)

In this composite we examine the Bosch NBN-921's image quality at five different CBR bit-rates (0.5, 1, 2, 4, and 8 mbps) across three different scenes (indoor low-light, indoor daytime, and busy intersection). Images are digitally zoomed to better reveal differences in detail. The reader is encouraged to click on the image to enlarge the composite for closer examination.

At the low-end of 512kbps CBR, the Bosch NBN-921 exhibits considerably more compression artifacts than the Axis P1344 does at the same 512kbps level. However, this is expected as the NBN-921 maintains full motion 30fps video throughout, and relies exclusively on increases in compression level to keep below the cap.

Setting the NBN-921's maximum exposure to 1/30s, causes its image to darken considerably under the extreme low-light conditions (~0.5 lux). As such little to no practical differences are realized between the different CBR bit-rate caps in this particular environment.

In the indoor daytime scenario, compression artifacts are clearly evident at the 512kbps and 1mbps cap. At the 2mbps level, compression artifacts are minimal for static objects, but still persist to a degree for the hand/arm that is in motion. Despite measured bit-rates continuing to rise at the 4mbps and 8mbps caps, the image quality tends to plateau, with little to no gain in detail for the additional bandwidth consumed.

Artifacts and 'blockiness' are heavily apparent at the 512kbps rate in the complex intersection scene. Strangely, the measured bit-rate is only ~360kbps. The 1mbps image clears up rather dramatically and manifests an actual bit-rate (0.93mbps) much more aligned to the user defined cap. Image quality improves at each subsequent step towards 4mbps. Going from 4 to 8 mbps produces little detectable material improvements to the image.

CBR Image Quality Analysis (Sony CH-140)

In this composite we examine the Sony CH140's image quality at five different CBR bit-rates (0.5, 1, 2, 4, and 8 mbps) across three different scenes (indoor low-light, indoor daytime, and busy intersection). Images are digitally zoomed to better reveal differences in detail. The reader is encouraged to click on the image to enlarge the composite for closer examination.

 

As the low-light progression indicates, the practical impact of increasing bit-rates is quite minimal as the scene's sub-optimal lighting imposes a relatively low upper limit on potential image quality.

In the indoor daytime scenario, compression artifacts persist quite noticeably until the 2mbps mark. Although they continue to steadily decrease at each step between 2mbps and 8mbps, the practical impact is quite minimal at this narrow FoV.

In the more complex intersection scenario, at the 512kbps mark the camera is oddly streaming at ~750kbps (~50% above its user defined cap). The image is still quite 'blocky' and clears up rather distinctly at the 1mbps and 2mbps mark. Unlike the indoor test, however, there is still quite a reduction in compression artifacts going from 2 to 4 mbps.

VBR Image Quality Analysis (Axis P1344)

In this composite we examine the Axis P1344's image quality at four different VBR compression levels (0, 30, 60 and 100) across three different scenes (indoor low-light, indoor daytime, and busy intersection). 30 is the Axis default level. Lower numbers means less compression (e.g., 0) while greater numbers mean more compression (e.g., 60, 100).

Images are digitally zoomed to better reveal differences in detail. The reader is encouraged to click on the image to enlarge the composite for closer examination.

In the low-light scenario, the 0, 30 and 60 compression levels result in images of practically equal utility. The key point to observe is the resulting differences in bandwidth consumption. Notably the 0 compression level results in an inordinately high ~24mbps; increasing compression to 30 still results in an exorbitant ~12mbps bandwidth. At 60 a more reasonable ~4.5mbps is reached. Thus, going from a 0 to 60 compression level results in little utility loss, but reduces bandwidth consumption by ~80%.

In the indoor daytime scenario, the 0 to 30 compression change results in the most significant bandwidth drop (~85%) with little loss of overall utility of the image.

The intersection exhibits the same behavior as the indoor daytime (0 to 30 results in ~75% bandwidth savings with little loss of image utility).

It is important to note that the 'sweet-spot' for low-light and indoor-daytime/intersection can be at different compression levels (~60 for low-light, and ~30 for indoor-daytime/intersection). Applications that vary in complexity and lighting are subject to this dilemma and require a more thoughtful optimization approach.

Below is a graph of the P1344's Compression Level vs. Bandwidth (3 Scenes):

One key behavior here is that the most precipitous drop in bandwidth for the two well-lit scenes (indoor daytime and intersection) occurs from 0 to 30 compression, while the low-light curve decreases in a steadier fashion - not really converging until the 100 compression level. The two 'well-lit' curves converge to a relative minimum (a reasonable average of ~2.2 mbps) at the 30 compression level.

Based on an extrapolation of this graph a camera viewing a mixture of these three scenes throughout the day may find a compromise, e.g. at the 45 level of compression. Of course decreasing max gain and/or increasing max exposure can also help mitigate bandwidth spikes in the low-light environment, as well.

Methodology

Cameras were normalized to 1/30s maximum exposure lengths. Each camera lens was adjusted to provide a uniform HFoV for analysis (lens angle ~50 degress).

Here are the cameras used in 'Quality vs. Bandwidth' tests:

• Axis P1344 (online $759) - 720p D/N; 1/4" CMOS; F1.2 Computar ; 0.3/0.05 Lux (Color/BW)
• Bosch NBN-921 [link no longer available] (online $775) - 720p D/N; 1/3" CCD; F1.3 Bosch; 1.0/0.4 Lux (Color/BW)
• Sony CH140 (online $800) - 720p D/N; 1/3" CMOS; F1.2 Fujinon; 0.1 Lux (BW)

In the CBR testing, all three cameras recorded the same scene at a fixed user defined CBR bit-rate. The scene was repeated for each of five ascending CBR bit-rates (512kbps, 1mbps, 2mbps, 4mbps, and 8mbps). Both actual bit-rates and fps were documented for analysis.

In the VBR testing, only the P1344 was used. The same three primary scenes used in CBR testing were recorded. Each scene was repeated at each of four compression level settings (0, 30, 60, and 100).

Here are their key default CBR/VBR settings:

• Axis P1344 - VBR at a compression of 30 (0 - 100)
• Bosch NBN-921 - CBR with a target of 5000 kbps and a maximum of 10000 kbps
• Sony CH140 - CBR with a maximum of 2 mbps