Sony Camera Shootout 2011
Sony has a strong reputation for video quality in consumer electronics. Indeed, in our ongoing shootout series, a Sony surveillance camera, the CH140, has done quite well in many tests.
However, in the last year, Sony has significantly expanded their network surveillance camera lineup. The Sony camera we tested was the 'premium' model. Now, Sony has 'mid' and 'entry' level options. The immediate plus is that the newer cameras are much less expensive - ~$900 online pricing for the premium HD version versus ~$300 for the entry level HD version.
Important question remains: What is the tradeoff in quality between the different versions? What impact does this have on real world surveillance deployments? When would I use one versus the other?
In this test, we took 3 Sony cameras for a simultaneous shootout to answer these questions. The specific cameras tested were:
We tested in several scenarios, encompassing a wide variety of real-world surveillance environments (e.g. indoors, outdoors, evenly-lit, low-light, and WDR scenes). Below are some representative snapshots of a few of the test scenes.
We tested indoors in an artificially lit 'easy' scenario:
We tested outdoor in both day and night (shown below is night):
We also tested in a 'difficult' WDR scene:
Note this is the same test plan we did for an Axis camera shootout. The final segment in this mini-series will be an Axis vs Sony shootout where we pit these two brands against each other to see who's better in what scenarios.
Analyzing Video Quality
We analyzed a series of scenes to see how performance varied across the cameras. We digitally zoomed each camera to the same level to show differences. See an example scene comparison from our parking lot test.
We conducted our analysis across 10 scenes. They are:
- Indoor Even Artificial Lighting (Far)
- Indoor Even Artificial Lighting (Near)
- Indoor Low-Light (Default)
- Indoor Low-Light (Normalized)
- Outdoor Daytime Even Lighting (Far)
- Outdoor Daytime Even Lighting (Near)
- Outdoor Nighttime (Far)
- Outdoor Nighttime (Near)
- WDR Scene (Bright)
- WDR Scene (Dark)
Analyzing Bandwidth Consumption
Not only do we examine which cameras provide the best quality in each scene, we now break down the bandwidth consumption, showing who's the best and worse for the amount of network bandwidth and storage consumed.
Our analysis covers 8 metrics:
- Even Artificial Lighting Bandwidth Consumption Per Camera
- Low-Light (Default Settings) Bandwidth Consumption Per Camera
- Low-Light (Normalized Settings) Bandwidth Consumption Per Camera
- Outdoor Even Lighting Bandwidth Consumption Per Camera
- Outdoor Nighttime Bandwidth Consumption Per Camera
- WDR Scene Bandwidth Consumption Per Camera
- Average Bandwidth Consumption Per Scene
- Average Bandwidth Consumption Per Camera Overall
Key Findings for Each Camera's Performance
First, let's summarize the relative performance of each camera:
- Entry level HD CH110: Application is highly limited due to a very wide (80.7 degree) integrated fixed lens (much wider than most FoVs used in practice). Given this, high levels of detail can only be achieved at points very close to the camera. Low-light performance was also considerably worse than the premium and mid-tier 720p HD models when normalized (1/30s shutter).
- HD CH120: Relative to the premium CH140, the CH120 exhibited similar levels of detail for scenes with narrower FoVs involving even indoor lighting. However, the CH120 was shown to lag the CH140 in details at the wider FoV (e.g. 70ft) even under favorable lighting conditions. The CH120 was also substantially less effective than the CH140 in outdoor and WDR scenarios.
- HD CH140: Overall the CH140 provided the greatest image clarity and detail levels throughout the majority of scenarios. Advantages over the CH120 were least pronounced, and as such nearly equal, in favorable lighting at narrower FoVs. The CH140 separated itself quite impressively during WDR scenarios. There was slight separation to the upside in low-light scenarios, where the CH140 produced relatively less noise and overall crisper images.
- Bandwidth was not a differentiating factor as all cameras used Constant Bit Rate streaming and nearly equivalent bandwidth consumption.
While the CH120 and CH140 have the same resolution and are both Day / Night cameras, the effective details captures varies noticeably in scenes with strong lighting variances (such as in outdoor scenes and indoor/outdoor entrances).
Recommendations
When assessing image quality tradeoffs within these Sony camera offerings, we recommend the following:
- General Daytime use: For a relatively even lighting and a narrower FoV (e.g. ~31ft and less) the CH120 is comparable to the CH140 at significantly lower cost (~$300-$400 less). If the application accepts a very wide FoV and does not need to capture facial or license plate details, the very low cost CH110 may be acceptable. Alternatively, if the subject / target is within 5 - 10 feet from the camera, the CH110 also would be attractive.
- WDR use: The CH140 delivered notably superior performance.
- Outdoor use: The CH140 delivered moderately superior performance than the CH120. We believe this is a function of the CH140's superior WDR capability to handle better outdoor lighting variances.
- Night time use: Either the CH140 or the CH120. While the CH140 delivered less noisy images under low-light conditions than the CH120, differences were not significant.
Indoor Even Artificial Lighting (Far)
In this scenario, cameras are digitally zoomed into a human subject ~33ft out, corresponding to a HFoV of ~31ft. The scene is illuminated via artificial lighting and is evenly distributed (~200 lux). Note, because the CH110 has a fixed lens angle unequal to the other cameras, it has been left fully zoomed out.
Below is the composite snapshot (download full video clip [link no longer available]):
The CH140 and CH120 deliver similar if not equal levels of details. The subject is readily identifiable in both images. There are some subtle qualitative differences to note in picture quality. For instance the E series CH120 image appears both slightly 'sharper' and more color saturated.
Indoor Even Artificial Lighting (Near)
In this scenario, cameras are digitally zoomed into a human subject ~17ft out, corresponding to a HFoV of ~16ft. The scene is illuminated via artificial lighting and is evenly distributed (~200 lux). Note, because the CH110 has a fixed lens angle unequal to the other cameras, it has been left fully zoomed out.
Below is the composite snapshot (download full video clip [link no longer available]):
At this narrower FoV, the practical differences are again immaterial, as both the V series CH140 and E series CH120 deliver a highly identifiable image of the human subject. As in the previous scenario, the same subtle qualitative picture differences are repeated (i.e. the CH120 is both slightly sharper and more color saturated).
Below is the bandwidth utilization graph:
All cameras operated at the full 30fps maximum in this scenario. All cameras also operated essentially at their maximum CBR rate of 2Mbps.
Indoor Low-Light (Default)
In this scenario, cameras are digitally zoomed into a human subject ~33ft out, corresponding to a HFoV of ~31ft. The scene is 'dark' with essentially no lighting (~0.1 lux). All cameras are set to default settings. Note that the CH110 is color only. Also note, because the CH110 has a fixed lens angle unequal to the other cameras, it has been left fully zoomed out.
Below is the composite snapshot (download full video clip [link no longer available]):
In practical terms no material differences in performance are evident. Both the CH140 and CH120 produce images revealing little to no details of the human subject. While, the CH120 does appear to be 'brighter' than the CH140, it also exhibits greater noise. Unlike the CH140 and CH120, the CH110 features a digital slow shutter by default. As a result, it operated at a reduced frame-rate (7.5fps) under these low-light conditions
Below is the bandwidth utilization graph:
Due to CH110's default DSS, its frame-rate operated at a reduced 7.5fps (down from 30fps max). The other two cameras (CH140 and CH120) both maintained their full 30fps. All cameras essentially operated at their maximum CBR bit-rate of 2Mbps. It is interesting to note the slight 'break-out' to 2.19Mbps experienced by the CH110.
Indoor Low-Light (Normalized)
In this scenario, cameras are digitally zoomed into a human subject ~33ft out, corresponding to a HFoV of ~31ft. The scene is 'dark' with essentially no lighting (~0.1 lux). Since the CH140 and CH120 have a fixed 1/30s shutter by default, no changes were made to them in this scenario. The CH110, however, has a digital slow shutter by default, and has been set to 1/30s fixed for this scenario. Note that the CH110 is color only. Also note, because the CH110 has a fixed lens angle unequal to the other cameras, it has been left fully zoomed out.
Below is the composite snapshot (download full video clip [link no longer available]):
The only change in this scenario is that the CH110 has been configured for a 1/30s fixed shutter. This normalizes it to the CH140 and CH120 fixed shutters of 1/30s. It produces a nearly all 'blacked-out' image as a result. The CH110 also features a color-only sensor, owing to the 'poorer' low-light performance in comparsion to the CH140/CH120 D/Ns, when shutter speeds are normalized.
Below is the bandwidth utilization graph:
In the normalized low-light scenario, all cameras operated at their full 30fps maximums. All cameras operated at their CBR maximums of 2Mbps, as well.
Outdoor Midday Even Lighting (Far)
In this scenario cameras are digitally zoomed into a human subject ~75ft out, corresponding to a HFoV of ~70ft. The scene is illuminated via natural sun light and is evenly distributed (~10,000 to 20,000 lux). Note, because the CH110 has a fixed lens angle unequal to the other cameras, it has been left fully zoomed out.
Below is the composite snapshot (download full video clip [link no longer available]):
At this wider FoV, significant differences develop between the V series CH140 and E series CH120. The CH140 produces a fair degree of details on the human subject, where some chance of identification is likely. In contrast, the CH120 appears rather blurry in comparison. Note the differences in overall sharpness in the background vehicle and license plate, as well.
Outdoor Midday Even Lighting (Near)
In this scenario cameras are digitally zoomed into a human subject ~33ft out, corresponding to a HFoV of ~31ft. The scene is illuminated via natural sun light and is evenly distributed (~10,000 to 20,000 lux). Note, because the CH110 has a fixed lens angle unequal to the other cameras, it has been left fully zoomed out.
Below is the composite snapshot (download full video clip [link no longer available]):
At this narrower FoV, the differences are found to diminish, whereby the CH140 and CH120 images are essentially equal in terms of utility. Subtle differences in quality do persist, however, with the CH140 appearing sharper, and appropriate exposure. In contrast, the CH120 seems a bit blurry and slightly washed-out. All in all, in both images the certainty level of the human subject identification is quite high.
Below is the bandwidth utilization graph:
In this rather ideal lighting situation all cameras maintained their maximum 30fps. Furthermore, all operated at around the CBR bit-rate maximum of 2Mbps.
Outdoor Nighttime (Far)
In this scenario cameras are digitally zoomed into a human subject ~75ft out, corresponding to a HFoV of ~70ft. The scene is illuminated via artificial lights (~0.5 to 10 lux). Note, because the CH110 has a fixed lens angle unequal to the other cameras, it has been left fully zoomed out.
Below is the composite snapshot (download full video clip [link no longer available]):
No practical differences between the CH140 and CH120 emerged at this wider FoV under low-light conditions. Both images provide essentially no facial details on the human subject. However, enough contrast exists to elucidate gross contours and thus indications of body type/build, and perhaps gender.
Outdoor Nighttime (Near)
In this scenario cameras are digitally zoomed into a human subject ~33ft out, corresponding to a HFoV of ~31ft. The scene is illuminated via artificial lights (~0.5 to 10 lux). Note, because the CH110 has a fixed lens angle unequal to the other cameras, it has been left fully zoomed out.
Below is the composite snapshot (download full video clip [link no longer available]):
Whereas the wider FoV previously, produced little practical differentiation, this narrower FoV indicates clear separation in favor of the V series CH140. Indeed, the CH140 produces greater contrast, sharpness and overall details on the subject. Identification is highly likely and license plate details are clear enough to make a reasonably strong guess as to its characters. In the CH120 we observe an overall lesser level of clarity and details, owing somewhat to the additional presence of noise in the image. Furthermore, the CH120 offers less contrast than the CH140 in this scenario. Note that in the CH120 the license plate is virtually unreadable, as well.
Below is the bandwidth utilization graph:
Both the CH140 and CH120 maintained their full 30fps operation and essentially stayed within or around the configured CBR bit-rate of 2Mbps. The CH110 adapted to low-light conditions by increasing its exposure length as suggested by the drop in frame-rate to 7.5fps.
WDR Performance (Bright)
In this scenario cameras are digitally zoomed into a human subject ~40ft out, corresponding to a HFoV of ~37ft. The scene's WDR character is produced by a ~2000 lux bright area and a ~200 lux interior 'dark' area. The subject is positioned in the bright area. Note, because the CH110 has a fixed lens angle unequal to the other cameras, it has been left fully zoomed out.
Below is the composite snapshot (download full video clip [link no longer available]):
In this scenario, a clear and dramatic separation emerges between the CH140 and CH120. Despite the extreme backlighting, the CH140 produces an appropriately exposed subject and facial details are fairly robust, allowing a fair likelihood of identification. In contrast, the subject's face is severly washed-out in the CH120. Note also that the CH140 actually reveals details in the extremely bright background, whereas the same details are completely absent in the CH120.
WDR Performance (Dark)
In this scenario cameras are digitally zoomed into a human subject ~40ft out, corresponding to a HFoV of ~37ft. The scene's WDR character is produced by a ~2000 lux bright area and a ~200 lux interior 'dark' area. The subject is positioned in the 'dark' area. Note, because the CH110 has a fixed lens angle unequal to the other cameras, it has been left fully zoomed out.
Below is the composite snapshot (download full video clip [link no longer available]):
In this 'dark' region of the WDR scene, the benefits of the CH140 are not as starkly perceived as they were in the 'bright' region. However, indications of over-exposure can still be observed in the CH120, whereas the CH140 is nearly perfectly exposed. As a result, a degree more clarity, contrast and details is observed in the subject produced by the CH140 versus the CH120.
Below is the bandwidth utilization graph:
In this WDR scenario all cameras maintained their maximum 30fps. Furthermore, all operated at around the CBR bit-rate maximum of 2Mbps.
Motion Analysis
Based on our understanding of the relationship between shutter speeds, fps, and motion blur, we expected smooth motion even under low-light conditions for the CH140 and CH120 at default settings. This is because of a default fixed shutter of 1/30s and 30fps.
The video clip below confirms the lack of motion blur even under low-light conditions (~10 lux). As a trade-off of fixing the shutter at a relatively fast 1/30s, a fair degree of noise or gain is introduced. This is seen as a grainier looking picture with a loss of overall clarity. The video sequence starts with the CH140 on the left, and then the CH120 on the right:
Bandwidth Analysis
Because Sony cameras employ CBR, we expect little variation between cameras and between scenes. The default CBR bit-rate is 2Mbps across all cameras used in this study. As the following graphs indicate, there are no material differences in bandwidth utilization either between camera models or between scenes of various types (from the particular set of scenarios used in this study).
Methodology
The following are the three (3) cameras used in the 'Sony Camera Shootout':
- Sony CH140 (online $800) - 720p D/N; 1/3" CMOS; F1.2 Fujinon; 0.1 Lux (BW)
- Sony CH120 (online $460) - 720p D/N; 1/3" CMOS; ; 0.3 Lux (BW)
- Sony CH110 (online $300) - 720p Color;1/3" CMOS; F2.8 integrated; 2.0 Lux (Color)
Each of the three (3) cameras were simultaneously recorded to an Exacqvision VMS during five (5) key scenarios. All cameras were set to their defaults during testing, except during the low-light tests. During low-light tests D/N cameras were 'forced' into B/W to ensure consistency of comparison. Furthermore, cameras were normalized to 1/30s shutter speed for one of the tests. All camera lenses were adjusted to a uniform lens angle (~50 degrees) and recorded/analyzed at their - VMS supported - maximum resolutions.
Here are the key default settings and firmware versions for each camera:
- Sony CH140 - 1/30s shutter; View-DR/VE On; AGC 'Middle'; fw 1.26.01
- Sony CH120 - 1/30s shutter; AGC '42dB'; fw1.12.03
- Sony CH110 - 1/8s auto slow-shutter; AGC '30dB' ; fw 1.27.00
Here are the five (5) key scenarios:
- Indoor Even Artificial Lighting - ~200 lux
- Indoor Low-Light - ~0.1 lux
- Outdoor Even Midday Lighting - ~ 10,000 to 20,000 lux
- Outdoor Nighttime - ~0.5 to ~10 lux
- WDR Scene - ~2000 lux (bright area) and ~200 lux (dark area)