Axis vs. Sony HD Shootout
When deciding on camera selection, one of the toughest decisions is determining which camera has better image quality. You can try to guess based on certain image features but it is an inexact science as 'good' marketing can cause confusion. You might choose on the strength of the brand name but that is becoming more difficult as major surveillance brands are offering multiple tiers or quality levels of products (e.g., how good is the big brand's entry level camera?)
In this report, we share our findings from a shootout of two of the biggest brands in the surveillance camera market: Axis and Sony. The focus was to see what image quality differences truly existed across a variety of real world scenes.
We used the following five cameras (listed in order from lowest to highest price):
- Sony CH120 (online ~$475) - 720p D/N
- Axis M1114 (online ~$530) - 720p Color
- Axis P1344 (online ~$800) - 720p D/N
- Sony CH140 (online ~$850) - 720p D/N
- Axis P1347 (online ~$1350) - 5MP D/N
Note: all cameras ship with varifocal lens and were tested with the lens included as part of the camera package. For both manufacturers, we included both their 'lower' and 'higher' end series as well as Axis's 5MP - the highest resolution offering of either brand.
We tested all of these cameras simultaneously in several scenarios, encompassing a wide variety of real-world surveillance environments (e.g. indoors, outdoors, evenly-lit, low-light, and WDR scenes).
Analyzing Video Quality
We analyzed a series of scenes to see how performance varied across the cameras. We exported the simultaneous recordings and then digitally zoomed each camera to the same level to show differences. The sample comparison below from this test shows an example of the analysis (camera names/models revealed inside the Pro section):
We conducted our analysis across 10 scenes. They are:
- Indoor Even Artificial Lighting (Moderately Wide FoV)
- Indoor Even Artificial Lighting (Narrow FoV)
- Indoor Low-Light (Default)
- Indoor Low-Light (Normalized)
- Outdoor Daytime Even Lighting (Wide FoV)
- Outdoor Daytime Even Lighting (Moderately Wide FoV)
- Outdoor Nighttime (Wide FoV)
- Outdoor Nighttime (Moderately Wide FoV)
- WDR Scene (Bright)
- WDR Scene (Dark)
Additionally, we examined bandwidth differences across the cameras and scenes to see any tradeoffs in bandwidth vs quality.
This report focuses on image quality tradeoffs. There are clearly other important factors - compatibility, feature sets, channel structure, etc - in choosing an IP camera. Those are easier to discern without testing and the reason why we have focused the report on image quality.
A few structural differences to keep in mind:
- VMS Compatibility: Axis hast the industry's broadest support while Sony is good but not as universal
- Software Feature Sets: Axis has a greater number of advanced software feature sets than Sony
- Hardware Feature Sets: Roughly similar across the series
- Channel: Sony uses a traditional security channel while Axis uses a more IT channel
With that noted, let us focus on the image quality tradeoffs.
Key Findings and Recommendations
Here are our key image quality and bandwidth findings:
- Best Overall: The 5MP Axis P1347 and the 720p Sony CH140 provided the best overall image quality across the scenes, roughly similar to one another and superior to the other 3 cameras tested. Given the significantly lower price of the Sony CH140, its price performance is attractive.
- Low Price Best: The Sony CH120's image quality provided moderately more details than the Axis M1114 across the scenes tested. Given the similar price, the Sony CH120's image quality provides a moderate advantage.
- Indoor Best: In indoor scenes, the difference in image quality between the lower and higher end cameras was modest. Of those, the Sony CH120 performed moderately better than the Axis M1114.
- Outdoor Best: The Axis P1347 provided modest increases in image quality at wider FoVs outdoors. The Sony CH140 was a close second. Given the significant price disparity, decisions should be based on the value of modest increases in coverage.
- Dark Best: Using the defaults, Axis P series cameras outperformed both Sony's. However, Axis's default max shutter speed is 500% longer than Sony's. When normalized (to 1/30s), Sony image quality was moderately better.
- WDR Scenes: The Sony CH140 provided the most significant details across strong sunlight scenes compared to all cameras (with the Axis P1347 delivering the next closest amount).
- Bandwidth Best: Axis bandwidth consumption was moderately lower than Sony primarily due to Axis's support of VBR versus Sony's CBR streaming.
Overall, Sony camera image quality was better than Axis quality, even factoring in variances in pricing. On the other hand, in most scenes, we would categorize the differences as modest.
Ultimately, decision makers should factor in both image quality variance and structural variances (compatibility, feature sets, channel structure, etc.)
Indoor Even Artificial Lighting (Moderately Wide FoV)
In this scenario, cameras are digitally zoomed into a human subject at a HFoV of ~31ft. The scene is illuminated via artificial lighting and is evenly distributed (~200 lux).
Below is the composite snapshot (download full video clip [link no longer available]):
All cameras deliver fairly strong levels of facial detail and a subsequent high likelihood of identification. The M1114 is the weakest performer of the lot, with overall less clarity. In general the Sony cameras appear to provide additional sharpness and contrast versus the Axis cameras. The 5MP Axis P1347 does not seem to provide any real advantages over the stronger performing 720p cameras (CH120 and CH140) under these conditions.
Indoor Even Artificial Lighting (Narrow)
In this scenario, cameras are digitally zoomed into a human subject at a HFoV of ~16ft. The scene is illuminated via artificial lighting and is evenly distributed (~200 lux).
Below is the composite snapshot (download full video clip [link no longer available]):
Differences in details captured are minimal, especially given the relative narrow FoV. Both the CH120 and P1344 provide slightly crisper images versus their higher tier counterparts, the CH140 and P1347, respectively. The Axis M1114, continues to lag the rest of the field. Despite subtle picture quality differences between Axis and Sony, utility remains essentially equal with high likelihoods of identification being delivered across the board.
Below is the bandwidth utilization graph:
Under near ideal lighting conditions, Axis's VBR streaming consumes less bandwidth than Sony's CBR.
Indoor Low-Light (Default)
In this scenario, cameras are digitally zoomed on a human subject at a HFoV of ~31ft. The scene is 'dark' with essentially no lighting (~0.1 lux). All cameras are set to default settings (Axis max 1/6s exposure, Sony max 1/30s exposure). Note that the M1114 is color only.
Below is the composite snapshot (download full video clip [link no longer available]):
Using a 1/6s max exposure default, the Axis cameras provide moderately better greater image details. The Sony cameras default to 1/30s max shutters times and result in relatively darker images. Indeed, both Sony cameras provide little detail outside of a silhouette. In contrast, Axis provides a modicum of high level details that may indicate gender, build, type of dress etc.
Below is the bandwidth utilization graph:
The relative low-light 'intolerance' of the P1347 results in a large spike in bandwidth (~10mbps), considerably higher than any of the other cameras (avg. ~2.5mbps). Also worthy of note is the triggering of digital slow-shutters and a subsequent fps decrease across the Axis cameras. In contrast, the Sony cameras with max shutter lengths of 1/30s retain 30fps and as such do not present the same risk for motion blur as do the Axis cameras.
Indoor Low-Light (Normalized)
In this scenario, cameras are digitally zoomed into a human subject at a HFoV of ~31ft. The scene is 'dark' with essentially no lighting (~0.1 lux). Cameras have been normalized to a 1/30s shutter speed. Note that the M1114 is color only.
Below is the composite snapshot (download full video clip [link no longer available]):
Now that normalization to a 1/30s maximum exposure length is in effect, Sony provides images with more details than Axis in this environment. The entire array of Axis images have become considerably dark, and offer little utility at this point. It can be fairly reliably inferred from these results that the Sony cameras provide a degree more low-light sensitivity.
Below is the bandwidth utilization graph:
After normalizing to 1/30s max shutter times, Axis' image quality 'collapsed' to effectively 'blackened' images. The upside is that bandwidth does not spike as is sometimes the case with other cameras (e.g., Arecont) in similar conditions. Note that the Sony shutter times were the same as in the 'default' case, so no changes were observed in either bandwidth or picture quality post-normalizations.
Outdoor Daytime (Wide FoV)
In this scenario cameras are digitally zoomed into a human subject ~75ft out, corresponding to a HFoV of ~70ft. The scene is illuminated via natural sun light and is evenly distributed (~10,000 to 20,000 lux).
Below is the composite snapshot (download full video clip [link no longer available]):
The 5MP Axis P1347 provided more detail than any of the other cameras. In fact, the key differentiating basis appears to be the higher resolution as the rest of the field are all 720p cameras. Moreover, aside from subtle differences the 720p cameras all provide an effectively equivalent level of utility.
Outdoor Daytime (Moderately Wide FoV)
In this scenario cameras are digitally zoomed into a human subject at a HFoV of ~31ft. The scene is illuminated via natural sun light and is evenly distributed (~10,000 to 20,000 lux).
Below is the composite snapshot (download full video clip [link no longer available]):
Across the board, high likelihoods of identification of both human and license plate characters are evident. While utility levels are essentially equal, subtle differences in picture quality favor the 5MP P1347 and 720p CH140, as they tend provide better overall clarity and sharpness.
Below is the bandwidth utilization graph:
Relatively favorable lighting conditions result in fairly economical bandwidth consumptions across the Axis VBR streams, averaging slightly less than the 2Mbps observed with Sony's CBR. The Axis bandwidths are slightly higher than seen in the earlier 'Indoor Even-Lit' scenario, suggesting slightly more complexity (e.g. motion, vegetation, and lighting effects) present in the outdoor environment.
Outdoor Nighttime (Far)
In this scenario cameras are digitally zoomed into a human subject ~75ft out, corresponding to a HFoV of ~70ft. The scene is illuminated via artificial lights (~0.5 to 10 lux).
Below is the composite snapshot (download full video clip [link no longer available]):
At this wider FoV, under low-light conditions, high levels of detail are not expected even for the multi-megapixel 5MP P1347. Indeed, effectively the images are of equivalent utility (e.g. indications of build, gender and perhaps dress type). The key difference to note perhaps is the higher noise level seen in the Sony cameras and their apparent higher brightness level.
Outdoor Nighttime (Near)
In this scenario cameras are digitally zoomed into a human subject ~33ft out, corresponding to a HFoV of ~31ft. The scene is illuminated via artificial lights (~0.5 to 10 lux).
Below is the composite snapshot (download full video clip [link no longer available]):
Although subtle, the 5MP Axis P1347 edges out the field, delivering slightly more facial details and clarity on the license plate. The license plate provides the most quantitative evidence of the P1347s added benefit, as it offers the only image in the composite where capture is near absolute. Notably, the entry level M1114 (color-only) performs quite well and offers a richness (e.g. complexion and clothing color) unavailable from the D/Ns (in b/w mode). The Sony CH120 is the laggard, as contrast is low and slight over-exposure is evident.
Below is the bandwidth utilization graph:
It may appear paradoxical that the outdoor nighttime bandwidths for the Axis VBR cameras (excluding the color only M1114) are actually lower than their outdoor daytime bandwidths. However, lighting conditions are not too severe given that artificial lighting provides roughly 5 to 10 lux. Furthermore the P1347 has adapted via its digital slow-shutter, and as a result overall gain/noise is relatively low. Indeed in the low-light (default) scenario (~0.1 lux), Axis bandwidths were higher than Sony across the board and even a heavy spike to ~10mbps was observed for the P1347.
WDR Scene (Bright)
In this scenario cameras are digitally zoomed into a human subject ~40ft out, corresponding to a HFoV of ~37ft. The scene's WDR character is produced by a ~2000 lux bright area and a ~200 lux interior 'dark' area. The subject is positioned in the bright area.
Below is the composite snapshot (download full video clip [link no longer available]):
The CH140 distinguishes itself solidly from the field in this WDR scenario. The CH140 delivers fairly strong facial and body details/contours despite the extreme back lighting, while also continuing to provide semblance of a background scene. In contrast, all others produce a washed out background that carries over onto the subject's face/body. The 5MP P1347 is perhaps the strongest of the 'weakest' yielding slightly more facial details.
WDR Scene (Dark)
In this scenario cameras are digitally zoomed into a human subject ~40ft out, corresponding to a HFoV of ~37ft. The scene's WDR character is produced by a ~2000 lux bright area and a ~200 lux interior 'dark' area. The subject is positioned in the 'dark' area.
Below is the composite snapshot (download full video clip [link no longer available]):
In terms of facial details/clarity the CH140 is slightly outdone by the P1347 as interest shifts to the darker region of the WDR scene. However, the CH140 continues to provide a key advantage by revealing details in the back-lit region of the scene. All others continue to lack in this regard.
Below is the bandwidth utilization graph:
Though a typically problematic scenario for image production, in terms of bandwidth the WDR scene is somewhat analogous to the 'Even-Lit' scene. Indeed, the Axis VBR cameras perform relatively economically at roughly ~1mbps versus the Sony CBR cameras at ~2mbps.
Motion Analysis
Due to Axis' exposure length defaults of 1/6s maximum, only Axis cameras produced video containing motion blur in certain low-light/nighttime scenarios. Of the Axis cameras the P1347 due to its lower light sensitivity and lower top frame-rate of 12fps, tended to be the most susceptible to motion blur.
In contrast, Sony's default 1/30s max exposure time resulted in a full 30fps throughout all scenarios. Of course its 'apparent' light sensitivity was lower than Axis in the 0.1 lux environment, but was shown to actually be inherently more sensitive to light once the Axis cameras were normalized to 1/30s.
Bandwidth Analysis
Average Bandwidth by Scene (Axis vs. Sony)
Overall, Axis's VBR streaming consumed less bandwidth across the majority of scenes compared to Sony's CBR streaming. As indicated by the low-light (default) scene where P1347 bandwidth spiked to ~10mbps, increasing levels of complexity will tend to drive VBR bit-rates upwards to possibly problematic levels.
Overall Average by Camera
Not surprisingly the Axis P1347 exhibited the highest overall average bit-rate of ~2.6mbps. However, it is not a particularly concerning number and is a reasonable expected bit-rate given the performance/resolution level. More notable, perhaps is the remarkable bandwidth efficiency exhibited by the Axis P1344, which averaged roughly ~1.5mbps throughout all low-light/nighttime tests and never broke beyond a mark of 3Mbps.
Methodology
The following are the five (5) cameras used in the 'Axis vs. Sony Shootout':
- Axis P1347 (online $1350) - 5MP D/N; 1/2.5" CMOS; F1.6 Kowa;0.08 Lux (BW)
- Axis P1344 (online $759) - 720p D/N; 1/4" CMOS; F1.2 Computar ; 0.05 Lux (BW)
- Axis M1114 (online $475) - 720p Color; 1/4" CMOS; F1.2 Fujinon; 0.6 Lux (Color)
- Sony CH140 (online $800) - 720p D/N; 1/3" CMOS; F1.2 Fujinon; 0.1 Lux (BW)
- Sony CH120 (online $460) - 720p D/N; 1/3" CMOS; ; 0.3 Lux (BW)
Each of the five (5) cameras were simultaneously recorded to an Exacqvision VMS during five (5) key scenarios. All cameras were set to their defaults during testing, except during the low-light tests. During low-light tests D/N cameras were 'forced' into B/W to ensure consistency of comparison. Furthermore, cameras were normalized to 1/30s shutter speed for one of the tests. All camera lenses were adjusted to a uniform lens angle (~50 degrees) and recorded/analyzed at their - VMS supported - maximum resolutions.
Here are the key default settings and firmware versions for each camera:
- Axis M1114 - 1/6s max exposure w/ AGC; fw 5.09
- Axis P1344 - 1/6s max exposure w/ AGW; fw 5.22
- Axis P1347 - 1/6s max exposure w/ AGC; fw 5.20
- Sony CH140 - 1/30s shutter; View-DR/VE On; AGC 'Middle'; fw 1.26.01
- Sony CH120 - 1/30s shutter; AGC '42dB'; fw1.12.03
Here are the five (5) key scenarios:
- Indoor Even Artificial Lighting - ~200 lux
- Indoor Low-Light - ~0.1 lux
- Outdoor Even Midday Lighting - ~ 10,000 to 20,000 lux
- Outdoor Nighttime - ~0.5 to ~10 lux
- WDR Scene - ~2000 lux (bright area) and ~200 lux (dark area)