Panoramic Camera Shootout 2013

Panoramic cameras are one of the hottest growing markets within surveillance. The combination of multi-megapixel resolution and super wide FoVs claims to deliver greater coverage than conventional cameras. But how well do they work and how do the options compare?

This report shares the result of our multi-month test of 8 different Panoramic cameras from 6 different providers including Axis (M3007), Immervision, Mobotix (Color and B/W Q24 models), Panasonic, Sentry360 (3MP, 5MP and 10MP versions) and Vivotek.

We did tests in even, well lit scenes, in multiple levels of darkness, and in WDR conditions. 

Moreover, we examined key issues including distortion, camera skew, dead pixel count, panoramic controls and VMS integration.

Here are two unmarked excerpts showing the range of differences in performance:

Inside, we offer full test results.

Key Findings

In this shootout, more than many others, citing overall winners is difficult. Every camera we tested had positives and negatives in image quality, WDR and low light performance, VMS integration, and control. Because of this, we separated our findings into the following:

• Well-lit image quality: The Axis M3007-PV, Mobotix Q24, and Vivotek FE8172V stood out as providing the all around best image quality in evenly, well-lit scenes. This can be seen in comparisons below, as the subject's face and/or eye chart are more easily discernable. The 10 MP Sentry360 FS-IP10k provides more detail in some scenes, but its tendency to overexpose the image limits it in even bright indoor light.
• Very low light performance: In our truly dark scene (<1 lx), no camera performed well. The Axis Q1604 with ImmerVision lens stood out, providing the best detection in these conditions, but it provides no usable details, due to noise and lower resolution. This is a good example of ImmerVision's main advantage: the ability to use top performing third-party cameras with the lens.
• Low light performance: In the 3 lx scene, the M3007-PV, Mobotix monochrome Q24, and Sentry360's Pro series (IP5000 and IP10K) provided moderately better details than others, with the chart more legible. The Vivotek FE8172V produced one of the brightest images, but details were lost beyond 6'. Note that Sentry360's pro models and the FE8172V are true day/night cameras, a likely reason for this performance gain.
• Wide dynamic range: WDR performance was almost universally poor. Not surprisingly, the Q1604 with ImmerVision lens performed best in our WDR scene.  Panasonic and Vivotek both captured details of our subject and strongly lit background as well, to a lesser extent. 

We will release individual manufacturer reports in an upcoming series, with more in-depth testing of each of these.

In more general terms, this shootout also revealed a number of general issues and observations:

• Resolution drops quickly: At even 6' range, the shortest distance we tested, panoramic cameras deliver far fewer details than standard cameras, due to their extremely wide FOV. At 12' and beyond, most cameras will provide only detection and situational awareness. For example, our previous tests of megapixel cameras in this room have shown facial identification is easily possible across the front wall (~22 feet) at nearly 30' range.
• Megapixels matter less: Increases in pixel count in 5 and 10 megapixel models had modest effect on performance. In well lit scenes, the increase helped with facial and chart details. However, as light decreased, these models were among the fastest to increase in noise and lose detail.
• Dewarping varies: All manufacturers showed our subject's height-to-width ratio differently, some more accurately than others, since each uses a different dewarping SDK.
• Panorama vs. dewarped views: All cameras distorted the image when viewing 360º or 180º panoramas, with horizontal lines becoming curved when the image is flattened. Generally speaking, we found fully dewarped vPTZ images or quad views to be easier to use because of this.
• Client side dewarping has definite advantages over camera side, in speed and smoothness of PTZ control, usability, and the ability to dewarp archived video. 

The Test

We tested nine cameras in our shootout, from six manufacturers, with varying resolutions and featuresets. Following is an overview of each model, outlining key differences:

Axis M3007

• Resolution: 5MP sensor, cropped to 2MP panoramic or quad views, or four 800x600 positionable view areas.
• Framerate: 12 FPS
• Dewarping: Camera side only, sending eight different views: Warped overview, panorama, dual panorama, quad view, and four configurable view areas.
• Notes: VMS must integrate integrate separate views and PTZ control of dewarping, or only the warped overview image is viewable.
• Price:~$560 USD standard, ~$650 vandal resistant

ImmerVision IMV1-1/3 + Axis Q1604

• Resolution: 1.3MP
• Framerate: 30 FPS
• Dewarping: Client side, with at least a dewarped view available. Some VMSs add panorama and quad view modes, as well.
• Notes: The IMV1-1/3 is rated to 1.3MP only. An upcoming 5MP version has been announced.
• Price: ~$525 USD online, plus cost of camera (~$900 online). This is one of the highest prices in our test, for the lowest-resolution camera.

Mobotix Q24

• Resolution: 3.1MP (color version) or 1.3MP (monochrome)
• Framerate: 20/30 (3.1MP/1.3MP) using MxPEG encoding. Significantly drops to 4/8 FPS when using MJPEG.
• Dewarping: Camera side via web interface and integrated VMSs. Client side via MxControlCenter.
• Notes: Only available in color or monochrome, not day/night. Very limited VMS support.
• Price: ~$950 online

Panasonic WV-SF438

• Resolution: 3.1MP
• Framerate: 30 FPS
• Dewarping: Camera side, controllable from VMS.
• Notes: Defaults to two streams, a dual panorama view, plus a controllable dewarped PTZ view of up to 1.3MP resolution.
• Price: ~$780 online. The outdoor-rated rated WV-SF458 is also available for ~$900.

Sentry360 FS-IP3000 Mini Series

• Resolution: 3MP
• Framerate: 15 FPS
• Dewarping: Client side, with configurable dewarping geometry, not found in other manufacturers.
• Notes: 
• Price: $980 MSRP

Sentry360 FS-IP5000/10K Pro Series

• Resolution: 5MP/10MP
• Framerate: 14/6 FPS
• Dewarping:Client side, with configurable dewarping geometry, not found in other manufacturers.
• Notes: Compact box camera, with optional dome. True day/night with IR cut filter.
• Price: $2,500/$3,050 MSRP

Vivotek FE8172

• Resolution: 5MP, cropped to 1920x1920 max.
• Framerate: 30 FPS
• Dewarping:Client side
• Notes: True day/night with IR cut filter.
• Price: ~$630 (non-vandal)/$800 (vandal-resistant) USD online.

We tested these cameras using Video Insight 5.5, since it is one of the only VMSs supporting all of them, in one way or another. Cameras were left defaulted, with only exposure settings standardized for low-light scenes. 

Common Imaging/Integration Differences

In the course of our tests, we found several notable difference in imaging and integration among the tested cameras. We've detailed these in a series of videos, below:

Geometry and Distortion

Since all dewarping, whether on camera or client side, is creating a rectangular image from a round fisheye view, various forms of distortion will occur. When using panoramic display modes, for example, horizontal lines will not be straight. On the other hand, when using fully dewarped views, vertical lines may not be vertical, and the image may be "keystoned." We examine these issues in this screencast:

Camera Skew

In addition to distortion from dewarping, cameras may be skewed by off-level mounting. In slight cases, it may not be an issue. In more severe cases, the camera's dewarping may be affected, or pre-defined views innefective. Some cameras allow adjustment for this, while others must simply be re-mounted level to the ground.

Dead Pixel Space

This screencast reviews an issue unique to fisheye cameras, created when the camera's lens extends past the imager, creating blank areas when the image is dewarped and flattened. Depending on how the camera is oriented, this blank "divot" may extend into the FOV, partially obscuring it. Of the models we tested, only Sentry360 allowed for adjustment to the dewarped area to compensate for this. This issue is best seen, as it's difficult to describe in words:

PTZ Control Camera vs. Client Side

In this screencast we show the differences in virtual PTZ control when using client and camera side dewarping. Client side dewarping resulted in much quicker pan/tilt control, since no commands needed to be sent to the camera, and dewarping was done on the faster processor of our client PC. Latency in camera side control varied, but was generally at least a second, at best.

VMS PTZ Integration

Some models which dewarped camera side could be controlled via VMS PTZ controls (Axis, Panasonic). Others, such as Mobotix, recorded the view which was set in the camera's web interface. This screencast illustrates the differences between these two.

Client vs. Camera Side Archived Video

This screencast illustrates one of the key advantages to client-side dewarping integration: virtual PTZ in archived video. While camera-side dewarping has an advantage in that it does not require custom SDK integration to dewarp video, the VMS only records what the camera was displaying at the time, whether it be a panoramic view, custom view area, etc.

A Note About Comparison Images

In our comparisons, every effort was made to set the camera's virtual "field of view" similarly. That is, the direction and digital zoom level of the dewarping, whether on camera or dewarped in the VMS client. However, since all cameras are dewarped slightly differently and no two lenses are identical, or identically mounted in every camera, differences do occur. This can be seen in our comparisons, with some looking more angled, not quite level, in different proportions, etc. To try and equalize comparison images as best as possible, we tried to set the scene using the size of the letter E on the test chart for reference, as this was the simplest objective size reference in our test shots.

Full Light Comparison

Our first, and easiest comparison, we tested all cameras at 6' range under full lights. Resolution was the most impactful factor here, with the 1.3MP Q1604 with ImmerVision lens immediately less clear than other cameras due to its lower pixel count. However, we found several cameras had issues compensating for the light in the room, with the chart becoming washed out in Mobotix's monochrome Q24 and Sentry360's cameras, reducing clarity. Most cameras were fairly accurate in their proportions of the subject. However, in this and other tests, Sentry360's 3MP FS-IP3000 displayed objects as a bit wider than they really are, and ImmerVision shows our subject as taller and thinner.

 At 12' range, we can see that the chart becomes much less readable on all cameras, and the subject's features become harder to distinguish.

 

At 18', details are futher reduced, but several cameras still display multiple rows of the chart at a legible level, including the Axis M3007, Mobotix Q24, Panasonic, Sentry360 10MP, and Vivotek FE8172V. Axis, Mobotix (color), and the FS-IP10k most accurately reveal details of the subject such as facial hair and glasses. At this range, however, resolution is likely much too low to recognize an unknown subject on any camera.

 

 30lx Comparison

We next lowered the lights in the room to approximately 30lx. Interestingly, using default settings, the 5MP Sentry360 FS-IP5000 switched to monochrome at this light level (it is true day/night), but no other cameras did. At 6', results are mostly consistent with results above. The background of the Sentry360 10MP camera is notably dimmer, but given its much higher resolution, this is to be expected.

 Results at 12' are consistent with full light images, as well:

 And at 18':

3lx Comparison

We dimmed the lights further to ~3lx. At this level, the Vivotek FE8172V and Sentry360 FS-IP10k switched to night mode, as well. The only other day/night cameras in this test are the Q1604 with ImmerVision lens, and the Panasonic SF438 (electronic day/night only, no mechanical cut filter). At this level, and 6' range, cameras still produced usable images, though Mobotix and Panasonic are notably darker, and the 10MP Sentry360 model has dimmed further, darkening the subject's face.

At 12' range, Panasonic and the FS-IP10k are dimmed so much the subject's facial details are obscured. The color Mobotix Q24 becomes much noisier at this light level, and the effects are pronounced at this distance to target.

At 18' range, the lower light levels impact all cameras, with the SF-438 barely showing the subject's face, and others very blurry. The monochrome Q24 is perhaps best at this level and range.

<1lx Comparison 

Finally, turning all the lights off, so illumination is at under 1 lx, only four cameras display the subject at all, as shown below. The Q1604, known for its low light performance, does best in this scene, with the subject and chart both visible, though without any detail. The subject also visible, but less so, in the 5MP Sentry360, along with the chart. The chart is also visible in the M3007 and FE8172V.

 At 12', only the Q1604 shows the subject at all.

And at 18', the subject and chart are barely visible even on the Q1604. 

WDR Scene

We set up the cameras in a garage with the overhead door rolled up to test WDR performance. Performance, on average, was poor against the strong backlighting. The M3007, Mobotix Q24s, and Sentry360 models had most trouble, with the background very washed out on all of these models, in most cases over exposing the subject's face, as well. The Q1604 with ImmerVision lens, not surprisingly, performed best in these scenes. The Panasonic WV-SF438, while washing out the background, still reveals details, such as the "LOADING ZONE" lettering, vans, and grass in the background. The Vivotek FE8172V shows distant object, but washes out the near background.

Moving the subject to the dark area next to the door (bottom row of images), cameras handled the lighting difference much better than backlight. Coincidentally, Panasonic and the Sentry360 10MP, which showed decreasing performance in low light the quickest, handle the dim area the worst.