VMD Camera Shootout

Video motion detection's inaccuracy can have disastrous effects on your VMS system, with recordings due to positives easily consuming far more storage than estimated. On the other hand, optimizing VMD to reject these alarms can decrease detection rates, missing real events.

In order to test whose on-camera VMD performed best, we used cameras from Avigilon, Axis, Bosch, and Sony in multiple scenes, day and night, starting with a simple indoor conference room:

Moving on to a more complex open office area:

And finally, an outdoor area with moving vegetation and headlights on a nearby road:

Our tests looked to answer the following key questions:

• What issues most commonly cause false positives?
• Whose VMD was best at rejecting these events?
• How did detection range vary from manufacturer to manufacturer?
• Can settings be optimized for better detection without increasing false alerts and vice versa?

Key Findings

False positives were hands down the largest issue in our tests, with the following specific issues discovered:

• No camera could truly be named "best" at rejecting false positives, since all suffered numerous activations from different conditions, day and night, at about the same rates. The Avigilon H3 1MP rejected alarms due to changes in light better than others. The Sony VB600 was activated by reflected headlights more than other cameras. Other than these two points, performance was about equal.
• All cameras suffered numerous false positives due to the movement of vegetation in the scene, day or night, causing near continuous recording. In our scene, this area could be masked out without affecting performance, as no human movement was possible in this area.
• However, shadows caused by moving branches caused many false positives which could not be excluded via motion masks, since they fell throughout the motion detection zone.
• At night, all cameras were sensitive to car headlights in our scene. The Sony VB600 was sensitive to this even when the road was excluded from the motion zone.
• Indoors, light changes were the largest cause of false alarms. These were due to doors opening/closing from a lit hallway, subjects moving past the door, or flashlights or headlights reflecting. The Sony VB600 as most prone to false alerts in this scene, followed by Bosch NBN-733V, Axis P3354, and Avigilon 1.0-H3-B2. 

The following key findings reflect performance in various test scenes using a human subject:

• The Avigilon 1.0-H3-B2 performed well in indoor and outdoor scenes at all light levels, ranging from full light to dark (<1lux). The camera had the longest detection range among tested cameras and it rarely, if ever, missed detection of humans and vehicles in the test conditions.  
• In outdoor daytime and full light indoor tests, Axis P3354, Bosch NBN-733V and Sony SNC-VB600 yielded similar motion detection performance in terms of detection ranges and false alert rates. 
• In low light, VMD performance of Bosch NBN-733V and Sony SNC-VB600 were also similar, with Bosch yielding only slightly longer detection ranges  and slightly lower false alert rates.
• Axis P3354, on the other hand, lagged significantly behind other cameras in low light, with much lower detection ranges.

Optimization Recommendations

Based on our tests, as well as discussions of best practices with each manufacturer, here are our recommendations on how each camera may be optimized for best performance:

• For the Avigilon 1.0-H3-B2, default settings resulted in best performance in all scenes, with no recommended changes.
• With Axis' P3354 Standard VMD, the default object size setting is too large for most scenes.  In order to get useful detection ranges, the object size setting needed to be reduced carefully to a level specific to the scene, generally 5 or below.
• For the Bosch NBN-733V, best VMD performance was also obtained by using the default settings in the scenes we tested. Lowering the object size resulted in significantly higher false alerts due to shadows and wind. 
• For the Sony SNC-VB600, the object size should be reduced to its minimum level for best performance. In the daytime, sensitivity can also be slightly increased from its default setting to get a longer detection range. Due to higher false alert rates in low light, default sensitivity is found to be more appropriate in these conditions.

Recommendations

Due to the prevalance of false alerts from moving leaves, branches, shadows, and light variations, users should carefully consider camera placement and exclude windows when planning to use video motion detection. Aiming cameras away from vegetation and active roadways when possible will greatly improve performance, even beyond using exclude windows on these areas.

Default VMD settings should not be assumed optimum for all cameras in all scenes. In some cases these settings will result in numerous false positives or missed detections. Users should plan to adjust these settings once cameras are in place, and settings should be revisited periodically to ensure performance is still satifactory.

For users requiring reliable motion detection at longer ranges (>100' during day time and >50' night-time), the Avigilon 1.0-H3-B2 stands out as the best option among the tested cameras. However, since it integrates with third parties via ONVIF, most VMSs will not receive motion events from the camera, limiting its use to end-to-end Avigilon systems.

Comparison to Built-In Analytics

Compared to the standard camera motion detection tested here, Axis Video Motion Detection 2.1 performed better in most scenes. Falses positives, rejection of shadows, and range were all improved, but low light performance was reduced somewhat. We recommend it be used with Axis cameras whenever possible, but its application is limited due to integration with very few VMSs (Axis and Genetec only).

Compared to their built-in VMD, Axis, Bosch, and Sony camera analytics such as cross line detection suffered from much lower false positive rates. This is due to the much narrow detection areas and specific rules required for activation. Users may want to consider these analytics instead of standard VMD where possible.

Configuration

The following videos review configuration of each manufacturer's VMD.

Avigilon H3

Axis P3354

This screencast reviews configuration of Axis' VMD. Note that this video also includes configuration of Axis Video Motion Detection 2.1, which we recommend over standard VMD whenever possible. However, this report focuses only on cameras' standard included VMD. 

Bosch NBN-733V

Sony 6th Gen

Outdoor Daytime 

Outdoors, in daylight, moving tree branches and leaves triggered continuous nuisance alarms on all cameras. We removed these areas using exclude windows, as testing of motion detection was not possible otherwise.

Even though we excluded the vegetation in the scene, all cameras gave false positives due to the movement of shadows in the scene. These areas could not be excluded as they were in the center of our motion zone.

The image below shows a section of a spider web in front of one camera. The movement of this web with the sun glaring off of it caused near continuous recording for two hours.

Detection Range

For our detection range test, we used a test scene about 25' wide from building to fence, and about 280' the nearby road. All cameras activated due to an object walking towards the cameras. However, Avigilon 1.0-H3-B2 picked up our subject at  ~ 100-120' away from the camera, while Axis P3354, Bosch NBN-733V and Sony SNC-VB600 triggered at the 45-50' range. Note that, Axis P3354 Standard VMD's object size needed to reduce from 15 (default) to 1 in order to increase detection range from 10' to 45'. Also, the Sony SNC-VB600 VMD's sensitivity setting was increased to 225 from the default 180, which enhanced the detection range by around 15'. Further increase in Sony's sensitivity resulted in continuous motion detection due to false positives.

Avigilon 1.0-H3-B2 also captured vehicles accurately on the nearby road (~280') during the day. Sony SNC-VB600, Bosch NBN-733V and Axis P3354 detected only about 40%, 15%, and 5% of vehicles, respectively.

Avigilon 1.0-H3-B2 was also able to detect trucks on the distant highway, over 600' from the camera.

 

Outdoor Nighttime

In our outdoor nighttime scene, there were two key triggers of false positives. As in the daytime, shadows from people and vegetation triggered motion:

Also, although we masked out the road to measure other factors in the test separately, Sony SNC-VB600 were still very sensitive to the headlights and it kept triggering continuously from the headlights.

Detection Range

In low light, Avigilon 1.0-H3-B2's human detection range decreased to about 50' but still yielded the longest range (Bosch at ~40', Sony at ~30', and Axis at ~20'). Note that we did not increase the sensitivity setting of Sony SNC-VB600 this time, since it had more tendency to generate false positives due to shadows in low light.

With our subject moving across the FOV at ~50', Avigilon 1.0-H3-B2 and Bosch NBN-733V detected him accurately. Sony SNC-VB600 caught him about half of the time, while Axis P3354 did not trigger at all.

At 70', none of the cameras triggered, with the subject simply too small to activate.

Indoor Conference Room

The Sony VB600 VMD had a false alarm rate of 10%-15% from the shadow on the background, while Bosch triggered about 5% of the time.

Opening the door changed the light in the scene and caused the following false alert rates: Sony 40%, Axis 20%, Bosch 20%. 

When waving a flashlight throughout the room, 

In our conference room in full light, all camera were able to reliably detect a human subject at  all ranges up to 20'. Note that the Axis P3354 could not detect our subject at maximum distance at default object size (15), and had to be reduced to 5 for accurate detection.

In low light (~1lx), all VMDs detected our object in 100% accuracy up to 20' in a similar manner to the full light test.

 

Open Office

Moving the cameras to an open office scene, about 40' long, full light performance of all cameras were reliable. Once again, the Axis P3354's must be reduced to 1 to improve detection range

In low light, the back of the office (~40') was lightened by a small background security light. Due to this, all cameras except Axis P3354 captured the object as soon as he entered the room 40' away.

However, as the object moved down the aisle to the darker side, Sony SNC-VB600 and Bosch NBN-733V lost detection at around 25' and re-captured the motion at about 15'. 

The Axis P3354, on the other hand, could only detect the object at 7' away from the camera. 

Cameras/Firmware Used

The following cameras and firmware versions were used in the test:

• Avigilon 1.0-H3-B2: 2.4.10.4
• Axis P3354: 5.40.17
• Bosch NBN-733V: 5.85
• Sony SNC-VB600: 1.5.1