VMD Camera Shootout

By: Ethan Ace, Published on Oct 28, 2013

Video motion detection's inaccuracy can have disastrous effects on your VMS system, with recordings due to positives easily consuming far more storage than estimated. On the other hand, optimizing VMD to reject these alarms can decrease detection rates, missing real events.

In order to test whose on-camera VMD performed best, we used cameras from Avigilon, Axis, Bosch, and Sony in multiple scenes, day and night, starting with a simple indoor conference room:

Moving on to a more complex open office area:

And finally, an outdoor area with moving vegetation and headlights on a nearby road:

Our tests looked to answer the following key questions:

  • What issues most commonly cause false positives?
  • Whose VMD was best at rejecting these events?
  • How did detection range vary from manufacturer to manufacturer?
  • Can settings be optimized for better detection without increasing false alerts and vice versa?

Key Findings

False positives were hands down the largest issue in our tests, with the following specific issues discovered:

Get Notified of Video Surveillance Breaking News
Get Notified of Video Surveillance Breaking News

  • No camera could truly be named "best" at rejecting false positives, since all suffered numerous activations from different conditions, day and night, at about the same rates. The Avigilon H3 1MP rejected alarms due to changes in light better than others. The Sony VB600 was activated by reflected headlights more than other cameras. Other than these two points, performance was about equal.
  • All cameras suffered numerous false positives due to the movement of vegetation in the scene, day or night, causing near continuous recording. In our scene, this area could be masked out without affecting performance, as no human movement was possible in this area.
  • However, shadows caused by moving branches caused many false positives which could not be excluded via motion masks, since they fell throughout the motion detection zone.
  • At night, all cameras were sensitive to car headlights in our scene. The Sony VB600 was sensitive to this even when the road was excluded from the motion zone.
  • Indoors, light changes were the largest cause of false alarms. These were due to doors opening/closing from a lit hallway, subjects moving past the door, or flashlights or headlights reflecting. The Sony VB600 as most prone to false alerts in this scene, followed by Bosch NBN-733V, Axis P3354, and Avigilon 1.0-H3-B2. 

The following key findings reflect performance in various test scenes using a human subject:

  • The Avigilon 1.0-H3-B2 performed well in indoor and outdoor scenes at all light levels, ranging from full light to dark (<1lux). The camera had the longest detection range among tested cameras and it rarely, if ever, missed detection of humans and vehicles in the test conditions.  
  • In outdoor daytime and full light indoor tests, Axis P3354, Bosch NBN-733V and Sony SNC-VB600 yielded similar motion detection performance in terms of detection ranges and false alert rates. 
  • In low light, VMD performance of Bosch NBN-733V and Sony SNC-VB600 were also similar, with Bosch yielding only slightly longer detection ranges  and slightly lower false alert rates.
  • Axis P3354, on the other hand, lagged significantly behind other cameras in low light, with much lower detection ranges.

Optimization Recommendations

Based on our tests, as well as discussions of best practices with each manufacturer, here are our recommendations on how each camera may be optimized for best performance:

  • For the Avigilon 1.0-H3-B2, default settings resulted in best performance in all scenes, with no recommended changes.
  • With Axis' P3354 Standard VMD, the default object size setting is too large for most scenes.  In order to get useful detection ranges, the object size setting needed to be reduced carefully to a level specific to the scene, generally 5 or below.
  • For the Bosch NBN-733V, best VMD performance was also obtained by using the default settings in the scenes we tested. Lowering the object size resulted in significantly higher false alerts due to shadows and wind. 
  • For the Sony SNC-VB600, the object size should be reduced to its minimum level for best performance. In the daytime, sensitivity can also be slightly increased from its default setting to get a longer detection range. Due to higher false alert rates in low light, default sensitivity is found to be more appropriate in these conditions.

Recommendations

Due to the prevalance of false alerts from moving leaves, branches, shadows, and light variations, users should carefully consider camera placement and exclude windows when planning to use video motion detection. Aiming cameras away from vegetation and active roadways when possible will greatly improve performance, even beyond using exclude windows on these areas.

Default VMD settings should not be assumed optimum for all cameras in all scenes. In some cases these settings will result in numerous false positives or missed detections. Users should plan to adjust these settings once cameras are in place, and settings should be revisited periodically to ensure performance is still satifactory.

For users requiring reliable motion detection at longer ranges (>100' during day time and >50' night-time), the Avigilon 1.0-H3-B2 stands out as the best option among the tested cameras. However, since it integrates with third parties via ONVIF, most VMSs will not receive motion events from the camera, limiting its use to end-to-end Avigilon systems.

Comparison to Built-In Analytics

Compared to the standard camera motion detection tested here, Axis Video Motion Detection 2.1 performed better in most scenes. Falses positives, rejection of shadows, and range were all improved, but low light performance was reduced somewhat. We recommend it be used with Axis cameras whenever possible, but its application is limited due to integration with very few VMSs (Axis and Genetec only).

Compared to their built-in VMD, Axis, Bosch, and Sony camera analytics such as cross line detection suffered from much lower false positive rates. This is due to the much narrow detection areas and specific rules required for activation. Users may want to consider these analytics instead of standard VMD where possible.

Configuration

The following videos review configuration of each manufacturer's VMD.

Avigilon H3

Axis P3354

This screencast reviews configuration of Axis' VMD. Note that this video also includes configuration of Axis Video Motion Detection 2.1, which we recommend over standard VMD whenever possible. However, this report focuses only on cameras' standard included VMD. 

Bosch NBN-733V

Sony 6th Gen

Outdoor Daytime 

Outdoors, in daylight, moving tree branches and leaves triggered continuous nuisance alarms on all cameras. We removed these areas using exclude windows, as testing of motion detection was not possible otherwise.

Even though we excluded the vegetation in the scene, all cameras gave false positives due to the movement of shadows in the scene. These areas could not be excluded as they were in the center of our motion zone.

The image below shows a section of a spider web in front of one camera. The movement of this web with the sun glaring off of it caused near continuous recording for two hours.

Detection Range

For our detection range test, we used a test scene about 25' wide from building to fence, and about 280' the nearby road. All cameras activated due to an object walking towards the cameras. However, Avigilon 1.0-H3-B2 picked up our subject at  ~ 100-120' away from the camera, while Axis P3354, Bosch NBN-733V and Sony SNC-VB600 triggered at the 45-50' range. Note that, Axis P3354 Standard VMD's object size needed to reduce from 15 (default) to 1 in order to increase detection range from 10' to 45'. Also, the Sony SNC-VB600 VMD's sensitivity setting was increased to 225 from the default 180, which enhanced the detection range by around 15'. Further increase in Sony's sensitivity resulted in continuous motion detection due to false positives.

Avigilon 1.0-H3-B2 also captured vehicles accurately on the nearby road (~280') during the day. Sony SNC-VB600, Bosch NBN-733V and Axis P3354 detected only about 40%, 15%, and 5% of vehicles, respectively.

Avigilon 1.0-H3-B2 was also able to detect trucks on the distant highway, over 600' from the camera.

 

Outdoor Nighttime

In our outdoor nighttime scene, there were two key triggers of false positives. As in the daytime, shadows from people and vegetation triggered motion:

Also, although we masked out the road to measure other factors in the test separately, Sony SNC-VB600 were still very sensitive to the headlights and it kept triggering continuously from the headlights.

Detection Range

In low light, Avigilon 1.0-H3-B2's human detection range decreased to about 50' but still yielded the longest range (Bosch at ~40', Sony at ~30', and Axis at ~20'). Note that we did not increase the sensitivity setting of Sony SNC-VB600 this time, since it had more tendency to generate false positives due to shadows in low light.

With our subject moving across the FOV at ~50', Avigilon 1.0-H3-B2 and Bosch NBN-733V detected him accurately. Sony SNC-VB600 caught him about half of the time, while Axis P3354 did not trigger at all.

At 70', none of the cameras triggered, with the subject simply too small to activate.

Indoor Conference Room

The Sony VB600 VMD had a false alarm rate of 10%-15% from the shadow on the background, while Bosch triggered about 5% of the time.

Opening the door changed the light in the scene and caused the following false alert rates: Sony 40%, Axis 20%, Bosch 20%. 

When waving a flashlight throughout the room, 

In our conference room in full light, all camera were able to reliably detect a human subject at  all ranges up to 20'. Note that the Axis P3354 could not detect our subject at maximum distance at default object size (15), and had to be reduced to 5 for accurate detection.

In low light (~1lx), all VMDs detected our object in 100% accuracy up to 20' in a similar manner to the full light test.

 

Open Office

Moving the cameras to an open office scene, about 40' long, full light performance of all cameras were reliable. Once again, the Axis P3354's must be reduced to 1 to improve detection range

In low light, the back of the office (~40') was lightened by a small background security light. Due to this, all cameras except Axis P3354 captured the object as soon as he entered the room 40' away.

However, as the object moved down the aisle to the darker side, Sony SNC-VB600 and Bosch NBN-733V lost detection at around 25' and re-captured the motion at about 15'. 

The Axis P3354, on the other hand, could only detect the object at 7' away from the camera. 

Cameras/Firmware Used

The following cameras and firmware versions were used in the test:

  • Avigilon 1.0-H3-B2: 2.4.10.4
  • Axis P3354: 5.40.17
  • Bosch NBN-733V: 5.85
  • Sony SNC-VB600: 1.5.1

 

Comments (8) : Members only. Login. or Join.

Related Reports

Mobotix 7 Line Camera Tested on Mar 12, 2020
Mobotix is attempting a turn-around, struggling for years, then releasing the OEMed Move line, Mobotix is aiming for its new 7th generation line to...
Dahua Smart Motion Detection Camera Tested on Mar 03, 2020
Dahua has introduced Smart Motion Detection, AI-based VMD, claiming to use an advanced algorithm to differentiate human and vehicular shapes within...
Camera Analytics Shootout 2020 - Avigilon, Axis, Bosch, Dahua, Hanwha, Hikvision, Uniview, Vivotek on Jan 22, 2020
Analytics are hot again, thanks to a slew of AI-powered cameras, but whose analytics really work? And how do these new smart cameras compare to top...
Bosch Budget 3000i Cameras Tested on Dec 05, 2019
Bosch has long had a hole in its lineup for, as it describes, "competitively-priced cameras". Now, Bosch has released its 3000i series cameras...
Illustra Pro Gen3 4K Camera Tested on Oct 23, 2019
Johnson Controls has released the latest generation in their Illustra Pro line, the Pro Mini-Dome Gen 3, a non-OEM model claiming to "optimize...
Avigilon H5A Analytic Cameras Tested on Oct 07, 2019
Avigilon has released its H5A analytic cameras, claiming to "detect more objects with greater accuracy even in crowded scenes." We tested the...
Vivotek 4K S-Series Camera Tested on Sep 30, 2019
Vivotek's highest-end S-series camera claims "Supreme Night Visibility", "Smart IR II", "Smart Stream II", "WDR Pro for unparalleled visibility in...
Axis ARTPEC-7 P1375-E Camera Tested on Jul 12, 2019
Axis claims the new P1375-E box camera with ARTPEC-7 chip delivers "clear, sharp images in any lighting condition." But how well does it do? We...
FLIR Saros Visible / Thermal Analytic Camera Tested on Jun 26, 2019
FLIR's Saros claims "accurate, actionable alerts" with a combination of 1080p visible and dual thermal sensors along with IR and white light...
Hikvision DeepinView Camera Analytics Tested on May 08, 2019
Hikvision is expanding its 'deep learning' offerings with a new camera series called 'DeepinView' claiming false alarm reduction and improved...

Most Recent Industry Reports

Pivot3 Mass Layoffs on Mar 27, 2020
Pivot3 has conducted mass layoffs, the culmination of grand hopes, a quarter of a billion dollars in VC funding, and multiple failures to gain...
Athena CEO Criticizes 'Deplorable' 'Nitpicking', IPVM Refutes on Mar 27, 2020
Athena Security's CEO Lisa Falzone has strongly objected to IPVM's reporting on Athena, calling it 'deplorable' and repeatedly criticizing IPVM's...
Hikvision Admits Sanctions Harming Its Financial Performance on Mar 27, 2020
While Hikvision initially downplayed being sanctioned for human rights abuses, the company is now admitting a significant impact in a new PRC...
New Axis M30 Cameras Tested on Mar 26, 2020
Axis has released a new generation of, for them, relatively low cost M30 series cameras, claiming to deliver "sharp video quality even in poor...
Coronavirus Shuts Down ADT Door Knockers on Mar 26, 2020
Coronavirus has another victim - this time, alarm giant ADT has stopped all door to door sales. Door knockers are a critical but controversial...
Access Control Course Spring 2020 - Save $50 Last Day on Mar 26, 2020
Register Now - Spring 2020 Access Control Course. Today, March 26th is the last day to save $50. IPVM offers the most comprehensive access...
Convergint Coronavirus Cuts on Mar 25, 2020
One of the world's largest security integrators, Convergint, has made a major move to handle the impact of coronavirus, with cuts across the...
VSaaS 101 on Mar 25, 2020
Video Surveillance as a Service (VSaaS) is the common industry term for cloud video. But what does it mean? How does it all work? Inside this...
TVT / InVid Facial Recognition Tested on Mar 25, 2020
Facial recognition is frequently sold for thousands of dollars per channel but some China manufacturers are offering full facial recognition...
IPVM Launches On-Demand Courses on Mar 24, 2020
For nearly a decade, IPVM has been a leader in online live courses. Now, we have added on-demand versions for all courses. The same course...