VMD Camera Shootout

By: Ethan Ace, Published on Oct 28, 2013

Video motion detection's inaccuracy can have disastrous effects on your VMS system, with recordings due to positives easily consuming far more storage than estimated. On the other hand, optimizing VMD to reject these alarms can decrease detection rates, missing real events.

In order to test whose on-camera VMD performed best, we used cameras from Avigilon, Axis, Bosch, and Sony in multiple scenes, day and night, starting with a simple indoor conference room:

Moving on to a more complex open office area:

And finally, an outdoor area with moving vegetation and headlights on a nearby road:

Our tests looked to answer the following key questions:

  • What issues most commonly cause false positives?
  • Whose VMD was best at rejecting these events?
  • How did detection range vary from manufacturer to manufacturer?
  • Can settings be optimized for better detection without increasing false alerts and vice versa?

Key Findings

False positives were hands down the largest issue in our tests, with the following specific issues discovered:

Get Notified of Video Surveillance Breaking News
Get Notified of Video Surveillance Breaking News

  • No camera could truly be named "best" at rejecting false positives, since all suffered numerous activations from different conditions, day and night, at about the same rates. The Avigilon H3 1MP rejected alarms due to changes in light better than others. The Sony VB600 was activated by reflected headlights more than other cameras. Other than these two points, performance was about equal.
  • All cameras suffered numerous false positives due to the movement of vegetation in the scene, day or night, causing near continuous recording. In our scene, this area could be masked out without affecting performance, as no human movement was possible in this area.
  • However, shadows caused by moving branches caused many false positives which could not be excluded via motion masks, since they fell throughout the motion detection zone.
  • At night, all cameras were sensitive to car headlights in our scene. The Sony VB600 was sensitive to this even when the road was excluded from the motion zone.
  • Indoors, light changes were the largest cause of false alarms. These were due to doors opening/closing from a lit hallway, subjects moving past the door, or flashlights or headlights reflecting. The Sony VB600 as most prone to false alerts in this scene, followed by Bosch NBN-733V, Axis P3354, and Avigilon 1.0-H3-B2. 

The following key findings reflect performance in various test scenes using a human subject:

  • The Avigilon 1.0-H3-B2 performed well in indoor and outdoor scenes at all light levels, ranging from full light to dark (<1lux). The camera had the longest detection range among tested cameras and it rarely, if ever, missed detection of humans and vehicles in the test conditions.  
  • In outdoor daytime and full light indoor tests, Axis P3354, Bosch NBN-733V and Sony SNC-VB600 yielded similar motion detection performance in terms of detection ranges and false alert rates. 
  • In low light, VMD performance of Bosch NBN-733V and Sony SNC-VB600 were also similar, with Bosch yielding only slightly longer detection ranges  and slightly lower false alert rates.
  • Axis P3354, on the other hand, lagged significantly behind other cameras in low light, with much lower detection ranges.

Optimization Recommendations

Based on our tests, as well as discussions of best practices with each manufacturer, here are our recommendations on how each camera may be optimized for best performance:

  • For the Avigilon 1.0-H3-B2, default settings resulted in best performance in all scenes, with no recommended changes.
  • With Axis' P3354 Standard VMD, the default object size setting is too large for most scenes.  In order to get useful detection ranges, the object size setting needed to be reduced carefully to a level specific to the scene, generally 5 or below.
  • For the Bosch NBN-733V, best VMD performance was also obtained by using the default settings in the scenes we tested. Lowering the object size resulted in significantly higher false alerts due to shadows and wind. 
  • For the Sony SNC-VB600, the object size should be reduced to its minimum level for best performance. In the daytime, sensitivity can also be slightly increased from its default setting to get a longer detection range. Due to higher false alert rates in low light, default sensitivity is found to be more appropriate in these conditions.

Recommendations

Due to the prevalance of false alerts from moving leaves, branches, shadows, and light variations, users should carefully consider camera placement and exclude windows when planning to use video motion detection. Aiming cameras away from vegetation and active roadways when possible will greatly improve performance, even beyond using exclude windows on these areas.

Default VMD settings should not be assumed optimum for all cameras in all scenes. In some cases these settings will result in numerous false positives or missed detections. Users should plan to adjust these settings once cameras are in place, and settings should be revisited periodically to ensure performance is still satifactory.

For users requiring reliable motion detection at longer ranges (>100' during day time and >50' night-time), the Avigilon 1.0-H3-B2 stands out as the best option among the tested cameras. However, since it integrates with third parties via ONVIF, most VMSs will not receive motion events from the camera, limiting its use to end-to-end Avigilon systems.

Comparison to Built-In Analytics

Compared to the standard camera motion detection tested here, Axis Video Motion Detection 2.1 performed better in most scenes. Falses positives, rejection of shadows, and range were all improved, but low light performance was reduced somewhat. We recommend it be used with Axis cameras whenever possible, but its application is limited due to integration with very few VMSs (Axis and Genetec only).

Compared to their built-in VMD, Axis, Bosch, and Sony camera analytics such as cross line detection suffered from much lower false positive rates. This is due to the much narrow detection areas and specific rules required for activation. Users may want to consider these analytics instead of standard VMD where possible.

Configuration

The following videos review configuration of each manufacturer's VMD.

Avigilon H3

Axis P3354

This screencast reviews configuration of Axis' VMD. Note that this video also includes configuration of Axis Video Motion Detection 2.1, which we recommend over standard VMD whenever possible. However, this report focuses only on cameras' standard included VMD. 

Bosch NBN-733V

Sony 6th Gen

Outdoor Daytime 

Outdoors, in daylight, moving tree branches and leaves triggered continuous nuisance alarms on all cameras. We removed these areas using exclude windows, as testing of motion detection was not possible otherwise.

Even though we excluded the vegetation in the scene, all cameras gave false positives due to the movement of shadows in the scene. These areas could not be excluded as they were in the center of our motion zone.

The image below shows a section of a spider web in front of one camera. The movement of this web with the sun glaring off of it caused near continuous recording for two hours.

Detection Range

For our detection range test, we used a test scene about 25' wide from building to fence, and about 280' the nearby road. All cameras activated due to an object walking towards the cameras. However, Avigilon 1.0-H3-B2 picked up our subject at  ~ 100-120' away from the camera, while Axis P3354, Bosch NBN-733V and Sony SNC-VB600 triggered at the 45-50' range. Note that, Axis P3354 Standard VMD's object size needed to reduce from 15 (default) to 1 in order to increase detection range from 10' to 45'. Also, the Sony SNC-VB600 VMD's sensitivity setting was increased to 225 from the default 180, which enhanced the detection range by around 15'. Further increase in Sony's sensitivity resulted in continuous motion detection due to false positives.

Avigilon 1.0-H3-B2 also captured vehicles accurately on the nearby road (~280') during the day. Sony SNC-VB600, Bosch NBN-733V and Axis P3354 detected only about 40%, 15%, and 5% of vehicles, respectively.

Avigilon 1.0-H3-B2 was also able to detect trucks on the distant highway, over 600' from the camera.

 

Outdoor Nighttime

In our outdoor nighttime scene, there were two key triggers of false positives. As in the daytime, shadows from people and vegetation triggered motion:

Also, although we masked out the road to measure other factors in the test separately, Sony SNC-VB600 were still very sensitive to the headlights and it kept triggering continuously from the headlights.

Detection Range

In low light, Avigilon 1.0-H3-B2's human detection range decreased to about 50' but still yielded the longest range (Bosch at ~40', Sony at ~30', and Axis at ~20'). Note that we did not increase the sensitivity setting of Sony SNC-VB600 this time, since it had more tendency to generate false positives due to shadows in low light.

With our subject moving across the FOV at ~50', Avigilon 1.0-H3-B2 and Bosch NBN-733V detected him accurately. Sony SNC-VB600 caught him about half of the time, while Axis P3354 did not trigger at all.

At 70', none of the cameras triggered, with the subject simply too small to activate.

Indoor Conference Room

The Sony VB600 VMD had a false alarm rate of 10%-15% from the shadow on the background, while Bosch triggered about 5% of the time.

Opening the door changed the light in the scene and caused the following false alert rates: Sony 40%, Axis 20%, Bosch 20%. 

When waving a flashlight throughout the room, 

In our conference room in full light, all camera were able to reliably detect a human subject at  all ranges up to 20'. Note that the Axis P3354 could not detect our subject at maximum distance at default object size (15), and had to be reduced to 5 for accurate detection.

In low light (~1lx), all VMDs detected our object in 100% accuracy up to 20' in a similar manner to the full light test.

 

Open Office

Moving the cameras to an open office scene, about 40' long, full light performance of all cameras were reliable. Once again, the Axis P3354's must be reduced to 1 to improve detection range

In low light, the back of the office (~40') was lightened by a small background security light. Due to this, all cameras except Axis P3354 captured the object as soon as he entered the room 40' away.

However, as the object moved down the aisle to the darker side, Sony SNC-VB600 and Bosch NBN-733V lost detection at around 25' and re-captured the motion at about 15'. 

The Axis P3354, on the other hand, could only detect the object at 7' away from the camera. 

Cameras/Firmware Used

The following cameras and firmware versions were used in the test:

  • Avigilon 1.0-H3-B2: 2.4.10.4
  • Axis P3354: 5.40.17
  • Bosch NBN-733V: 5.85
  • Sony SNC-VB600: 1.5.1

 

Comments (8) : Members only. Login. or Join.

Related Reports

Uniview Deep Learning Camera Tested on Jul 14, 2020
Uniview's intrusion analytics have performed poorly in our shootouts. Now,...
Use Access Control Logs To Constrain Coronavirus on Apr 09, 2020
Access control users have included capabilities that are not commonly used...
Directory of 206 "Fever" Camera Suppliers on Aug 04, 2020
This directory provides a list of "Fever" scanning thermal camera providers...
FLIR A Series Temperature Screening Cameras Tested on Jun 04, 2020
FLIR is one of the biggest names in thermal and one of the most conservative....
FLIR Screen-EST Screening Software Tested on Jun 30, 2020
In our FLIR A Series Test, the cameras' biggest drawback was their lack of...
Door Fundamentals For Access Control Guide on Aug 24, 2020
Doors vary greatly in how difficult and costly it is to add electronic access...
False: Verkada: "If You Want To Remote View Your Cameras You Need To Punch Holes In Your Firewall" on Jul 31, 2020
Verkada falsely declared to “3,000+ customers”, “300 school districts”, and...
US Startup Fever Inspect Examined on Aug 03, 2020
Undoubtedly late to fever cameras, this US company, Fever Inspect, led by a...
Faked Coronavirus Fever Detection, Athena Used Hikvision; Responds - Selling NDAA Compliant Cameras, Pledging 50% Of Profits to Victims on Mar 24, 2020
US company, Athena Security, faked its coronavirus fever detection marketing,...
Surveillance Storage 101 on Mar 23, 2020
This guide teaches the fundamentals of video surveillance...
Verkada Falsely Claims "First Native Cloud-based Access Control and Video Security Solution" on Jun 18, 2020
Verkada's false claims continue, this time to be the first native cloud-based...
Injes Tiny Temperature Terminal Tested on Jul 17, 2020
While temperature terminals have trended bigger, the Injes DFace801 is...
Face Masks Increase Face Recognition Errors Says NIST on Aug 04, 2020
COVID-19 has led to widespread facemask use, which as IPVM testing has shown...
FLIR Markets Windows Temperature Screening, Violates IEC And Causes Performance Problems on Jul 17, 2020
FLIR, one of the largest thermal screening manufacturers, is marketing...
Don't Be Fooled By Hot Water Bottle Fever Camera Demos on Aug 24, 2020
Fever camera salesmen like to fool buyers (and themselves) with hot water...

Recent Reports

Axis Compares Fever Camera Sellers to 9/11 on Sep 18, 2020
Axis Communications, the West's largest surveillance camera manufacturer, has...
Avigilon Elevated Temperature Detection Camera Tested on Sep 17, 2020
Avigilon has entered the temperature screening market with the release of...
Chilean Official Investigated for Motorola And Hikvision Contracts on Sep 17, 2020
A corruption investigation is underway in Chile after a crime prevention...
Huawei HiSilicon Production Shut Down on Sep 17, 2020
Huawei HiSilicon chips are no longer being manufactured or supplied to...
Virtual ISC West and GSX+ Exhibiting Contrasted on Sep 17, 2020
Both ISC West and ASIS GSX are going virtual this year, just weeks apart, but...
X.Labs Sues FLIR on Sep 16, 2020
X.Labs, the maker of Feevr, has sued FLIR, the publicly traded thermal...
Video Surveillance 101 September Course - Last Chance on Sep 16, 2020
Today is the last chance to sign up for the Fall Video Surveillance 101...
No Blackbody Mistake, Half Million Dollar, Hikvision Fever Camera System in Georgia on Sep 16, 2020
A Georgia school district touted buying Hikvision fever screening "about...
Costar Technologies / Arecont H1 2020 Financials Examined on Sep 16, 2020
Costar's financial results have been hit by the coronavirus with the company...
Startup Cawamo Presents Live Alerts With Edge AI and Cloud VMS on Sep 15, 2020
Cawamo, an Israeli edge-to-cloud analytics and VMS startup, presented its...
Favorite Access Control Credentials 2020 on Sep 15, 2020
Credential choice is more debated than ever, with hacking risk for 125kHz and...
Dangerous Hikvision Fever Screening Marketing In Africa on Sep 15, 2020
A multi-national African Hikvision distributor is marketing dangerously...
New Products Show Fall 2020 Announced - Register Now on Sep 14, 2020
IPVM's sixth online show will feature New Products from over 25...
Hanwha 8K / 33MP Camera Tested on Sep 14, 2020
Hanwha Techwin has released an 8K / 33MP resolution camera, the TNB-9000 with...
Gait Recognition Examined on Sep 14, 2020
Facial recognition faces increasing ethical and political criticisms while...