Camera Analytics Tested: Axis vs. Bosch vs. SonyBy Ethan Ace, Published on Sep 17, 2013
With the move to IP and HD now commonplace, companies are looking for new ways to compete. While analytics has long disappointed, many still hold out hope for it being the next big thing.
Many major camera manufacturers offer analytics built-in or for a minimal additional charge. However, skepticism of their performance is commonplace.
In this report, we tested 3 camera manufacturers analytics - Axis cross line detection, Bosch's Intelligent Video Analytics [link no longer available] and Sony's Video Motion Filter (VMF).
The test focuses on tripwire / perimeter violation analytics as this is most common type used in security to alert against intruders entering into a restricted area.
We tested in 3 different scenes.
To start, we did a simple indoor tripwire at an entrance:
Then we progressed to an outdoor scene of moderate complexity and distance:
Finally, we tried a very long area with vegetation and shadows to maximize the challenge:
Here are our key findings from these tests:
- Cross line detection performs best when the angle of incidence is as high as possible. This reduces false or early activations due to shadows or shallow perspective.
- Cross lines perform best when configured for outside-in crossing, not inside-out, with the object moving toward the camera.
- False alerts from background objects such as moving foliage and headlights were possible, but not common. Lines should be drawn to avoid such areas where possible.
- All of the above combined mean that typical camera locations may not always provide best performance for cross line analytics.
- Axis P3354 Cross Line Detection detected objects at more ranges in more scenes under varying lighting than either Bosch NBN-733V or Sony SNC-VB600. However, it suffered from false activations, caused by shadows or shallow angle of incidence, and often activated multiple times.
- Bosch NBN-733V's IVA triggered reliably in well lit scenes, far better than either Sony SNC-VB600 or Axis P3354 , with no false or multiple activations. However, in low light detection range was drastically reduced, and missed activations increased.
- Sony SNC-VB600's built-in VMF lagged in our tests, only able to reliably detect people at very close range. Vehicles were detectable at longer distances, though short of Bosch and Axis. This is due at least in part to object size settings in Sony SNC-VB600's analytics not required in other cameras, which failed to detect objects if set to large, and detected single objects as multiple if set too small, missing detections.
Cost of the built in analytics in this test vary by manufacturer:
- Axis P3354 Cross Line Detection: ~$30 USD, with discounts on 10-packs of licenses.
- Bosch NBN-733V IVA: 900 series cameras ship with IVA installed at no additional charge. 700 and 800 series models may be licensed at an additional cost of ~$100.
- Sony SNC-VB600 VMF: Sony SNC-VB600's built-in analytics are available at no additional cost in 6th generation cameras. Fifth generation cameras previously shipped with DEPA analytics.
Cross line analytics may offer some useful advantages, but must be applied carefully, as mounting locations and FOV of typical cameras may not always be ideal. Users should carefully consider these factors and test analytics in place before relying upon them in the field.
In most scenes, with the exception of very low light, Bosch NBN-733V's IVA was most accurate, with almost no false or missed activations. Users should be aware of its low light limitations before deploying, however.
Axis P3354 Cross Line Detection was reliable in most scenes, and performed better in very low light than Bosch NBN-733V. However, users should be aware of its false activations due to headlights, shadows, and moving foliage, before deploying.
We do not recommend Sony SNC-VB600's VMF analytics in scenes aside from well lit indoor areas due to its issues detecting objects.
In these screencasts, we review configuration for each of the three manufacturers tested.
First, Axis P3354 Cross Line Detection. Configuration is very simple, only including the line (1 or 2 segments) and activation direction, with no other setup.
Next, Bosch NBN-733V IVA. The NBN-733V offered the most configuration options, including multiple lines, fine tuning of speed, size, contrast, and other parameters. It also presented information regarding objects detected and their path through the scene, making troubleshooting easier compared to others which did not display this.
Finally, Sony SNC-VB600's VMF setup contains adjustment for object size and speed, as well as standard settings such as line length and direction. Up to three lines can be set per camera, and combined into events. Users should be careful when configuration object size, as it can drastically affect detection performance.
We tested the cross line analytics in several common outdoor scenarios.
First, we began with a parking lot/driveway scene at ranges up to 250', shown in this overhead view:
During the day, beginning at 25', all cameras are accurate.
Moving about halfway down the parking lot to ~120', Axis P3354 and Bosch NBN-733V once again trigger reliably on the vehicle. At this distance and beyond, Axis P3354 occasionally tirggered twice or more in quick succession, as the vehicle approached and crossed the line. Bosch NBN-733V triggered only once. The VB600 was unable to reliably detect objects at this range, so did not trigger.
Finally, with the line located all the way at the entrance to the lot from the road, both Axis P3354 and Bosch NBN-733V were still accurate, though Axis P3354 still activated multiple times as it did at 120'.
At night, at close range, Axis P3354 and Bosch NBN-733V both trigger, while Sony SNC-VB600 does not. Bosch NBN-733V performed better when the vehicle was moving away from the camera, not toward, as it detected the object much more easily in the foreground.
At ~120' and beyond, only Axis P3354 triggers. Some activations were slightly early due to the reflection of headlights moving toward the camera.
At long distance, ~250', Axis P3354 still triggers. However, occasional false alarms were caused by cars passing on the roadway beyond the parking lot. Moving the line back from the road partway remedies this.
Long Range Tripwire
We also tested the cameras in the rear of our office, drawing a virtual tripwire inside the vegetation at the perimeter, another common application for cross line analytics.
At 100', both Axis P3354 and Bosch NBN-733V activated, while Sony SNC-VB600 did not (nor did it at any longer ranges):
At 200', Bosch NBN-733V continued to track and alert on our subject, while Axis P3354 was unable to:
At 300', none of the cameras triggered, with the subject simply too small to activate:
At night, with the outdoor lighting on, approximately 2-3 lux, both Axis P3354 and Bosch NBN-733V triggered reliably at 50' range.
At 75', Bosch NBN-733V still triggered, while Axis P3354 did not. Beyond this, no cameras were able to detect our subject.
Switching off the outdoor lighting drops light levels below 1 lux. Detection range of Bosch NBN-733V drops to only 20', shown here.
Axis P3354 was able to detect our subject reliably up to 75', however.
One key application often cited for crossline analytics is detection of subjects entering a doorway. We set up our cameras in front of the office doors to simulate this, drawing our tripwire at the bottom of the door, with the cameras about 12' away. In this setup, Axis P3354 and Sony SNC-VB600 worked reliably, but Bosch NBN-733V did not trigger:
This is due to the way Bosch NBN-733V tracks objects, with the object path drawn higher, between the subject's knees and waist. At the above angle, the subject's path simply never crossed the line. Moving it partway up the doorway, Bosch NBN-733V triggers reliably, as does Axis P3354 . However, depending on the exact installation scenario, this line location may cause false triggers on object moving outside the door. Users should be careful to avoid this.
Thought not a common use for cross line analytics, we performed some tests in our controlled conference room in order to demonstrate some issues common throughout our tests.
First, this image shows Axis P3354 Cross Line Detection's sensitivity to shadows. In this case, though the line was set to trigger only when the subject moved toward the camera, the motion of his shadow on the floor caused false activations as he moved across the line in the opposite direction:
This occurred as he moved across the room, as well.
In this case, the line was drawn to trigger as the subject moved out of the door at the other end of the room. However, all three cameras failed to activate if he moved straight out of the door perpendicular to the cameras.
Detection was more reliable if he moved at more of an angle or toward the camera.
However, users are better served using a single vertical line next to the door, so the subject crosses no matter what their path once in the room.
We used the following cameras and firmware versions for this test:
- Axis P3354, firmware 5.40.17
- Bosch NBN-733V, firmware 5.80 build 73
- Sony SNC-VB600, firmware 1.5.1
All cameras used default settings, with exposures standardized to 1/30s maximum.