Analytics vs Rain: Axis, Bosch & Sony

Rain. This was a key objection to our Camera Analytics Test: Axis vs. Bosch vs. Sony. Members rightfully noted it is one thing for an analytics to do well indoors and outdoors at night but what about in the rain?

In this report, we share our findings of testing the same camera analytic in moderate to heavy rain. Here's what the day looked like:

We set up crosslines / tripwires from 20' to 200' away from our cameras.

Additionally, we tested a crossline perpendicular to the camera from 20' to 200' away.

Here are our key findings from this test:

• No false positives were generated due to either rain (moderate-to-heavy) or droplets falling from camera housings.
• Cross line performance was slightly reduced at long ranges (at and over ~150') in rain, when compared to the results from our Camera Video Analytics shootout. 
• With a human subject, the Bosch NBN-733V's IVA and Axis Crossline Detection started to have missed activations at 150' and completely missed activations at 200'. With no rain conditions, the true alert rate was 100% until 150' and 200' for Axis Crossline and Bosch IVA, respectively. Detection success rates of Sony SNC-VB600's VMF continued to be low even at close ranges (70% at 20', 40% at 45' and 0% at 100') consistent with the no-rain results. 
• When the object was a vehicle, no major differences in the detection performance of each camera were observed, compared to the test in no-rain condition.  

Recommendations

In addition to our overall recommendations of the previous report, these tests show that rain and other weather conditions will also be an important factor to consider when designing a reliable cross-line detection system. The maximum detection distance will be reduced in rain and it should be taken into account, together with other factors, such as mounting location, FOV and lighting conditions. Particular attention is needed when the target object is crossing an imaginary tripwire at long distances over 100-150' away.

However, false positive / alerts, while understandably feared, did not demonstrate increased problems due to the rain.

Test

We tested the cross line analytics in two common outdoor scenarios:

Tripwire

First, we began with setting a virtual tripwire along our parking lot/driveway, as shown in the overhead view below. 

Our subject crossed the tripwire 10 repetitive times at various distances from 20' to 250' away from the camera.

At 20', both Axis P3354 and Bosch NBN-733V detected with full accuracy, while Sony SNC-VB600's success rate was 7/10.

At distances between 20' and 75', success rate of VB600 decreased gradually and it missed all activations at 100'. P3354 and NBN-733V continued to perform reliably at this range.

At 150', both the P3354 and NBN-733V started to miss some activations. This is lower performance from both when compared to their accuracy at this range in our previous no rain test.

At 200', none of the cameras triggered at all. In clear conditions, the NBN-733V was found to be 100% accurate 200'.

Detection success rates of each camera from the virtual tripwire tests have been tabulated below. Check marks mean 100% success rate, while an X denotes full failure to detect. Since we have not observed any false alerts in this test, success rate is based on true alerts in the summary table.

Vehicle on Driveway

We also tested the accuracy of the crossline analytics of three cameras, targeting vehicle detection at three distances; 25', 100' and 250'. This scene was similar to the tripwire test, but lines were set up on the driveway, across the FOV.

Neither the Axis P3354 or Bosch NBN-733V missed any of the vehicle activations at any of the three distances. The Sony VB600, on the other hand, had missed activations even at the closest test range of 25', and gradually more at longer distances. These results were similar to what we saw in the no rain test. Therefore, we have not observed any effect of the rain in the detection range of bigger objects, such as vehicles, until 250'.

At ~100' Sony's accuracy drops to ~50%.

Finally, at 250', the SNC-VB600 is unable to detect our vehicle at all.

The table below summarizes overall performance of vehicles in the rain: