We are seeing artifacts on video when using Avigilon H3 platform fixed dome cameras on a very well performing wireless network and using ACC 4.0+ Enterprise software. We have not seen such symptoms with other camera manufacturers and VMS platforms even in cases where there was much more packet loss.
Has anyone run into a similar issue before? Is there a solution you'd recommend?
What is the bit rate of that Avigilon camera? What is the throughput of the wireless network?
When are these artifacts occurring? Continuously? During certain times of the day?
What wireless are you using?
What type antennes are you using?
Why are you having packet loss? A well functioning Wireless Link should work equally well as a layer 1 physical replacement for copper wire. If that isn't working correctly you can't pass video across that link with any camera.
"you can't pass video across that link with any camera."
Andrew, as I am sure you know, not all cameras act the same, not all wireless links have the same issues with video. Most obviously, cameras have different bandwidth utilizations. He might be trying to send a 1Mb/s stream or he may be sending a 12Mb/s stream.
Before we dismiss this, I would like to understand more about the specifics of what he is doing.
I would expect if you put any other camera up on this same wireless network you would experience the same issue. We have literally hundreds of Avigilon and other cameras installed over wireless networks with no problems. The only time you will see artifacts in the cameras is if you have packet loss which you eluded to the fact you do have some packet loss in the statement "...where there was much more packet loss." Any packet loss in an H.264 or other interframe compression (really any compression) video transmission will cause artifacting and will continue through the GOP until the packet loss stops on the affected stream(s).
What version of HDSM y'all running?
It's an H3 camera, so that means H.264 and multi-streaming.
Handy reference to HDSM:
The underlying protocols of all cameras and other devices are held to the same standard and must respond in accordance to the requirements of the associated RFC's of TCP or UDP. The implication another camera works fine, and the reference to packet loss leads on to jump to a conclusion of which I'm guilty. It Video and VoIP applications Latency is everything. Before you blame any camera, diagnosing the wireless link, developing some good tests is critical. Starting with Ping and the option to increase packet sizes is a good first step. Below is a four hour graph of basic latency testing of four Avigilon 5MP H3 cameras as part of a "Civic Surveillance" solution linked with a 50Mb wireless link at a distance of 1,200' By day the four cameras generate about 35Mbs traffic and 20Mbs by night. The IPS rates are at 7, and the image quality (less compression) is near max, and preference is image quality vs. Frame rate. When we shoot over a mile, latency tends to drift upward of 10ms.
Everything Wireless is really starts with having a clear shot, followed by the right antenna's, frequency and options to secure, like WORP, or other proprietary FreqSkip protocols... the artifact is a symptom of the underlying issue, and I would focus on getting basic PING responses to be consistent and predictable. An interesting note, setting the output power on a radio to MAX, isn't always the best setting.
Thank you Andrew. Additional details provided above.
Out of curiosity, what amount of latency do you find acceptable for Avigilon cameras?
Thank you all who responded with requests for more information and comments. Here's additional data for those interested in digging deeper with us.
The bit rate on Avigilon cameras is set not to exceed 10 Mbps. There are 2 cameras on this particular link. The following screenshot shows 11.9 Mbps used at that point in time by 2 cameras on this link.
The throughput on this wireless link is capped at 60 Mbps:
The artifacts occur throughout the day and night. We are in the process of seeing if there is a pattern. While most of the video contains no artifacts and has high level of detail, an occasional artifact does occur.
We are using Hautespot Wireless equipment with MIMO directional antennas (dual polarity V + H).
The following screenshot shows additional detail for the wireless link with high rate of availability, which is normal for this link. Having said that, this is a busy metropolitan area with 160+ wireless access points which we identified during our interference survey.
Further detail on link quality:
Here's a weekly view showing traffic generated by 2 cameras over a 60 Mbit/s link
Sir, may we request the following additional data:
One possible test if feasible is to run just one camera at a time to see the effect.
Thank You
Packet count, and throughput do not tell the story, I assume these radios have some built in diags, Pinging from Radio to Radio is the issue.
A quick read of the NV2 protocol indicates they may be forcing resyncs occassionally with new and existing client radios. That may be a random process, and might be burning CPU cycles on the radios. I see these are set as WDS mesh, can you configure these radios in Pure Bridge Mode? I would also see if the link can be set in V2 with a share encryption key, and forgo their auto sync stuff in their proprietary TDMA scheme.
Max rates for anything is inversely related to latency. drive up the latency, something has to give.... frame rates, image quality or both.
We use ubiquiti nsm5s - 90 bucks each 150mbs throughput never any problems. I would highly recommend them. Rock solid.
There are a number of questions this raises:
1. How are the cameras configured to stream? RTP over TCP, RTP over UDP, Multicast, other? If the cameras are using TCP over wireless, you can get delay due to acknowledgement round trip from server. Even if router to router ping round trip is low, this could still be trouble. And some camera/server combinations have issues even using UDP where the application layer is responsible retransmissions.
2. How much interference is there? Have you coorelated noise floor, BER, etc to video performance. Could be that you are just not seeing some interference when you are checking the network. Turn on NTP and make sure that cameras and routers all pull time from the same source. Then turn on syslogd with debug logging from your router and cameras. Then compare the two.
3. Have you tested this specific wireless link with another camera brand of same resolution and frame rate?
4. Can you run dynamic frequency selection or use a script to change frequencies when BER hits a certain threshold?
With wireless, in heavily congested areas, you cannot always find easy solutions. Something that works in the parking lot where there is little interference may not work in the center of the shopping center where every shop has an AP.
Thank you all for your input. As a follow up - the issue was managed by reducing the number of "hops" and limiting wireless point-to-point connections to a single link.
Ask questions and get answers to your physical security questions from IPVM team members and fellow subscribers.