Avigilon System - Flickering Video And Unresponsive USB Joystick

What are the common reasons for the following problems?

1. Flickering of Video display (even on recorded video)

2. Unresponding usb Joystick Controller

The brand is Avigilon.

Specs:

100+ cameras

4MN-HD-RMWS (Workstation) x 6

21.0TB-HD-NVR2 HD NVR, 21.0TB Storage, 2U Rack Mount(VMS Server) x 6

Backbone : Fiber Optic

Cat 6 UTP

HP 1920-24G-PoE+ (370W)

Can you also give me probable solutions or ways to troubleshoot this. I dont even know if these problems is common to IP CCTV System.

Thanks in Advance.


Have you talked with their tech support? Their tech support is good and answers anyone's questions so it would be helpful to start with them. If the problem still exists, share feedback from what they suggested and then we can provide better advice.

I already advised our system integrator to get technical support from avigilon.

is the flickering video on all cameras ? - then it is most likely network based

is it just a few coloured lines through the image like a tear ? - bandwidth from camera to NVR not enough

or when it is flicking does the whole image go black ? - ive had this issue when i had the switches in a stack

Flickering of Video display (even on recorded video).

So there is flickering when playing back recorded video even when on the viewed directly on the VMS server machine?

Or just the client? Does it flicker if you export a clip? If so can you export a clip and post a link to it?

What kind of monitor/video cards do you have?

Check the line frequency setting - 50hz vs 60hz vs off. Many cameras have an anti-flicker or flickerless setting in the camera image adjustment for this...

If it is on both live and recorded video it is most likely packet loss from the cameras to the server.

Your integrator should help you with this. However, you can call Avigilon technical support directly at 1.888.281.5182 - they can remote into your system and help you.

We have dealt with this issue before it has nothing to do with 50hz or 60hz setting in the cameras.

If this issue is happening on both live and recorded video on multiple machines it is a bandwidth issue from your cameras to the server. What is your total throughput on your servers? Are your servers connected to the switch via Gig ports? Are your uplink ports between your switches connected via Gig ports?

Contact your integrator first as they should be able to provide you with a network map and confirm these questions. Then contact Avigilon support.

If all else fails I would be more then happy you help you out off line.

So is the flickering caused by black frames or partially black frames generated during the packet loss?

Like Mike,

We've seen this issue, and in 100% of all cases it was transport related. That includes updates applied to a server, that resulted in the LAN drivers having an issue. I can't find some screen captures we've taken in the past, but the symptom is more a horizontal tearing that is recorded in the video. With every new server, or a server we are responsible for, and at any chance we are getting the latest LAN drivers from BC or Intel.

We've also seen where a client had two NICS, each with an IP and gateway on the same subnet...
I would start by looking at the Totalbytes received using performance monitor. We've ran the Dell OEM 7xx and 5xx servers up to 35Mbytes/sec for almost a year before adding another server.

Never let a user view video on the server console monitor.

LAN drivers are equally important on a multi-monitor viewer (quad 30' in @ 2550 x 1600) for example... Older XP machines (disable hardware excel) (viewing only issues of course)

To better isolate the issue, crank up a handful of cameras to the best settings and highest frame rates..

We've tested this with 6 7K cameras, all set to MAX, plus many more... running a 1G segment up to near 50%, and over 500mbits into a server... after that we started having record issues... Duh...

Avigilon has decent disk bandwidth tool that will saturate . test the disk subsystem. You might have a failing disk or under performing disk.... Tech support can collect the data....

Happy to comment privately, or share some tools...

That is packet loss.

Mostly it occurs only at night

Chances are this is a easy fix you just need to find the current choke point for the bandwidth. Check server NICs for total bandwidth and then check your switches. Good luck.

This could be what is known as "Black blocking"

I had this issue several years ago and stacking the switches was the cause.
I still see this whenever a new camera is connected to the network but dies down once the camera has been added.

Create new views of the cameras that are connected to each switch, and also each server. look at the views and see which one is the worst one and it could point you to the switch or nvr that has the issue.

good luck

We know what Anixter would say...;)

I have no idea :)

How can i contact you guys? Thanks

You can click the mailbox icon to the right of their name to send a message.

Im using a mobile phone, icant.see it

First,

You may have tried to send me a PM, but McAfee email blocked it for a content violation....

Let make assumption - all of the displays are on a local workstation displaying video from one or more servers. - Correct?

If you go back and watch the same footage, the errors are recorded? - Right.

Is the frequency of these events consistent throughout a 24 hour period?

Do you know how to get the received bytes using performance monitor?

Did someone add an additional INTEL dual port NIC card? There are several SERVER models from INTEL (they acknowledge) were bad on PCI slots we've discarded for this and other problems.

Assuming these events are randomly consistent, and across all switches and cameras, I am hair triggered to make sure you are using the latest release drivers direct from Intel... I can confirm what we are using on the Dell OEM7xx and OEM5xx HD servers.....

One more thing to check is the make/model of the switches and the age of their firmware, and the current UPTIME... We've seen switches,, CISCO, NETGEAR, and others that become flacky after a a year or so of uptime..... STG grapher by leonid Mikhailov is a great tool to have to plot Switch CPU,RUNTS,Collisions and speed too..... A fresh reboot on the switches is in order too.

Let make assumption - all of the displays are on a local workstation displaying video from one or more servers. - YES

If you go back and watch the same footage, the errors are recorded? - Right. YES

Is the frequency of these events consistent throughout a 24 hour period? No. Only during at night

Do you know how to get the received bytes using performance monitor? Nope. but i think ACC have a Server status where you can monitor the performance of the server and each camera connected on the server.

Did someone add an additional INTEL dual port NIC card? There are several SERVER models from INTEL (they acknowledge) were bad on PCI slots we've discarded for this and other problems. Each server have two Network adapters

Every time I have seen this issue it has been a simple fix.

Uplink port plugged into 10/100 port

Missed matched fiber patch cable (single mode fiber patch cable on multi mode fiber or 50/125 patch used on 62/125 fiber)

Bad patch cable

Faulty punch on a patch panel

Miss configured switch

Hopefully you contacted your integrator as they should be able to fix this. Without having a network diagram and the bandwidth info on your servers ports its hard to trouble shoot on a forum.

Video streams use a lot more bandwidth at night if quality isn't constrained so you might be seeing bursts of data whenever cameras send their key frames. A Gigabit switch can only handle 10 Megabits per millisecond so there needs to be buffering. Assuming you're not exceeding the capacity of your switches, open the driver properties for your network cards and increase the number of receive buffers to the maximum. The default receive buffers on Intel cards is less than 6 Megabits and the max is around 24 Megabits which gives the Avigilon software 2.4 milliseconds to pick up worst case burst data, plenty of time.

Let's talk about your Fiber BackBone for a minute...
Are we simply connecting switches together using SFP Modules installed in the switches?
Is this older FDDI/OM1 62.5 Multi-Mode? It has physical speed limits? Only recently have we found reliable SFP that can drive MM @ 1Gb out to 2.1k. Older 1Gb MM modules created issues near 500 meters.. This would create the same result.... packet latency....

We can easily duplicate the problem on our wireless installations, and its the same symptom..

The Night Clue, is key,,, START-PERFMON click the GREEN PLUS and add the BYTES RECEIVED for Interfaces under NETWORK Interface.

The rated speed is 256mbits or 32mbytes.... We have pushed this to 35Mbytes for almost 8 months, with no noticeable effect on quality. (ignore the scale and simply look at the counters, last average, Multiply the value by 8 for bits)

The default gain on Micro Domes, and others will drive bandwidth way up. Any Indoor camera that is seeing near darkness at night is a culprit.

We use two NICS Bonded/team to a switch LAG for better backup performance, or failover protection from a NIC failure, or having cameras and production networks separated.

Else a single NIC is more that enough. I have seen many attempts at Bonded /teaming with a switch LAG that were done wrong. BroadCom Teaming

Check out you inbound BYTES/BITS per second below....

Perform Monitor

The Avigilon screenshot shows 70.2 Mbps incoming total, in a perfect world would this number be the same as the adapter statistics number?

Of course it's not a perfect world, so are you thinking that the difference between the two represents the packets not read out of the in the buffer in time by ACC?

If 70 represents the number in the Bytes/received for the given interface, "There's your problem." Whomever set this system up did not know the recommended limits of the Avigilon OEM series services. We've ran these at 35Mbytes/Sec for a long time, but 70Mbytes would certainly be a killer...as that is over 550Mbits, second. That could be impacting your network as well.

"This NVR comes in a standard rack mount enclosure and can record up to 32MB/s of image data from up to 128 cameras"

taken directly from spec sheets for the NVR servers. It's been this value for as long as I can remember...I would start cutting my frame rates back, and reducing the gain on all interior cameras.

...but 70Mbytes would certainly be a killer...as that is over 550Mbits, second

But it says 70Mbps. Small b = bit, no?

Yes it is 70Mbps

What does BYTES/RECEIVED per second for the Chosen Ethernet interface in Performance Monitor say?

I just did a screen capture of one of our servers... I've overlaid the performance monitor counters over a just ran server status report. Note how the status report shows 20.6Mbs . The performance monitor chart shows last sample of 2,508,034 Multiply that value time 8 to get BITS / SECOND... this equals 20,064,272 MBytes. The value in the Server Status Report is BYTES, not bits, ignore the case.... If Bytes / Rec in Perfmon is 70,xxx,xxx then your killing it..

Don't know where you're are, and if reasonable, happy to supply a much needed nTH server. :)

speed

No, you proved that it is labeled correctly

20 Million BITS = 2.5 Million BYTES

therefore

70 Million BITS = ~9 Million BYTES

Check your math

Michael Miller's right. You've got a bottleneck in your network somewhere.

Say one of the uplink ports is miss configured as 10/100 and in the day time you have 90Mbps of bandwidth. Night time rolls around and the bandwidth spikes to 110Mbps but your uplink can't handle it.

Check your Layer 1 first. Start from the server and work your way out to the cameras.

You said you have 6 server and 100+ cameras. So I would be shocked if your servers were anywhere near the 256Mbps limit. Check the switches as I have found those 1910s to be fickel. We use the 2920s for much larger systems than this without issue.

Won't the ACC logs show something as well? I would think that ACC knows whenever it has an incomplete frame. Though never having seen personally an overloaded ACC server I wouldn't know for sure...

Does the performance monitor values confirm your numbers? Unsure how Avigilon reports two un-teamed NICS. I guess you need to wait until dark and re-validate the numbers. Default gain is a cuprit. If all this confirms you're not exceeding the maximum input values, the revisit all previous group advice. Good luck... AT

The recommended Network interface performance counters are in BYTES. Avigilon Dell OEM R7xx R5xx servers with the Perc controllers and the default array are rated at 32MBYTES. if you performance counters are showing 70, then you've exceeded the ratings by a lot. the Avigilon Server status report is showing the rate in BITS.

Andrew the server report is Mbps.

Is TCP or UDP being used?

I must have drank to much mull cider... The report below is for a small system with 2 encoders and 2 other cameras. The incoming rate is 17.7Mbps.

report

Below are the performance counters (BYTES RECEIVED per Second) LAST=2,074,862 BYTEs per second X 8 BITS per packet equals 16,598,896 bits per second. Please include the term BITS pr Bytes in terms, this leaves on room for confusion. If the undisclosed person is seeing a value with 70,???,??? in the performance counters, this is BAD.....

fron counters

If the undisclosed person is seeing a value with 70,???,??? in the performance counters, this is BAD.....

He's not. He's seeing 70 Mbps in ACC.

The problem is on your transport... (By chance do both LAN interfaces on your server have gateways listed?) I find it strange to use the common gateway address of 192.168.11.1 as a usable IP on a NIC..... I bet the routing table looks a mess.... Does the LAN have an L3 switch?

load

Did I misunderstand you or didn't you say that the ACC recording limitation was 256Mbps?

ACC recording limit is not 256Mbps. Current Avigilon servers which this customer has is limited to 256Mbps. This is a hardware limit not a software limit.

Do you understand what is meant by the graphic in Andrew's post above?

192.168.x.254 is also a common gateway address.

UD2 - Of course other gateways exists, and ruling that out, or assuming anything is a poor scientific method. Better have IPVM reevaluate your service engineer credentials.

If I'm not mistaken, UE1 is having a problem that began sometime prior and has yet to be resolved. By now he/she has received several thousand dollars worth of free advise and a wealth of education from well meaning individuals.

Why did you post this above? What does it show?

Sample numbers from systems that have been in place for years. The numbers posted for the failing system are well under our typical servers, and further confirms they are dealing with a issue that is addressed in the many previous posts.

Thanks for clarifying.

To be clear then, the "much needed nTH server" isn't so much needed anymore.

I agree that it appears to be a network failure. Since we haven't heard from UE1 in a bit, I'm thinking he (hopefully) has determined the culprit and is in the process of remedying the situation.

This could be bandwidth or broadcast storm / spanning tree convergence. If you unplug something from the network (a camera or something) do the packets start appearing on the screens?

Undisclosed 1,

Has the problem been fixed?

Can you share any insights?

The video flickering were already fixed, they (SI) changed the default baudrate into half.

The joystick problem is till occurring.

We just found out the most critical problem:not synchronize time.

Do you think a gps time server would be a good idea?

We have a stand alone CCTV system (not connected to an internet)

Do you think a gps time server would be a good idea?

Yes, for a system of that size, absolutely. Veracity's Timeseal is one that comes up a lot, but there are others. See NTP / Network Time Guide For Video Surveillance.

The video flickering were already fixed, they (SI) changed the default baudrate into half.

Can you explain what you mean by this? Are you saying that the bitrate from each camera was reduced in half? What was the effect on image quality?

Although the flickering may have been solved by such a measure, it should be considered that you still have some network issue underneath, since the 70Mbps looked well within the spec.

Your server report is now showing 35mb vs 72 previously?
If they changed the max bit rate, you have a band-aid fix, the underlying problem remains..

Do you have more than 1 Avigilon Server?
Any Domain Server is a default SNTP Server.Accurate time on two Avigilon servers is required only when using fail-over or shared recording.

If they changed the max bit rate, you have a band-aid fix, the underlying problem remains...

I agree.

Interesting ethical dilemma though, since his SI is most likely saying 'problem solved' should we push the point?

Add the SI comments to the definition of "me·di·oc·ri·ty."