Have you talked with their tech support? Their tech support is good and answers anyone's questions so it would be helpful to start with them. If the problem still exists, share feedback from what they suggested and then we can provide better advice.
is the flickering video on all cameras ? - then it is most likely network based
is it just a few coloured lines through the image like a tear ? - bandwidth from camera to NVR not enough
or when it is flicking does the whole image go black ? - ive had this issue when i had the switches in a stack
Flickering of Video display (even on recorded video).
So there is flickering when playing back recorded video even when on the viewed directly on the VMS server machine?
Or just the client? Does it flicker if you export a clip? If so can you export a clip and post a link to it?
What kind of monitor/video cards do you have?
Check the line frequency setting - 50hz vs 60hz vs off. Many cameras have an anti-flicker or flickerless setting in the camera image adjustment for this...
If it is on both live and recorded video it is most likely packet loss from the cameras to the server.
Your integrator should help you with this. However, you can call Avigilon technical support directly at 1.888.281.5182 - they can remote into your system and help you.
We have dealt with this issue before it has nothing to do with 50hz or 60hz setting in the cameras.
If this issue is happening on both live and recorded video on multiple machines it is a bandwidth issue from your cameras to the server. What is your total throughput on your servers? Are your servers connected to the switch via Gig ports? Are your uplink ports between your switches connected via Gig ports?
Contact your integrator first as they should be able to provide you with a network map and confirm these questions. Then contact Avigilon support.
If all else fails I would be more then happy you help you out off line.
We've seen this issue, and in 100% of all cases it was transport related. That includes updates applied to a server, that resulted in the LAN drivers having an issue. I can't find some screen captures we've taken in the past, but the symptom is more a horizontal tearing that is recorded in the video. With every new server, or a server we are responsible for, and at any chance we are getting the latest LAN drivers from BC or Intel.
We've also seen where a client had two NICS, each with an IP and gateway on the same subnet...
I would start by looking at the Totalbytes received using performance monitor. We've ran the Dell OEM 7xx and 5xx servers up to 35Mbytes/sec for almost a year before adding another server.
Never let a user view video on the server console monitor.
LAN drivers are equally important on a multi-monitor viewer (quad 30' in @ 2550 x 1600) for example... Older XP machines (disable hardware excel) (viewing only issues of course)
To better isolate the issue, crank up a handful of cameras to the best settings and highest frame rates..
We've tested this with 6 7K cameras, all set to MAX, plus many more... running a 1G segment up to near 50%, and over 500mbits into a server... after that we started having record issues... Duh...
Avigilon has decent disk bandwidth tool that will saturate . test the disk subsystem. You might have a failing disk or under performing disk.... Tech support can collect the data....
Happy to comment privately, or share some tools...
Mostly it occurs only at night
This could be what is known as "Black blocking"
I had this issue several years ago and stacking the switches was the cause.
I still see this whenever a new camera is connected to the network but dies down once the camera has been added.
Create new views of the cameras that are connected to each switch, and also each server. look at the views and see which one is the worst one and it could point you to the switch or nvr that has the issue.
We know what Anixter would say...;)
How can i contact you guys? Thanks
Im using a mobile phone, icant.see it
You may have tried to send me a PM, but McAfee email blocked it for a content violation....
Let make assumption - all of the displays are on a local workstation displaying video from one or more servers. - Correct?
If you go back and watch the same footage, the errors are recorded? - Right.
Is the frequency of these events consistent throughout a 24 hour period?
Do you know how to get the received bytes using performance monitor?
Did someone add an additional INTEL dual port NIC card? There are several SERVER models from INTEL (they acknowledge) were bad on PCI slots we've discarded for this and other problems.
Assuming these events are randomly consistent, and across all switches and cameras, I am hair triggered to make sure you are using the latest release drivers direct from Intel... I can confirm what we are using on the Dell OEM7xx and OEM5xx HD servers.....
One more thing to check is the make/model of the switches and the age of their firmware, and the current UPTIME... We've seen switches,, CISCO, NETGEAR, and others that become flacky after a a year or so of uptime..... STG grapher by leonid Mikhailov is a great tool to have to plot Switch CPU,RUNTS,Collisions and speed too..... A fresh reboot on the switches is in order too.
Let's talk about your Fiber BackBone for a minute...
Are we simply connecting switches together using SFP Modules installed in the switches?
Is this older FDDI/OM1 62.5 Multi-Mode? It has physical speed limits? Only recently have we found reliable SFP that can drive MM @ 1Gb out to 2.1k. Older 1Gb MM modules created issues near 500 meters.. This would create the same result.... packet latency....
We can easily duplicate the problem on our wireless installations, and its the same symptom..
The Night Clue, is key,,, START-PERFMON click the GREEN PLUS and add the BYTES RECEIVED for Interfaces under NETWORK Interface.
The rated speed is 256mbits or 32mbytes.... We have pushed this to 35Mbytes for almost 8 months, with no noticeable effect on quality. (ignore the scale and simply look at the counters, last average, Multiply the value by 8 for bits)
The default gain on Micro Domes, and others will drive bandwidth way up. Any Indoor camera that is seeing near darkness at night is a culprit.
We use two NICS Bonded/team to a switch LAG for better backup performance, or failover protection from a NIC failure, or having cameras and production networks separated.
Else a single NIC is more that enough. I have seen many attempts at Bonded /teaming with a switch LAG that were done wrong. BroadCom Teaming
Check out you inbound BYTES/BITS per second below....
If 70 represents the number in the Bytes/received for the given interface, "There's your problem." Whomever set this system up did not know the recommended limits of the Avigilon OEM series services. We've ran these at 35Mbytes/Sec for a long time, but 70Mbytes would certainly be a killer...as that is over 550Mbits, second. That could be impacting your network as well.
"This NVR comes in a standard rack mount enclosure and can record up to 32MB/s of image data from up to 128 cameras"
taken directly from spec sheets for the NVR servers. It's been this value for as long as I can remember...I would start cutting my frame rates back, and reducing the gain on all interior cameras.
What does BYTES/RECEIVED per second for the Chosen Ethernet interface in Performance Monitor say?
I just did a screen capture of one of our servers... I've overlaid the performance monitor counters over a just ran server status report. Note how the status report shows 20.6Mbs . The performance monitor chart shows last sample of 2,508,034 Multiply that value time 8 to get BITS / SECOND... this equals 20,064,272 MBytes. The value in the Server Status Report is BYTES, not bits, ignore the case.... If Bytes / Rec in Perfmon is 70,xxx,xxx then your killing it..
Don't know where you're are, and if reasonable, happy to supply a much needed nTH server. :)
Michael Miller's right. You've got a bottleneck in your network somewhere.
Does the performance monitor values confirm your numbers? Unsure how Avigilon reports two un-teamed NICS. I guess you need to wait until dark and re-validate the numbers. Default gain is a cuprit. If all this confirms you're not exceeding the maximum input values, the revisit all previous group advice. Good luck... AT
The recommended Network interface performance counters are in BYTES. Avigilon Dell OEM R7xx R5xx servers with the Perc controllers and the default array are rated at 32MBYTES. if you performance counters are showing 70, then you've exceeded the ratings by a lot. the Avigilon Server status report is showing the rate in BITS.
Is TCP or UDP being used?
I must have drank to much mull cider... The report below is for a small system with 2 encoders and 2 other cameras. The incoming rate is 17.7Mbps.
Below are the performance counters (BYTES RECEIVED per Second) LAST=2,074,862 BYTEs per second X 8 BITS per packet equals 16,598,896 bits per second. Please include the term BITS pr Bytes in terms, this leaves on room for confusion. If the undisclosed person is seeing a value with 70,???,??? in the performance counters, this is BAD.....
The problem is on your transport... (By chance do both LAN interfaces on your server have gateways listed?) I find it strange to use the common gateway address of 192.168.11.1 as a usable IP on a NIC..... I bet the routing table looks a mess.... Does the LAN have an L3 switch?
This could be bandwidth or broadcast storm / spanning tree convergence. If you unplug something from the network (a camera or something) do the packets start appearing on the screens?
Has the problem been fixed?
Can you share any insights?
Your server report is now showing 35mb vs 72 previously?
If they changed the max bit rate, you have a band-aid fix, the underlying problem remains..
Do you have more than 1 Avigilon Server?
Any Domain Server is a default SNTP Server.Accurate time on two Avigilon servers is required only when using fail-over or shared recording.