Avigilon System: Cameras Not Reconnecting To Server After Disconnect

I had a new Avigilon system installed that has Avigilon cameras, analog cameras connected to an Avigilon digital converter and 3rd party IP cameras connected to it. Whenever a camera (any camera) loses connection to the server (such as power interruption, network drops, camera reboots) the camera cycles between Error and Not Present. I have to go in and disconnect the camera in the software and then reconnect it to make it show up again.

Has anyone ever run into that problem? Avigilon tech support says they've never seen this happen.


What version of ACC software ?

Post NVR hardware specification .

What is the max distance of cable runs ?

Does this happen with the encoders too or just the cameras?

For the record I have never seen this issue.

Yes it happens with the encoders as well.

I've seen this (or something like it) with some Dahua cams on Vigil NVRs. Cycle camera power too quickly, sometimes Vigil can't grab their streams again. Power down for a couple of minutes, they come back fine. It's happened mainly with the little UFO mini-domes, and it's not consistently repeatable. Newer firmware versions seem to have at least reduced the problem.

Matt - yes that was one of the theories of the Avigilon tech support too but it happens with short as well as long interruptions. We disconneted a camera for 5 minutes in a test and it still did not come back up. And we have upgraded to the latest firmware though it's hard to keep up as they come out with a new one every month.

Claudia - What version of ACC are you using? What is you network layout? We have thousands of Avigilon cameras in the field on both very large and small systems and haven't seen this issue. If you like please email me mike@thewireguys.tv I can provide more insight.

I've seen this before in a system with non Avigilon cameras. In my case was a problem with network NIC from the server in one case and a problem with grounding of CAT5e cables in another case. Also can be due of faulty patch cable between NVR/encoders and switches. In all the cases was something from network.

Are the cameras static or dhcp, are you using dual nics on the server/workstation?

Cameras are all static and we are only using one NIC, other one is inactive

Claudia,

Did your issue get resolved?

You appear to the be an end-user member of IPVM, and that's great, but as an integrator, if a client ever felt the need to post a support question here, it's time to retire. Without naming names, did the integrator abandon you, or was this system a DIY project?

Is there such a thing as Avigilon DIY?

Yes

I mean where they start out as DIY not end up as.

Yes we have many DIY Avigion customers both small and enterprise.

The problem has not been resolved and the integrator has been more than willing to help, they just don't know how to fix it. They've spent probably 40 hours or more on calls with Avigilon. Avigilon is saying they have never encountered this problem before and I was just curious if that is really the case or if it's one of those where it is a known problem that they just don't know how to fix. I am quite impressed with Avigilon tech support though I have to say, especially with their willingness to keep looking at it. Other companies I've dealt with they pretty much leave you hanging if they can't find the problem in one or two calls.

By the way, I'm not your typical end user. My integrator calls it proactive...

"By the way, I'm not your typical end user. My integrator calls it proactive..."

:) Claudia, thanks for the update! Hopefully a solution will arise soon.

Let me make some assumptions, correct where necessary

1. All cameras and other devices are on a flat /c LAN or CIDR range.
2. We are using L2 managed switch/POE with mostly default settings.
Is this an Avigilon Server? (RACK or TOWER)?
In almost every case of the (TOWER) servers, we've experienced connectivity issues on links with greater than 5ms responses. (We use lots of wireless) In every case, the issues were the Intel NIC drivers, and resolved with the latest Generic Intel Driver Pack from Intel Downloads.
V 19.5.303.0 Intel Network connections drivers..

As a test - Assign an additional IP address 169.254.200.200 /16 to the NIC serving the cameras. (This is an apipa address) then assign an apipa address of 169.254.100.100/16 to a camera. (I would want to know if this camera on a different IP, not using DHCP experiences the same issue) Perhaps a duplicate IP may be randomly occuring.. Check DHCP logs.

Also, disabling Windows FILE/PRINT client in the NIC
Disable IPv6
re-order the providers on the NIC (viewing the local area connection properties press ALT)
advanced settings move IPV4 to the top.

Can you make the problem occur is the cats meow to determine what is happening.
If you can, then m theory of IP conflict isn't valid.

If you can unplug a camera, and it fails to reconnect, then the answer lies elsewhere.

Are you experienced with WireShark?
Can you set a mirror port on the switch and capture all traffic from a known camera that fails?
TX/RV

1. We have multiple VLANs and the camera system is on it's own separate VLAN. All the cameras and/or encoders have static IP addresses. It's definitely not a duplicate IP address, because we did that by accident during installation and at that point neither camera would connect at all.

2. yes it's a L2 managed switch with Avigilon provided OS. It's rack mounted.

Upgrading the NIC driver sounded like it was worth a shot so we tried that but all it did was disconnect every single camera and the problem is still the same. IPv6 was already disabled.

And yes we have wire shark and Avigilon took gigabytes worth of recordings during the multiple tests we have done so far (short disconnect, long disconnect, different IP range, firewall and antivirus completely turned off, with camara connected to power injector...I forget the rest but there were multiple scenarios).

Avigilon's latest theory is that something in our internal group policies is preventing the reconnect so we sent them a copy of all policies. Haven't heard back yet.

Thanks for the suggestions!

More assumptions, and correct where necessary.

You identified the switch as rack mounted, but is this a 100% Avigilon branded Dell OEM Rack Mount Server Model # beginning with storage size and ending in HD-NVR2?

If so, these systems use W7embedded, and Avgilon maintains the latest tested drivers, firmware, etc, all that apply via a bootable USB onto the Dell Lifecycle Controller.

Or is this a Tower OEM Avigilon NVR.

You have joined these servers to you AD?
Are they is a seperate OU - completely isolated from GP's

Do you use a network management system? IE, something that is pinging devices, or using sythentic ICMP packets to determine health?

Earlier you said power loss, how is this being allowed to happen? Bad or NO UPS's on switches or servers? (Not a healthy things)

regardless of all answers, assuming the physical layer is sound, then moving up to to L2, check out ARP packets...

https://ask.wireshark.org/questions/7277/arp-broadcast

I would suggest you create a trace of a camera that correctly reconnects, on 1 or more VLANs, then run a real-time filter capturing future arp traffic.

When you plug the cameras on to your VLAN'd network do the cameras/encoders get auto-discovered in ACC?

I would recommend you disconnect the server from your network and connect one camera using the same nic you use to connect camera network. Use a POE injector or extra switch you have laying around. Once you get the single camera connected to the server disconnect the power to the POE device and then reconnect it and see if the camera reconnects to ACC. This test will remove your network and VLANs from the equation and will tell you which direction to look.

I don't remember if we did this scenario, but we did the opposite. We disconnected the server and the integrator set up his laptop to act as the server, with same IP address and on our network. We then connected one camera from each of the different switches we use. In that scenario the reconnection of the cameras worked just fine. So it is definitely something in the server that's causing the problem. The server came from Avigilon but we did join it to our domain and apply group policies to it.

And yes ACC does auto discover the cameras in the connect/disconnect section, I just have to manually add them to the site from there. They also automatically reconnect if I shut down and then restart the services in the ACC admin tool. So it's not hard to do but the problem is when the disconnect happens at night or during the weekend I lose video for an extended period of time.

Your assumption was GP's were the issue, did you unjoin the system from the domain?

Do a series of GPUPDATES to see if this triggers the issue from the NVR console?

Is this a RACK mount Server?

Did Avigilon Send you the Files to make a bootable USB to upgrade everything in the system?

It also wasn't clear, but you have multiple switches, switches with VLANS, do you by chance have these configured in an RSTP ring? if so, can you break the failover link?

Do you have VoIP phones on the switches?

Capturing and seeing the ARP packets on startup of any device is really important?

Also what is the MAC table timeout values on the switch? 5 minutes or less? I would add time to that?

Sure - search Avigilon on Ebay or for that matter, and I never understood how this happened, but check out www.clearpix.com. Does anyone see the resemblence?

I suspect most Avigilon Dealers would sell software license to a good client. Knowing they may be buying Non Avigilon cameras.

Or if you need exactVision, checkout http://inaxsys.com/vms-softwares-s26/ and see another resemblence, and available through distribution.

The resemblance is rather striking, isn't it?

When we first became Avigilon dealers at my old job, I remember them offering private labeling services. Do they still mention that to integrators at all?

Hmmm could you try setting up a different NVR and see if it experiences the same issues? Maybe a free NVR software or a low-cost standalone? Something like Dahua's PSS software should be able to find and add any ONVIF-compliant cameras... you could even install copies on both the Avigilon server and a separate server to narrow down anything specific to that machine.

Claudia - What's the status?

As an FYI, we have a similar problem with some of our non-Avigilon cameras not reconnecting to one of our NVRs (Avigilon ACC 5) after a power interruption to the cameras. Cycling the power to the cameras again has always resolved the issue. This usually only happens in one location (building) and those installations are different from most of our others.

In this building we are using external power supplies at the camera, and the power supply is powering both the camera and an IFS fiber-to-Ethernet media converter. Fiber runs back to a telecom room and connects to an IFS fiber switch, which is in turn connected to the greater network via fiber.

The NVRs and the switch are on UPSs. The error message displayed in ACC 5 varies between "not connected" or "not present". We have noted that it is not always the same camera(s) that do not recover after a power outage (all cameras are on one circuit in this location), but in every instance we can cycle the power again and all will regain connectivity.

I have seen a similar instance in one other location where power to a PoE switch was inadvertently interrupted. Some of those non-Avigilon cameras did not automatically reconnect when power was restored, but in that instance I was able to stop and restart the Avigilon server service on the NVR and the cameras reconnected.

We have not talked with Avigilon about the problem, but I thought it might be helpful to know others are experiencing a similar problem to that of the original poster.

I didn't think it was still going to happen but it seems we have solved the issue. In what must have been the 10th remote session with Avigilon the tech had an Eureka moment and checked the Rules that come as default from Avigilon.

In those rules under Device Actions the option Reboot Camera was selected. What that does apparently is send a reboot command from the server to the camera every time the camera loses connection. In our case, a momentary loss of connection set off a cycle where the server sends the reboot signal, the camera reboots and with that loses connection again, which triggered another reboot command and so on. Once the Reboot camera option was deselected the cameras reconnected without issue. I have no idea why that option is even there...

Avigilon - Default Rules

The Avigilon tech stated that this option is not supposed to be selected and he doesn't know why it was in our case. It's not anything we or our integrator adjusted, as it is in the default rules.

I have to say though that through all this I am impressed with the Avigilon support and our Integrator. I have had issues with other software products and after the 3rd call (not even a remote session) the answer usually is the problem must be on your side, sorry about your luck. Avigilon and our integrator were always willing to keep looking and working with us.

Very interesting feedback, thanks for sharing and I am glad this was resolved!