Subscriber Discussion

Cameras Dropping On One Switch Only - What Is Wrong?

UI
Undisclosed Integrator #1
Oct 08, 2016

Hi All,

Here is the situation: I have a 12-floor building with cameras on every floor. Every floor has around 8-10 cameras and one network switch, CAT6 is homerun from each floor's switch to the parkade.

Initially, the problematic floor had a PoE switches. I started getting cameras dropping on just one floor - all cameras would drop, but at different times. Sometimes I will have 2 or 3 cameras drop out, other times I will have just one, some times all 8. The cameras will drop multiple times per day and then not at all for a few days, then they will drop again.

I thought it might be the switch, so I changed from PoE to a standard switch and powered all 8 cameras - 12VDC for the indoor cameras and 24VAC for the two outdoor cameras. Funny enough, the issue persisted - I still have cameras dropping.

Some other notes:
- Live streaming the cameras on two or three different computers, only issue is on this one floor
- No IP address conflict, camera IPs are static
- If the cameras are down for more than 60 seconds, I receive an e-mail
- Drops and restores at random intervals - timing is never the same
- Firmware is latest and greatest (5.4.0)
- Hikvision cameras (2112's indoor, 4324's outdoors), lose them in SADP software as well as VMS
- All power is within voltage spec, cables are homerun to electrical closet
- Camera network is completely isolated

Does anyone have any idea how else I can troubleshoot this? Short of replacing all cameras, I am at somewhat of a loss.

Thanks.

 

(1)
U
Undisclosed #2
Oct 08, 2016
IPVMU Certified

Maybe the home-run to the parkade is flaky, maybe you can just run a temp cable down a floor to another switch to test?

(1)
(1)
Avatar
Vincent Tong
Oct 08, 2016

have you checked the wire run from that switch to the parkade? that seems to be where i would check first. You might have intermittent issues causing packet losses

(1)
UI
Undisclosed Integrator #1
Oct 08, 2016

Hi Vincent,

All runs have been checked and verified - I am wondering if the issue crept up with the building shifting and/or other people working in the building.

I believe I can use wireshark to check for packet losses, correct? I have literally never had problems until now.

Thanks for the reply!

(1)
MB
Matt Bischof
Oct 08, 2016

Now, I have a good idea of what's going on here, so humor me... and no, it has nothing to do with any current conspiracies ;-) ...

What exactly is going on in this 12-story building from the standpoint of business, industry, etc? If there are multiple businesses, a summary of businesses would be helpful.

Also, did you use the elevator shaft as your riser?

U
Undisclosed #2
Oct 08, 2016
IPVMU Certified

What exactly is going on in this 12-story building from the standpoint of business, industry, etc? If there are multiple businesses, a summary of businesses would be helpful.

And contact info as well, might be helpful ;)

(1)
MB
Matt Bischof
Oct 08, 2016

Funny guy.

Im looking for a source of EMI. Ran wire through an elevator shaft? DC lift motor throwing out massive amounts of EMI each time the elevator actuates...

Recent one I encountered, 10-story building, mink pelts being dried in a large tumbler on the 8th floor... every time it's operated, every IP camera in the building drops, while every camera on coax in the building is fine.

Integrator 1... have you used chokes (magnets) in this installation?

(1)
UI
Undisclosed Integrator #1
Oct 08, 2016

Hi Matt,

IPVM is always good for a chuckle in the morning!

We did not use the elevator shaft (building codes prevent it where we are). The facility is a long-term care center, we need to know if the residents are going into other rooms or if anyone is doing something they should not be :).

EMI is very possible, there is a lot of equipment that went in to this building.

I have not used chokes... did not even occur to me. Are all brands practically the same or is there a "better" brand?

(1)
(1)
U
Undisclosed #2
Oct 08, 2016
IPVMU Certified

Which floor is it? Nothing special about it? Are all the floor aggregating into the same conduit?

UI
Undisclosed Integrator #1
Oct 08, 2016

Floor 4, nothing special (same wander guard, wireless, etc. located on 4F and other floors), three conduits for home run to parkade, down the side of the stairwell (located behind drywall)

(1)
U
Undisclosed #2
Oct 08, 2016
IPVMU Certified

you said "humor me"...

(1)
(1)
MM
Michael Miller
Oct 08, 2016

Maybe the Chinese government is checking in on things :) JK

I would check the network drop that connects the problem floor with your Core switch. Do you have a network tester? Can you run a new cable? What switches are you using at the edge and core?

(1)
(3)
UI
Undisclosed Integrator #1
Oct 08, 2016

Hi Michael,

I prefer not to run a new cable as the floors are concrete and everything is fire cocked. Not to say that I am opposed - I want the issue sorted and, if a new cable is what it takes, than so be it. A new cable to the parkade is all but impossible, but one between the floors might only take 6 hours to run :S

Edge I am using a Netgear 16-port gigabit switch (notes left at the office) and the core switch is a 24-port Netgear gigabit switch.

I did check each and every cable and they all tested fine- I am wondering if it might be heavy EMI (as Matt stated above) or just a bad cable that tested fine one year ago.

(1)
MM
Michael Miller
Oct 08, 2016

I would start by re-punching down or terminating the ends of the cable from switch to core switch. I would also replace both patch cables if you using them.

(2)
RB
Rich Buchanan
Oct 08, 2016

We had issues with techs using an RJ45 connector and tool that allowed the wires to be pushed all of the way thru and then cut when crimped. Often there would be "stubbies" that caused intermittent connections but did not show up when tested and sometimes not for months.

(3)
MM
Michael Miller
Oct 08, 2016

We use a ton of EZ-RJ45 connectors. Key is to make sure your blade is sharp and adjusted properly.

(2)
RB
Rich Buchanan
Oct 08, 2016

Agreed. Secondly, techs that realize that and care that it is dull and take pride in their work. Sometimes you don't discover that until it's late in the game...

(2)
UI
Undisclosed Integrator #1
Oct 09, 2016

I will add re-terminating the cables to the list of things to try - thanks!

RB
Rich Buchanan
Oct 08, 2016

A few questions come to mind: Which floor is having issues? What is the distance from that switch to the parkade? When they are down can you connect to that camera with a browser or even ping it? Does a reboot resolve it? If not, what does? Time? Have you tried connecting with a laptop from the switch on that floor? Have you tried swapping that switch with one from another floor?

(1)
(1)
UI
Undisclosed Integrator #1
Oct 08, 2016

Hi Richard,

- Floor 4, not the shortest run and not the longest run.
- Cannot ping or connect when they are down
- Reboot does seem to resolve it, when I am there.
- Switch has been swapped with ones from other floors and finally replaced

When it comes to connecting with the laptop to the floor, I do not believe I have been on site when this was happening.

(1)
(3)
HL
Horace Lasell
Oct 08, 2016

Someone voted, "Disagree!" What do they know?

1) Floor 4 IS the shortest or longest run

2) You CAN ping or connect when they are down

3) Reboot DOES NOT seem to resolve it

4) Switch HAS NOT been swapped or replaced

or, you HAVE been on site when this was happening.

What else is there to disagree with? That voter must have AMAZING insight into your practices!

RB
Rich Buchanan
Oct 08, 2016

Sorry Horace

that might have been me and my fat fingers as I was replying from my phone and tried to hit informative and may have hit disagree. I didn't think it registered as I did not notice the usual confirmation but couldn't see a way to undo it. I did hit informative. Or maybe it wasn't me?

HL
Horace Lasell
Oct 08, 2016

No worries -- it gave us all a humorous moment. Just FYI, clicking a voting button again toggles its state.

(1)
RB
Rich Buchanan
Oct 08, 2016

Thanks, didn't know that. I'm obviously a newbie to IPVM!

RB
Rich Buchanan
Oct 08, 2016

Which VMS? How many go down simultaneously?

RB
Rich Buchanan
Oct 08, 2016

I had a similar scenario twice using HIK + Avigilon. It was always one camera plus a different camera. I replaced the one camera and it resolved the issue. The next time it happened on a different job as I was connecting the cameras to the VMS in the initial setup. The one camera would not coexist with some of the others. Could not see any obvious reason for it. I defaulted the camera and started over and it resolved it. Sometimes products are not properly defaulted after bench testing from the manufacturer.

(1)
UI
Undisclosed Integrator #1
Oct 08, 2016

Hi Richard,

The old Luxriot VMS (budget VMS for 100+ cameras, it works) is what we are using. I will absolutely try to default the cameras on this floor and see what happens.

Thanks!

(1)
RB
Rich Buchanan
Oct 08, 2016

Sorry, not familiar with the VMS.

One last thought. I wonder if the switch is struggling.

What are the camera resolution and record rate settings?

Is there any relationship to what is being viewed by the two workstations when the cameras go down?

UI
Undisclosed Integrator #1
Oct 09, 2016

No worries, as I said it tends to be considered a budget VMS and not widely talked about. Their new iteration seems to be pretty good, but everyone has their favorites :).

Cameras are 1.3MP, 20fps. I do not believe the switch is struggling, but it would not be the first time a defective product was replaced with a defective product. The switches were both new in box from the distributor, but that doesn't mean they can't be defective.

MM
Michael Miller
Oct 08, 2016

Richard... did you call Avigilon support about this? We had a similar issue where 2 Hikvision cameras had the same unique ID and could not be installed on the same network. They worked fine on different systems but would not work when connected to the same server.

(1)
RB
Rich Buchanan
Oct 08, 2016

Hi Michael

Yes, I called Avigilon support. I think they have arguably one of the best support teams in the business. Most of the time they are quite helpful. Occasionally you get a tech who goes down the wrong path as he is convinced about a hunch. Early releases of 5.8 were a little ONVIF buggy. They were baffled and wanted bug reports and packet captures as there were issues with IP addresses and camera titling if you rebooted the camera. Later versions resolved it. That's why I wondered which VMS. Avigilon eventually put out a tech bulletin with a protocol for connecting HIK cameras saying to stop and restart the service to add a HIK camera. That helped but did not resolve all issues. This was also when HIK had just upgraded their firmware. I had 30 cameras on this job and had four different default IP addresses in the bunch. Two were neither of their standard defaults. I joke but there's some truth to today's way of resolving issues: reboot, firmware uografe, default, replace...

i replaced the camera in that one scenario without trying to default it. It was when I found four different default IP addresses that I concluded they were reset properly after bench testing. That can happen with any product from any manufacturer. Flash back to Radionics in the lates 80's early 90's but seen it several times since. No product is better than the people making it. There will always be issues, the difference from one manufacturer to the next is how they handled them. Avigilon does a pretty darn good job if it most of the time. I just hate that they penalize me with license fees simply for selling more cameras, but love the end user functions of their VMS. Their dynamic timeline always makes the customer say WOW!

(1)
MM
Michael Miller
Oct 08, 2016

I just hate that they penalize me with license fees simply for selling more cameras

You hate that Avigilon charges a fee for their product :(

(1)
RB
Rich Buchanan
Oct 08, 2016

Well, no, I can understand that, it's quite good, but when I need a 25th camera on a server I'm forced to use more expensive and feature rich licensing when I don't need the features, just another camera added. That is not always an easy sell to the end user. My focus is on the end user experience and providing a solution to their pain or needs. At the end of the day, it's all about relieving their anxiety.

(1)
(1)
JH
John Honovich
Oct 08, 2016
IPVM

We had a similar issue where 2 Hikvision cameras had the same unique ID

Michael, how 2 Hikvision cameras have the same unique ID? And by unique ID, what ID specifically is that?

(1)
RB
Rich Buchanan
Oct 08, 2016

John, do you play the drums? You don't miss a beat! 😉

MM
Michael Miller
Oct 08, 2016

I am not sure how but it did happen. When you plugged in camera A everything worked fine. When you plugged camera B in while camera A was plugged in both cameras dropped offline and would not stay connected. When you unplugged camera A then camera be would work. All cameras on this system had static IPs and checked multiple times for IP address conflicts which was confirmed not the issue. We swapped both cameras out and tested at our office and had the same issue. This time, the cameras were setup using DHCP.

This was over 2 years ago so I don't have much more info since we didn't track our service tickets like we do now.

MS
Mark Strange
Oct 08, 2016
IPVMU Certified

Swap cameras one at a time with one on another floor. Does the camera continue to drop when moved? Does a well-behaved camera moved to the 12th floor begin to misbehave? When a camera is moved to another floor, do other cameras on that floor start to misbehave? If a 12th floor camera encourages other well behaved cameras on another floor to misbehave, maybe Michael Miller is correct and this is Chinese malware.

Are all cameras on the same subnet? If so, re read the above after doing a mental search/replace "another floor" with "another subnet".

If all else fails, well, maybe this is really the 13th floor, not the 12th.

(2)
MB
Matt Bischof
Oct 08, 2016

I stand by my EMI theory, especially given the fact that you said there's a lot of equipment in the building. As far as choke "brands", can't help you there-- a ring of ferrite is a ring of ferrite. Here's a 10-pack on the Amazon for $6.99 that might tickle the gentleman's fancy...

https://www.amazon.com/dp/B015E8DHC0/ref=cm_sw_r_cp_api_OOr-xb5E9MZY6

(2)
HL
Horace Lasell
Oct 08, 2016

You've probably already run through the basic thinking...

What has changed?

It's not the switch, because you replaced that,

It's not power to each camera, because you changed that.

What is the same?

Cameras

IPs

Ethernet from Cameras to Switch

Switch Power

Ethernet from Closet Switch to Central Switch

Of those elements that are the same, what is unlikley to be the problem?

Since all cameras seem similarly affected, it seems extremely unlikely that all cameras on the floor developed the same kind of fault at the same time.

Conclusion: the camera hardware is probably OK

Conclusion: the ethernet from each camera to the local switch is probably OK

You said that "All power is within voltage spec." I take this to imply the switch power (otherwise a prime suspect) has been verified to be OK.

That leaves Ethernet from Closet Switch to Central Switch. Do you have a way to certify this drop?

If so, that leaves IPs.

Could a DHCP range (somewhere on your network) have changed to collide with this floor's static IPs?

Maybe someone's playing Pokemon online through a handy ethernet drop, and hitting your camera IPs? (but, unlikely since you indicate it can take out multiple cameras at the same time, which would require multiple IP conflicts).

Do your static IP ranges segment by floor? e.g. 192.168.1.001 to 010=floor 1, 011-020=floor 2, etc? This would make it more likely that someone who has established a colliding DHCP range could affect only one floor. If, on the other hand, your camera IPs are all over the place, with any floor's IP assignments interleaved with any other floor, then this theory would be unlikely because if someone was hitting a range, they would more likely affect a variety of floors.

If you're on-site, then when you lose a camera, take it offline and then ping its IP, to discover who just collided with your gear's IP (because if you get a ping response, it won't be the camera you just took ofline, right?) .

Just running through this thinking, I'm leaning toward IP conflict.

Also, I forgot to ask if each floor's network is separate. That would make some of my comments (about assigned IPs per floor) irrelevant.

I'm sure I'm missing plenty of options. I'm an end user, not an installer. I'd love to be better educated. Anyone else see flaws or missing elements in this meandering troubleshooting chain?

Hope this helps.

(1)
HL
Horace Lasell
Oct 08, 2016

When I started this rambling post, there weren't any replys, but by the time I've posted, you've received enough professional responses that my post isn't really adding much value :)

IPVM sure is a great resource!

(1)
UI
Undisclosed Integrator #1
Oct 09, 2016

Hi Horace,

Sometimes integrators "know it all" and do not think about the simple stuff. I appreciate the post!

(1)
UI
Undisclosed Integrator #3
Oct 08, 2016

Are the switches managed in any way?

Avatar
Robert Baxter
Oct 08, 2016

Have you checked your cables with a tester designed for this purpose? The intermittent nature suggests to me problems with your cables. The following handbook might be of help.

Fluke Test Instruments

(2)
UI
Undisclosed Integrator #1
Oct 09, 2016

Hi Robert,

All cables tested with a Fluke certifier. Who knows what has happened in the last year with construction and building shifting, I will certainly be checking again.

Thanks for the PDF, might come in handy!

UI
Undisclosed Integrator #4
Oct 08, 2016

Are the cameras still powered when they go offline? Could be possible you have a broadcast storm.

CC
Christopher Cincochan
Oct 08, 2016

try checking the core switch if there are any network chokes happening. this is under the assumption, that you have thouroughly checked all possibilities of cable issues and termination issues. we had a similar problem before with a client. we finally found the root cause of the problem, by using a high end network tester that discovered that there are two defective ip devices that are intermitently flooding the network. eventually have to replace the two defective hardware, problem solved after that.

Avatar
Armando Perez
Oct 09, 2016
Hoosier Security and Security Owners Group • IPVMU Certified

I'm on mobile, so it's possible I missed replies. If I'm redundant, then please take that as additional confirmation...

I would look at the wire first. Terminations and the wire itself. Can you drop a temp wire to another switch?

Have you calculated bandwidth? Are these switches choking up?

Chances are the cameras are fine.

UI
Undisclosed Integrator #1
Oct 09, 2016

Alright friends - thanks to all for posting on a Saturday. My long absence was due to family commitments (Thanksgiving weekend in Canada).

Items I will try in the coming week (or two, depending when I can get back on site):

1) Default all cameras
2) Re-terminate the cables at both ends (just in case) and test them well
3) Add ferrite cores to each end of the CAT6
4) Run a new cable to the floor above or below (PIA, but anything to ensure the system is 100%)
5) If all else fails, replace the switch again
6) If none of the above works, use a tool like wireshark to see what the heck is going on. I may still choose to do this first, we'll see.

If I don't forget, I will edit or do a big reply to let everyone know how it goes. Thanks again for all the suggestions - have a great weekend!

(1)
U
Undisclosed #5
Oct 10, 2016

Try reducing the camera bandwidth settings on all cameras on that switch. It usually solves network issues until you can upgrade to fiber trunks and Cisco switches.

(3)
UI
Undisclosed Integrator #1
Oct 10, 2016

While I appreciate the reply, not all clients/companies have the budgets to transition to fibre trunks in a finished building and all Cisco equipment.

U
Undisclosed #2
Oct 10, 2016
IPVMU Certified

Yes but Cisco switches are well known to work on ALL floors, not just 11 out of 12, like some other manufacturer's gear. ;)

(1)
(1)
UI
Undisclosed Integrator #1
Oct 10, 2016

Alright, I may have solved the biggest portion of the problem - the home-run cable was damaged somehow. The connection to the server room would go on for about a second then off for many seconds (up to around 45 seconds). I tried re-terminating both ends, but nothing helped. I ended up running a cable to the floor above and things have been steady for the last hour.

I also defaulted all cameras to see if that was the issue and it did not seem to help.

Thanks for everyone's feedback - I will update again if the issue repeats itself.

(1)
Avatar
Armando Perez
Oct 10, 2016
Hoosier Security and Security Owners Group • IPVMU Certified

Do I get a cookie? :)

(1)
U
Undisclosed #2
Oct 10, 2016
IPVMU Certified

Do I get a cookie? :)

No, but you can have some of mine. :)

(1)
UI
Undisclosed Integrator #1
Oct 10, 2016

Next time we're both at ISC East or West (maybe 2017), I'll buy you a drink :)

(1)
Avatar
Armando Perez
Oct 11, 2016
Hoosier Security and Security Owners Group • IPVMU Certified

Hell I'm in even if I have to buy my own!

DO
David O'Neil
Oct 10, 2016

I just got back from troubleshooting a similar issue on a site that had used an Exaq Vision NVR and Bosch 5M cameras.

The main issue was similar to yours, intermittant camera drops, playback and recording drops (even at unrelated intervals to the live feed drop).

Upon further investigation, cabling and power checked out, we would find that a camera reboot would bring the camera back online.

Next we reviewed playback and noticed the problem cameras would ramp up or down brightness until they went offline.

After checking Exaqs list of supported cameras I found that the original camera integrator used cameras not officially on Exaqs support page. The integrator assumed ONVIF type s meant the two products should pair without trouble.

Well much testing with firmware up and downgrades on both exaq and bosch, and messing around with camera resolution and bandwidth settings it was found that there was an issue with how the camera was handling or not handling the day/night part of the stream..

The problem persisted though much of the changes we made and would transition between a select group of cameras.

Finally recently there was another firmware update for the bosch cameras as well as the Exaq box... so we went back and redefaulted, updated firmware and re added cameras back to Exaq, this time Exaq's camera finder found some ONVIF cameras but quickly discarded the ONVIF for I believe the UPnP stream and the cameras were not able to be added. When we went to the bosch camera network setting and turned off UPnP, and refreshed the Exaq camera finder , the cameras now were all visable to add as ONVIF cameras.

While programming it seemed that the camera quality and speed had improved and were more responsive through the variety of menus on Exaq.

At this point it seems out problem with cameras dropping out has been resolved.

Could this have been found sooner with a different approach to troubleshooting, perhaps like others have said a wireshark capture on the network may have revealed a packet flood of some kind due to onvif protocol issues in firmware which was causing the packet missmatch and eventual drop? Note our cameras stayed powered when the feed had dropped.

We are waiting and watching to see if the latest firmware updates have resolved our problem.

Sometimes there is only so much we can do ourselves without requiring a firmware patch or upgrade.

Also , properly disassociating the cameras from the VMS services and programming, prior to defaulting , doing any updates may be a useful troubleshooting step, then re add the cameras only after rebooting and restarting the VMS.

Let us know if the problem ever got resolved as well as how you resolved it as we can always learn from others troubles.

Dave.

(3)
UM
Undisclosed Manufacturer #6
Oct 10, 2016

The integrator assumed ONVIF type s meant the two products should pair without trouble.

That's the dream, right?

(1)
New discussion

Ask questions and get answers to your physical security questions from IPVM team members and fellow subscribers.

Newest discussions