Subscriber Discussion
Why Are My Cameras Randomly Falling Offline?
Hello
I am having an ongoing technical issue with an installation that I did a few months back. The system consists of a Digital Watchdog NVR (Blackjack Blade - Spectrum) with eight Digital Watchdog DWC-MV421D 2.1 MP outdoor cameras. I am using two Trendnet eight port gigabit POE switches. I'm using outdoor CAT5E cable to all of the cameras with the exception of two which have standard CAT5E. The system is installed in a small commercial environment and is connected to their network for internet access and local viewing. All of the cameras have been assigned static IP addresses that I was given by the IT people that manage their network.
Periodically (every three to seven days) one of the cameras falls off line. It is not always the same camera and it is not always a camera on the same switch. The camera that falls off line can be pinged but the you can not access the camera's web browser. The only way that I can put the camera back on line is to power cycle it by shutting down the POE switch or unplugging the camera's ethernet cable.
I have spoken with Digital Watchdog technical support people on several occasions and most recently they suggested to me that it may be a power issue so I replaced the switches. I also connected one of the cameras to an Axis midspan POE injector but even that camera at one point went down as well. The camera cable lengths vary between 50 to 250 feet but cameras on short cable runs will go down as well.
The most recent episode happened yesterday (after a week of having no issues) and in the past 24 hours it has happened twice.
Any help would be greatly appreciated!
Pinging can work while web browsing is not, as these are two different processes in side the camera. I'd suspect a firmware or an integration issue with the DVR if everything else is alright. Are the cameras using DVR manufacturer's recommnded/tested firmware versions.
Also, this sort of behaviour can happen due to misconfigured IP/gateway/mask settings. Make sure there are no IP address conflicts, dbl check subnet mask is correct and gateway is correct if any. Is any other devices on the network other than cameras?
Thanks for the reply. These cameras initially were running on DHCP and I was having IP conflicts with other devices on the network. At this point I had system administrator assign me static IP addresses outside of their DHCP range. This solved the conflict problems but the issue that I described above has continued. The camera's firmware is up to date.
"Periodically (every three to seven days) one of the cameras falls off line."
Can you elaborate on this? How do you know it went off line? Gaps in recording? Or some form of alert?
"The only way that I can put the camera back on line is to power cycle it by shutting down the POE switch or unplugging the camera's ethernet cable."
When the camera is offline, are you able to reach the camera's web interface and see video? Or is that offline / not accessible as well?
Yes. It was a thought of mine and DWD technical support. There are two eight port gigabit POE switches with only four cameras on each. The camera's power consumption is about half of the manufacturer's per port rating. There are not even any IRs on these cameras. I even put an Axis midspan (T8120) POE power supply on one of the cameras in place of the switch POE power. At some point that camera ended up going off line. When a camera falls off line it can still be pinged but you can not access the web browser. The only way that I can get it going again is to power cycle it.
Hello, when the camera's go offline is it a power loss issue or a signal loss issue? If you have lost the signal you may have an IP conflict with another device depending on how the network is set up, this would cause the camera to be on but not able to deliver a signal to the DVR thus causing loss of recording. Most camera's have a light that will help indicate if the power is still being provided to the device. If they do not you can check the switch, if there is a power and connection status light on the switch this may help you diagnose where and when the problem is occuring.
Undisclosed,
Are both the Gigabit PoE switches and the Axis Midspan going through the same power outlet?
"The system is installed in a small commercial environment and is connected to their network for internet access and local viewing"
For troubleshooting purposes, try separating it from the customers network for a week and reevaluate. You might have traffic issues impacting the cameras streaming/NVR receiving them. if the NVR has a single NIC, even more so.
I ran into this issue last year with a set of Axis M1054 cameras. Four of eight M1054 cameras would go offline, same symptoms as you describe. Over a 2 to 3 week period, one would go offline, sometimes two. Cycling the port would restart the camera and get it back online. Usually they would not respond to ping when offline, but sometimes they would even though the web interface was down.
Network monitoring showed a really high rate of link errors for the four cameras, like 1,000 per day when we should have seen zero or one per week. Plus for some of the cameras packets were getting dropped but not consistently.
It turned out to be a combination of three problems.
(1) Some of the cable was bad when we re-tested it with a cable tester. That was replaced and the packet issue cleared up. The link errors remained. It may have been that the cable problems caused the connections to degrade to 10 Mbits from 1Gbits, as we experience that with some other workstation connections that had bad cable. The client software would connect, but getting recorded video from the server was slow as molasses in January.
(2) The system was originally running one VMS, and when the company was bought out their service dropped to zero, and we switched to Axis Camera Station software (it's a 50-camera system, no moitoring, just infrequent review of recorded video). It turned out that these cameras had not been reset back to factory defaults for the VMS change, and the previous VMS software had created settings inside the camera that were visible in the System Report, but were seen in the camera's setup pages unless you knew where to look for them or caught them in the System Report. The server names were no longer reachable, and this was the source of the link errors.
(3) We went months without an issue, then a camera went offline. We upgraded to the latest version of firmware at the time, because per the release notes there were "improvements to camera robustness" -- which we interpreted to mean. we fixed some bugs related to the camera going offline.
It's now been a year since we had the issues, and no problems.
If you have Axis cameras, do you see anything unusual in the System Report? Depending upon what the cause of the problem is, you may get some insight if you set up a syslog server and enable syslog on the cameras.
At the same site we also had an outdoor Axis P1347 that would go offline every few weeks. We found that the POE switch had unstable power, and replacing the switch solved the problem.
I don't or won't know if this helps or not but you can see you're not alone. I had a similar issue with GeoVision VMS and a new camera. I had been dropping signals on just one camera of 16 total no POE 24 VAC Power. I was on the phone with Tech support for an hour updating the VMS. Everything was grand stayed there two more hours pictured stayed on. Whereas before VMS would say signal loss picture would cycle on and off right in front of you. Sometimes with IE 9 I could log into camera and see the video and not on the VMS and then the issue came back hour after I left.
So another trip and call to GeoV and they said to hard reset the camera which I did then I reconfigure with my static IP worked great for about two hours just like before then here we go again it's off after I left premise already. My network stands alone from clients and I had a guy certify it for my own piece of mine however I did verify cable as it could have failed or not been seated properly. Hopefully so I would avoid conflicts such as this, as it made sense to me at the time.
Well I decided to delete all 16 cameras to "query" cameras and learn them back in VMS as there are two methods for assignment I chose this one. I have not had the camera lose signal since for about two weeks. I am on this forum to learn and reading your problem seems so time consuming I really understand how difficult it can get. One job works great the others well it only takes one to ruin the day from time to time.
You may want to check if you are using inline couplers and if someone other than yourself crimped connections sometimes the clip does not seat or "snap-in" correctly and I have found that if your looking at the connection if done with tool you should see shinny copper ends after the crimp is performed...that way you know the wires haved passed through if using "easy crimp sytle" where wire comes through RJ45 vrs. the crimps that do not pass trough and are seated they also will be shinny. I use a jewelers 4x google when I crimp mine. I even seen the whole cable nicked where it was stripped if done poorly.
Many great suggestions I think it is software related hopefully... and not a hardware issue. I'm a great cable guy not to technical yet HA! What a great place to learn it boggles my mind just how complicated IP issues can get...forget explaining it the the customer.
I have found that I have far greater success at connecting remotely with Firefox. IE is always asking for additional activeX plugins.
I think that disconnecting my system from their network for test purposes is a great idea. It certainly would make sense trying it before investing in new switches. Thanks for all of the great suggestions! One thing that I didn't mention in my first post is that all of the eight cameras are fitted with micro SD cards for data back-up in case of network loss. Is there a possibility that these may have anything to do with the issue? I really don't think so but I thought that I'd mention it. Thanks again!
Maybe also trying connecting another brand camera. See if another manufacturer of camera does the same thing.
Yes, using the wrong SD card can cause your cameras to behave in odd ways, including lockup. Check your camera specs for the recommended SD Card Class.
Harold, in support of your comments, I know that Axis states, "Extensive testing has shown that SanDisk SD cards work very well with Axis network video products. SD cards from other manufacturers may also work, but before deploying them in real installations the cards need to be extensively tested."
Our experience is similar. We've had Transcend cards fail to work, but never had SanDisk ever fail.
Here's another possible troubleshooting technique. I take it you have two unused Trendnet POE switches laying around ("... I replaced the switches.") Based on the theory of "...using low-end switches once they start seeing a lot of traffic", why not use all 4 instead of 2 switches? Even when balanced, you'll cut each switch's traffic in half. Beyond that, I'd mix up the setup. I'd have one camera each on two switches, two on one switch, and four on one switch. If both traffic and power loading make no difference, cameras should continue to fail without preference for which switch they're on. If you find cameras failing on the 4-camera one but not the others, re-balance all to 2 cameras per switch and re-check failure rates. Who knows, might clear up the problem.
However, if the greatest cost is the value of your time, then this is not your best troubleshooting approach because it depends upon observations over time, and will likely require multiple site visits.
Thanks Harold and Horace. Although the SD cards are the proper class, I may try removing one or two of them to see if that may be the issue.
I see lots of good suggestions here. I don't disagree with any of the contributed advice. Some comments... (1)if you're using DHCP, your failure description comes close to describing the moment the DHCP lease expires (and so maybe someone else gets the IP address). (2) Trendet switches are, to be kind, low end consumer grade. use a more serious switch vendor. No, not Linksys with a Cisco logo. (3) grab the network stats and confirm there aren't network glitches (some suggests explored this topic, I know.)
Try putting the cameras in the LAN DHCP range, I've seen routers cause connectivity issues with cameras assigned static IP addresses if its outside the range its looking to assign everything else.
Thanks Rodney and Steve. The cameras are using static IP addresses outside of the DHCP range. I may try changing one of them to an address at the high end of the lease range. I do agree that the Trendnet switches are lower end.
The ideal IP solution is to have all Security devices on a separate VLAN from Data & Voice with a different Private subnet. With the cameras, switches and VMS server/storage with a totally different IP subnet and all these devices set with static IPs, you can eliminate conflicts with the rest of the non-security devices on the network.
Just because you have a pool of static IP to be utilized for security devices, does not mean that other devices on the network can't have that same IP manually assigned to them as well.
On the topic of switches, just because there's enough power are on each port does not mean the switch is robust enough to handle multiple megapixel camera streaming video. You have to consider the throughput capabilities of the switch. Additionally, the uplink port has to be able to support the aggregate of the cameras across the gigabit ports.
given this was a net new installed including cabling, why wasn't CAT6 Cable pulled?
did you check if this happens if there were big changes in the scene.
Why did you use CAT5e? I learned from Anixter that CAT5e is not optimal for 8 cameras, each 2.1 MP, better is CAT6a, in special for MP cameras. I don't think you easaly can change the cabling. Try this instead: use, if available, the bit rate settings in the camera, best is CBR, not VBR. (VBR = Variable Bit Rate; CBR=Constant Bit Rate, value can be configured by user). Set this to a level max. 2000 kbps.
This will work!
Why did Anixter tell you this? Cat5e supports +1000Mbps, and is fine for connecting cameras to switches.
See our Cat 5e vs. Cat 6 for IP Cameras? tutorial for an explanation.
Setting CBR to 2000 kbps is bad advice knowing nothing about the scene. That's equivalent to ~1.95 Mbps.
Setting a camera's CBR at 2000 kbps and running Cat6 to it is the equivalent of turning a water spigot to trickle while running it through 24" pipe.
IPVMU Certified | | 11/11/13 05:41pm
LOL, most cameras only come with 100 mbps connections and streams out less than 10 mbps. Cat5e is rated at 1 gigabit and is way above the spec required to deliver the data. Cat6 only provides better shielding in highly noise environments to prevent crosstalk, it does not give you any additional length nor speed for IP camera installations. Anixter just want you to spend more money on cabling. Try to get your info from wikipedia instead of a sales person.
The more I read into this the more I think is an IP issue like some of the responders point out.
We had a situation at one site with about 200 cameras where cameras would go offline every few days, no rhyme or reason. We could ping the offline camera (or so we thought, more on that later) but could not bring up the camera configuration page or the video.
After a lot or troubleshooting and hair loss, we figured out that the customer had connected the Video VLAN to a Cisco router on the building so they could view video remotely and it so happened that even though we were given an specific range of reserved IP addresses, there were existing IP addresses on the customer side of the router that matched the IP address on a lot of the cameras on the site, therefore causing a conflict resulting on the cameras to randomly go offline.
The way we initially noticed there was something wrong with the IP addresses was by looking at the ARP tables on the switches. We started noticing that the ARP tables had the IP addresses of the offline cameras associated with MAC addresses that did not match the MAC on the cameras on those IP addresses. This explained why we were able to ping the IP address but not able to view video or open the camera’s configuration page. We were not pinging the camera but some other device on the other side of the router. The moment we disconnected the router from the Video VLAN and refreshed all the switches, all the offline cameras came online and all the cameras stayed online for several weeks.
I am not familiar with the switches you are using and whether they have a management console or not but my recommendation would be to check their ARP tables or the connection list on the switch’s management console if there is one and make sure the MAC addresses listed match the MACs on your cameras.
Cheers
Thanks guys!
in reference to the comments about CAT6 cabling, there is more to difference with CAT5e than just supported transmission speed ( 1 vs 10 Gigabit ).
a cable that meets Cat6 specifications provides significantly lower interference or near end crosstalk (NEXT) in the transmission. It also improves equal level far end crosstalk (ELFEXT), return loss and insertion loss compared with Cat5e. The result is less noise, fewer errors and higher data rates in the transmission of the signal.
as more data is transmitted in a streaming mode (constant video streaming) the more noise and interference become an issue.
worth a viewing.
I don't know if this has been raised already, but I avoid using command line ping tests to check connectivity. The ping test will tell me if some sort of device is connected to that ip - but that is all it tells me. Within it's network limitations, I find using an arp scan tool much more reliable because it gives me mac address detail. That way I can be sure that the "ping" is from the specific device in question. (hope that makes sense)
Col Jones
agree, when you have connection problems you need to check all the possibilities and remove them one by one.
guide line that I use, and works in most cases:
---
1. check the power, network cable, and switches
- Impedance matching will be an issue at times. use CAT6 whenever possible
- Replace power/cable/switches one by one to isolate the problem
- For PoE, make sure the distance is w/in the spec, and when in doubt use PoE injector closer to the IP camera
- Switches are basically a small computer which handles swiching, and the firmware could be buggy and coud result in intermittent operation. the switches are handling hundreds of MB of data every seconds, and it is a lot of work. you are better off using a decent/proven vendor for the switches (it is notoriously difficult to isolate a problem to a switch, i.e., you will be wasting A LOT of time)
- You may want to use one of those tools used for network analysis, to assess the health of your network, but usually an over kill
2. check arp table between the camera and the NVR
- ping and then check arp -a to check if that IP is actually from/for the MAC address of the device
3. suspect the IP camera FW
- successful ping only means, there is a device using that IP, and it is responding to the ICMP packet
- but there are other applications running in the camera such as http server, rtsp server, etc
- it is possible that only the http is working but the other servers like rtsp server is dead
- check w/ vendors for known issues for the current FW, and check for bug free FW
4. always a good practice to have a separate subnet for video traffic
- VLAN is good, but remember, VLAN shares the bandwidth
- I try to stay away from using VLAN, because it adds complexity and one additional management point
- Using a dedicated subnet is the best way to go, and you don't have to ask for help from the IT dept
IPVMU Certified | | 01/05/14 06:32pm
"I don't know if this has been raised already, but I avoid using command line ping tests to check connectivity."
PING is an indispensible and useful tool for checking connectivity. Many "intermittent connection issues" can be found with PING using the continuous PING "-t" switch.
ping <ip address> -t
A general PING will give back only 4 replies by default. But doing a continuous PING can show intermittent problems that may not show up until every 10th or 15th PING for example.
I'm been able to discern many a bad network port, bad line connection, bad modem connection and bad Internet connection using PING, at least isolating the problem to the customer's Internet connection (when it's a remote access problem) and being able to negate the security equipment as the problem.
I think constant bit rate change could be a troubleshooting method, perhaps maybe not as low as 2mb but chaning it could isolate the throughput capcity of the switch, of course a switch upgrade would be best. Also is your customer using any 3rd party camera apps like ipcamviewer or anything to make thier phones compatible to the camera? I have seen that cause the stream capcity of the camera to max out and lock up the video but stay somewhat responsive to other connectivity measures.
Instead of a vlan if your switches don't have that option, try just assigning your own range on the same switch hopefully your nvr has dual nic keep one on the customer subnet and the 2nd on your newly created subnet this way your only tracing down one possible duplicate ip address
Also always label the cables when installed that way you can be sure your disconnecting the right device when you loose a camera and it mysteriously still responds to a ping.
I think constant bit rate change could be a troubleshooting method, perhaps maybe not as low as 2mb but chaning it could isolate the throughput capcity of the switch, of course a switch upgrade would be best. Also is your customer using any 3rd party camera apps like ipcamviewer or anything to make thier phones compatible to the camera? I have seen that cause the stream capcity of the camera to max out and lock up the video but stay somewhat responsive to other connectivity measures.
Instead of a vlan if your switches don't have that option, try just assigning your own range on the same switch hopefully your nvr has dual nic keep one on the customer subnet and the 2nd on your newly created subnet this way your only tracing down one possible duplicate ip address
Also always label the cables when installed that way you can be sure your disconnecting the right device when you loose a camera and it mysteriously still responds to a ping.
Hi Joe. Thanks for your input. I believe that the bit rate is at default which I believe may be 4Mb. I am slowly and methodically implementing some of the suggestions. It will be a slow process because the system will run for up to a week without issue.
Undisclosed Integrator,
Spectrum is some great software, have you looked at the log files on the server? Are they saying someting about streaming stopping, or is it just saying the camera is offline?
When I am troubleshooting these types of things, I usally go through and change one thing at a time, or at least take something out of the equation. In this case, I would say take DW Spectrum out of the equation and stream the video using VLC, or another VMS to record the video for a time period. Download the free version of Exacq or Milestone and have it record those cameras for a week. So if the camera continues to go offline, you have now eliminated the software as the culprit. Then move on and replace other single parts, like the camera, the network switch.
I guess it really depends on how much time you have to fix this issue...
Thanks guys!
I have replaced on of the switches and it has greatly improved the situatuation. I have had one incident in several weeks whereas before it was every few days.
Thanks again
Hi
I may have posted in this thread already ...
I had a simialr issue with one installation. Hik cameras and VMS on COTS PC .. Probems of cameras getting offline randomly. We changed the system to a Hik NVR.. Same issues. We changed the switch to a Cisco 48-port SC-300 MP or PP.. Same issue. We tested the network cabling, they passed some of those failed we redid the connectors and it was that .... Same issues ... We installed a UPS with a longer autonomy up to 2 hours since the site experienced brownouts and black-outs quite often ... I think by now you know .. Sma e issues ..meanwhile similar site with similar number of cameras and same NVR are functioning without a hitch ... We despaired, ran a network test during the day.. network wasn't the best, no one would recommend it for a Gigabit speed applications but it would pass 100 Mb/s with flying colors ... we changed all the cameras ( at our cost) same issues ...
Finally on a whim we decided to check the cabling visually ... OMG!!! OMG!! It must have been the worst cabling job I have ever seen. With cables manually spliced, cables turning at odd angles and water in the conduits ... staying for a while until it evaporated into a muddy mess ... Cabling was performed by a different contractor and when we tested it summarily it works but not after rainfalls (subtropical climate , lot of rain). The network center is in an adjacent building where the all cables land. They pass through conduits (another botched job BTW) ... cables are indoor cables ... Turning radius are not respected .. etc
We had to redo part of the cabling. We ran fiber to the main building and installaour own cables inside the building .. Problem solved
Lesson learned : PLEASE PAY ATTENTION TO YOUR CABLING!! If it was performed by another contractor, test and retest it. Inspect it visually. Make sure it is certified or certifiable .. use the best tool you can have for this purpose .. I have come to be wary of many off-brand cable certification tools. Buy the bst tested you can afford ..CABLING people .> CABLING it was .. How's that for shouting! :D
I have seen such many times when the camera buffer get full and shut down the cameras. Only hard power restart got the camera back to work.
Your options are:
1. have local recording on an SD card
2. Replace your camera and NVR supplier
3. Try to use our PdbU which is designed to prevent such
as a start I will replace the DVR and camera to use a “different brand”
p.s. try to use cat 6 and up higher quality cables
good luck
Newest Discussions
| Discussion | Posts | Latest |
|---|---|---|
|
Started by
Brian Rhodes
|
12
|
less than a minute by Undisclosed Integrator #2 |
|
Started by
John Honovich
|
19
|
1 minute by Taylor May |
|
Started by
Alex Jason
|
1
|
less than a minute by Alex Jason |
|
Started by
Undisclosed #1
|
1
|
6 minutes by Undisclosed #1 |
|
Started by
John Honovich
|
11
|
less than a minute by Undisclosed Manufacturer #6 |
