Subscriber Discussion

Why Do VMSes Sometimes Have Difficulty Reconnecting To IP Cameras?

UM
Undisclosed Manufacturer #1
Mar 01, 2018

Why do IP cameras just go offline?

I ask myself the same question often. I can understand a technical "hiccup" occurring on the communication from time to time. What I don't understand is restarting the VMS service restores that one camera connection. Why can't doesn't the system have some sort of error correction in it to retry on it's own when it looses the stream? Why do I have to restart the VMS to restore that one camera?

NOTICE: This comment was moved from an existing discussion: Top Video Surveillance Service Call Problems (Statistics)

JH
John Honovich
Mar 01, 2018
IPVM

I made this its own discussion because I am hoping to get feedback from some VMS or IP camera manufacturers. We've definitely seen this as well - IP cameras will become disconnected but sometimes will not reconnect until a person manually disconnects and reconnects the camera from the VMS, etc.

I am sure it varies based on implementations of the VMS and IP camera but suspect there are some common issues underlying this.

(3)
(1)
UM
Undisclosed Manufacturer #1
Mar 01, 2018

Thank you!

Avatar
Michael Budalich
Mar 01, 2018
Genetec

I agree with you it is weird sometimes the methods one has to go through to reconnect a camera to the VMS. Most of the time it is just a firmware or credentials issue but there are still those occasions when everything is entered properly and the camera still will not connect.

 

What VMS are you having this issue with currently?

UM
Undisclosed Manufacturer #1
Mar 01, 2018

Michael, it's not a matter of getting the camera connected the first time, but when it happens to cameras that were already connected and working (so it's not credentials), and the VMS looses the stream. And not all cameras, but just one or two, and I've seen it happen randomly. You'll log into the camera web page and the video is fine. You sometimes get the stream back by rebooting the VMS, but I've gotten it back many times by simply doing something like changing the IP address in the channel setting to something different, then changing it back, and the stream comes back. So altogether it leads me to believe it wasn't so much a problem in the camera. Maybe it was a hiccup in the VMS stream module, and just a little "kick" brings it back. My question and I believe others is, why doesn't the VMS have something that will retry the stream automatically without you having the tickle it?

I've seen this in several name brand VMS's we've worked with other the years. I don't want to say which because it would be too revealing and I don't want it to seem I'm blaming a particular brand when it looks to me like it's not that uncommon across multiple brands.

MM
Michael Miller
Mar 01, 2018

Yea I would be interested in hearing what VMS you're having this issue with. Is this on one job or does it happen over multiple jobs? What does your network layout look like?   It's not something we experience with Avigilion. 

(1)
UI
Undisclosed Integrator #2
Mar 01, 2018

I've found that if you're able to spend a little more up front and purchased a managed switch you're able to remotely log into it to cycle the port the camera is on.  This is very helpful unless you're installing a VMS in which the NVR has integrated ports.

(1)
UM
Undisclosed Manufacturer #1
Mar 01, 2018

Yes, we do have managed switches and we can do that, but with all due respects, that's not the question. The question is why does someone have to manually do this? If a simple hiccup of some sort occurs to disrupt the VMS receiving a good stream, but network communication is otherwise good, it's not an issue with the camera, then why doesn't the VMS reestabblish the stream on it's own.

(2)
Avatar
Josh Hendricks
Mar 01, 2018
Milestone Systems

 

This is a difficult question to answer in a general way as there can be a lot of causes for this symptom. But I suppose the very simplest answer I can give you is the VMS probably has a "bug".

Restarts should never be necessary, regardless of the component (camera, software, switch, router, etc). But since software is written by imperfect people, it usually ends up being imperfect as well.

I put quotes around the word "bug" above because it might not necessarily be something wrong with the software. Sometimes cameras do unexpected things like send malformed video, or respond to HTTP GET/POST only partially, or in otherwise unexpected ways. It might not be that the VMS did anything wrong to cause the issue, but there's usually something you can do to recover gracefully. And sometimes it's easier said than done depending on the type of failure and the software design.

Milestone has the ability to detect and recover from errors. For example, you can almost always sever the network connection and restore it later, and the camera will come back online. I've even used Clumsy to make the video stream so corrupt that it introduces weird blocky colors and eventually kills the video stream and when I turn off Clumsy, it returns to normal. If a broad driver-wide issue occurs somehow which can affect multiple cameras, we'll even reload that entire driver. Still, we might not be prepared to handle every possible error condition. So if restarting the server solves the problem, we want to know about it so we can make the software even more robust.

 

(4)
(5)
UM
Undisclosed Manufacturer #1
Mar 01, 2018

That's good insight, Josh, thanks! And bonus points for introducing a new software utility to me, which I love.

(1)
UI
Undisclosed Integrator #3
Mar 02, 2018

Really great information to hear Josh.  It's rare to get an unfiltered response such as this.  I have seen the "it's not up but I can see it via browser" symptom a few times in the past on different VMS and it has always confounded me.

(1)
JC
Jesse Crawford
Mar 01, 2018
OpenEye

I'm no longer a manufacturer, but I can give you the sales answer I would have given you when I was at Avigilon. It's also not really an answer to your question as to why the connection is restored when you restart your VMS 

Avigilon cameras are programmed to reboot periodically when they don't have a connection to a server.  This way, in the instance where a camera drops off, it will reboot itself, which usually restores the connection.  This was a great feature in all but one application over the years.  We had one customer that bought some Avigilon cameras because they didn't want to buy new analog cameras.  They used the analog connection to connect to their DVRs.  Since there was no connection to an ACC Server, the cameras were continually rebooting.  

(3)
(1)
UM
Undisclosed Manufacturer #1
Mar 01, 2018

Interesting feature and I can see it being useful in that circumstance. But then that begs the question, if rebooting the camera solves the issue, then is the issue at the camera then, and if so how hard is it to write firmware that just reloads the stream without having to do a reboot of the whole camera? Or is it more for overcoming a network issue?

(1)
MM
Michael Miller
Mar 01, 2018

Keep in mind this only happens if the camera is not connected to the VMS.  If the camera is connected and the stream stops the auto reboot would not happen.

(1)
Avatar
Brian Levy
Mar 02, 2018

Sometimes cameras lose connection due to bandwidth problems on the network. In cases where the camera system shares a network with the main business or home network, there is just not enough bandwidth to go around. With the current popularity in multi-megapixel cameras, networks are being challenged and in some cases there is just too much competition from other services. TCP/IP was never designed for streaming video. 

(2)
UM
Undisclosed Manufacturer #1
Mar 02, 2018

Brian, what if there is plenty of bandwidth and room for the cameras? Even if it were bandwidth that caused the initial problem, why do some VMS systems fail to re-connect to the camera when the network and camera both seem to be working fine?

Avatar
Josh Hendricks
Mar 02, 2018
Milestone Systems

This is true, TCP/IP wasn't specifically designed for streaming video, in the same way it wasn't designed for Spotify, VoIP or Battlefield 3. But it accommodates video extremely well. Occasionally I do see a saturated network where there are obvious bandwidth issues, but it's surprisingly rare. With gigabit networking being the standard now, it takes a lot of cameras and/or very high bitrate to saturate a network. In these cases, the connection would be intermittent. Restarting the VMS would probably also work, but network congestion doesn't typically result in a permanently-down camera.

There's a really useful tool for testing network throughput called iPerf when you need to verify whether the network is really struggling to handle the throughput. Run iPerf in server mode on one computer, and client mode on another, and it will tell you the throughput and packetloss pretty reliably.

(1)
Avatar
Brian Levy
Mar 02, 2018

There have been issues from time to time with large camera implementations where cameras lose occasional connections. I’m referring to 50-100 camera implementations with some wireless links and 2+ MP cameras. If a camera is permanently down it tends to be due to a crashed camera. If I reboot the camera it fixes this issue which tends be rare. Temperary loss could be due to limited resources like memory leaks, bandwidth, etc. I have not not had too many issues when I build and manage the network. Most of the problems occur on shared networks that are overwhlemed like in hospitals. Networking equipment, including cameras, do not always fall into their claimed specifications either, in the sense that I have seen networking equipment struggle under loads that should theoretically be very doable. I believe that networking is not always a 1:1 ratio. Put enough devices that send and receive a lot of bytes and the equipment will start to struggle far sooner than you would estimate. I have seen this time and again. I would love to hear other integrators opinions on this idea. 

(1)
Avatar
Josh Hendricks
Mar 02, 2018
Milestone Systems

I have seen networking equipment struggle under loads that should theoretically be very doable.

 

You are not alone. I have seen switches drop frames when no individual port is even close to capacity. Generally, for gigabit switches I consider the capacity to be around 800Mb/s.

I don't think I've seen this on enterprise level equipment, but my guess as to what's happening is that some of the consumer grade light duty switches tend to have a thinner backplane and they can't forward frames as fast as they come in. So you'll start seeing jitter, intermittent connections, and at the network level you'll see dropped packets, retransmissions, resets, etc.

Avatar
Bobby Mancia Jr.
Mar 03, 2018
MIZELA CORPORATION • IPVMU Certified

It happened to us lately. We can't blame neither the VMes nor the camera since it works well with the other. However, we figured it out that the rj45 was installed loose at the camera side.

Avatar
Marc Pichaud
Mar 04, 2018

When you have Gb Network never use more than 600 Mb real throughputs..

When using 100 Mb ports, never exceed 60/70 Mbit .....! you will avoid to saturate switch and RJ45 port buffers and avoid  data overflow

When you are doubting on your network , I do agree with Brian , use Iperf (most guys trust in Ethernet without even testing is or just with Fluke devices) LOL

Then the non auto reconnection is most of the time coming from the VMS editor...

Try to connect with VLC using rtsp .(what is doing your VMS by the way) .. if in meanwhile you change the streams in the camera, Vlc is not able to get the changes without somebody relaunching the connection manually... so if the VMS doesn't have an internal loop to checkout the rtsp and http presence , you are done. It's software issue.

Additionnaly , Using UNicast UDP to save bandwidth and latency,  an somestimes Multicast in large installations  for multiple Security Destinations(Typical of Genetec's)  you are in passive mode using D range IP of addresses, you can't get the TCP errors if the Vms isn't looping regularly the camera presence to re-activate the rtsp streams

UM
Undisclosed Manufacturer #4
Mar 04, 2018

Have you ever gone to playback and checked the stored video leading up to the loss of connection? Is the video leading up to the lost connection clear and stable or tiling and distorted? Tiling would be indicative of significant packet loss and either network saturation, inadequate switch fabric (or poor connector) or over taxation of the server.

Why a camera does not reconnect after such an event would suggests the VMS is not polling the camera connection(s) adequately enough as indicated by Marc above or the camera is failing to recover from some internal overload. 

(1)
New discussion

Ask questions and get answers to your physical security questions from IPVM team members and fellow subscribers.

Newest discussions