Subscriber Discussion

IP Cameras Self-Maintenance/Watchdog/Rebooting

BH
Bohan Huang
Mar 12, 2013

An IP camera consists of a lens system, sensor and ISP system, a H264 encoder, and a SoC running some sort of OS (usually Linux).

I have seen some IP cameras (including our own) that come default from the factory with a settting that reboots the camera to ensure stability on a regular basis.

I am aware of the problem that this can cause evidence not to be captured. I fully support the notion that this is bad.

What I really want to know is how the majority of brands deal, or not deal with the possibly of the camera's OS crashing for whatever reason during operation.

I am most interested in hearing from manufactuers/distributors/resellers/integrators of the top tier brands (with the assumption that they lead the market in reliability) like Sony, Axis, Bosch, Samsung, Arecont, Avigilon, etc.

BH
Bohan Huang
Mar 12, 2013

I will most likely be crucified for this, but I am going to say this anyway for the sake of honesty:

As a manufactuer/vendor/distributor in the vertical market that is video surveillance, my upmost concern is for the wellbeing of the resellers/integrators that are our true direct customers. if our cameras have a feature that will gauranteed 100% reliability and no warranty service calls due to crashed cameras at the cost of a 0.06% chance of loosing evidence (60 seconds per day (60/86400) for a daily reboot), I will probably turn it on.

JH
John Honovich
Mar 12, 2013
IPVM

One related aspect is how to figure out if there is a problem with cameras. Typically, even 'top tier brands' make this hard to do. Getting real time diagnostics of camera performance is not very common nor is typically very detailed. For instance, we had a discussion recently on 'Any cameras show real time CPU consumption?" and we only found one who did.

MI
Matt Ion
Mar 12, 2013

I'll say this, Bohan, your honesty is refreshing :)

Just offhand, I can't think of any specific instances where I've had a camera "crash" such that a simple reboot would fix the problem. I have seen a few where a hard power cycle (ie. completely remove the power for a few seconds) would recover from a crash, but almost without exception, those proved to be cameras that failed completely sooner or later.

Going over it now, there may be one or two instances of IQEye cameras that lost image, and when I rebooted them from the web interface, image came back... but there again, those were cameras that died within a short time anyway.

Based on this, for me, if a camera is "crashing", it probably needs to be replaced ASAP. If a soft reboot "fixes" a problem temporarily, then that's not really a fix, is it? Better to get in there and take it down for a few minutes to do the trade, rather than risk to going completely, say, in the middle of a long weekend when a tech may not be able to get a replacement for a couple days, or may not be able to get to site for a couple days.

But if it's rebooting automatically anyway, I may never see the potential failure coming, until it DOES finally kack... inevitably during the weekend.

JH
John Honovich
Mar 12, 2013
IPVM

I've had many problems with cameras where a reboot fixed it. Often cameras get in weird states, where the quality degrades or you have problems connecting it. Then you reboot it and it works perfect.

Do some of them indicate that the camera should be replaced? Sure but I don't think it means all of them.

Also, Matt your advice runs counter to what many people do with many computers. People do soft reboots all the time that fix problems.

MI
Matt Ion
Mar 12, 2013

As I said, I'm just going from my own experience. If there are examples similar to yours, I can't recall any of them - with all but a couple exceptions, if I've had a camera lock up, crash, go offline, etc., it's taken a hard power cycle to bring it back (indeed, if a camera is crashed, accessing the web interface is usually difficult if not impossible), and those cameras have usually failed before too long.

This goes back to the analog days as well - I can think of numerous times with a wide variety of analog cameras, where a simple power cycle has cleared up what appears to be a dead camera... at least temporarily.

I can think of ONE instance where I actually put a camera on a timer that would cycle its power daily, to address a problem: an old Capture Fastrax II PTZ that after a few days, would fail to focus properly; focus would be stuck at one point until power was removed and restored (even a soft boot failed to help). It actually ran like for two, maybe two and a half years, until it died completely just recently, and was replaced by a Spectra III.

MI
Matt Ion
Mar 12, 2013

I will add to the above: I've seen plenty of times, with both IP and analog cameras, where a camera will lock/freeze/crash/stop working right, where a hard power cycle has brought it back, and it's continued to work fine for many years. Sometimes you get that little anomaly that causes a camera to do something weird, and it's not a sign of the camera being near death.

I'm talking more above about cameras that seem to need this attention on a regular basis, since the discussion started off talking about cameras that reboot automatically *once a day*.

BH
Bohan Huang
Mar 12, 2013

Thanks Matt and John for detailed responses so far. However what I really wanted to know was whether this self-rebooting that "plagues" our IP cameras is a "feature" of most IP cameras in general.

I have a feeling that most inegrators (no offence) do not take the time to look through all the web interface options in a IP camera to determine whether such a feature exists and is activated by default, since there is so many other things they have to worry about (like how to run the cable, waterproof the exterior connections, testing all the cable terminations, etc...)

I am interested in comments from those who actually know for sure whether the cameras they make/sell/use have this feature. We are debating whether to deactivate this feature on our cameras and of course would like to know as much about the general consensis on this subject as we can.

Thinking a bit deeper about the issue - shouldn't the camera keep working even if the OS carshes? since the video encoder is a hardware IC? If its ends up that crashing only causes the web interface to be unresponsive then we would rather switch off the feature to reduce the chance of loosing evidence. The issue is we have not yet had a camera crash out of the ~300 IP cameras we have sold this year (new models) but that statistics is pointless as all these cameras has the auto-reboot self maintenance activated as per factory defaults. Furthermore I have not though of a good repeatable way to cause the camera to go into a unsresponsive state except by drawing 10+ HD streams from a single camera (bandwidth saturation) and running a DoS attack against it - both of these methods do not reflect real world use.

BH
Bohan Huang
Mar 12, 2013

An even more tricky question is whether a randomly timed monthly or quarterly reboot is better then no reboot at all. the random timing would reduce the likelihood of a professional attempt to evade capture to almost zero but will not reduce the chance of missing an event (which for the purpose of this discussion can only be assumed to be completely random).

MI
Matt Ion
Mar 12, 2013

An even more tricky question is whether a randomly timed monthly or quarterly reboot is better then no reboot at all. the random timing would reduce the likelyhood of a professional attempt to evade capture to almost zero but will not reduce the chance of missing an event (which for the purpose of this discussion can only be assumed to be completely random).

Well look at it this way: if a crook wanted to figure out your camera reboot schedule, he'd probably need access to your VMS to and/or network to do it, in which case you have far more serious problems to worry about!

Other than that, if you have two or more cameras with overlapping views, you could always offset their reboot times, so one is always active while the other is rebooting.

MI
Matt Ion
Mar 12, 2013

Bohan, just glancing through the cameras I have here, or have easy remote access to...

My IQ511 and IQ753 do not have any auto-reboot function that I can see, although they do support some significant scripting ability and it may be possible to "add" such a feature that way.

The Dahua cameras on my bench do have an auto-reboot function automatically enabled to go at 2:00am Tuesday... seems to me I have noticed this before but have never bothered changing it. Once a week in the wee hours doesn't seem like a significant issue.

The Axis P3384s do not appear to have an automatic reboot feature.

I can find no automatic reboot in older HIKvision models... don't have a newer one handy to see if they support it.

Looking at a Vivotek IP8162, there's no auto-reboot there either, unless it's well-hidden.

And finally, an Arecont... nope, nothing there either (not surprised there, though - they're very limited in administration).

You're right though, it's something I've never actually looked for, as there are far too many other things that require tweaking just in a camera's admin interface, nevermind in the NVR itself, or the physical installation. As I say, now that I look at it, I believe I have noticed it in the Dahua, but never paid attention to it.

As for whether the camera keeps working when the OS crashes... well, I suppose it depends on the actual cause of the crash, and just WHERE it happens. At some point the interface OS has to interact with the video processor. A bug in firmware code for the encoding and/or compression itself could cause a crash at that level. Of the camera failures I've seen, few have been simply an inability to get into the admin interface while the camera keeps sending video to the NVR.

Then again, maybe the interface crashes a lot and nobody ever notices, because nobody tries to access it except to see why a camera is no longer showing up in the VMS?

SP
Sean Patton
Mar 12, 2013

The only cameras I have ever run into (out of Bosch, Axis, Panasonic, Arecont, Basler, Verint, Sony, Canon, Mobotix, IQeye, VideoIQ, Pelco, Cisco) that had a programmed in a reboot was Mobotix. I don't like the idea of that, even though its probably not that big of a deal on 99% of cameras. I would actually never sell a Mobotix camera because of that reason (my apologies to Mobotix if this was something a previous vendor had installed on the cameras I was working on), or really any other camera that requires it. I think it just points to weak design or selection of poor hardware/chips. These are not, or shouldn't be in the same catagory as a PC with Windows OS with bloat and memory leaks and 25 background programs running that requires a software client PC to be rebooted daily; this just shouldnt be the case with a camera.

JH
John Honovich
Mar 12, 2013
IPVM

Sean, Mobotix daily reboot is a 'feature'.

Btw, just because a camera vendor does not show a setting for reboots, does not mean they do not do automatic reboots. For instance, I've seen DVRs in the past that do this without any disclosure.

HB
Harris Bond
Mar 12, 2013

This might be interesting as a generic poe autoping solution for any camera. I have used their 230v power autoping products and they work great. The only problem is that you can sometimes ping a device even those it has crashed but this is not common - anyway I think you can also schedule daily/weekly reboots with also.

JH
John Honovich
Mar 13, 2013
IPVM

Harris, thanks for sharing that product. Does anyone else have any experience with it?

Btw, they have the most bizarre commercial I've ever seen for a tech product:

BH
Bohan Huang
Mar 13, 2013

Boy that is one rude advertisement - a bit too brazen for promoting a serious piece of networking gear!? Also it s 24V PoE - very few IP cameras use this I think?

I did see this a while ago which I think would be a great feature to have - unfortunately it is very expensive for the residential/SMB market we concentrate on:

HB
Harris Bond
Mar 13, 2013

The one I posted from Digital Loggers is 48V PoE.

We have used the following Digital Loggers pieces in about 50 installations so far and they work very well. We have actually standardized on these for our equipment racks (please note that these reboot mains power when autoping fails):

Avatar
Brian Rhodes
Mar 13, 2013
IPVMU Certified
I thought it interesting that the Digital Loggers midspan uses BASIC:

The ability to hook up batteries or solar directly to a midspan is a little weird. I can do that with any switch or midspan in the form of a UPS. The other features are fairly standard on enterprise switches. I know the Cisco Catalyst Chassis supports remote power management.
Also, Pricing looks to be low, but the site is unclear. They advertise "just $295", but that price is for a PDU, not a midspan.
How does overall pricing compared to other options, Harris? Comparatively high? Low? The same?
DM
Duncan Miller
Mar 14, 2013

Brian,

It is more reliable to use batteries directly hooked up to the midspan then a UPS to power a midspan for backup. A full online UPS has to rectify the power into DC then invert back into AC only to then be rectified back into DC again in the midspan. You are eliminating one rectifying stage and one invering stage by using batteries connecting direcly to the midspan thus removing multiple failure points. Thats why the Cisco also includes DC connections on many of there switches.

MI
Matt Ion
Mar 14, 2013

^In addition to points of failure, you're also reducing inefficiencies. Granted, your average switching inverter and switching power supply tend to be 85-90% efficient, but that's still a 10-15% loss occurring in two different places.

HB
Harris Bond
Mar 20, 2013

> How does overall pricing compared to other options, Harris? Comparatively high? Low? The same? >

Brian, apologies for the late reply. The Digital Loggers pricing on autoping power rebooting is as good as it gets. On autoping switch port rebooting I am not aware if there are alternatives. I will post some other alternatives when i am back on the office for autoping power rebooting, there are even some solutions for AV like Furman and PS Audio that they have a hosted solution so you do not need to setup access (port forwarding or VPN) to the site.

MI
Matt Ion
Mar 13, 2013

One thing I WOULD like to see within the camera is a "watchdog" that would reboot the camera if it loses ping to a specified IP (probably the NVR). I have seen instances where a camera is working but just looses connection and needs a reboot to get it back. I've also seen instances where a camera doesn't recover properly from an outside-caused connection loss and needs to be rebooted.

Obviously you'd want some parameters on this, so, for example, it's not rebooting any time there's a brief interruption, or so it doesn't keep rebooting constantly for something that requires on-site attention, like an unplugged cable. Maybe something that reboots it if ping is lost for more than 'x' minutes, and will only reboot once or twice before giving up, or once or twice within a set time period before "sleeping" for a while...

I see Dahua cameras have an option to trigger SD recording if network connection is lost - that's a pretty good idea right there.

CA
Chris Anderson
Mar 14, 2013

Great Discussion. We have found through the deployment of primary cellular and wireless video systems, increasing reliability of that system by having both automated and manual power control critical. In every system we deploy we include an automated pinging relay which will power cylcle all of the components in the box (NVR, Modem, Camera, etc) if it fails to get response to a major DNS server. To take this idea further, in our mission critical and enterprise grade solutions it has become so important that we've developed power control boards which can report real-time power draw information and indivually power cycle ports for specific devices which may be experiencing issues. These two capabilities combined have saved the day on many an occasion, especially in those deployments which end up being remote in nature. There is no reason this functionality can't be used in fixed situations as well. Thank! Chris

Avatar
Luis Carmona
Mar 20, 2013
Geutebruck USA • IPVMU Certified

I would like it if a lot more network devices; cameras, switches, access points, etc had an automatic reboot feature. Might have saved a $900 plane ticket that I know of.

HB
Harris Bond
Mar 20, 2013

> I would like it if a lot more network devices; cameras, switches, access points, etc had an automatic reboot feature. Might have saved a $900 plane ticket that I know of.

Louis,

please see above for some solutions. For us the 2 DL pieces below have saved us many service calls.

I will post other alternatives but these are as cheap as they come and they work.

We design all our AV and networking systems with these, and we use additional dedicated surge protection (from Furman) and UPS (mainly APC) on all "sensitive" and networking equipment. I think this design (along with good networking gear) has helped a lot in the reliability of our systems.

DL even now have a solution that can autoping and reboot individual PoE ports which could be great for CCTV.

Avatar
Luis Carmona
Mar 20, 2013
Geutebruck USA • IPVMU Certified

Thanks, Harris, I'll give it a lookover.

Avatar
Jeffrey Hinckley
Mar 20, 2013

I had a sales guy stop in an give me one of these (Videogenix) couple of years ago. I had some NET350 Pelco devices that would always freeze up (he said he made a career of sending these to Iraq for Pelco systems).

I have gone through most cameras and have never had a reboot. Have never seen a camera freeze up that was not bad. Always use smart switches so I can remotely reboot cameras and iBoot devices in remote solar/other cameras. Once reason is that I used to use heated enclosures (Maine), but tested a bunch of IQEye mx domes in a parking garage when the temperature dipped to -25 F. All worked, but if power were to go out, and the cameras got cold and then rebooted, I would have to make a trip (tested this theory with a freezer). So now I save the hassle of AWG18 cabling and heaters and just use bullets/domes for exterior use. (Note: They usually are rated to -4 F, but I believe that must be the recovery/boot-up spec temperature - Bohan, you may be able to elaborate on this).

Oh, my vote - leave it off by default. (And for goodness sake, set default for AGC at lower level/low noise and shutter speed at night for at least 1/15 - most vendors set to sell cameras at low light).

JH
John Honovich
Mar 21, 2013
IPVM

Jeffrey, I looked at the Videogenix units and they appear to run ~$150 per camera. That's a lot of money, no? Is this mainly for specific high risk / high value cameras, then?

New discussion

Ask questions and get answers to your physical security questions from IPVM team members and fellow subscribers.

Newest discussions