Subscriber Discussion

How Often Do Cameras Fail To Come Up After Reboot?

U
Undisclosed #1
Sep 23, 2015
IPVMU Certified

How often do cameras fail to come up after reboot?

Not because they are damaged in any known way, but just because?

I have an Axis camera that doesn't boot at least 1/5 times and a 3svision one that is batting 1/4. Other cameras definitely haven't come back one time or another but I can't say exactly.

I am considering having a camera power-off automatically because it is not needed for days at at time, but if there is more than a 1% chance of it not coming back, it's probably not worth it.

Whats your gut, do cameras come up 99/100 times?

MW
Mike Weymer
Sep 26, 2015
IPVMU Certified
99%. I've had issues and either poe power cycle it, or a firmware update helps.
(1)
(1)
Avatar
Ross Vander Klok
Sep 28, 2015
IPVMU Certified

My AXIS cameras come up more like 99.9% of the time. If it is less than that there is an issue somewhere, but where that "somewhere" is.....???

(1)
U
Undisclosed #1
Sep 28, 2015
IPVMU Certified

Ross, that is impressive.

How many do you have? Are you saying when they don't come back up, that even then it's usually not the fault of the camera?

Avatar
Ross Vander Klok
Sep 28, 2015
IPVMU Certified

We have a little over a hundred AXIS cameras and a few dozen analog on AXIS encoders. I cannot remember a single time where a camera has not come up after a reboot. If I can reach the camera over the network to reboot it, it comes back up.

I have had cameras that lost their POE switch or their POE injector, cameras that were fried by surges or lightning, cameras whose ports were swapped by “helpful” IT techs and numerous other issues, but if I can reach a camera to reboot it, it comes back up.

(1)
U
Undisclosed #1
Sep 28, 2015
IPVMU Certified

You must be doing something right!

Side question, would you say over or under a minute reboot time on average?

TS
Todd Smith
Sep 28, 2015
IPVMU Certified

When you restart an Axis camera by direct ip to it the camera will give you a running clock when it should come online in seconds.. off hand I think it is something like 149 seconds.

(1)
U
Undisclosed #1
Sep 28, 2015
IPVMU Certified

True.

At the 120 second mark I start to lay on the Ctrl-F5's with full typomatic action. Maybe that's my problem.

Avatar
Ross Vander Klok
Sep 28, 2015
IPVMU Certified

Over a minute for sure. I would put it around three minutes as the sweet spot. Those seconds go by pretty slow though.

(1)
MM
Michael Miller
Sep 28, 2015

I can't recall any Avigilon cameras that we have installed not come back up after a reboot.

(1)
U
Undisclosed #1
Sep 28, 2015
IPVMU Certified

Do you mean to say that they always show back up in the VMS automatically?

Or that they are just alive to some degree, thru the direct camera interface perhaps?

U
Undisclosed #2
Sep 28, 2015

Yes, they always show back up in the VMS automatically

(1)
(1)
Avatar
Ross Vander Klok
Sep 28, 2015
IPVMU Certified

Some times it may require a restart of the VMS Client for them to display, but they normally show right back up on the server side.

U
Undisclosed #1
Sep 28, 2015
IPVMU Certified

Just to clarify, in case it matters, I don't only mean warm booting, i.e. rebooting from the user interface. I mean cold-booting because your powering down the switch or whatever.

I would say I have seen more problems with cold-boots.

UM
Undisclosed Manufacturer #3
Sep 30, 2015

Last year, I powered up approx. 700 cameras, in 2 groups of 16. Each group was on its own 24-port PoE switch. Every single one of them powered up fine (cold).

After performing the necessary firmware upgrade, they rebooted (warm). Out of the 700, I had 12 that didn't report the new firmware version/didn't take it. I put them aside, and at the end, plugged them back in. Each one powered up fine, and then took the firmware update.

(1)
U
Undisclosed #1
Sep 30, 2015
IPVMU Certified

Last year, I powered up approx. 700 cameras, in 2 groups of 16. Each group was on its own 24-port PoE switch.

So two groups with 16 switches in each group?

Whose cameras? (Please say Arecont.)

U
Undisclosed #2
Sep 30, 2015

I have no doubt that was Avigilon cameras :)

(1)
UM
Undisclosed Manufacturer #3
Sep 30, 2015

Sorry. I should have been more clear. I did 2 groups of 16 cameras at a time. I only had 2x-24 port switches available. The cameras were in distribution warehouse boxed up. So, as each pallet arrived, I would remove the camera, and connect 16 cameras to a switch and procede to upgrade. Then while that was running, I would connect the next group of 16, etc...

If I had more switches, I would have done more at once. It was to get the camera firmware up to date for deliver to the customer before installation (long story).

The camera brand did not start with an A.

U
Undisclosed #1
Sep 30, 2015
IPVMU Certified

Ok, that makes sense.

Were you able to use a mass update tool or a script at least?

Otherwise it sounds like a pain in the capital A ass...

UM
Undisclosed Manufacturer #3
Sep 30, 2015

Yes, I used the manufacturer's tool to mass update all 16x cameras at the same time, so it took 4-5 minutes, same as updating 1 camera.

Avatar
Jeffrey Hinckley
Oct 01, 2015

If you are talking about a camera running, that is hard reset, for me the number is probably 1 out of 1000+. I have probably put in and manage (since 2006 for IP) several thousand cameras ranging from ACTI-Vivotek-Axis-Arecont-Panasonic-Hikvision-Pelco-ISD-Vivotek-Sony. If they are down, we first look at POE indicators the web based portals of the POE switches, to see if power is being pulled. Then we may reboot that POE port. Normally 50% of the time, camera comes back up. If a tech did not label camera names in the switch, we have to reboot the entire switch. Never seen a camera not come back up. (Of that list, the Pelco cameras were the most problematic though). All cameras normally 45-90 seconds for reboot (Pelco 3 minutes).

Maybe it is your POE negotiation/switch brand?? Overloaded the POE capacity of the switch?

Cabling? Is it a certified installation (<100 meters, proper termination, 4-5-7-8).

Duplicate IP addresses on network (I assume you may be plugged into network?).

Wrong or conflicting subnet/gateway information if accessing through a router or maybe a loop somehow in network if not accessing direct to camera? Sounds like all cameras working.

Multiple VMS or applications trying to connect to camera??

Browser is not set to refresh page every visit?

Non-heated camera in a very cold environment (this would mean you are in southern hemisphere right now, I guess).

Your problem is not the cameras, it is something else.

(1)
(2)
U
Undisclosed #1
Oct 01, 2015
IPVMU Certified

They are cameras on a test bed, so they do get the worst treatment in terms of strange settings, being connected to any and all switches. I just did a test on this re-defaulted axis 3006 till it failed, 1/10, which is better than I thought. But maybe it's shot.

I curious to know though, from you and everyone else who have less than < .1% reboot failures is:

When your cameras DO fail, assuming they do, do they just die an instant death, never to boot again, even once? Don't they ever permanently fail WHEN they are rebooting? Like because of corruption to the flash or something that doesn't get accessed UNTIL you boot that last time?

Most plane crashes happen during takeoff and landing, I assumed that cameras likewise crash more often during startup and shutdown, but I never really investigated it.

Thanks for the info!

KL
Keefe Lovgren
Oct 02, 2015
IPVMU Certified

i have the same experience as many in this discussion, very few if any have not come back up... not to say that the possibility of it not coming back up isn't in the back of my mind...

Avatar
Jeffrey Hinckley
Oct 02, 2015

Looking at the history since 2007, when I first went all IP, most IP cameras that failed (not due to damage or power surges) were due to manufacturer (Vivotek had a run of FD8134 cameras where the POE negotiation would fail 1-18 months after install, faulty diode on board I believe, was about 80% failure which killed us, finally recovering) or installation (non-certifed cabling install - many times by EC), or just plain faulty products (Pelco Sarix). 99% of replacements for cameras 5-8 years old are either due to degraded video or firmware incompatibility (or because they were just not good cameras and functionality to begin with compared to current offerings - ACTI ACM3401/5611).

All of our installs are done with certified cable testing, patch panels, patch cables, and proper mounting (no exterior flush mount - pendant exterior - liquite - no exposed cabling). We use POE Smartswitches with proper wattage for overhead on all cameras (thank god we have finally replaced all of those FS116 switches).

Believe me, I have just about every type of camera out there, and have never seen reboots required, or a camera not come up after a reboot (unless failed firmware upgrade or techniques doing so).

SM
Steve Mitchell
Oct 02, 2015

You need to be specific about the make/model and even the firmware version of software you're talking about before you're going to get a real sense of what the risks might be of losing a camera from a reboot. It depends almost completely on the manufacturer and software. With the network environment being a wild card.

As a manufacturer, I feel like even very very low failure rates of our cameras is unacceptable. We happen to monitor every one of our cameras deployed, and try to do a root cause analysis on any failures.

Most manufucturers use techniques common in embedded systems to ensure availability, including:

* Use a 'factory default' OS image to fall back to in the event the current firmware image fails to boot a certain number of times in a row--this is designed so a 'bad' firmware update will not brick the device.

* Use a hardware watchdog that must be 'fed' periodically by higher level software functions or the camera will reboot--this technique tends to work around deadlock or other failure conditions of the higher level software.

* Use of image based (rather than incremental) updates--designed to better manage the final configuration of the software and prevent bugs in the update process itself from inadvertently causing the software to diverge into an untested configuration.

Even the most reliable IP devices do have a hurdle to jump when they reboot though, and that's rejoining the network. It can occasionally be like walking outside to get the newspaper and finding you accidently locked the door behind you. This depends on the network and the camera's setup. I just point it out because it's an area of risk where even the most reliable cameras may stumble.

My advice is if you 'feel' like you're at risk of losing cameras after a reboot, then you should find a new vendor. The manufacture's job is to make you feel like their equipment is reliable and trustworthy. If they can't do that then somebody else can.

U
Undisclosed #1
Oct 02, 2015
IPVMU Certified

Ok, so I will do a little test.

I'm thinking one switch, one test camera, one control camera, one recorder.

Recorder set to record continuous on both cameras. Video loss alarm enabled.

Control camera FOV will include switch, test camera and monitor of recorder.

A timer process wil cycle POE to the test camera every 5 minutes.

Let run for 24 hours.

Count video loss alarms, verify from control video any gaps found. The number found / 300 = reboot reliability rate.

Is this an unfair test?

New discussion

Ask questions and get answers to your physical security questions from IPVM team members and fellow subscribers.

Newest discussions