Subscriber Discussion

Should You Always Update Firmware Before Troubleshooting?

U
Undisclosed #1
Jan 03, 2015
IPVMU Certified

There is varying opinion on whether you should keep a camera's firmware at the latest version as part of scheduled service/cleaning, but when it comes to troubleshooting at least, conventional wisdom says, update first, ask questions later. Additionally, vendor tech-support may insist strongly suggest on it before processing the problem ticket.

But is mindlessly complying before working the problem really wise? Considering the fact that doing so can needlessly expose you to:

  • New bugs
  • Undesired 'features'
  • VMS version incompatibility
  • Bricking
  • Difficulty establishing proximate cause

Given:

Scenario #1

Camera working fine for months. Suddenly audio is dropping out intermittently.

Scenario #2

Camera working fine for months. Then owner wants to start using VMD on camera, once enabled camera rebooting intermittently.

Case, 1 I say hold off at first, case 2, go for it...

I submit that you should establish a likelihood that a firmware update could actually solve the problem before beginning. This mainly translates into a skeptical attitude towards things that were working fine and then just stop working without anything in the setup or environment changing. In those cases it may be clearer to work other remedies, (check the microphone /cables), BEFORE proceeding with something that more likely to hurt than help.

Agree or disagree anyone?

(4)
Avatar
Brian Rhodes
Jan 03, 2015
IPVMU Certified

I would opt to contact manufacturer tech support first. Do what they say.

(1)
U
Undisclosed #1
Jan 03, 2015
IPVMU Certified

In my experience that means "always update" the firmware, since typical call center support personel are taught to suggest this by rote. No doubt it helps front-liners prune the decision tree. Helpful, but to whom? Maybe a more senior engineer might delay a bit...

Just my experience though, have you reached front line tech-support where they actually advised against getting current first?

(2)
JH
John Honovich
Jan 03, 2015
IPVM

To be clear, I would do basic troubleshooting first, check configs, reboot, etc.

Beyond that, I'd factory default before I'd do anything else in both cases. I've had too may experiences were cameras started acting 'crazy', there was no obvious reason for it, but once I factory defaulted it, the camera worked fine.

As for firmware upgrade, I'd want to hear a manufacturer SE or read a release note that said the firmware release directly improved / fixed / added something relevant to me.

That said, statistically speaking, we've done lots and lots of upgrades (easily hundreds), I don't think there were that many times were it made things worse. I am not saying it never happens but your odds are lot better than the Russian Roulette it is sometimes made out to be.

(1)
U
Undisclosed #1
Jan 03, 2015
IPVMU Certified

...odds are lot better than the Russian Roulette.

(4)
Avatar
Mike Dotson
Jan 05, 2015
Formerly of Seneca • IPVMU Certified

John has a good list of steps here. The steps even apply to other system components that have firmware...such as network devices and RAID cards to name two other important ones.

Indeed vendor help desk folks always say to have the latest firmware. There is a real reason behind that...and that is because when it gets escalated and needs more digging...the software they are looking at matches what is installed. It is too cumbersome for them to back load a particular software version...so they take the easy way out.

(1)
(1)
Avatar
Michael Votaw
Jan 05, 2015

I'd agree with John. If it worked fine, it's unlikely firmware will fix it and you should always consult the firmware release notes to understand what was changed. I've deployed firmware and software updates numerouse times and broken something that worked fine prior.

(2)
DW
David Westberry
Jan 05, 2015
IPVMU Certified

If it isnt broke dont fix it first. Unless, as stated there is a listed fix/feature in the notes that relates to the problem that is occuring then I would not upgrade. Restoring factory defaults, if an option, accomplishes what updating does by reloading a fresh copy of the software. If it was a software related problem this usually corrects it. Ive been told from at least a couple manufacturers do not update the firmware unless absolutley needed to address a security issue or a new feature.

Realistically though, this will always have to be case by case and depend on the manufaturers recommendation. If you can reprogram settings easily enough the factory defaulting should be, in my opinion, the first step after diagnosing and before contacting technical.

(2)
Avatar
Aaron Saks
Jan 07, 2015

Defaulting a camera usually does not reload the software in the camera. It simply puts the settings back to a known state. One reason why I like upgrading firmware is that this then load a new copy of software on the camera. Over the years, I have seen this fix some issues.

I have also seen defaulting a camera and then specifically programming the necessary settings to test for the issue fix a problem.

I have also seen many times where a new firmware fixes an issue -whether due to to defaulting a setting (some cameras or firmware may do this), or reloading the software or whatever....

I also have seen times where a manufacturer, myself included will recommend to NOT upgrade firmware. Sometime new features in firmware add new hassles that are not needed in troubleshooting this specific issue.

I will often recommend reloading the existing firmware. If VMS integration is questioned, then I may recommend downgrading firmware to a known version listed as compatible or to match other cameras...

(1)
U
Undisclosed #1
Jan 07, 2015
IPVMU Certified

I will often recommend reloading the existing firmware.

Is that because the image on the EEPROM/FePROM becomes corrupt at some point, but the EEPROM itself is still viable? How often does that occur?

Avatar
Aaron Saks
Jan 07, 2015

Just what I have found from my experience. I don't know the technical specifics if the data on the flash memory becomes corrupt or what. I do know that when firmware is written, other processes also take place, such as extracting the web page files from the firmware image, reloading the kernel, restart or reinstall services, or cleaning up temp files or logs, etc....

Some firmware depending on version or manufacturer may default settings as well, which could fix an issue as well....

Avatar
Carl Lindgren
Jan 06, 2015

That is also a holdover from the IT side. I don't have enough appendages to count the number of times I've been told virtually the same thing by computer manufacturers' Level 1 support. It has been a source of much frustration on my part. Examples:

  • "I have a failed hard drive and need a replacement"
    • "Upgrade your firmware and software."
  • "The server keeps rebooting due to a conflict between the fiber HBA and storport.sys"
    • "Upgrade your firmware and software."
  • "Not all network shares load on bootup"
    • "Upgrade your firmware and software."

Etc. etc.

JH
John Honovich
Jan 06, 2015
IPVM

"My camera is on fire."

"Upgrade the....

(1)
(5)
U
Undisclosed
Jan 11, 2015

if you're running end-of-life unsupported software, then have a good time with "if it ain't broke don't fix it".

if you're buying cameras in 2015 that have >0% chance of bricking if you update, change camera vendors. Make sure you can roll back the firmware version if you have an issue. If your vendor claims you can't roll back versions, change camera vendors.

I would restate it as "call tech support, tell them what changed, and THEN discuss the "we should update the firmware..." path. If calling tech support is painful, change camera vendors. I agree with "first get the system up to date" as a strategy. If you feel that's too risky get the vendor in the loop on your strategy. "The vendor recommended we not update" is a tolerable (not great) integrator excuse to present to the customer in the middle of troubleshooting something.

If you have a VMS which is finicky about versions and doesn't support the latest version, then the VMS vendor AND the camera vendor are letting you down. Within reason. If Axis's 5.81.1 came out 2 days ago, cut your VMS vendor some slack if they don't have support yet.

"if it ain't broke don't fix it" is debatable. What ain't broken might include vulnerabilities.

"It's been working for years" is debatable. If you're on somebody else's network then you really shouldn't assume the environment is unchanging. This is why we preach the use of "change control" so you can catch that. "It's been working for years but we just updated the Cisco 2950's..." is a change, eh?

(1)
(2)
U
Undisclosed #1
Jan 11, 2015
IPVMU Certified

"if it ain't broke don't fix it" is debatable. What ain't broken might include vulnerabilities.

Agree with that wholeheartedly. Though that's a worthy but different question.

I'm advocating more of "if it's broke, don't fix something else."

JH
John Honovich
Jan 11, 2015
IPVM

"Make sure you can roll back the firmware version if you have an issue."

How many IP camera vendors allow this? I don't know. I suspect a number do not, yes/no? Ethan?

Avatar
Carl Lindgren
Jan 11, 2015

IndigoVision, for one. They have two methods and sets of firmware files: one for updating and one that completely installs the firmware from scratch. I don't remember the details of the second method but have a document saved on my work PC. I have used that method when updating a beta version of a camera.

U
Undisclosed
Feb 15, 2015

I agree, the fantastically primitive world of security video cameras frequently has not yet discovered the concept of software rollback, even though it was invented last century, before Axis^^^^Canon "invented" the camera.

jw
jim warner
Jan 11, 2015

Our policy (public University) says that devices on the network must be supported with vendor patches for security and that they need to be kept updated. That said, I have seen assorted scans for buffer overrun vulnerabilities crash network connected devices. A specially crafted packet to port 80 that is intended to exercise a bug in MS IIS can leave a camera "toes up" waiting for a power cycle. The bug is not apparent until a vulnerability sweep hits the mgmt web page on the camera. Of course, the bug was always there.

It is my belief that not everything that gets fixed in vendor firmware is reflected in the release notes. Sometimes, stuff gets fixed silently, and even tech support doesn't know about the bug. Whether or not the bug is firmware related, the guys trying to help you have to know what you're running and be able to do tests on their bench. That does not require that you run the latest code, but it augers against anything beyond N and N-1.

jim warner, U of Cal network engineer

(1)
U
Undisclosed #1
Feb 09, 2015
IPVMU Certified

Related: In a rare move that Carl may appreciate, Foscam includes this interesting advice in their later user manuals:

Although I agree with the sentiment, I must admit that actually seeing it in writing doesn't exactly fill me with confidence about Foscam firmware in general...

(2)
(1)
Avatar
Carl Lindgren
Feb 10, 2015

Woohoo! Good for Foscam!

UI
Undisclosed Integrator #2
Feb 09, 2015
I believe Jim Warner hit the nail on the head. Policy is a reason for sure. In my experience updates solve many issues and the known issues are mentioned in release notes. For every exception there is a rule. Too often a quick reboot, install a patch and default the unit are offered when the phone tech is missing information. I'll agree with that also.
(1)
SM
Steve Mitchell
Feb 10, 2015

I'm with the poster who said, if you don't trust that a firmware upgrade will be an improvement, get a new vendor.

Unless you see your cameras as disposable (which might be okay in some circumstances) then you're in it for the long haul with your vendor. The expectation that new firmware will not brick the unit, and has a chance at fixing your latest issues is completely reasonable.

(1)
New discussion

Ask questions and get answers to your physical security questions from IPVM team members and fellow subscribers.

Newest discussions