Should You Always Update Firmware Before Troubleshooting?

There is varying opinion on whether you should keep a camera's firmware at the latest version as part of scheduled service/cleaning, but when it comes to troubleshooting at least, conventional wisdom says, update first, ask questions later. Additionally, vendor tech-support may insist strongly suggest on it before processing the problem ticket.

But is mindlessly complying before working the problem really wise? Considering the fact that doing so can needlessly expose you to:

  • New bugs
  • Undesired 'features'
  • VMS version incompatibility
  • Bricking
  • Difficulty establishing proximate cause

Given:

Scenario #1

Camera working fine for months. Suddenly audio is dropping out intermittently.

Scenario #2

Camera working fine for months. Then owner wants to start using VMD on camera, once enabled camera rebooting intermittently.

Case, 1 I say hold off at first, case 2, go for it...

I submit that you should establish a likelihood that a firmware update could actually solve the problem before beginning. This mainly translates into a skeptical attitude towards things that were working fine and then just stop working without anything in the setup or environment changing. In those cases it may be clearer to work other remedies, (check the microphone /cables), BEFORE proceeding with something that more likely to hurt than help.

Agree or disagree anyone?


I would opt to contact manufacturer tech support first. Do what they say.

In my experience that means "always update" the firmware, since typical call center support personel are taught to suggest this by rote. No doubt it helps front-liners prune the decision tree. Helpful, but to whom? Maybe a more senior engineer might delay a bit...

Just my experience though, have you reached front line tech-support where they actually advised against getting current first?

To be clear, I would do basic troubleshooting first, check configs, reboot, etc.

Beyond that, I'd factory default before I'd do anything else in both cases. I've had too may experiences were cameras started acting 'crazy', there was no obvious reason for it, but once I factory defaulted it, the camera worked fine.

As for firmware upgrade, I'd want to hear a manufacturer SE or read a release note that said the firmware release directly improved / fixed / added something relevant to me.

That said, statistically speaking, we've done lots and lots of upgrades (easily hundreds), I don't think there were that many times were it made things worse. I am not saying it never happens but your odds are lot better than the Russian Roulette it is sometimes made out to be.

...odds are lot better than the Russian Roulette.

John has a good list of steps here. The steps even apply to other system components that have firmware...such as network devices and RAID cards to name two other important ones.

Indeed vendor help desk folks always say to have the latest firmware. There is a real reason behind that...and that is because when it gets escalated and needs more digging...the software they are looking at matches what is installed. It is too cumbersome for them to back load a particular software version...so they take the easy way out.

I'd agree with John. If it worked fine, it's unlikely firmware will fix it and you should always consult the firmware release notes to understand what was changed. I've deployed firmware and software updates numerouse times and broken something that worked fine prior.

If it isnt broke dont fix it first. Unless, as stated there is a listed fix/feature in the notes that relates to the problem that is occuring then I would not upgrade. Restoring factory defaults, if an option, accomplishes what updating does by reloading a fresh copy of the software. If it was a software related problem this usually corrects it. Ive been told from at least a couple manufacturers do not update the firmware unless absolutley needed to address a security issue or a new feature.

Realistically though, this will always have to be case by case and depend on the manufaturers recommendation. If you can reprogram settings easily enough the factory defaulting should be, in my opinion, the first step after diagnosing and before contacting technical.

Defaulting a camera usually does not reload the software in the camera. It simply puts the settings back to a known state. One reason why I like upgrading firmware is that this then load a new copy of software on the camera. Over the years, I have seen this fix some issues.

I have also seen defaulting a camera and then specifically programming the necessary settings to test for the issue fix a problem.

I have also seen many times where a new firmware fixes an issue -whether due to to defaulting a setting (some cameras or firmware may do this), or reloading the software or whatever....

I also have seen times where a manufacturer, myself included will recommend to NOT upgrade firmware. Sometime new features in firmware add new hassles that are not needed in troubleshooting this specific issue.

I will often recommend reloading the existing firmware. If VMS integration is questioned, then I may recommend downgrading firmware to a known version listed as compatible or to match other cameras...

I will often recommend reloading the existing firmware.

Is that because the image on the EEPROM/FePROM becomes corrupt at some point, but the EEPROM itself is still viable? How often does that occur?

Just what I have found from my experience. I don't know the technical specifics if the data on the flash memory becomes corrupt or what. I do know that when firmware is written, other processes also take place, such as extracting the web page files from the firmware image, reloading the kernel, restart or reinstall services, or cleaning up temp files or logs, etc....

Some firmware depending on version or manufacturer may default settings as well, which could fix an issue as well....

That is also a holdover from the IT side. I don't have enough appendages to count the number of times I've been told virtually the same thing by computer manufacturers' Level 1 support. It has been a source of much frustration on my part. Examples:

  • "I have a failed hard drive and need a replacement"
    • "Upgrade your firmware and software."
  • "The server keeps rebooting due to a conflict between the fiber HBA and storport.sys"
    • "Upgrade your firmware and software."
  • "Not all network shares load on bootup"
    • "Upgrade your firmware and software."

Etc. etc.

"My camera is on fire."

"Upgrade the....

if you're running end-of-life unsupported software, then have a good time with "if it ain't broke don't fix it".

if you're buying cameras in 2015 that have >0% chance of bricking if you update, change camera vendors. Make sure you can roll back the firmware version if you have an issue. If your vendor claims you can't roll back versions, change camera vendors.

I would restate it as "call tech support, tell them what changed, and THEN discuss the "we should update the firmware..." path. If calling tech support is painful, change camera vendors. I agree with "first get the system up to date" as a strategy. If you feel that's too risky get the vendor in the loop on your strategy. "The vendor recommended we not update" is a tolerable (not great) integrator excuse to present to the customer in the middle of troubleshooting something.

If you have a VMS which is finicky about versions and doesn't support the latest version, then the VMS vendor AND the camera vendor are letting you down. Within reason. If Axis's 5.81.1 came out 2 days ago, cut your VMS vendor some slack if they don't have support yet.

"if it ain't broke don't fix it" is debatable. What ain't broken might include vulnerabilities.

"It's been working for years" is debatable. If you're on somebody else's network then you really shouldn't assume the environment is unchanging. This is why we preach the use of "change control" so you can catch that. "It's been working for years but we just updated the Cisco 2950's..." is a change, eh?

"if it ain't broke don't fix it" is debatable. What ain't broken might include vulnerabilities.

Agree with that wholeheartedly. Though that's a worthy but different question.

I'm advocating more of "if it's broke, don't fix something else."

"Make sure you can roll back the firmware version if you have an issue."

How many IP camera vendors allow this? I don't know. I suspect a number do not, yes/no? Ethan?

IndigoVision, for one. They have two methods and sets of firmware files: one for updating and one that completely installs the firmware from scratch. I don't remember the details of the second method but have a document saved on my work PC. I have used that method when updating a beta version of a camera.

I agree, the fantastically primitive world of security video cameras frequently has not yet discovered the concept of software rollback, even though it was invented last century, before Axis^^^^Canon "invented" the camera.

Our policy (public University) says that devices on the network must be supported with vendor patches for security and that they need to be kept updated. That said, I have seen assorted scans for buffer overrun vulnerabilities crash network connected devices. A specially crafted packet to port 80 that is intended to exercise a bug in MS IIS can leave a camera "toes up" waiting for a power cycle. The bug is not apparent until a vulnerability sweep hits the mgmt web page on the camera. Of course, the bug was always there.

It is my belief that not everything that gets fixed in vendor firmware is reflected in the release notes. Sometimes, stuff gets fixed silently, and even tech support doesn't know about the bug. Whether or not the bug is firmware related, the guys trying to help you have to know what you're running and be able to do tests on their bench. That does not require that you run the latest code, but it augers against anything beyond N and N-1.

jim warner, U of Cal network engineer

Related: In a rare move that Carl may appreciate, Foscam includes this interesting advice in their later user manuals:

Although I agree with the sentiment, I must admit that actually seeing it in writing doesn't exactly fill me with confidence about Foscam firmware in general...

Woohoo! Good for Foscam!

I believe Jim Warner hit the nail on the head. Policy is a reason for sure. In my experience updates solve many issues and the known issues are mentioned in release notes. For every exception there is a rule. Too often a quick reboot, install a patch and default the unit are offered when the phone tech is missing information. I'll agree with that also.

I'm with the poster who said, if you don't trust that a firmware upgrade will be an improvement, get a new vendor.

Unless you see your cameras as disposable (which might be okay in some circumstances) then you're in it for the long haul with your vendor. The expectation that new firmware will not brick the unit, and has a chance at fixing your latest issues is completely reasonable.