"Severely Impacted" Mercury Security 2020 Leap Year Firmware Issue

By Brian Rhodes, Published Jan 17, 2020, 09:27am EST (Info+)

One of the largest access controller manufacturers has a big problem: February 29th.

Mercury Security, owned by HID, is alerting partners of the problem that will have a severe impact caused by not being able to handle 2020's upcoming 'Leap Day' if not fixed quickly.

IPVM spoke with HID about the problem, and we share their feedback inside, including:

  • An Overview Of Mercury's Severe Leap Year Date Issue
  • Which Mercury Security Intelligent Controllers Are Impacted
  • Mercury's Instructions For Fixing The Problem
  • Which Access 'OEM Partners' Use Mercury Hardware
  • How To Quickly Check If Equipment Is Vulnerable
  • Do Dealers Plan To Eat Service Costs To Fix?

'Severely ********' ******** **** **** *****

*** **** * ****** ** *******:

**** ********* *** ******* ** * time/date ****** ***** **** *** ******** hits ******** ****, ******** ***** ** 'Leap ***':

** *********** *********** **** *** ******** handle **** **** **** ************. ***** February ****, ****, **** ***, *** intelligent ********** **** ****** **** *** date ***********.

*** ******* **** *** *****, ** left **********, **** **** '******* ******' on *** ****** ******* ***** ******* LP ***********.

** ******** ** ****, ******* ********* current ******** ** *** *******'* ******, and **** *** ******* ** ** the ******* ** ********* ********:

** *** ******* **** ** ***** was ********** ** *** ********* ******** firmware ******* (**.**.*) **** **** ** the ** *********** ***********.

***** **** ** * ****** *****, in **** ****** ***, ** ** fortunate **** ***** *** ***** ***, giving ** **** ** ******* ** and ****** *** ********/********* *** *** discovered ***** *** ****.

OEMs **** ***/******** ******

*** ******** ******* ******* **** **** to ** ******* *** *** ** firmware ******** **** *** ********** *** partner.

******* ***** **** **** *********** ******** has **** **** ** *** ******** who *** ********* *** ************ *** firmware ** *******:

****** *** ******** ** *********** *********** to ******** ******* *.**.*.****.

* *** *** *********** *** ****** last **** **** *** ******** ****** been ******** ** *** *** ******** for **** *** ************.

Recommend '*********' *******

***** *** ******* **** *** ***** until ******** ****, ******* ******** ******** that ******* ****** ****** ***, ** advance ** '**** ***':

** *** ****** ****all ** *********** ** ******* *********** to reduce any risk. [Emphasis Added]

LP *********** *********** ********

*** *********** *********** ******** *********, *** physical ***** ** *** ******* ***** distinguishes *** '**' ****** **** *******, unaffected ******.

** *** ******* ****** *** ***, then **** *** '**' *** **** need ** ** ********. *******, ** the ****** *** *****, **** *** older *** ********** '**' ****** ***********.

******* ************ ***** *** ******** ****** as '******, ******, ******, *** ****** Intelligent ***********'.

Legacy ****, *** *** */*, *******, *** ** ****** *** ********

*******, *** *** ******* ******** *** impacted, ********* ** *********** ******** (**********) in '******' ******** ************* ****, *** company's ****-**** '******' ***********, *** ******** ********** '********' */* *******:

** *********** *********** **** *** ******** in “******” **** *** *** ********. Additionally, ** ****** ***********, ****** ******** (M5, **, *** **) *** ****** controllers **** ** *** *** ****** are *** ******** ** **** *****.

Over ****** ****** ****** ********

******* ******** ** ********* **** * large ******* ** *** ****** ******* market, **** ** ****** ******.

***** ******* ** ***** ****** ******* multiple *** ***** ********** *******, ** any ** ****** *********** *** **** even ** * '*****' *********** ** multiple *****, *** ******** *** ****** by *** ******* ***********.

*** ****** ***** ***** ***** ***** common ******* ********, ******** *** **** may *** ** *************:

Two ******* ** ******** ******

** ***** ** ****** *** ****** firmware *******, ***** ******** ********** ***** to ** ********.

******* ******** ***** *** *** ******* of ************ *** *** ******:

***** *** *** **** ** ****** panel ********:
  1. ****** **** **** *** **** ********.
  2. ******* **** *** ***** *******.

*******, *** *** ******** ******* *********** are ****** ** ** ******** **********, and * *********** ****** ** ******** updates *** ****** ** ** **** onsite, ********* ******* '**** ******' ** customer ********* ** ****** *** ***.

Poll: ** **** ******** *******?

*********** *** ******** **** ** ******* impacted *********, *** ********** ******* *************, and *** ****** **** ** ****** equipment **** ** *** ******** **********, the **** ** *******'* '**** ****' problem ***** ** *********** *** **** dealers ** *******.

***********, ** *** **** ** **** your ********* * ******* **** ** do *** ****** ** ** ** free?

****:

Comments (80)

With Mercury, the party never ends.

Agree
Disagree
Informative
Unhelpful
Funny: 3

This should be billable, but the only account I have affected is fairly sizable for us and has been a customer since 2002. All other accounts I have are on older EP based controllers. I will not be invoicing. Others in our company are much more exposed and I imagine they will have to invoice. With our Mercury-based systems remote firmware upgrades en masse is an option. As with I assume every integrator, this gives me great anxiety each time we pursue a mass firmware upgrade.

Agree
Disagree
Informative: 4
Unhelpful
Funny

Whatever you do, don't do system upgrades on a Friday. Nothing good came from system upgrades on a Friday...

Agree: 23
Disagree
Informative: 1
Unhelpful
Funny: 9

Whatever you do, don't do system upgrades on a Friday. Nothing good came from system upgrades on a Friday...

Learned that lesson, more than once, already.

Agree: 4
Disagree
Informative
Unhelpful
Funny: 2

Or close to lunch time!

Agree
Disagree
Informative
Unhelpful
Funny

its one of my ten commandment of operation. never start an important job on Friday, unless you want to work that Saturday and possibly sunday

Agree: 5
Disagree
Informative
Unhelpful
Funny

I can tell you we at Identiv also have a policy (that we usually adhere to) that we never release new software or firmware updates on Fridays either.

You're welcome.

Agree
Disagree
Informative
Unhelpful
Funny

Great information Brian. At this point Lenel hasn't released into the wild yet but hopefully they will soon.

Agree
Disagree
Informative: 1
Unhelpful
Funny

Thanks. A member alerted us, and individual partner notifications are trickling out, like this example from RS2:

Agree
Disagree
Informative
Unhelpful
Funny

Here is the even bigger issue on my side of the house. Where in the heck have we installed all of the new X series controllers. At some locations having to download firmware, reboot, download database almost takes an act of congress since the system is so busy downtime is critical.

Some sites this isn't a bit deal but there are some that this will be. Some can be done remotely and others can't.

This is going to take some strategic planning that is for certain. Good luck to me fellow integrators in implementing this task.

Agree: 6
Disagree
Informative
Unhelpful
Funny

This is going to take some strategic planning that is for certain.

look before you leap ;)

Agree
Disagree
Informative
Unhelpful
Funny: 1

Mercury firmware upgrades shouldn't require a database download. This may be OEM software specific, but it is not a necessity. You should confirm with the software vendor in question.

Agree
Disagree
Informative
Unhelpful
Funny

We use Lenel so you never know what may happen or what you will have to do. As always when dealing with Lenel you prepare for the worst and hope for The best.
I now with the new X series controllers this is supposed to be better but as with everything you never know. The bigger question is what will possibly break with this update.

Agree: 3
Disagree
Informative
Unhelpful
Funny: 1

Some manufacturers have been pretty good about the notification as soon as possible like cited above. Big surprise here: LenelS2 is sending mixed messages to different personnel when asked. If you are a LenelS2 partner assume this update is needed and applied unless otherwise proven on any X-series boards.

Agree
Disagree
Informative: 1
Unhelpful
Funny

I was born on that day. It’s amazing how often I can’t signup for something using my actual birthday!

Agree
Disagree
Informative
Unhelpful
Funny: 12

Next topic: Adobe Flash EOL in 2020.

Think about LenelS2's maps.

Agree: 2
Disagree
Informative
Unhelpful
Funny: 6

Genetec notice on this:

Agree
Disagree
Informative: 1
Unhelpful
Funny

Mercury is only supplying their OEM Partners with the patch. Not all potential end users are getting this update leaving numerous customers with this significant defect and risk to their Access Control Sites . So, if a previous Mercury reseller sold customers any affected Mercury hardware and aren't current OEM partner, Mercury's position is that you aren't going to get any updates. It's a pretty poor decision on their part for not protecting all end users with Mercury Hardware firmware issues.

Agree: 1
Disagree
Informative: 5
Unhelpful
Funny

Who was a Mercury partner but now no longer is? I didn't think that was even an option for a manufacturer. It seems like once you have enough boards deployed it's really hard to extract yourself for the very reason you mention.

Agree
Disagree
Informative
Unhelpful
Funny

The point remains valid if end-users were customers of OEM partners but now are not.

For example, if they stopped buying maintenance plans and adding/replacing doors to a system and just let the system operate in-situ without formal support.

Granted, it would likely be a small number of doors given how new LP boards are, but those impacted doors would stop working correctly with no prior notification from Mercury to the end-user.

Agree: 1
Disagree
Informative
Unhelpful
Funny

keep in mind that his only affects LP series controllers, which have been out for a relatively short timeline. there doesn't appear to have been any turnover with Merc OEMs in that time, so this shouldn't really be a concern.

Agree
Disagree
Informative
Unhelpful
Funny

Ah, I see, it's end-user specific. This seems fair to me.

Agree
Disagree
Informative
Unhelpful
Funny

Mercury response:

Mercury is supplying to all the OEMs that have sold the effected products which goes back to the start of sales in July 2018. All of those companies are still current OEMs.

Agree
Disagree
Informative: 1
Unhelpful
Funny

if [ $$getdate == '02-28-2020' ] ; then

setdate '02-27-2020'

fi

;)

Agree
Disagree
Informative
Unhelpful
Funny: 5

To Brian's point this current issue affects a very limited number of customers. My point was Mercury's position not to support older OEM Partners with firmware updates that are critical is not a good business practice nor is it normal in today's marketplace. Anyone of their current OEM Partners may one day move away from Mercury and could have a larger exposure to this type of issue in the future.

Agree
Disagree: 1
Informative
Unhelpful
Funny

if someone moves away from Mercury, why would a Mercury firmware problem even factor in? if they don't use mercury then they don't have mercury firmware at all right?

also older, unsupported, OEM partners wouldn't be using LP series controllers in non-legacy mode, unless they have done so on their own without consent from Mercury so i don't see why you think Mercury should theoretically support them.

Agree
Disagree: 1
Informative
Unhelpful
Funny

Your explain doesn't make any sense for a security product or any computer product. If a defect or potential cyber security risk shows up on a product that was install & purchased in good faith and now since the end user hasn't purchased more or enough of any of their products that company now isn't not responsible to their older customers. This isn't a good business practice. How would you feel if a company such as Microsoft implemented a policy that since you haven't purchase enough products from them or their partners nor have a service agreement, you are not going to get any system updates or critical system patches? Most companies follow a process to provide critical updates and patches for their hardware and software until they are end of life.

Agree
Disagree
Informative
Unhelpful
Funny

Isn’t that going to endlessly loop? 🤣

Agree: 3
Disagree
Informative
Unhelpful
Funny

Isn’t that going to endlessly loop?

Image result for groundhog day not bad for a quadruped

Agree
Disagree
Informative
Unhelpful
Funny: 1

Isn’t that going to endlessly loop?

hopefully ;)

i'm still running a 30 day sql server 2000 trial on a gateway pc with much success.

Agree: 1
Disagree
Informative
Unhelpful
Funny: 3

Mercury has posted this pop-up notification to their portal:

However in terms of broadcasting the issue, Mercury says their 'partner OEMs' are the ones responsible for getting word to end users. In a followup to IPVM, Mercury says:

  • The notification process was first through our OEM partners. They needed to do their testing and confirm our fix.
  • The OEMs are also the only ones who know their dealer channel. While we encourage the channel to be registered on our Portal, that is not a majority of the industry.

They close by stating 'we are trying to be as proactive as possible to get the word out.'

Agree
Disagree
Informative
Unhelpful
Funny

It took less than a day for a cloud company sales rep to market cloud's benefit here:

It's certainly a fair point and will help, at least, somewhat in making the case for more cloud access control.

Agree: 3
Disagree
Informative: 3
Unhelpful
Funny

You don't need a cloud to have a centralized update solution. It was called "Windows Update", back in the day...

How many of these PACS solutions have bulk update capability? Is that common?

Agree
Disagree
Informative
Unhelpful
Funny

You don't need a cloud to have a centralized update solution. It was called "Windows Update", back in the day...

"Windows Update" always came from the cloud.

we just didn't have the word, back in the day.

Agree: 2
Disagree
Informative
Unhelpful
Funny

windows update for your onsite server came from outside your site, yes. that's not the same as a cloud solution.

also it's a bit disturbing that the updates are popping up at different times from different vendors. some info suggests the legit version didn't exist until last night so what was Feenix pushing...

Agree: 1
Disagree
Informative
Unhelpful
Funny

some info suggests the legit version didn't exist until last night so what was Feenix pushing...

HID sent updated firmware to their OEM partners mid-January. Individual partners release the subsequent 'tested and verified' firmware to dealers on their own internal schedule.

Agree
Disagree
Informative: 1
Unhelpful
Funny

Glad to learn FEENICS has pro-actively tested and released the fix. Saved me a phone call!

Agree
Disagree
Informative
Unhelpful
Funny

I coworker that was handling a global customer genetec system upgrade and was embedded on a monthly basis to bring all of their systems update with firmware and software. they stopped half way though the process and cancelled the support contract last month saying unofficially it wasn't cost effective and they didn't see a difference.

they have do have an awful lot of these boards scattered throughout the world in their buildings...... oh well I am sure it will be fine we do change orders and new contracts pretty quickly

Agree
Disagree
Informative
Unhelpful
Funny

Has anyone simulated this on the bench yet? If so, tell us what happens.

Agree: 1
Disagree
Informative
Unhelpful
Funny

Somebody's done this, right? If nothing else one might want to test the rest of your solution to confirm this is the only leap-issue .. customers will presumably ask...

Agree
Disagree
Informative
Unhelpful
Funny

It’s been a few years since I worked on Lenel systems, but back then you could load a firmware update into the servers and schedule when it would push it.

Seems like this isn’t as big an issue for them, unless I’m wrong.

Agree: 1
Disagree
Informative
Unhelpful
Funny

The real problem will come to Lenel platforms that are running unsupported versions (Pre 7.2) if the firmware hasn't been tested there. The step from 6.x to 7.x takes some decent coordination on some of the enterprise platforms, especially with large thick-client deployments.

Agree
Disagree
Informative
Unhelpful
Funny

Are people even reading the releases from vendors? older versions of Lenel (7.4 and prior) operate in Legacy mode so are NOT affected. older versions of Genetec (5.7 and prior) operate in Legacy mode so are NOT affected. this means that all your old customers who don't bother to keep up with their SUSP/Care Package/SUP or whatever the vendor refers to it as DO NOT HAVE THIS PROBLEM! unless you have a customer that had their service agreement expire very recently and has kept up to date, this is not an issue at all.

as for the argument of 'what if the customer moves to a different platform in the future' that is also ridiculous to call out here as mercury's problem. regardless of what vendor/platform it is, you will have to download some sort of firmware when you switch over, so this point is also nonsense.

Agree: 2
Disagree: 1
Informative: 1
Unhelpful
Funny

Agreed. Also when and if they do finally upgrade to a new version then you will be pushing firmware downloads at the time anyway and the customer will also be prepared for this.

Agree
Disagree
Informative
Unhelpful
Funny

UI6, thank you for the detail! There are those of us from different verticals here, receiving different communication on this.

My communication from the Synergis product team doesn't include any notes on legacy mode. Additionally, Genetec's posted KBA-79024 has no mention of Legacy mode.

From a customer perspective, that's all the information they have.

Agree
Disagree
Informative
Unhelpful
Funny

Hello, this is Despina Stamatelos, Product Marketing Manager, access control at Genetec.

I would like to clarify the comment above stating that older Security Center versions (5.7 and prior) operated in legacy mode and are not affected by this issue. This statement is inaccurate.

Thanks to the decoupling feature available through the Synergis Cloud Link three-tiered architecture, the LP panels have been genuinely compatible with older versions of Security Center in backward compatibility mode. Therefore LP panels appear as EP panels in older versions, prior to 5.8 (released in May 2019). As such, Genetec has never used the legacy mode to integrate the Mercury LP panels in Security Center.

We confirm that ALL LP panels are affected by this issue and Genetec customers must upgrade to Mercury firmware 1.29.0 regardless of the Security Center version being used. “

Best Regards,

Despina

Agree
Disagree
Informative: 6
Unhelpful
Funny

but can you do bulk updates.

Agree
Disagree
Informative
Unhelpful
Funny

Yes we support bulk firmware update and we also linked the KB on how do it in our notification.

Agree
Disagree
Informative
Unhelpful
Funny

Here is the deal as we have all been through. It doesn’t matter what system it is, you can NEVER push a firm download and it always comes out roses.

Agree: 1
Disagree
Informative
Unhelpful
Funny

And that would be when one trots out the "you have a staging system, right?" interview. Let me guess, you're a global integrator, you have 8000 offices and they're all silos and nobody has budget for a test system...

Agree
Disagree
Informative
Unhelpful
Funny

I guess in the end we should be happy they figured this out before Feb 29th has passsed!!

Agree: 1
Disagree
Informative
Unhelpful
Funny: 1

LenelS2 has sent a tech bulletin to dealers on this issue:

Agree
Disagree
Informative
Unhelpful
Funny

Cursory research suggests Lenel "can" do bulk update but it's not advisable more than (5?) at a time. It would be interesting to hear some feedback from Lenel users who have pushed the button...

Agree
Disagree
Informative
Unhelpful
Funny

That notice is a full week and a half behind our other Mercury-based manufacturers. The notice is also about a week and a half after the mixed messages we received when we queried LenelS2 on the same item. That's slow... but at least it is not late-February!

We have one Mercury partner that not only told us how many affected boards there are but also listed all of the POs we purchased them on so we knew with certainty the sites involved. All unprovoked. To me, that was impressive.

Agree: 3
Disagree
Informative: 1
Unhelpful
Funny

Would you mind sharing the manufacturer that did that? Agree it's impressive and well-deserved marketing.

To be clear, I'm not asking a leading question here...I'm not a manufacturer, and I have no idea who did that...but am legitimately curious to learn.

Agree
Disagree
Informative
Unhelpful
Funny

Anyone know if Honeywell's Pro-Watch boards are affected? I brought this issue up with my managers, but they don't seem concerned. Honeywell is notoriously slow with sending out information.

Agree: 1
Disagree
Informative
Unhelpful
Funny

Honeywell default is to consider the worst-case solution..........in my experience......

Agree: 1
Disagree
Informative
Unhelpful
Funny

(Someone check this...) Honeywell uses their own hardware (EP1501 clone?) running Mercury's software. I would think they are not using Linux (i.e. the software in the LP boards.)

Agree
Disagree
Informative
Unhelpful
Funny

Honeywell boards are not effected. However, Honeywell also sells and supports Authentic Mercury, so it's possible customers could have LP.

Agree
Disagree
Informative: 1
Unhelpful
Funny

I applied the Avigilon supplied firmware update to our LP1502 panels. In short, the update caused problems for our installation; the panels stopped working properly. I rolled the firmware back and we returned to normal. Avigilon performed some trouble shooting, but were unable to explain the failures. I recommend testing the patches closely.

Agree: 2
Disagree
Informative: 4
Unhelpful
Funny

Sherman:

Thanks! We have asked Avigilon for clarity/comment on this.

Agree
Disagree
Informative
Unhelpful
Funny

Thanks for the reply, Brian. Avigilon has been in touch with me and is actively researching the matter.

Agree
Disagree
Informative
Unhelpful
Funny

After some testing, Avigilon identified the issue. I have some controllers with no readers attached (for example, vehicle exit gates). The Mercury board with the updated firmware did not respond properly until Avigilon added in a software "dummy reader". Avigilon forwarded the information to Mercury, but as of yet there has not been any indication they will publish updated firmware. At least I know the work-around.

Agree
Disagree
Informative: 2
Unhelpful
Funny

Thanks Sherman. We have reached out to Mercury for comment on this.

Agree
Disagree
Informative
Unhelpful
Funny

[Update] Bosch was incorrectly listed as an affected partner. They are not a Mercury partner and have been removed.

Thanks.

Agree
Disagree
Informative: 2
Unhelpful
Funny

I just updated 5 x LP1502 controllers used in an S2 installation.
The new version is: 1.27.8 (629)
The installation went without any issues.
Guess I'll know if it's fixed on March 1st.
Only issue I had was physically setting a DIP switch on each board so I could login with the default credentials. I went ahead and created a user on each of them to make future updates simpler.

Agree
Disagree
Informative: 1
Unhelpful
Funny

It is my understanding they are advising everyone to go to 1.29. You probably want to verify you're supposed to be on a (now older) version.

Agree
Disagree
Informative
Unhelpful
Funny

I just asked my supplier to verify the version he sent me is correct.

I’m relieved that I created a user on each of the boards... just in case!

Agree
Disagree
Informative
Unhelpful
Funny

My integrator is still saying that S2 recommends being on 1.27.8
If anyone from S2 knows otherwise, I'd appreciate the feedback.

Agree
Disagree
Informative
Unhelpful
Funny

I will ask LenelS2 and report back here.

Agree
Disagree
Informative
Unhelpful
Funny

Here is S2's response:

We ship 1.27.8. Also, we typically ship with panels in legacy mode, which would avoid the issue altogether, but an installer could change that.

Please let the integrator know that our support team is available to answer these questions should this sort of detail come up.

Agree
Disagree
Informative
Unhelpful
Funny

Seems a good time for this shameless plug:

Agree
Disagree
Informative
Unhelpful: 2
Funny

This is promotional but I am going to leave it with the statement this is in poor taste to trash your competitor.

As an email to customers / partners, I can see why it makes sense though beyond the marketing angle as some people might genuinely be concerned about who is impacted.

Agree: 2
Disagree
Informative
Unhelpful
Funny

I've learned never to gloat when your competitors slip, you could be the next one...especially in this industry.

Agree: 3
Disagree
Informative
Unhelpful
Funny

Agreed. Good thing there was no gloating there. Just informing the industry that we support Leap Year, and always have :)

Agree
Disagree
Informative
Unhelpful: 2
Funny: 1

Yes until a programmer makes a mistake and this mistake isn't caught until the system is running out in the wild. Everyone makes mistakes it is how you handle the mistake which separates the competition. I was happy we had over a months notice about this issue.

Agree: 2
Disagree
Informative
Unhelpful
Funny

Interestingly, Tyco (Software House) is also making similar statements via Twitter:

Agree
Disagree
Informative
Unhelpful
Funny

Curious how these firmware updates are going for folks. Anyone reporting significant issues with the firmware upgrade process? Panels dropping and not returning etc..? What is the typical down time of a panel during this upgrade?

Agree
Disagree
Informative
Unhelpful
Funny

I've done upgrades on two separate systems (Avigilon and Genetec) and had no issues with either one. Time to complete depends on a variety of factors, but rarely more than just a couple of minutes. The only one that took longer was a panel in Australia, and everything takes longer to get there so not that surprising.

I also keep panel firmware updated pretty regularly, so I wasn't coming from multiple revisions to get there.

Agree
Disagree
Informative: 1
Unhelpful
Funny

We did some Lenel boards today and one took a few minutes and others, which were across the country took almost an hour.

Agree
Disagree
Informative: 1
Unhelpful
Funny
Login to read this IPVM report.
Why do I need to log in?
IPVM conducts reporting, tutorials and software funded by subscriber's payments enabling us to offer the most independent, accurate and in-depth information.
Loading Related Reports