Subscriber Discussion

No Brainer Idea For Manufacturers To Stay Current With Cyber Security: Automatic Firmware Upgrades

Avatar
Sean Nelson
Sep 28, 2017
Nelly's Security

Why cant manufacturers figure this out?

Every other IoT device does this. Computers do this. Cell phones do this. Sonos does this. "you name it" does this. But for some reason, Surveillance Manufacturers have not figured this out yet.

AUTOMATIC FIRMWARE UPGRADES!! 

We need more than just a button to check to see if their is new firmware, that isnt effective either. 
We need a way for firmware upgrades to either be pushed to the devices or atleast give us a notifcation on our monitors, cell phones, web interfaces, warning us to upgrade our firmwares due to new security threats.

As displayed by the recent Dahua example:

TELLING PEOPLE TO MANUALLY UPGRADE THEIR FIRMWARE DOES NOT WORK EFFECTIVELY!!

The problem is, their is no real easy way to effectively and feasibly communicate this information to all the users in the world. Alot of people dont read their emails. Its not feasible for distrubutors, resellers and dealers to call all of their customers to tell them to upgrade their firmware either. Then you get the people who do read their emails but dont take action, or dont really know how to take action. 

 

We live in a world where new Cyber Threats arise all the time.  Atleast this would give your customers a way to effectively stay on top of these threats. Not to mention this would be much less exhaustive on all parties involved.

(1)
JH
John Honovich
Sep 28, 2017
IPVM

Sean, I definitely agree about the benefits.

It is worth noting risks in that process:

  • If something goes wrong with the update process, cameras can malfunction, lose settings or worse be bricked
  • This requires trusting the vendor / provider doing the updates not to introduce vulnerabilities or misuse their access inside one's network.

I am not arguing against it overall but wanted to note the concerns / other side.

(3)
Avatar
Sean Nelson
Sep 28, 2017
Nelly's Security

I understand their are concerns, but i feel the risk to not have it is much greater than having it. 

My suggestion would be to make an optional checkbox to receive the upgrades, that way if some users dont want them, its their choice.

Then send the user a push notification of some sort to notify them their is a new firmware with clear instructions on how to properly receive the upgrade.

 

 

(1)
Avatar
Michael Budalich
Sep 29, 2017
Genetec

I agree with you Sean. This would definitely help especially in the case of Dahua and Hik who have tons of models in the field that have old vulnerable firmware versions. Even if a couple cameras were bricked in the process the benefits auto updating to the latest firmware are much higher. Maybe they make it an option that you can select in the browser so that the integrator can choose auto update or not?

Avatar
Sean Nelson
Sep 29, 2017
Nelly's Security

exactly, i would much rather deal with the occasional bricked phone call here and there as opposed to weeks full of "unhacking DVR" calls

Avatar
John Bazyk
Sep 29, 2017
Command Corporation • IPVMU Certified

The way our alarm panels upgrade is this. We send the command via the dealer cloud portal. The cloud sends the firmware to the panel. The alarm panel operation is not stopped during this time. The panel confirms that the firmware is good and was recieved completly. Then it actually upgrades the panels firmware. We have never had a firmware update screw up an alarm panel since this platform came online over a year ago. 

(1)
(1)
U
Undisclosed #2
Sep 29, 2017

How secure is the portal? I suppose it's HTTPS everywhere, at least?

Avatar
John Bazyk
Sep 29, 2017
Command Corporation • IPVMU Certified

The connection from the cloud to the device uses a 2048RSA key and 256AES VPN.

U
Undisclosed #2
Sep 29, 2017

Avatar
John Bazyk
Sep 29, 2017
Command Corporation • IPVMU Certified

I can’t tell if this is a good response or bad. 😂😂

U
Undisclosed #2
Sep 29, 2017

It's good 😅

U
Undisclosed #2
Sep 29, 2017

That something goes wrong is an accepted risk in the process, as long as it can be managed in a good way, ie. when you're about to update cameras/whatever in quantity, you could do a batch of maybe 20 at most with a single click, then make sure everything works, then continue or fix any issues if they arise. If typically everything works out perfect, while not ideal, this wouldn't be too much administrative overhead for someone who wants to have control over the applied updates.

The admin and users of a particular system can probably work out a patching schedule, but there perhaps should be some easy guidelines available on how to implement those things, from a small store to a multi-site enterprise environment.

If only the tools and assistance to do the actual update reliably were there, much of the other stuff concerns the procedures in place and the support available in the VMS/whatever. Of course one couldn't really know much about the content of the updates or whether they break things or not, but if it's simple to handle them like "./update_cam 10.1.1.55 firmware_2017_09_29.bin" one could in some cases put this up in Jenkins or so for acceptance testing with a real camera, to test certain important things from the end-user perspective, like that the camera can be accessed with the configured IP, image is recorded, certain settings are enabled and so on.

This is an ongoing process for sure.

UD
Undisclosed Distributor #1
Sep 28, 2017

You would trust companies like hik and dahua to randomly send out any trash firmware updates that they wish to your customers???  These companies have a long established history of bungling these things on a regular basis.  There is also the concern of having all of these DVR/NVR/IP camera devices out there "calling home" on a regular basis to god knows where, asking for updates.  Would you also have this "call home" process provide these companies with direct contact information for your customers?  You would have to be pretty naive to not believe that they would make a disaster out of this and also use it steal more your customers.

If done properly this could be a nice idea, but I think we know that neither of these companies could do this without making it a complete nightmare.

(1)
MM
Michael Miller
Sep 28, 2017

So how is this automatically upgrade going to work when all the cameras are on a segmented network?

This is one of the reasons I love Avigilon.  Every time we upgrade the server software all Avigilon cameras automatically get firmware upgrades.  Fast and simple.

(2)
(1)
Avatar
Sean Nelson
Sep 28, 2017
Nelly's Security

You can push firmwares thru to the cameras via the NVR. So any camera that is connected to the NVR can receive upgrades as well.

MM
Michael Miller
Sep 28, 2017

What happens when your not using Hik NVRs?

Avatar
Sean Nelson
Sep 28, 2017
Nelly's Security

I dont get your question.

MM
Michael Miller
Sep 28, 2017

If you are using Hikvision cameras on a 3rd party VMS and the cameras are on a segmented network.  How are the automatic upgrades going to work?

Avatar
Sean Nelson
Sep 28, 2017
Nelly's Security

I'm more concerned about devices that are open to the internet.

If the cameras are not port forwarded and on a different segmented network, the risk is quite low.

(2)
(1)
U
Undisclosed #7
Oct 04, 2017
IPVMU Certified

What happens when your not using Hik NVR

What happens with Avigilon cameras when you're not using Avigilon ACC?

(1)
U
Undisclosed #2
Sep 29, 2017

Yes, it's convenient with <particular_brand> to update the firmware of many cameras at once.

MM
Michael Miller
Sep 29, 2017

There are a lot of management tools that can upgrade may cameras at once but I don't know of other VMS solutions that include the latest firmware in the server software upgrade package and then push the firmware to the cameras automatically without having to use a management tool. 

(2)
U
Undisclosed #2
Sep 29, 2017

Yeah, it's hard to do well without some common interface. Not to say it couldn't be done the traditional way, by implementing support for various manufacturer's firmware upgrade feature even if there were just minor differences and making it nice and manageable, but that's a lot of work. And you have to keep up with the changes others make.

If an open source and free VMS software existed that would compare with the enterprise features of the current market leaders (or hopefully go much beyond them), it would be very nice indeed...

Avatar
Brian Karas
Sep 28, 2017
IPVM

I think this is a good suggestion, but you have to step back a bit first.

Automated firmware updates are not trivial, and require a lot of additional QA testing (if you want to do it properly) to ensure there will be no scenarios where devices hang on reboot or run into other problems.

Another issue relative to the devices you mentioned as examples is that it is hard to know when it is 'safe' to reboot someone's camera or recorder. What if the reboot happens just as a crime occurs and video is never captured? Sure, it sounds like an edge-case, but factor in 10's of millions of units, and odds are there is not a universal ideal time to do a reboot (e.g.: at noon, or 1AM, or Saturday evening, whatever). This could open manufacturers up to other liabilities.

But I think the biggest issue is that automated firmware upgrades require a certain amount of overall additional diligence and forethought in engineering. The kind of diligence and forethought in process that tends to eliminate the kinds of low-complexity exploits we have seen from the two manufacturers who could most benefit from this.

In other words, if the companies that most needed automated updates could build an automated update system, they might not actually need automated updates.

I do not really want a manufacturer pushing random updates down to my devices to fix vulnerabilities. I want them to write software that does not have gaping security issues. This is not a difficult or extreme request. In fact, large companies with thousands of engineers should be able to release firmware that does not require frequent patches.

(2)
(1)
Avatar
Sean Nelson
Sep 28, 2017
Nelly's Security

I think thats overthinking it. All you do is send a push notification to the end user asking if they want to upgrade the firmware or not. If they say yes, then they receive the upgrade. Im not a programmer by any means so I dont know the complexitities but what I do know is that there are already tons of devices that do this and have been doing it for years.

(1)
(1)
U
Undisclosed #2
Sep 28, 2017

The complexities arise mostly from varying usage scenarios as Brian detailed, but if a well managed update system was in place, it could be a reasonable default setting, and for enterprise customers there could be a coordinated way to manage those updates. If the camera is not internet connected, it's not going to get the updates anyway, but I think it probably should be one of the first settings when you first set up the camera, ie. set the admin password and choose if you wish to have it updated automatically or otherwise communicate somewhere on its own.

Most of the time you'd want to have full control over the updates and how and when they're applied, but those who click-click through the settings anyway might be safer if they enable it as default. Or then again, it might just mean that the next hack comes as an update.

(1)
Avatar
Brian Karas
Sep 28, 2017
IPVM

All you do is send a push notification to the end user asking if they want to upgrade the firmware or not.

"All you do..." is greatly summarizing the complexities involved in this.

Can the automatic update system itself be compromised? Those old enough to remember sending pop-up message to remote PC's using "NET SEND" could understand how a poorly implemented alert system could be compromised to trick a user into downloading a malicious update instead of valid firmware.

Implementing a reliable auto-update firmware checking/distribution mechanism tends to require the kinds of skills and engineering that naturally eliminate low-complexity firmware exploits in the first place.

For some cases it could be beneficial. But right now, I would not trust Dahua or Hikvision to push automatic updates, or deploy any kind of a function that puts dynamic messages from the internet in front of users.

Concentrate on fixing the issues that already exist in firmware before you layer a new set of unproven deployment complexities on top of it. The fact that other companies have gotten automatic updates (mostly?) right does not mean it is by any means easy to do properly, or indicate this could be a simple fix to dealing with ongoing vulnerabilities.

(1)
Avatar
Sean Nelson
Sep 28, 2017
Nelly's Security

Like I said im not a programmer so I have no idea of the complexities involved but this has existed for a long time from manufacturers that are smaller and have less money than some of these big Surveillance Manufacturers, so I know it can be done. Is it difficult? Maybe. Is it impossible? Obviously not. These manufacturers have the money to figure out how to do this properly. Its time to wake up and get with the rest of the world and do it.

I understand your point of view but I think your overthinking it and these manufacturers probably are thinking the same thoughts which is why perhaps they have not implemented this. I also agree they should make more secure firmwares in the first place but what is secure now may not be secure in a year from now, thus the need to update. When you have millions of devices all over the world, alot of which are not professionaly installed with cyber security best practices in mind, this seems like a no brainer. 

What I can tell you is informing people to upgrade their firmwares is obviously not an effective way to have people keep their firmwares up to date. I do feel that this is  no-brainer approach in the right direction. 

(1)
(1)
UI
Undisclosed Integrator #5
Sep 29, 2017

Like Microsoft or Apple?

Avatar
Brian Karas
Sep 29, 2017
IPVM

Are you comparing Hikvision and Dahua to Microsoft and Apple?

UI
Undisclosed Integrator #5
Sep 29, 2017

Of course not.  HIK writes much more reliable software! (Just kidding)

You had previously stated something to the effect of “if companies could develop an automated software update system they likely wouldnt need it.”

I just painfully updated to iOS 11.

 

Avatar
Brian Karas
Sep 29, 2017
IPVM

What I was saying is that if they were developing an automated update process specifically to deal with ongoing vulnerabilities, they would likely be able to just write better software with no vulnerabilities in the first place.

This is because it is not easy to do an automated software update properly (though of course several companies make it look easy, which is a testament to their engineering).

If Hikvision and Dahua cannot find these basic vulnerabilities through their own internal testing and development efforts, I would not trust them to develop and deploy a system to push updates to camera/recorders automatically, something more complex to implement.

(1)
UI
Undisclosed Integrator #5
Sep 29, 2017

I agree fully.  

Which now starts the question of “did they miss it or was it there intentionally”

 

U
Undisclosed #3
Sep 28, 2017

I would love this, but I also know it is much more complex than it sounds if you want to do it "right" and do it safely.

First, the phrase "automatic firmware upgrades" gives me heartburn. I do not want public safety equipment upgrading it's own software based on some external signal. What happens if the auto-update functionality is hacked. Maybe the manufacturer forgets to renew a domain registration, DNS is taken over, and firmware is pushed out by a malicious 3rd party?

Still, I would love it if I happened to access the web interface and an alert was shown about an available firmware update. My phone does this, my router does this, my NAS, etc.

But how often in a commissioned surveillance system would anyone be logging into the web interface of individual hardware components? Not often. So what if, in the case of a camera, it overlaid a message at the top or bottom of the feed that a firmware update was available?

It may be an annoyance to the user, but you can be sure the administrator for the system will hear about it. Maybe you could delay the overlay message for a couple of weeks and only do an overlay if the firmware update is not applied sooner, or only if the update is a security update?

For cameras without internet access, I would like to see a day where ONVIF-enabled cameras could receive firmware updates through the VMS. I imagine it would work like this...

ONVIF provides an interface for retrieving the individual hardware manufacturers firmware server address

VMS polls firmware server for the latest firmware version number once per day

VMS sees there is a newer firmware version available, downloads it, and notifies the administrator of the availability

VMS Administrator pushes new firmware to all applicable ONVIF devices via another interface which allows to upload a binary firmware file to the hardware

The hardware then applies the firmware update the same way it would if you had uploaded the firmware from the web interface

 

* I just checked the ONVIF specification and it provides a firmware update mechanism. So all that is needed to accomplish this is a standardized ONVIF interface for getting the hardware manufacturers firmware server address and a standardized method for manufacturers to host and deliver firmware.

(1)
(2)
U
Undisclosed #2
Sep 28, 2017

A compatible way to update firmware from the VMS would be nice indeed, it's a pain when you have a wide range of models and can't manage all of the updates in one place, instead you can perhaps handle a bunch of them with one software and another with something else, and some you just have to resort to manually update via the web interface. And some you can't update at all :(

(1)
CM
Corey McCormick
Sep 29, 2017

ONVIF is not as well implemented as I think we would all like.  I think part of the reason is the complexity and the lack of proprietary lock-in.  Once everything is plug and play compatible, it becomes often a race to the commodity bottom.  Suppliers do not like creating a system that may hasten their demise.  (Uber drivers should take note...)

Supporting ONVIF fully as well as your proprietary methods used for brand loyalty support, is labor intensive from a development point of view.  If your "in" into the market is disruptive pricing, then this expense gets in the way of your marketing plan.

Look at Ubiquiti networks, who does just that.  At times they have great products with horribly buggy software that they let the "community" beta test and work out the kinks in for several months before most folks should depend on it.  They have no real SNMP support (the networking semi-equivalent to ONVIF) for most of their products, but still they are successful.  Partly because of their low price points and partly because unlike similarly priced low-end products with more robust features, once they are stable, they tend to be reliable long-term and prevent truck rolls.  They eschew standard features and industry compatibility for proprietary solutions.  Listen to their quarterly call-ins where they boast about their customer lock-in as one key to their success.  Note that as their software complexity goes up and they join the wider community of standards, their prices go up as well.  There is no such thing as a free lunch.

Long term there needs to be enough margin to support the upgrades of the older buggy software as defects are discovered and that makes the ultra-lowcost solutions harder to build/purchase.  You get what you pay for?  Sometimes.  Some high priced vendors take lots of money up front and still do a very poor job of support and security.  Where is their excuse as they are the ones generally complaining loudly about their low costs competition bad practices?  A look in their mirrors could be enlightening.

How hard does that low-cost vendor really want to work for a $70 IP camera?  How many purchasers will be able to see an ONVIF certification stamp and decide that means all of them are equal so their job is to pick the lowest priced option.

I think underestimating how truly difficult and labor intensive software development and support is, creates a gap in expectation vs. reality.  We can write books on the subject, but in summary I think it can be like trying to nail Jello to the wall in a hurricane on a hot day with thumb tacks much of the time.  Lots of pressures fighting against you and little support from anyone.

Long term I agree that managed updates are the only reliable way for larger installations to succeed.  You will need a human to read the release notes, understand the business requirements, satisfy the logistical and political environment needs, etc...   

For mom and pop installations the automatic update would help, but should not be the only option.  These issues put more pressure to migrate to SAAS again since many end users do not want to deal with these sorts of details.

 

(1)
(1)
U
Undisclosed #4
Sep 28, 2017

A software needs to be created that can quarantine any device connected to a network port, perform firmware updates and then test the device. After the updated device is completed it is placed back in to production. This is done 1:1 basis with reports, automatic emails and journal. If a firmware update is faulty or hostile the updates are stopped and human interaction is needed.

U
Undisclosed #3
Sep 28, 2017

How effective would a piece of software be in recognizing whether a firmware update was hostile? Even antivirus has difficulty recognizing malicious files until a good signature can be developed for that particular threat. Considering physec hardware threats have considerably less volume, I'm not sure any application could be effective in automatically detecting threats - especially if the threat was designed to wait a few weeks before contacting a botnet for example.

Probably the best protection against this is to use PKCS to sign the firmware so the hardware will only apply firmware which was signed by the organization's (hopefully) secure private key. Ideally there would be multiple layers of protection. There was a thread a while back about kernel support for executing only properly signed code as well for example.

(1)
CM
Corey McCormick
Sep 29, 2017

Multiple layers is still the only reliable method, I agree.  Humans and their systems are fallible and always will be.  Only by layering them can we mask the individual flaws and provide a robust environment.

Look at your running shoes.  The hard synthetic rubber sole protects feet from sharp object penetration.  The EVA foam layer protects feet from blunt force trauma we call walking/running (controlled falling?  :-)).  The interior arch support helps distribute the loads to the correct parts of the feet where it can be best handled.  The liner helps prevent friction/blister damage, but it really needs the sock to act as the proper mating surface.  None of these would rally work without the others.

Simple solutions for complex problems rarely exist no matter how simple we want them to be.

(1)
UM
Undisclosed Manufacturer #6
Sep 29, 2017

Automatic upgrades could corrupt an integration if the camera updates to an unsupported version on a VMS.  A VMS or NVR that gets upgraded could then not work with an access control or other platform that is integrated....

 

It is a good idea, but difficult to implement unless using a single manufacturer end to end.

I have worked with NVRs that display a message to update, if they are connected to the internet. 

Avatar
Joseph Henry
Oct 04, 2017

As anyone seen this within the newest Dahua Firmwares? Is it going to notify me via email, just check and do nothing, or automatically update it? No information on dahuawiki.com and nothing on IPVM.

 

Dahua Online Upgrade

(1)
Avatar
Ethan Ace
Oct 04, 2017

Thanks for posting this. I hadn't seen it yet. We'll see if we can get some information on what the auto-check actually does. What was that firmware running on?

UM
Undisclosed Manufacturer #8
Oct 04, 2017

Yes in most of there newest firmware's they have auto checking of updates. 

I haven't seen a machine do a update from a pushed update yet but they are for sure rolling it out. 

Uniview has had cloud updating for a long time and it works well.

Avatar
Sean Nelson
Oct 04, 2017
Nelly's Security

Thats great! Hopefully it actually works! curious how one receives notification to receive update. What they really need is to be shown on their phone or whenevet they login to their DVR thru any method

(1)
New discussion

Ask questions and get answers to your physical security questions from IPVM team members and fellow subscribers.

Newest discussions