Windows 10 Update Failures Taking VMS Servers Offline

Avatar
Ethan Ace
Jul 11, 2017

Most of our machines run Windows 10, which we have few problems with except the way Windows Update works now. There is now no simple way to turn automatic updates off in Control Panel:

Which wouldn't be a problem if they worked right all the time. But over the course of a couple months, we've had 4-5 instances of machines getting stuck on updates and requiring manual intervention to finish or reboot. In the meantime, VMS services were down, so no video was recorded.

Updates can be disabled by turning off the service, but then you run the risk of missing potentially critical updates when they're released. 

How is everyone else handling Windows updates? Are we the only ones having such extreme issues?

Summary

Windows 10 updating issues are significant, according to various reports of industry professionals. Many are staying with Windows 7 for the time being.

(2)
SD
Shannon Davis
Jul 11, 2017
IPVMU Certified

Ethan see here (How to stop Windows 10 auto updates). Our problem was that windows would reboot out of the blue an update no matter what we told the system to do. We typically tell Windows to download the update then we will choose when to install. Well through the usual settings window you can only tell the system when to reboot after an update, not just download. If you have Windows Pro then through the "Local Group Policy Editor" you can customize your windows update preferences like previous versions of windows. Was hard to find at first but easy to do. See link below:

 

 

 

(1)
(2)
SC
Scott Clingan
Jul 11, 2017
IPVMU Certified

So far not many of our servers are running Windows 10.  Although I believe that is about to change.  Most of our work laptops are on it though and it can be annoying sometimes.  I wonder if Windows 10 IoT handles updates differently?

I believe with Win 10 Pro and Enterprise you can use Group policy editor to make some changes to how it updates.  But I think it would be an all or nothing approach again.  Although perhaps less drastic than turning the service off. 

 

 

SD
Shannon Davis
Jul 11, 2017
IPVMU Certified
  1. Under Options, you'll find a number of ways to configure automatic updates, including:

    • 2 - "Notify for download and notify for install."
    • 3 - "Auto download and notify for install."
    • 4 - "Auto download and schedule the install."
    • 5 - "Allow local admin to choose setting."
(4)
Avatar
Craig Mc Cluskey
Jul 11, 2017

And then you can have a non-critical "guinea pig" system to manually perform the updates and see if there are problems with it which would take a production system off line.

Avatar
Brian Rhodes
Jul 11, 2017
IPVMU Certified

Does anyone still use WSUS?

It gives a local Windows server the role of managing updates and checking for errors with them.  You do need to have a server version OS and machine to use it:

I have not used it in a long time (pre Win10), but it worked well on other Microsoft OS for server/workstation environments.

(2)
U
Undisclosed #1
Jul 11, 2017

Check this one

http://www.wsusoffline.net/

 

 

UM
Undisclosed Manufacturer #5
Jul 14, 2017

I used to use SUS & WSUS back in the day on Win XP & 2000 & Server 2003.  It was great because you had full control of which updates to allows or block, and options to reboot or not, etc.

If you have a network that you want control of the desktop and servers, definately go ahead and download and install it so you are not at the complete whim of MS.  Even if you decide to auto-install updates, you have more control of reboot options, forcing updates to deploy in a certain period for critical updates, and reporting.  The reporting in critical for large corp. or government networks where they have to show their due diligence for critical updates.

You also don't have dozens or hundreds of PCs all downloading the same updates from the internet.  If an update is failing again and again, you can see if from the reports.

Avatar
Rainald Schulte-Eppendorf
Feb 14, 2019
Santa Cruz Video Security LLC

We used it in Windows SBS 2011 environments. Sometimes there were issues with cleaning up "old update files" before running out of storage.

Avatar
Ethan Ace
Jul 13, 2017

So only a day after posting this, multiple servers hung on the latest updates again. Manual intervention required to get them rebooted, video lost in the meantime (thankfully nothing critical).

UI
Undisclosed Integrator #2
Jul 14, 2017

We don't have any VMS running on Windows 10 yet though this certainly doesn't seem to bode well...  Not looking forward to this.  Is the same issue applicable to server 2016?

Avatar
Ethan Ace
Jul 14, 2017

We have not checked Server 2016 yet, but a quick Google search doesn't make me feel too confident:

I'm asking VMS vendors now what their best practices are and will report back.

(1)
Avatar
Josh Hendricks
Jul 14, 2017
Milestone Systems

I haven't heard of many Windows Update related support cases to be honest, but I'm not on the front line of support anymore so I'm not necessarily seeing everything.

Our recommendation on Windows Updates in general has always been to ensure recommended/security updates are applied regularly. We don't have a fixed policy on how "up to date" a system should be, or provide a specific recommendation on approved updates as it is nearly impossible to test all supported products/components/drivers on every supported OS to guarantee compatibility. In reality I can recall only one Windows Update, back in ~2007, which definitively caused a failure relating to a specific driver in the Device Pack, and IIRC I believe it was a bug in the update which we reported and then quickly worked around.

Ethan's comment about disabling the Windows Update service seems like the best approach for an internet-connected Windows 10 machine which is outside of a domain environment where more controls can be placed on update policy.

Most organizations (should) have a maintenance schedule for their servers which include performing Windows Updates, so IMO it is fine to perform updates once a month during a planned maintenance window. If a high priority security update is released, you can perform an emergency update out of band.

When needed, enable the Windows Update service or install the updates manually when possible, reboot, and disable the service again.

Having said that, I saw nothing wrong with Microsoft's previous approach where you could choose to download the updates but not install them. I don't even mind a periodic nag window where you can choose to be reminded about the update. But having to go to extraordinary measures to avoid an update is a problem for critical systems.

Maybe Microsoft is taking this approach to discourage the use of Windows 10 home/professional for non-desktop applications? While support on desktop OS's should be provided, one could argue that a critical surveillance system really should be on either a server OS or on embedded/IoT versions.

(2)
(3)
JH
John Honovich
Jul 14, 2017
IPVM

Ethan, please email a few of the VMS manufacturers and ask what general guidance they have for these issues. Also, VMS manufacturers reading, please share feedback direct here if you have any recommendations.

(1)
U
Undisclosed #3
Jul 14, 2017

At the end of 2016, I was informed Exacq was shipping Win7 until their agreement with Microsoft expired, which I think was July 2017.

(2)
Avatar
Ryan Hulse
Jul 20, 2017

I am the product manager for exacqVision. 

As of today (7/20/17) we ship with Windows Embedded 7 Standard.  Our images have Windows automatic update disabled by default.  If internet access is available, our best practice is to enable automatic download of updates, but not automatically apply the updates, allowing the administrator to determine when recording downtime is acceptable. 

In August of 2017, our Windows builds will introduce Windows 10 IoT Enterprise.  Most relevant to this discussion, our images use the Long Term Service Branch (LTSB) and automatic updates are still disabled by default. 

Article on Windows 10 branches

Article on LTSB

This branch of Windows 10 is designed for business-critical server applications that should not be taken offline frequently.  It does not include certain features of Windows that are more end-user facing (Universal apps, Cortana, Windows Store for example).

When the admin is checking for updates, or if they enable automatic updates they will find that pretty much only critical fixes and security patches are available for LTSB, and these updates are coming out on a less frequent basis.

As discussed elsewhere in this discussion, if these updates require a reboot, the machine will normally do that unless that setting is changed in the registry or through group policy settings. 

(4)
U
Undisclosed #3
Jul 14, 2017

When I deployed another in-house VMS in Jan'17 I was so cranky about the desktop user issues with Win10 that I went with Win7 as the OS.  6+ months later and I'm really glad I did it.  Lately it feels like I cannot find many positive things to say about Win10 (and Excel 2016) as I'm groaning about it or hearing about it from a user about every other day...

My only suggestion would be to manually/periodically check a desktop PC for Win10 updates and if you find any, plan to run a manual update on the VMS where you can control the update and reboot.

(1)
(1)
UI
Undisclosed Integrator #4
Jul 14, 2017

Not VMS server specific, but doesn't it seem like Windows 10 is undergoing much more updates than previous versions?  Our inter office machines, Quickbooks, laptops, etc are constantly showing the "update and restart" on the power button.  Or you'll come in one morning and find out that it rebooted the night before  and is stuck at the login screen.  

Avatar
Christopher Uiterwyk
Jul 14, 2017
IPConfigure

 

(6)
(2)
(5)
U
Undisclosed #6
Jul 15, 2017

Windows 10 burned me in the middle of many presentations with auto updates. It seemed no matter what I or my IT department did, it would always try to update at most inopportune moment - just after you thought the problem was finally squashed. 

(1)
UI
Undisclosed Integrator #7
Jul 17, 2017

Unless our customers have a managed services contract with us specifically to handle their IT requirements we do not setup Automatic Updates on NVRs or Servers. Before the NVR or server goes out to the site we get all the Windows Updates applied and then install our application software as needed. Once it is in the possession of the customer, we make it very clear that they are responsible for making sure the computer is maintained and updated. Most companies, even small ones, have a part time IT person. If they have nothing, then this is where we offer our services to maintain the NVR. It becomes part of a managed services contract. We require the ability to remote into the system for this to be available...We use GoToAssist Express. On the third Tuesday/Wednesday of each month we remote into all our manage service customers and run Windows Update manually and reboot. The customer is made aware via email that during the reboot there will be a loss of video. If a major exploit or zero day vulnerability is in the wild we will patch ASAP and let the customer know of the required down time via email.

(1)
(2)
Avatar
Randall Raszick
Jul 17, 2017

It is not only VMS servers, it seems that the updates pushed out three weeks ago also disabled display drivers. This seems to have affected older computers originally shipped with Windows 7 Pro and upgraded to Win10.

(2)
UE
Undisclosed End User #8
Jul 17, 2017

I found it a bit odd that one would let these servers sit on a network that can reach Windows Updates, not very Cyber Conscious.  A good solid Patch Management and Vulnerability Program should be in place following NIST Standards.  

Times have changed, this type of forward thinking about Cyber Security Awareness is now #1 priority in my role, if you follow the news that you know that Shadow Brokers is keeping us busy patching stuff including Windows 10.

(1)
RG
Richard Galatas
Jul 19, 2017

I have avoided using Windows 10 for VMS builds specifically because of the way MS Updates are handled. While there are decent work arounds for the Pro version to regain control there are none for the home version. The only choice on that is to disable the update service.

Unfortunately now Microsoft is forcing the issue. They are supporting the newest CPU's only on Windows 10. Intel 7th generation CPU's (Kabylake) are not supported at all on Wndows 7 or 8. When you install Windows 7 on those you get the first batch of updates (about 200+) but then the second round of updates triggers an alert that the CPU is not supported and won't get any further security updates. Even worse they are dropping support early for 6th generation (Skylake) CPU's except a select list of OEM manufacturers who have agreed (been forced) to provide their own validation testing of updates on specific models. if your mfg or model is not on the list or like me you build your own computers they are now threatening to stop providing updates before the scheduled EOL date. I guess they learned their lesson from all us bitter clingers who held onto XP for dear life when they were trying to foist Vista on us. For all practical purposes you now have to start building Windows 10 computers regardless of how reluctantly. 

(4)
JT
J. Tar
Jul 29, 2017

We use Shutdown Guard, a small Windows app, that pervents auto reboots.

With this, the updates are loaded, but Shutdown Guard does not allow the system to reboot. Then on other (unguarded) systems we know when updates are arriving and schedule the reboots on all systemsvin an orderly manner.  This works for us. Hope this helps. Good luck! 

(2)
Avatar
Ethan Ace
Feb 14, 2019

Resurrecting this thread because I've had three servers install a major Windows update in the past month and make changes to the SNMP service, which kicks them off our PRTG instance, so we end up with this:

In one case, it removed the SNMP service from Windows. In the others, it changed SNMP security settings, so it wasn't accepting packets from the PRTG server.

Cool feature, Windows.

(3)
Avatar
Craig Mc Cluskey
Feb 14, 2019

As

Can you set up any switch/router between the servers and the outside world to block any traffic from the servers to the outside world (and the M$ update servers specifically if you know their IPs)?

U
Undisclosed #9
Feb 14, 2019

Was the update automatic? Who approved the update? Who is managing the system?

I have a customer, with an entire IT group outsourced and internal to manage over 500 Windows 2012 R2 deployments using Cisco UCS, they consistently manage to fail.

Every application server/vm/ftp is unique and any update, reboot or change needs to be piloted before distributed to production servers.

People are lazy, full of excuses.

The thought of using Linux invokes that a REAL administrator is watching his underlings, kingdom and domain and will assure there is no shenanigans.

If any of you guys, girls, integrators, trunk slammers, hikua experts, distributors or avigilon fan babies have YOUR SOFTWARE set it and forget it deployment on AUTO.....You should be exposed and fired for being absolutely incompetent. 

Instance 1: Customer does not like the failures on their system (Access Control, CCTV, PSIM, Integrations). So you go on site, tap the network and pen test the crap out of it.

Once your survey, report, analysis is completed you can totally exhaust the integrator excrement that original set the system up with basic understanding configurartions from a technician boot camp. Just another trunkslammer or even a LARGE corporation (with sub contractors) weaseling along until they get that project closure money. At that point it is the service departments job to fix and in comes the next round of DIY (ADT/CONVERGINT/STANLEY/JCI) noob techs to double charge for troubleshooting a system that was never commissioned in the first place, had no plan for system updates (OS and Software / Firmware).

Last, IF YOUR SYSTEM FLATLINES AFTER A WINDOWS UPDATE, ANTI-VIRUS UPDATE, FIRMWARE UPDATE....then you are not qualified on that hardware. Please stick to your b-connector splices, Tri-ed box sale, ADI hikua designs and GTFO the arena. Right now. Get to the choppa.

It seems those who cannot IT are at the bottom of the barrel bumping their gums on all the technological services they provide. Windows or Linux....know your game and stop shrugging your shoulders over system outages. KNOW YOUR GAME OR GTFO. YES YOU.

(1)
(2)
(2)
(1)
Avatar
Ethan Ace
Feb 14, 2019

Doesn’t anyone else think it’s a little crazy that we are all so used to stories of updates breaking things that we now blame the victim when things break? Heaven forbid I run a Windows update. I deserved to have my SNMP settings changed for no apparent reason! 

(4)
UE
Undisclosed End User #12
Feb 14, 2019

I like UD#9's post, while a bit harsh it is spot on.  Own you're domain, while its likely many of you may not be dealing with LARGE ENTERPRISES with tens of thousands of end point devices that need patching you can still employ some of the same best practices.  The lack of understanding is only ignorance.

You need the proper environment with DEV, UAT and BCP so you don't have mishaps when pushing any update not just MS Updates.

DEV - test the obvious, does it brick stuff right out the gate from an OS and stack prospective.

UAT - Let the end user test it out and do their thing to see what's fixed and what's not.

BCP - what do you fail back to if something goes wrong even after properly vetting updates?

Lastly, READ THE RELEASE NOTES.  Does anyone really know what they are allowing to be updated? 

For MS admins/users I hope you follow Patch Tuesday's and read the CVE's to know what you are exposed to.....this month there are 69 vulnerabilities (20 critical) that are patched...read them......  https://blog.talosintelligence.com/2019/02/microsoft-patch-tuesday-february-2019.html 

(1)
U
Undisclosed #10
Feb 14, 2019

Ethan, thanks for the heads up.

Can you post the OS version(s) affected? and, if you happen to find it, the root cause (e.g., rollup, out-of band patch…)?

UI
Undisclosed Integrator #11
Feb 14, 2019

All our machines run on a closed network. Basically...we are not doing updated. We want to...but are now like two years behind.

That and I am personally concerned Honeywell is not quick to approve new updates, so...even if we did update, what the odds it would break our VMS

(2)
New discussion

Ask questions and get answers to your physical security questions from IPVM team members and fellow subscribers.

Newest discussions