Subscriber Discussion

Should I Disable Windows Updates On NVR?

Now that more and more Windows based NVR's are out in the wild, I'm curious as to what other technicians are doing with Windows updates on their systems. With most of our builds, we update the unit during initial OS and VMS installation and than turn off windows updates to avoid any prompts etc. being displayed to the end user. I know that we can schedule updates to take place at a given time (say 4am weekly), but I am worried that there may be a problem with updates and the machine will not come back online, and thus no longer be recording footage. How do you guys handle this? And what are some positives/negatives in either situation.



From my side of thiings in local government and a few small private jobs, I have found it best to disable the auto update in windows. The majority of client computers used to access the VMS are usually dedicated to the camera system and really don't need to update. If left alone, Windows will be auto updating quite often and the kiosk computer I set up last week ended up in a call the very next day due to a windows update that I forgot to disable. If the end users are familiar with the system or computers in general it's not an issue but most just call for help.

Windows Updates should always be disable on any server. We have had automatic updates cause systems to crash and hang on multiple occasions. The other thing you should be carefully about is virus software and backups. These pieces of software can dramatically reduce performance of a server and if you scan or backup a live database of a VMS you could cause that VMS system to crash.

Mark, Duncan, thanks, good points!

Can any VMS manufacturer comment on what their standard recommendations are for updating Windows OS? Do you periodically send out information on what can and should not be done?

I know larger end user IT departments will do their own testing on machines running VMS to see if the update breaks their recorder before considering a roll out.

This is from the Exacq Knoweldge Base.

"Are Microsoft Windows Updates Enabled on exacqVision Systems?

By default, Microsoft's automatic Windows updates are not enabled on exacqVision systems.

There are no known exacqVision compatibility issues with any currently available Windows updates. However, if you decide to apply Windows updates your exacqVision system, Exacq Technologies suggests that you apply only the critical updates. Do not apply optional updates."

I agree it shouldn't be allowed to be auto but there should be a process in place to patch the OS. That's just stuff that needs to be done in any modern computing and one that lots of people (particularly in the security realm ironically) don't do becuase of a set it and forget it approach which should not be taken. We patch our system during set and announced maintenance windows. Some software manufacturers are pickier than other about "supporting" patches. I personally haven't seen major patch issues for years now at least with our stuff. I still do know of a few manufactures that want to approve patches before you implement but those have grown few and far between now a days. I don't care of the systems are one highly secured networks, they need to be kept up to date.

Note however:I'm speaking as part of a large IT shop that also handles all video surveillance for our organization. A little easier situation with regard to this topic than an integrator with hundreds of separate clients systems they may or may not fully manage after implementation.

Be careful about "Critical Updates", too, if you do them. When I had a computer store we had an isuse where Microsoft catagorized their version of an nVidia video driver as a critical update, replacing nVidia's driver with their own. Once working and stable computers because unstable and we had a flood of customer computers coming in for service that turned out to be Microsoft's critical driver update. Uninstall the Microsoft version, reinstall the nVidia version and the systems were working fine again.

If you have a closed system architecture (as we do) and the system is stable, I've found that there is no reason to update Windows beyond whatever is required by the VMS and any other installed programs. Regular updates are dangerous themselves. I can't count the number of failed Windows updates I've experienced on my home computers and sometimes the failures cause other systems to fail.

For example, I use Firefox on this computer - not because it is any better than IE, but because a Windows Update installed in 2011 caused functional problems with IE-8 and I've never been able to get it working properly; even after multiple System Restores, uninstalling and re-installing multiple versions of IE, etc. Yes, I could do a "Repair Install" but that itself can cause issues.

Carl, to play devil's advocate, how many systems today are truly closed? Even if there is a dedicated network connecting IP cameras to recorder, frequently (usually?) there is a connection for the recorder to the corporate LAN / network to enable remote monitoring.

Yeah, well that's their problem, wink wink! Casinos often have closed domains; especially Tribal Casinos.

I know that is not always acceptable policy and once any computer is connected to the outside world, it opens up a large can of worms. Firewalls and Ironports can only do so much, especially given the huge number of security flaws being discovered almost daily.

Still, it's caveat emptor when it comes to installing updates. Best practice would be to install each update separately and test the system thoroughly after each update, crossing one's fingers that any conflict can be cured by a System Restore. Automatic Updates are a definite "no-no".

If you you a properly designed VMS thats runs the server as a service (e.g. Milestone, Genetec, Luxriot, HD Witness, etc...) and lock the server down by running everything else and logging in as a limited/standard user and having a well thoughout group policy applied, then viruses/malware will not be able to run in 95%+ of cases. Only dedicated manned hacker exploits will be able to breakthrough, which is in the rare case that an attacker is deliberately targeting the system. In these cases the management system of the PoE switch and router are possibly easier targets.

An ISP we work with has a very transparent maintenance process and every month we are notified of router/switch firmware/hardware upgrades.

Here's a freebie for my fellow IPVM'ers (though some of you already know).

Two primary ways viruses will infect a system; access a nefarious website with a web browser, and "worms" that come from other virus infected computers on a network that break through the login requirements. One of the things you can do for the worms is go into the properties of the network adapter and uncheck the "File and Printer Sharing for Microsoft Networks" box. This denies RPC (remote procedure calls) access to the server. (This also means you won't able able to access shared printers or folders on the vms server, but then why would you.) You can also uncheck "Client for Microsoft Networks" so if the server does become infected, it severaly mitgates the server's ability to pass on worms.

Patching of the OS does not necessarily directly/always correspond to the prevention of viruses and worms. There are a several potential security things that patches can address. Also there are worms, or to be more generic: exploits, of all kinds to take advantage of vulnerabilities in all walks of software, not just RPC. In fact, I would say in today’s IT Security climate that it is lower on the list of things are commonly being exploited. Network/IT Security best practices, some good items have been mentioned here could (and does) take up volumes but is always interesting discussion. (To me anyway) Security in all forms should be multifaceted and layered. Someone else mentioned vulnerabilities on the network gear as well which is a good point. It all matters.

Warning: I'm making a generalization here but it's one I've unfortunately found to be true too often. If it doesn’t apply to you, just be proud of yourself. :) I've always found it interesting if not ironic that lots of physical security folks don't seem to care about, and some even do things that exacerbate, IT security issues. I feel that some things like this are still the cause of lots of head butting between IT and physical security and why in many cases IT ?is having to /choosing to? take over certain applicable aspects of physical security.

Physical security isn't the only market that people want to play IT but don’t want to be bothered with a lot of the things that go with that world. The AV market nowadays is pretty bad with that as well. Of course I’ve also seen plenty of sysadmins, network, PC support people, etc. that never seems to think of security either but hopefully those are slowly going the way of the dodo. Of course, YMMV. My view point is different from that of most integrators and of course is based on the assumption that there are good "IT resources" for all your clients which I know isn't true.

In our organization, we manage Windows Critical Updates with WSUS. No updates are installed automatically, rather pushed from our server on a quarterly basis. We have a test group, which contains a handful on NVR's and DVR's (Windows XP and Server 2008), that receives the updates one week prior to the rollout. This allows to identify any potential issues with updates.

Our VMS is Pelco and thus far, we have not had any issues with critical updates negatively impacting a system. I did have to disable any SQL updates, as I ran into a couple of odd issues whereby the server would hang while SQL updates were being applied.

Thank you for all your comments guys! Great community of technicians and integrators here, very nice to have so much insight from so many different people.

Next question is what about VMS manufacturer issued patches and upgrades? Do you install them? When, how? What if they address security vulnerabilities?

This is something that should be a written policy. You can a) never patch; b) patch after testing - define the test in the procedures; c) patch the production system, and back out if it fails; d) patch production, and if it is a security vulnerability patchm, and the system fails, then be down until a new patch is out; e) something else. You may have multiple production servers, and can do one as the test.

Regardless, put the policy and procedure in writing and follow it.

How do you determine which option? Which is a higher priority issue? Continuous video monitoring while risking a patched problem, or risking an update which may stop your continuous video monitoring? You may have criminal activity to monitor, you may have regulatory issues with patching or non-patching.

Same sort of thing may apply to your client workstations. Do they get patches, updates, upgrades, etc? Again, written policy and procedures are critical. I've seen desktops where the user couldn't install anything for security reasons, but then couldn't install Java, Flash, or Firefox updates, and that opened the door to vulnerabilities, leading to a security hole.

Think about which scenario you would rather explain to your boss. "I did not run Windows Update, and we got a piece of malware that shut down the whole video system, that would not have gotten in if we updated." or "I installed the latest Microsoft-issued Windows updates, and one of them didn't work. I'll back out the update, though we won't be able to monitor the mob/CBRNE/gambling activity for 3-5 hours."

Next question is do you run security vulnerability tools, such as MBSA, Nessus, Qualys, etc? Do you run it privileged or not? Do you have a qualified system admin, such as a MCSA or MCITP managing the server? Do you use a server OS, or desktop OS? What if you have a Linux OS?

Do you have a BC/DR plan? Do you have a system partition backup? Do you backup the video files? Do you use RAID?

There may be more policy and procedure issues than any of us are aware of...

Never abandon maintaining systems. If you don't apply updates you're doing poor maintenance. Try expaining that the next time someone tries to take your video into court. Closed systems aren't. Video recording systems are cyber attack targets in and of themselves. Also, vendors who advocate avoiding windows update tend to have other (scary) technology flaws. Sure, automatic updates are sometimes clunky. That's why you're supposed to use maintenance processes on closed-network sytems.

At least there are now two of us in here now Rodney. :)

We have our equipment on a secure vlan which requires the user to have access granted to the specific vlan. If you are off campus you would then need to VPN in which also requires the access to the vlan be added to your profile. We also push updates to windows with tools that allow us to schedule the updates and filter them. The windows update feature is pointing to our internal server to look for updates by means of a simple registry key. This allows us to not only poll the devices to see their status but also keeps us from getting calls in the middle of the night due to an update hosing the system.

Reading all of this, I feel like I've gone back to 1998. Windows updates and antivirus on a security appliance? Classic.

Really? 1998? Let's hope not, Windows 98 'First Edition' was a horrific platform, and Windows NT 4 didn't do enough.

Rodney said the key thing here: Closed systems aren't. There are enough vectors for security issues on both closed and open systems that you will not account for them all. This is why we have a concept called defense-in-depth

As for Michael's post, the answer is always apropos to the environment. You have a support team consisting of 1? Do you have the resources to have test and production? Then you may not be testing your patches on test equipment- maybe you will use a non-critical system to verify operation before pushing out.

Patch. Test. Have written processes. Run a closed network. Use network segmentation. Implement firewalls. Default to deny-all policies. Users do not run in privledged mode. The list goes on.

And to answer the original question... disable automatic installation of updates, but have a scheduled, manual process in place to continue these updates. And don't stop at O/S patching- think drivers, firmware, and software upgrades.

Vulnerabilities exist at every layer. You have to decide the risk in your environment, the cost associated to mitigate that risk, and implement.

One solution we are trying out for a customer right now:

Exacq NVR, two client PCs and cameras are all on a dedicated network, with no connection to the outside world save via VPN that only I can connect to (and only when remote troubleshooting becomes necessary). NVR has updates turned off (it's Linux, but not important here).

We have Exacq Web Server installed on a separate machine that bridges the dedicated network to the outside world. This means remote access is browser only, but it also means we can aggressively auto-update the Web Server machine as we are only risking loss of remote connectivity - not interruption of recording.

Obviously, this isn't going to fly with the majority of users. For most others we just lock down access as best we can and manually apply updates on a schedule (how often depends on customers budget). Performing manual updates via ssh on Linux is relatively painless, but even more important is the "out-of-band management" we include on all our servers. This allows us to perform updates remotely, while helping to mitigate the OP's original concern that the system might hang or otherwise fail to boot after an update. IPMI is basically as good as being there. I even use it to install the OS (Windows and Linux both). The servers leaving our shop now never have and never will be connected to keyboard, mouse, or monitor.

Our preference is Supermicro motherboards with IPMI 2.0.