Intel Meltdown / Spectre Patch Tested on Avigilon, Exacq and Milestone VMSes

By IPVM Team, Published on Jan 23, 2018

Significant concern exists about the impact on VMS servers and video surveillance systems of patching the Meltdown and Spectre flaws. Generally, reports have claimed a significant increase in CPU loads, though the amounts have varied depending on the source and services being used. This could be very problematic for VMS servers as it risks problems in recording, watching or playing back video.

Meltdown / Spectre Summary

Meltdown is a flaw in protections allowing unauthorized applications to access OS kernel protected memory. Spectre allows unauthorized applications to trick other programs into divulging private data. For technical details on these, see The Meltdown and Spectre CPU Bugs, Explained [link no longer available] or various other online tech sources.

How To Patch

Users may see if their Windows machine is vulnerable using PowerShell, following the steps in this guide.

If so, in order to avoid both of these vulnerabilities, users must install two updates:

Windows Update KB4056892: This update should have been included in regular Windows Updates as of early January. This fixes the Meltdown vulnerability.
BIOS update: To fix the Spectre vulnerability, users must update their system BIOS, provided by their hardware manufacturer.

However, many devices have not yet received a BIOS update. Out of four different hardware manufacturers we checked (Acer, Asus, HP, and Intel NUCs), only Intel had released a BIOS update for our machines.

Tests Performed

IPVM measured CPU load of server and client processes prior to applying Windows and BIOS updates, and again after updating, using two configurations:

Recording only, 25 cameras, ~150 Mb/s
Recording and viewing, 10 cameras, ~30 Mb/s

Test Machine Specs

IPVM tested on three separate machines, all with the same hardware configuration, as follows:

Operating System: Windows 10 Pro 64-bit (10.0, Build 16299) (16299.rs3_release.170928-1534)
BIOS: RYBDWi35.86A.0350.2015.0812.1722 (pre-update)/RYBDWi35.86A.0368.2017.1220.0950 (post-update)
Processor: Intel(R) Core(TM) i7-5557U CPU @ 3.10GHz (4 CPUs), ~3.1GHz
Memory: 8192MB RAM

Note that these machines are fairly high spec, with relatively new processors (1-2 generations old). Older generation processors and lower spec machines are likely to see a greater impact.

Inside this report, we examine the results achieved across Avigilon, Exacq and Milestone plus our recommendations.

Test *******

** *** *****, ******** system **** ** ***** the ******* ************* *** a *********** ****** ** CPU ****, ********** *** load ** ********* ******** by ** **** **-**%. ****** on ****** **** **** viewing **** ** ******** video *** **** **** significant, nearly ******** *** **** ** low ****** ***** ******* using ******** *** ***** (an ******** ** **-**% load). ******** *******' ******** update ** ***** ******** alone had * ********** ******, increasing **** **** **** 1%.

****** *******, ** *** no *** ******* **** these *******/******** *******, **** as ******* ******, **** loading *****, ******** ** video, ** ***** *** malfunctions ***** *** ***** when **********. *******, **** that *** ******* *** load **** ** **** testing *** ***** ***** 50% ***********. ***** **** higher ********** ******* ** lower **** ******* *** see ****** ** ***** camera ******/***********.

***************

******* ** *** *********** impact ** **** ******* on *** ****, ** strongly ********* **** ***** of ****** **********/****** ****** count ******* ****** ***** system *** ******** ***** to ********. ***** ********* running ** **-**% **** are ****** ** ******** to **-***%, **** *** potential *** ****** **** higher.

***** **** *** ****** *****/********** systems *** ****** ** see ****** *********** ********** unless **** ******* ******* on *** ****** *******.

Intel ***** ******** ******* - ****** *****

******* *** **** ******* have ****** ****** ****** on ******* ******** (****: IPVM *** *** ********** this, *** ** *** be ******** *********). ***** released * ********* ** January **** ******** *** ***** ******** BIOS ********** **** ****** **** updated ******* ****. ******* indications *** **** *** updated ******* **** **** roughly ******* *********** *******, and ***** ** ************ partners ** ******** ** test **** *** ******* patches ** *** **** data ** *********** *******.

Biggest ******: **** ******

** ***** ** ***** repair *** ******** *** Spectre ***************, ***** **** apply **** ******* ******* and **** *******. ** these ***, *** **** update *** * *********** impact ** *** ****, while ******* ******* *** minimal ******, **** ***** *% on *** ***** ******, both ********* *** *******.

Significant ****** ** **** ********** ****

** *** ***** ** 24 ****** *******, ~*** Mb/s ***** **********, ****** process *** **** ********* significantly ***** ******** *** BIOS ** **** *******, with *********** ************ ********** by ~*% *** ********* XProtect ********* ** ~**%.

**** **** ** ******** no ****** ** *********** in ****** *** ****** testing (******* ******, **** loading, ********/******* *****, ***.), but ***** **** ****** throughput *** *** ****** as **** ***** ***** updating.

Low ********** ********* *******

******* **** ** ******* at ***** ** **/* total, ****** **** *** usage ********* **** *****, but *************** ****** **** ** ****** testing, **** ***** ********** by 4% **** (* **% increase) *** ********* ** 5% (* ~**% ********). However, ******** ******* ****** jumped *************, **** ~*% to *%, *** ** with ****** ********** *******, these ********* *** *** result ** ********** *********** issues.

Live **** ******* *** ******

*** **** ********* *** those ******* ***** ***** viewing **** *** **** significant, **** ******** *** Exacq ********** ** **-**% when **** ******* ** playing **** *** ******* on *** ********* ******. Milestone ********* ** ***** the **** *** **** percentage, *** * *************** smaller ******.

Test ********* ****** ********

***** ** *** ******* state ** *****'* **** patches **** ***** ******* users ** **** ********* before ******** ******* ** production ********. ***** *** cannot ***** **** ** a ***-********** *********** *** want ** **** *** the ******* **** ******* to ******** **** ** unanticipated *******.

Manufacturer ********

**** *** **** ** communication **** *** ************* regarding *********** ******* **** patches. ** ***, ************* **** indicated **** *** ***** in *** ******* ** determining ******* *********** ******, and *********** ********** *** applying *******.

Comments (21)

John Honovich

IPVM | 01/23/18 05:48pm

What do you do, if after you upgrade / patch, you realize you have a performance problem on your VMS? Replace the CPU? Replace the server? What are one's practical options?

Brian Karas

IPVM | In reply to John Honovich | 01/23/18 05:55pm

It is hard to say with limited data being released so far. In some cases, you may be able to upgrade to a faster processor. Similarly, upgrading to faster RAM (if possible) may also help.

Until VMS manufacturers can issue some solid guidance on this, I would limit investments in upgrading components, or servers, as much as possible. It could turn out that server upgrades are the most practical approach for older hardware, as you would likely want to leave a decent chunk of free CPU to absorb any future enhancements that also add to processor utilization.

Also, this has not been addressed so far, but if the server is supporting multiple applications (e.g.: VMS server and also mobile gateway server, etc.), it may be cost-effective to off-load some applications to other hardware to free up CPU for the core VMS.

Undisclosed Manufacturer #4

In reply to John Honovich | 01/26/18 04:33pm

With Milestone 2018 R1 you have the option to add one or more NVidia cards to a client machine to increase processing power. Milestone automatically load balances the GPU's and uses CPU for leftover needs. Due out Feb 20, 2018

With Milestone 2018 R2, you have the option to drop one or more NVidia cards into your recording servers to offload the decompression tasks. This is not due out till mid-summer so a bit of a gap for those patching now, but at least it is a real and relatively low-cost option that we have.

Tom Sharples

01/23/18 06:13pm

Why are these exploits even an issue in a closed environment such as a VMS server? No one is going to be using one to surf the net or adding random apps, or am I missing something?

Undisclosed Integrator #2

In reply to Tom Sharples | 01/24/18 01:38pm

Unfortunately from the integrator's point of view, security networks are not so closed environments anymore.

IT departments are pushing towards opening security networks to other segments of the network, and sometimes yes they are open to the Internet as well either for remote assistance, remote monitoring or connecting remote sites.

Undisclosed #1

01/23/18 07:07pm

Impact on server load when viewing live or playback video was even more significant, increasing CPU load by 80-90% on low camera count systems using Avigilon and Exacq

Forgive my low reading comprehension but am I understanding this right? I'm hoping what is meant is that if we're at 10% load it is jumping to 18%-19%.

Ethan Ace

IPVM | In reply to Undisclosed #1 | 01/23/18 10:19pm

Yes, that's correct. I'll edit that so it's more clear.

Rob Kilpatrick

IPVM | IPVMU Certified | In reply to Undisclosed #1 | 01/23/18 10:28pm

Yes, you were reading it correctly.

For example, the exacq server in the low throughput test increased by 4% cpu usage, going from 8% pre-patch to 12% post patch, this was a 50% increase.

Bob Schenck

IPVMU Certified | 01/24/18 04:18am

The problem right now with BIOS updates is Intel released code that is causing a reboot problem. Dell, HP, and Lenovo have all removed the latest BIOS patches from their downloads and are recommending anyone who installed them to roll back. If you recently patched your BIOS, you should go back and check to be sure you didn't install a version that has been retracted.

Dell and HP purge Intel’s Spetcre patch over reboot woes

Michael Miller

01/24/18 02:39pm

“We recommend that OEMs, cloud service providers, system manufacturers, software vendors and end users stop deployment of current versions, as they may introduce higher than expected reboots and other unpredictable system behavior,” he wrote.

https://www.scmagazine.com/intel-advises-companies-to-stop-installing-spectremeltdown-update/article/738625/

Alex Knapik

Milestone Systems
01/24/18 11:11pm

Purely from a testing methodology standpoint, was testing done with Milestone Smart Client hardware acceleration enabled? It could stand to reason CPU usage and impact would be less than this if GPUs would being used to decode and render...

Undisclosed #1

In reply to Alex Knapik | 01/24/18 11:17pm

Alex, since you are with Milestone, what does your internal testing show?

Campbell Chang

01/25/18 01:25am

Don't suppose you can build an AMD Ryzen machine and retest this?

Intel has done a great job convincing everyone that Meltdown affects everyone despite AMD being entirely unaffected by this. So I'd be super curious to find out how much the Meltdown patch affects the speed vs Spectre.

Undisclosed Manufacturer #3

01/25/18 11:17am

We are currently evaluating this situation and discussing with customers whether or not to actually applying the BIOS patches.

The reason for this is that we want the customers to have a choice. Many of our customers have distinctly disconnected networks for CCTV and very high control over what can be installed on their machines, like web browsers.

To do this we need to call on our corporate security experts to look at the attack vectors and recommend actions.

With these high impacts we are seeing, many of our customers will have to do expensive and time consuming upgrades if there is no way around the BIOS patches.

Tim Cook

IPVMU Certified | 01/25/18 02:25pm

This BIOS vulnerability, can it affect dedicated Linux OS recorders/NVR's with Intel chips or only Windows/Intel devices?

How about the Meltdown vulnerability on these same devices?

Brian Karas

IPVM | In reply to Tim Cook | 01/25/18 02:36pm

These are processor vulnerabilities, so they can affect any OS really.

Linux can certainly be affected, and Linus Torvalds has been a bit critical of Intel on this.

Undisclosed #1

In reply to Brian Karas | 01/26/18 02:34am

Brian,

From what I have been reading it appears a much smaller set of ARM chips are affected as well. Does this have any potential impact to embedded DVRs/NVRs or does their embedded nature prevent this?

Brian Karas

IPVM | In reply to Undisclosed #1 | 01/26/18 02:40am

Yes, some ARM chips are affected also. Embedded recorders of the sort that do not allow you to load external software/apps/etc., and instead distribute software as complete firmware images are less likely to be infected. A caveat of course being that you do not have things like telnet, ftp, ssh, open where an attacker could upload a binary directly to the system.

Currently exploits rely in some malicious software being executed on the device that is designed to leverage these vulnerabilities. (note it may not technically need to be installed, as there have been some browser-based PoC's). With most embedded NVRs you do not have a way to load random software or even execute a browser, so they are safer than PC-based systems in that regard.

Also keep in mind the above is based on current knowledge of the vulnerabilities and exploits. As more details about these vulnerabilities circulate among black-hat hackers there could be new exploits devised that put embedded NVRs at greater risk.

bashis mcw

01/29/18 07:44pm

To my reading and understanding, this is a bit hyped... sure it's serious bug, when you have someone or something executing on your host.

Campbell Chang

In reply to bashis mcw | 02/01/18 05:01am

The biggest drama out of this is not the bug itself but the resultant performance of the CPUs post fix.

In Intels case, testing has revealed a decrease of 25% performance I/O intensive tasks. This is where the real issue with Meltdown and Spectre lies.

Boris Biryuchkov

02/01/18 04:24am

I tried to reproduce it myself.

Realized that my NUC (nuc7i3bnh) doesn't have Meltdown/Spectre fix available on the Intel website (latest Bios update is Nov 2017).

Tried to find your firmware (RYBDWi35.86A.0368.2017.1220.0950) and failed again.

The only link from Google leads to 404 https://downloadmirror.intel.com/27426/eng/RY_0368_ReleaseNotes.pdf

If check downloads for your model (NUC5i7RYH)

https://downloadcenter.intel.com/product/87570/Intel-NUC-Kit-NUC5i7RYH

the latest Bios update is dated September 2017.

Looks like Intel takes down the Bios updates as there is too many issues with the upgrade.

Read this IPVM report for free.

This article is part of IPVM's 6,602 reports, 890 tests and is only available to members. To get a one-time preview of our work, enter your work email to access the full article.