Intel Meltdown / Spectre Patch Tested on Avigilon, Exacq and Milestone VMSes

By: IPVM Team, Published on Jan 23, 2018

Significant concern exists about the impact on VMS servers and video surveillance systems of patching the Meltdown and Spectre flaws. Generally, reports have claimed a significant increase in CPU loads, though the amounts have varied depending on the source and services being used. This could be very problematic for VMS servers as it risks problems in recording, watching or playing back video.

Meltdown / Spectre Summary

Meltdown is a flaw in protections allowing unauthorized applications to access OS kernel protected memory. Spectre allows unauthorized applications to trick other programs into divulging private data. For technical details on these, see The Meltdown and Spectre CPU Bugs, Explained [link no longer available] or various other online tech sources.

How To Patch

Users may see if their Windows machine is vulnerable using PowerShell, following the steps in this guide.

If so, in order to avoid both of these vulnerabilities, users must install two updates:

Windows Update KB4056892: This update should have been included in regular Windows Updates as of early January. This fixes the Meltdown vulnerability.
BIOS update: To fix the Spectre vulnerability, users must update their system BIOS, provided by their hardware manufacturer.

However, many devices have not yet received a BIOS update. Out of four different hardware manufacturers we checked (Acer, Asus, HP, and Intel NUCs), only Intel had released a BIOS update for our machines.

Tests Performed

IPVM measured CPU load of server and client processes prior to applying Windows and BIOS updates, and again after updating, using two configurations:

Recording only, 25 cameras, ~150 Mb/s
Recording and viewing, 10 cameras, ~30 Mb/s

Test Machine Specs

IPVM tested on three separate machines, all with the same hardware configuration, as follows:

Operating System: Windows 10 Pro 64-bit (10.0, Build 16299) (16299.rs3_release.170928-1534)
BIOS: RYBDWi35.86A.0350.2015.0812.1722 (pre-update)/RYBDWi35.86A.0368.2017.1220.0950 (post-update)
Processor: Intel(R) Core(TM) i7-5557U CPU @ 3.10GHz (4 CPUs), ~3.1GHz
Memory: 8192MB RAM

Note that these machines are fairly high spec, with relatively new processors (1-2 generations old). Older generation processors and lower spec machines are likely to see a greater impact.

Inside this report, we examine the results achieved across Avigilon, Exacq and Milestone plus our recommendations.

*********** ******* ****** ***** the ****** ** *** servers *** ***** ************ systems ** ******** *** Meltdown *** ******* *****. Generally, ******* **** ******* a *********** ******** ** CPU *****, ****** *** amounts **** ****** ********* on *** ****** *** services ***** ****. **** could ** **** *********** for *** ******* ** it ***** ******** ** recording, ******** ** ******* back *****.

Meltdown / ******* *******

******** ** * **** in *********** ******** ************ applications ** ****** ** kernel ********* ******. ******* ****** unauthorized ************ ** ***** other ******** **** ********* private ****. *** ********* details ** *****, *** The ******** *** ******* CPU ****, ********* [**** no ****** *********] ** ******* other ****** **** *******.

How ** *****

***** *** *** ** their ******* ******* ** vulnerable ***** **********,********* *** ***** ** this *****.

** **, ** ***** to ***** **** ** these ***************, ***** **** install *** *******:

******* ****** *********: **** ****** should **** **** ******** in ******* ******* ******* as ** ***** *******. This ***** *** ******** vulnerability.
**** ******: ** *** *** Spectre *************, ***** **** update ***** ****** ****, provided ** ***** ******** manufacturer.

*******, **** ******* **** not *** ******** * BIOS ******. *** ** four ********* ******** ************* we ******* (****, ****, HP, *** ***** ****), only ***** *** ******** a **** ****** *** our ********.

Tests *********

**** ******** *** **** of ****** *** ****** processes ***** ** ******** Windows *** **** *******, and ***** ***** ********, using *** **************:

********* ****, ** *******, ~150 **/*
********* *** *******, ** cameras, ~** **/*

Test ******* *****

**** ****** ** ***** ******** machines, *** **** *** same ******** *************, ** follows:

********* ******:******* ** *** **-*** (10.0, ***** *****) (*****.***********.******-****)
****:********.***.****.****.****.**** (***-******)/********.***.****.****.****.**** (****-******)
*********:*****(*) ****(**) **-***** *** @ *.***** (* ****), ~3.1GHz
******:****** ***

**** **** ***** ******** are ****** **** ****, with ********** *** ********** (1-2 *********** ***). ***** generation ********** *** ***** spec ******** *** ****** to *** * ******* impact.

****** **** ******, ** examine *** ******* ******** across ********, ***** *** Milestone plus *** ***************.

[***************]

Test *******

** *** *****, ******** system **** ** ***** the ******* ************* *** a *********** ****** ** CPU ****, ********** *** load ** ********* ******** by ** **** **-**%. ****** on ****** **** **** viewing **** ** ******** video *** **** **** significant, nearly ******** *** **** ** low ****** ***** ******* using ******** *** ***** (an ******** ** **-**% load). ******** *******' ******** update ** ***** ******** alone had * ********** ******, increasing **** **** **** 1%.

****** *******, ** *** no *** ******* **** these *******/******** *******, **** as ******* ******, **** loading *****, ******** ** video, ** ***** *** malfunctions ***** *** ***** when **********. *******, **** that *** ******* *** load **** ** **** testing *** ***** ***** 50% ***********. ***** **** higher ********** ******* ** lower **** ******* *** see ****** ** ***** camera ******/***********.

***************

******* ** *** *********** impact ** **** ******* on *** ****, ** strongly ********* **** ***** of ****** **********/****** ****** count ******* ****** ***** system *** ******** ***** to ********. ***** ********* running ** **-**% **** are ****** ** ******** to **-***%, **** *** potential *** ****** **** higher.

***** **** *** ****** *****/********** systems *** ****** ** see ****** *********** ********** unless **** ******* ******* on *** ****** *******.

Intel ***** ******** ******* - ****** *****

******* *** **** ******* have ****** ****** ****** on ******* ******** (****: IPVM *** *** ********** this, *** ** *** be ******** *********). ***** released * ********* ** January **** ******** *** ***** ******** BIOS ********** **** ****** **** updated ******* ****. ******* indications *** **** *** updated ******* **** **** roughly ******* *********** *******, and ***** ** ************ partners ** ******** ** test **** *** ******* patches ** *** **** data ** *********** *******.

Biggest ******: **** ******

** ***** ** ***** repair *** ******** *** Spectre ***************, ***** **** apply **** ******* ******* and **** *******. ** these ***, *** **** update *** * *********** impact ** *** ****, while ******* ******* *** minimal ******, **** ***** *% on *** ***** ******, both ********* *** *******.

Significant ****** ** **** ********** ****

** *** ***** ** 24 ****** *******, ~*** Mb/s ***** **********, ****** process *** **** ********* significantly ***** ******** *** BIOS ** **** *******, with *********** ************ ********** by ~*% *** ********* XProtect ********* ** ~**%.

**** **** ** ******** no ****** ** *********** in ****** *** ****** testing (******* ******, **** loading, ********/******* *****, ***.), but ***** **** ****** throughput *** *** ****** as **** ***** ***** updating.

Low ********** ********* *******

******* **** ** ******* at ***** ** **/* total, ****** **** *** usage ********* **** *****, but *************** ****** **** ** ****** testing, **** ***** ********** by 4% **** (* **% increase) *** ********* ** 5% (* ~**% ********). However, ******** ******* ****** jumped *************, **** ~*% to *%, *** ** with ****** ********** *******, these ********* *** *** result ** ********** *********** issues.

Live **** ******* *** ******

*** **** ********* *** those ******* ***** ***** viewing **** *** **** significant, **** ******** *** Exacq ********** ** **-**% when **** ******* ** playing **** *** ******* on *** ********* ******. Milestone ********* ** ***** the **** *** **** percentage, *** * *************** smaller ******.

Test ********* ****** ********

***** ** *** ******* state ** *****'* **** patches **** ***** ******* users ** **** ********* before ******** ******* ** production ********. ***** *** cannot ***** **** ** a ***-********** *********** *** want ** **** *** the ******* **** ******* to ******** **** ** unanticipated *******.

Manufacturer ********

**** *** **** ** communication **** *** ************* regarding *********** ******* **** patches. ** ***, ************* **** indicated **** *** ***** in *** ******* ** determining ******* *********** ******, and *********** ********** *** applying *******.

Comments (21)

John Honovich

IPVM | 01/23/18 05:48pm

What do you do, if after you upgrade / patch, you realize you have a performance problem on your VMS? Replace the CPU? Replace the server? What are one's practical options?

Brian Karas

IPVM | In reply to John Honovich | 01/23/18 05:55pm

It is hard to say with limited data being released so far. In some cases, you may be able to upgrade to a faster processor. Similarly, upgrading to faster RAM (if possible) may also help.

Until VMS manufacturers can issue some solid guidance on this, I would limit investments in upgrading components, or servers, as much as possible. It could turn out that server upgrades are the most practical approach for older hardware, as you would likely want to leave a decent chunk of free CPU to absorb any future enhancements that also add to processor utilization.

Also, this has not been addressed so far, but if the server is supporting multiple applications (e.g.: VMS server and also mobile gateway server, etc.), it may be cost-effective to off-load some applications to other hardware to free up CPU for the core VMS.

Undisclosed Manufacturer #4

In reply to John Honovich | 01/26/18 04:33pm

With Milestone 2018 R1 you have the option to add one or more NVidia cards to a client machine to increase processing power. Milestone automatically load balances the GPU's and uses CPU for leftover needs. Due out Feb 20, 2018

With Milestone 2018 R2, you have the option to drop one or more NVidia cards into your recording servers to offload the decompression tasks. This is not due out till mid-summer so a bit of a gap for those patching now, but at least it is a real and relatively low-cost option that we have.

Tom Sharples

01/23/18 06:13pm

Why are these exploits even an issue in a closed environment such as a VMS server? No one is going to be using one to surf the net or adding random apps, or am I missing something?

Undisclosed Integrator #2

In reply to Tom Sharples | 01/24/18 01:38pm

Unfortunately from the integrator's point of view, security networks are not so closed environments anymore.

IT departments are pushing towards opening security networks to other segments of the network, and sometimes yes they are open to the Internet as well either for remote assistance, remote monitoring or connecting remote sites.

Undisclosed Integrator #1

01/23/18 07:07pm

Impact on server load when viewing live or playback video was even more significant, increasing CPU load by 80-90% on low camera count systems using Avigilon and Exacq

Forgive my low reading comprehension but am I understanding this right? I'm hoping what is meant is that if we're at 10% load it is jumping to 18%-19%.

Ethan Ace

IPVM | In reply to Undisclosed Integrator #1 | 01/23/18 10:19pm

Yes, that's correct. I'll edit that so it's more clear.

Rob Kilpatrick

IPVM | IPVMU Certified | In reply to Undisclosed Integrator #1 | 01/23/18 10:28pm

Yes, you were reading it correctly.

For example, the exacq server in the low throughput test increased by 4% cpu usage, going from 8% pre-patch to 12% post patch, this was a 50% increase.

Bob Schenck

IPVMU Certified | 01/24/18 04:18am

The problem right now with BIOS updates is Intel released code that is causing a reboot problem. Dell, HP, and Lenovo have all removed the latest BIOS patches from their downloads and are recommending anyone who installed them to roll back. If you recently patched your BIOS, you should go back and check to be sure you didn't install a version that has been retracted.

Dell and HP purge Intel’s Spetcre patch over reboot woes

Michael Miller

01/24/18 02:39pm

“We recommend that OEMs, cloud service providers, system manufacturers, software vendors and end users stop deployment of current versions, as they may introduce higher than expected reboots and other unpredictable system behavior,” he wrote.

https://www.scmagazine.com/intel-advises-companies-to-stop-installing-spectremeltdown-update/article/738625/

Alex Knapik

Milestone Systems | 01/24/18 11:11pm

Purely from a testing methodology standpoint, was testing done with Milestone Smart Client hardware acceleration enabled? It could stand to reason CPU usage and impact would be less than this if GPUs would being used to decode and render...

Undisclosed Integrator #1

In reply to Alex Knapik | 01/24/18 11:17pm

Alex, since you are with Milestone, what does your internal testing show?

Campbell Chang

01/25/18 01:25am

Don't suppose you can build an AMD Ryzen machine and retest this?

Intel has done a great job convincing everyone that Meltdown affects everyone despite AMD being entirely unaffected by this. So I'd be super curious to find out how much the Meltdown patch affects the speed vs Spectre.

Undisclosed Manufacturer #3

01/25/18 11:17am

We are currently evaluating this situation and discussing with customers whether or not to actually applying the BIOS patches.

The reason for this is that we want the customers to have a choice. Many of our customers have distinctly disconnected networks for CCTV and very high control over what can be installed on their machines, like web browsers.

To do this we need to call on our corporate security experts to look at the attack vectors and recommend actions.

With these high impacts we are seeing, many of our customers will have to do expensive and time consuming upgrades if there is no way around the BIOS patches.

Tim Cook

IPVMU Certified | 01/25/18 02:25pm

This BIOS vulnerability, can it affect dedicated Linux OS recorders/NVR's with Intel chips or only Windows/Intel devices?

How about the Meltdown vulnerability on these same devices?

Brian Karas

IPVM | In reply to Tim Cook | 01/25/18 02:36pm

These are processor vulnerabilities, so they can affect any OS really.

Linux can certainly be affected, and Linus Torvalds has been a bit critical of Intel on this.

Undisclosed Integrator #1

In reply to Brian Karas | 01/26/18 02:34am

Brian,

From what I have been reading it appears a much smaller set of ARM chips are affected as well. Does this have any potential impact to embedded DVRs/NVRs or does their embedded nature prevent this?

Brian Karas

IPVM | In reply to Undisclosed Integrator #1 | 01/26/18 02:40am

Yes, some ARM chips are affected also. Embedded recorders of the sort that do not allow you to load external software/apps/etc., and instead distribute software as complete firmware images are less likely to be infected. A caveat of course being that you do not have things like telnet, ftp, ssh, open where an attacker could upload a binary directly to the system.

Currently exploits rely in some malicious software being executed on the device that is designed to leverage these vulnerabilities. (note it may not technically need to be installed, as there have been some browser-based PoC's). With most embedded NVRs you do not have a way to load random software or even execute a browser, so they are safer than PC-based systems in that regard.

Also keep in mind the above is based on current knowledge of the vulnerabilities and exploits. As more details about these vulnerabilities circulate among black-hat hackers there could be new exploits devised that put embedded NVRs at greater risk.

bashis mcw

01/29/18 07:44pm

To my reading and understanding, this is a bit hyped... sure it's serious bug, when you have someone or something executing on your host.

Campbell Chang

In reply to bashis mcw | 02/01/18 05:01am

The biggest drama out of this is not the bug itself but the resultant performance of the CPUs post fix.

In Intels case, testing has revealed a decrease of 25% performance I/O intensive tasks. This is where the real issue with Meltdown and Spectre lies.

Boris Biryuchkov

02/01/18 04:24am

I tried to reproduce it myself.

Realized that my NUC (nuc7i3bnh) doesn't have Meltdown/Spectre fix available on the Intel website (latest Bios update is Nov 2017).

Tried to find your firmware (RYBDWi35.86A.0368.2017.1220.0950) and failed again.

The only link from Google leads to 404 https://downloadmirror.intel.com/27426/eng/RY_0368_ReleaseNotes.pdf

If check downloads for your model (NUC5i7RYH)

https://downloadcenter.intel.com/product/87570/Intel-NUC-Kit-NUC5i7RYH

the latest Bios update is dated September 2017.

Looks like Intel takes down the Bios updates as there is too many issues with the upgrade.

Login to read this IPVM report.

Why do I need to log in?

IPVM conducts unique testing and research funded by member's payments enabling us to offer the most independent, accurate and in-depth information.

Related Reports

Bandwidth Fundamentals For Video Surveillance on Jan 13, 2020

Bandwidth is the most fundamental element of computer networking for video surveillance systems. Because video surveillance can consume an immense...

Testing Bandwidth vs. Frame Rate on Jan 23, 2019

Selecting frame rate has a major impact on surveillance bandwidth and storage consumption. But with smart codecs now common and cameras more...

Bandwidth vs Low Light Shootout - Avigilon, Axis, Bosch, Dahua, Geovision, Hanwha, Hikvision, Uniview, Vivotek on Feb 08, 2019

Nighttime bandwidth spikes are a major concern in video surveillance, but do all manufacturers' cameras perform the same? Are some more consistent...

Exacq Raises VMS Software Pricing Twice in Less Than a Year on Feb 18, 2019

Most VMSes regularly release new features, but rarely increase their prices. For the 3rd time in 4 years, and 2nd time in 8 months, since being...

Subnetting for Video Surveillance on Apr 30, 2019

This guide explains when subnetting is used on security networks, and how it works. We explain how to add or remove IP addresses to your range,...

H.265 Usage Statistics on Apr 19, 2019

H.265 has been available in IP cameras for more than 5 years and, in the past few years, the number of manufacturers supporting this codec has...

Average Frame Rate Video Surveillance 2019 on May 23, 2019

What is the average frame rated used in video surveillance systems? In IPVM's 2011 statistics, the average was 6-8fps increasing to ~10fps in...

Verkada Video Quality Problems Tested on May 23, 2019

Verkada suffers from numerous video quality problems, not found in commercial IP cameras, new IPVM testing of Verkada vs Axis and Hikvision...

Smart CODEC Usage Statistics 2019 on Jun 03, 2019

Smart codecs are now nearly a standard feature in IP cameras, but our statistics show integrator adoption has not increased at the same rate. In...

Dahua Wiretapping Vulnerability on Aug 02, 2019

IPVM has validated, with testing, and from Dahua, that many Dahua cameras have a wiretapping vulnerability. Even if the camera's audio has been...

Most Recent Industry Reports

IronYun AI Analytics Tested on Feb 17, 2020

Taiwan startup IronYun has raised tens of millions for its "mission to be the leading Artificial Intelligence, big data video software as a service...