Intel Meltdown / Spectre Patch Tested on Avigilon, Exacq and Milestone VMSes

By IPVM Team, Published on Jan 23, 2018

Significant concern exists about the impact on VMS servers and video surveillance systems of patching the Meltdown and Spectre flaws. Generally, reports have claimed a significant increase in CPU loads, though the amounts have varied depending on the source and services being used. This could be very problematic for VMS servers as it risks problems in recording, watching or playing back video.

Meltdown / Spectre Summary

Meltdown is a flaw in protections allowing unauthorized applications to access OS kernel protected memory. Spectre allows unauthorized applications to trick other programs into divulging private data. For technical details on these, see The Meltdown and Spectre CPU Bugs, Explained [link no longer available] or various other online tech sources.

How To Patch

Users may see if their Windows machine is vulnerable using PowerShell, following the steps in this guide

If so, in order to avoid both of these vulnerabilities, users must install two updates:

  • Windows Update KB4056892: This update should have been included in regular Windows Updates as of early January. This fixes the Meltdown vulnerability.
  • BIOS update: To fix the Spectre vulnerability, users must update their system BIOS, provided by their hardware manufacturer.

However, many devices have not yet received a BIOS update. Out of four different hardware manufacturers we checked (Acer, Asus, HP, and Intel NUCs), only Intel had released a BIOS update for our machines. 

Tests Performed

IPVM measured CPU load of server and client processes prior to applying Windows and BIOS updates, and again after updating, using two configurations:

  • Recording only, 25 cameras, ~150 Mb/s
  • Recording and viewing, 10 cameras, ~30 Mb/s

Test Machine Specs

IPVM tested on three separate machines, all with the same hardware configuration, as follows:

  • Operating System: Windows 10 Pro 64-bit (10.0, Build 16299) (16299.rs3_release.170928-1534)
  • BIOS: RYBDWi35.86A.0350.2015.0812.1722 (pre-update)/RYBDWi35.86A.0368.2017.1220.0950 (post-update)
  • Processor: Intel(R) Core(TM) i7-5557U CPU @ 3.10GHz (4 CPUs), ~3.1GHz
  • Memory: 8192MB RAM

Note that these machines are fairly high spec, with relatively new processors (1-2 generations old). Older generation processors and lower spec machines are likely to see a greater impact.

Inside this report, we examine the results achieved across Avigilon, Exacq and Milestone plus our recommendations.

Test *******

** *** *****, ******** system **** ** ***** the ******* ************* *** a *********** ****** ** CPU ****, ********** *** load ** ********* ******** by ** **** **-**%. ****** on ****** **** **** viewing **** ** ******** video *** **** **** significant, nearly ******** *** **** ** low ****** ***** ******* using ******** *** ***** (an ******** ** **-**% load). ******** *******' ******** update ** ***** ******** alone had * ********** ******, increasing **** **** **** 1%. 

****** *******, ** *** no *** ******* **** these *******/******** *******, **** as ******* ******, **** loading *****, ******** ** video, ** ***** *** malfunctions ***** *** ***** when **********. *******, **** that *** ******* *** load **** ** **** testing *** ***** ***** 50% ***********. ***** **** higher ********** ******* ** lower **** ******* *** see ****** ** ***** camera ******/***********.

***************

******* ** *** *********** impact ** **** ******* on *** ****, ** strongly ********* **** ***** of ****** **********/****** ****** count ******* ****** ***** system *** ******** ***** to ********. ***** ********* running ** **-**% **** are ****** ** ******** to **-***%, **** *** potential *** ****** **** higher. 

***** **** *** ****** *****/********** systems *** ****** ** see ****** *********** ********** unless **** ******* ******* on *** ****** *******.

Intel ***** ******** ******* - ****** *****

******* *** **** ******* have ****** ****** ****** on ******* ******** (****: IPVM *** *** ********** this, *** ** *** be ******** *********). ***** released * ********* ** January **** ******** *** ***** ******** BIOS ********** **** ****** **** updated ******* ****. ******* indications *** **** *** updated ******* **** **** roughly ******* *********** *******, and ***** ** ************ partners ** ******** ** test **** *** ******* patches ** *** **** data ** *********** *******. 

Biggest ******: **** ******

** ***** ** ***** repair *** ******** *** Spectre ***************, ***** **** apply **** ******* ******* and **** *******. ** these ***, *** **** update *** * *********** impact ** *** ****, while ******* ******* *** minimal ******, **** ***** *% on *** ***** ******, both ********* *** *******.

Significant ****** ** **** ********** ****

** *** ***** ** 24 ****** *******, ~*** Mb/s ***** **********, ****** process *** **** ********* significantly ***** ******** *** BIOS ** **** *******, with *********** ************ ********** by ~*% *** ********* XProtect ********* ** ~**%.

**** **** ** ******** no ****** ** *********** in ****** *** ****** testing (******* ******, **** loading, ********/******* *****, ***.), but ***** **** ****** throughput *** *** ****** as **** ***** ***** updating.

Low ********** ********* *******

******* **** ** ******* at ***** ** **/* total, ****** **** *** usage ********* **** *****, but *************** ****** **** ** ****** testing, **** ***** ********** by 4% **** (* **% increase) *** ********* ** 5% (* ~**% ********). However, ******** ******* ****** jumped *************, **** ~*% to *%, *** ** with ****** ********** *******, these ********* *** *** result ** ********** *********** issues.

Live **** ******* *** ******

*** **** ********* *** those ******* ***** ***** viewing **** *** **** significant, **** ******** *** Exacq ********** ** **-**% when **** ******* ** playing **** *** ******* on *** ********* ******. Milestone ********* ** ***** the **** *** **** percentage, *** * *************** smaller ******.

 

Test ********* ****** ********

***** ** *** ******* state ** *****'* **** patches **** ***** ******* users ** **** ********* before ******** ******* ** production ********. ***** *** cannot ***** **** ** a ***-********** *********** *** want ** **** *** the ******* **** ******* to ******** **** ** unanticipated *******.

Manufacturer ********

**** *** **** ** communication **** *** ************* regarding *********** ******* **** patches. ** ***, ************* **** indicated **** *** ***** in *** ******* ** determining ******* *********** ******, and *********** ********** *** applying *******.

Comments (21)

What do you do, if after you upgrade / patch, you realize you have a performance problem on your VMS? Replace the CPU? Replace the server? What are one's practical options?

It is hard to say with limited data being released so far.  In some cases, you may be able to upgrade to a faster processor.  Similarly, upgrading to faster RAM (if possible) may also help.

Until VMS manufacturers can issue some solid guidance on this, I would limit investments in upgrading components, or servers, as much as possible. It could turn out that server upgrades are the most practical approach for older hardware, as you would likely want to leave a decent chunk of free CPU to absorb any future enhancements that also add to processor utilization.

Also, this has not been addressed so far, but if the server is supporting multiple applications (e.g.: VMS server and also mobile gateway server, etc.), it may be cost-effective to off-load some applications to other hardware to free up CPU for the core VMS.

 

With Milestone 2018 R1 you have the option to add one or more NVidia cards to a client machine to increase processing power. Milestone automatically load balances the GPU's and uses CPU for leftover needs. Due out Feb 20, 2018

With Milestone 2018 R2, you have the option to drop one or more NVidia cards into your recording servers to offload the decompression tasks. This is not due out till mid-summer so a bit of a gap for those patching now, but at least it is a real and relatively low-cost option that we have.

Why are these exploits even an issue in a closed environment such as a VMS server? No one is going to be using one to surf the net or adding random apps, or am I missing something?

Unfortunately from the integrator's point of view, security networks are not so closed environments anymore.

IT departments are pushing towards opening security networks to other segments of the network, and sometimes yes they are open to the Internet as well either for remote assistance, remote monitoring or connecting remote sites.

 

Impact on server load when viewing live or playback video was even more significant, increasing CPU load by 80-90% on low camera count systems using Avigilon and Exacq

Forgive my low reading comprehension but am I understanding this right?  I'm hoping what is meant is that if we're at 10% load it is jumping to 18%-19%.

Yes, that's correct. I'll edit that so it's more clear. 

Yes, you were reading it correctly.

For example, the exacq server in the low throughput test increased by 4% cpu usage, going from 8% pre-patch to 12% post patch, this was a 50% increase.

The problem right now with BIOS updates is Intel released code that is causing a reboot problem.  Dell, HP, and Lenovo have all removed the latest BIOS patches from their downloads and are recommending anyone who installed them to roll back.  If you recently patched your BIOS, you should go back and check to be sure you didn't install a version that has been retracted.

Dell and HP purge Intel’s Spetcre patch over reboot woes

 

“We recommend that OEMs, cloud service providers, system manufacturers, software vendors and end users stop deployment of current versions, as they may introduce higher than expected reboots and other unpredictable system behavior,” he wrote.

https://www.scmagazine.com/intel-advises-companies-to-stop-installing-spectremeltdown-update/article/738625/

Purely from a testing methodology standpoint, was testing done with Milestone Smart Client hardware acceleration enabled? It could stand to reason CPU usage and impact would be less than this if GPUs would being used to decode and render...

Alex, since you are with Milestone, what does your internal testing show?

Don't suppose you can build an AMD Ryzen machine and retest this?

Intel has done a great job convincing everyone that Meltdown affects everyone despite AMD being entirely unaffected by this.  So I'd be super curious to find out how much the Meltdown patch affects the speed vs Spectre.

 

We are currently evaluating this situation and discussing with customers whether or not to actually applying the BIOS patches.

The reason for this is that we want the customers to have a choice. Many of our customers have distinctly disconnected networks for CCTV and very high control over what can be installed on their machines, like web browsers.

To do this we need to call on our corporate security experts to look at the attack vectors and recommend actions.

With these high impacts we are seeing, many of our customers will have to do expensive and time consuming upgrades if there is no way around the BIOS patches.

 

This BIOS vulnerability, can it affect dedicated Linux OS recorders/NVR's with Intel chips or only Windows/Intel devices?

How about the Meltdown vulnerability on these same devices?

These are processor vulnerabilities, so they can affect any OS really.

Linux can certainly be affected, and Linus Torvalds has been a bit critical of Intel on this.

 

 

Brian,

From what I have been reading it appears a much smaller set of ARM chips are affected as well.  Does this have any potential impact to embedded DVRs/NVRs or does their embedded nature prevent this?

Yes, some ARM chips are affected also. Embedded recorders of the sort that do not allow you to load external software/apps/etc., and instead distribute software as complete firmware images are less likely to be infected. A caveat of course being that you do not have things like telnet, ftp, ssh, open where an attacker could upload a binary directly to the system.

Currently exploits rely in some malicious software being executed on the device that is designed to leverage these vulnerabilities. (note it may not technically need to be installed, as there have been some browser-based PoC's). With most embedded NVRs you do not have a way to load random software or even execute a browser, so they are safer than PC-based systems in that regard.

Also keep in mind the above is based on current knowledge of the vulnerabilities and exploits. As more details about these vulnerabilities circulate among black-hat hackers there could be new exploits devised that put embedded NVRs at greater risk.

To my reading and understanding, this is a bit hyped... sure it's serious bug, when you have someone or something executing on your host.

The biggest drama out of this is not the bug itself but the resultant performance of the CPUs post fix.

In Intels case, testing has revealed a decrease of 25% performance I/O intensive tasks.  This is where the real issue with Meltdown and Spectre lies.

I tried to reproduce it myself.

Realized that my NUC (nuc7i3bnh) doesn't have Meltdown/Spectre fix available on the Intel website (latest Bios update is Nov 2017).

Tried to find your firmware (RYBDWi35.86A.0368.2017.1220.0950) and failed again.

The only link from Google leads to 404 https://downloadmirror.intel.com/27426/eng/RY_0368_ReleaseNotes.pdf

If check downloads for your model (NUC5i7RYH)

https://downloadcenter.intel.com/product/87570/Intel-NUC-Kit-NUC5i7RYH

the latest Bios update is dated September 2017.

Looks like Intel takes down the Bios updates as there is too many issues with the upgrade.

Read this IPVM report for free.

This article is part of IPVM's 6,602 reports, 890 tests and is only available to members. To get a one-time preview of our work, enter your work email to access the full article.

Already a member? Login here | Join now

Related Reports

Door Fundamentals For Access Control Guide on Aug 24, 2020
Doors vary greatly in how difficult and costly it is to add electronic access...
Uniview Deep Learning Camera Tested on Jul 14, 2020
Uniview's intrusion analytics have performed poorly in our shootouts. Now,...
FLIR Markets Windows Temperature Screening, Violates IEC And Causes Performance Problems on Jul 17, 2020
FLIR, one of the largest thermal screening manufacturers, is marketing...
Exit Devices For Access Control Tutorial on Aug 25, 2020
Exit Devices, also called 'Panic Bars' or 'Crash Bars' are required by safety...
Drain Wire For Access Control Reader Tutorial on Sep 23, 2020
An easy-to-miss cabling specification plays a key role in access control, yet...
Ubiquiti Access Control Tested on Oct 21, 2020
Ubiquiti has become one of the most widely used wireless and switch providers...
Free Online NFPA, IBC, and ADA Codes and Standards 2020 on Sep 03, 2020
Finding applicable codes for security work can be a costly task, with printed...
Verkada Disruptive Embedded Live Help on Sep 24, 2020
Call up your integrator? Have someone come by the next day? Verkada is...
K3 Pro Wall Mounted IR Gun Tested on Aug 28, 2020
The original K3 model was lacking in features that the K7 model had and was...
Face Detection Shootout - Dahua, Hanwha, Hikvision, Uniview, Vivotek on Jul 30, 2020
Face detection analytics are available from a number of manufactures...
YOLOv5 Released Amidst Controversy on Jul 27, 2020
YOLO has gained significant attention within video surveillance for its...
Injes Tiny Temperature Terminal Tested on Jul 17, 2020
While temperature terminals have trended bigger, the Injes DFace801 is...
China's SMIC Hit By US Trade Restrictions, Impact On Video Surveillance on Oct 13, 2020
US trade restrictions have hit Semiconductor Manufacturing International...
Forced Door Alarms For Access Control Tutorial on Aug 17, 2020
One of the most important access control alarms is also often ignored....
The Future of H.266 For Video Surveillance Examined on Aug 17, 2020
First H.264, now H.265, is H.266 next? H.266 was recently announced amid...

Recent Reports

Recruiters Online Show LIVE Today! on Oct 29, 2020
IPVM's 7th online show resumes today with 12 recruiters presenting themselves...
Hikvision AcuSense G2 Camera Test on Oct 29, 2020
Hikvision has released their next generation of AcuSense analytic cameras...
Biggest Problems Selling Access Control 2020 on Oct 29, 2020
Access control can cause integrators big headaches. What practical issues do...
Taiwan Geovision AI Analytics and NDAA Examined on Oct 29, 2020
Taiwan manufacturer Geovision's revenue has been falling for years. However,...
Bedside Cough and Sneeze Detector (Sound Intelligence and CLB) on Oct 28, 2020
Coronavirus has increased interest in detecting symptoms such as fever and...
Fever Tablet Thermal Sensors Examined (Melexis) on Oct 28, 2020
Fever tablet suppliers heavily rely on the accuracy and specs of...
Verkada Fires 3 on Oct 28, 2020
Verkada has fired three employees over an incident where female colleagues...
Eagle Eye Networks Raises $40 Million on Oct 27, 2020
Eagle Eye has raised $40 million aiming to "reinvent video...
Hikvision Q3 2020 Global Revenue Rises, US Revenue Falls on Oct 27, 2020
While Hikvision's global revenue rises driven by domestic recovery, its US...
VICE Investigates Verkada's Harassing "RawVerkadawgz" on Oct 26, 2020
This month, IPVM investigated Verkada's sexism, discrimination, and cultural...
Six Flags' FDA Violating Outdoor Dahua Fever Cameras on Oct 26, 2020
As Six Flags scrambled to reopen parks amid plummeting revenues caused by the...
ISC Brasil Digital Experience 2020 Report on Oct 23, 2020
ISC Brasil 2020 rebranded itself to ISC Digital Experience and, like its...
Top Video Surveillance Service Call Problems 2020 on Oct 23, 2020
3 primary and 4 secondary issues stood out as causing the most problems when...
GDPR Impact On Temperature / Fever Screening Explained on Oct 22, 2020
What impact does GDPR have on temperature screening? Do you risk a GDPR fine...
Security And Safety Things (S&ST) Tested on Oct 22, 2020
S&ST, a Bosch spinout, is spending tens of millions of dollars aiming to...