What do you do, if after you upgrade / patch, you realize you have a performance problem on your VMS? Replace the CPU? Replace the server? What are one's practical options?
Intel Meltdown / Spectre Patch Tested on Avigilon, Exacq and Milestone VMSes
*********** ******* ****** ***** *** ****** on *** ******* *** ***** ************ systems ** ******** *** ******** *** Spectre *****. *********, ******* **** ******* a *********** ******** ** *** *****, though *** ******* **** ****** ********* on *** ****** *** ******** ***** used. **** ***** ** **** *********** for *** ******* ** ** ***** problems ** *********, ******** ** ******* back *****.
Meltdown / ******* *******
******** ** * **** ** *********** allowing ************ ************ ** ****** ** kernel ********* ******. ******* ****** ************ ************ to ***** ***** ******** **** ********* private ****. *** ********* ******* ** these, *** *** ******** *** ******* CPU ****, ********* [**** ** ****** available] or ******* ***** ****** **** *******.
How ** *****
***** *** *** ** ***** ******* machine ** ********** ***** **********,********* *** ***** ** **** *****.
** **, ** ***** ** ***** both ** ***** ***************, ***** **** install *** *******:
- ******* ****** *********: **** ****** ****** **** been ******** ** ******* ******* ******* as ** ***** *******. **** ***** the ******** *************.
- **** ******: ** *** *** ******* *************, users **** ****** ***** ****** ****, provided ** ***** ******** ************.
*******, **** ******* **** *** *** received * **** ******. *** ** four ********* ******** ************* ** ******* (Acer, ****, **, *** ***** ****), only ***** *** ******** * **** update *** *** ********.
Tests *********
**** ******** *** **** ** ****** and ****** ********* ***** ** ******** Windows *** **** *******, *** ***** after ********, ***** *** **************:
- ********* ****, ** *******, ~*** **/*
- ********* *** *******, ** *******, ~** Mb/s
Test ******* *****
**** ****** ** ***** ******** ********, *** with *** **** ******** *************, ** follows:
- ********* ******:******* ** *** **-*** (**.*, ***** 16299) (*****.***********.******-****)
- ****:********.***.****.****.****.**** (***-******)/********.***.****.****.****.**** (****-******)
- *********:*****(*) ****(**) **-***** *** @ *.***** (4 ****), ~*.****
- ******:****** ***
**** **** ***** ******** *** ****** high ****, **** ********** *** ********** (1-2 *********** ***). ***** ********** ********** and ***** **** ******** *** ****** to *** * ******* ******.
****** **** ******, ** ******* *** results ******** ****** ********, ***** *** Milestone plus *** ***************.
Test *******
** *** *****, ******** ****** **** to ***** *** ******* ************* *** a *********** ****** ** *** ****, increasing *** **** ** ********* ******** by ** **** **-**%. ****** ** ****** load **** ******* **** ** ******** video *** **** **** ***********, ****** ******** *** load ** *** ****** ***** ******* using ******** *** ***** (** ******** of **-**% ****). ******** *******' ******** update ** ***** ******** ***** *** * negligible ******, ********** **** **** **** 1%.
****** *******, ** *** ** *** effects **** ***** *******/******** *******, **** as ******* ******, **** ******* *****, smearing ** *****, ** ***** *** malfunctions ***** *** ***** **** **********. However, **** **** *** ******* *** load **** ** **** ******* *** still ***** **% ***********. ***** **** higher ********** ******* ** ***** **** servers *** *** ****** ** ***** camera ******/***********.
***************
******* ** *** *********** ****** ** BIOS ******* ** *** ****, ** strongly ********* **** ***** ** ****** throughput/higher ****** ***** ******* ****** ***** system *** ******** ***** ** ********. Those ********* ******* ** **-**% **** are ****** ** ******** ** **-***%, with *** ********* *** ****** **** higher.
***** **** *** ****** *****/********** ******* *** likely ** *** ****** *********** ********** unless **** ******* ******* ** *** server *******.
Intel ***** ******** ******* - ****** *****
******* *** **** ******* **** ****** reboot ****** ** ******* ******** (****: IPVM *** *** ********** ****, *** it *** ** ******** *********). ***** released * ********* ** ******* **** that**** *** ***** ******** **** ********** **** ****** **** ******* ******* soon. ******* *********** *** **** *** updated ******* **** **** ******* ******* performance *******, *** ***** ** ************ partners ** ******** ** **** **** the ******* ******* ** *** **** data ** *********** *******.
Biggest ******: **** ******
** ***** ** ***** ****** *** Meltdown *** ******* ***************, ***** **** apply **** ******* ******* *** **** patches. ** ***** ***, *** **** update *** * *********** ****** ** CPU ****, ***** ******* ******* *** minimal ******, **** ***** *% ** *** VMSes ******, **** ********* *** *******.
Significant ****** ** **** ********** ****
** *** ***** ** ** ****** systems, ~*** **/* ***** **********, ****** process *** **** ********* ************* ***** patching *** **** ** **** *******, with *********** ************ ********** ** ~*% and ********* ******** ********* ** ~**%.
**** **** ** ******** ** ****** in *********** ** ****** *** ****** testing (******* ******, **** *******, ********/******* video, ***.), *** ***** **** ****** throughput *** *** ****** ** **** jumps ***** ********.
Low ********** ********* *******
******* **** ** ******* ** ***** 30 **/* *****, ****** **** *** usage ********* **** *****, *** *************** ****** than 24 ****** *******, **** ***** ********** by 4% **** (* **% ********) *** Milestone ** *% (* ~**% ********). However, ******** ******* ****** ****** *************, from ~*% ** *%, *** ** with ****** ********** *******, ***** ********* did *** ****** ** ********** *********** issues.
Live **** ******* *** ******
*** **** ********* *** ***** ******* using ***** ******* **** *** **** significant, **** ******** *** ***** ********** by **-**% **** **** ******* ** playing **** *** ******* ** *** recording ******. ********* ********* ** ***** the **** *** **** **********, *** a *************** ******* ******.
Test ********* ****** ********
***** ** *** ******* ***** ** Intel's **** ******* **** ***** ******* users ** **** ********* ****** ******** patches ** ********** ********. ***** *** cannot ***** **** ** * ***-********** environment *** **** ** **** *** the ******* **** ******* ** ******** risk ** ************* *******.
Manufacturer ********
**** *** **** ** ************* **** VMS ************* ********* *********** ******* **** patches. ** ***, ************* **** ********* **** are ***** ** *** ******* ** determining ******* *********** ******, *** *********** approaches *** ******** *******.
It is hard to say with limited data being released so far. In some cases, you may be able to upgrade to a faster processor. Similarly, upgrading to faster RAM (if possible) may also help.
Until VMS manufacturers can issue some solid guidance on this, I would limit investments in upgrading components, or servers, as much as possible. It could turn out that server upgrades are the most practical approach for older hardware, as you would likely want to leave a decent chunk of free CPU to absorb any future enhancements that also add to processor utilization.
Also, this has not been addressed so far, but if the server is supporting multiple applications (e.g.: VMS server and also mobile gateway server, etc.), it may be cost-effective to off-load some applications to other hardware to free up CPU for the core VMS.
With Milestone 2018 R1 you have the option to add one or more NVidia cards to a client machine to increase processing power. Milestone automatically load balances the GPU's and uses CPU for leftover needs. Due out Feb 20, 2018
With Milestone 2018 R2, you have the option to drop one or more NVidia cards into your recording servers to offload the decompression tasks. This is not due out till mid-summer so a bit of a gap for those patching now, but at least it is a real and relatively low-cost option that we have.
Why are these exploits even an issue in a closed environment such as a VMS server? No one is going to be using one to surf the net or adding random apps, or am I missing something?
Unfortunately from the integrator's point of view, security networks are not so closed environments anymore.
IT departments are pushing towards opening security networks to other segments of the network, and sometimes yes they are open to the Internet as well either for remote assistance, remote monitoring or connecting remote sites.
Impact on server load when viewing live or playback video was even more significant, increasing CPU load by 80-90% on low camera count systems using Avigilon and Exacq
Forgive my low reading comprehension but am I understanding this right? I'm hoping what is meant is that if we're at 10% load it is jumping to 18%-19%.
Yes, you were reading it correctly.
For example, the exacq server in the low throughput test increased by 4% cpu usage, going from 8% pre-patch to 12% post patch, this was a 50% increase.
The problem right now with BIOS updates is Intel released code that is causing a reboot problem. Dell, HP, and Lenovo have all removed the latest BIOS patches from their downloads and are recommending anyone who installed them to roll back. If you recently patched your BIOS, you should go back and check to be sure you didn't install a version that has been retracted.
Dell and HP purge Intel’s Spetcre patch over reboot woes
“We recommend that OEMs, cloud service providers, system manufacturers, software vendors and end users stop deployment of current versions, as they may introduce higher than expected reboots and other unpredictable system behavior,” he wrote.
Purely from a testing methodology standpoint, was testing done with Milestone Smart Client hardware acceleration enabled? It could stand to reason CPU usage and impact would be less than this if GPUs would being used to decode and render...
Don't suppose you can build an AMD Ryzen machine and retest this?
Intel has done a great job convincing everyone that Meltdown affects everyone despite AMD being entirely unaffected by this. So I'd be super curious to find out how much the Meltdown patch affects the speed vs Spectre.
We are currently evaluating this situation and discussing with customers whether or not to actually applying the BIOS patches.
The reason for this is that we want the customers to have a choice. Many of our customers have distinctly disconnected networks for CCTV and very high control over what can be installed on their machines, like web browsers.
To do this we need to call on our corporate security experts to look at the attack vectors and recommend actions.
With these high impacts we are seeing, many of our customers will have to do expensive and time consuming upgrades if there is no way around the BIOS patches.
This BIOS vulnerability, can it affect dedicated Linux OS recorders/NVR's with Intel chips or only Windows/Intel devices?
How about the Meltdown vulnerability on these same devices?
These are processor vulnerabilities, so they can affect any OS really.
Linux can certainly be affected, and Linus Torvalds has been a bit critical of Intel on this.
Brian,
From what I have been reading it appears a much smaller set of ARM chips are affected as well. Does this have any potential impact to embedded DVRs/NVRs or does their embedded nature prevent this?
Yes, some ARM chips are affected also. Embedded recorders of the sort that do not allow you to load external software/apps/etc., and instead distribute software as complete firmware images are less likely to be infected. A caveat of course being that you do not have things like telnet, ftp, ssh, open where an attacker could upload a binary directly to the system.
Currently exploits rely in some malicious software being executed on the device that is designed to leverage these vulnerabilities. (note it may not technically need to be installed, as there have been some browser-based PoC's). With most embedded NVRs you do not have a way to load random software or even execute a browser, so they are safer than PC-based systems in that regard.
Also keep in mind the above is based on current knowledge of the vulnerabilities and exploits. As more details about these vulnerabilities circulate among black-hat hackers there could be new exploits devised that put embedded NVRs at greater risk.
To my reading and understanding, this is a bit hyped... sure it's serious bug, when you have someone or something executing on your host.
The biggest drama out of this is not the bug itself but the resultant performance of the CPUs post fix.
In Intels case, testing has revealed a decrease of 25% performance I/O intensive tasks. This is where the real issue with Meltdown and Spectre lies.
I tried to reproduce it myself.
Realized that my NUC (nuc7i3bnh) doesn't have Meltdown/Spectre fix available on the Intel website (latest Bios update is Nov 2017).
Tried to find your firmware (RYBDWi35.86A.0368.2017.1220.0950) and failed again.
The only link from Google leads to 404 https://downloadmirror.intel.com/27426/eng/RY_0368_ReleaseNotes.pdf
If check downloads for your model (NUC5i7RYH)
https://downloadcenter.intel.com/product/87570/Intel-NUC-Kit-NUC5i7RYH
the latest Bios update is dated September 2017.
Looks like Intel takes down the Bios updates as there is too many issues with the upgrade.