Intel Meltdown / Spectre Patch Tested on Avigilon, Exacq and Milestone VMSes

Published Jan 23, 2018 17:47 PM

Significant concern exists about the impact on VMS servers and video surveillance systems of patching the Meltdown and Spectre flaws. Generally, reports have claimed a significant increase in CPU loads, though the amounts have varied depending on the source and services being used. This could be very problematic for VMS servers as it risks problems in recording, watching or playing back video.

Meltdown / Spectre Summary

Meltdown is a flaw in protections allowing unauthorized applications to access OS kernel protected memory. Spectre allows unauthorized applications to trick other programs into divulging private data. For technical details on these, see The Meltdown and Spectre CPU Bugs, Explained [link no longer available] or various other online tech sources.

How To Patch

Users may see if their Windows machine is vulnerable using PowerShell, following the steps in this guide

If so, in order to avoid both of these vulnerabilities, users must install two updates:

  • Windows Update KB4056892: This update should have been included in regular Windows Updates as of early January. This fixes the Meltdown vulnerability.
  • BIOS update: To fix the Spectre vulnerability, users must update their system BIOS, provided by their hardware manufacturer.

However, many devices have not yet received a BIOS update. Out of four different hardware manufacturers we checked (Acer, Asus, HP, and Intel NUCs), only Intel had released a BIOS update for our machines. 

Tests Performed

IPVM measured CPU load of server and client processes prior to applying Windows and BIOS updates, and again after updating, using two configurations:

  • Recording only, 25 cameras, ~150 Mb/s
  • Recording and viewing, 10 cameras, ~30 Mb/s

Test Machine Specs

IPVM tested on three separate machines, all with the same hardware configuration, as follows:

  • Operating System: Windows 10 Pro 64-bit (10.0, Build 16299) (16299.rs3_release.170928-1534)
  • BIOS: RYBDWi35.86A.0350.2015.0812.1722 (pre-update)/RYBDWi35.86A.0368.2017.1220.0950 (post-update)
  • Processor: Intel(R) Core(TM) i7-5557U CPU @ 3.10GHz (4 CPUs), ~3.1GHz
  • Memory: 8192MB RAM

Note that these machines are fairly high spec, with relatively new processors (1-2 generations old). Older generation processors and lower spec machines are likely to see a greater impact.

Inside this report, we examine the results achieved across Avigilon, Exacq and Milestone plus our recommendations.

Test Summary

In our tests, updating system BIOS to patch the Spectre vulnerability had a significant impact on CPU load, increasing CPU load of recording services by as much 25-30%. Impact on server load when viewing live or playback video was even more significant, nearly doubling CPU load on low camera count systems using Avigilon and Exacq (an increase of 15-16% load). Applying Windows' security update to patch Meltdown alone had a negligible effect, increasing load less than 1%. 

During testing, we saw no ill effects from these patches/security updates, such as dropped frames, slow loading video, smearing of video, or other VMS malfunctions which may occur when overloaded. However, note that the highest CPU load seen in IPVM testing was still below 50% utilization. Those with higher throughput systems or lower spec servers may see issues at these camera counts/throughputs.

Recommendations

Because of the significant impact of BIOS updates on CPU load, we strongly recommend that users of higher throughput/higher camera count systems should check system CPU headroom prior to updating. Those currently running at 60-70% load are likely to increase to 90-100%, with the potential for errors much higher. 

Those with low camera count/throughput systems are likely to see little performance difference unless also viewing locally on the server machine.

Intel Still Refining Patches - Reboot Issue

Patches for some systems have caused reboot issues on patched machines (note: IPVM did not experience this, but it can be hardware dependent). Intel released a statement on January 22nd that they are still refining BIOS patches and will likely have updated patches soon. Current indications are that the updated patches will have roughly similar performance impacts, and Intel is recommending partners to continue to test with the current patches to get more data on performance impacts. 

Biggest Impact: BIOS Update

In order to fully repair the Meltdown and Spectre vulnerabilities, users must apply both Windows updates and BIOS patches. Of these two, the BIOS update had a significant impact on CPU load, while Windows Updates had minimal impact, only about 1% on all VMSes tested, both recording and viewing.

Significant Impact On High Throughput Load

In our tests of 24 camera systems, ~150 Mb/s total throughput, server process CPU load increased significantly after patching the BIOS of test servers, with ExacqVision Professional increasing by ~7% and Milestone XProtect Corporate by ~11%.

Note that we observed no issues in performance in either VMS during testing (dropped frames, slow loading, smearing/tearing video, etc.), but those with higher throughput may see issues as load jumps after updating.

Low Throughput Increases Similar

Testing with 10 cameras at about 30 Mb/s total, server load CPU usage increases were small, but proportionately larger than 24 camera testing, with Exacq increasing by 4% load (a 50% increase) and Milestone by 5% (a ~70% increase). However, Avigilon Control Center jumped significantly, from ~4% to 8%, but as with higher throughput testing, these increases did not result in noticeable performance issues.

Live View Drastic CPU Impact

CPU load increases for those systems using local viewing were far more significant, with Avigilon and Exacq increasing by 15-16% when live viewing or playing back ten cameras on the recording server. Milestone increased by about the same CPU load percentage, but a proportionately smaller amount.

 

Test Carefully Before Patching

Based on the current state of Intel's BIOS patches IPVM would caution users to test carefully before applying patches to production machines. Users who cannot fully test in a non-production environment may want to wait for the updated BIOS patches to minimize risk of unanticipated reboots.

Manufacturer Feedback

IPVM has been in communication with VMS manufacturers regarding performance impacts from patches. So far, manufacturers have indicated they are still in the process of determining overall performance impact, and recommended approaches for applying patches.

Comments are shown for subscribers only. Login or Join