VMS Server Load Fundamentals Tested

By Ethan Ace, Published Feb 11, 2014, 12:00am EST (Research)

How powerful of a server do you need to run VMS software? What drives VMS server load?

There's manufacturer recommendations but, until now, there's has never been an independent test of VMS server load.

In this report, we share our test findings experimenting with 4 VMSes (Exacq, Milestone, Axxon and Network Optix/DW) and an examination of:

  • Camera count impact
  • VMD impact (full stream, substream and key frame only included)
  • Live viewing impact 
  • Overall, there are three key recommendations that come out of this test:

    • CPU > RAM: When building VMS server machines, money is better spent increasing processing power than adding huge amounts of memory. RAM was a factor only when advanced analytics were running on all frames.
    • VMD Impact: When possible, server side motion detection should be run on secondary, lower quality streams, or keyframes only to reduce CPU usage. We recommend running motion detection on high quality or full framerate streams only when performance issues are a factor.
    • Separate viewing: Viewing should be performed on a machine separate from the VMS server unless the PC is upsized with faster processing and dedicated graphics to handle this increased load. Processor usage of our low end test machine maxed at 100% at very low camera counts.

    Note that this is not an exhaustive test of all VMSs, and performance of others may vary. Users should always benchtest server configurations prior to any field deployment.

    Key Findings

    A few key trends came out of our server load testing of four VMSs:

    • Per camera load was an average of ~1% and under when running recording only, no viewing, and no motion detection or analytics on our test server (dual quad core i7, 16 MB RAM).
    • RAM usage remained close to constant regardless of number of cameras attached. Motion detection only increased memory usage in Axxon Next, and only when detecting on all frames instead of only key frames. No other VMSs showed a steady increase in RAM usage.
    • Increases due to server-side motion detection and video analytics varied widely by VMS and exact method used (all frames, key frames only, secondary stream vs. main stream, etc.).
    • Increases were low (~1% CPU usage per camera or less) when performing motion detection on key frames only (or using a detection interval of 1000ms in the case of Axxon).
    • Analyzing motion on all frames resulted in much higher consumption, resulting in ~4-6% increase in CPU per camera over running no server side detection.
    • CPU impact when using a secondary lower resolution stream for motion detection was much lower than using high resolution streams, about a 1% increase per camera.
    • Viewing client impact was moderate on our high end test machine, varying from no noticeable increase in CPU utilization to about a 17% increase (from 10% to 27% total CPU utilization) across 12 cameras at 720p30, depending on the VMS.
    • However, CPU usage reached 99-100% at only five cameras when using smaller test machines typical of those used for recording in small systems (dual core AMD, 1.6 GHz, 2GB RAM).

    Per Camera Impact

    We began our test by adding cameras one at a time to each server, using both low end and high end machines:

    High End Test Server

    On average, adding a single camera increased load by approximately 1% when no local viewing, motion detection, or analytics were running. The actual observed average ranged from as little as 0.3-0.4% to ~3%. Load per camera scaled down slightly as camera count increased in some cases, with larger increases for the first or second camera (as high as 3-4%), but less so for cameras 11 and 12 (closer to the 1-2% average).

    Increases in Exacq were small per camera, from 1% with one camera added, to 5% when adding 12. Axxon Next runs separate database services, accounting for a higher starting load  (~5%), but per-camera increases are still low, less than 1% per camera for a total of 10%.

    Low End Mini PC

    Increases per camera were greater when using our AMD powered mini PCs. On average, each camera added between 1-2% CPU usage on this lower powered machine:

    Server Side VMD Load

    We tested server-side video motion detection using three different VMSs, each with different options for processing video.

    All Frames vs. Key Frames

    Analyzing motion on all frames resulted in much higher CPU consumption than only key frames (a setting allowed by Milestone and Axxon). This graph visualizes this drastic increase across eight cameras using Milestone Enterprise for no motion, keyframe only motion detection, and all frames.

    Secondary Stream Analysis

    Both Axxon and DW Spectrum allow motion to be analyzed using a lower resolution secondary stream, instead of the full resolution high quality stream, reducing CPU consumption. Using this feature in DW Spectrum, for example, results nearly no increase in CPU consumption for server-side motion detection on all cameras. The increase when using the high quality stream is much higher, ~11% across 12 cameras. This chart compares CPU consumption for both low and high quality streams in DW Spectrum:

    Viewing Client Load

    We tested VMSs on both our higher end i7 machine, as well as a lower end dual-core AMD PC. Each was tested with their viewing client open, viewing all connected cameras, to check how this affected system load.

    High End Server Viewing Performance

    Impact on the higher end machine, with its greatly increased processor and dedicated graphics card was moderate at most, with numbers varying by VMS. Exacq had the lowest viewing overhead, increasing only a few percent, while Axxon and DW Spectrum both increased substantially.

    Mini PC Viewing Performance

    However, using our smaller mini PCs, CPU usage with local viewing maxed out at 100% after only 4-5 cameras with Milestone and Exacq, clearly emphasizing the need for separate viewing client machines in even small systems.

    Note that these numbers may vary, depending on VMS client settings, server hardware, and video card performance. This is intended only as a general indication of how load increases using default settings, not a definitive comparison.

    Test Procedure 

    When testing load, we took the following steps to ensure measurements were as accurate and repeatable as possible:

    • Our test PC was rebooted, with no VMS services started automatically, before each test, to produce an accurate baseline CPU consumption metric.
    • Once booted, services for the individual VMS were started. This includes both specific VMS recording services, as well as any database or other background services required, as these contribute to load, as well.
    • Measurements were taken with no cameras added, to get a baseline of performance with no archiving taking place.
    • Cameras were plugged in and configured in the VMS one at a time to explicitly test impact of individual streams as they were added.
    • This process was repeated for each VMS, as well as tests of different VMS functions (VMD, keyframe vs. all frame processing, etc.), with all cameras removed and re-added in the same order.
    • The laptop and cameras were isolated on a separate switch, with no connection to any other networks.

    Test Machine Specs

    High End:

    • Operating System: Windows 8 Pro 64-bit
    • Processor: Intel(R) Core(TM) i7-3630QM CPU @ 2.40GHz (8 CPUs), ~2.4GHz
    • Memory: 16384MB RAM
    • Graphics card: Nvidia Geforce GT 650M, 2GB

    Low End:

    • Operating System: Windows 7 Pro 64-bit
    • Processor: AMD G-T56N Processor (2 CPUs), ~1.6GHz
    • Memory: 4096MB RAM
    • Integrated graphics only, no dedicated GPU

    VMS Version Information

    The following versions of each VMS were used in this test:

    • Axxon Next: 3.5
    • DW Spectrum:
    • Exacq:
    • Milestone: 2013 R2

1 report cite this report:

Milestone XProtect 2014 Tested on Sep 03, 2014
In this third entry in our ongoing VMS test series we provide in-depth...

Comments (31)

Only IPVM Subscribers may comment. Login or Join.

This is the first of our server load testing, and most fundamental. We know we've gotten many questions about this topic and many of you are excited about this and more advanced aspects. Let us know what you'd like to see in the comments.

Great start, looking forward for part two

Why don’t you have all four VMSs on the same chart for the CPU utilization - the low end PC and the High end PC? And, the same thing for the local viewing performance?

Why did you use these vms? What drove the selection ?

Josh, these were the VMSs we've had installed for prior tests, and we felt they covered a good part of the spectrum, with various performance, viewing, VMD differences.

Great test and a good start if you are going to do more.

One thing, just to be certain, it sounds like for the high end you used a machine with (2) physcial CPU's, each with quad cores, is that correct?

Also, as to RAM, you might want to list which VMS's use the operating system's file system for database files, which I know Exacq does, and which uses actual database container files, like some version of SQL or other native version. I have a feeling that would affect RAM requirements, or maybe not as further testing may or may not show.

Off the top of my head, I'm going to agree with you. Simply because the CPU/RAM impact of those databases must be accounted for when considering total CPU load. This is why we did not simply record the consumption of specific services, but instead took overall system load, since many of these VMSs required multiple services of their own plus database services in order to operate.

And yes, two quad core CPUs.

Right on-- any way in the future we can do a bench test on all VMS Client / Server i would like to understand independtly how Genetec, Milestone, Aviglon, Exacq , Lenel etc stack up from a server / client hardware usage and it would be nice to do a review on failover capabilities- how well they work etc.

Good work guys!

Hi Ethan, Your chart for vms load with client running doesn't match your written results right above it. In the paragraph above it you attribute lowest CPU load result to DW Spectrum but in the chart you credit Exacq. Which one was which?

I've clarified that paragraph. DW Spectrum was had the lowest increase in CPU utilization when the client was running. Exacq was lower overall.

That being said, we really are not aiming to make this a shootout, but that chart lends itself to that. I'm editing to show the difference between client open/client closed CPU usage for each individually instead.

Incredible article Ethan, I think this highlights the importance of head end hardware and infrastructure on large systems, obviously, with more cameras and higher resolutions, the problem is compunded.

I didn't realise that you could use motion detection on key frames and separate resolutions, which is quite interesting.

In the past I have used this link as a guide for CPU's in servers and workstations, and found that display cards don't necessarily play a huge role in the processing of video (about 20%). We have used some high end Nvidia Quadro cards in the past but found that video streams (2D) require more CPU processing to unpack the H.264 compression, and the high end cards (for 3D modelling etc) need to load the file in memory, beside the fact that quite a few features in high end cards require the software used, to 'unlock' them (such as Adobe etc). This might help to an extent, but not in the way Dual CPU's or CPU's with more cores do.

You might find that choosing CPU's carefully would allow for dramatic increases in performance. IE: - One Intel Core i7-4930 could do the load of your dual CPU machine (at a best price).

Did you use Axxon's 'Green Stream' as part of the test ? I think it's simliar to DW, in that you can use a lower quality stream, when you have split tiles over multiple camera layouts

I know that I've also (on occasion) had to record with H.264, but streamed live viewing with the MJPEG to assist with CPU loads. Perhaps it could also be nice to test.

Great article. I am curious for the graph in this section : High End Server Viewing Performance

You mentioned the best is DW spectrum (no processor usage) but your graph shows Exacq is the best. Does your graph show a wrong legend perhaps?

I have been doing independent testing of VMS perfomance on hardware in our lab for a few years now and have amassed quite a bit of data on Server side AND Client side metrics. Our sales staff has access to this data.

For the server side, we have done Milestone Corporate, ONSSi CS, Genetec SC (Omnicast) and a couple of OEMs.

The server side has many peculiarities as you start to push beyond 100Mbits throughput. What happens is that the CPU bottleneck becomes less important than a bottleneck in another piece of hardware...usually the HDD or NIC paths depending on the VMS.

The Clients side are the above VMS plus Pelco Control Point.

As onserved on the Client side...the CPU is king, but one DOES have to make available a good video card for the VMSs that want to use it. The new Haswell CPUs actually do a decent job at the low levels, and if you need a bit more boost go to the K600 then the K2000 cards.

Eventually the client application itself simply can not handle the 'housekeeping' needed for a larger number of streams and slows down.

"The server side has many peculiarities as you start to push beyond 100Mbits throughput. What happens is that the CPU bottleneck becomes less important than a bottleneck in another piece of hardware...usually the HDD or NIC paths depending on the VMS."

Mike, good feedback. Thanks. That makes sense. Do you see a big variance on when/how quickly the HDD becomes a bottleneck across VMSes?

The way I start my testing does not lend itself to answering a 'rate' type question. I am typically starting at what we 'think' the maximum is and add/delete streams until the system is at it's highest stable point.

Other factors for the HDD are how fragmented a storage array is over time. When it is highly fragmented..the performance drops. Some VMSs more than others. Defragging helps a lot but this has to be a planned maintenence activity in a real installation.

TimR below mentions bridging NICs.... and one has to be careful here. Windows 7 does not truly aggregate 2 NICs and give you 2GB. In actual operation, I saw it simply switch from using one port to the other...thus still only getting 1GB.

To get a true 2GB capacity, one needs to define the NICs to be on different subnets. Doing this works well.

WinServer aggregates NICs differently, but I dont have test data on that since I use the different subnet method instead.

I would assume you could alleviate this by using a RAID configuration, however it would also depend on what drives you're using (but we should be talking Gb/s here and not Mb/s). IE - SATA, SAS or SSD, if you're using a Seagate Surveillance drive (SV), the read/write speed should be at least 6Gb/s.

My assumption would be that the bottleneck should be with the NIC, you could always add an extra card and Bridge the connections for more throughput.

*Correction ... I see the average data rate is 156Mb/s, up to 210Mb/s for a max sustained rate.

Tim... in my lab we test with RAID sets on any of our systems that we expect to support 200Mbits and beyond. Our smaller systems are the JBODs. We establish a conservative RAID5 with 4 drives even though a RAID10 with 4 drives is a better performer.... but want to be conservative. Drives are 6G SATA and/or SAS (enterprise class) depending on the VMS support. We do not see a demand for RAIDs with SSD.

Where we see SSD requests are for our mobile server since it makes sense for SSD in those environments.

I would like to clarify my own mind. When you speak of VMS, I interpret that as the video management system which encompasses any manager database and local recorders. In the tests above I am presuming its referring to the component that is actually recording the video streams.

Great feedback, thanks Mike.

Where is the encoding being done for these VSMs? Are we getting direct streams to the cameras on live view? Is the decoding done on the client or server side side for live/recorded video?

If all of the decoding is being done on the server side that is typically where you have the lower max camera limits and higher CPU usage. I believe Exacq is 96 or 128, no matter that your server specs are. Same with Lenel OnGuard video and Prism. OnGuard video is 63 and I believe Prism is 96 or 128 total cameras.


This question can only be answered with 'it depends on the VMS'. Some VMSs will not perform any encoding of the streams going out to thick remote clients while others will.

Some VMSs, like Exacq, have the Client built in to the Recording server installation, which promotes using the server as a viewstation. As observed, this Local viewing is a hit to the CPU and it is best to limit this type of viewing if you are trying to maximize the recording performance.

Additionally, one has to be careful when changing the resolution or FPS of a stream from what the Recorder is getting because this will cause the Server to transcode the stream to the requested type.

If one has a thin client (ie Web/tablet/laptop) as part of the Client infrastructure...be prepared to either limit the usage to a small number (less than 10 streams) or install this service on its own machine. This service is a BIG hit to the CPU of the machine and if it is running on the recording server, then there will be loss of data being stored because the CPU can not keep up with the 'transcoding' task and the recording task at top speed.

For systems that have just a small number of streams (say 16, or less than 100Mbits) , it is possible to size a system to be able to handle the local viewing and the recording. We do this sort of sizing in the lab for our customers that need to know what to build for the VMS that will be used.

Nice report! Just wondering if you have compare the CPU usage between single channel and dual channel memory (or even quad channel)? It should make great difference

In addition, the VMS that supports Intel media SDK should help to decrease the CPU loading whe doing video decoding

I'll add some real world stats to this based on our experience with Avigilon.

The following graph is pull from our N-Central Management System from Three servers at a client. Each Server is the Avigilon Branded OEM Dell R710 Server running windows xp64Bit, or Windows7 Embedded. Each server is running the Dell Perc6i, each with 6 Sata 7200K WD enterprise (black Drives) By default Avigilon Ships with a Raid5 (with spare) and they build a 75GB C: volume for the OS and the remaining RAID space is a single volume preallocated to all video...

The heavier load is about 50 JP2000 (1MP or better)Cameras 30 which are 4 sensor dome 180 or 360 domes.

the next heavier is about 25 JP2000 1/2 being Dome 180/360, and remaining are 1MP or better

the next is roughly the same as before, but has about 1/2 being H264

In the Avigilon World, Streams are counted individually. Instead the total Bandwidth is considered the MAX.

125 Cameras or (32Mbytes/Second or 256Kbits/second) is the stated maximum on this server. CPU (wmi) never exceed 50%

We know for a fact with 90 cameras, all Jp2k domes, many being dome180/360 we have seen this server configuration support 35MBytes Second, 24x7 for 6 months, prior to installing a second server and relieving the load.

3-Aviligon Servers

Our own office system is a small embedded windows 7 box with 2GB of ram, and a 750GB 2.5" WD black, and we push 80Mbits/Second across our Jp2000/and H.264 cameras....

It isn't all about CPU. In our IT support business we measure and predict reliability by watching the ratio of WMI (committed memory / maximum memory) in the servers above that value is about 20% and remains flat indefinitely.

Andrew, thanks! That's very helpful and makes sense. It looks like you are easily managing lots of cameras on these boxes.

Related, you might find this discussion interesting or be able to shed some light on it: 32 NVRs Needed For ~1000 Avigilon Cameras?


How did you make sure that different clients display same megapixels per second while doing viweng client load tests comparing the cpu usage?

16 camera Exacq system (4x D1 analogs from an encoder - remainder are 1.3MP and 2MP) running on an Atom D510. It is playing double duty as a Ubiquiti Unifi AP manager as well. CPU load is pretty stable at 5%.

97 camera Exacq system (10x 3MP + 87x 2MP) running on a single Xeon E5-2650. It usually runs between 10 and 20% CPU.

25 camera Exacq System (5x 5MP + 3x 3MP + 17x 2MP) running on a Core i3-3240. Most cameras are dual streaming a second 2CIF stream (46 total streams). CPU is around 6%.

First. Linux FTW!

Second. It really makes me question how they come up with the recommended specs.

One of the challenges with performance testing is what to use as the definition of a 'load' on the system and how to report the result. Since there is no 'standard', we can all use what we think is best.

As can be seen in this discussion...there are folks who simply use 'camera count', others who refer to the Client running on the server and others who talk about throughput...and even these are a mix of 'system level' and NIC references.

Back when I started out on my VMS performance testing mission several years ago, I asked several integrators who were buying our servers what they would like to see. Their answer was a 'system throughput' value doing constant recording and a camera load was a 1Mbyte/sec D1/VGA MJPEG stream.

This combo helped them 'think' about how many cameras this was and the math was fairly straightforward since it used values that the analog folks were used to working with. I still test servers doing the recordings this way and when I introduce client operations, I use H264 streams of 2Mbit size.

Perhaps it is time that a 'defacto standard' be developed so that a comparison of system performance can be done more easily. Anyone for another 'Up To' spec? Perhaps this should be a discussion on its own?

Hi guys,

Thanks for the informative testing report.

Just got 2 questions if I am not too late to ask about it, since it is already in June.

1) Having MD configured on camera itself and transferred the video streaming to the VMS would it reduce the CPU consumption on the high end PC compare to MD running and handle by the VMS Server itself ?

2) The resulting graph seems to show that Exacq VMS is most efficient VMS, since it shows the least CPU comsumption, am I right?

Thanks again.

You always want to do MD on the camera side. To my understanding CPU consumption on the client side (I assume that is what you mean by PC) is determined by the architecture of the VMS software; is the decoding/encoding done on the server side or client side? I've found that 90% of salesman do not know the answer to this and the manufacturer wont always give a straight answer. If the decoding is done on the client side the client specs will be higher.

Regarding VMD location, see: Motion Detection: Camera vs. Server-Side

Marcus, as to whom was most efficient, this was not a shootout, so we are refraining from picking a 'most efficient VMS'.

I know this is an older article, but how do you get Milestone to use keyframes only on VMD? I don't see that as an option anywhere. Is this an Enterprise and up only feature?


Informative: 2
Loading Related Reports