VMS Server Load Fundamentals Tested

This is the first of our server load testing, and most fundamental. We know we've gotten many questions about this topic and many of you are excited about this and more advanced aspects. Let us know what you'd like to see in the comments.

Great start, looking forward for part two

Why don’t you have all four VMSs on the same chart for the CPU utilization - the low end PC and the High end PC? And, the same thing for the local viewing performance?

Great test and a good start if you are going to do more.

One thing, just to be certain, it sounds like for the high end you used a machine with (2) physcial CPU's, each with quad cores, is that correct?

Also, as to RAM, you might want to list which VMS's use the operating system's file system for database files, which I know Exacq does, and which uses actual database container files, like some version of SQL or other native version. I have a feeling that would affect RAM requirements, or maybe not as further testing may or may not show.

Right on-- any way in the future we can do a bench test on all VMS Client / Server i would like to understand independtly how Genetec, Milestone, Aviglon, Exacq , Lenel etc stack up from a server / client hardware usage and it would be nice to do a review on failover capabilities- how well they work etc.

Good work guys!

Incredible article Ethan, I think this highlights the importance of head end hardware and infrastructure on large systems, obviously, with more cameras and higher resolutions, the problem is compunded.

I didn't realise that you could use motion detection on key frames and separate resolutions, which is quite interesting.

In the past I have used this link as a guide for CPU's in servers and workstations, and found that display cards don't necessarily play a huge role in the processing of video (about 20%). We have used some high end Nvidia Quadro cards in the past but found that video streams (2D) require more CPU processing to unpack the H.264 compression, and the high end cards (for 3D modelling etc) need to load the file in memory, beside the fact that quite a few features in high end cards require the software used, to 'unlock' them (such as Adobe etc). This might help to an extent, but not in the way Dual CPU's or CPU's with more cores do.

You might find that choosing CPU's carefully would allow for dramatic increases in performance. IE: - One Intel Core i7-4930 could do the load of your dual CPU machine (at a best price).

Did you use Axxon's 'Green Stream' as part of the test ? I think it's simliar to DW, in that you can use a lower quality stream, when you have split tiles over multiple camera layouts

I know that I've also (on occasion) had to record with H.264, but streamed live viewing with the MJPEG to assist with CPU loads. Perhaps it could also be nice to test.

Great article. I am curious for the graph in this section : High End Server Viewing Performance

You mentioned the best is DW spectrum (no processor usage) but your graph shows Exacq is the best. Does your graph show a wrong legend perhaps?

I have been doing independent testing of VMS perfomance on hardware in our lab for a few years now and have amassed quite a bit of data on Server side AND Client side metrics. Our sales staff has access to this data.

For the server side, we have done Milestone Corporate, ONSSi CS, Genetec SC (Omnicast) and a couple of OEMs.

The server side has many peculiarities as you start to push beyond 100Mbits throughput. What happens is that the CPU bottleneck becomes less important than a bottleneck in another piece of hardware...usually the HDD or NIC paths depending on the VMS.

The Clients side are the above VMS plus Pelco Control Point.

As onserved on the Client side...the CPU is king, but one DOES have to make available a good video card for the VMSs that want to use it. The new Haswell CPUs actually do a decent job at the low levels, and if you need a bit more boost go to the K600 then the K2000 cards.

Eventually the client application itself simply can not handle the 'housekeeping' needed for a larger number of streams and slows down.

I would assume you could alleviate this by using a RAID configuration, however it would also depend on what drives you're using (but we should be talking Gb/s here and not Mb/s). IE - SATA, SAS or SSD, if you're using a Seagate Surveillance drive (SV), the read/write speed should be at least 6Gb/s.

My assumption would be that the bottleneck should be with the NIC, you could always add an extra card and Bridge the connections for more throughput.

*Correction ... I see the average data rate is 156Mb/s, up to 210Mb/s for a max sustained rate.

I would like to clarify my own mind. When you speak of VMS, I interpret that as the video management system which encompasses any manager database and local recorders. In the tests above I am presuming its referring to the component that is actually recording the video streams.

Where is the encoding being done for these VSMs? Are we getting direct streams to the cameras on live view? Is the decoding done on the client or server side side for live/recorded video?

If all of the decoding is being done on the server side that is typically where you have the lower max camera limits and higher CPU usage. I believe Exacq is 96 or 128, no matter that your server specs are. Same with Lenel OnGuard video and Prism. OnGuard video is 63 and I believe Prism is 96 or 128 total cameras.

Nice report! Just wondering if you have compare the CPU usage between single channel and dual channel memory (or even quad channel)? It should make great difference

In addition, the VMS that supports Intel media SDK should help to decrease the CPU loading whe doing video decoding

I'll add some real world stats to this based on our experience with Avigilon.

The following graph is pull from our N-Central Management System from Three servers at a client. Each Server is the Avigilon Branded OEM Dell R710 Server running windows xp64Bit, or Windows7 Embedded. Each server is running the Dell Perc6i, each with 6 Sata 7200K WD enterprise (black Drives) By default Avigilon Ships with a Raid5 (with spare) and they build a 75GB C: volume for the OS and the remaining RAID space is a single volume preallocated to all video...

The heavier load is about 50 JP2000 (1MP or better)Cameras 30 which are 4 sensor dome 180 or 360 domes.

the next heavier is about 25 JP2000 1/2 being Dome 180/360, and remaining are 1MP or better

the next is roughly the same as before, but has about 1/2 being H264

In the Avigilon World, Streams are counted individually. Instead the total Bandwidth is considered the MAX.

125 Cameras or (32Mbytes/Second or 256Kbits/second) is the stated maximum on this server. CPU (wmi) never exceed 50%

We know for a fact with 90 cameras, all Jp2k domes, many being dome180/360 we have seen this server configuration support 35MBytes Second, 24x7 for 6 months, prior to installing a second server and relieving the load.

Our own office system is a small embedded windows 7 box with 2GB of ram, and a 750GB 2.5" WD black, and we push 80Mbits/Second across our Jp2000/and H.264 cameras....

It isn't all about CPU. In our IT support business we measure and predict reliability by watching the ratio of WMI (committed memory / maximum memory) in the servers above that value is about 20% and remains flat indefinitely.

Ethan,

How did you make sure that different clients display same megapixels per second while doing viweng client load tests comparing the cpu usage?

16 camera Exacq system (4x D1 analogs from an encoder - remainder are 1.3MP and 2MP) running on an Atom D510. It is playing double duty as a Ubiquiti Unifi AP manager as well. CPU load is pretty stable at 5%.

97 camera Exacq system (10x 3MP + 87x 2MP) running on a single Xeon E5-2650. It usually runs between 10 and 20% CPU.

25 camera Exacq System (5x 5MP + 3x 3MP + 17x 2MP) running on a Core i3-3240. Most cameras are dual streaming a second 2CIF stream (46 total streams). CPU is around 6%.

First. Linux FTW!

Second. It really makes me question how they come up with the recommended specs.

One of the challenges with performance testing is what to use as the definition of a 'load' on the system and how to report the result. Since there is no 'standard', we can all use what we think is best.

As can be seen in this discussion...there are folks who simply use 'camera count', others who refer to the Client running on the server and others who talk about throughput...and even these are a mix of 'system level' and NIC references.

Back when I started out on my VMS performance testing mission several years ago, I asked several integrators who were buying our servers what they would like to see. Their answer was a 'system throughput' value doing constant recording and a camera load was a 1Mbyte/sec D1/VGA MJPEG stream.

This combo helped them 'think' about how many cameras this was and the math was fairly straightforward since it used values that the analog folks were used to working with. I still test servers doing the recordings this way and when I introduce client operations, I use H264 streams of 2Mbit size.

Perhaps it is time that a 'defacto standard' be developed so that a comparison of system performance can be done more easily. Anyone for another 'Up To' spec? Perhaps this should be a discussion on its own?

Hi guys,

Thanks for the informative testing report.

Just got 2 questions if I am not too late to ask about it, since it is already in June.

1) Having MD configured on camera itself and transferred the video streaming to the VMS would it reduce the CPU consumption on the high end PC compare to MD running and handle by the VMS Server itself ?

2) The resulting graph seems to show that Exacq VMS is most efficient VMS, since it shows the least CPU comsumption, am I right?

Thanks again.

I know this is an older article, but how do you get Milestone to use keyframes only on VMD? I don't see that as an option anywhere. Is this an Enterprise and up only feature?