How Should IPVM Test VMS Server Performance?

JH
John Honovich
Oct 08, 2013
IPVM

IPVM is going to start testing VMS server performance. Obviously, there are a lot of factors that come into play (CPU, RAM, motion detection - on or off, number of cameras, resolution, bitrate, to name just a few).

This discussion is to gather feedback from the community about how we should do it. If you are a manufacturer, integrator or a technologist with an opinion here, please speak up now.

We will be testing a number of VMSes, with the goal of better understanding key performance drivers.

A few immediate questions:

  • What hardware should we use? Certain brands? Certain components?
  • What cameras should we use?
  • What issues should we beware of?

If there are things that you find essential, let us know.

It will be impossible to make everyone happy as there are far too many possible combinations. However, we want to have a reasonable set of equipment and tests to cover the most common use cases.

JC
Jason Clement
Oct 08, 2013

What's the server load going to look like ie number of cameras and clients viewing live video?

JH
John Honovich
Oct 09, 2013
IPVM

There's a lot there - How many cameras? What resolution for the cameras? What birate for the cameras? Will the bit rate change (VBR - spikes at night for higher traffic scenes)? Does it matter which brand or model of manufacturer?

To start, we are going to try out a bunch of combinations, measure the results and see what patterns are uncovered.

JC
Jason Clement
Oct 09, 2013

I would definitely recommend Dell but there's so many different combinations. I guess a 520 with a single quad core or six core processor would be about an average build for 40-80 cameras depending on the VMS and whether server side motion was used. A typical supermicros server build would be good to test on as well. Other than that I'm not super familiar with any of the other server manufaturers out there.

As far as cameras most people have a mix of 720p and higher res cameras but it would be coool to see a server loaded down with 5 MP cameras as to what that would do with performance especially if you have server side motion turned on and ahigh frame rate.

RAID performance would be good to see too I think. RAID 10 vs RAID5 vs RAID6.

There's so much like you said it's hard to get it all in :)

JH
John Honovich
Oct 09, 2013
IPVM

Dell or HP strike me as safe choices since they are so commonly used. I am not as sure about quad or six core processors as most NVR appliances seem to run Atoms or i3/i5/i5 dual core CPUs.

JC
Jason Clement
Oct 09, 2013

True but when you buy like a R520 server, which is pretty common, I believe the lowest end processor you cna buy is a sandy bridge quad core hyper threading processor. It's been about four months since I spec'd one though so I may be wrong.

DM
Duncan Miller
Oct 09, 2013

What do you think about using VMWare to test and measuring the performace of the VMS systems?

JH
John Honovich
Oct 09, 2013
IPVM

Definitely not virtual machines, only because it's going to cause a lot of objections about performance impact. It would certainly make it easier to swap VMSes in and out but almost certainly manufacturers will cry foul if their products do badly, blaming it on the virtual machine.

DM
Duncan Miller
Oct 09, 2013

Good point. I think Dell or IBM Server would be a good pick but then again you might be better off picking a Server board from Tyan or Supermicro to keeps your costs down as I don't think the motherboard will make a big difference on the end results. I would probably try to use a lower end CPU so the cpu loads are noticiable between VMS's.

It would be interesting to see the difference multiple cores makes on the different VMS to see if the VMS systems are acually taking advantage of the multiple cores.

How are you gong to pick which OS to use? I am assuming you are going to have to keep that consistant between comparisons. Will you use or 64bit OS or 32bit OS? Server OS or Desktop OS? What if the VMS is only a 32bit application?

JH
John Honovich
Oct 09, 2013
IPVM

I am not interested in saving a few dollars at the expense of creating greater debates on the test.

My gut feel is to buy a mainstream COTS manufacturer line and then swap up some internal components like CPU or RAM, for different experiments.

As for OS, we'd like to keep that as consistent as possible, but I'd be curious to hear what manufacturers claim.

JH
John Honovich
Oct 09, 2013
IPVM

I want to reiterate for VMS manufacturers. If you want to have input on this, speak up here and NOW. Do not make excuses later. If there is some specific way your VMS must be tested or some specific situations you believe should not be done, say it now so we can evaluate before we start. If you don't like your results after we test, and you didn't speak up now, it's your fault.

RM
Ryan McGovern
Oct 09, 2013

Awesome, I look forward to seeing the results.

As this technology develops and more products become avalable it is great to have you guys testing products and sharing the results with the community.

Suggestions on Hardware: HP Z220, HP DL380, Dell R210 v aplliances such as Razberi.

Cameras: single & multi-lens including Arecont 8185DN.

VMS - Milestone, Exacq, Genetec, Avigilon, Geutebruck, Mobotix

Software raid controller v hardware raid controller

MN
Morten Nielsen
Oct 09, 2013

You can't expect the manufacturers to spec the test for you; and even they did, you'd be running 5 different setups, rendering the test somewhat meaningless.

As an end user, I probably don't care that vendor A has a super-optimized server side motion detection algo, if vendor B supports camera side detection and thus incur no processing cost on the server. If my TCO on the system is lower using vendor B, why would I care that A has "magic software" in it?

That said, I'd go with fairly standard equipment, so that you can reliably max out the system. If system A gets maxed out at 20 cameras on an i5, while system B can do 40, I'd be pretty confident that system B would also perform better on a higher end system. If your CPU is crazy fast, you might never max out the CPU, but instead be seeing bottlenecks in the storage system.

I'd stay with just one camera type for this test, but it might be hard to get 50-60 identical cameras. Most VMS manufacturers support a very wide range of cameras, some may be supported very well, and others may have a shitty driver. If you are mixing up cameras, your results may be valid for that particular permutation of cameras, while a - say Axis only - setup might give other results. I'd go with Axis as the drivers for these cameras should be pretty mature by now (for all VMS's)

Then I'd look at the following scenarios:

1) Record everything. Start with one camera af full res, max fps, lowest compression, and keep adding cams until the VMS can't keep up. Some will max out the drive and others probably wont be able to. This will test the efficiency of the storage system. Depending on the write-strategy on the different VMS's you will probably see different results.

2) Server side motion detection: This one is hard, because some will have a more robust (or fine-grained) motion detection, while others will offer only very coarse detection. I suppose that any "shortcut" in the code is ok, as long as the detection is "good enough" (whatever that means). Again, start with one camera and then keep adding cameras until the CPU maxes out.

3) Record on motion: Again, add cameras until you are maxed out. You can use your little train to trigger motion deterministically, or alternatively, point the cameras to a screen that plays the same video loop over and over again.

Testing a client is a project in itself; I've often heard that system A was inferior, because at X cameras, it was at 100% CPU utilization, while system B was only at 80%. Upon further investigation, it turns out that system B is only drawing half the number of frames on the screen. The way that the clients present video is very different from one another. One vendor didn't (until recently) support stepping in reverse frame by frame, which surely gave them an edge in terms of CPU and memory use, caused a huge problem in usability.

Clearly, starting a client will impact the VMS. One could set up a baseline - say 16 cameras, and then connect one, then two, then three clients showing a 4x4 view of the cameras. But you should then keep your eyes peeled on the SERVERs CPU utilization (and database performance too).

Just my 2 cents.

JC
Jason Clement
Oct 09, 2013

The client impact on the server really depends on what VMS you're using and what the server is doing. In the simplest form a client simply requests video streams from the VMS and the VMS then sends those raw video streams to the client machine. I know with OnSSI you can actually send different resolutions/framerates but the server alters the stream so that greatly impacts CPU. It all depends on what you're using and what you're doing :)

JH
John Honovich
Oct 09, 2013
IPVM

I do not want them to 'spec the test'. I want to hear their opinions now. Otherwise, we get into excuse mode after the test - "Oh, our system did really bad because you used X. If you used Y, our system would have worked great."

The camera vs server side motion detection is definitely going to be an interesting issue as real world users seem to be split between the two. We will have to do different series of experiments to see what the relative impact of adjusting those settings are.

If we are going to only run 1 camera, then is it Axis? Or is it Axis virtual camera emulator? Are people going to complain that Axis is better or worse?

JC
Jason Clement
Oct 10, 2013

No matter what camera manufactuer you use people will complain. I have no problem with Axis being used.

MN
Morten Nielsen
Oct 10, 2013

I agree 100% and I, for one, am going to complain - nomatter the outcome.

JH
John Honovich
Oct 10, 2013
IPVM
RF
Robert Fuller
Oct 24, 2013

I do not understand AT ALL why server side motion is even an option. The whole purpose of IP cameras is to keep the bulk of the processing and encoding on the edge. I would never even consider running motion detection on the camera side.

I am currently running an HP server and am currently working on an upgrade as I type this. I can provide some input, let me know.

JC
Jason Clement
Oct 09, 2013

VMS - Milestone, Exacq, Video Insight, Genetec, Avigilon

BB
Boris Biryuchkov
Oct 09, 2013

First of all, all VMSes must be tested in 100% identical environment. It means that the same physical machine, OS, network and set of cameras must be used for all tests.

SETUP

1) We suggest to use at least two different computers

- low end like atom with 2GB ram and 1-2 HDDs

- middle-end i3 or i5 with 4-8Gb of Ram and at least 3 HDDs

Tips: some VMSes are optimized for certain CPU instructions that may not be supported by low end systems ( like atom ), this is why simple results extrapolation would not work.

We do not think it is necessary to perform tests on high-end server equipment (like XEON processors, huge RAID etc) because:

a) Results obtained from tests on middle-end systems can be extrapolated to high-end segment,

b) It may require too many setup and too any cameras to really feel the difference.

2) As for camera setup: the many different cameras you use, the better it will be. One note: all cameras should be properly discovered and set up in all VMSes.

3) Network. We suggest to use 2 gigabit NICS: one for cameras and one for client connections. In low-end systems just one NIC could be used. All switches and VMS servers must be connected by CAT6 cables.

TESTING

The major bottlenecks that may be tested are:

- HDD usage (direct correlation with the maximum number of cameras that can be recorded simultaneously)

- CPU usage

To estimate HDD usage:

a) Find a software to measure HDD usage.

b) Isolate OS and VMS archive: do not write archive on OS hard disk drive (do not mistake partition with physical HDD!). That’s why we suggest you to use 2 HDDs.

c) Set up VMSes to record with the same FPS, bitrate, resolution. We suggest to set up continuous recording on max fps/resolution/quality that cameras may support (not motion recording).

d) Remember that maximum HDD usage occurs when HDD is writing and erasing the existing archive in the same time. So the disk must be full in the moment of measurement and activate the corresponding feature in each VMS (the oldest recording should be erasing).

e) To ensure that the software is able to handle cameras try to open streams from cameras in client (archived footage). The playback should be smooth and frames must not be dropped.

f) The measurement must be done at the moment while no client is connected to cameras.

g) Also it will be useful to compare HDD load while the recording is in progress and all cameras are viewed by client (archived footage).

To estimate CPU usage:

a) CPU usage can be measured by Windows Process Monitor.

b) It may be necessary to record many cameras to overload CPU so either use a lot of HDD or compare the average CPU load on the different VMSes.

c) Pay attention that some VMSes do software motion detection or transcoding.

As for memory usage estimation, different manufacturers may use some tricks to increase memory usage but decrease CPU one. The memory is cheap in comparison to CPU or HDD so I suggest to omit the estimation because the overall picture may be blurred.

Avatar
Sergey Bystrov
Oct 09, 2013
NetworkOptix

I'm very agree with Boris.

Avatar
Kevin Bennett
Oct 21, 2013

I disagree on the use of separate OS HDD for the test - or at least as the only configuration - though I agree that it is a good practice. In reality, most of the VMS servers I have encountered have had the OS on a partition of the RAID5 array. I think that making the test closely resemble actual environments would better than using more "ideal" environments.

Perhaps the separate OS HDD configuration would be an alternate configuration that could be evaluated.

JH
John Honovich
Oct 21, 2013
IPVM

Kevin, I agree with you about the baseline (not using a separate HDD for the OS). We can certainly do an alternate test to compare how performance differs with a separate HDD for the OSS.

There's also the case of some VMSes running on an SSD card, segregated from the video recordings stored on hard drives. That might be interesting to consider as well.

JC
Jason Clement
Oct 09, 2013

Saying that you can simply take a i3 and i5 and "scale it up" to a server grade processor is overly simplistic. Processors have become more and more complex over the years and there's a lot more behind it than just simply processing. I'm not an expert but I know enough that there is more involved.

MN
Morten Nielsen
Oct 09, 2013

@Jason : I disagree that it is overly simplistic in this case. What you are proposing is that one or more of the VMSs will suddenly gain a significant advantage on a server grade CPU (or gaming CPU for that matter). I doubt that you'll see that with any of the VMSs.

Consider an i5 based rig. System A does 30 cameras before maxing out, while system B does 20. Would going to a XEON suddenly cause System B to be able to do more than A? While theoretically, that could be the case, I would be very surprised to see this happen in practice.

JC
Jason Clement
Oct 09, 2013

On the small and mid range VMS I totally agree with you. When you start getting into VMS that allow for unlimitted cameras per server based on hardware then it becomes extremely important. Also running server side motion and/or analytics it becomes critical as well.

*edit* It also becomes critical if you do any virtualization. I know that's not being included but just had to throw that out there.

ng
nidal gharzeddine
Oct 09, 2013

What about network utilization, how many Mbps can the VMS handle?

That should be a main factor into deciding the number of cameras that could be added onto a single server. The cost impact of upgrading components on the same server is minor when compared to the costs of having to add a second server.

I would recommend considering, the number of licenses per OS? Some VMS maximize their license per OS, while they haven’t utilized the hardware resources, and hence virtualization and all its complexity.

RF
Robert Fuller
Oct 24, 2013

Mbps should not be limited by the VMS. Mbps liits are dictated by hardware/network.

BB
Boris Biryuchkov
Oct 09, 2013

Initially VMS should be able to utilize all available network bandwidth. However the need to record 1Gbit per second (especially if it is many concurrent streams) will result in bottleneck in HDD performance. This will show how VMSes are able to handle the recording of several streams simultaneously.

But I agree it may be useful to measure network utilization during tests.

ng
nidal gharzeddine
Oct 10, 2013

I do agree with the fact that 1Gbit will cause a bottle neck at the storage level, but keep in mind that 1Gbit would require almost 500 cameras (+/-), consequently multiple storage heads would be required to contain all of these video streams, hence the bandwidth would be split, and the HDD bottle neck removed.

If a VMS can support more than a thousand video streams on a standard off the shelf server and single OS, that (in my opinion) could be a decisive factor for the end user and the integrator, the cost impact is not negligible.

I guess, what i would like to see is what is the license limitation by the VMS for a single OS (if any) and what is the bandwidth limitations (if any).

JC
Jason Clement
Oct 10, 2013

Are you taking into account the HDD/RAID controller? All the data still has to flow through that beore going through to the drives. It's true the higher spindle count the more performance you get hence huge database programs have a ton of small size fast drives as they need crazy I/O.

TS
Tariq Saleh
Oct 11, 2013

Hello, this is a very big topic and I don't expect to be covered by one article. Anyhow, we could start testing the CPU and memory unitization regardless of the server vender since many vendors are using the same Intel/AMD processors. Selecting specific CPU will help to establish a baseline for further benchmark. Also, choosing the Xeon processor will be a good choice since it is a server grade processor.

Also, we could have an abstraction layer on the camera side. So it doesn't matter which camera I'm using or settings or frame rate; we have to focus on the overall bit rate and do stress testing to see how much traffic could the server handle. Also we have to see the impact of the MJPEG/MPEG4/H.264 decoding impact on the server/client. Also, we have to consider the storage sub system in term of IOPS, the throughput, the OS block size, and so on as this will impact the processor and the memory as well.


I have couple of documents that could help.

JH
John Honovich
Oct 16, 2013
IPVM

Let's talk specific hardware to test. As mentioned earlier, we want to buy either Dell or HP, simply because they are common real world choices.

Incorporating some of what was said above, there seem to be 3 possible levels:

  • Low end - Atom processor
  • Mid end - I3/I5/I7 processor
  • High end - Xeon processor

What about specific models?

I will suggest a few from Dell to discuss:

Any specific problems or limitations?

BB
Boris Biryuchkov
Oct 16, 2013

OptiPlex 160 Tiny Desktop - its base RAM size is 1Gb. I would recommend to extend this up to 4 Gb. Most VMS'es are Windows so Windows will need at least 2 Gb to run. Also low amount of memory will result in high disk usage by OS (swapping) so this will affect testing results. To avoid this add another hard drive for archive or add SSD drive to install OS on.

OptiPlex 9010 Small Form Factor. As you're gonna load it up, add more RAM. 8 GB (4 GB for OS and 4 GB for VMS) will be sufficient. Do not write archive on the system drive.

As for high end server - it would be interesting to test various RAID modes.

JH
John Honovich
Oct 16, 2013
IPVM

Thanks, Boris.

For the Optiplex 160, the idea would be to use 2GB and 4GB RAM, i.e., see what performance differences happens when everything else is equal but the RAM is increased from 2GB to 4GB.

Same thing for all the units, varying RAM to monitor performance.

As for RAID, it's certainly worth testing. The question is: what configuration? How do we accurately test it comparatively?

Avatar
Sergey Bystrov
Oct 17, 2013
NetworkOptix

John,

What is the thought behind testing VMS with different RAM configuration?

Here are my thoughts against:

As I understand the purpose of your article is performance comparison, in other words to test how cheap server could be to handle this and that...

Taking into account the memory price at the moment is 20$ per 2 gig, may be it does not worth to:

1) make you report more blurry with such details
2) mess with OS swapping on same HDD and such

Maybe more fair would be give VMS as much memory as it needs?

Just a thought

JH
John Honovich
Oct 17, 2013
IPVM

It would be easier for us to only test a higher amount of RAM. However, many people choose lower amounts of RAM (some go 2, 4, 6, etc.). It would be helpful to know if a VMS will have any problems with lower RAM levels so we can make recommendations to our readers of the tradeoffs within.

BB
Boris Biryuchkov
Oct 17, 2013

John,

If you are going to test VMSes on a low-end computer with 1 gig of RAM, I'm not sure you will be able even to install Windows (perhaps Win XP or Win 2003 server)...

I suggest to run the test on Linux Server (not desktop) on such machines. This OS does not consume much resources so the test will be good enough. I will be interesting to compare the same VMSes performance on Windows and Linux as well.

Not all VMSes you mentioned support Linux. However it is still worth testing. For instance, if VMS A is both Windows and Linux and VMS B is Windows only and:

VMS A - 12 Cameras on Windows

VMS A - 16 Cameras on Linux

VMS B - 14 Cameras on Windows

In this case VMS A looks much more good for end user because he won't need to buy Windows license and can operate with more cameras comparing to VMS B.

JH
John Honovich
Oct 17, 2013
IPVM

We're not going to run any tests with 1GB of RAM but we most likely will do 2GB as a number of the low end NVRs use 2GBs.

For the ~10% of VMSes that support both Linux and Windows, will test on both, because it will be interesting to see what the performance differences are.

Avatar
Sergey Bystrov
Oct 17, 2013
NetworkOptix

I forgot to say that as I understand majority of VMSes installations are done not on random end users machines but on well designed servers ( well designed - optimized from price point ). So VMS always have enough memory almost for free

JH
John Honovich
Oct 17, 2013
IPVM

I think you understand wrong or are optimistic :)

I am not saying that most are done on 'random' end user machines, but a lot of them are not particularly well designed.

As for VMSes always having enough memory, that implies the VMS is not a resource hog. VMS's resource consumption varies and that's a big part of the test to better quantify who are the hogs and how bad of an issue it is.

Avatar
Sergey Bystrov
Oct 17, 2013
NetworkOptix
TS
Tariq Saleh
Oct 17, 2013

The storage performance will need separate articles to be covered. As an example, RIAD 5 has higher throughout than RAID 6; what will be RAID controller memory; Are we talking about internal storage, DAS, NAS, or SAN;

I have couple of documents that I would like to share and will shed more light into this subject, but I don't know how to share it. Thanks

Avatar
Joel Kriener
Oct 21, 2013
IPVMU Certified

Scalability over time. How does the system react to adding cameras? (for testing purposes assume all the same cameras, configuration and settings) Select one or more appropriate performance metrics (i.e. CPU and memory usage) and plot those values as you increase the number of cameras.

Simplistic metric but should give at least an empirical profile reference to use and understand the impact on a particular hardware / software setup.

Joel

Avatar
Joel Kriener
Oct 21, 2013
IPVMU Certified

Impact of using Virtual Machine environment for the applications would be good to see.

Joel

JH
John Honovich
Oct 21, 2013
IPVM

VMs will come much later in our load testing as the priority first will be the most common use case - running VMS software directly on hardware.

Avatar
Kevin Bennett
Oct 21, 2013

Things like RAID controller performance (software RAID vs. mid-tier hardware; 512MB RAID RAM vs. 1GB; internal drives vs. external SAN/NAS) would be good to include from my perspective. I know the goal is to test VMS, but how it interacts with different hardware/software combinations would be helpful to know.

I would like to see MS Windows Server 2012 vs. 2008 vs. Win 8 vs. Win 7 Pro vs. Win 7 embedded OS.

For processors, I recommend using a dual-core, a quad-core such as i3, and a high-end quad such as XEON. Then test one process vs. two. THEN, I would recommend testing with similar AMD as well. Some software performs better in one processor environment than the other. Some hardware, for that matter.

For hardware, I concur with the recommendations for Dell PowerEdge equipment. HP would be my other recommendation as well.

Avigilon's 15TB NVR is, I believe, a rebranded Dell R520. I prefer the newer R720xd which allows for additional internal drives that can be dedicated to OS, but understand that many installations are not going to go with something that "beefy".

As are many others here, I'm interested in seeing how much bandwidth each VMS is capable of handling, total number of connected cameras and clients, etc. We are moving from 100% analog to about a 50/50 mix of analog-to-IP encoder and true IP megapixel cameras.

JH
John Honovich
Oct 21, 2013
IPVM

Kevin, I agree with the value of your suggestions.

That said, I want to set appropriate expectations. It is going to take months to test all these combinations so we are going to start on the most basic and then work our way up.

Avatar
Kevin Bennett
Oct 21, 2013

I'm sure most of us here understand that with so many potential variables it can't be done quickly. No more than two or three months, though, right? :-) Seriously, though, I think the VMS / server performance comparison is an excellent endeavor and thank IPVM in advance for undertaking what can potentially be a huge project. Talk about scope creep potential . . .

BH
Bohan Huang
Oct 29, 2013

1. Start with a common entry level E3-1230v3, 8gb ram and single 3TB 7200rpm sata hdd

2. Simulate 32ch 1080p 3mbps load with software motion detect recording

3. Log CPU, RAM and storage IOPs usage for server and client processes.

4. Compare!

AK
Alex K
Oct 29, 2013

Here is # 1 for me
Develop very clear, detail
standard testing procedure
and publish for members to read

JH
John Honovich
Oct 29, 2013
IPVM

While we will have that, regardless of how clear or testing procedures are, the challenge will be is that there are so many permutations, that people will obviously have their personal preferences (you should have used 2GB instead of 8GB or vice versa, you should have use Windows 8 instead of Server 2008 or vice versa). I am not expecting to please everyone and if Avigilon does not come in first place, I am sure you will promptly object Alex.

AK
Alex K
Oct 29, 2013

I do not expect for you to "please" anybody ( specially myself )

I simply suggested to do in real engineering way

also I would suggest may be to hire Hardware computer engineer or at least consultant

and to me it does not matter if Avigilon takes "FIRST " place or whatever

I made choice for my self long time ago

JH
John Honovich
Oct 29, 2013
IPVM

Again, "very clear, detail standard testing procedure" does not ensure a 'real engineering way' nor will it eliminate debate about what should be tested.

So, yes, we can and will publish testing procedure but that will certainly not stop debate or dissent because there are literally hundreds of combinations of hardware, components, OSes, etc. that people will want / ask / prefer.

As for your 'hardware computer engineer' or 'consultant', if you do not think we can handle doing these tests, ignore the results. This discussion with 50+ comments is designed to get feedback from experts of different perspectives. I am not going to play a game of what consultant we hire because it will be just as easy to dispute consultant's credentials as well.

And, yes, it does matter to you about Avigilon because anytime we say something critical about them, you jump in it. Anytime, we say something positive, you say nothing.

AK
Alex K
Oct 29, 2013

You right about me and Avigilon

my Point is very simple

It is does not Matter what you think about Avigilon

We all made our choices

good luck with testing

New discussion

Ask questions and get answers to your physical security questions from IPVM team members and fellow subscribers.

Newest discussions