How Should IPVM Test VMS Server Performance?
IPVM is going to start testing VMS server performance. Obviously, there are a lot of factors that come into play (CPU, RAM, motion detection - on or off, number of cameras, resolution, bitrate, to name just a few).
This discussion is to gather feedback from the community about how we should do it. If you are a manufacturer, integrator or a technologist with an opinion here, please speak up now.
We will be testing a number of VMSes, with the goal of better understanding key performance drivers.
A few immediate questions:
- What hardware should we use? Certain brands? Certain components?
- What cameras should we use?
- What issues should we beware of?
If there are things that you find essential, let us know.
It will be impossible to make everyone happy as there are far too many possible combinations. However, we want to have a reasonable set of equipment and tests to cover the most common use cases.
What's the server load going to look like ie number of cameras and clients viewing live video?
There's a lot there - How many cameras? What resolution for the cameras? What birate for the cameras? Will the bit rate change (VBR - spikes at night for higher traffic scenes)? Does it matter which brand or model of manufacturer?
To start, we are going to try out a bunch of combinations, measure the results and see what patterns are uncovered.
I would definitely recommend Dell but there's so many different combinations. I guess a 520 with a single quad core or six core processor would be about an average build for 40-80 cameras depending on the VMS and whether server side motion was used. A typical supermicros server build would be good to test on as well. Other than that I'm not super familiar with any of the other server manufaturers out there.
As far as cameras most people have a mix of 720p and higher res cameras but it would be coool to see a server loaded down with 5 MP cameras as to what that would do with performance especially if you have server side motion turned on and ahigh frame rate.
RAID performance would be good to see too I think. RAID 10 vs RAID5 vs RAID6.
There's so much like you said it's hard to get it all in :)
Dell or HP strike me as safe choices since they are so commonly used. I am not as sure about quad or six core processors as most NVR appliances seem to run Atoms or i3/i5/i5 dual core CPUs.
What do you think about using VMWare to test and measuring the performace of the VMS systems?
Definitely not virtual machines, only because it's going to cause a lot of objections about performance impact. It would certainly make it easier to swap VMSes in and out but almost certainly manufacturers will cry foul if their products do badly, blaming it on the virtual machine.
Good point. I think Dell or IBM Server would be a good pick but then again you might be better off picking a Server board from Tyan or Supermicro to keeps your costs down as I don't think the motherboard will make a big difference on the end results. I would probably try to use a lower end CPU so the cpu loads are noticiable between VMS's.
It would be interesting to see the difference multiple cores makes on the different VMS to see if the VMS systems are acually taking advantage of the multiple cores.
How are you gong to pick which OS to use? I am assuming you are going to have to keep that consistant between comparisons. Will you use or 64bit OS or 32bit OS? Server OS or Desktop OS? What if the VMS is only a 32bit application?
I am not interested in saving a few dollars at the expense of creating greater debates on the test.
My gut feel is to buy a mainstream COTS manufacturer line and then swap up some internal components like CPU or RAM, for different experiments.
As for OS, we'd like to keep that as consistent as possible, but I'd be curious to hear what manufacturers claim.
I want to reiterate for VMS manufacturers. If you want to have input on this, speak up here and NOW. Do not make excuses later. If there is some specific way your VMS must be tested or some specific situations you believe should not be done, say it now so we can evaluate before we start. If you don't like your results after we test, and you didn't speak up now, it's your fault.
Awesome, I look forward to seeing the results.
As this technology develops and more products become avalable it is great to have you guys testing products and sharing the results with the community.
Suggestions on Hardware: HP Z220, HP DL380, Dell R210 v aplliances such as Razberi.
Cameras: single & multi-lens including Arecont 8185DN.
VMS - Milestone, Exacq, Genetec, Avigilon, Geutebruck, Mobotix
Software raid controller v hardware raid controller
You can't expect the manufacturers to spec the test for you; and even they did, you'd be running 5 different setups, rendering the test somewhat meaningless.
As an end user, I probably don't care that vendor A has a super-optimized server side motion detection algo, if vendor B supports camera side detection and thus incur no processing cost on the server. If my TCO on the system is lower using vendor B, why would I care that A has "magic software" in it?
That said, I'd go with fairly standard equipment, so that you can reliably max out the system. If system A gets maxed out at 20 cameras on an i5, while system B can do 40, I'd be pretty confident that system B would also perform better on a higher end system. If your CPU is crazy fast, you might never max out the CPU, but instead be seeing bottlenecks in the storage system.
I'd stay with just one camera type for this test, but it might be hard to get 50-60 identical cameras. Most VMS manufacturers support a very wide range of cameras, some may be supported very well, and others may have a shitty driver. If you are mixing up cameras, your results may be valid for that particular permutation of cameras, while a - say Axis only - setup might give other results. I'd go with Axis as the drivers for these cameras should be pretty mature by now (for all VMS's)
Then I'd look at the following scenarios:
1) Record everything. Start with one camera af full res, max fps, lowest compression, and keep adding cams until the VMS can't keep up. Some will max out the drive and others probably wont be able to. This will test the efficiency of the storage system. Depending on the write-strategy on the different VMS's you will probably see different results.
2) Server side motion detection: This one is hard, because some will have a more robust (or fine-grained) motion detection, while others will offer only very coarse detection. I suppose that any "shortcut" in the code is ok, as long as the detection is "good enough" (whatever that means). Again, start with one camera and then keep adding cameras until the CPU maxes out.
3) Record on motion: Again, add cameras until you are maxed out. You can use your little train to trigger motion deterministically, or alternatively, point the cameras to a screen that plays the same video loop over and over again.
Testing a client is a project in itself; I've often heard that system A was inferior, because at X cameras, it was at 100% CPU utilization, while system B was only at 80%. Upon further investigation, it turns out that system B is only drawing half the number of frames on the screen. The way that the clients present video is very different from one another. One vendor didn't (until recently) support stepping in reverse frame by frame, which surely gave them an edge in terms of CPU and memory use, caused a huge problem in usability.
Clearly, starting a client will impact the VMS. One could set up a baseline - say 16 cameras, and then connect one, then two, then three clients showing a 4x4 view of the cameras. But you should then keep your eyes peeled on the SERVERs CPU utilization (and database performance too).
Just my 2 cents.
The client impact on the server really depends on what VMS you're using and what the server is doing. In the simplest form a client simply requests video streams from the VMS and the VMS then sends those raw video streams to the client machine. I know with OnSSI you can actually send different resolutions/framerates but the server alters the stream so that greatly impacts CPU. It all depends on what you're using and what you're doing :)
I do not want them to 'spec the test'. I want to hear their opinions now. Otherwise, we get into excuse mode after the test - "Oh, our system did really bad because you used X. If you used Y, our system would have worked great."
The camera vs server side motion detection is definitely going to be an interesting issue as real world users seem to be split between the two. We will have to do different series of experiments to see what the relative impact of adjusting those settings are.
If we are going to only run 1 camera, then is it Axis? Or is it Axis virtual camera emulator? Are people going to complain that Axis is better or worse?
No matter what camera manufactuer you use people will complain. I have no problem with Axis being used.
I do not understand AT ALL why server side motion is even an option. The whole purpose of IP cameras is to keep the bulk of the processing and encoding on the edge. I would never even consider running motion detection on the camera side.
I am currently running an HP server and am currently working on an upgrade as I type this. I can provide some input, let me know.
First of all, all VMSes must be tested in 100% identical environment. It means that the same physical machine, OS, network and set of cameras must be used for all tests.
SETUP
1) We suggest to use at least two different computers
- low end like atom with 2GB ram and 1-2 HDDs
- middle-end i3 or i5 with 4-8Gb of Ram and at least 3 HDDs
Tips: some VMSes are optimized for certain CPU instructions that may not be supported by low end systems ( like atom ), this is why simple results extrapolation would not work.
We do not think it is necessary to perform tests on high-end server equipment (like XEON processors, huge RAID etc) because:
a) Results obtained from tests on middle-end systems can be extrapolated to high-end segment,
b) It may require too many setup and too any cameras to really feel the difference.
2) As for camera setup: the many different cameras you use, the better it will be. One note: all cameras should be properly discovered and set up in all VMSes.
3) Network. We suggest to use 2 gigabit NICS: one for cameras and one for client connections. In low-end systems just one NIC could be used. All switches and VMS servers must be connected by CAT6 cables.
TESTING
The major bottlenecks that may be tested are:
- HDD usage (direct correlation with the maximum number of cameras that can be recorded simultaneously)
- CPU usage
To estimate HDD usage:
a) Find a software to measure HDD usage.
b) Isolate OS and VMS archive: do not write archive on OS hard disk drive (do not mistake partition with physical HDD!). That’s why we suggest you to use 2 HDDs.
c) Set up VMSes to record with the same FPS, bitrate, resolution. We suggest to set up continuous recording on max fps/resolution/quality that cameras may support (not motion recording).
d) Remember that maximum HDD usage occurs when HDD is writing and erasing the existing archive in the same time. So the disk must be full in the moment of measurement and activate the corresponding feature in each VMS (the oldest recording should be erasing).
e) To ensure that the software is able to handle cameras try to open streams from cameras in client (archived footage). The playback should be smooth and frames must not be dropped.
f) The measurement must be done at the moment while no client is connected to cameras.
g) Also it will be useful to compare HDD load while the recording is in progress and all cameras are viewed by client (archived footage).
To estimate CPU usage:
a) CPU usage can be measured by Windows Process Monitor.
b) It may be necessary to record many cameras to overload CPU so either use a lot of HDD or compare the average CPU load on the different VMSes.
c) Pay attention that some VMSes do software motion detection or transcoding.
As for memory usage estimation, different manufacturers may use some tricks to increase memory usage but decrease CPU one. The memory is cheap in comparison to CPU or HDD so I suggest to omit the estimation because the overall picture may be blurred.
I'm very agree with Boris.
I disagree on the use of separate OS HDD for the test - or at least as the only configuration - though I agree that it is a good practice. In reality, most of the VMS servers I have encountered have had the OS on a partition of the RAID5 array. I think that making the test closely resemble actual environments would better than using more "ideal" environments.
Perhaps the separate OS HDD configuration would be an alternate configuration that could be evaluated.
Kevin, I agree with you about the baseline (not using a separate HDD for the OS). We can certainly do an alternate test to compare how performance differs with a separate HDD for the OSS.
There's also the case of some VMSes running on an SSD card, segregated from the video recordings stored on hard drives. That might be interesting to consider as well.
Saying that you can simply take a i3 and i5 and "scale it up" to a server grade processor is overly simplistic. Processors have become more and more complex over the years and there's a lot more behind it than just simply processing. I'm not an expert but I know enough that there is more involved.
@Jason : I disagree that it is overly simplistic in this case. What you are proposing is that one or more of the VMSs will suddenly gain a significant advantage on a server grade CPU (or gaming CPU for that matter). I doubt that you'll see that with any of the VMSs.
Consider an i5 based rig. System A does 30 cameras before maxing out, while system B does 20. Would going to a XEON suddenly cause System B to be able to do more than A? While theoretically, that could be the case, I would be very surprised to see this happen in practice.
On the small and mid range VMS I totally agree with you. When you start getting into VMS that allow for unlimitted cameras per server based on hardware then it becomes extremely important. Also running server side motion and/or analytics it becomes critical as well.
*edit* It also becomes critical if you do any virtualization. I know that's not being included but just had to throw that out there.
What about network utilization, how many Mbps can the VMS handle?
That should be a main factor into deciding the number of cameras that could be added onto a single server. The cost impact of upgrading components on the same server is minor when compared to the costs of having to add a second server.
I would recommend considering, the number of licenses per OS? Some VMS maximize their license per OS, while they haven’t utilized the hardware resources, and hence virtualization and all its complexity.
Initially VMS should be able to utilize all available network bandwidth. However the need to record 1Gbit per second (especially if it is many concurrent streams) will result in bottleneck in HDD performance. This will show how VMSes are able to handle the recording of several streams simultaneously.
But I agree it may be useful to measure network utilization during tests.
I do agree with the fact that 1Gbit will cause a bottle neck at the storage level, but keep in mind that 1Gbit would require almost 500 cameras (+/-), consequently multiple storage heads would be required to contain all of these video streams, hence the bandwidth would be split, and the HDD bottle neck removed.
If a VMS can support more than a thousand video streams on a standard off the shelf server and single OS, that (in my opinion) could be a decisive factor for the end user and the integrator, the cost impact is not negligible.
I guess, what i would like to see is what is the license limitation by the VMS for a single OS (if any) and what is the bandwidth limitations (if any).
Are you taking into account the HDD/RAID controller? All the data still has to flow through that beore going through to the drives. It's true the higher spindle count the more performance you get hence huge database programs have a ton of small size fast drives as they need crazy I/O.
Hello, this is a very big topic and I don't expect to be covered by one article. Anyhow, we could start testing the CPU and memory unitization regardless of the server vender since many vendors are using the same Intel/AMD processors. Selecting specific CPU will help to establish a baseline for further benchmark. Also, choosing the Xeon processor will be a good choice since it is a server grade processor.
Also, we could have an abstraction layer on the camera side. So it doesn't matter which camera I'm using or settings or frame rate; we have to focus on the overall bit rate and do stress testing to see how much traffic could the server handle. Also we have to see the impact of the MJPEG/MPEG4/H.264 decoding impact on the server/client. Also, we have to consider the storage sub system in term of IOPS, the throughput, the OS block size, and so on as this will impact the processor and the memory as well.
I have couple of documents that could help.
Let's talk specific hardware to test. As mentioned earlier, we want to buy either Dell or HP, simply because they are common real world choices.
Incorporating some of what was said above, there seem to be 3 possible levels:
- Low end - Atom processor
- Mid end - I3/I5/I7 processor
- High end - Xeon processor
What about specific models?
I will suggest a few from Dell to discuss:
- Low end - OptiPlex 160 Tiny Desktop
- Mid end - OptiPlex 9010 Small Form Factor
- High end - PowerEdge R210 or 420 rack server
Any specific problems or limitations?
OptiPlex 160 Tiny Desktop - its base RAM size is 1Gb. I would recommend to extend this up to 4 Gb. Most VMS'es are Windows so Windows will need at least 2 Gb to run. Also low amount of memory will result in high disk usage by OS (swapping) so this will affect testing results. To avoid this add another hard drive for archive or add SSD drive to install OS on.
OptiPlex 9010 Small Form Factor. As you're gonna load it up, add more RAM. 8 GB (4 GB for OS and 4 GB for VMS) will be sufficient. Do not write archive on the system drive.
As for high end server - it would be interesting to test various RAID modes.
Thanks, Boris.
For the Optiplex 160, the idea would be to use 2GB and 4GB RAM, i.e., see what performance differences happens when everything else is equal but the RAM is increased from 2GB to 4GB.
Same thing for all the units, varying RAM to monitor performance.
As for RAID, it's certainly worth testing. The question is: what configuration? How do we accurately test it comparatively?
John,
What is the thought behind testing VMS with different RAM configuration?
Here are my thoughts against:
As I understand the purpose of your article is performance comparison, in other words to test how cheap server could be to handle this and that...
Taking into account the memory price at the moment is 20$ per 2 gig, may be it does not worth to:
1) make you report more blurry with such details
2) mess with OS swapping on same HDD and such
Maybe more fair would be give VMS as much memory as it needs?
Just a thought
It would be easier for us to only test a higher amount of RAM. However, many people choose lower amounts of RAM (some go 2, 4, 6, etc.). It would be helpful to know if a VMS will have any problems with lower RAM levels so we can make recommendations to our readers of the tradeoffs within.
John,
If you are going to test VMSes on a low-end computer with 1 gig of RAM, I'm not sure you will be able even to install Windows (perhaps Win XP or Win 2003 server)...
I suggest to run the test on Linux Server (not desktop) on such machines. This OS does not consume much resources so the test will be good enough. I will be interesting to compare the same VMSes performance on Windows and Linux as well.
Not all VMSes you mentioned support Linux. However it is still worth testing. For instance, if VMS A is both Windows and Linux and VMS B is Windows only and:
VMS A - 12 Cameras on Windows
VMS A - 16 Cameras on Linux
VMS B - 14 Cameras on Windows
In this case VMS A looks much more good for end user because he won't need to buy Windows license and can operate with more cameras comparing to VMS B.
I forgot to say that as I understand majority of VMSes installations are done not on random end users machines but on well designed servers ( well designed - optimized from price point ). So VMS always have enough memory almost for free
I think you understand wrong or are optimistic :)
I am not saying that most are done on 'random' end user machines, but a lot of them are not particularly well designed.
As for VMSes always having enough memory, that implies the VMS is not a resource hog. VMS's resource consumption varies and that's a big part of the test to better quantify who are the hogs and how bad of an issue it is.
The storage performance will need separate articles to be covered. As an example, RIAD 5 has higher throughout than RAID 6; what will be RAID controller memory; Are we talking about internal storage, DAS, NAS, or SAN;
I have couple of documents that I would like to share and will shed more light into this subject, but I don't know how to share it. Thanks
Scalability over time. How does the system react to adding cameras? (for testing purposes assume all the same cameras, configuration and settings) Select one or more appropriate performance metrics (i.e. CPU and memory usage) and plot those values as you increase the number of cameras.
Simplistic metric but should give at least an empirical profile reference to use and understand the impact on a particular hardware / software setup.
Joel
Impact of using Virtual Machine environment for the applications would be good to see.
Joel
VMs will come much later in our load testing as the priority first will be the most common use case - running VMS software directly on hardware.
Things like RAID controller performance (software RAID vs. mid-tier hardware; 512MB RAID RAM vs. 1GB; internal drives vs. external SAN/NAS) would be good to include from my perspective. I know the goal is to test VMS, but how it interacts with different hardware/software combinations would be helpful to know.
I would like to see MS Windows Server 2012 vs. 2008 vs. Win 8 vs. Win 7 Pro vs. Win 7 embedded OS.
For processors, I recommend using a dual-core, a quad-core such as i3, and a high-end quad such as XEON. Then test one process vs. two. THEN, I would recommend testing with similar AMD as well. Some software performs better in one processor environment than the other. Some hardware, for that matter.
For hardware, I concur with the recommendations for Dell PowerEdge equipment. HP would be my other recommendation as well.
Avigilon's 15TB NVR is, I believe, a rebranded Dell R520. I prefer the newer R720xd which allows for additional internal drives that can be dedicated to OS, but understand that many installations are not going to go with something that "beefy".
As are many others here, I'm interested in seeing how much bandwidth each VMS is capable of handling, total number of connected cameras and clients, etc. We are moving from 100% analog to about a 50/50 mix of analog-to-IP encoder and true IP megapixel cameras.
Kevin, I agree with the value of your suggestions.
That said, I want to set appropriate expectations. It is going to take months to test all these combinations so we are going to start on the most basic and then work our way up.
I'm sure most of us here understand that with so many potential variables it can't be done quickly. No more than two or three months, though, right? :-) Seriously, though, I think the VMS / server performance comparison is an excellent endeavor and thank IPVM in advance for undertaking what can potentially be a huge project. Talk about scope creep potential . . .
1. Start with a common entry level E3-1230v3, 8gb ram and single 3TB 7200rpm sata hdd
2. Simulate 32ch 1080p 3mbps load with software motion detect recording
3. Log CPU, RAM and storage IOPs usage for server and client processes.
4. Compare!
Here is # 1 for me
Develop very clear, detail
standard testing procedure
and publish for members to read
While we will have that, regardless of how clear or testing procedures are, the challenge will be is that there are so many permutations, that people will obviously have their personal preferences (you should have used 2GB instead of 8GB or vice versa, you should have use Windows 8 instead of Server 2008 or vice versa). I am not expecting to please everyone and if Avigilon does not come in first place, I am sure you will promptly object Alex.
I do not expect for you to "please" anybody ( specially myself )
I simply suggested to do in real engineering way
also I would suggest may be to hire Hardware computer engineer or at least consultant
and to me it does not matter if Avigilon takes "FIRST " place or whatever
I made choice for my self long time ago
Again, "very clear, detail standard testing procedure" does not ensure a 'real engineering way' nor will it eliminate debate about what should be tested.
So, yes, we can and will publish testing procedure but that will certainly not stop debate or dissent because there are literally hundreds of combinations of hardware, components, OSes, etc. that people will want / ask / prefer.
As for your 'hardware computer engineer' or 'consultant', if you do not think we can handle doing these tests, ignore the results. This discussion with 50+ comments is designed to get feedback from experts of different perspectives. I am not going to play a game of what consultant we hire because it will be just as easy to dispute consultant's credentials as well.
And, yes, it does matter to you about Avigilon because anytime we say something critical about them, you jump in it. Anytime, we say something positive, you say nothing.
Newest Discussions
| Discussion | Posts | Latest |
|---|---|---|
|
Started by
Jay Hobdy
|
6
|
about 6 hours by Undisclosed Integrator #1 |
|
Started by
John Honovich
|
24
|
about 4 hours by John Honovich |
|
Started by
Jeffrey Hinckley
|
6
|
about 8 hours by John Honovich |
|
Started by
Undisclosed Manufacturer #1
|
4
|
less than a minute by John Honovich |
|
Started by
Undisclosed Integrator #1
|
5
|
less than a minute by Undisclosed #2 |