How Should IPVM Test VMS Server Performance?

I would definitely recommend Dell but there's so many different combinations. I guess a 520 with a single quad core or six core processor would be about an average build for 40-80 cameras depending on the VMS and whether server side motion was used. A typical supermicros server build would be good to test on as well. Other than that I'm not super familiar with any of the other server manufaturers out there.

As far as cameras most people have a mix of 720p and higher res cameras but it would be coool to see a server loaded down with 5 MP cameras as to what that would do with performance especially if you have server side motion turned on and ahigh frame rate.

RAID performance would be good to see too I think. RAID 10 vs RAID5 vs RAID6.

There's so much like you said it's hard to get it all in :)

I want to reiterate for VMS manufacturers. If you want to have input on this, speak up here and NOW. Do not make excuses later. If there is some specific way your VMS must be tested or some specific situations you believe should not be done, say it now so we can evaluate before we start. If you don't like your results after we test, and you didn't speak up now, it's your fault.

You can't expect the manufacturers to spec the test for you; and even they did, you'd be running 5 different setups, rendering the test somewhat meaningless.

As an end user, I probably don't care that vendor A has a super-optimized server side motion detection algo, if vendor B supports camera side detection and thus incur no processing cost on the server. If my TCO on the system is lower using vendor B, why would I care that A has "magic software" in it?

That said, I'd go with fairly standard equipment, so that you can reliably max out the system. If system A gets maxed out at 20 cameras on an i5, while system B can do 40, I'd be pretty confident that system B would also perform better on a higher end system. If your CPU is crazy fast, you might never max out the CPU, but instead be seeing bottlenecks in the storage system.

I'd stay with just one camera type for this test, but it might be hard to get 50-60 identical cameras. Most VMS manufacturers support a very wide range of cameras, some may be supported very well, and others may have a shitty driver. If you are mixing up cameras, your results may be valid for that particular permutation of cameras, while a - say Axis only - setup might give other results. I'd go with Axis as the drivers for these cameras should be pretty mature by now (for all VMS's)

Then I'd look at the following scenarios:

1) Record everything. Start with one camera af full res, max fps, lowest compression, and keep adding cams until the VMS can't keep up. Some will max out the drive and others probably wont be able to. This will test the efficiency of the storage system. Depending on the write-strategy on the different VMS's you will probably see different results.

2) Server side motion detection: This one is hard, because some will have a more robust (or fine-grained) motion detection, while others will offer only very coarse detection. I suppose that any "shortcut" in the code is ok, as long as the detection is "good enough" (whatever that means). Again, start with one camera and then keep adding cameras until the CPU maxes out.

3) Record on motion: Again, add cameras until you are maxed out. You can use your little train to trigger motion deterministically, or alternatively, point the cameras to a screen that plays the same video loop over and over again.

Testing a client is a project in itself; I've often heard that system A was inferior, because at X cameras, it was at 100% CPU utilization, while system B was only at 80%. Upon further investigation, it turns out that system B is only drawing half the number of frames on the screen. The way that the clients present video is very different from one another. One vendor didn't (until recently) support stepping in reverse frame by frame, which surely gave them an edge in terms of CPU and memory use, caused a huge problem in usability.

Clearly, starting a client will impact the VMS. One could set up a baseline - say 16 cameras, and then connect one, then two, then three clients showing a 4x4 view of the cameras. But you should then keep your eyes peeled on the SERVERs CPU utilization (and database performance too).

Just my 2 cents.

Saying that you can simply take a i3 and i5 and "scale it up" to a server grade processor is overly simplistic. Processors have become more and more complex over the years and there's a lot more behind it than just simply processing. I'm not an expert but I know enough that there is more involved.

On the small and mid range VMS I totally agree with you. When you start getting into VMS that allow for unlimitted cameras per server based on hardware then it becomes extremely important. Also running server side motion and/or analytics it becomes critical as well.

*edit* It also becomes critical if you do any virtualization. I know that's not being included but just had to throw that out there.

I do agree with the fact that 1Gbit will cause a bottle neck at the storage level, but keep in mind that 1Gbit would require almost 500 cameras (+/-), consequently multiple storage heads would be required to contain all of these video streams, hence the bandwidth would be split, and the HDD bottle neck removed.

If a VMS can support more than a thousand video streams on a standard off the shelf server and single OS, that (in my opinion) could be a decisive factor for the end user and the integrator, the cost impact is not negligible.

I guess, what i would like to see is what is the license limitation by the VMS for a single OS (if any) and what is the bandwidth limitations (if any).

Are you taking into account the HDD/RAID controller? All the data still has to flow through that beore going through to the drives. It's true the higher spindle count the more performance you get hence huge database programs have a ton of small size fast drives as they need crazy I/O.

Hello, this is a very big topic and I don't expect to be covered by one article. Anyhow, we could start testing the CPU and memory unitization regardless of the server vender since many vendors are using the same Intel/AMD processors. Selecting specific CPU will help to establish a baseline for further benchmark. Also, choosing the Xeon processor will be a good choice since it is a server grade processor.

Also, we could have an abstraction layer on the camera side. So it doesn't matter which camera I'm using or settings or frame rate; we have to focus on the overall bit rate and do stress testing to see how much traffic could the server handle. Also we have to see the impact of the MJPEG/MPEG4/H.264 decoding impact on the server/client. Also, we have to consider the storage sub system in term of IOPS, the throughput, the OS block size, and so on as this will impact the processor and the memory as well.

I have couple of documents that could help.

The storage performance will need separate articles to be covered. As an example, RIAD 5 has higher throughout than RAID 6; what will be RAID controller memory; Are we talking about internal storage, DAS, NAS, or SAN;

I have couple of documents that I would like to share and will shed more light into this subject, but I don't know how to share it. Thanks

Impact of using Virtual Machine environment for the applications would be good to see.

Joel

1. Start with a common entry level E3-1230v3, 8gb ram and single 3TB 7200rpm sata hdd

2. Simulate 32ch 1080p 3mbps load with software motion detect recording

3. Log CPU, RAM and storage IOPs usage for server and client processes.

4. Compare!

We have released preliminary VMS server load test results.