First of all, all VMSes must be tested in 100% identical environment. It means that the same physical machine, OS, network and set of cameras must be used for all tests.
1) We suggest to use at least two different computers
- low end like atom with 2GB ram and 1-2 HDDs
- middle-end i3 or i5 with 4-8Gb of Ram and at least 3 HDDs
Tips: some VMSes are optimized for certain CPU instructions that may not be supported by low end systems ( like atom ), this is why simple results extrapolation would not work.
We do not think it is necessary to perform tests on high-end server equipment (like XEON processors, huge RAID etc) because:
a) Results obtained from tests on middle-end systems can be extrapolated to high-end segment,
b) It may require too many setup and too any cameras to really feel the difference.
2) As for camera setup: the many different cameras you use, the better it will be. One note: all cameras should be properly discovered and set up in all VMSes.
3) Network. We suggest to use 2 gigabit NICS: one for cameras and one for client connections. In low-end systems just one NIC could be used. All switches and VMS servers must be connected by CAT6 cables.
The major bottlenecks that may be tested are:
- HDD usage (direct correlation with the maximum number of cameras that can be recorded simultaneously)
- CPU usage
To estimate HDD usage:
a) Find a software to measure HDD usage.
b) Isolate OS and VMS archive: do not write archive on OS hard disk drive (do not mistake partition with physical HDD!). That’s why we suggest you to use 2 HDDs.
c) Set up VMSes to record with the same FPS, bitrate, resolution. We suggest to set up continuous recording on max fps/resolution/quality that cameras may support (not motion recording).
d) Remember that maximum HDD usage occurs when HDD is writing and erasing the existing archive in the same time. So the disk must be full in the moment of measurement and activate the corresponding feature in each VMS (the oldest recording should be erasing).
e) To ensure that the software is able to handle cameras try to open streams from cameras in client (archived footage). The playback should be smooth and frames must not be dropped.
f) The measurement must be done at the moment while no client is connected to cameras.
g) Also it will be useful to compare HDD load while the recording is in progress and all cameras are viewed by client (archived footage).
To estimate CPU usage:
a) CPU usage can be measured by Windows Process Monitor.
b) It may be necessary to record many cameras to overload CPU so either use a lot of HDD or compare the average CPU load on the different VMSes.
c) Pay attention that some VMSes do software motion detection or transcoding.
As for memory usage estimation, different manufacturers may use some tricks to increase memory usage but decrease CPU one. The memory is cheap in comparison to CPU or HDD so I suggest to omit the estimation because the overall picture may be blurred.