> The biggest tuning performance issue is the disk I/O, which is independent of the virtual appliance. We've seen huge variances between using generic iSCSI initiators and purpose built VMware storage packages such as StorMagic. The higher end packages generally utilize VMware's API for higher throughput rather than their generic I/O drivers. Scott, so do you recommend StorMagic and if so, in what applications? All? Higher camera count / throughput ones? StorMagic seems pretty expensive - http://www.stormagic.com/shop.php
Virtualization fundamentally reduces system performance because it consumes resources itself and acts as middleware application for all I/O operations. Properly configured this impact is minimal. A virtual appliance contains default settings like minimum CPU, RAM, I/O requirements so the system administrator can't allocate less than the minimum resources. We've got a video on our website that shows the VMware screens with these parameters during installation here: exacqVision Virtual VMS Software for VMware host video servers | Exacq
. BTW, there's also a link to a great VMware video that explains virtual appliance basics. The biggest tuning performance issue is the disk I/O, which is independent of the virtual appliance. We've seen huge variances between using generic iSCSI initiators and purpose built VMware storage packages such as StorMagic. The higher end packages generally utilize VMware's API for higher throughput rather than their generic I/O drivers. A system administrator with a VCP certification has at least been trained on how to monitor this bottleneck in VMware, though they obviously don't train users on the configuration of 3rd party storage software.
No, it's a combination of software addressing and disk I/O capacity. The server software is capable of addressing 255 cameras, but we only rate it for 128 because disk I/O bottlenecks become common over 128 cameras (depending on data rate of course) The server application normally doesn't consume much memory unless there's a disk I/O backup. It's normal operation for this to fluctuate such as when file reads and deletes are occurring or a user is doing a search. If the disk I/O system is overloaded, the writes back up into RAM and our software is forced to throttle. This phenomenon occurs with any software when the hardware won't support the disk I/O requirements of the application. Therefore more RAM generally only delays the inevitable, it won't prevent it.
I'll take a shot but I'm assuming 128 cameras could be a RAM issue because exacq is a 32 bit app? We have about half a dozen clients right now running exacqVision in a virtual environment, all started before the appliance even came out and they've been running great. Granted we're talking between 12-24 cameras here. Exacq hardware utilization (CPU/RAM/etc) is very low compared to other VMS I have used it seems to make sense that you could get great results through virtualization. Not to mention a good way to get High Availability if you have multiple physical hosts.
Scott, very helpful and detailed feedback. Is the 128 camera limitation a Windows OS issue or an Exacq application one? Also, one other thing. Is the tuning on VMWare deliver better performance than just running on Ubuntu without virtualization and without any tuning. The reality is most people are not going to spend the time or get the training to tune. To that end, I am curious if the 'built-in' tuning has performance benefits?
* A virtualized appliance is mainly targeted to customers with existing IT managed VMware environments. These customers strongly prefer, or require, utilizing common hardware, software and architectures for ease of maintenance.
In regards to channel count,virtualization saves server hardware if video is being written to a networked storage device. We do not support multiple video server services running in the same OS instance, so a physical server is limited to 128 cameras where a virtual server is only hardware limited (512 cameras is a reasonable ceiling, though theoretically it can go much higher). Consolidation is not possible when using local storage because a CPU and OS are required to manage disk I/O.
* Sandra doesn't perfectly simulate all VMS operations (create and fill container files, read large files, scan the directory, small random reads/writes to databases), but I think you can ballpark it with the following:
1) Configure Sandra for 90% writes with parameters equal to your existing system (ex: 1GB files in 64K blocks). Set it to 50/50 sequential/non-sequential writes.
2) Configure Sandra for 10% reads, everything same as above.
3) Divide the above number by at least 3, probably 5, to arrive at the usable throughput for the VMS. This divisor compensates for two things: A) Sandra tests on disks with free space, which is far more efficient than the VMS environment of a full drive system with constant file deletion B) The VMS has some random reads/writes (like scanning a file directory) that are less efficient than large reads/writes tested above.
* With equal configurations, there is no performance difference between a virtual appliance (made by Exacq) and a virtual machine (custom built by user). The difference is the time to reach parity and support. An experienced VMware administrator may get there with 8 hours of work over the course of a week, but will get there in an hour or two with the appliance. Also, we only support virtualization with our appliance because it removes a ton of variables that cause performance problems and we include necessary support tools.
Lastly, VMware has some nice performance monitoring tools that give users a quick visual reference to resource usage or overload. We have some VMware screen shots in appendix C of the exacqVision virtual manual if you're interested: https://exacq.com/auto/specsheet/uploads/exacqVision%20Virtual%20Appliance%20User%20Manual.pdf
I have a couple of questions: - If physical servers are greatly underutilized with Exacq, why do I need to use a virtualized appliance? Can't I just add more cameras to a powerful physical machine. (Note: I recognize the HA benefits but I am looking at just channel capacity alone here). - Good reference to third party disk benchmarking programs. Is there any specific setting or type of test to use with Sandra SiSoftware – Windows, GPGPU, Android, iOS analysers, diagnostic and benchmarking apps
How applicable are the results to video streaming? - Why an Exacq virtulized appliance vs. just running a Ubuntu VM and loading Exacq on? Outside of the modest time savings in setup, any specific performance or reliability benefits?