Virtualization And Exacq

Exacq recently announced a VMWare appliance exacqVision Virtual VMS Software for VMware host video servers | Exacq In a discussion in the old LinkedIn group, an Exacq product manager posted a detailed technical explanation, copied below: "Here are several important points related to virtualization that we learned while creating our VMware certified exacqVision virtual appliance. * Most physical servers are greatly underutilized, so virtualization can reduce hardware costs. Our server application is very CPU/RAM efficient, so we were able consolidate 4 physical servers into one virtualized server (128 vs. 512 cameras). If you use a VMS with server side motion detection, then the ratio is lower because more CPU power is required. Some video analytic packages don't virtualize because they offload video analysis to the GPU, which virtualization abstracts from the application. * Virtualization is great if you need high uptime. Jayson mentioned High Availability, which will restart your guest (application) from a failed host to an operational host. Think 5 minute failover. For a lot more money, Fault Tolerance will transfer your application to another host in a second or two using memory mapping. * Any brand of VMS will require significant disk I/O capacity to write/read large volumes of data. You can measure your disk I/O on a current physical server in MBps, then consult with your iSCSI provider to determine the IOPS (Inputs/Outputs Per Second) rating required of your storage system. A general rule of thumb is adding all of your incoming camera traffic (remember to convert from Mbps to MBps), add on 25-50% for video searches, plus any other overhead required by your VMS for maintenance. If you're building your own storage system, run a third party disk benchmarking program like Sandra on your host to measure throughput to the SAN to make sure you're in the ballpark before going live. - Virtualization requires a moderate/high level of technical competency to maintain. It only took us hours to get our software initially working in VMware, but weeks to build a completely stable, maintainable virtual appliance with all the "gotcha's" worked out. If you use VMware, I strongly suggest having a system engineer with a VCP (VMware Certified Professional). Virtualization software is like any other complex software in that there's a significant learning curve that may not be obvious at first."

> The biggest tuning performance issue is the disk I/O, which is independent of the virtual appliance. We've seen huge variances between using generic iSCSI initiators and purpose built VMware storage packages such as StorMagic. The higher end packages generally utilize VMware's API for higher throughput rather than their generic I/O drivers. Scott, so do you recommend StorMagic and if so, in what applications? All? Higher camera count / throughput ones? StorMagic seems pretty expensive - http://www.stormagic.com/shop.php
Virtualization fundamentally reduces system performance because it consumes resources itself and acts as middleware application for all I/O operations. Properly configured this impact is minimal. A virtual appliance contains default settings like minimum CPU, RAM, I/O requirements so the system administrator can't allocate less than the minimum resources. We've got a video on our website that shows the VMware screens with these parameters during installation here: exacqVision Virtual VMS Software for VMware host video servers | Exacq . BTW, there's also a link to a great VMware video that explains virtual appliance basics. The biggest tuning performance issue is the disk I/O, which is independent of the virtual appliance. We've seen huge variances between using generic iSCSI initiators and purpose built VMware storage packages such as StorMagic. The higher end packages generally utilize VMware's API for higher throughput rather than their generic I/O drivers. A system administrator with a VCP certification has at least been trained on how to monitor this bottleneck in VMware, though they obviously don't train users on the configuration of 3rd party storage software.
No, it's a combination of software addressing and disk I/O capacity. The server software is capable of addressing 255 cameras, but we only rate it for 128 because disk I/O bottlenecks become common over 128 cameras (depending on data rate of course) The server application normally doesn't consume much memory unless there's a disk I/O backup. It's normal operation for this to fluctuate such as when file reads and deletes are occurring or a user is doing a search. If the disk I/O system is overloaded, the writes back up into RAM and our software is forced to throttle. This phenomenon occurs with any software when the hardware won't support the disk I/O requirements of the application. Therefore more RAM generally only delays the inevitable, it won't prevent it.
I'll take a shot but I'm assuming 128 cameras could be a RAM issue because exacq is a 32 bit app? We have about half a dozen clients right now running exacqVision in a virtual environment, all started before the appliance even came out and they've been running great. Granted we're talking between 12-24 cameras here. Exacq hardware utilization (CPU/RAM/etc) is very low compared to other VMS I have used it seems to make sense that you could get great results through virtualization. Not to mention a good way to get High Availability if you have multiple physical hosts.
Scott, very helpful and detailed feedback. Is the 128 camera limitation a Windows OS issue or an Exacq application one? Also, one other thing. Is the tuning on VMWare deliver better performance than just running on Ubuntu without virtualization and without any tuning. The reality is most people are not going to spend the time or get the training to tune. To that end, I am curious if the 'built-in' tuning has performance benefits?
* A virtualized appliance is mainly targeted to customers with existing IT managed VMware environments. These customers strongly prefer, or require, utilizing common hardware, software and architectures for ease of maintenance. In regards to channel count,virtualization saves server hardware if video is being written to a networked storage device. We do not support multiple video server services running in the same OS instance, so a physical server is limited to 128 cameras where a virtual server is only hardware limited (512 cameras is a reasonable ceiling, though theoretically it can go much higher). Consolidation is not possible when using local storage because a CPU and OS are required to manage disk I/O. * Sandra doesn't perfectly simulate all VMS operations (create and fill container files, read large files, scan the directory, small random reads/writes to databases), but I think you can ballpark it with the following: 1) Configure Sandra for 90% writes with parameters equal to your existing system (ex: 1GB files in 64K blocks). Set it to 50/50 sequential/non-sequential writes. 2) Configure Sandra for 10% reads, everything same as above. 3) Divide the above number by at least 3, probably 5, to arrive at the usable throughput for the VMS. This divisor compensates for two things: A) Sandra tests on disks with free space, which is far more efficient than the VMS environment of a full drive system with constant file deletion B) The VMS has some random reads/writes (like scanning a file directory) that are less efficient than large reads/writes tested above. * With equal configurations, there is no performance difference between a virtual appliance (made by Exacq) and a virtual machine (custom built by user). The difference is the time to reach parity and support. An experienced VMware administrator may get there with 8 hours of work over the course of a week, but will get there in an hour or two with the appliance. Also, we only support virtualization with our appliance because it removes a ton of variables that cause performance problems and we include necessary support tools. Lastly, VMware has some nice performance monitoring tools that give users a quick visual reference to resource usage or overload. We have some VMware screen shots in appendix C of the exacqVision virtual manual if you're interested: https://exacq.com/auto/specsheet/uploads/exacqVision%20Virtual%20Appliance%20User%20Manual.pdf
I have a couple of questions: - If physical servers are greatly underutilized with Exacq, why do I need to use a virtualized appliance? Can't I just add more cameras to a powerful physical machine. (Note: I recognize the HA benefits but I am looking at just channel capacity alone here). - Good reference to third party disk benchmarking programs. Is there any specific setting or type of test to use with Sandra SiSoftware – Windows, GPGPU, Android, iOS analysers, diagnostic and benchmarking apps How applicable are the results to video streaming? - Why an Exacq virtulized appliance vs. just running a Ubuntu VM and loading Exacq on? Outside of the modest time savings in setup, any specific performance or reliability benefits?