VMS Remote Monitoring TestedBy: Sarit Williams, Published on Aug 21, 2013
Remote monitoring of VMS systems is often critical. Large organizations have campuses, offices or branches scattered across a country, with a need to access that surveillance video. Indeed, all organization, large and small, frequently want to watch video from their homes or while on the road.
Limited bandwidth remains a key problem, with remote locations often only providing 1Mb/s upstream to send video out. Worse, many locations (e.g., banks and retailers) want to constrain how much bandwidth video takes up to ensure transactions are always promptly processed.
At the same time, megapixel adoption has surged, becoming the de facto standard for new IP video surveillance systems, raising the question of how well remote monitoring will work with increased strain.
We decided to test performance of remotely monitored VMS systems. We used a tool to throttle bandwidth to 3 levels:
- 56kb/s: Worst case scenario for legacy systems
- 256kb/s: Challenging but common scenario with limited bandwidth
- 1.5Mb/s: "Good" bandwidth for remote monitoring
We then tested the following VMSes: Avigilon, Exacq, Genetec, Milestone, Network Optix/Digital Watchdog and VideoInsight.
We started with thick clients and then tested web clients. (Note: a future report will be dedicated to mobile applications).
The Key Questions
This report examines and answers the following:
- How good or bad was performance at 56kb/s, 256kb/s and 1.5Mb/s?
- How did performance vary between a single and 4 camera view?
- How usable was live monitoring vs investigations?
- What difference in performance occurred across cameras and resolutions?
- What optimization techniques were available to improve performance?
- How did manufacturer performance vary?
Here are the key lessons from our testing:
- Higher definition cameras clearly and significantly increased load times over SD ones. 3MP took about twice as long on average as SD.
- Going from 56Kb/s to 256Kb/s made a dramatic difference in usability in almost all thick clients. Going to 1.5Mb/s made all VMS thick client almost as responsive and fast as having unthrottled connections.
- Thick clients were far less usable than web clients, largely because web clients offered better options to reduce bandwidth demand (primarily transcoding).
- Even among web clients using transcoding, performance differences were sizeable with Exacq loading the fastest and providing the most granular options as well as ability to transcode for search as well.
- Among the thick clients, Genetec Security Center exhibited serious problems maintaining a connection at 56kb/s and 256kb/s. Client connections kept dropping not giving cameras a chance to load. Milestone had similar issues.
- While multi-streaming with a secondary low stream is billed to improve remote monitoring, in our tests, many VMS's low stream took almost as long to load as the high stream with negligible benefits in real-time performance.
Throttling Bandwidth Tool
This short video shows the tool (WAN Connection Emulator) we used to throttle bandwidth and simulate various network conditions. We highly recommend it for anyone interested in understanding remote monitoring problems.
Thick Client Performance
The lowest bandwidth used, 56kb/s, was the most stressful for the VMSes in both a single camera view and especially in 4-camera layout. However, marked improvement can be seen when bandwidth increased to 256kb/s and 1.5Mb/s.
Single Camera Performance
- 56kb/s - Performance at 56kb/s across all VMSes was very slow. In a single camera view, the HD streams took as long as 100 seconds to load, with 30 - 40 seconds average. This can seem like an eternity when waiting for video to display. Cameras dropped constantly and or flickered and showed stale images 10 seconds old or longer. The lowest was when using a low quality secondary stream defaulted was 10 seconds.
- 256kb/s - At 256kb/s there was at least 50% and no dropped cameras, though occasional flickering still occurred.
- 1.5Mb/s- At 1.5Mb/s performance was greatly improved when compared to prior restrictions. Flickering and dropped cameras were rare and overall improvement was dramatically better than 56kb/s.
4 Camera View Performance
- 56kb/s – Layout view at 56kb/s was almost non-existent. As soon as one camera appeared, another disappeared maxing the streaming to one camera at a time. Cameras dropped constantly and or flickered and showed stale images 20 seconds old or longer. Time to load a layout with two or more at one time took 102 seconds for the longest and 26 seconds for the quickest one where multistreaming was built in (HD Witness/DW Spectrum).
- At 256kb/s, there was a major improvement, though flickering and dropped cameras still occurred.
- At 1.5Mb/s performance was greatly improved when compared to prior restrictions. Flickering and dropped cameras was rare and overall improvement was ~90% when compared to 56kb/s
The chart below shows the difference between views and bandwidth available (in seconds):
Optimization Techniques Web Clients
Overall, web clients offered significant better usability and quicker load time that thick clients, as shown in this graph:
There are 3 main techniques VMSes offer to improve performance at very low bandwidth.
- Resolution (Image size) and quality (compression) reductions
Web Client / Transcoding
This option showed the most improvement in overall performance. Although all VMSes, (except Video Insight) employ this method in their thin client. However, transcoding implementation effectiveness differed significantly.
Exacq’s implementation of Transcoding was the most notable, allowing the user to dynamically change size and quality of the stream for individual cameras, layouts and playback thus allowing some streaming, albeit at a lower quality instead of no streaming or significantly delayed as in the other VMSes.
In contrast, the majority of VMSes used transcoding on the backend automatically, transcoding the H.264 stream to a JPEG or MJPEG keeping quality and size the same- not taking into account reducing the image size or quality or let the end user choose a lower stream.
Some thin clients are stripped of major functionality and admin level management features, but for the purpose of viewing critical video remotely and/or investigations they suffice.
Here is an overview video:
Touted as a major enhancement, multistreaming showed slight improvement in single camera view (typically half the resolution as full), with upward savings of 5-10 seconds per camera (versus 25 - 30 at full). However, there was a collective improvement in single camera load time when a layout was used, and the cameras low stream was configured to use the lowest resolution and lowest quality. However, once a camera came up the other one disappeared and so on- even with lower streams.
Moreover, how the low stream is configured and whether the user can select it remotely hugely impacts the benefit of having such a feature. In, Avigilon the second stream is added automatically, and the user cannot control its settings relying on the VMS to choose. [Update: Network Optix added controls in their new 2.0 release.]
In Milestone’s Web Client the default stream is used (multistreaming in Corporate only). If low stream is set as the default it affects local users as well- not just Remote.
Exacq’s implementation in the Web Client allows a user to “configure” low streams dynamically affecting only their session, not local users. On the other side of the spectrum Video Insight doubles the users number of cameras in the server list and expects the end user to know which has the lower stream (based on visual trial/error and good naming conventions).
Here is a video:
As we dropped stream resolution from 3MP down to QCIF for a given camera, load times radically decreased. This should not be surprising as bitrate is a key driver of load time. The tests confirmed the impact higher resolution has:
VMS Specific Issues
A few other VMS specific issues are worth noting:
- Genetec: Genetec exhibited unstable connectivity between thick client/server; dropping connection every 2-6 minutes due to low bandwidth of 56k and 256k making it difficult for cameras to even try loading. At 1.5Mb, the cameras wasted time in loading an image in Remote stream due to fetching entity details and a stock photo that appears before the first image is displayed.
- Milestone: Milestone's thick Client exhibited similar stability issues: dropping the server/client communication as soon as bandwidth was throttled to the lower 56k and 256k speed rates. Additionally, the web client in IE10 had difficulty trying to load the image. Unlike IE, Chrome was able to load the initial image.
- Avigilon: Avigilon's Web Client constantly crashed in IE10 upon Activex installation attempts; the only browser supported for remote viewing. Tech Support is investigating the issues.
- Exacq: Exacq's Web Client performance at the lowest bandwidth showed the best implementation and consideration of remote users. They offered two types of web clients: Advanced (reviewed in Transcoding video) and Simple to accommodate even 56k. The Simple Web Client option delivers JPEG images without any advanced UI options or playback to quickly view streaming video while allowing users to select the Transcoding size and quality as well.
- Network Optix/Digital Watchdog: As one VMS where low streams are built in and added automatically it does not allow the user to choose the configuration of the low stream (fps/res/quality) and camera loading at lower bandwidth resulted in warped images initially, though the VMS was fluid and responsive at the lowest bandwidth save camera images. Network Optix also transcodes recorded video. [UPDATE: The newest version 2.0, released the same day as this report, adds in controls for the secondary stream.]
- Video Insight: Video Insight's Web Client does not support transcoding and serves the same image as their thick client which showed consistent increased load time. Furthermore, at lower bandwidth rates (56k and 256k) interacting with the either client (web or thick) via digital zoom or switching cameras/layouts results in a frozen, unresponsive client.
VMS Versions Tested
- Genetec version Server/Client 5.2.1045.11, Web Client 3.1
- Milestone Enterprise version 8.1a and Corporate server version 6.0a/Client 8.0a, Web Client 2.5.
- Avigilon Server/Client/Web Client version 126.96.36.199
- Exacq Server/Client/Web Client version 188.8.131.52600
- Network Optix/Digital Watchdog Server/Client/Web Client version 1.51.3031
- Video Insight Server/Client/Web Client version 184.108.40.206