ONVIF Mega Test 2014

Author: Ethan Ace, Published on Apr 16, 2014

In this groundbreaking report, we share findings of in-depth ONVIF testing.

ONVIF Real World Integration

We tested 14 camera manufacturers with 5 VMSes, performing 70 total integrations.

The camera manufacturers included: ACTi, Avigilon, Axis, Bosch, Dahua, DRS, Geovision, Hikvision, Honeywell, Pelco, Samsung, Sony, Wodsee (the super cheap Chinese offering).

The VMSes included: Exacq, Milestone, VideoInsight, Axxon and Wodsee. The first 2 are ONVIF Profile S conformant, the middle 2 are 1.x / archived products and the last is not officially conformant, just claims it in marketing. This provided a range of products / conformance to evaluate.

The testing checked (1) connectivity, (2) video streaming, (3) setting resolution, (4) setting frame rate and (5) enabling camera side motion detection.

ONVIF Test Tool

ONVIF has an official test tool that we accessed by paying $500 to become a member. This is the tool that each camera manufacturer uses to verify their conformance. We tested this with 15 cameras to determine how useful this tool is for verifying conformance and understanding problems.

ONVIF Device Manager

Finally, we tested the ONVIF Device Manager, an open source tool, to understand what value it offered and how it compared to the official ONVIF test tool.

Get Video Surveillance News In Your Inbox
Get Video Surveillance News In Your Inbox

Key Findings

Here are our key findings from this test:

  • Connecting a sampling of cameras from 14 different manufacturers to 5 recorders, the majority of cameras, 90%+, connected and streamed, regardless of ONVIF version and Profile conformance.
  • Profile S VMSs were far more likely to support motion detection, with Exacq and Milestone supporting 50% or more cameras. Non-Profile VMSs supported motion on nearly no cameras.
  • Support for changing framerate, resolution, and image settings (brightness, contrast, WDR) varied widely depending on VMS and ONVIF version used.
  • Compared to our 2011 ONVIF test, performance was radically better, with basic problems essentially eliminated.
  • In our tests using the ONVIF Device Test Tool, no cameras (out of 15+) passed a conformance test, even after following ONVIF recommendations and settings manufacturers used for their own conformance tests.
  • The tool itself is complex to operate, with little guidance on what settings to use (timeouts, event settings, etc.), and little clear feedback on what caused failures without intimate knowledge of low level camera and network performance information.
  • ONVIF Device Manager allows basic management, viewing, and configuration of ONVIF cameras, but does not provide conformance information and little in the way of diagnostics.
  • However, ODM may report features are available which are not, or are not properly implemented, limited its use to basic functions only.

Recommendations

ONVIF, especially the introduction of Profile S, has done much to increase camera/VMS compatibility, with most listed cameras and VMSs able to connect and stream. However, basic functions such as the ability to change resolution and framerate are not implemented across the board, and motion detection worked in few combinations, less than 25% of the time.

The ONVIF Device Test Tool is likely useful only to manufacturer technical staff, as the low level technical details required in order to properly run a test and interpret results are not available to outside parties.

ONVIF DM may be useful for basic camera configuration and feature checks, but does not perform any conformance testing, limiting its use to these tasks.

Unfortunately, the only way to truly test if a specific camera/recorder combination will work is to test it in the field. This is especially true of camera side motion detection, as neither tool discussed here can confirm that the VMS has integrated motion events specific to each manufacturer.

ONVIF Camera/VMS Field Test

We tested 14 cameras with 4 VMS systems and an NVR, varying in ONVIF conformance from Profile S to version 1.02 to claiming conformance without official listing. We tested the following parameters:

  • Does the camera connect?
  • Does the camera stream?
  • Can resolution and framerate be modified?
  • Can other image settings such as brightness, contrast, etc. be changed?
  • Does motion detection work?

The following charts detail each of these features for each VMS:

Connect/Stream

First and most basic, we tested whether the camera could be added to the recorder using proper credentials, and video streamed. In the most instances (90%+), this worked fine, with some glaring exceptions. The Wodsee NVR, which is not officially listed as conformant, crashed when the Avigilon H3 camera (Profile S) was connected. The Wodsee WIP130 would not connect to 3 out of 5 recorders.

Framerate/Resolution

Next, we tested whether each VMS had the ability to change framerate and resolution of each camera from the client. In Exacq and Milestone, the Profile S VMSs, most cameras could be changed, with few exceptions. Axxon Next, an ONVIF 1.02 VMS, was able to change resolution and framerate on all but Avigilon and Vivotek. Video Insight does not allow changing of framerate and resolution from the client, nor does the Wodsee DVR.

The non-conformant (but marketed as "ONVIF") Wodsee WIP130 could not be changed from any client whatsoever.

Motion Detection

Milestone integrates motion from most cameras in this test (9/14), while Exacq handles six (though Avigilon may be integrated by inserting a few lines in one file). In the version 1.02 VMSs, motion detection does not work at all.

Most surprisingly, the Wodsee NVR received events from Bosch, Dahua, and Honeywell cameras, though it is not officially listed as conformant.

Imaging Settings

Finally, we tested whether the VMS could adjust imaging settings such as brightness, contrast, etc., from the client. These features are purely optional in the VMS client, with Exacq, Milestone, and Axxon integrating the majority of cameras, while VI and Wodsee could not adjust any.

ONVIF Device Test Tool

This video provides a brief overview of setup and operation of the ONVIF test tool:

Full tests take about an hour or more in many cases, depending on how many features are being tested, timeout settings, and camera performance.

Test Tool Usage Issues

Though it appears straightforward, we found several issues impacted conformance testing and we could not get any tested cameras to pass a conformance test. We contact ONVIF with these issues and failed test reports for feedback, but even after following their recommendations, we found issues still remained.

These issues include the following:

Timeout and Event Settings

On the "Management" tab of the Device Test Tool are several timeout settings impacting testing. These settings include the time between the tool beginning tests, the time allowed for the device to reboot (occurs several times during testing), time the tool awaits for response to commands, etc.

Running a test with default settings resulted in large numbers of failed test steps. ONVIF pointed out that we refer to the "Feature List" XML document which is attached to each device in the conformance listing (see sample from Sony SNC-VB630) which shows what settings were used in the test. However, even though these settings decreased the number of failed steps, conformance still failed.

This screencast details how to find this XML document and adjust settings accordingly, and demonstrates how much manufacturers may differ.

Limited Feedback on Failures

Limited detailed pass/fail information is provided by the test tool, except in raw XML data for each step. This leaves users without intimate knowledge of the camera and the ONVIF spec unable to understand exactly why the test has failed.

In this screencast we show a failed test, which was run using recommended manufacturer specific settings and review the detailed feedback that is received on each step:

Test Environment

As per ONVIF, tests should be run on a closed network. Other traffic may cause unpredictable results, and increase latency, affecting the timeouts that were set in test configuration. Working DHCP and NTP servers must be available on the network, as the test tool uses these in the conformance test. Finally, ONVIF also recommends the test tool should be run as administrator.

ONVIF Device Manager

The ONVIF Device Manager (ODM) was created by a third party developer for basic viewing and configuration of ONVIF devices. This screencast shows basic use of ONVIF DM:

Unlike the official test tool, ODM does not perform any conformance tests. Instead, it is used for basic configuration and viewing of ONVIF devices. Some of this information may be quickly used to check if devices support ONVIF functions, but almost no diagnostic information is provided if specific functions do not work.

Methodology

We used the following VMS versions in this test:

  • Axxon Next: 3.5.0.819
  • Exacqvision: 6.0.8.58249
  • Milestone: 2013 (8.5e)
  • Video Insight: 5.5.26.4
  • Wodsee: V4.02.R11.63400083.12001

And camera firmware versions as follows:

  • ACTi D11: V6.06.16
  • Avigilon 2.0W-H3-BO1: 2.6.0.14
  • Avigilon 3.0W-H3-B: 2.6.0.18
  • Axis P3354: 5.40.17
  • Axis M3004: 5.40.5.4
  • Bosch NBN-733V: 5.90
  • Bosch NBN-932V: 5.90
  • Dahua IPC-HF3101N: 2.210.General 01.0R
  • DRS WatchMaster IP: v2.1.5770
  • Geovision GV-UBX1301: v2.11 2014-02-14
  • Hikvision DS-2CD2232-I5: V5.0.2
  • Honeywell HCW1F: 40.0.6
  • Pelco IME119: 2.0.0.11.8280-A0.0
  • Samsung SNB-5004: 1.13
  • Samsung SNB-6004: 2.22
  • Sony SNC-VB600: 1.12.0
  • Sony SNC-VB600B: 1.12.0
  • Sony SNC-VB630: 1.12.0
  • Sony SNC-CH140: 1.82
  • Vivotek IP8371E: IP8371E-VVTK-0100j
  • Wodsee WIP130-S20: 1.2.1.227383

7 reports cite this report:

DeskCamera ONVIF Screen Capture on Jan 05, 2017
Recording PC screens have historically been an expensive proposition, requiring dedicated hardware or specialized, proprietary software. However,...
ONVIF Video Surveillance Tutorial on Dec 19, 2016
ONVIF is well known within the surveillance industry as an interface to connect IP cameras and VMS systems but: Is ONVIF a 'Standard'? Why...
HD Analog DVRs With Exacq, Milestone and Genetec VMSes Tested on Aug 23, 2016
HD analog offerings have proven to deliver HD video at radically lower prices. However, the main limitation for professional applications is their...
Milestone ONVIF Bridge Tested on Jul 06, 2016
The Milestone ONVIF Bridge could be a breakthrough. Enterprise VMSes have long restricted 3rd party access making larger system integration more...
Universal Camera Tester Tested (IPC-8600) on May 20, 2016
Setting up cameras is often a pain. A number of test tools exist but are frequently limited on camera support or have a small screen, slow...
You Cannot Trust the ONVIF Chairman on Jul 17, 2015
A fascinating question: Can you trust salesman? Posed SourceSecurity. Unfortunately and ironically, the ONVIF chairman shows you why should...
Milestone Acquired by Canon on Jun 13, 2014
Wow. Industry insiders have been speculating about Milestone being acquired for weeks. But the outcome is surprising. Milestone has been...
Comments (38) : PRO Members only. Login. or Join.

Related Reports

Axis Guardian - Cloud VMS And Alarm Monitoring - Released on Jun 19, 2018
Axis has struggled to deliver a cloud-based managed service video platform. Video service providers have utilized AVHS for over a decade, and have...
Axis Releases First New Access Controller In 5 Years (A1601) on Jun 15, 2018
It has been 5 years since Axis 2013 entry in the physical access control market, with the A1001 (IPVM test). Now, Axis has released its second...
ReconaSense - The AI / Access Control / Analytics / IoT / Video Company Profile on Jun 12, 2018
One company's ISC West booth stood out for displaying a light-up tower of buzzwords. The company, ReconaSense, pledged to be 'making sense of it...
H.265 / HEVC Codec Tutorial on Jun 07, 2018
H.265 support has improved significantly in 2018, with H.265 camera/VMS compatibility increased compared to only a year ago, and more manufacturers...
Bosch IVA Video Analytics And Motion+ VMD Tested on Jun 06, 2018
Bosch's video analytics now ship on nearly every model, from indoor domes to high-end 5MP starlight cameras.  In this test, we evaluate Bosch's...
Hikvision PanoVu 20MP Flexible Camera Tested on Jun 01, 2018
Hikvision has released their first repositionable multi imager cameras with integrated IR included, atypical in competitors. We bought and tested...
Oncam 12MP Fisheye Camera Tested on May 29, 2018
Oncam has made their name since the early 2000s as a fisheye specialist, focusing only on panoramic cameras. To see how this specialist stacks up...
VMS Server Sizing on May 25, 2018
Specifying the right sized PC/server for VMS software is one of the most important yet difficult decisions in IP video surveillance. In the past...
Hanwha Wisenet X Analytics and VMD Test on May 24, 2018
Continuing our updated testing of camera analytics, we tested Hanwha's Wisenet X analytics for over two weeks in multiple scenes, indoors and out,...
Software Only VMS vs NVR Appliances on May 23, 2018
Should you buy your own PC/server and load VMS software on it or get a turnkey appliance (both hardware and software, e.g., NVR, Hybrid DVR) from a...

Most Recent Industry Reports

IFSEC Show Report Day 2 Report on Jun 20, 2018
IPVM is live from London reporting on the IFSEC show. The Chinese have taken over the UK, centered on Hikvision, flanked by Dahua, Huawei and a...
Mobotix Releases 'Move' Into 21st Century on Jun 20, 2018
For years, Mobotix stood resolutely against, well, every other manufacturer, selling it as a virtue: MOBOTIX equipment is designed with no...
Cybersecurity Startup VDOO Disclosing 10 Manufacturer Vulnerabilities Starting With Axis And Foscam on Jun 20, 2018
Cybersecurity startup VDOO has uncovered significant vulnerabilities in Axis cameras along with many others not yet disclosed. In this report, we...
July 2018 IP Networking Course on Jun 19, 2018
The last chance to save $50 on registration is this Thursday, June 21st. Register now and save. This is the only networking course designed...
Axis Guardian - Cloud VMS And Alarm Monitoring - Released on Jun 19, 2018
Axis has struggled to deliver a cloud-based managed service video platform. Video service providers have utilized AVHS for over a decade, and have...
IPVM Vulnerability Scanner Released on Jun 18, 2018
IPVM is proud to announce video surveillance's first and only cybersecurity vulnerability scanner. This tool allows quickly and simply...
Hikvision Corrects False Cybersecurity Announcement on Jun 18, 2018
Hikvision has corrected a false cybersecurity announcement that claimed a British government-sponsored program endorsed the cybersecurity of...
The Dumb Ones: PSA's Bozeman On Cybersecurity on Jun 15, 2018
The smart ones are the hundred people who flew to Denver and spent $500+ on a 1.5-day conference featuring Dahua as a 'cyber responsible partner',...
Amazon Ring Launches $10 Monthly Professional Alarm Monitoring on Jun 15, 2018
Amazon's Ring has announced an alarm system with 24/7 professional alarm monitoring for $10 per month, a fraction of the $30+ per month traditional...
Axis Releases First New Access Controller In 5 Years (A1601) on Jun 15, 2018
It has been 5 years since Axis 2013 entry in the physical access control market, with the A1001 (IPVM test). Now, Axis has released its second...

The world's leading video surveillance information source, IPVM provides the best reporting, testing and training for 10,000+ members globally. Dedicated to independent and objective information, we uniquely refuse any and all advertisements, sponsorship and consulting from manufacturers.

About | FAQ | Contact