You Cannot Trust the ONVIF Chairman

JH
John Honovich
•Published Jul 17, 2015 04:00 AM

A fascinating question:

Can you trust salesman?

Posed SourceSecurity.

Unfortunately and ironically, the ONVIF chairman [link no longer available] shows you why should not.

*** '*********'

*** ***** ******** *** *****, "*** End ***** ***** ****** **** ******** Salesmen?" *** ****** ** ********* **** you *** ***** ******** ******* *** can ***** *****, **** *** ************ it:

"** ***** **** ******, ********* ******* minimum *********** ************, ** ***conforming ******* *** ********* **************** at the most fundamental level."

***** ** **** ******, **** ***** positive, *** ************* ********* ** ****************. **** ** **** understood ** **** *********** *** *********** by*** ***** *****. **** ***** *********** *** *** guaranteed *** *********** ******** **** ****** detection *** *** ******* ********** ** not '************'.

** ****, ** ** ******** *** he ***** ******* ****, ********** ***** that *** *** **** *** ******** posed *** ******* * ******** ****** about ***** ****** ****** * ********** person ** ******* * ***********, ****** answer.

Update: ***** *** ******** ** **********

****** **** *** ******** ***** *** [link ** ****** *********]: 

"***** **** *** ******* ************ ** conformance ** ***** ********** ** ***********."

*** **** *** *****'* *** ** any ******* ****** ***** ******* **** 'guarantees' **** ****.

Not * '*********'

*** **** ** *** ********* ******* wrong, *** *** ***** ******** *********** to ***** ******** ****. **** ****,*** ******** ************ "***** ** *** * ******** organization", ****** ********** "** ** *******" who "****(*) *** ******** ** ****** that ** *** **** ******** ***." As **** **** ******* *** ** our **** ** ************* ****** ***** ***********, *****'* **** ******** ** ********* their '********' ********** ************* ** *** be *************.

The **** ********

*** ***** ******** [**** ** ****** available], Per **ö****** [**** ** ****** *********], [link ** ****** *********] ** * long **** **** ******** *** **** made ******** ** *****, ***** ***** last ******** / ******** **** ** Samsung.

*** ****** *** *** **** ********* ONVIF ** ******* ****** ***** * little ************** ** ** *** **** you *** *** ********'* ********, ** does **** *** ******.

The ******** *********

*********, **** *********** *** ***********. ** can ** ****, ******, ** ********* which **** *** ******, ******* *** good ***** *** *** ***** *******.

** *****, ******, ******** *** ** honest ********* * ******** ***** ***** honest.

Comments (22)
DW
Dale White
•Jul 17, 2015

Standard, Schmandard. I couldn't find my rant from a year or two ago on video standards, but I'm sure you don't want to hear it again anyway.

This is on the manufacturers, of course.

Today, there are two video standards; NTSC and PAL. When manufacturers decide that good products are what sustains their business, not the latest gadgetry, this situation will correct itself. It's not a thousand more pixels, or 6 more feet of IR illumination. It's a good product at a good price that plays with the other products that are available.

Keep engineering systems that are made up of proprietary components that are incompatible with the rest of the world and the standards situation will continue to languish. And, using “booth babes” to sell the latest, not very useful gadgets, is ineffectual as well.

I concede that this is a far more complex problem than when the previous standards were conceived. From the imager all the way through to the RJ45 connector, the configurations are endless. There are answers though. It is not an insurmountable task.

PTZs may be the easiest example. If you build a PTZ camera that is only compatible with your protocol and I’m not a big fan of your user interface or controller, you’re just asking the customer to keep looking.

I do believe we’ll get there but it’s going to take a shift in attitude. (see Apple vs. PC)

(4)
FY
Frank Yeh
•Jul 19, 2015

When it comes to standards there is a difference between intent and execution.

Having a standard gives everyone in the industry the same target. Whether they decide to shoot at it and how close they come to a bull's eye is up to the manufacturer, not the standards body.

So let's look at Per's statement a little more carefully...

"By their very nature, standards specify minimum operational requirements"

IE ONVIF defines the target.

"so the conforming devices can guarantee interoperability at the most fundamental level."

IE it's not ONVIF that is guarunteeing ineroperability, it is the devices.

So I see nothing inaccurate with the statement, but people could definitely interpret it incorrectly.

The fundamental problem is that device manufacturers can claim conformance without the need to back it up. We saw a similar situation in the 90's when IP was becoming the standard for networking. The IETF defined the standards, manufacturers would claim to be compliant, and no one was confirming their claims.

Into this void stepped an independent testing organization (I want to say it was Network Testing Labs but could be wrong) and that really changed the game. Eventually successful test results became required on RFPs and manufacturers could no longer simply claim to be conformant, they ended up paying to have their devices independently tested and made sure they were compliant before submitting them for testing.

So today there is only one independent organization that has tested various manufacturers' devices for ONVIF compliance and if you're reading this you should already know who it is.

If system integrators and consultants could specify a requirement for independent compliance testing and certification, that would improve the situation immensely.

(1)
JH
John Honovich
•Jul 19, 2015
IPVM
In reply to Frank Yeh

"IE it's not ONVIF that is guarunteeing ineroperability, it is the devices."

You would make a good defense lawyer, Frank :)

It's the devices using ONVIF, so when the ONVIF chairman is pitching ONVIF as a solution, it's hard to disassociate the ONVIF standard being the enabling component of the 'guarantee'.

"they ended up paying to have their devices independently tested and made sure they were compliant before submitting them for testing."

That's one approach. Or they could have a real reference implementation making it simpler for companies to implement and reducing the amount of variance.

"If system integrators and consultants could specify a requirement for independent compliance testing and certification, that would improve the situation immensely."

Even if they could, how many are going to pay for this testing and certification? That was part of the aim of ONVIF, that this could be mature and robust enough that each individual party could avoid the expense and time of doing this.

FY
Frank Yeh
•Jul 19, 2015
In reply to John Honovich

LOL When you work for IBM you learn to watch those words very carefully. Half of the battle for releasing new product is getting legal approvals. There's a lot more lines of code than lines of legal text but we have compilers and testing tools to verify the code and only have lawyers to validate the legalese.

I totally agree that a reference implementation is essential. It's something we spent a lot of effort on when I was working with the PSIA.

Manufacturers will not pay for testing & certification unless it becomes a barrier to winning business. This is where SI's and consultants come in_ they could influence RFPs and customer requirements to make certification a business requirement if there was a certification.

So there is a chain of dependencies...

  • manufacturers won't need to get certification until the SI's and consutants make it necessary
  • they can't make it necessary until there is a certification and someone who can certify
  • nobody can realistically certify with objectivity if there is no reference implementation_ if someone writes one and becomes the certifying body, then they essentially own the standard which is not what we want

Right now we have a 'standard' and it is becoming necessary to claim compliance but not to certify it. If ONVIF provided a reference implementation then the dominos could start being toppled.

U
Undisclosed #1
•Jul 19, 2015
IPVMU Certified
In reply to Frank Yeh

...they can't make it necessary until there is a certification.

There IS certification already and most manufacturers DO certify, but not because SI's have not made it necessary.

IMHO, device manufacturers DO certify because of the ability to instantly integrate their hardware into third-party recorders without waiting (or paying) for a driver to be written. At least that's the theory... ;)

John's point, to which I agree, is that it would reduce the incompatibility and allow developers to write more robust code with an official reference implementation.

JH
John Honovich
•Jul 19, 2015
IPVM
In reply to Undisclosed #1

Though all of us like the reference implementation idea, the problem is that it is a little late at this point, given that there are ~5000 conformant products already with code that is mostly but not always compatible.

I still think it would help going forward, but there would be some limits of what it could rectify for previous releases.

U
Undisclosed #1
•Jul 19, 2015
IPVMU Certified
In reply to John Honovich

Profile S is not the final word in any event. There's talk that a new profile will be created to accomodate H.265. If and when so, let's hope they dont repeat the mistakes of the past.

FY
Frank Yeh
•Jul 19, 2015
In reply to Undisclosed #1

I'm not aware of any independent certification available. Are you referring to the ONVIF compliance process?

The ONVIF web site says:

How do you ensure that manufacturers are conforming to the specification?

ONVIF provides a conformance process specification, a test specification and a test tool. Conformance is based on a self-declaration process. After meeting all the requirements defined in the conformance process and specifications, members can declare conformance to one or more ONVIF Profiles, as applicable.

How can I tell if a product is ONVIF conformant?

The only way to verify that a product has been declared to be ONVIF conformant (by ONVIF) is via the ONVIF conformant product search page at http://www.onvif.org/FindaProduct/ProfileProducts.aspx. ONVIF DOES NOT provide certificates of conformance or other guarantees of conformance. Products that have been verified to be ONVIF conformant via the ONVIF conformance process are able to utilize the appropriate Profile mark on that product.

The ONVIF Conformance Process Specification is full of language that clearly states that the statement of conformance is the member's responsibility and not ONVIF's. Given IPVM's findings of multiple manufacturers faking ONVIF conformance and the less than stellar results in their ONVIF Mega Test, it's not outlandish to infer that some of the products who have declared conformance would not pass independent testing.

I will retract my statement that a reference implementation is needed for an independent testing organization (but I still think one would be valuable). The IPVM ONVIF Mega Test was performed using the official ONVIF test tool and an open source tool (that did not really provide any usable conformance results). This finding in particular is a cause for concern:

"In our tests using the ONVIF Device Test Tool, no cameras (out of 15+) passed a conformance test, even after following ONVIF recommendations and settings manufacturers used for their own conformance tests."

So ONVIF provides the specs and a test tool and manufacturers self-declare that they passed the conformance testing but then IPVM tests products that are supposedly conformant and finds they do not pass the tests.

To me it seems like there would be value in an independent organization like IPVM performing the testing and certifying conformance, but @John Honovich has not taken the bait (yet).

(1)
JH
John Honovich
•Jul 19, 2015
IPVM
In reply to Frank Yeh

The problem with testing is the sheer number of devices involved. We know that many devices pass the ONVIF conformance test and then fail when integrating certain other devices that also passed the ONVIF conformance test. So how do you test / verify that? Test every combination?

And, again, I am not against a reference implementation. It would help going forward but there's already so much out there.

They need to tighten things for future versions and then a reference implementation might help reduce issues in real world integration.

And still, though, you have companies claiming to be ONVIF conformant yet are not, like OnSSI, who I have told repeatedly about this. You think OnSSI cares? Evidently not. You think ONVIF cares. Evidently not. Same deal with Cisco's VMS.

An organization and its leadership need to have pride and determination to deliver a quality offering. ONVIF obviously does not.

FY
Frank Yeh
•Jul 20, 2015
In reply to John Honovich

Yeah there are a whole lot more "ONVIF devices" out there than networking gear that was certified so your point about combinations is certainly valid.

In a meeting with the development manager of Windows NT a long time ago he said that the vast majority of Windows BSOD's were caused by incompatible 3rd party device drivers. While they tested about 100,000 combinations of these regularly, they figured they were covering maybe 10% of the possible combinations.

I tend to try to look for precedents (there's that legal influence again) and other standards bodies (EG IETF, W3C) do not police claims of conformance. Even with well-known and pervasive standards you see major differences in the way products implement them. EG the same "standard" web pages often do not run the same in IE, Chrome, and Firefox.

A major difference between ONVIF and IETF/W3C is that the latter do not make any attempt to say who has properly implemented the standard while ONVIF has their conformant product search page.

When vendors claim to be conformant but are not listed on the page that is a problem, which I think is your point. It would seem incumbent on the consultants and SI's to validate the claims of vendors against the page, or since IPVM has done this, avail themselves of yet another piece of IPVM value. :)

U
Undisclosed #1
•Jul 19, 2015
IPVMU Certified
In reply to Frank Yeh

I'm not aware of any independent certification available. Are you referring to the ONVIF compliance process?

Yes. Sorry for the sloppy language.

But my point is: do we really need an independent certification of an independent organization's standards?

Independent meaning independent of the independent organization.

Wouldn't they just use automated tools like the one used to self-conform anyway? Of course they could improve them, but so should ONVIF.

Remember, this is not a toaster being stressed to see whether it will spontaneously combust. It's an abstract messaging protocol. It can be tested, virtually thru a simulation suite, for conformance without requiring human intervention.

And if found conformant then it should be certified by the same organization who defined the compliant behavior to begin with.

U
Undisclosed #1
•Jul 20, 2015
IPVMU Certified
In reply to Frank Yeh

...ONVIF DOES NOT provide certificates of conformance.

To be clear. They provide them on their website. They are certificates generated automatically when passing the test tool. The manufacturer has camera in hand and swears it's the one on the document. The detail results of the test tool are here.

Frank I understand your concern for the proccess, but I wanted to make it clear that there was some documentation available from ONVIF. What it's worth, I'm not sure...

(1)
FY
Frank Yeh
•Jul 20, 2015
In reply to Undisclosed #1

Thanks for the clarification and references. I have never had to register a device with ONVIF so have never actually gone through the process. So my perspective comes frmo reading the ONVIF web site and specs.

Interesting that the artifacts generated by the ONVIF tool seem to be very simplistic. Do you know if these are digitally signed by the tool? IE how does ONVIF know that someone is not faking these?

U
Undisclosed #1
•Jul 20, 2015
IPVMU Certified
In reply to Frank Yeh

Do you know if these are digitally signed by the tool?

They don't appear to be. The conformance statement itself can be digitally signed, but this I believe would only authenticate the person signing it.

Here's the conformance spec and here's a verbose version of the conformance test.

So, yes the tool could be easily faked with a different camera or even a PC simulator.

One thing that might at least bring the self-conformance testing full-circle, would be a tool whereby an end-user, possibly after having difficulties and getting onvif's website address off the box, could allow their camera to be probed by an ONVIF verifier. If the camera-in-the-wild wasn't recognized by ONVIF as having a matching conformance doc, then ONVIF could pull the whole vendor off their site.

JH
John Honovich
•Jul 20, 2015
IPVM
In reply to Undisclosed #1

"Test Environment

As per ONVIF, tests should be run on a closed network. Other traffic may cause unpredictable results, and increase latency, affecting the timeouts that were set in test configuration. Working DHCP and NTP servers must be available on the network, as the test tool uses these in the conformance test. Finally, ONVIF also recommends the test tool should be run as administrator."

U
Undisclosed #1
•Jul 21, 2015
IPVMU Certified
In reply to John Honovich

As per ONVIF, tests should be run on a closed network.

*Just to be clear I'm not suggesting that a full conformance test be run over the Internet. Rather, this check would be just to make sure that what comes out of the box was what was tested originally.

The model number could be entered by the end-user, and then a few well chosen probes for firmware version / date etc. combined with a few random checks on available features etc, should generally indicate if this device/firmware is recognized and if it has ever passed conformance.

This would also root out any OEM piggybacking.

*Since this thread has long ago exceeded max_nesting_level, I cannot be entirely sure that you are responding to my last comment about an end-user tool...(or an earlier one), but I am assuming you are.

FY
Frank Yeh
•Jul 21, 2015
In reply to Undisclosed #1

Thanks, I have read the conformance spec but had never seen a fully detailed report.

After rethinking things this whole issue of conformance and certification may be overrated. If someone claims to be ONVIF compliant but is not registered on the site that does not necessarily mean their kit does not implement the protocols, only that they have not gone through the conformance procedure. Conversely, if they are registered on the site that does not guaruntee they will interoprate successfully.

There does seem to be some value to running things through the test tool but perhaps vendors should just place their reports on their own web sites and ONVIF should recommend that people just check the vendors' sites for the report. That would save them from having to maintain their page and people would still have to pay for a membership to get the tool. This would work better if the tool could digitally sign the reports.

Ultimately it comes down to what this thread started as_ a matter of who can you trust. If a vendor were to go so far as to fake a report or post a faked report, hopefully word would get out, the vendor's reputation would be ruined and nobody would buy from them.

JH
John Honovich
•Jul 21, 2015
IPVM
In reply to Frank Yeh

"If a vendor were to go so far as to fake a report or post a faked report, hopefully word would get out, the vendor's reputation would be ruined and nobody would buy from them."

Well considering dozens of manufacturers fake even being ONVIF conformant, and have for years, evidently not :(

U
Undisclosed #1
•Jul 21, 2015
IPVMU Certified
In reply to John Honovich

...considering dozens of manufacturers fake even being ONVIF conformant...

Which group of fakers do you refer to?

A. Manufacturers who use the ONVIF logo even though they are not members and their products are not listed?

B. Manufacturers who are members and products are listed, with certificates on file, but suspiciously still have trouble interoperating?

I suspect that you are talking about group A and Frank is talking about group B.

Group A's reputation doesn't get ruined as long as their products actually work. No one cares that they are NOT listed on the ONVIF site, if they (or the 'original' manufacturer) got it right.

Group B, on the other hand might make a quick buck, but will soon be buried by tech support and returns. No one cares that they ARE on the ONVIF site when their stuff doesn't work. They will have to remedy or their reputation will be ruined.

IMHO.

(1)
FY
Frank Yeh
•Jul 21, 2015

Great point about Group A and Group B.

IMHO Group B is more troubling_ they have the ONVIF "seal of approval" but customers end up with bad product.

Group A using the ONVIF logo without going through the ONVIF conformance process is probably a violation of some ONVIF terms but it would be up to ONVIF to seek action against them, and if they are ONVIF members I'm not sure if they would want to do that.

JH
John Honovich
•Jul 21, 2015
IPVM
In reply to Frank Yeh

"it would be up to ONVIF to seek action against them, and if they are ONVIF members I'm not sure if they would want to do that."

See: ONVIF Launches Enforcement Campaign

FY
Frank Yeh
•Jul 21, 2015

Thanks for the link to the ONVIF Enforcement Campaign, I had not read that one. So this campaign has basically brought us to the current point where manufacturers use the ONVIF logo with impunity. Not real effective.

Do you know if there are manufacturers who claim to be ONVIF compatible but do not use their logo? They might not be guilty of anything but false advertising if their products do not work.

Those who do use the logo and are not ONVIF members or are members who have not passed the conformance process are in violation of the ONVIF terms for using their trademark and legal action could be taken against them. It would be up to ONVIF to seek this action (I meant legal action in my previous comment). So ONVIF may haven taken action with their enforcement campaign but as far as I know nobody has been sued for trademark infringement. Not sure how effective this would be against the likes of Cantonk (the Chinese legal system is pretty strange) who really have no reputation to protect either so there you go.

BTW I got a kick out of reading the thread on Cantonk. Not that the situation is funny but the thread sure is. :)