Hikvision Upgrade Breaks ONVIF VMS Integration
By Ethan Ace, Published Oct 31, 2017, 11:32am EDT
Comments (53)
At this point we’ve installed thousands of Hikvision cameras. It’s too bad I have to subscribe to the IPVM blog to get news about Hikvision updates. While we typically test new firmware ourselves for stability with the VMS and read through all documentation prior to deployment. I may have spent hours trying to figure this out.
Great reporting by team IPVM once again. Making my membership well worth it. Moving away from Hikvision over the last two years was the best decision we’ve made.
You may want to talk with your HikVision Representative. When I spoke with HikVision back in early September they mentioned they were turning ONVIF off by default to help improve the overall security of the device. I haven't played with the new firmware, but they said it was a simple check box correction. It would be nice if you could hit it in SADP, or from the NVR though.
Do you not read the release notes for each firmware update?
Do you not read the release notes for each firmware update?
In this case, the release notes should be clearer / more explicit. It says "ONVIF function is disabled by default" but it is not clear whether that means the upgrade will disable ONVIF for existing cameras that have ONVIF already enabled.
Generally, technical users expect firmware upgrades to fix or add things, not to break things that already work. Arguably, such an action is a bug in its own right.
Minimally, if you are going to break existing functionality for devices already deployed being upgraded, it should be prominently disclosed in the release notes. Yes/no?
Did you not read my comment?
I agree, release notes clearly stated that ONFIV is disabled now. I believe, 95% of the installers just don't read it
I believe, 95% of the installers just don't read it
Dennis, I do agree that most installers don't read release notes (I don't think anyone can guess at the exact percentages but it's certainly reasonable to think it is common).
The reason why most do not read release notes is that firmware upgrades typically fix broken things and add new features. It is much less common for a firmware upgrade to actually break things (which is what Hikvision has done here). To that end, installers can typically presume firmware upgrades are benign.
You and I agree that installers typically don't read release notes. Presumably, Hikvision, with 10,000 'engineers', also knows this.
And Hikvision has to know that turning off ONVIF on existing cameras is going to break many VMS integrations (again 10,000 'engineers'). Ergo, if Hikvision wants to do this, the least they can do is prominently and proactively make this known in mainstream marketing communications. Yes/no?
It is much less common for a firmware upgrade to actually break things...
It is much less common for a firmware upgrade to actually *knowingly* break things...
actually *knowingly* break things...
For sure, lol. Even then, unknowingly breaking things is less common. However, things can break, so I do agree that firmware should be checked by integrator before deploying.
However, if a manufacturer knows they are breaking something significant, e.g., that will disconnect VMS systems, they have a responsibility to make an extra effort to communicate that.
That noted, there's a line of thought that 'APIs are a contract' and that you do not break the 'API'/'contract' when you do upgrades, i.e., the internals can be changed but the public interface (in this case ONVIF out) would not be changed. I tend to agree with that because when you do break the API (which is effectively what Hikvision did here), you cause pain and money to be spent by the user / customer.
Is there a way to mass configure the enabling of ONVIF? I ask because if my customer has a few hundred cameras deployed at their location, going back to upgrade for a vulnerability could knock all the cameras off the VMS. Having to go back to each individual camera would be a nightmare.
Im sure it will be available in batch config option, hopefully.
Yes, you can use Batch Config Tool v2.0.0.6 to bulk enable/disable ONVIF accounts.
Now, to use Hikvision with ONVIF, ONVIF needs to be enabled and an admin account and password need to be created, as shown in the clip below. These credentials are used to add the camera to the VMS/NVR, not the usual admin credentials which have historically been used.
What if you create an ONVIF account named “admin” with the same password as the non-ONVIF admin account, before doing the upgrade as a way to minimize interaction and downtime?
Anyone got the time to run that test case?
That doesn't work, because there is no ONVIF account prior to the upgrade. ONVF users weren't separate from other users prior to 5.5.
If you upgrade and then create the user, it does indeed work, though. But it still won't minimize downtime.
I'm downloading the latest iVMS-4200 from Europe to see if this can be done in bulk.
Just so I know I am understanding this correct. I can create an account called admin/password for the ONVIF account and it will load up into the VMS, but I still have to manually create the ONVIF account?
That's correct. And that account can't be created until after you do the 5.5 upgrade.
I just checked in iVMS v.2.6.2.7 and there is no way to bulk config ONVIF enable/disable or users that I can see. So as of now this is a one at a time process.
wait wait you're saying OnVIF was enabled with no credentials? You've been running naked/unauthenticated OnVIF in your lab? Are you saying it is your assumption OnVIF is supposed to run with no credentials?
No... HikVision uses a special alpha/numerical code to send API commands to and from the camera with admin level access. Oh Son of a B****!
Who is saying that? We're saying it used to use the main admin or other accounts. Now it does not.
If it makes it more secure, than Im ok with it, regardless of the pain of additional setup. Although it is a doozy that it will break integration of already installed cameras.
Can you see if there is an option on the 5.5 for optional automatic firmware updates. Until they get this done, I wont get overly excited.
Related from Sean: No Brainer Idea For Manufacturers To Stay Current With Cyber Security: Automatic Firmware Upgrades
This is, though, an example of the risks of automatic firmware upgrades because if you automatically disable something that a customer is currently using, you risk silently breaking systems.
I think this is not a big deal at all. Simply use the next backdoor to create the required user accounts and set the right settings automatically.
Using 2 sets of user accounts is a bad idea IMHO. Also, they are doing this to increase security. This to me seems like they are bandaiding their system, without solving the problem. If they simply solved the backdoor from the API, then who cares if the API command for a ONVIF snapshot is still in the API. It is protected by a username/password. Once ONVIF is re-enabled, is the backdoor still removed? Does the API command still work?
Also, for Axis, I thought that it is whatever protocol is used first. If you first connect to the camera via ONVIF, doesn't it stay enabled?
Do they have complex password rules for the ONVIF user accounts. It is easy for an admin to change a camera password when need arises, but they forget about the ONVIF account, which could lead to unauthorized access.
Also, for Axis, I thought that it is whatever protocol is used first. If you first connect to the camera via ONVIF, doesn't it stay enabled?
ONVIF is on by default, with root/pass as credentials. If you log into the camera's web interface and create a root password, it disabled ONVIF until it's re-enabled. It does not remain enabled.
Do they have complex password rules for the ONVIF user accounts. It is easy for an admin to change a camera password when need arises, but they forget about the ONVIF account, which could lead to unauthorized access.
Yes, complex passwords are required. And you cannot use ONVIF users to log into the web interface or access the RTSP stream. You can only use it for ONVIF purposes.
You could use old ONVIF credentials that admins didn't update and use ONVIF device manager to change settings in the camera.
Yes, you can use Batch Config Tool v2.0.0.6 to bulk enable/disable ONVIF accounts.
Thanks for that. I just confirmed myself before I saw your post. We'll add a note on this to the report.
Update: Network Optix Issues Notice
VMS developer Network Optix has issued a Hikvision v5.5 Firmware Fix Notice, citing this report, explaining that "Nx Witness uses ONVIF to discover, configure, and stream video from Hikvision cameras" and that therefore, upgrades will break VMS integration with Network Optix. Network Optix than explains steps to fix. Given that ~30% of Network Optix connected cameras are Hikvision, this could be a non-trivial support issue for the company.
That's very good of them! Also, linking to the source and utilizing your graphic (with credits) saves everyone's time.
Makes it evident that the release notes are not enough to notify about changes that are expected to break things.
A person called Ryan Flagler commented on the Network Optix page:
We tried what he's saying, and it's not correct. We turned off Hik-CGI, created a user named "Onvifuser" for ONVIF, and did not add it to the main user list.
Camera adds just fine to Nx Witness:
So you're saying this VMS vendor relies on an open known password to access the cameras. Sounds like a security issue.
Hikvision, they have all tried to bash you for this, but I would like to commend you on your forward thinking. I do not use ONVIF anyway, since all the VMS systems I use have excellant direct driver support. When I buy a new car, toy, or tool, I read the manual. Yup, I do. Same holds true for firmware upgrades. Who would upgrade without reading release notes. (sounds like IPVM did not read this before writing this article, and if they did, should have reported this at the beginning. That does not create conflict though, which sells subscriptions). Heck, most companies do not even provide accurate if any release notes. Also, who would be bulk updating a customers system before testing it out in the office lab/demo area. Okay, many do not have one. So try maybe one camera first.? When it does not work, THEN check release notes? Imagine that, when you download the new firmware, it forces you to also download the release notes. How does anyone not notice them in that zip file. This artcle should have been more of a congratulations than a regular Hik-bashing.
Look at the title of this article. Huh? Maybe something like “Hikvision addresses security vulnerability with ONVIF compatability”. Instead, IPVM seems to address the .01 percent of Hikvision cameras integrated through ONVIF to a VMS system and in those cases, the 1 percent of integrators or customers that do not read release notes, have not designed secure networks in the first place (so do not need to upgrade), or have not tested the upgrade.
Hikvision, keep up the good work. (I do get a little grumpy with this sensationaism article “titling” which brings out the Trump in every body. This site actually used to promote, advance, and educate in our industry with frequent positive collaboration. If you can believe it, manufacturers used to converse and provide valuable information to us. Really, Im not kidding. Now, this only directs them to the unemployment line).
Jeffrey, thanks for your comment.
We indeed did read the release notes. That's how we found out about ONVIF being disabled in the first place. The release notes clearly mention ONVIF being disabled by default. They do not, however, mention disabling it upon upgrade. This is the entire section
ONVIF function is disabled by default. The path of ONVIF configuration in web component is Configuration->Netwrk->Advanced Settings->Integration Protocol. ONVIF user accounts need to be created for application of ONVIF. Three ONVIF user levels of Administrator, User and Operator are selectable, with up to 32 user accounts. Web component, iVMS-4200 and Batch Configuration tool are available for ONVIF configuration
I actually personally support ONVIF being disabled by default, and turning off services that aren't going to be used - that's good security practice, but disabling it on cameras which are connected via ONVIF is bad. At the least, a warning should be given in release notes (it's not), and at best, the camera should detect if ONVIF connections are in use and not disable it.
Second, even if you do read the release notes, and test the firmware (we did, again), how does that help here? Systems are still going to see several minutes of downtime per camera, at least, while the upgrade runs and ONVIF users are created and applied. And given Hikvision's track record on security, if I were a Hikvision user, I'd apply every firmware update as it comes.
Also, it might be 0.01% of Hikvision cameras are connected via ONVIF, but it's far more than enough to cause substantial issues for dealers, as you can gather from responses here. Avigilon and Network Optix both use ONVIF to integrate Hikvision. That is a non-trivial number of cameras. About a third of cameras connected to Network Optix are Hikvision, by their stats, so this is impacting thousands of channels.
Agreed, it is a disappointing headline.
Im surprised that more VMS'es just dont use a direct Hikvision protocol instead of using Onvif. Biggest surveillance manufacturer in the world for gosh sakes. That would be a big marketing slogan they could use "Hikvision Conformant"
Genetec could use this and their sales would skyrocket.
They could have foreseen the problems such as ONVIF users losing streams and try to come up with something to migrate the settings smoothly, but maybe it would have ruined their release schedule. When updating cameras you expect them to just become better, not require action from you to restore their function to what it used to be a minute ago. Maybe it won't cause so many problems now that IPVM wrote a heads-up about it.
Perhaps they'll fix it in the next release.
Look at the title of this article. Huh? Maybe something like “Hikvision addresses security vulnerability with ONVIF compatability”.
Jeffrey thanks for the extensive feedback! Ethan has already well addressed your other concerns. I'll just focus on the title of the article comment.
It might be that Hikvision did this to improve security but it is also certain that they broke VMS integrations. The latter can be certainly proved, the former cannot.
To your point, though, if Hikvision thinks disabling ONVIF is the only way to solve security problems, they should communicate that. The reason being is that no other major manufacturer hard disables ONVIF. So are all those manufacturers vulnerable or did Hikvision make a mistake in this implementation? I mean that seriously. I am genuinely interested in hearing that answer.
Ultimately, that is the job of Chuck Davis and I am looking forward to his publishing on this matter as he claimed in the press coverage Hikvision issued when he was introduced last month.
Follow up item / discussion: Guy Spends $500 And Hours Fixing Hikvision ONVIF Upgrade Break
Got this notification from NX the other day. Here's their response to Hikvision breaking ONVIF. I would be curious to know how they did this and if other VMS will follow. I underlined and bolded the applicable line below.
3.1.0.17256 Video Management System
PUBLISHED 29 Nov 2017
Release Notes:IMPROVEMENTS
Audio output support for Sony SNC-CX600.
New AXIS devices support: P1367, P1368, F40-Q1765, XF60-Q1765, P40-Q1765, F34, FA54, M5525, Q8742, P1275, Q8741, FA1105, FA4115, P1245, P1265, F1004, M3048, F8804, P3375, F4005, F1025, F1005, Q3517, Q8685, P3374, FA1125, F1015, F1035, Q3504, Q8642, Q8641, XF60-Q2901, XP40-Q1942, XF40-Q2901.
Hikvision 5.5+ ONVIF automatic re-enabling
"Do HTTP Request" action improvements. Now user can specify request and autorization type manually.
The first thing software does - enables ONVIF on the camera.
This is our short-term fix. We call it short term, because we concerned about the security and waiting for more info on this topic.
The document was provided to us by one of the Hikvision officials.
The document has technical errors(such as invalid xml etc), but you can get the idea...
Besides you can reverse engineer such things with the camera, Wireshark, and the browser.
Wow, make an onvif user, that is really inconvenient on a large job...
Read this IPVM report for free.
This article is part of IPVM's 6,728 reports, 907 tests and is only available to members. To get a one-time preview of our work, enter your work email to access the full article.
What is not documented here is that you must also check "Enable Hikvision-CGI" right above the "Enable ONVIF" box.
Additionally, the user you setup for ONVIF must also exist in the Configuration -> Usermanagement list. If it ONLY exists as an ONVIF user, it will not work.