Alternatively stated, is Axis better than Dahua, is Dahua better than Hanwha, is Hanwha better than Hikvision, etc.? And how should the companies be judged when it comes to cybersecurity?
We will be doing more analysis in this area and wanted to open this discussion up to IPVM members (and especially cybersecurity researchers) first to see what they recommend to be used?
I'll add some initial comments inside and look forward to responses and suggestions.
Some areas to assess:
Agree/disagree? Others?
John,
I think the criteria that you suggest is excellent. I would also like to know where in the channel the manufacturer chooses to push out the information concerning vulnerabilities. Does notice stop at the integrator level, or does it go to the end-user of the surveillance device?
I know of many, many cases where there is no longer a relationship between the integrator who originally installed the device and the end-user client. There could be cases where the integrator channel hears about vulnerabilities, but the end-user never gets notified.
Maybe another case where it would be beneficial for a manufacturer to know the final installed location of every one of its products through a registration process. (even though many integrators are likely to oppose this because its "their" customer.)
Does notice stop at the integrator level, or does it go to the end-user of the surveillance device?
Some manufacturers consider that a feature...
It is a good point, though, and it reflects a tension between manufacturers trying to be responsible vs taking a PR hit.
As for registration, I do think one factor should be how easy the manufacturer makes it for people to be notified.
I think that when talking about the "security industry" in general, meaning mostly the manufacturers of devices and the effort they put in their software, the attitude towards security and handling disclosure is roughly at least 15 years behind the curve of pure software companies.
Is there any camera manufacturer or VMS vendor for example that actually has a page like this online with a clear history and details of their flaws across products, and fixes for them: https://www.mozilla.org/en-US/security/known-vulnerabilities/
You could probably pick any big name in software and they have such. They can't afford not to. I think that sincerely disclosing your own dum-dums without blaming everyone else - publicly and responsibly with no BS or marketing taint - speaks volumes for any company. I would place much more trust on a company that doesn't try to make themselves appear like a bunch of robots who never make mistakes, and instead expose the humans within who we can trust to not deceive us and perhaps learn from their mistakes.
When those humans tell us: "sorry, we goofed up and have now fixed this issue, it also affects these other versions and please apply the free patch immediately because we care about you", the fair among us don't judge them for that. We judge them for not telling us, be it PR reasons or whatever.
If Mozilla, for example, were ever to randomly start making VMS software, I think their development processes would be much more mature with security issues than any other company in the VMS business. I very much hope that these things will heal in time, and with some setting examples for others to follow.
#1, that's good feedback and a good example of a template to compare to for security manufacturers.
Another point to note of is features - e.g., does a device enforce strong passwords? (good) does a device include manufacturer controlled security codes to override admin passwords? (bad)
Related, we had this report - 10 Manufacturer Cyber Security Compared
In the end, passwords aren't the ultimate way of securing anything anyway, but if some manufacturer made having an easy to understand, cryptographically impenetrable-for-now system their main asset, it would surely turn heads.
We might trust a traditional ballot box to handle casual tampering, because it seems to us that not just anyone could physically fit their hand through the slot, and any other way would be impractical for achieving anything more than a small disturbance. We need the same confidence about cybersecurity too.
I would personally just create a matrix. List each manufacture and then create a table for the products each manufacture represents.
List firmware updates that the product currently has available from the most current version going back until it's original firmware release and color code them.
Red = Known Security Vulnerability.
Green = Currently in Good Standing.
Then it is just a matter of counting cells and getting percentages.
To create the table, you need the manufacturers to disclose the information or find it out yourself. And it would be a huge table indeed, but the data has to come from somewhere first.
Very true, but it would be pretty useful to just glance at before purchasing a camera. One could also integrate into a "Camera Finder" on their webpage if they were so inclined.
List firmware updates that the product currently has available from the most current version going back until it's original firmware release and color code them....
Then it is just a matter of counting cells and getting percentages.
What if one company issues more firmware updates than another that has nothing to do with vulnerabilities? Does the company issuing more firmware update get an advantage? And how does one factor in the severity of vulnerability using this approach? Is every vulnerability considered equal?
Law of averages. If I release more firmware in the same state as my competition then it should come out to be the same in the end
1/10 = 10/100 = 100/10000
(Doesn't work very well when you have BS Manufacturers that say "all old firmware is compromised", but it will still give you some useful data.)
I would look over time instead of number of firmware releases. How long a vulnerability exists and how long since it has been fixed is more important than the number of firmware versions.
Also, you have to factor in the severity of the vulnerability, it is why, e.g., there exists CVSS scores.
That's a nice calculator there. It might deem the Mirasys vulnerability at 8.8 assuming an unelevated position, 9.6 otherwise.
Very true. I was just thinking of a binary table based on a "safety level" threshold value, but you want something much more in depth.
This is an interesting topic, as someone has already said I think the security industry is a long way behind pure software companies.
It's only in the last few years that a serious number of CCTV devices have been connected to the internet (at least in the UK) and up until quite recently they where from a large number of different manufactures. As the quantity of units connected increases and the number of manufactures is reduced then they become targets just like companies in the software industry. And to this end I have some sympathy with the manufactures of all shapes and size.
However I think the response from the whole industry in general has been rather poor. I include manufactures, distributors and installers in this statement. We could and should have done more, from making sure vulnerabilities are not there in the first place, to informing installers about updates available for known issues, to installers taking basic precautions like changing default passwords.
It would seem we (the industry) have found it difficult to respond and in some cases even do basic things when notified of a problem. That's not to say we have been deliberately secretive, but rather we have not known how to respond, it's like the cat caught in the headlights of a car at night.
The software industry has had decades to formulate policies to deal with cyber security, for most security manufactures, distributors and installers this is the first time we have had to think about it seriously, and I think it's fare to say it's taken some of us by surprise.
Now I am not making excuses, but there are many lessons we must all learn from the recent cyber security issues and some of these will take time to implement, we have a lot of catching up to do.
But we must also be under no illusion that these cyber attacks will continue and if anything get worse. As more equipment is fitted and connected to the internet the target becomes bigger.
There are a lot of questions we need to ask as an industry and then consider the answers carefully before we can truly be in a position to say we are doing all we can.
Question. Once a vulnerability is identified who is notified, installer or end user.
Question. If the end user is notified are they given access to update without an installer.
Question. If an installer is to take responsibility for maintaining the software who will pay for the additional cost, end user or installer.
Question. Is adding auto update function (like your phone) a good idea, and who gets to make the update decision, or is it updated automatically.
Question. If the update is automatic what happens if the unit won't reboot of is slow to reboot and an incident is missed.
These questions raise many more questions and there is no simple answer, don't forget we are not talking about DIY systems here, some of these are security systems fitted in highly sensitive areas.
But I think over the coming months we will see some big changes in the way the industry looks at cyber security and we will all have to take it a lot more seriously.
You have insightful points, there has for sure been a rapid transition from cabling to protecting against hackers, and there are not only new technical challenges, but also many related to management and communicating with the public.
Question. Once a vulnerability is identified who is notified, installer or end user.
It is a tough question indeed and there is no general answer. Some of it is dictated by business, some by ethics, some by just common sense. There are some serious vulnerabilities that must be disclosed with haste (imagine pacemaker, car...), but before that, there must be a fix for the issue or they must be prepared to recall their devices for public safety.
If the end user doesn't know or care, the installer should. If the installer doesn't know or care, the manufacturer should. If the manufacturer doesn't know or care, someone else should.
Question. If the end user is notified are they given access to update without an installer.
I think that anyone who owns the device is entitled for a security update. Problem with some companies is that they think all of their users are idiots and couldn't handle a firmware update without hiring someone, and if their 10-year old mailing list with hardly a few valid addresses is their only way of disclosing serious issues, things are not well. Even if just a few needed or wanted the update/warning in the end, it's critical information.
Question. If an installer is to take responsibility for maintaining the software who will pay for the additional cost, end user or installer.
I don't think an installer could take any responsibility for maintaining the software of some manufacturer whose devices they configure for money - their job is to install a particular system the best they can for the customer who needs one. Their job is to also know what the customer needs and can do with. If there is a huge problem with a Brand™ device that comes up in the news a week after some company just happened to install them for their customer, yeah, the customer might be pissed for a while but the installer can just blame the Brand™ unless it was way too obvious that they're crap. If the Brand™ can't manage to fix those issues, maybe they shouldn't sell their systems any more.
Question. Is adding auto update function (like your phone) a good idea, and who gets to make the update decision, or is it updated automatically.
I think it's not a bad idea to add an auto-update feature (which wouldn't work anyway in a restricted network without extra effort), and it should always be up to the end user to decide how the system operates. I'm not buying any security system I can't configure myself when the people who did it before are gone. Just don't dumb down things for the enterprise user and give enough options, defaulting with safe ones, and everyone is happy.
Question. If the update is automatic what happens if the unit won't reboot of is slow to reboot and an incident is missed.
Sure, there are some cases where power is cut at the right moment, or surges and hardware fails, and everyone blames each other for a while. Perhaps there is some other general problem here, related to customer communication. If things work as they should, there should be minimal complaints, and it is not impossible to achieve even in things as complicated as camera surveillance.
What I would like to see is an end user application that resides on the network that will monitor the health and firmware status of all security components: cameras, NVRs, ACAMS servers, even the workstations. This application could keep track of the firmware versions and what the manufacturer’s current versions are. It would also notify the user of new updates. The VMS folks would be in a good position to add this app to their software.
It is not reasonable to expect integrators or dealers to manage the customer security for all, or probably even most end users. A security app would be the best way to take the middlemen out of the loop, and let the end user manage the security of their devices.
All good stuff, but it doesn't answer the central question "What if somthing goes wrong with an update"
Is the end user to take responsability if they upgrade there unit and for some reason it goes wrong.
Will the installer accept site visit to rectify this.
Will the distributor or manufacture be expected to cover this with warranty claim or compensation.
Ask your self how often an update from microsoft or other software causes a problem and you need to reboot or reconfigure somthing to fix it. This is relatively easy with a PC, but a security system is a little different.
Not excuses but the software industry has had decaids to get this right and it's still not fool proof.
I probably misinterpreted some things in my previous response, sorry.
In the future we will be concerned with other things, but for now I think there should be awareness that these systems are mostly broken beyond belief, and installers educated and informed about that.
In the case of installers, they can offer the customer an optional maintenance contract. If they're not paid to update the system, I'm not expecting them to fix a system the user messed up, but if there was a reason to alert their user because of a widespread issue for example, they should at least try to do that within reasonable limits. Some responsibility like that should be mandated by law for companies operating in the sector, and simple procedures provided so it's not hard to do.
If a manufacturer has disregarded repeated proof of problems and need their head banged on the desk for them to understand that they are at fault, I have no sympathy. They deserve a big lawsuit for all those issues. In the end, with proprietary software it is the responsibility of the party owning the code to disclose and admit any problems with it.
It is surely a sensitive business. If I try to imagine I was manufacturing parachutes and someone dies because I never actually jumped out of a plane myself and didn't know what I was doing when I designed it, I would probably feel bad. In software there are also very serious, even classical examples of software faults causing casualties, and there will never be fool proof software as we know it.
With cameras there's also the problem of costly maintenance if something goes wrong, but if we can reduce the possibility of something going wrong with proper software development procedures, testing and being open about any issues while thinking 'like a criminal' to foresee problems, it might not take long until we actually feel nostalgic about climbing ladders to replace a bricked camera.
It's still not the same as pure software business, but many common things we pass by in the physical world every day used to be really hazardous and suboptimal a long time ago, but have developed since, and we don't think about them anymore.
UD #3,
Agreed, there are big risks with doing firmware updates. I want a tool to advise when an update is needed, that is not dependent on having a relationship with a company that may not be responsive to the user’s needs. The application is just a tool in the tool box.
Most users should not be installing updates, they should be contracting it out to qualified companies, either through a service contract, or an a per visit basis.
The manufacturer could actually issue a proper description of an update so that whoever may apply it can consider beforehand if it's required. If it's non-critical for the environment, no need to bother, but key thing is knowing what issues the update addresses. Often it's just "bugs fixed", while hoping no one ever asks what the "bug" was.
I'm not sure if I agree with "most users should not be installing updates" as the "user" is vague. These systems scale from a single camera to many tens of thousands and "most users" may not be the "most important users". All of them should know if there is a problem with their security system, even if they didn't have a contract. If they can't handle it, maybe they'll hire someone after they hear about it.
I think IPVM should create a 5 point system for rating security manufacturers on cyber security -something along these lines:
Level 1
The manufacturer has a cyber security person on staff who is "public facing"
Level 2
The manufacturer provides current high quality information on securing their systems (no port forwarding please)
Level 3
The manufacturer has a commitment or history of publicly disclosing cyber security faults in their products
Level 4
The manufacturer has a means of notifying their clients of a cyber security issue (not just posting on their web site)
Level 5
The manufacturer has a commitment to, or history of, promptly patching cyber security issues.
Ask questions and get answers to your physical security questions from IPVM team members and fellow subscribers.
