Axis, Pay Integrators To Fix Your Critical Defect

IPVM

Isn't Axis already living up to their "self proclaimed standards"? It seems like Axis has far fewer scramble-the-techs vulnerabilities and critical upgrades than the majority of their competitors. Unless we expect Axis to produce a flawless product I would say that they are earning their price as it relates to this.

Presumably if you are an Axis fan you are also following their cyber security guide (Axis Cybersecurity Hardening Guide Examined), which recommends setting IP address filters once you get beyond "demo" or "small office" deployments.

If you have properly filtered/firewalled your cameras the probability of exploit drops dramatically. Theoretically customers are only liable to an internally-executed exploit, or a compromised VMS server being used to launch an attack against the camera. Both of these are scenarios that neither Axis, or the integrator, should really be seen as liable for.

While the ideal is non-exploitable firmware always, I do not think that is a realistic expectation. Customers and integrators need to understand that devices may need to have an emergency update done on one or more occasions over the life of the install, and set pricing for handling those issues accordingly.

Now, if this was the 3rd time Axis had a vulnerability like this in a year, I might agree that they owe integrators some recourse to justify their premium.

Semi-related, incidents like this are why integrators may want to offer service/emergency service programs to their customers. Perhaps the really savvy integrators can use Axis' weakness here as a new revenue stream for themselves.

Reactions:

(12)

(1)

John Honovich

IPVM

In reply to Brian Karas

Isn't Axis already living up to their "self proclaimed standards"?

It depends what we see those standards to be and how it compares to their competitors.

It seems like Axis has far fewer scramble-the-techs vulnerabilities and critical upgrades than the majority of their competitors.

I do not think Axis is worse but it also does not strike me that the average competitor is all that more problematic (excluding Arecont).

Take Axis vs Hikvision. Axis charges 50 - 100% more than the equivalent tech specs of Hikvision. Axis claims to differentiate on more subtle things like 'quality'.

To me, that's like Wal-Mart vs Nieman Marcus / Nordstrom. You go to Wal-Mart, Wal-Mart makes an error, you figure you eat it because it's Wal-Mart, everyday low prices, barebones service, etc. But Axis prices itself like Nieman Marcus / Nordstrom and the expectation at a place like that is that extra payment you make goes to white glove service and going the extra mile to solve mistakes.

I do not think Axis has to do this but if they want their buyers to remain confident why they are paying a premium for Axis, Axis should see paying integrators here as an investment in justifying the continued premium for Axis products.

Reactions:

(2)

(1)

Brian Karas

IPVM

I think it is worth keeping in mind here that this issue relates to the fact that while Axis may be really good at making cameras, they are less good at making firewalls, in a sense.

This vulnerability is something that appears to be somewhat difficult to stumble upon (or else Axis would have found it in earlier testing, or it would have been found/exploited much sooner).

If this was something that caused the camera to stop being a camera, like a bug that caused the iris to close any time the checksum of the date was a prime number, I might feel that Axis did not deliver on their position of building a superior product.

But this is the kind of vulnerability that almost every network device maker has encountered at one time or another, and it is why it is not recommended to have any device like this directly connected to the wide-open internet. This why firewalls exist, if devices could all be relied upon to manage their own security (and be easily manageable) we would not need firewalls (IPv4 address space and NAT simplicity aside).

While every Axis customer should upgrade their firmware to remove this vulnerability, the ones that are in a critical position are those that also seem to be lacking good network security in general. For this specific case I do not think Axis is really at fault here where they should be paying integrators. They encountered an exploit, announced it, fixed it, and have also previously given recommendations in their cyber security guide that help to minimize the impact of situations like this.

Reactions:

(3)

Ari Erenthal

Chesapeake & Midlantic

Do you want manufacturers to keep vulnerabilities secret? Because that's how you get manufacturers to keep vulnerabilities secret.

Reactions:

(8)

(3)

(1)

John Honovich

IPVM

In reply to Ari Erenthal

Axis has no choice to keep this 'secret'. The researcher is releasing this in 4 days after following responsible disclosure.

Reactions:

(2)

Ari Erenthal

Chesapeake & Midlantic

They can't keep this one secret, but they can keep the next one secret.

If you make it expensive for a manufacturer to admit fault and release a patch, they'll only do it when forced to.

Axis releases what, dozens of firmware updates yearly? Why bother once they've set the precedent that they'll pay you to install it?

Reactions:

(4)

John Honovich

IPVM

In reply to Ari Erenthal

Axis releases what, dozens of firmware updates yearly? Why bother once they've set the precedent that they'll pay you to install it?

Because the typical firmware release is incremental tech advances (e.g., slightly better WDR) or minor bug fixes (now works with browser x for niche feature y).

This firmware release is to stop attackers from gaining root access and taking control of Axis cameras. The severity of this is far beyond any typical firmware upgrade.

An extraordinary problem merits an extraordinary solution.

Reactions:

(1)

John Bazyk

Command Corporation • IPVMU Certified

If Axis made a statement and said, "We guarantee zero vulnerabilities." Then yes, they should pay integrators to fix the issue, because "zero vulnerabilities" was guaranteed. If they didn't make a statement similar to this, then they shouldn't have to or be expected to by their partners. As businesses, we assume certain risks, while we do our best to mitigate risk by installing quality products and hiring good people. Sometimes things happen and we have to deal with it.

Agree with it or not, every Hikvision dealer has more fuel to combat Axis. If Axis were smart, they would take council from Tylenol and get ahead of this.

Speaking of Nordstrom, I bought a $1,200 suit there a couple of years ago. Lost 30 lbs and the suit didn't fit me anymore. I went back and was shopping for another suit. I told the sales associated what happened, and he got the manager, and the manager asked me to bring the suit back. They credited my credit card and took the suit back (it was well worn for a year). Since then I have spent lots of $$$ at Nordstrom.

Reactions:

(9)

(3)

John Honovich

IPVM

In reply to John Bazyk

Agree with it or not, every Hikvision dealer has more fuel to combat Axis.

I agree. That's why I think paying integrators here would reduce that fuel and make integrators feel more confident and likely to spend more with Axis, like you did with Nordstrom.

Reactions:

(2)

Richard Friend

Any estimates on the financial impact of even offering $200/site to integrators globally? Would it materially impact the Axis numbers in 2016?

Interesting and controversial suggestion. Seems like it could open a Pandora's Box of issues with some integrators servicing smaller customers and happy with the $200 and others with sites in the 100's or 1000's of cameras then wanting more compensation. Or finding a way to split the site into smaller sites... And many of the sites could be covered under service agreements which may already include application of f/w upgrades.

Reactions:

John Honovich

In reply to Richard Friend

IPVM

Any estimates on the financial impact of even offering $200/site to integrators globally?

Let's say there are 100,000 sites with Axis cameras (just to use a rough number). At $200 per site, that's $20 million total, compared to Axis ~$800 million revenue. So a hit but survivable. Also, you would suspect some will not upgrade, some will ignore getting payment, not now, not want to share end user details with Axis, etc.

Seems like it could open a Pandora's Box of issues with some integrators servicing smaller customers and happy with the $200 and others with sites in the 100's or 1000's of cameras then wanting more compensation.

I agree. That's why I framed it as an average payment. If you have to emergency upgrade all of Target or City of Chicago, etc. it would be a lot more money.

And many of the sites could be covered under service agreements which may already include application of f/w upgrades.

Even if the integrator has a service agreement, it still costs them money to do the upgrade, which would be unnecessary if this defect did not exist.

Reactions:

(1)

Richard Friend

The comparison shouldn't be versus revenue though but against profit: this would come straight off their bottom line and $20M profit is a much bigger hit. Also survivable but pretty painful.

Reactions:

(1)

John Honovich

In reply to Richard Friend

IPVM

Axis 2015 operating profit was ~$100 million USD, so a fraction of their annual profit.

Think about it this way, though, if Axis steps up and makes their partners whole, how much revenue and profits will they save going forward? How many Axis partners are on the verge of leaving Axis for lower cost alternatives and will see this as one more reason to switch?

Reactions:

(1)

Richard Friend

If - and that's a big assumption - the cost impact was in the region of $20M then that's a pretty material impact on $100M profit, for what's basically a goodwill gesture. Maybe it's more or less but the estimated fraction of profit would be an important part of the decision.

I understand your perspective and the points supporting the argument, I just don't see it being practically possible for them to implement. I'd also be surprised if anyone thinking of defecting to a lower cost alternative were swayed by a token payment of roughly $200 per site: they could be saving that (or more) per camera already. Maybe it would hold off the move temporarily.

Reactions:

John Honovich

In reply to Richard Friend

IPVM

I just don't see it being practically possible for them to implement.

I agree with you about that. I highly highly doubt that they will implement a flat policy like that.

My ultimate goal is to spur awareness / discussion / 'thought leadership' that encourages integrators to ask Axis for some form of credit / comp and motivates Axis to do something.

Or they do not, and Hikvision can use this defect as another entry point.

Reactions:

(1)

Eddie Perry

I am torn on the whole "where does the responsibility start and end" thing only because it vary between manufacturers and it is in part also the owner/maintainers responsibility to take care of the maintenance of the camera.

Other electronic devices like phones, TV's, computers and Photography cameras in order to honor warranties require you to register your products with their respective manufacturer. When you do this they also send you information regarding your products as well as recalls. After reading the AXIS website warranty information all you need is a receipt and the serial number for the camera. with the way cyber security is going this may need to change as most other manufacturers follow this same model.

Best solution is to have a secure method to register these products so the manufacturer can send you emails saying hey you need to upgrade you firmware and here is how. This puts all of the liability on the owner and not the manufacturer. This could lead to some bad relations with integrators though as it cuts them out of service calls but IMHO that will go away sooner or later anyway with the way things are progressing.

So no I dont think it is AXIS's job to chase down people and let them know that the camera is security vulnerability, But I will say if what AXIS says:

"Axis is committed to providing high quality products, trouble-free ownership and better control of total costs by offering exceptional global service covering questions and trouble-shooting concerning both the installation and use of the products"

Then they need to come up with a better way to do what they are claiming

Reactions:

(2)

(1)

John Honovich

IPVM

In reply to Eddie Perry

Eddie, good feedback.

For members, that quote is from their warranty page:

Reactions:

Undisclosed Integrator #1

Do Jeep owners pay for their cars to be fixed?

Reactions:

(1)

John Honovich

IPVM

If the car has a defect and there is a recall, absolutely.

This is the camera equivalent of a car defect / recall.

Reactions:

(1)

(3)

Brian Rhodes

IPVMU Certified

The auto repair analogy isn't cogent here without qualification.

If Jeep owners break their Jeep, they fix it.

If Jeep designs a part that is faulty and/or could result in damages, injury, or death they pay for the repair under a vehicle recall.

In the same way, a critical security vulnerability is not normal 'wear and tear', and I don't think the owner is expecting to pay to fix it.

Reactions:

(1)

Brian Karas

IPVM

If a vehicle has a defect/recall, the manufacturer fixes it under warranty without cost. Similarly, Axis has released this firmware update for free.

Jeep, does not pay the owner for their time to come into the dealership and wait for the car to be serviced. Most of them don't even have convenient hours, requiring the owner to potentially take time off of work to deal with the defect.

If you are lucky, you might get a free loaner car, or maybe the dealership will drop you off and pick you up when the work is complete, but you still incur costs in the form of time lost and inconvenience.

A typical vehicle also costs more than all the cameras in an "average" smaller system.

Reactions:

(6)

(1)

Brian Rhodes

IPVMU Certified

In reply to Brian Karas

The car company pays for the part to be installed.

They don't just toss out a box of redesigned good parts and hang a 'free' sign over it.

Reactions:

(1)

Richard Friend

In reply to Brian Rhodes

If you carried through with the analogy though, would that not mean Axis offering to install the firmware free of charge, provided that you brought your affected cameras in to your closest Axis location?

Reactions:

(1)

Luis Carmona

Geutebruck USA • IPVMU Certified

In reply to Brian Karas

"Jeep, does not pay the owner for their time to come into the dealership and wait for the car to be serviced. Most of them don't even have convenient hours, requiring the owner to potentially take time off of work to deal with the defect."

That is a great point I was actually thinking of posting before Brian R. and Brian K. (B&B) chimed in on it.

Maybe "higher ned" manufacturers should consider a firmware option to automatically download and install critical updates, either from Axis' website or from an optional location the installer sets up. That way Internet facing cameras, which are more at risk than network closed cameras can get updates easier and quicker.

Reactions:

John Honovich

IPVM

In reply to Luis Carmona

In the context of the jeep example, the integrators are the equivalent of dealers, not users. Do car dealers have to do the recall work for free?

This source says typically no with some debate:

The basic pattern that has emerged is for dealers to fix recalled autos and automakers to pay dealers for the service.

Reactions:

(1)

John Scanlan

IPVM • IPVMU Certified

Jeep put a bounty on vulnerabilities within their own software.

Reactions:

(1)

Luis Carmona

Hackers exploit the lucrative software vulnerability trade

Geutebruck USA • IPVMU Certified

In reply to John Scanlan

That they're only paying up to $1,500 shows how cheap and not-serious Jeep is.

This 2013 article reports bounties of 5 to 6 figures being paid for finding exploits, and I have read that in other sources, too.

Reactions:

Eddie Perry

I think the best to solve this is to go to the "registering your product model"

If you have a camera or NVR and it needs a firmware upgrade then you get an email saying "hey you need to upgrade your firmware or we are not liable for whatever may happen.

also this would cover the warranty as well

1)so in short you buy a camera or NVR

2) there is a slip of paper in the manual or packet that says " if you want warranty or to avoid security vulnerabilities you need to register you products and upgrade according.

3) customer either does #2 and gets the upgrades and is put on an email list for upgrades that may come out or doesn't and assumes responsibility for what ever may happen.

4)Manufacturers get Brownie and advertising points saying they give a damn about the end user/owner, and they are doing it better than all the other Manufacturers.

this cleans up a lot of gray area's in the who is responsible for what questions. Manufacturer says we will take care or owner as long as you cooperate with rules. Owner gets great service from manufacturer and feels safer..... hopefully.

Reactions:

Undisclosed #2

IPVMU Certified

Should Axis pay TWICE for any integrators who have already downloaded the latest firmware, erroneously thinking this would solve their problem?

PRO TIP: When you actually are able to secure your camera, MAKE SURE TO RESET TO FACTORY DEFAULTS before upgrading the firmware. Although not mentioned by Axis, once the vulnerability is disclosed and script-kiddies probe every known Axis camera on the web, there may be nothing stopping them from making the firmware webpage non-functional except to change what is SAYS is the version.

Resetting to factory defaults, using the hard button, gives a better chance of eradicating any installed malware first. Though it should be noted that once the camera has been rooted, even the recovery partition can be corrupted, still that's unlikely to be exploited in the early days.

Reactions:

(1)

Undisclosed #2

IPVMU Certified

Jeep's aside, does anyone know of any software company, in any industry, ever paying for someone's time and labor of installing a patch?

If not, why do we think Axis will be the first?

Reactions:

(3)

Undisclosed Integrator #3

In reply to Undisclosed #2

I have backcharged camera manufacturers for our time to remedy their hardware defects quite successfully. Perhaps it is simply because we represent several large accounts for them, but this particular manufacturer has covered our labor costs on many occasions. It's not Axis.

Reactions:

(1)

Undisclosed #2

In reply to Undisclosed Integrator #3

IPVMU Certified

Was the remedy a software patch?

Reactions:

Undisclosed Integrator #3

In reply to Undisclosed #2

In one instance, yes. However, it was a firmware update that required significant work on our part. Not something as easy as clicking to update in Axis Camera Management.

Reactions:

(1)

Undisclosed Manufacturer #4

How does it differ from other software updates? Take Windows, for example. Granted, many/most are "functional" updates/patches, although many are security patches/updates to block/fix vulnerabilities.

While many a consumer can set their machine to update Windows automatically, many Enterprises do not allow such action, and have Network Admins who spend tons of time on installing/administering patches/updates.

Should Microsoft be reimbursing companies for Network Admin time spent applying/adminstering their constant patching? How does it differ from this situation and Axis? (and maybe others......)

Reactions:

(3)

John Honovich

In reply to Undisclosed Manufacturer #4

IPVM

Should Microsoft be reimbursing companies for Network Admin time spent applying/adminstering their constant patching?

But as you say before that:

Granted, many/most are "functional" updates/patches, although many are security patches/updates to block/fix vulnerabilities.

The difference here, compared to 99% of Windows updates and pretty much all prior Axis upgrades is that this is for defect that allows taking over control of the device.

The other issue is structural. Most PCs are managed by techs in-house. In IP video surveillance, it is the opposite. If upgrading all these defected Axis cameras was simply a matter of Axis pushing a button or the integrator pushing a button from their office, than the cost would be trivial for all parties. Unfortunately, that is not the case. Most of these are going to require truck rolls to deal with the typical IP camera not being remotely accessible to the typical integrator.

Reactions:

Undisclosed #2

IPVMU Certified

Most of these are going to require truck rolls to deal with the typical IP camera not being remotely accessible to the typical integrator.

If the camera is not remotely accessible then the patch is not urgent.

Reactions:

(4)

(2)

Undisclosed Integrator #1

Can be seen as another huge plus for using HD Coax to some extent......

Reactions:

John Honovich

IPVM

With 100 votes received, integrators and manufacturers (not surprisingly) have taken opposing views here.

While 71% of integrators vote yes, only 41% of manufacturers agree.

Reactions:

John Richardson

•Jul 16, 2016

I am not a fan of these sensationalist articles.

All complex software has flaws, with the right research effort holes could be picked in almost any product. If it is a bug in the core of an open source product the exact same bug may appear in other brand's firmware, would you expect the same from every manufacturer?

I think that AXIS turning around a patched firmware for practically every model with a matter of days shows their commitment to holding themselves at a very high standard.

Reactions:

(3)

Alberto Alonso

Let's stick to the announced vulnerability. It will only be applicable to cameras (video devices) directly accessible through the net (internet). The vulnerability will be removed by just upgrading those cameras to the latest firmaware version which is downloadable for free from the axis site. Threfore the job to do is to connect with those cameras (remotely? yes, if they are not accessible there's no risk) and upgrade the firmware. So costs will only be time spending in the process rather than moving to "far away" sites.

Axis has a free tool to do that in a semi automatic way (Axis Camera Management) which can be programmed to upgrade hundreds (thousands) of units in sequence at desired times. That would reduce time spent in the process dramatically.

On the other hand, it's suppossed that the integrator will take care of maintenance (that includes firmware upgrades). Yes, this is not an improvement upgrade for new features, but an upgrade anyway.

Does anybody pay for the upgrade of those cameras which were supplied with default password in the past until most manufacturers found out it was risky and changed the firmware?

Reactions:

John Honovich

In reply to Alberto Alonso

IPVM

It will only be applicable to cameras (video devices) directly accessible through the net (internet).

Just because the integrator does not have remote access to the cameras does not mean there is no remote access. Secondly, a prudent manager is not going to ignore doing the upgrade on such a critical vulnerability just because the cameras are behind a firewall. Mistakes can happen and also someone internally could take advantage of this as well.

Reactions:

Undisclosed Integrator #5

I voted yes, but that's just wishing. I wound't expect that to happen. Its like taking a medicine and having it not work and going to the doctor and asking for a a free replacement.

Reactions:

Damon Hood

Not really sure if Axis should pay integrators to do this. I have been in the industry over 20 years and I have never seen an Integrator roll trucks to do software or firmware upgrades to products.

As far as this vulnerability we upgraded over 1600 Axis cameras in the past week. We did this with out an integrator. But we always do our own firmware upgrades.

Reactions:

(2)

John Honovich

IPVM

In reply to Damon Hood

I have been in the industry over 20 years and I have never seen an Integrator roll trucks to do software or firmware upgrades to products.

Do you mean solely to do a software or firmware upgrade? I assume so since it is routine part of support and or maintenance for integrators to come up on site and do upgrades.

If you mean solely, for sure, that is rare. On the other hand, it is also rare to have a critical security vulnerability of this level.

Reactions:

(1)

Undisclosed #2

IPVMU Certified

It's the 18th, did Axis ask the researcher for more time?

Reactions:

Undisclosed End User #6

In reply to Undisclosed #2

http://seclists.org/bugtraq/2016/Jul/71

Reactions:

(1)

Undisclosed #2