Axis, Pay Integrators To Fix Your Critical Defect

Author: John Honovich, Published on Jul 14, 2016

Axis should pay integrators to fix Axis' recent disclosed critical security vulnerability / defect

There are millions of Axis cameras deployed with this defect. The risk is high as the soon to be published disclosure (July 18th) will allow attackers to take over the devices.

It falls on integrators to upgrade Axis cameras and eliminate this risk camera by camera.

Why Axis?

Axis should live up to their self proclaimed standards. Axis markets their high 'quality'. Integrators and customers pay a premium for that quality.

**** ****** *** *********** ** *******' ****** ********* ******** ******** ************* / ******

***** *** ******** ** **** ******* ******** **** **** ******. The **** ** **** ** *** **** ** ** ********* ********** (July ****) **** ***** ********* ** **** **** *** *******.

** ***** ** *********** ** ******* **** ******* *** ********* this **** ****** ** ******.

Why ****?

**** ****** **** ** ** ***** **** ********** *********. **** markets ***** **** '*******'. *********** *** ********* *** * ******* *** **** *******.

[***************]

******, ************ ********** '*********** ****** *******':

** ********, **** ***, ** ***** *** *****, * '********' vulnerability', * *********** ****** ** ***** *******. ** **** ***** at **** *** ***** ******* ** ** ***** ****, *** broader ******** ** **** *** ***** ******* ** ** **** to ****** ******* ******* *** *** ********** ** **** *********** and ********* *** ***** ****** ** ****.

**** ****** **** ** ** *** ********* *** ******* ******** by ************ *********** *** ***** *** **** ** *********** **** vulnerability ** ******* ****** *** *****.

Integrator *****

*** ***** *** *********** *** ** ***********:

  • ****, ** *** ****, ********* **** ******* **-**** ******, ********* customers *** ***** **** *** ** *** ****.
  • *** ********** *** **** ** ******* *** ************* ** *** ******** which **** ********* **** ** ***** ********* *****.
  • **** ****** **** ******* ******* ***** ********, **** ********* **** *** want ******** ******* ***** ******* ** **** **** *** * *** minutes *** *** ******* ********** ***** **** ***** ** ********* up ********. 
  • ***** *********, *** ********** **** **** ** ****** **** *** cameras ******** ** ****, ***** **** **** **** **** *** if *** ******* **** ** ********* ***** ** * *********** problem.

* ********** ******* ******* **** **** ** *********** ***** ** $200 *** ****. ** ******, ***** **** ** ********** ** time *** ****** *** * **** ***** *** **** ** hour ** **** ***** **** ** **** *****.

Vote: ****** **** ***?

 

 

Comments (47)

Isn't Axis already living up to their "self proclaimed standards"? It seems like Axis has far fewer scramble-the-techs vulnerabilities and critical upgrades than the majority of their competitors. Unless we expect Axis to produce a flawless product I would say that they are earning their price as it relates to this.

Presumably if you are an Axis fan you are also following their cyber security guide (Axis Cybersecurity Hardening Guide Examined), which recommends setting IP address filters once you get beyond "demo" or "small office" deployments.

If you have properly filtered/firewalled your cameras the probability of exploit drops dramatically. Theoretically customers are only liable to an internally-executed exploit, or a compromised VMS server being used to launch an attack against the camera. Both of these are scenarios that neither Axis, or the integrator, should really be seen as liable for.

While the ideal is non-exploitable firmware always, I do not think that is a realistic expectation. Customers and integrators need to understand that devices may need to have an emergency update done on one or more occasions over the life of the install, and set pricing for handling those issues accordingly.

Now, if this was the 3rd time Axis had a vulnerability like this in a year, I might agree that they owe integrators some recourse to justify their premium.

Semi-related, incidents like this are why integrators may want to offer service/emergency service programs to their customers. Perhaps the really savvy integrators can use Axis' weakness here as a new revenue stream for themselves.

Isn't Axis already living up to their "self proclaimed standards"?

It depends what we see those standards to be and how it compares to their competitors.

It seems like Axis has far fewer scramble-the-techs vulnerabilities and critical upgrades than the majority of their competitors.

I do not think Axis is worse but it also does not strike me that the average competitor is all that more problematic (excluding Arecont).

Take Axis vs Hikvision. Axis charges 50 - 100% more than the equivalent tech specs of Hikvision. Axis claims to differentiate on more subtle things like 'quality'.

To me, that's like Wal-Mart vs Nieman Marcus / Nordstrom. You go to Wal-Mart, Wal-Mart makes an error, you figure you eat it because it's Wal-Mart, everyday low prices, barebones service, etc. But Axis prices itself like Nieman Marcus / Nordstrom and the expectation at a place like that is that extra payment you make goes to white glove service and going the extra mile to solve mistakes.

I do not think Axis has to do this but if they want their buyers to remain confident why they are paying a premium for Axis, Axis should see paying integrators here as an investment in justifying the continued premium for Axis products.

I think it is worth keeping in mind here that this issue relates to the fact that while Axis may be really good at making cameras, they are less good at making firewalls, in a sense.

This vulnerability is something that appears to be somewhat difficult to stumble upon (or else Axis would have found it in earlier testing, or it would have been found/exploited much sooner).

If this was something that caused the camera to stop being a camera, like a bug that caused the iris to close any time the checksum of the date was a prime number, I might feel that Axis did not deliver on their position of building a superior product.

But this is the kind of vulnerability that almost every network device maker has encountered at one time or another, and it is why it is not recommended to have any device like this directly connected to the wide-open internet. This why firewalls exist, if devices could all be relied upon to manage their own security (and be easily manageable) we would not need firewalls (IPv4 address space and NAT simplicity aside).

While every Axis customer should upgrade their firmware to remove this vulnerability, the ones that are in a critical position are those that also seem to be lacking good network security in general. For this specific case I do not think Axis is really at fault here where they should be paying integrators. They encountered an exploit, announced it, fixed it, and have also previously given recommendations in their cyber security guide that help to minimize the impact of situations like this.

Do you want manufacturers to keep vulnerabilities secret? Because that's how you get manufacturers to keep vulnerabilities secret.

Axis has no choice to keep this 'secret'. The researcher is releasing this in 4 days after following responsible disclosure.

They can't keep this one secret, but they can keep the next one secret.

If you make it expensive for a manufacturer to admit fault and release a patch, they'll only do it when forced to.

Axis releases what, dozens of firmware updates yearly? Why bother once they've set the precedent that they'll pay you to install it?

Axis releases what, dozens of firmware updates yearly? Why bother once they've set the precedent that they'll pay you to install it?

Because the typical firmware release is incremental tech advances (e.g., slightly better WDR) or minor bug fixes (now works with browser x for niche feature y).

This firmware release is to stop attackers from gaining root access and taking control of Axis cameras. The severity of this is far beyond any typical firmware upgrade.

An extraordinary problem merits an extraordinary solution.

If Axis made a statement and said, "We guarantee zero vulnerabilities." Then yes, they should pay integrators to fix the issue, because "zero vulnerabilities" was guaranteed. If they didn't make a statement similar to this, then they shouldn't have to or be expected to by their partners. As businesses, we assume certain risks, while we do our best to mitigate risk by installing quality products and hiring good people. Sometimes things happen and we have to deal with it.

Agree with it or not, every Hikvision dealer has more fuel to combat Axis. If Axis were smart, they would take council from Tylenol and get ahead of this.

Speaking of Nordstrom, I bought a $1,200 suit there a couple of years ago. Lost 30 lbs and the suit didn't fit me anymore. I went back and was shopping for another suit. I told the sales associated what happened, and he got the manager, and the manager asked me to bring the suit back. They credited my credit card and took the suit back (it was well worn for a year). Since then I have spent lots of $$$ at Nordstrom.

Agree with it or not, every Hikvision dealer has more fuel to combat Axis.

I agree. That's why I think paying integrators here would reduce that fuel and make integrators feel more confident and likely to spend more with Axis, like you did with Nordstrom.

Any estimates on the financial impact of even offering $200/site to integrators globally? Would it materially impact the Axis numbers in 2016?

Interesting and controversial suggestion. Seems like it could open a Pandora's Box of issues with some integrators servicing smaller customers and happy with the $200 and others with sites in the 100's or 1000's of cameras then wanting more compensation. Or finding a way to split the site into smaller sites... And many of the sites could be covered under service agreements which may already include application of f/w upgrades.

Any estimates on the financial impact of even offering $200/site to integrators globally?

Let's say there are 100,000 sites with Axis cameras (just to use a rough number). At $200 per site, that's $20 million total, compared to Axis ~$800 million revenue. So a hit but survivable. Also, you would suspect some will not upgrade, some will ignore getting payment, not now, not want to share end user details with Axis, etc.

Seems like it could open a Pandora's Box of issues with some integrators servicing smaller customers and happy with the $200 and others with sites in the 100's or 1000's of cameras then wanting more compensation.

I agree. That's why I framed it as an average payment. If you have to emergency upgrade all of Target or City of Chicago, etc. it would be a lot more money.

And many of the sites could be covered under service agreements which may already include application of f/w upgrades.

Even if the integrator has a service agreement, it still costs them money to do the upgrade, which would be unnecessary if this defect did not exist.

The comparison shouldn't be versus revenue though but against profit: this would come straight off their bottom line and $20M profit is a much bigger hit. Also survivable but pretty painful.

Axis 2015 operating profit was ~$100 million USD, so a fraction of their annual profit.

Think about it this way, though, if Axis steps up and makes their partners whole, how much revenue and profits will they save going forward? How many Axis partners are on the verge of leaving Axis for lower cost alternatives and will see this as one more reason to switch?

If - and that's a big assumption - the cost impact was in the region of $20M then that's a pretty material impact on $100M profit, for what's basically a goodwill gesture. Maybe it's more or less but the estimated fraction of profit would be an important part of the decision.

I understand your perspective and the points supporting the argument, I just don't see it being practically possible for them to implement. I'd also be surprised if anyone thinking of defecting to a lower cost alternative were swayed by a token payment of roughly $200 per site: they could be saving that (or more) per camera already. Maybe it would hold off the move temporarily.

I just don't see it being practically possible for them to implement.

I agree with you about that. I highly highly doubt that they will implement a flat policy like that.

My ultimate goal is to spur awareness / discussion / 'thought leadership' that encourages integrators to ask Axis for some form of credit / comp and motivates Axis to do something.

Or they do not, and Hikvision can use this defect as another entry point.

I am torn on the whole "where does the responsibility start and end" thing only because it vary between manufacturers and it is in part also the owner/maintainers responsibility to take care of the maintenance of the camera.

Other electronic devices like phones, TV's, computers and Photography cameras in order to honor warranties require you to register your products with their respective manufacturer. When you do this they also send you information regarding your products as well as recalls. After reading the AXIS website warranty information all you need is a receipt and the serial number for the camera. with the way cyber security is going this may need to change as most other manufacturers follow this same model.

Best solution is to have a secure method to register these products so the manufacturer can send you emails saying hey you need to upgrade you firmware and here is how. This puts all of the liability on the owner and not the manufacturer. This could lead to some bad relations with integrators though as it cuts them out of service calls but IMHO that will go away sooner or later anyway with the way things are progressing.

So no I dont think it is AXIS's job to chase down people and let them know that the camera is security vulnerability, But I will say if what AXIS says:

"Axis is committed to providing high quality products, trouble-free ownership and better control of total costs by offering exceptional global service covering questions and trouble-shooting concerning both the installation and use of the products"

Then they need to come up with a better way to do what they are claiming

Eddie, good feedback.

For members, that quote is from their warranty page:

Do Jeep owners pay for their cars to be fixed?

If the car has a defect and there is a recall, absolutely.

This is the camera equivalent of a car defect / recall.

The auto repair analogy isn't cogent here without qualification.

If Jeep owners break their Jeep, they fix it.

If Jeep designs a part that is faulty and/or could result in damages, injury, or death they pay for the repair under a vehicle recall.

In the same way, a critical security vulnerability is not normal 'wear and tear', and I don't think the owner is expecting to pay to fix it.

If a vehicle has a defect/recall, the manufacturer fixes it under warranty without cost. Similarly, Axis has released this firmware update for free.

Jeep, does not pay the owner for their time to come into the dealership and wait for the car to be serviced. Most of them don't even have convenient hours, requiring the owner to potentially take time off of work to deal with the defect.

If you are lucky, you might get a free loaner car, or maybe the dealership will drop you off and pick you up when the work is complete, but you still incur costs in the form of time lost and inconvenience.

A typical vehicle also costs more than all the cameras in an "average" smaller system.

The car company pays for the part to be installed.

They don't just toss out a box of redesigned good parts and hang a 'free' sign over it.

If you carried through with the analogy though, would that not mean Axis offering to install the firmware free of charge, provided that you brought your affected cameras in to your closest Axis location?

"Jeep, does not pay the owner for their time to come into the dealership and wait for the car to be serviced. Most of them don't even have convenient hours, requiring the owner to potentially take time off of work to deal with the defect."

That is a great point I was actually thinking of posting before Brian R. and Brian K. (B&B) chimed in on it.

Maybe "higher ned" manufacturers should consider a firmware option to automatically download and install critical updates, either from Axis' website or from an optional location the installer sets up. That way Internet facing cameras, which are more at risk than network closed cameras can get updates easier and quicker.

In the context of the jeep example, the integrators are the equivalent of dealers, not users. Do car dealers have to do the recall work for free?

This source says typically no with some debate:

The basic pattern that has emerged is for dealers to fix recalled autos and automakers to pay dealers for the service.

Jeep put a bounty on vulnerabilities within their own software.

That they're only paying up to $1,500 shows how cheap and not-serious Jeep is.

Hackers exploit the lucrative software vulnerability trade

This 2013 article reports bounties of 5 to 6 figures being paid for finding exploits, and I have read that in other sources, too.

I think the best to solve this is to go to the "registering your product model"

If you have a camera or NVR and it needs a firmware upgrade then you get an email saying "hey you need to upgrade your firmware or we are not liable for whatever may happen.

also this would cover the warranty as well

1)so in short you buy a camera or NVR

2) there is a slip of paper in the manual or packet that says " if you want warranty or to avoid security vulnerabilities you need to register you products and upgrade according.

3) customer either does #2 and gets the upgrades and is put on an email list for upgrades that may come out or doesn't and assumes responsibility for what ever may happen.

4)Manufacturers get Brownie and advertising points saying they give a damn about the end user/owner, and they are doing it better than all the other Manufacturers.

this cleans up a lot of gray area's in the who is responsible for what questions. Manufacturer says we will take care or owner as long as you cooperate with rules. Owner gets great service from manufacturer and feels safer..... hopefully.

Should Axis pay TWICE for any integrators who have already downloaded the latest firmware, erroneously thinking this would solve their problem?

PRO TIP: When you actually are able to secure your camera, MAKE SURE TO RESET TO FACTORY DEFAULTS before upgrading the firmware. Although not mentioned by Axis, once the vulnerability is disclosed and script-kiddies probe every known Axis camera on the web, there may be nothing stopping them from making the firmware webpage non-functional except to change what is SAYS is the version.

Resetting to factory defaults, using the hard button, gives a better chance of eradicating any installed malware first. Though it should be noted that once the camera has been rooted, even the recovery partition can be corrupted, still that's unlikely to be exploited in the early days.

Jeep's aside, does anyone know of any software company, in any industry, ever paying for someone's time and labor of installing a patch?

If not, why do we think Axis will be the first?

I have backcharged camera manufacturers for our time to remedy their hardware defects quite successfully. Perhaps it is simply because we represent several large accounts for them, but this particular manufacturer has covered our labor costs on many occasions. It's not Axis.

Was the remedy a software patch?

In one instance, yes. However, it was a firmware update that required significant work on our part. Not something as easy as clicking to update in Axis Camera Management.

How does it differ from other software updates? Take Windows, for example. Granted, many/most are "functional" updates/patches, although many are security patches/updates to block/fix vulnerabilities.

While many a consumer can set their machine to update Windows automatically, many Enterprises do not allow such action, and have Network Admins who spend tons of time on installing/administering patches/updates.

Should Microsoft be reimbursing companies for Network Admin time spent applying/adminstering their constant patching? How does it differ from this situation and Axis? (and maybe others......)

Should Microsoft be reimbursing companies for Network Admin time spent applying/adminstering their constant patching?

But as you say before that:

Granted, many/most are "functional" updates/patches, although many are security patches/updates to block/fix vulnerabilities.

The difference here, compared to 99% of Windows updates and pretty much all prior Axis upgrades is that this is for defect that allows taking over control of the device.

The other issue is structural. Most PCs are managed by techs in-house. In IP video surveillance, it is the opposite. If upgrading all these defected Axis cameras was simply a matter of Axis pushing a button or the integrator pushing a button from their office, than the cost would be trivial for all parties. Unfortunately, that is not the case. Most of these are going to require truck rolls to deal with the typical IP camera not being remotely accessible to the typical integrator.

Most of these are going to require truck rolls to deal with the typical IP camera not being remotely accessible to the typical integrator.

If the camera is not remotely accessible then the patch is not urgent.

Can be seen as another huge plus for using HD Coax to some extent......

With 100 votes received, integrators and manufacturers (not surprisingly) have taken opposing views here.

While 71% of integrators vote yes, only 41% of manufacturers agree.

I am not a fan of these sensationalist articles.

All complex software has flaws, with the right research effort holes could be picked in almost any product. If it is a bug in the core of an open source product the exact same bug may appear in other brand's firmware, would you expect the same from every manufacturer?

I think that AXIS turning around a patched firmware for practically every model with a matter of days shows their commitment to holding themselves at a very high standard.

Let's stick to the announced vulnerability. It will only be applicable to cameras (video devices) directly accessible through the net (internet). The vulnerability will be removed by just upgrading those cameras to the latest firmaware version which is downloadable for free from the axis site. Threfore the job to do is to connect with those cameras (remotely? yes, if they are not accessible there's no risk) and upgrade the firmware. So costs will only be time spending in the process rather than moving to "far away" sites.

Axis has a free tool to do that in a semi automatic way (Axis Camera Management) which can be programmed to upgrade hundreds (thousands) of units in sequence at desired times. That would reduce time spent in the process dramatically.

On the other hand, it's suppossed that the integrator will take care of maintenance (that includes firmware upgrades). Yes, this is not an improvement upgrade for new features, but an upgrade anyway.

Does anybody pay for the upgrade of those cameras which were supplied with default password in the past until most manufacturers found out it was risky and changed the firmware?

It will only be applicable to cameras (video devices) directly accessible through the net (internet).

Just because the integrator does not have remote access to the cameras does not mean there is no remote access. Secondly, a prudent manager is not going to ignore doing the upgrade on such a critical vulnerability just because the cameras are behind a firewall. Mistakes can happen and also someone internally could take advantage of this as well.

I voted yes, but that's just wishing. I wound't expect that to happen. Its like taking a medicine and having it not work and going to the doctor and asking for a a free replacement.

Not really sure if Axis should pay integrators to do this. I have been in the industry over 20 years and I have never seen an Integrator roll trucks to do software or firmware upgrades to products.

As far as this vulnerability we upgraded over 1600 Axis cameras in the past week. We did this with out an integrator. But we always do our own firmware upgrades.

I have been in the industry over 20 years and I have never seen an Integrator roll trucks to do software or firmware upgrades to products.

Do you mean solely to do a software or firmware upgrade? I assume so since it is routine part of support and or maintenance for integrators to come up on site and do upgrades.

If you mean solely, for sure, that is rare. On the other hand, it is also rare to have a critical security vulnerability of this level.

It's the 18th, did Axis ask the researcher for more time?

http://seclists.org/bugtraq/2016/Jul/71

Thanks!

# I don't say that Axis Communication has made this hidden format string by this purpose.
# I can only believe it was a really stupid mistake from Axis side, after I have seen one screen-dump of the CVS
changelog of SSI Daemon,
# and another screen-dump with the change made late 2009, from non-vulnerable to vulnerable, in the affected code of
logerr().
#

Login to read this IPVM report.
Why do I need to log in?
IPVM conducts unique testing and research funded by member's payments enabling us to offer the most independent, accurate and in-depth information.

Related Reports on Integrators

Favorite Request-to-Exit (RTE) Manufacturers 2018 on Sep 19, 2018
Request To Exit devices like motion sensors and lock releasing push-buttons are a part of almost every access install, but who makes the equipment...
Favorite Intercom Manufacturers 2018 on Sep 14, 2018
Intercoms are certainly increasing in popularity, driven by the integration of video and IP networking. But who is the favorite? On the one side,...
Stanley Security Acquires 3xLogic, Kushner Becomes Product President on Sep 10, 2018
Stanley Security acquired 3xLogic a few months ago. However, the company has still not officially publicly announced it, leading many to wonder...
Dell Launches IoT for Surveillance on Sep 05, 2018
Historically, Dell has been a PC and server provider (e.g., "Dude, you're getting a Dell") and widely used for surveillance storage. However, in...
Sell Dahua or Hikvision At All, Banned From Selling to US Federal Government, Says US HASC on Aug 29, 2018
The US House Armed Services Committee (HASC) Communications Director has confirmed to IPVM that if a company sells Dahua or Hikvision at all, they...
France Political Scandal Reveals Video Surveillance Problems on Aug 22, 2018
In what French media describes as "the most damaging crisis yet for" French President Marcon, a political scandal has revealed major gaps in the...
ISS VMS / Video Analytics Company Profile on Aug 16, 2018
Who is ISS? In the past few months, they had one of the craziest ISC West promo items in years. Then, they hired industry veteran and ex-Dahua...
RealNetworks Free School Facial Recognition on Aug 03, 2018
The company that created RealPlayer is moving beyond media delivery and into the security space with a new facial recognition platform they have...
Improved Security And Surveillance Bidding - 2018 MasterFormat Divisions Examined) on Jul 19, 2018
Navigating the world of system specifications and bidding work can be complex and confusing, but a standard format exists, and understanding it...
4 Most Difficult Camera Installs (Statistics) on Jul 12, 2018
Heavy housings, cumbersome brackets, heavy ladders required, and tricky field of view requirements will cause difficulties no matter the camera...

Most Recent Industry Reports

Alexa Guard Expands Amazon's Security Offerings, Boosts ADT's Stock on Sep 21, 2018
Amazon is expanding their security offerings yet again, this time with Alexa Guard that delivers security audio analytics and a virtual "Fake...
UTC, Owner of Lenel, Acquires S2 on Sep 20, 2018
UTC now owns two of the biggest access control providers, one of integrator's most hated access control platforms, Lenel, and one of their...
BluePoint Aims To Bring Life-Safety Mind-Set To Police Pull Stations on Sep 20, 2018
Fire alarm pull stations are commonplace but police ones are not. A self-funded startup, BluePoint Alert Solutions is aiming to make police pull...
SIA Plays Dumb On OEMs And Hikua Ban on Sep 20, 2018
OEMs widely pretend to be 'manufacturers', deceiving their customers and putting them at risk for cybersecurity attacks and, soon, violation of US...
Axis Vs. Hikvision IR PTZ Shootout on Sep 20, 2018
Hikvision has their high-end dual-sensor DarkfighterX. Axis has their high-end concealed IR Q6125-LE. Which is better? We bought both and tested...
Avigilon Announces AI-Powered H5 Camera Development on Sep 19, 2018
Avigilon will be showcasing "next-generation AI" at next week's ASIS GSX. In an atypical move, the company is not actually releasing these...
Favorite Request-to-Exit (RTE) Manufacturers 2018 on Sep 19, 2018
Request To Exit devices like motion sensors and lock releasing push-buttons are a part of almost every access install, but who makes the equipment...
25% China Tariffs Finalized For 2019, 10% Start Now, Includes Select Video Surveillance on Sep 18, 2018
A surprise move: In July, when the most recent tariff round was first announced, the tariffs were only scheduled for 10%. However, now, the US...
Central Stations Face Off Against NFPA On Fire Monitoring on Sep 18, 2018
Central stations are facing off against the NFPA over what they call anti-competitive language in NFPA 72, the standard that covers fire alarms....
Hikvision USA Starts Layoffs on Sep 18, 2018
Hikvision USA has started layoffs, just weeks after the US government ban was passed into law. Inside this note, we examine: The important...

The world's leading video surveillance information source, IPVM provides the best reporting, testing and training for 10,000+ members globally. Dedicated to independent and objective information, we uniquely refuse any and all advertisements, sponsorship and consulting from manufacturers.

About | FAQ | Contact