Axis, Pay Integrators To Fix Your Critical Defect

Author: John Honovich, Published on Jul 14, 2016

Axis should pay integrators to fix Axis' recent disclosed critical security vulnerability / defect

There are millions of Axis cameras deployed with this defect. The risk is high as the soon to be published disclosure (July 18th) will allow attackers to take over the devices.

It falls on integrators to upgrade Axis cameras and eliminate this risk camera by camera.

Why Axis?

Axis should live up to their self proclaimed standards. Axis markets their high 'quality'. Integrators and customers pay a premium for that quality.

**** ****** *** *********** ** *******' ****** ********* ******** ******** ************* / ******

***** *** ******** ** **** ******* ******** **** **** ******. The **** ** **** ** *** **** ** ** ********* ********** (July ****) **** ***** ********* ** **** **** *** *******.

** ***** ** *********** ** ******* **** ******* *** ********* this **** ****** ** ******.

Why ****?

**** ****** **** ** ** ***** **** ********** *********. **** markets ***** **** '*******'. *********** *** ********* *** * ******* *** **** *******.

[***************]

******, ************ ********** '*********** ****** *******':

** ********, **** ***, ** ***** *** *****, * '********' vulnerability', * *********** ****** ** ***** *******. ** **** ***** at **** *** ***** ******* ** ** ***** ****, *** broader ******** ** **** *** ***** ******* ** ** **** to ****** ******* ******* *** *** ********** ** **** *********** and ********* *** ***** ****** ** ****.

**** ****** **** ** ** *** ********* *** ******* ******** by ************ *********** *** ***** *** **** ** *********** **** vulnerability ** ******* ****** *** *****.

Integrator *****

*** ***** *** *********** *** ** ***********:

  • ****, ** *** ****, ********* **** ******* **-**** ******, ********* customers *** ***** **** *** ** *** ****.
  • *** ********** *** **** ** ******* *** ************* ** *** ******** which **** ********* **** ** ***** ********* *****.
  • **** ****** **** ******* ******* ***** ********, **** ********* **** *** want ******** ******* ***** ******* ** **** **** *** * *** minutes *** *** ******* ********** ***** **** ***** ** ********* up ********. 
  • ***** *********, *** ********** **** **** ** ****** **** *** cameras ******** ** ****, ***** **** **** **** **** *** if *** ******* **** ** ********* ***** ** * *********** problem.

* ********** ******* ******* **** **** ** *********** ***** ** $200 *** ****. ** ******, ***** **** ** ********** ** time *** ****** *** * **** ***** *** **** ** hour ** **** ***** **** ** **** *****.

Vote: ****** **** ***?

 

 

Comments (47)

Isn't Axis already living up to their "self proclaimed standards"? It seems like Axis has far fewer scramble-the-techs vulnerabilities and critical upgrades than the majority of their competitors. Unless we expect Axis to produce a flawless product I would say that they are earning their price as it relates to this.

Presumably if you are an Axis fan you are also following their cyber security guide (Axis Cybersecurity Hardening Guide Examined), which recommends setting IP address filters once you get beyond "demo" or "small office" deployments.

If you have properly filtered/firewalled your cameras the probability of exploit drops dramatically. Theoretically customers are only liable to an internally-executed exploit, or a compromised VMS server being used to launch an attack against the camera. Both of these are scenarios that neither Axis, or the integrator, should really be seen as liable for.

While the ideal is non-exploitable firmware always, I do not think that is a realistic expectation. Customers and integrators need to understand that devices may need to have an emergency update done on one or more occasions over the life of the install, and set pricing for handling those issues accordingly.

Now, if this was the 3rd time Axis had a vulnerability like this in a year, I might agree that they owe integrators some recourse to justify their premium.

Semi-related, incidents like this are why integrators may want to offer service/emergency service programs to their customers. Perhaps the really savvy integrators can use Axis' weakness here as a new revenue stream for themselves.

Isn't Axis already living up to their "self proclaimed standards"?

It depends what we see those standards to be and how it compares to their competitors.

It seems like Axis has far fewer scramble-the-techs vulnerabilities and critical upgrades than the majority of their competitors.

I do not think Axis is worse but it also does not strike me that the average competitor is all that more problematic (excluding Arecont).

Take Axis vs Hikvision. Axis charges 50 - 100% more than the equivalent tech specs of Hikvision. Axis claims to differentiate on more subtle things like 'quality'.

To me, that's like Wal-Mart vs Nieman Marcus / Nordstrom. You go to Wal-Mart, Wal-Mart makes an error, you figure you eat it because it's Wal-Mart, everyday low prices, barebones service, etc. But Axis prices itself like Nieman Marcus / Nordstrom and the expectation at a place like that is that extra payment you make goes to white glove service and going the extra mile to solve mistakes.

I do not think Axis has to do this but if they want their buyers to remain confident why they are paying a premium for Axis, Axis should see paying integrators here as an investment in justifying the continued premium for Axis products.

I think it is worth keeping in mind here that this issue relates to the fact that while Axis may be really good at making cameras, they are less good at making firewalls, in a sense.

This vulnerability is something that appears to be somewhat difficult to stumble upon (or else Axis would have found it in earlier testing, or it would have been found/exploited much sooner).

If this was something that caused the camera to stop being a camera, like a bug that caused the iris to close any time the checksum of the date was a prime number, I might feel that Axis did not deliver on their position of building a superior product.

But this is the kind of vulnerability that almost every network device maker has encountered at one time or another, and it is why it is not recommended to have any device like this directly connected to the wide-open internet. This why firewalls exist, if devices could all be relied upon to manage their own security (and be easily manageable) we would not need firewalls (IPv4 address space and NAT simplicity aside).

While every Axis customer should upgrade their firmware to remove this vulnerability, the ones that are in a critical position are those that also seem to be lacking good network security in general. For this specific case I do not think Axis is really at fault here where they should be paying integrators. They encountered an exploit, announced it, fixed it, and have also previously given recommendations in their cyber security guide that help to minimize the impact of situations like this.

Do you want manufacturers to keep vulnerabilities secret? Because that's how you get manufacturers to keep vulnerabilities secret.

Axis has no choice to keep this 'secret'. The researcher is releasing this in 4 days after following responsible disclosure.

They can't keep this one secret, but they can keep the next one secret.

If you make it expensive for a manufacturer to admit fault and release a patch, they'll only do it when forced to.

Axis releases what, dozens of firmware updates yearly? Why bother once they've set the precedent that they'll pay you to install it?

Axis releases what, dozens of firmware updates yearly? Why bother once they've set the precedent that they'll pay you to install it?

Because the typical firmware release is incremental tech advances (e.g., slightly better WDR) or minor bug fixes (now works with browser x for niche feature y).

This firmware release is to stop attackers from gaining root access and taking control of Axis cameras. The severity of this is far beyond any typical firmware upgrade.

An extraordinary problem merits an extraordinary solution.

If Axis made a statement and said, "We guarantee zero vulnerabilities." Then yes, they should pay integrators to fix the issue, because "zero vulnerabilities" was guaranteed. If they didn't make a statement similar to this, then they shouldn't have to or be expected to by their partners. As businesses, we assume certain risks, while we do our best to mitigate risk by installing quality products and hiring good people. Sometimes things happen and we have to deal with it.

Agree with it or not, every Hikvision dealer has more fuel to combat Axis. If Axis were smart, they would take council from Tylenol and get ahead of this.

Speaking of Nordstrom, I bought a $1,200 suit there a couple of years ago. Lost 30 lbs and the suit didn't fit me anymore. I went back and was shopping for another suit. I told the sales associated what happened, and he got the manager, and the manager asked me to bring the suit back. They credited my credit card and took the suit back (it was well worn for a year). Since then I have spent lots of $$$ at Nordstrom.

Agree with it or not, every Hikvision dealer has more fuel to combat Axis.

I agree. That's why I think paying integrators here would reduce that fuel and make integrators feel more confident and likely to spend more with Axis, like you did with Nordstrom.

Any estimates on the financial impact of even offering $200/site to integrators globally? Would it materially impact the Axis numbers in 2016?

Interesting and controversial suggestion. Seems like it could open a Pandora's Box of issues with some integrators servicing smaller customers and happy with the $200 and others with sites in the 100's or 1000's of cameras then wanting more compensation. Or finding a way to split the site into smaller sites... And many of the sites could be covered under service agreements which may already include application of f/w upgrades.

Any estimates on the financial impact of even offering $200/site to integrators globally?

Let's say there are 100,000 sites with Axis cameras (just to use a rough number). At $200 per site, that's $20 million total, compared to Axis ~$800 million revenue. So a hit but survivable. Also, you would suspect some will not upgrade, some will ignore getting payment, not now, not want to share end user details with Axis, etc.

Seems like it could open a Pandora's Box of issues with some integrators servicing smaller customers and happy with the $200 and others with sites in the 100's or 1000's of cameras then wanting more compensation.

I agree. That's why I framed it as an average payment. If you have to emergency upgrade all of Target or City of Chicago, etc. it would be a lot more money.

And many of the sites could be covered under service agreements which may already include application of f/w upgrades.

Even if the integrator has a service agreement, it still costs them money to do the upgrade, which would be unnecessary if this defect did not exist.

The comparison shouldn't be versus revenue though but against profit: this would come straight off their bottom line and $20M profit is a much bigger hit. Also survivable but pretty painful.

Axis 2015 operating profit was ~$100 million USD, so a fraction of their annual profit.

Think about it this way, though, if Axis steps up and makes their partners whole, how much revenue and profits will they save going forward? How many Axis partners are on the verge of leaving Axis for lower cost alternatives and will see this as one more reason to switch?

If - and that's a big assumption - the cost impact was in the region of $20M then that's a pretty material impact on $100M profit, for what's basically a goodwill gesture. Maybe it's more or less but the estimated fraction of profit would be an important part of the decision.

I understand your perspective and the points supporting the argument, I just don't see it being practically possible for them to implement. I'd also be surprised if anyone thinking of defecting to a lower cost alternative were swayed by a token payment of roughly $200 per site: they could be saving that (or more) per camera already. Maybe it would hold off the move temporarily.

I just don't see it being practically possible for them to implement.

I agree with you about that. I highly highly doubt that they will implement a flat policy like that.

My ultimate goal is to spur awareness / discussion / 'thought leadership' that encourages integrators to ask Axis for some form of credit / comp and motivates Axis to do something.

Or they do not, and Hikvision can use this defect as another entry point.

I am torn on the whole "where does the responsibility start and end" thing only because it vary between manufacturers and it is in part also the owner/maintainers responsibility to take care of the maintenance of the camera.

Other electronic devices like phones, TV's, computers and Photography cameras in order to honor warranties require you to register your products with their respective manufacturer. When you do this they also send you information regarding your products as well as recalls. After reading the AXIS website warranty information all you need is a receipt and the serial number for the camera. with the way cyber security is going this may need to change as most other manufacturers follow this same model.

Best solution is to have a secure method to register these products so the manufacturer can send you emails saying hey you need to upgrade you firmware and here is how. This puts all of the liability on the owner and not the manufacturer. This could lead to some bad relations with integrators though as it cuts them out of service calls but IMHO that will go away sooner or later anyway with the way things are progressing.

So no I dont think it is AXIS's job to chase down people and let them know that the camera is security vulnerability, But I will say if what AXIS says:

"Axis is committed to providing high quality products, trouble-free ownership and better control of total costs by offering exceptional global service covering questions and trouble-shooting concerning both the installation and use of the products"

Then they need to come up with a better way to do what they are claiming

Eddie, good feedback.

For members, that quote is from their warranty page:

Do Jeep owners pay for their cars to be fixed?

If the car has a defect and there is a recall, absolutely.

This is the camera equivalent of a car defect / recall.

The auto repair analogy isn't cogent here without qualification.

If Jeep owners break their Jeep, they fix it.

If Jeep designs a part that is faulty and/or could result in damages, injury, or death they pay for the repair under a vehicle recall.

In the same way, a critical security vulnerability is not normal 'wear and tear', and I don't think the owner is expecting to pay to fix it.

If a vehicle has a defect/recall, the manufacturer fixes it under warranty without cost. Similarly, Axis has released this firmware update for free.

Jeep, does not pay the owner for their time to come into the dealership and wait for the car to be serviced. Most of them don't even have convenient hours, requiring the owner to potentially take time off of work to deal with the defect.

If you are lucky, you might get a free loaner car, or maybe the dealership will drop you off and pick you up when the work is complete, but you still incur costs in the form of time lost and inconvenience.

A typical vehicle also costs more than all the cameras in an "average" smaller system.

The car company pays for the part to be installed.

They don't just toss out a box of redesigned good parts and hang a 'free' sign over it.

If you carried through with the analogy though, would that not mean Axis offering to install the firmware free of charge, provided that you brought your affected cameras in to your closest Axis location?

"Jeep, does not pay the owner for their time to come into the dealership and wait for the car to be serviced. Most of them don't even have convenient hours, requiring the owner to potentially take time off of work to deal with the defect."

That is a great point I was actually thinking of posting before Brian R. and Brian K. (B&B) chimed in on it.

Maybe "higher ned" manufacturers should consider a firmware option to automatically download and install critical updates, either from Axis' website or from an optional location the installer sets up. That way Internet facing cameras, which are more at risk than network closed cameras can get updates easier and quicker.

In the context of the jeep example, the integrators are the equivalent of dealers, not users. Do car dealers have to do the recall work for free?

This source says typically no with some debate:

The basic pattern that has emerged is for dealers to fix recalled autos and automakers to pay dealers for the service.

Jeep put a bounty on vulnerabilities within their own software.

That they're only paying up to $1,500 shows how cheap and not-serious Jeep is.

Hackers exploit the lucrative software vulnerability trade

This 2013 article reports bounties of 5 to 6 figures being paid for finding exploits, and I have read that in other sources, too.

I think the best to solve this is to go to the "registering your product model"

If you have a camera or NVR and it needs a firmware upgrade then you get an email saying "hey you need to upgrade your firmware or we are not liable for whatever may happen.

also this would cover the warranty as well

1)so in short you buy a camera or NVR

2) there is a slip of paper in the manual or packet that says " if you want warranty or to avoid security vulnerabilities you need to register you products and upgrade according.

3) customer either does #2 and gets the upgrades and is put on an email list for upgrades that may come out or doesn't and assumes responsibility for what ever may happen.

4)Manufacturers get Brownie and advertising points saying they give a damn about the end user/owner, and they are doing it better than all the other Manufacturers.

this cleans up a lot of gray area's in the who is responsible for what questions. Manufacturer says we will take care or owner as long as you cooperate with rules. Owner gets great service from manufacturer and feels safer..... hopefully.

Should Axis pay TWICE for any integrators who have already downloaded the latest firmware, erroneously thinking this would solve their problem?

PRO TIP: When you actually are able to secure your camera, MAKE SURE TO RESET TO FACTORY DEFAULTS before upgrading the firmware. Although not mentioned by Axis, once the vulnerability is disclosed and script-kiddies probe every known Axis camera on the web, there may be nothing stopping them from making the firmware webpage non-functional except to change what is SAYS is the version.

Resetting to factory defaults, using the hard button, gives a better chance of eradicating any installed malware first. Though it should be noted that once the camera has been rooted, even the recovery partition can be corrupted, still that's unlikely to be exploited in the early days.

Jeep's aside, does anyone know of any software company, in any industry, ever paying for someone's time and labor of installing a patch?

If not, why do we think Axis will be the first?

I have backcharged camera manufacturers for our time to remedy their hardware defects quite successfully. Perhaps it is simply because we represent several large accounts for them, but this particular manufacturer has covered our labor costs on many occasions. It's not Axis.

Was the remedy a software patch?

In one instance, yes. However, it was a firmware update that required significant work on our part. Not something as easy as clicking to update in Axis Camera Management.

How does it differ from other software updates? Take Windows, for example. Granted, many/most are "functional" updates/patches, although many are security patches/updates to block/fix vulnerabilities.

While many a consumer can set their machine to update Windows automatically, many Enterprises do not allow such action, and have Network Admins who spend tons of time on installing/administering patches/updates.

Should Microsoft be reimbursing companies for Network Admin time spent applying/adminstering their constant patching? How does it differ from this situation and Axis? (and maybe others......)

Should Microsoft be reimbursing companies for Network Admin time spent applying/adminstering their constant patching?

But as you say before that:

Granted, many/most are "functional" updates/patches, although many are security patches/updates to block/fix vulnerabilities.

The difference here, compared to 99% of Windows updates and pretty much all prior Axis upgrades is that this is for defect that allows taking over control of the device.

The other issue is structural. Most PCs are managed by techs in-house. In IP video surveillance, it is the opposite. If upgrading all these defected Axis cameras was simply a matter of Axis pushing a button or the integrator pushing a button from their office, than the cost would be trivial for all parties. Unfortunately, that is not the case. Most of these are going to require truck rolls to deal with the typical IP camera not being remotely accessible to the typical integrator.

Most of these are going to require truck rolls to deal with the typical IP camera not being remotely accessible to the typical integrator.

If the camera is not remotely accessible then the patch is not urgent.

Can be seen as another huge plus for using HD Coax to some extent......

With 100 votes received, integrators and manufacturers (not surprisingly) have taken opposing views here.

While 71% of integrators vote yes, only 41% of manufacturers agree.

I am not a fan of these sensationalist articles.

All complex software has flaws, with the right research effort holes could be picked in almost any product. If it is a bug in the core of an open source product the exact same bug may appear in other brand's firmware, would you expect the same from every manufacturer?

I think that AXIS turning around a patched firmware for practically every model with a matter of days shows their commitment to holding themselves at a very high standard.

Let's stick to the announced vulnerability. It will only be applicable to cameras (video devices) directly accessible through the net (internet). The vulnerability will be removed by just upgrading those cameras to the latest firmaware version which is downloadable for free from the axis site. Threfore the job to do is to connect with those cameras (remotely? yes, if they are not accessible there's no risk) and upgrade the firmware. So costs will only be time spending in the process rather than moving to "far away" sites.

Axis has a free tool to do that in a semi automatic way (Axis Camera Management) which can be programmed to upgrade hundreds (thousands) of units in sequence at desired times. That would reduce time spent in the process dramatically.

On the other hand, it's suppossed that the integrator will take care of maintenance (that includes firmware upgrades). Yes, this is not an improvement upgrade for new features, but an upgrade anyway.

Does anybody pay for the upgrade of those cameras which were supplied with default password in the past until most manufacturers found out it was risky and changed the firmware?

It will only be applicable to cameras (video devices) directly accessible through the net (internet).

Just because the integrator does not have remote access to the cameras does not mean there is no remote access. Secondly, a prudent manager is not going to ignore doing the upgrade on such a critical vulnerability just because the cameras are behind a firewall. Mistakes can happen and also someone internally could take advantage of this as well.

I voted yes, but that's just wishing. I wound't expect that to happen. Its like taking a medicine and having it not work and going to the doctor and asking for a a free replacement.

Not really sure if Axis should pay integrators to do this. I have been in the industry over 20 years and I have never seen an Integrator roll trucks to do software or firmware upgrades to products.

As far as this vulnerability we upgraded over 1600 Axis cameras in the past week. We did this with out an integrator. But we always do our own firmware upgrades.

I have been in the industry over 20 years and I have never seen an Integrator roll trucks to do software or firmware upgrades to products.

Do you mean solely to do a software or firmware upgrade? I assume so since it is routine part of support and or maintenance for integrators to come up on site and do upgrades.

If you mean solely, for sure, that is rare. On the other hand, it is also rare to have a critical security vulnerability of this level.

It's the 18th, did Axis ask the researcher for more time?

http://seclists.org/bugtraq/2016/Jul/71

Thanks!

# I don't say that Axis Communication has made this hidden format string by this purpose.
# I can only believe it was a really stupid mistake from Axis side, after I have seen one screen-dump of the CVS
changelog of SSI Daemon,
# and another screen-dump with the change made late 2009, from non-vulnerable to vulnerable, in the affected code of
logerr().
#

Login to read this IPVM report.
Why do I need to log in?
IPVM conducts unique testing and research funded by member's payments enabling us to offer the most independent, accurate and in-depth information.

Related Reports on Integrators

ISC East 2018 Mini-Show Report on Nov 15, 2018
ISC East, by its own admission, is not a national or international show, billed as the "Largest Annual Northeast U.S. Security...
Magos Radar Company Profile on Nov 12, 2018
Magos America General Manager Yaron Zussman admits when he first came across Magos, he asked himself: "What's innovative about radar?" Be that as...
Video Surveillance Hard Drive Size Statistics 2018 on Nov 08, 2018
What is the most common hard drive size for video surveillance? 150+ integrators answered: What size hard drive do you most commonly use? What...
Favorite Video Surveillance Hard Drive Manufacturer 2018 on Nov 06, 2018
Who is the favorite hard drive for video surveillance use? 150+ integrators answered: What is your preferred brand/model of hard drive for...
Worst Products on Nov 03, 2018
Security integrators periodically report on their favorite and worst products to IPVM. These are known integrators who IPVM pays to answer surveys....
Video Surveillance Hard Drive Failure Statistics 2018 on Nov 02, 2018
Hard drive failures can be significant service problems but how common of an issue are they in video surveillance? How long do drives last when...
Cloud Video Storage Usage 2018 on Oct 31, 2018
Storing email and documents in the cloud have long been common, with on-site email or file servers increasingly eliminated. However, what about...
ADT Acquires Red Hawk, Becomes Billion Dollar Commercial Business on Oct 25, 2018
With its ~7th commercial acquisition in the past year, ADT is transforming itself from a residential provider to a commercial player with now more...
Exacq Co-Founders Return, Start Qumulex on Oct 24, 2018
Exacq co-founders Dan Rittman, Tom Buckley, and David Underwood are back, starting up Qumulex. In 2000, they sold Integral to Andover...
Geutebruck Company Profile on Oct 22, 2018
Geutebrück has been in business for nearly 50 years, but they are not well known within the US surveillance market. In this report, we profile...

Most Recent Industry Reports

Throughtek P2P/Cloud Solution Profile on Nov 15, 2018
Many IoT manufacturers either do not have the capabilities or the interest to develop their own cloud management software for their devices....
ASIS Offering Custom Research For Manufacturers on Nov 15, 2018
Manufacturers often want to know what industry people think about trends and, in particular, the segments and product they offer.  ASIS and its...
ISC East 2018 Mini-Show Report on Nov 15, 2018
ISC East, by its own admission, is not a national or international show, billed as the "Largest Annual Northeast U.S. Security...
Hikvision Silent on "Bad Architectural Practices" Cybersecurity Report on Nov 14, 2018
A 'significant vulnerability was found in Hikvision cameras' by VDOO, a startup cybersecurity specialist. Hikvision has fixed the specific...
French Government Threatens School with $1.7M Fine For “Excessive Video Surveillance” on Nov 14, 2018
The French government has notified a high-profile Paris coding academy that it risks a fine of up to 1.5 million euros (about $1.7m) if it...
Integrator Credit Card Alternative Divvy on Nov 13, 2018
Most security integrators are small businesses but large enough that they have various employees that need to be able to expense various charges as...
Directory of Video Intercoms on Nov 13, 2018
Video Intercoms, also known as Video Door-Phones or Video Entry Systems, have been growing in the past decade as more and more IP camera...
Beware Amazon Go Store Hype (Tested) on Nov 13, 2018
IPVM's trip to and testing of Amazon Go's San Francisco store shows a number of significant operational and economic issues that undermine the...
Magos Radar Company Profile on Nov 12, 2018
Magos America General Manager Yaron Zussman admits when he first came across Magos, he asked himself: "What's innovative about radar?" Be that as...
Genetec Privacy Protector Tested on Nov 12, 2018
Genetec has built Kiwi Security's Privacy Protector into Security Center, an analytic which anonymizes individuals in cameras' fields of view...

The world's leading video surveillance information source, IPVM provides the best reporting, testing and training for 10,000+ members globally. Dedicated to independent and objective information, we uniquely refuse any and all advertisements, sponsorship and consulting from manufacturers.

About | FAQ | Contact