Axis, Pay Integrators To Fix Your Critical Defect

By John Honovich, Published on Jul 14, 2016

Axis should pay integrators to fix Axis' recent disclosed critical security vulnerability / defect

There are millions of Axis cameras deployed with this defect. The risk is high as the soon to be published disclosure (July 18th) will allow attackers to take over the devices.

It falls on integrators to upgrade Axis cameras and eliminate this risk camera by camera.

Why Axis?

Axis should live up to their self proclaimed standards. Axis markets their high 'quality'. Integrators and customers pay a premium for that quality.

******, ************ ********** '*********** ****** *******':

** ********, **** ***, in ***** *** *****, a '********' *************', * fundamental ****** ** ***** product. ** **** ***** at **** *** ***** cameras ** ** ***** over, *** ******* ******** at **** *** ***** cameras ** ** **** to ****** ******* ******* and *** ********** ** Axis *********** *** ********* for ***** ****** ** this.

**** ****** **** ** to *** ********* *** premium ******** ** ************ integrators *** ***** *** work ** *********** **** vulnerability ** ******* ****** the *****.

Integrator *****

*** ***** *** *********** can ** ***********:

  • ****, ** *** ****, customers **** ******* **-**** visits, ********* ********* *** sites **** *** ** far ****.
  • *** ********** *** **** ** explain *** ************* ** the ******** ***** **** customers **** ** ***** concerned *****.
  • **** ****** **** ******* support batch ********, **** ********* will *** **** ******** ******* going ******* ** **** even *** * *** minutes *** *** ******* scheduling ***** **** ***** or ********* ** ********. 
  • ***** *********, *** ********** will **** ** ****** that *** ******* ******** to ****, ***** **** take **** **** *** if *** ******* **** to ********* ***** ** a *********** *******.

* ********** ******* ******* from **** ** *********** would ** $*** *** site. ** ******, ***** will ** ********** ** time *** ****** *** a **** ***** *** even ** **** ** site ***** **** ** that *****.

Vote: ****** **** ***?

 

 

Comments (47)

Isn't Axis already living up to their "self proclaimed standards"? It seems like Axis has far fewer scramble-the-techs vulnerabilities and critical upgrades than the majority of their competitors. Unless we expect Axis to produce a flawless product I would say that they are earning their price as it relates to this.

Presumably if you are an Axis fan you are also following their cyber security guide (Axis Cybersecurity Hardening Guide Examined), which recommends setting IP address filters once you get beyond "demo" or "small office" deployments.

If you have properly filtered/firewalled your cameras the probability of exploit drops dramatically. Theoretically customers are only liable to an internally-executed exploit, or a compromised VMS server being used to launch an attack against the camera. Both of these are scenarios that neither Axis, or the integrator, should really be seen as liable for.

While the ideal is non-exploitable firmware always, I do not think that is a realistic expectation. Customers and integrators need to understand that devices may need to have an emergency update done on one or more occasions over the life of the install, and set pricing for handling those issues accordingly.

Now, if this was the 3rd time Axis had a vulnerability like this in a year, I might agree that they owe integrators some recourse to justify their premium.

Semi-related, incidents like this are why integrators may want to offer service/emergency service programs to their customers. Perhaps the really savvy integrators can use Axis' weakness here as a new revenue stream for themselves.

Isn't Axis already living up to their "self proclaimed standards"?

It depends what we see those standards to be and how it compares to their competitors.

It seems like Axis has far fewer scramble-the-techs vulnerabilities and critical upgrades than the majority of their competitors.

I do not think Axis is worse but it also does not strike me that the average competitor is all that more problematic (excluding Arecont).

Take Axis vs Hikvision. Axis charges 50 - 100% more than the equivalent tech specs of Hikvision. Axis claims to differentiate on more subtle things like 'quality'.

To me, that's like Wal-Mart vs Nieman Marcus / Nordstrom. You go to Wal-Mart, Wal-Mart makes an error, you figure you eat it because it's Wal-Mart, everyday low prices, barebones service, etc. But Axis prices itself like Nieman Marcus / Nordstrom and the expectation at a place like that is that extra payment you make goes to white glove service and going the extra mile to solve mistakes.

I do not think Axis has to do this but if they want their buyers to remain confident why they are paying a premium for Axis, Axis should see paying integrators here as an investment in justifying the continued premium for Axis products.

I think it is worth keeping in mind here that this issue relates to the fact that while Axis may be really good at making cameras, they are less good at making firewalls, in a sense.

This vulnerability is something that appears to be somewhat difficult to stumble upon (or else Axis would have found it in earlier testing, or it would have been found/exploited much sooner).

If this was something that caused the camera to stop being a camera, like a bug that caused the iris to close any time the checksum of the date was a prime number, I might feel that Axis did not deliver on their position of building a superior product.

But this is the kind of vulnerability that almost every network device maker has encountered at one time or another, and it is why it is not recommended to have any device like this directly connected to the wide-open internet. This why firewalls exist, if devices could all be relied upon to manage their own security (and be easily manageable) we would not need firewalls (IPv4 address space and NAT simplicity aside).

While every Axis customer should upgrade their firmware to remove this vulnerability, the ones that are in a critical position are those that also seem to be lacking good network security in general. For this specific case I do not think Axis is really at fault here where they should be paying integrators. They encountered an exploit, announced it, fixed it, and have also previously given recommendations in their cyber security guide that help to minimize the impact of situations like this.

Do you want manufacturers to keep vulnerabilities secret? Because that's how you get manufacturers to keep vulnerabilities secret.

Axis has no choice to keep this 'secret'. The researcher is releasing this in 4 days after following responsible disclosure.

They can't keep this one secret, but they can keep the next one secret.

If you make it expensive for a manufacturer to admit fault and release a patch, they'll only do it when forced to.

Axis releases what, dozens of firmware updates yearly? Why bother once they've set the precedent that they'll pay you to install it?

Axis releases what, dozens of firmware updates yearly? Why bother once they've set the precedent that they'll pay you to install it?

Because the typical firmware release is incremental tech advances (e.g., slightly better WDR) or minor bug fixes (now works with browser x for niche feature y).

This firmware release is to stop attackers from gaining root access and taking control of Axis cameras. The severity of this is far beyond any typical firmware upgrade.

An extraordinary problem merits an extraordinary solution.

If Axis made a statement and said, "We guarantee zero vulnerabilities." Then yes, they should pay integrators to fix the issue, because "zero vulnerabilities" was guaranteed. If they didn't make a statement similar to this, then they shouldn't have to or be expected to by their partners. As businesses, we assume certain risks, while we do our best to mitigate risk by installing quality products and hiring good people. Sometimes things happen and we have to deal with it.

Agree with it or not, every Hikvision dealer has more fuel to combat Axis. If Axis were smart, they would take council from Tylenol and get ahead of this.

Speaking of Nordstrom, I bought a $1,200 suit there a couple of years ago. Lost 30 lbs and the suit didn't fit me anymore. I went back and was shopping for another suit. I told the sales associated what happened, and he got the manager, and the manager asked me to bring the suit back. They credited my credit card and took the suit back (it was well worn for a year). Since then I have spent lots of $$$ at Nordstrom.

Agree with it or not, every Hikvision dealer has more fuel to combat Axis.

I agree. That's why I think paying integrators here would reduce that fuel and make integrators feel more confident and likely to spend more with Axis, like you did with Nordstrom.

Any estimates on the financial impact of even offering $200/site to integrators globally? Would it materially impact the Axis numbers in 2016?

Interesting and controversial suggestion. Seems like it could open a Pandora's Box of issues with some integrators servicing smaller customers and happy with the $200 and others with sites in the 100's or 1000's of cameras then wanting more compensation. Or finding a way to split the site into smaller sites... And many of the sites could be covered under service agreements which may already include application of f/w upgrades.

Any estimates on the financial impact of even offering $200/site to integrators globally?

Let's say there are 100,000 sites with Axis cameras (just to use a rough number). At $200 per site, that's $20 million total, compared to Axis ~$800 million revenue. So a hit but survivable. Also, you would suspect some will not upgrade, some will ignore getting payment, not now, not want to share end user details with Axis, etc.

Seems like it could open a Pandora's Box of issues with some integrators servicing smaller customers and happy with the $200 and others with sites in the 100's or 1000's of cameras then wanting more compensation.

I agree. That's why I framed it as an average payment. If you have to emergency upgrade all of Target or City of Chicago, etc. it would be a lot more money.

And many of the sites could be covered under service agreements which may already include application of f/w upgrades.

Even if the integrator has a service agreement, it still costs them money to do the upgrade, which would be unnecessary if this defect did not exist.

The comparison shouldn't be versus revenue though but against profit: this would come straight off their bottom line and $20M profit is a much bigger hit. Also survivable but pretty painful.

Axis 2015 operating profit was ~$100 million USD, so a fraction of their annual profit.

Think about it this way, though, if Axis steps up and makes their partners whole, how much revenue and profits will they save going forward? How many Axis partners are on the verge of leaving Axis for lower cost alternatives and will see this as one more reason to switch?

If - and that's a big assumption - the cost impact was in the region of $20M then that's a pretty material impact on $100M profit, for what's basically a goodwill gesture. Maybe it's more or less but the estimated fraction of profit would be an important part of the decision.

I understand your perspective and the points supporting the argument, I just don't see it being practically possible for them to implement. I'd also be surprised if anyone thinking of defecting to a lower cost alternative were swayed by a token payment of roughly $200 per site: they could be saving that (or more) per camera already. Maybe it would hold off the move temporarily.

I just don't see it being practically possible for them to implement.

I agree with you about that. I highly highly doubt that they will implement a flat policy like that.

My ultimate goal is to spur awareness / discussion / 'thought leadership' that encourages integrators to ask Axis for some form of credit / comp and motivates Axis to do something.

Or they do not, and Hikvision can use this defect as another entry point.

I am torn on the whole "where does the responsibility start and end" thing only because it vary between manufacturers and it is in part also the owner/maintainers responsibility to take care of the maintenance of the camera.

Other electronic devices like phones, TV's, computers and Photography cameras in order to honor warranties require you to register your products with their respective manufacturer. When you do this they also send you information regarding your products as well as recalls. After reading the AXIS website warranty information all you need is a receipt and the serial number for the camera. with the way cyber security is going this may need to change as most other manufacturers follow this same model.

Best solution is to have a secure method to register these products so the manufacturer can send you emails saying hey you need to upgrade you firmware and here is how. This puts all of the liability on the owner and not the manufacturer. This could lead to some bad relations with integrators though as it cuts them out of service calls but IMHO that will go away sooner or later anyway with the way things are progressing.

So no I dont think it is AXIS's job to chase down people and let them know that the camera is security vulnerability, But I will say if what AXIS says:

"Axis is committed to providing high quality products, trouble-free ownership and better control of total costs by offering exceptional global service covering questions and trouble-shooting concerning both the installation and use of the products"

Then they need to come up with a better way to do what they are claiming

Eddie, good feedback.

For members, that quote is from their warranty page:

Do Jeep owners pay for their cars to be fixed?

If the car has a defect and there is a recall, absolutely.

This is the camera equivalent of a car defect / recall.

The auto repair analogy isn't cogent here without qualification.

If Jeep owners break their Jeep, they fix it.

If Jeep designs a part that is faulty and/or could result in damages, injury, or death they pay for the repair under a vehicle recall.

In the same way, a critical security vulnerability is not normal 'wear and tear', and I don't think the owner is expecting to pay to fix it.

If a vehicle has a defect/recall, the manufacturer fixes it under warranty without cost. Similarly, Axis has released this firmware update for free.

Jeep, does not pay the owner for their time to come into the dealership and wait for the car to be serviced. Most of them don't even have convenient hours, requiring the owner to potentially take time off of work to deal with the defect.

If you are lucky, you might get a free loaner car, or maybe the dealership will drop you off and pick you up when the work is complete, but you still incur costs in the form of time lost and inconvenience.

A typical vehicle also costs more than all the cameras in an "average" smaller system.

The car company pays for the part to be installed.

They don't just toss out a box of redesigned good parts and hang a 'free' sign over it.

If you carried through with the analogy though, would that not mean Axis offering to install the firmware free of charge, provided that you brought your affected cameras in to your closest Axis location?

"Jeep, does not pay the owner for their time to come into the dealership and wait for the car to be serviced. Most of them don't even have convenient hours, requiring the owner to potentially take time off of work to deal with the defect."

That is a great point I was actually thinking of posting before Brian R. and Brian K. (B&B) chimed in on it.

Maybe "higher ned" manufacturers should consider a firmware option to automatically download and install critical updates, either from Axis' website or from an optional location the installer sets up. That way Internet facing cameras, which are more at risk than network closed cameras can get updates easier and quicker.

In the context of the jeep example, the integrators are the equivalent of dealers, not users. Do car dealers have to do the recall work for free?

This source says typically no with some debate:

The basic pattern that has emerged is for dealers to fix recalled autos and automakers to pay dealers for the service.

Jeep put a bounty on vulnerabilities within their own software.

That they're only paying up to $1,500 shows how cheap and not-serious Jeep is.

Hackers exploit the lucrative software vulnerability trade

This 2013 article reports bounties of 5 to 6 figures being paid for finding exploits, and I have read that in other sources, too.

I think the best to solve this is to go to the "registering your product model"

If you have a camera or NVR and it needs a firmware upgrade then you get an email saying "hey you need to upgrade your firmware or we are not liable for whatever may happen.

also this would cover the warranty as well

1)so in short you buy a camera or NVR

2) there is a slip of paper in the manual or packet that says " if you want warranty or to avoid security vulnerabilities you need to register you products and upgrade according.

3) customer either does #2 and gets the upgrades and is put on an email list for upgrades that may come out or doesn't and assumes responsibility for what ever may happen.

4)Manufacturers get Brownie and advertising points saying they give a damn about the end user/owner, and they are doing it better than all the other Manufacturers.

this cleans up a lot of gray area's in the who is responsible for what questions. Manufacturer says we will take care or owner as long as you cooperate with rules. Owner gets great service from manufacturer and feels safer..... hopefully.

Should Axis pay TWICE for any integrators who have already downloaded the latest firmware, erroneously thinking this would solve their problem?

PRO TIP: When you actually are able to secure your camera, MAKE SURE TO RESET TO FACTORY DEFAULTS before upgrading the firmware. Although not mentioned by Axis, once the vulnerability is disclosed and script-kiddies probe every known Axis camera on the web, there may be nothing stopping them from making the firmware webpage non-functional except to change what is SAYS is the version.

Resetting to factory defaults, using the hard button, gives a better chance of eradicating any installed malware first. Though it should be noted that once the camera has been rooted, even the recovery partition can be corrupted, still that's unlikely to be exploited in the early days.

Jeep's aside, does anyone know of any software company, in any industry, ever paying for someone's time and labor of installing a patch?

If not, why do we think Axis will be the first?

I have backcharged camera manufacturers for our time to remedy their hardware defects quite successfully. Perhaps it is simply because we represent several large accounts for them, but this particular manufacturer has covered our labor costs on many occasions. It's not Axis.

Was the remedy a software patch?

In one instance, yes. However, it was a firmware update that required significant work on our part. Not something as easy as clicking to update in Axis Camera Management.

How does it differ from other software updates? Take Windows, for example. Granted, many/most are "functional" updates/patches, although many are security patches/updates to block/fix vulnerabilities.

While many a consumer can set their machine to update Windows automatically, many Enterprises do not allow such action, and have Network Admins who spend tons of time on installing/administering patches/updates.

Should Microsoft be reimbursing companies for Network Admin time spent applying/adminstering their constant patching? How does it differ from this situation and Axis? (and maybe others......)

Should Microsoft be reimbursing companies for Network Admin time spent applying/adminstering their constant patching?

But as you say before that:

Granted, many/most are "functional" updates/patches, although many are security patches/updates to block/fix vulnerabilities.

The difference here, compared to 99% of Windows updates and pretty much all prior Axis upgrades is that this is for defect that allows taking over control of the device.

The other issue is structural. Most PCs are managed by techs in-house. In IP video surveillance, it is the opposite. If upgrading all these defected Axis cameras was simply a matter of Axis pushing a button or the integrator pushing a button from their office, than the cost would be trivial for all parties. Unfortunately, that is not the case. Most of these are going to require truck rolls to deal with the typical IP camera not being remotely accessible to the typical integrator.

Most of these are going to require truck rolls to deal with the typical IP camera not being remotely accessible to the typical integrator.

If the camera is not remotely accessible then the patch is not urgent.

Can be seen as another huge plus for using HD Coax to some extent......

With 100 votes received, integrators and manufacturers (not surprisingly) have taken opposing views here.

While 71% of integrators vote yes, only 41% of manufacturers agree.

I am not a fan of these sensationalist articles.

All complex software has flaws, with the right research effort holes could be picked in almost any product. If it is a bug in the core of an open source product the exact same bug may appear in other brand's firmware, would you expect the same from every manufacturer?

I think that AXIS turning around a patched firmware for practically every model with a matter of days shows their commitment to holding themselves at a very high standard.

Let's stick to the announced vulnerability. It will only be applicable to cameras (video devices) directly accessible through the net (internet). The vulnerability will be removed by just upgrading those cameras to the latest firmaware version which is downloadable for free from the axis site. Threfore the job to do is to connect with those cameras (remotely? yes, if they are not accessible there's no risk) and upgrade the firmware. So costs will only be time spending in the process rather than moving to "far away" sites.

Axis has a free tool to do that in a semi automatic way (Axis Camera Management) which can be programmed to upgrade hundreds (thousands) of units in sequence at desired times. That would reduce time spent in the process dramatically.

On the other hand, it's suppossed that the integrator will take care of maintenance (that includes firmware upgrades). Yes, this is not an improvement upgrade for new features, but an upgrade anyway.

Does anybody pay for the upgrade of those cameras which were supplied with default password in the past until most manufacturers found out it was risky and changed the firmware?

It will only be applicable to cameras (video devices) directly accessible through the net (internet).

Just because the integrator does not have remote access to the cameras does not mean there is no remote access. Secondly, a prudent manager is not going to ignore doing the upgrade on such a critical vulnerability just because the cameras are behind a firewall. Mistakes can happen and also someone internally could take advantage of this as well.

I voted yes, but that's just wishing. I wound't expect that to happen. Its like taking a medicine and having it not work and going to the doctor and asking for a a free replacement.

Not really sure if Axis should pay integrators to do this. I have been in the industry over 20 years and I have never seen an Integrator roll trucks to do software or firmware upgrades to products.

As far as this vulnerability we upgraded over 1600 Axis cameras in the past week. We did this with out an integrator. But we always do our own firmware upgrades.

I have been in the industry over 20 years and I have never seen an Integrator roll trucks to do software or firmware upgrades to products.

Do you mean solely to do a software or firmware upgrade? I assume so since it is routine part of support and or maintenance for integrators to come up on site and do upgrades.

If you mean solely, for sure, that is rare. On the other hand, it is also rare to have a critical security vulnerability of this level.

It's the 18th, did Axis ask the researcher for more time?

http://seclists.org/bugtraq/2016/Jul/71

Thanks!

# I don't say that Axis Communication has made this hidden format string by this purpose.
# I can only believe it was a really stupid mistake from Axis side, after I have seen one screen-dump of the CVS
changelog of SSI Daemon,
# and another screen-dump with the change made late 2009, from non-vulnerable to vulnerable, in the affected code of
logerr().
#

Read this IPVM report for free.

This article is part of IPVM's 6,604 reports, 890 tests and is only available to members. To get a one-time preview of our work, enter your work email to access the full article.

Already a member? Login here | Join now

Related Reports

OnTech Smart Services Partners With Google and Amazon To Compete With Integrators on Sep 25, 2020
A pain point for many homeowners to use consumer security and surveillance is...
Free Online NFPA, IBC, and ADA Codes and Standards 2020 on Sep 03, 2020
Finding applicable codes for security work can be a costly task, with printed...
Avigilon Aggressive Trade-In Program Takes Aim At Competitors on Oct 20, 2020
Avigilon has launched one of the most aggressive trade-in programs the video...
AHJ / Authority Having Jurisdiction Tutorial on Aug 06, 2020
One of the most powerful yet often underappreciated characters in all...
Biggest Problems Selling Access Control 2020 on Oct 29, 2020
Access control can cause integrators big headaches. What practical issues do...
Axis License Plate Verifier Tested on Jul 21, 2020
Axis has historically left license plate verification to their partners, but...
Drain Wire For Access Control Reader Tutorial on Sep 23, 2020
An easy-to-miss cabling specification plays a key role in access control, yet...
Dedicated Vs Converged IP Video Networks Statistics 2020 on Sep 10, 2020
Running one's video system on a converged network with other devices can save...
Verkada Access Control Tested on Sep 09, 2020
Verkada raised $80 million earlier in 2020, expanding from video into access...
Ubiquiti Access Control Tested on Oct 21, 2020
Ubiquiti has become one of the most widely used wireless and switch providers...
Seek Scan Thermal Temperature Screening System Tested on May 28, 2020
Now that IPVM has tested Dahua, Hikvision, and Sunell, we are returning to...
Axis Exports To China Police Criticized By Amnesty International on Sep 21, 2020
Axis Communications and other EU surveillance providers are under fire from...
Door Fundamentals For Access Control Guide on Aug 24, 2020
Doors vary greatly in how difficult and costly it is to add electronic access...
Kentix SmartXcan Fever Screening Device Tested on Jun 22, 2020
German manufacturer Kentix has declared "FEVER-SCREENING REINVENTED" with its...
Network Cable Usage Statistics 2020 (Cat 5e vs Cat 6 vs Cat 6a) on Sep 02, 2020
Integrators are split between using Cat 5e, 6, and 6a but 2 of them have...

Recent Reports

Motorola Solutions Total Revenue Down, Video Revenue Up on Oct 30, 2020
Motorola Solutions' total revenue is down, but video (both fixed and...
Recruiters Show 2020 On-Demand Recordings on Oct 30, 2020
Recordings from the 12 recruiter presentations are now available...
Consultants Show 2020 On-Demand Recording on Oct 29, 2020
Recordings from the consultant show are available on-demand at the end of...
Hikvision AcuSense G2 Camera Test on Oct 29, 2020
Hikvision has released their next generation of AcuSense analytic cameras...
Biggest Problems Selling Access Control 2020 on Oct 29, 2020
Access control can cause integrators big headaches. What practical issues do...
Taiwan Geovision AI Analytics and NDAA Examined on Oct 29, 2020
Taiwan manufacturer Geovision's revenue has been falling for years. However,...
Bedside Cough and Sneeze Detector (Sound Intelligence and CLB) on Oct 28, 2020
Coronavirus has increased interest in detecting symptoms such as fever and...
Fever Tablet Thermal Sensors Examined (Melexis) on Oct 28, 2020
Fever tablet suppliers heavily rely on the accuracy and specs of...
Verkada Fires 3 on Oct 28, 2020
Verkada has fired three employees over an incident where female colleagues...
Eagle Eye Networks Raises $40 Million on Oct 27, 2020
Eagle Eye has raised $40 million aiming to "reinvent video...
Hikvision Q3 2020 Global Revenue Rises, US Revenue Falls on Oct 27, 2020
While Hikvision's global revenue rises driven by domestic recovery, its US...
VICE Investigates Verkada's Harassing "RawVerkadawgz" on Oct 26, 2020
This month, IPVM investigated Verkada's sexism, discrimination, and cultural...
Six Flags' FDA Violating Outdoor Dahua Fever Cameras on Oct 26, 2020
As Six Flags scrambled to reopen parks amid plummeting revenues caused by the...
ISC Brasil Digital Experience 2020 Report on Oct 23, 2020
ISC Brasil 2020 rebranded itself to ISC Digital Experience and, like its...
Top Video Surveillance Service Call Problems 2020 on Oct 23, 2020
3 primary and 4 secondary issues stood out as causing the most problems when...