Subscriber Discussion

Axis/Exacqvision Problems (Encoders Mostly) Freeze Or Go Offline, Requiring Reboots And Firmware Upgrades To Stay Online

Avatar
Jeffrey Hinckley
Oct 17, 2016

Has anyone been seeing Axis devices (encoders mostly) freeze or go offline, requiring reboots and firmware upgrades to stay online. 

In 8+ years using Axis encoders with Exacqvision, I have never seen the frequency of problems we have been seeing the last 3-4 months (mostly M7014).  In these cases, there have been no recent upgrades (Exacqvision Server or Axis firmware) or configuration changes. 

 

 

UI
Undisclosed Integrator #1
Oct 17, 2016

I have seen this issue. I only have a few of them out there but one has gone offline multiple times. That is a serious problem.

Avatar
Jeffrey Hinckley
Oct 24, 2016

We have encountered additional examples of this problem in the last week. This not only included M7014 encoders (only Axis product we sell) but other Axis cameras from previous installers. I can only conclude that this has to be from a malware/botnet situation such as that which we have been reading about recently. This probably takes advantage of the security vulnerability which IPVM reported on months ago.

Axis Critical Security Vulnerability

Do you think I am jumping to conclusions? Not likely.

  • All instances of this occur at sites where the cameras are on the network connected to other client computers (PC) which have gateway access to the Internet. (The theory is that one of these is compromised and is scanning the network for eligible devices). Sites where the cameras are segregated, with no gateway and network access to PCs are not affected. I am assuming that, if this is the case (malware/botnet), a PC on the network (which one) had a user clicking a "link" which caused the infection.
  • First site we saw this at was in late July. Two M7014 locked up soon after an install. Symptoms are that you can ping the device, but web access is restricted. Rebooting device brings it back online, but it is compromised soon after (the malware is at work). We do change passwords, but with the reported vulnerability, this does not matter. Firmware update corrected this. (I assume that Axis corrected the flaw on new firmware).
  • Since then, it is weekly that we see Axis devices exhibiting the same. This has been accelerating and now we have seen this at about 10-15 customer sites. The last one had 300 devices on the subnet/VLAN, with the only Axis devices being 20 241Q and 3 M7104. All of the M7014 devices locked up within a 1 week period. There have been no updates/changes on this system in the last year. In all cases, firmware updates seem to have corrected.
  • We have been questioning a system we took over with Axis P1346 and Q series PTZ cameras which have been going down in succession the last couple of months. That system has been running with no updates for 6 years. They wanted a quote for replacement cameras, and I had to go myself to troubleshoot. These cameras should not be failing like this. I have never seen Axis cameras or devices fail like this in the past. Reboots corrected but soon they were locked up again. I brought one back to office, updated firmware, and works great.
  • In the last week (since I started this post) I have had 6+ customer sites exhibit the same. When I get an email about camera outages, I do not even have to research. I know the manufacturer is Axis. Sure enough, it is. Older devices (241Q, M3028, etc) do not seem to be affected. As I said before, I have not sold many Axis cameras over the years, only a stint in 2010 when the M3 series cameras came out. We have standardized on the encoders though.
  • On one site, it was apparent that, if this was the case, the PC would have had to push/scan/infect over an internal router. This means that the problem would not be only affected on the existing subnet, but the PC/malware is able to scan through all routers to different subnets.
  • I know in the white paper (Axis), they indicate that this vulnerability is only valid with devices connected via DMZ or direct connect to the Internet. I do not see why a hacker, however, would not go after the "Microsoft" of the camera world with their malware, especially after publicized information regarding the security breach. They can probably quickly update infected computers to include this device in their "Dahua buster" software.

If anybody has any input on how to 1) analyze to find out what PC may be infected or 2) test the camera to see if infected (is it removed with a reboot, or is the firmware affected), I would appreciate it.

If anybody wants to help me disprove this, that would be great too. (I need to alert customers ASAP if malware is the problem). I find it hard to believe that I am the only one seeing this.

JH
John Honovich
Oct 24, 2016
IPVM

Jeffrey,

To clarify, all the Axis units upgraded to current firmware no longer have issues? Have you upgraded all or?

Avatar
Jeffrey Hinckley
Oct 25, 2016

All devices upgraded to newest firmware no longer have issues. I assume this addresses the security issues reported in July.

Hey, if I was a hacker, and a public report such as that came out with how to do it, I would probably sent it to my malware minions.

JH
John Honovich
Oct 25, 2016
IPVM

Jeffrey,

I just forwarded this to the Axis HQ team that dealt with the critical security vulnerability asking them for feedback. Will update when I receive.

JH
John Honovich
Oct 25, 2016
IPVM

Axis HQ said to file this to tech support and note your suspicion of possible breached cameras.

Avatar
Jeffrey Hinckley
Oct 25, 2016

Just to clarfy, we originally thought it may be Exacq with later updates. We have seen this with Pelco and mpeg4 devices. We started to realize that this was affecting Axis/Exacq systems with no updates, but have been running fine for 1 plus years (last problem was 7.4).

I need to prove this, and the public announcement in July should be too juicy a temptation for these hackers with infected computers controlled on the inside.

I have been seeing the recent newscasts about ip devices. People have to realize that it is not the ip devices, but the person that clicked on the link or opened the file that created the inside access for the hackers. Common users with admin control of their workstations, especially in commercial and municipal settings, is mind blowing. I am always amazed when I am able to install client software on a secretaries computer. When I was an IT admin, this would never happen. My kids hated having to get me to grant them permission to install apps or software on the devices/pcs i gave them. Now they have school computers where they can install anything they want. (thank god for opendns, until they figure out how to put a static 8.8.8.8 on their devices).

New discussion

Ask questions and get answers to your physical security questions from IPVM team members and fellow subscribers.

Newest discussions