10,500 Accounts Down - 8 Weeks To Fix

Published Feb 09, 2017 21:42 PM

At the Barnes Buchanan Conference, Alarms Detections Systems explained that 10,500 of their accounts, consisting of half of their RMR, went down and took 8 weeks to fix.

This chart shows the start of the outage:

** *** * ************ ************** ******* **** could *** ** ***** ********. *******, and ******* *********, *** ** ** to **** ** *** **,*** ******** down:

*** ********* **** ******** *******, ********, nursing *****, ***. *** **** ******** fire ******. **** *** *** ********* for *** ****.

Product ******

*** **** *** **** **** ** go ** ****, **** ****** ** replace *** ******* / ***********. *****, not ****** ******** **** ********* **** the ************ ** *** *** *,*** of *** ******* ******* **** ***** dealers ** ****.

****, ** *** ********, ******** **** deployed ****** *** ****** ** * temporary ********.

** *** *****, *,*** ** *** units **** **** *** ************ *** not **** ******, ** *** ****** staff ***** * ***** *** ******* to **** ***** ** *** ******:

*******, * ***** *****, ** ********, they **** ****.

****

** ******** *** **** **** *** fixing **** ** ** * *** hundred ********* ** *******, ***** *** 8 ***** ** * *** ********* of ~** *********** ** ** ** hours * ***, ********* ********. *** that ** **** ******* *** ***** cost ** **** ******** *** ********** of ******* ** *** ********, ***.

Manufacturer ******** ***********

*** ******** ** ********* *** *** manufacturer *** ****** ******* **** **** that ************. *******, ***** ****** ********** ***-**********, ****** * ****** **** *** manufacturer:

* ****** *********, ***** ********* ** the **** ***.**** *** ********* ** our ********’* ***-*****************, ******* *** *****’* ******* ** that ******** **** ** ******** **** un-decodable **** *******. ** ** **** that **** ********* *** **** ************ over **** **** ********* *** **** the ***** ***** ****, ** ******* those ************* ***** **** ****** ******** to **** ****** *** ******** **** occurred.

**** **** **** ***** ****** *** a ************ ******** ** ******, ******-***** radio ********* ************ (***) ** *** same ******* **** ** ********. ***** have **** ******* ******* *** ****** noted ** *** **** ******* **** over ****** ******. **** *** **** sufficient ** ********* ** ***-*****************. ******* *** *** *** ********* in *** **** **** ********** *** concerns ** *** **** **** *** of ***** *** ****** ***** **** most ****** ***** *****.

** **** ** *********** ** *** what ********** ** ************ *** ************ will make ** *** *** ****.

Comments (18)
Avatar
Brandon Knutson
Feb 09, 2017
IPVMU Certified

2G dialer no-longer-supported issue?

Avatar
Ethan Ace
Feb 09, 2017

It was an AES-Intellinet issue. I'm trying to find some related articles about it now and we'll update.

JH
John Honovich
Feb 09, 2017
IPVM

Just found an article in SSI from a few months ago confirming AES. Given that, strange that ADS would not disclose the manufacturer.

(2)
MC
Marty Calhoun
Feb 09, 2017
IPVMU Certified

Exactly what I was thinking, AES had to be them.Only long range wireless that I considered worth a hoot in hell, but now??

Avatar
Joel Kriener
Feb 09, 2017
IPVMU Certified

Very interested in hearing more background on this .....

Avatar
Paul Curran
Feb 09, 2017

Seems to be caused by a rogue French satellite. 

(1)
EP
Eddie Perry
Feb 10, 2017

no it was totally the Russians, they made the French satellite go rouge

(4)
(6)
Ml
Mendy lewis
Feb 10, 2017
Safezone24

What wen wrong with the radio frequency, is it the alarm communicator with central station , or the wireless alarm system?

Avatar
Ethan Ace
Feb 10, 2017

AES-Intellinet is an alternative to telephone, cellular, or IP dialers. It uses spread spectrum, 450-470 Mhz radios, which act as a mesh. So if you have customers clustered in one area, they all act as transmitter and receiver, til the signal riches an uplink, which is usually IP to the central station.

That frequency is licensed in the United States, so nothing else should have been transmitting on it, but according the AES' investigation, a satellite malfunctioned and sent a packet which was received and rebroadcast by the network. Instead of ignoring it, essentially the radios just didn't know what to do with it, so they kept trying to rebroadcast it until they finally went down. 

SSI has a followup article with AES in which they talk about what happened/how they're preventing it from happening again, etc.

Avatar
Paul Curran
Feb 10, 2017

I wonder why the network wasn't encrypted? How could a external packet drop into the network and what's the stop anyone land based doing this? 

Avatar
Ethan Ace
Feb 10, 2017

According to AES, it was a combination of that rogue packet and some ground based interference. 

I'm not an RF genius and I'm not that well versed on AES' product, but...I'd tend to agree with you. I don't get a lot of warm fuzzies about it not being possible.

(1)
UE
Undisclosed End User #1
Feb 10, 2017

You cannot even imagen how much stuff, that is "certified" by various manufacturers and agencies who is using ISM band (433 MHz), for instance burglar alarms that are totally B.S, so many of these system that can easily be "disabled" by using using one simple handheld radio and transmitting on the right (or near) frequency...

(1)
UE
Undisclosed End User #1
Feb 12, 2017

The french satellite? http://www.hydrowise.com.cn/images/images/Argos3PMT-KHD.pdf

AES-IntelliNet Data Radio: https://fccid.io/L9N-7085UE5

I tend to believe it was a combination (or only #1)

#1. Satellite blocked the Data Radios receiver (when passing over).

#2. "Ground station" made reverse engineering of the traffic (most likely clear text), injected an "packet" that caused "broadcast storm" and overloaded the [low speed] network. (I assume it is Mesh as it says here, otherwise this will can go out from this hypothesis)

 

(1)
(1)
JH
John Honovich
Feb 13, 2017
IPVM

#2. "Ground station" made reverse engineering of the traffic (most likely clear text), injected an "packet" that caused "broadcast storm" and overloaded the [low speed] network.

So do you think this was malicious then or?

(1)
UE
Undisclosed End User #1
Feb 13, 2017

Frankly speaking, no.

I did only take the "report" in consideration, and besides, I have no clue how the "network" works, could be "Mesh" AKA repeating of packets - I have no idea how the protocol they are using works. (would be interesting to know tough)

The Radio is simple analogue stuff (as can been seen on FCC site), nothing special, this leave me personally to believe they has interference from the Satellite alone, and there was no "packet" was entered into their "network". And that would also make sense with their "temporary" vehicles all around, only to pickup the radio traffic from the customers and relay in. (with 3G / GSM?)

(Depending what's behind the radios (on the serial port), it could be possible to relay packets (most probably not even IP), but personally I don't think so)

Anyhow, that's the price to pay when using license free frequencies AKA ISM band for pretty serious applications. - Maybe embarrassing enough to make up a story to justify the change... 

 

Avatar
Robert Baxter
Feb 14, 2017

The issues raised are well known in the Central Station Community. There were actually two different situations. Issue 1 was from a few years ago: The discussion about the satellite interference was related to tracking of marine life fitted with communicators that happened to be on the same frequency as the one FCC granted to ADS. The french satellite did not have approval to use the frequency. 

Issue #2 is a recent incident that occurred around Sept of last year. This is what I have heard was the cause - there was a faulty packet of data introduced by one of the radios that was being distributed throughout the network that caused the receivers to not respond or rebroadcast. The network was very large and should have been isolated onto a few different frequencies, also a number of the radios were very old and did not have up to date firmware.

(1)
UE
Undisclosed End User #1
Feb 14, 2017

#1

"The french satellite did not have approval to use the frequency."

Well, the frequency seems to me (to what I have been reading) is free frequency to use for anyone, i.e. license to use this frequency are not required.

#2

I have been thinking about the "faulty packet", and personally i cannot find out how the "faulty packet" can enter one another "network" to be "rouge" packet that would put the "legal" network in faulty state.

What I can only think of, is simply "RF choke" of the central receiver, since this is listening on the frequency that the "rouge" french satellite transmitting on.

My personal conclusion

 Very good example on how manufactures think they can use free frequencies for implementing pretty serious applications without any drawbacks. The frequency is free to use for anyone, and anyone who will implement anything on these frequencies must be aware of the highly risk to have another applications on same frequency.

I personally know several cases when "the only legal" application is disturbed by a nearby hamradio repeater, who is transmitting 50 watt on near frequency and locking out AFSK/FSK receivers with totally lacking of any input filters. (of course, the "legal" application is the hamradio repeater and the ISM stuff, but there is absolutely nothing that can be said "I have priority of this frequency" - but the power of 50watt of registered repeater beats easily all other ISM stuff with lack of proper bandpass filter) 

As I said before, this is the price to pay to use free frequencies for pretty serious applications...

 

Avatar
Robert Baxter
Feb 15, 2017

"The frequency is free to use for anyone, and anyone who will implement anything on these frequencies must be aware of the highly risk to have another applications on same frequency."

This is not a Wi-Fi frequency. The frequencies in these bands are not free, it is regulated, meaning it is assigned to users. Users of the AES-Intellinet system have to get approval from the FCC or Industry Canada (or both if near the border) and pay annually for the band requested and then assigned to them and the assigned geographical area they can operate their transmitters by the regulators. 

(1)