Axis Suffers Outage, Provides Postmortem

By Ethan Ace and John Honovich, Published Aug 15, 2019, 09:36am EDT (Info+)

This week, Axis suffered an outage impacting their website and cloud services.

axis outage_2

Inside this note, we examined what happened, what was impacted and what response Axis has provided, contrasting it to recent outages for Alarm.com and Hikvision.

** *******, ****** ****, **** ******* started ** **** ** ***** *******:

************, ********** **** ** ****** *****, *.*.:

***** ***** ******** **** ******** ** well ******* **** *** ** **** to *****. ***** ********** ****** **** cache, *** **** **** **** ** seconds *** **. ******* ***** **** a **** **** ** ****, *** eventually ********, ** ************ *****:

*** ****** ****** ******* * *****.

**********

**** ******** *** ********* ********** ** IPVM:

******* ******* ** *** * ******** network ******* ** *** ** *** sites. *** ******** ********* *** **** performing **** ******** ****** ******* *** stopped ******* ** * *** **** should ***** ** ********. *** ****** is ******** *** ******* ******, ******* performance *** ******** **********. ************, ********** went **** ******* ******* ** ************* 18:45 ***.******** **** *** ******* ***** *** the **** ******/******* ******** ** ******* working ** ** ******** *** ***. Given *** **** ** *** **** and **********, **** **** ***** ************* 21:00 *** ******* *******, ********* ******* within **** *** ****** ******* **** our ********, **** ** ******** *** remotely **** ***.*** **** ** ******* ********** ** working ********** ******* ** **** ** we *** ** ************* ** *** cause. ** ************* **:** ***, *** core ******* *** ******** *** ** could ***** ******** ******* ** *****. As ** **** ******* ****** **:** CET, ** **** **** ** **** back ** **** ******** **** * few ********* ******. ** ********* (*****) 12:00 ***, *** ***** ****** *** been ******** *** ********** **** ** normal.*** ******** **** ***** ******** **** be ********* ** *** ********’* ****; the ******** **** **** **** **** and ** *** ********** **** *******. All ******* **** **** ******** *** complete **********. ******* ****, ** *** problems, ***** ** ****** ** * humbling **********. ** **** ******* *** design, ******** **** *** ********, *** make ********* *******.

** ***** ***** ********** *** **** elaborated:

*** ***** ****** ** ***, ** have ***** **** ********** *****. *** slightly ****** ****** ** **** *** multi-site ***** ** *** *** ***** uses * ****** ******* *** **** all ***** ****** **** ** ****** to * ******* ******* *******. ****** by ******* * ***** ***** ****** or ** ******** *** ***** *** renewing *** *******. ** *** ********* roadmap *** *** ***, *** ***** caches **** ** **** ** ** more **********, **** **** **** ******* content. ** ** * ******* ******* ensuring **** ******* ** ********* *** current. ******** ** ***, ** **** look **** **** ******* *****, *** we *** ********** ****** ** ******* even ******* ********** **** ** *** most ****** **********.

*********** (*** **** ****), ** **** ** ****** ****** notifications, *** *** **** ** **, though ** ***** ************ *** ******** they *** **** ******.

Alarm.com *** ********* *******

**** ** *** *** ****** (** least **** ** **** ********** ** the **** *** ******). *****.*** *** the **** ******* ***, ********* ******** of ********'* ******* **** * *****. Hikvision *** *** ******* *** ********* one ** ********* *****'* ******* *** down ** ***** * ****, *** Hikvision *** *** **** ************, *** hours ** * ****, **** * month.

** *** *, **** *** *** only *** *** ******** ****** ****** while ** *** ********* *** ******* any **********.

** *** *** *** ****** ******** become ************ ******** *** **********, *** ability ** ************ ** **** ***** services ****** *** ******** ****** ** issues *** *********** **** ** ** increasing **************.

Comments (4)

What is this amateur hour? I like Axis, but they have solutions for this very problem without searching, Apache Ignite (and Gridgain), Hazelcast, Terracotta, and Infinispan come to mind all provide In Memory Data Grids that are designed to be fault tolerant (and able to work with other systems like Etcd, ZooKeeper etc), atomic, with tunable consistency, etc and can be written to disk if "ram is too expensive"

Agree: 4
Disagree: 1
Informative: 4
Unhelpful: 2
Funny

I was trying to download cut sheets from Axis for submittals it took some extra searching on suppliers websites to get everything I Needed and cost we 45 min or so of time

Agree
Disagree
Informative: 1
Unhelpful
Funny: 4

Crappy cloud. Just thankful none of our customers want cloud access and demand we use this "reliable" service.

Agree: 1
Disagree
Informative
Unhelpful
Funny: 1

Did I miss something or did they just say...

"Our site went down, the switches were horked, our alleged redundancy had a single point of failure, and we are throwing the switch vendor under the bus even though our non-spectacular website apparently has an excessively complex configuration."

Agree: 2
Disagree
Informative
Unhelpful
Funny: 2
Login to read this IPVM report.
Why do I need to log in?
IPVM conducts reporting, tutorials and software funded by subscriber's payments enabling us to offer the most independent, accurate and in-depth information.
Loading Related Reports