Subscriber Discussion

Recent Netgear Vuezone Outage

SM
Steve Mitchell
May 14, 2015

Back on May 8 I received a message from Netgear VueZone support apologizing for an ongoing outage with their service that apparently prevented the ability to retrieve/view video 'clips.' I do not have an active account, but am probably listed as a customer from before Netgear acquired Avaak:

We apologize for the continued issue, impacting your ability to retrieve and view video clips. Due to the complexity of the issue, it has taken us longer than anticipated to troubleshoot. We have identified the root cause and are in the process of testing the fix. We expect testing to continue for 2-3 more days. Once completely resolved, we will send you a follow up email confirming the issue has been fixed. Again, we sincerely apologize for this inconvenience.

Thank you,
The NETGEAR VueZone Team

Then May 10 a follow-up that said the issue was resolved, but required an update to their apps:

The issue that has impacted the ability to view videos from the VueZone library is now resolved.

The complexity of the problem required us to perform a substantial amount of testing to ensure that the issue was adequately resolved and that no other areas of the system were negatively impacted. Resolution of the issue also required changes to both the Android and ios apps. Please take a moment to download the latest apps.

Then on May 11, another notice:

As you may be aware, VueZone services have not been available since approximately 10:00am (PST) this morning. We sincerely apologize for the inconvenience and appreciate your patience.

The root cause has been identified and our Engineering team is in the process of developing a code fix. We will send you an update once we have an estimated time when all services will be restored.

Then a whopping three days later May 13:

We apologize for the continued disruption of your VueZone service. The outage is an outcome of a rare occurrence which resulted in a complex issue that is requiring extensive Engineering and Quality Assurance efforts to resolve. We value the trust you place in us and take our operations responsibilities very seriously. We sincerely apologize for the inconvenience this has caused and for the extended length of this outage.

And finally this morning (May 14):

VueZone services are again available. Our sincerest apologies for the length of service interruption and for any inconvenience this may have caused.

We want to assure you that the length of downtime in no way reflects a lack of focus or Operational support for VueZone. Our teams have been working around the clock to resolve the problem. Unfortunately, the extent of this issue impacted many areas of the VueZone system which required extensive Engineering and Quality Assurance efforts to resolve. And although we are confident that the condition which caused the issue was a rare event, we are performing a full system audit in an effort to avoid similar potential issues in the future.

Outages and service interruptions strike the biggest and best of the cloud service providers (Google, Facebook, etc), but 7 days is a very long time for issues to persist on any professionally run, modern service.

Obviously this illustrates a risk of VSaaS: if the service provider's having problems, every one of their customers are having problems.

Caveat emptor.

(1)
JH
John Honovich
May 14, 2015
IPVM

I got those emails as well, and was confused but primarily ignored them because I have not used Zuevone for years.

I agree with the lesson and also why I tell integrators who think about hosting video themselves in their offices or basements or the data center nearby not to do it.

(2)
New discussion

Ask questions and get answers to your physical security questions from IPVM team members and fellow subscribers.

Newest discussions