Verkada Notification Outage

By: Sean Patton, Published on Dec 12, 2019

Verkada is suffering an event notification outage and analytic search failures.

Inside, we examine what the issues are, what Verkada told IPVM and the risks of these outages for VSaaS users.

******* ** ********* ** event ************ ****** *** analytic ****** ********.

******, ** ******* **** the ****** ***, **** Verkada **** **** *** the ***** ** ***** outages *** ***** *****.

[***************]

Update **/** - ***** *** *** ************* ********* / ** ******

** ** *** ******* of **/** ***** *** SMS ************* *** ****** have **** ******** **** our *******. ****** *** notifications *** ***** *** working. ** ******* *** to ******* *** ********** feedback *** **** ****** the ****** **** ** is ********.

************, ****** ****** ** also ******* ********, ******* for *** ***(*) **** it *** *** ***********, people ********* ****** *** still *** *****.

Outage ********

*************, *** *****, *** and ****** ***, *** all *** *******, ********** users **** ******* ****** about ********** *** ******** risks.

**** ********* *** ****** has ****** ** ***** 1 **** *** ** the **** ** **** publication. ******* ******* *** still ********* ** *******'* servers *** ********* ** functioning ********, ** ********* notifications ** *** ******** were **** ** *******.

*******, **** ** ********* Verkada ******* ** **/** at * ** **, they ********* **** **** aware ** *** ***** and **** ********** * fix:

******* *** ****** ** provide ** *** *** the ***, *** ***** it *** *** *** priority *** ***********:

*************, *** ******* ***** noted **** ****** *** issue ******* ******* **** are ***** ******* ******* in ***** *****:

** ******** * ************* at ~* ** **, by *** *** *****, but **** *** ******** any *****. ******* ****** App ************* *** **** not *******.

****** **** ***** ** the ***** *** *** trigger *** ******:

No ******* ************

******* *** *** **** email ************* ** ********/*** users ** ****** * blog ***** ***** *** outage ** ** *** morning ** **/**, ~** hours ***** **** ****** aware ** **. ******* this *******.*** ******* ** ***** to ******** *.* ***** after ***** ****** *******(* **** ***** ** was ********).

*** ********** ***** *****.*** experienced * **** ****** outage, ***** ******* ** suffering * ******* ****** of ***** ************* *** analytic ********.

** ***** ******* **** the ***** ** *** issue *** **** **** not ******** * ***** response. ** **** ****** with ***************** **** ******* ***, which ** ****** ** the ** ********.

Cloud - ***** ************ *****

*** ************ ****** ***** cause ***** ** **** critical ****** (****** ** vehicles ** ***** ********), especially ** ****** ***** or ***** ****** ******** hours **** ** *** may ** ******** ********** their *******. **** ** especially ********* *** ***** users *** **** ***** ** *******'* *** 5 ********, "***** ******** *** Alerts", **:

***** ***** ** ******** situational ********* ******* ****** to *********** ******* ************ around *** *****. **** notifications **** *********** ***** stakeholders ** ******* **********, such ** * ****** detected ** *** ******** of * ****** ***** hours, *** ********** ** on-site ********* ** ******* or ********** **********.

**** ** *** *** same ***** ** *** loss ** *** ******** at * ********, ***** could ********* ****** **** traditional ***/*** *******. ******** events *** ****** *** very ***** *** ******** that ********* *** ************* have ********* *** *******, and *** ********* ******** are *** *********** ** complicated.

Person ********* ****** ******

** ******** ** *** notification *******, ****** ****** is **** *** *********** properly. ***** ** *** basic ******** ******* ******, people *** ******* *** highlighted ** *** ******, switching ** ****** ****** returns "** ****** ******** within **** **** *****":

******* **** **** ***** was ******* ** *** camera *** ******* * fix ********, ******* ** of *** **** ** publishing, *** ***** ***** persists.

Cloud ** **-**** *******

******* ********* ******* ***** the ******** ** ******** outages *** ****** **** come **** ************ *************. And **** *** ********* right **** ***** *** such *****.

** *** ***** ****, as **** ******* ******* notification ****** *****, ***** are **** ***** **** one's ***** ******** *** problems **** ***** **** to ****** **** *** hope **** ** ******** as ******* ** ********.

Comments (36)

Undisclosed #1

12/12/19 04:04pm

** ******** ** *** notification *******, ****** ****** is **** *** *********** properly.

** *** ****** ** anything ***** **** **** notifications ** ******? **** what ** ******* ** the ******, **** ****** as ** ** *** be ********* **** **** just "*************", *** *** be ********* **** **** of *** ********* ******* in *** ***** ** well.

** **** ** *********** to *** ** *** search ******** ***** ***** things *** ********, ************ on ***** ******** ****** the **** ** *** outage.

Avatar

Sean Patton

IPVM | In reply to Undisclosed #1 | 12/12/19 04:17pm

****** ************* ******* ********* restored *** ******** ****** today, *** *** *** some ********** ******** *****.

Avatar

Sean Patton

IPVM | In reply to Undisclosed #1 | 12/12/19 04:20pm

*****, *** *** ***** question, *** ****** *** email/SMS/app ************* ** ******** events (******, ********) ***** on * **** ******* schedule (******, ******** *****, after *****, ***). *** analytics **** ** ** working ********, ** *** people *** *********** **** bounding ***** ** *** timeline, *** *** ****** results **** *** **********.

Undisclosed End User #2

12/12/19 04:06pm

***** *** *** ********** to ****** ** *** cons ** *** ******* technologies. ****** **** **** didn't *****.

Undisclosed Manufacturer #3

12/12/19 04:13pm

** ***** ** ***** beneficial ** **** **** this ******* ******.

Undisclosed Manufacturer #4

12/12/19 04:21pm

** ** ************ **** any ********** ***** *** a ***** ********** ******** for ********* *** ****** or * ******** ***-***** interface ****** ********. **** is *** ** **** ways ******** ****** *** go *********.

* ****** **** ****** on *** ********** *** it ******* ***** **** useless ** **** *** using ******* *** *** like. **********, ******** ** foreign, **** **** * field *** **** ******* who ****** ***** *******, just **** **** ****** anything **** ******** ***************.

*** **** ***** ****** will *** **** ******** of ****** *** ****** vulnerable.

Undisclosed #1

12/12/19 05:55pm

** ** ************ **** any ********** ***** *** a ***** ********** ******** for ********* *** ****** or * ******** ***-***** interface ****** ********.

"************", ** *** ** all. ******** **** ***** payroll, ***, ***, *****, and ***** ******** ********* are *** ****** ******** outsourced ** *****-****** *********. Why ***** ** ** "unbelievable" **** ***** ******** cameras ***** ** *** different?

** ***** **************** **** ** *** cloud-based *******, ** ***** short-sighted ****, *** ** you ***** ***** ** is "************" **** *** as * ************ *** in *** * **** rough **** *****.

Undisclosed Integrator #5

In reply to Undisclosed #1 | 12/12/19 06:04pm

* *** ******** ** watching ****** ** ******* for ***** **** "****** 365/Google ***** ** ****, guess *'** ** ****!" It ******* **** ***** than ***'* *****, *** a *** ** ************* seem ** ** **** with **.

Undisclosed Manufacturer #4

In reply to Undisclosed #1 | 12/12/19 07:05pm

** ** "***********" *** you ** ***** **** losing *****, *******, ***, etc ***** ****** ** loss ** **** **** losing * ************ ****** using ********* *** ******** areas *** *** **** you ****** *** ** would ** *** *********. Or ***** "************", ********* on **** **** **** in *** ******** **.

***'* **** ********* *******, fire ***** **** ********, and ***** ********* **** to *** ******* *** cloud, ***. ***'* **** access ******* ******* **** to ** ** *** cloud *****, ****** ******* a **** *** ********* egress. **'* *** *** same ** ***** *** CRM.

******** ************ ***** *** (or **** *** ***) for ******** *** * very ***** **** *****.

Undisclosed #1

In reply to Undisclosed Manufacturer #4 | 12/12/19 07:08pm

** ** "***********" *** you ** ***** **** losing *****, *******, ***, etc ***** ****** ** loss ** **** **** losing * ************ ****** using ********* *** ******** areas *** *** **** you ****** *** ** would ** *** *********.

* ***** *** ****** my *****. * ***** that *** *** ***** of ********* ******** ****** not ** ******** ** the **** ***, *** I *** ******* *** how *** *** ***-***** are ***** ******* ****. Calling ** "************" ******* that *** *** *** understanding *** ****** ****** of *** ******, *** are ******** ** ***** accordingly, ***.

Undisclosed Manufacturer #6

In reply to Undisclosed #1 | 12/12/19 08:08pm

** *** **** ** minimize ** ******* ********** demands *** *** ********* costs ** *** ****** that *** ****** ** manage *** ********* ** networked *******, **** ******* cloud *** ****** ************ operating *****.

** *** **** ** own *** ******* ** your ** *** *******, data *********, ********, **********, fault *********, **-****, ***, to ******** ****, *** have ** *** *** price *** ** ** in-house.

****'* ** ****-************** ** two ********* ** ******, but **** ** *** essense ** **. *** lesser ** *** ***** or *** ******* ** two *****.

***** *********** **** **** it's *** *** ** risks ** ******** *** the *********** ********* ** mitigate ***** *****.

** *** ********* ***** as * **** ****** technology, ** ** ** house ** ** ******** to ****** *** ************ of ********** ** ****** (the *****). ** *** does *** ******** ***** as * **** ****** technology, ***** *** **** sense.

** ****** *****'* **** more ** **** ** each ************ ***, *****, I ***** ****'* *** essence ** **.

Undisclosed Manufacturer #4

In reply to Undisclosed #1 | 12/12/19 09:56pm

***, **** ***** *** ill-communicated. * ***** **** that ****** *** *** a ****** ** **** industry ***** **** ********** and *********** ** * huge ****** ****** ** IT. * ****'* ***** I *** ** ***** the *******. **** ** unbelievable ** *** **** that **** ****** ****** has ******* ******** **** reliability ** * ****** used ** ******* ***** and ******. **** ** ERM ***** ** ****, "unadvisable" ** ** ********** adjective ** ******** *** ignorant *** ********* **** is, ***.

*** ******* ** **** executives\board ******* ***'* ********** technology **** * **** perspective ** ******** ** says, *******. ** *** cloud ******** ***** ** that ***** ***** **** be ****** ******* *********** logical ********, ** **** for ** ******* ******** about *** ********** ** physical ********.

Undisclosed Manufacturer #4

In reply to Undisclosed Manufacturer #4 | 12/13/19 06:58am

***, * ***** ***** based ******** *** *** time, **** *********** ** the **** *****.

Undisclosed Manufacturer #6

In reply to Undisclosed Manufacturer #4 | 12/12/19 10:17pm

**** * **** **** I *** *** ******** in *** ************ ** undesirable *********, **** ****** to **** * **** to ******** *** ************ and ************* ** *** given ******** ** *** risks *** ********** ** those ***** ** **** particular ***********.

Undisclosed #1

In reply to Undisclosed Manufacturer #4 | 12/12/19 10:52pm

***, **** ***** *** ill-communicated.

*** **** ******** **** was * ***** ** vague *********** ******** ** with ************ * *****-***** botnet **** ***** **** corporations *** ***** *****-********* surveillance *******.

**** ** *** *********** I **** **** **** their *** **** **** multiple ********* *** *** not *********** ** * "simple" **** ******. ****, Verkada ****** **** ***** viewing, ***** **** *** have * ***** **********.

**** ** ************ ** the **** **** **** buying ****** *** ******* priority **** *********** ** a ****** **** ** protect ***** *** ******.

******** ******. **** "*****" are *** *********** **** show ****** ****** ***************** *** *********** ** the ******?

Avatar

Sean Patton

IPVM | 12/13/19 04:00pm

Update **/** - ***** *** *** ************* ********* / ** ******

** ** *** ******* of **/** ***** *** SMS ************* *** ****** have **** ******** **** our *******. ****** *** notifications *** ***** *** working. ** ******* *** to ******* *** ********** feedback *** **** ****** the ****** **** ** is ********.

************, ****** ****** ** also ******* ********, ******* for *** ***(*) **** it *** *** ***********, people ********* ****** *** still *** *****.

Undisclosed #1

In reply to Sean Patton | 12/13/19 04:07pm

* ** **** ******* to *** *** ******* is ***** ** ******* the ****-****** ** ****.

********* **** ******** *** events ** ********* ****: Person ******** --> ****(*) matched --> ***** **** logged ** ******** --> Event **** ********** ** notification *******.

* **** ** ********** that *** ***** ************ stack ***** ** ** somehow ********** **** *** rules *****, *** ***** separate **** *** ****** detection **********.

**** **** **** *** been ******, ** ***** more **** * ******* in *** ***** ****** that *** *** *********** events *** ******* *** event *** ** *** notification **********, *** ** course * ***** *** say *** ****.

** ** **** **** common ** *** * third ***** ******* *** the ************ ******** (**:******), ** ***** ** interesting ** **** ** Verkada ** ******** ** have ****** (** ******) their *** **** ************ stack.

John Honovich

IPVM | 12/13/19 11:54pm

******* *** * *** funny, ********* ******* *** not **** * ** team ***** ********, **** a**** ********* ******* *********:

** ******* ***** ** a ***-****** ***** **** before ****** * ** team.

Avatar

Sergey Bystrov

NetworkOptix | In reply to John Honovich | 12/18/19 11:58pm

***** * *** ** people **** ** *****, sometimes **'* ** *** to **** * **.
***** *** * **** driven ********** **********
**** ****** *********** - Google ******

******* ********, **** ** 100% ******* **** *** autotests, ***** ** **** more ******** *** ********* than ****** **. ** should **** ** **** if ********* ***** *** be *********.

John Honovich

IPVM | In reply to Sergey Bystrov | 12/19/19 12:16am

******, **** ******* ***** have * ** ****? Why ** *** ***?

Avatar

Sergey Bystrov

NetworkOptix | In reply to John Honovich | 12/19/19 12:44am

** **. *** * few ******* *** * cannot *** ***** *** is *** ***** *** more ********* ***, * just **** **** ***. And **'* *** **** binary...

*) ** *******. ***** TDD ** **** **** efficient, ********** ** * startup ** ****** ** ship *** **** **** and **** **** ** manually ****** ** ******** with ***********. ****** ***** term ****, *** **** term **** ****. ** we ***** ** ** again, ** ***** **** a **** ******* **.

*) ** *** ****** client *** **** ** the ****, ** *** much ****** ** ********.

*) *****, ***** *** things ***** ****** ** is **** *********(********** ***** term).

*) ** *** ************, QA **** ***** *** role ** ********* *******. They **** *** ********* to ****** ********** ***** and ****** ********** ***** to **** ****** **** simple *** *****.

***** **** * *** of *** ******** ** members ******** **** ****** QA ** **** *******.

*** ****** * **** some *********/******** ** *** use ****** ** ** all. * ******* ***** would ** ** ******* form *** ********. **'* totally ****, **** *********.

John Honovich

IPVM | In reply to Sergey Bystrov | 12/19/19 01:37am

**** ******** ** ********* reasonable. *** "********* **'* ** *** to **** * **" is ***.

** *** ******** ** the ******-**, ***** ********** test ******** ** *********, in ********, *** ****** any ***** ** ******* an ************'* ********* ** all ***** ** **** and ********.

*'* *** ** **** list, * ***** *****, that *** ****** ********** and *** * **** for ***** ********* ** interaction, ****** ******* *** can't ***** ******* *** the ******** ******** ******* that ** *** ***** will ******** ***.

********* * **** *** are **** ******* *****'* advocate, * ***'* ******* you ****** ***** **'* sensible ** **** *** salespeople *** ** *** in **...

Avatar

Sergey Bystrov

NetworkOptix | In reply to John Honovich | 12/19/19 01:47am

**** *** ***** ********* interview ** **** **** they **** * **** of************(*** **** *****). ** means **** ****** ***** have ****** *** ** role ** ************ *******(*'* sure **** ***).

** **'* *** ***** "no *** ** **.." or "*** ****** *** human **".

John Honovich

IPVM | In reply to Sergey Bystrov | 12/19/19 01:54am

***, * ***** **** someone ****** ** ******* even ******** ***** **** out ** ******* ****** deploying ** **********. ** still ** *****, ********* to **** ** **, that ** *********** **** them **** **** ** formalize **.

**** ****, *** ********** advocated *** **********:

******* ********, **** ** 100% ******* **** *** autotests

*****, *** ********** ** detracting **** *** **** of **********, **** ****** that *** ********** ********, whether ******* ***** ** now *******, ** ***** to **** * ********** QA *******, ****** ***.

Avatar

Sergey Bystrov

NetworkOptix | In reply to Sergey Bystrov | 12/19/19 05:41pm

* ******** **'* ***** to **** * ***** good ****** ***** ***** product *******.

Undisclosed Manufacturer #7

12/16/19 08:37pm

**** ******* ** ***'* that *** **** ***** oil??

Avatar

Sean Patton

IPVM | 12/20/19 04:58pm

** **** ******** ******* of ********* ************ *******, with ** *** *** resolution ******** **** *******. A ******* ******* **** us **** *** ***** having ****** *** **** not ******** * ******** in * ****.

** **** ********* ** receive *****/*** *************, **** still ** ****** *** notifications.

******* *** *** ******** a ****** ** ********** for *** ******, *** we **** ****** *** report **** ** ******* one, ** ******* ********* a ****-******.

Undisclosed #1

In reply to Sean Patton | 12/20/19 05:37pm

**** ****** ** ***** like **** ****** * software ****** **** **** massively ******* * ******** schema, ** ******* **** other *********** ************* ** the ************ **** ** very **** **** ** roll ****.

** **** *** *** fault ** * *** party ******* ********, ** would **** **** ****** been **** **** ** a **** *****, ******** along **** ******* ********* to * ********* *** party. *** ** *******, we ***** ** ******* of ***** ********* ****** notification ******** *** ***** products.

** **** *** *** result ** * ***** bug, *****, ***., **** would **** ****** **** back, ** ***** *** issue ** ***.

** **** **** ******* out ** ****** *********, they ***** **** **** a ***** ** **** tumblers ** **** ** fund **** **** *** cycles :)

**** ****** ** ** believe **** ** **** very ******* ** **** architecture, *** ******* **** Verkada **** **** *** very *********, ** **********, to *******.

* ****** ** ******* customers **** **** ** given *** ****** ***** how *** **** *** changes, ************, ***. *** rolled *** ** ***** personal ********* *** ********. This ** *** ** the ***** ********* ** SaaS-style *******, **** ******** experience ** **** ** good ** *** ******* link ** **** ********* resources.

John Honovich

IPVM | 12/24/19 12:25pm

*********** *******'* ********* ***** them ** ***** '****** reliable, ****** **':

****** *** ******* ***** of **** *********, ********* rarely ***** ** ** 'always' ******** ** '**' as *** ******* (******* its ******* ** ***** or ***, ***.) ** any ******* *** **** issues ********.

Undisclosed #1

In reply to John Honovich | 12/24/19 12:38pm

*********** *******'* ********* ***** them ** ***** '****** reliable, ****** **'

****, ** **** ***** the ******* *** **** impacted **** ****** **** you ***************** ** ***'* *** fixed *******.

Undisclosed Manufacturer #8

In reply to John Honovich | 12/24/19 05:04pm

*'* ******** **** *** rethinking ******** **** ** sign ** ** * customer ***** ***.

Avatar

Sean Patton

IPVM | In reply to Undisclosed Manufacturer #8 | 12/24/19 07:10pm

** ********* *** ******* through * ***** ******* dealer *** ********** *** system ******* ******** **********.

***** * ******* ***** they ***** ***** ******** created **** *** ***** address/domain, * ** *** know ** **** ***. They **** ** **** channel, ****** *** ******** through ***/************.

Undisclosed Integrator #5

In reply to Sean Patton | 12/24/19 07:39pm

***** *** **** ******** partnering **** * ***** integrator ** ** **** covert *******, ** * manufacturer ***** ** **** their ******** **** **** you?

*'* **** ****** *** of *********; ** ******* doesn't ******* ** ************.

Undisclosed Integrator #5

In reply to John Honovich | 12/24/19 07:34pm

*** **** ** ** uptime ** **** ********* in ***** ***? ** they *** *** ******* like **********.** ** ****** outages?

** **** ** *** advertising ******* **** ** SLA ** ********** ** back ** **, ****'*... not ********** ** *** from **** *******.

Avatar

Sean Patton

IPVM | In reply to Undisclosed Integrator #5 | 12/27/19 07:59pm

* *'* ** ******, we **** ***** **** are ******** ******* ******* for ***** **** **** not **** ********* *************.

**** ******* *** *********'* *** ******* ****.

Undisclosed Integrator #9

12/30/19 05:16am

* ** * ******* partner. ***.

******: **** ******* *** been ***** ** *** own **********:* ** * ******* Partner. ***.

Login to read this IPVM report.
Why do I need to log in?
IPVM conducts unique testing and research funded by member's payments enabling us to offer the most independent, accurate and in-depth information.

Related Reports

ROG Security - Cloud AI For Remote Monitoring on Jan 28, 2020
ROG Security is offering cloud-based AI analytics to remote guard companies, by touting having "nothing to install" to "add virtual guards." We...
Apple Acquires XNOR.ai, Loss For The Industry on Jan 16, 2020
Apple has acquired XNOR.ai for $200 million, reports GeekWire. This is a loss for the video surveillance industry. XNOR.ai stunned the industry...
Wyze Massive Data Leak on Dec 26, 2019
Wyze has exposed millions of user's data, as reported by Twelve Security, and confirmed by IPVM, who has spoken with Twelve Security and reviewed...
nFlux AI Startup Profile on Oct 28, 2019
nFlux, an Amazon, Google, and Microsoft funded AI startup told IPVM their mission is to build the most intelligent video analytics platform in the...
Axis Suffers Outage, Provides Postmortem on Aug 15, 2019
This week, Axis suffered an outage impacting their website and cloud services. Inside this note, we examined what happened, what was impacted...
ZeroEyes Gun Detection Startup on Jul 16, 2019
A gun detection video analytics startup, ZeroEyes, is being led by a group of 6 former Navy SEALs, aiming to "save lives" by using AI to assist...
Calipsa - UK AI Startup Profile on Jul 10, 2019
Analytic startups are a major industry trend. One UK company, Calipsa is aiming to use AI to filter out false positive alarms for live video...
Carnegie Mellon AI Startup Zensors Profile on Jun 11, 2019
Zensors is a startup formed by Carnegie Mellon graduates from a Carnegie Mellon research project, offering customized models per camera that they...
Actuate (formerly Aegis AI) Gun Detection Video Analytics Startup on May 07, 2019
Gun detection analytic startups are increasing as the promise of AI and the threats of active shooters grow. One company, Actuate (formerly Aegis...
Bezos-Funded Deep Sentinel Tested on Mar 28, 2019
Backed by Jeff Bezos, the Silicon Valley startup, Deep Sentinel, has declared: No One Does Home Security Like We Do Our Surveillance Team has...

Most Recent Industry Reports

Athena CEO Criticizes 'Deplorable' 'Nitpicking', IPVM Refutes on Mar 27, 2020
Athena Security's CEO Lisa Falzone has strongly objected to IPVM's reporting on Athena, calling it 'deplorable' and repeatedly criticizing IPVM's...
Hikvision Admits Sanctions Harming Its Financial Performance on Mar 27, 2020
While Hikvision initially downplayed being sanctioned for human rights abuses, the company is now admitting a significant impact in a new PRC...
New Axis M30 Cameras Tested on Mar 26, 2020
Axis has released a new generation of, for them, relatively low cost M30 series cameras, claiming to deliver "sharp video quality even in poor...
Coronavirus Shuts Down ADT Door Knockers on Mar 26, 2020
Coronavirus has another victim - this time, alarm giant ADT has stopped all door to door sales. Door knockers are a critical but controversial...
Access Control Course Spring 2020 - Save $50 Last Day on Mar 26, 2020
Register Now - Spring 2020 Access Control Course. Today, March 26th is the last day to save $50. IPVM offers the most comprehensive access...
Convergint Coronavirus Cuts on Mar 25, 2020
One of the world's largest security integrators, Convergint, has made a major move to handle the impact of coronavirus, with cuts across the...
VSaaS 101 on Mar 25, 2020
Video Surveillance as a Service (VSaaS) is the common industry term for cloud video. But what does it mean? How does it all work? Inside this...
TVT / InVid Facial Recognition Tested on Mar 25, 2020
Facial recognition is frequently sold for thousands of dollars per channel but some China manufacturers are offering full facial recognition...
IPVM Launches On-Demand Courses on Mar 24, 2020
For nearly a decade, IPVM has been a leader in online live courses. Now, we have added on-demand versions for all courses. The same course...