Alarm.com "AI" Video Analytics Tested

Avatar
Brian Rhodes
•Published Nov 30, 2018 12:15 PM

*****.****** ************* ** ***** ** "********** ************ (AI) ************ *** ***** ********* *******", touting ****

*****.***'* ******** ****** ******** **** ******* the ** ****** **** ******** ** frames ** ***** ******* **** *****-******** cameras, *** ********* ******** **** ******* provider ******** *** **** ******* ************.

******, **** ******* *****.***'***** *********** ** ****** *****, *** ********, *** ****-********, ***** surveillance ********* ********.

*** **** ** *****.***'* "**" ******? We *** *******.*** ****** (****** ****), ********* * ********* ******, ******* the "**" ********* *** ****** *** system.

IPVM Image

** **** ******, ** ***** *** results *********:

  • ******** *********** ********
  • ***** ***** ********** ****** *******, *******, animals, ******, *** ****
  • ********* *********/*** ******
  • *************/**** ******** *** ****
  • *** ******* ********, *** ******* *** cloud *******

Key ********

*** ******* ****** **** *** ****** was ******* ****** ******** ** ************ VMD *** *** ****** **** ***********, suffering **** * **** ******:

  1. *** ****** *** ****** *********** / unrealistic ************ '***************', **** ******* ** - **° ********.
  2. **** **** *** ******** **************, ***** were ***** ***** ********* (*.*., ****** being *********** **** ** ****** **** present).
  3. *******, *** ********* *** *********** ******** recommendation, *** ***** ********* **** **** more ***********.

**** *****, ******** ** ************ ***, overall ***** ****** **** *****.

Solid ***** ***** **********

** *** *******, *****.***'* ***** ****** Detection (***** *********) *** ********* ********* in ******** ****** ***** ****** ****** by ******* *******, ****** ******, *****, insects, *******, *** ****** ***** ********** triggered ************** ********* ** *** *******.

********** *** *********** *** *********, ***** the ***** ********* ****** **** *** down ** *** **** '***** *****' events. *******, ** *** ******* *** reliable ****** ** ********* *** ******.

Classification ******

*** ******* ******** *********** ** *** accurate ***-***-****, **** **** ******* *** installed ********* ** *************. **** *******/****** were ***** ** ** **** ****** inaccurate **** ******. *** *******, *** camera ******** ***** ******** *** ******* 'people' ** *** **** ***** ** vehicles:

IPVM Image

** ******* ** ***** ******, ** people **** *******, ******* *** *****. The ******** ** ** **-******* **** Gas *****, ** *********** **** * moving *******, ***** ******** ** *** system ********* *** *** ***** ******** as * '******':

IPVM Image

** ***** *****, *** ****** ****** people *** ******* ********, ** ** this *****:

IPVM Image

****** *** ********** ** *** *** with * ****** ******* ** ** classification ****.

Small ****** *********

*** **** ***** ******** *** '******' classification ** *******. *** ****** *** not **** ** ****** *** ****** animal ******** ********** (** **** *********, dogs *** *******), *** *** ****** did *** ******** *** ******** ** 'animal' ********* ** *** *** ** handled ****** *** ********.

*** **** ***** ** ** ******* of ********, *** *** **********, ****** movement:

********** *********** ***** ****** *** ***** usability ********, ********** ** ***** *** rules ** ****** *** ******* ****** events **** * **** *****-***** ***********, but ******* ******* ****** ** ***** on ******* ** **** **** *** be ********* ** **** **** **** gotten *****.

** ***** *****, *** **** ****** detection *** **** ** ** *** as **** ** **** '****' ***** times ** *.* ******* ** **** an ***** ** ****** ***** ** fast-moving *******, *** ***** ******** *** also ************* ******** ******** ****** ** the **** ****.

Steep ******** *********

****** **** *** ******** ******** ****** we ******, *****.*** *** *********** *********** requirements ** **-**° ********, ******* ** least * **** **** *** ** coverage ****** **' ** *****:

IPVM Image

*** *********** ** ***** ***** ** quite *********** ** *** ** *** be ***** *** *** *** ** can *****, **** ******** ***** ** consumer *********.

Nonconformance ********* ******

******* *** ****** ** ** *** up **** ********* *********, ******** **-**° for ****** *** ** ***** ** installers *** ******** **** ***'* **************.

*******, ** **** *****, ** *********** other ******** ****** ** * ******. For *******, ** **** **** * vehicle *** *** ********** ** ** approached *** ******, *** *** ********** depicted ***** *** **** ** ** was *******:

**** **** ***** ******* *** ******* as ** ******** ***** ** *** system, *** ********* *** **** ***** no *********** ******** ** *** ****** at ***:

***** ******** ******** ****** *** *** reoccur **** ******** ** *** ******** specification *********** ** ***, *** ***** atypical ***** *** *** ***** ** highlighted ** ********* ********* ****** *****.

Rain/Snow ********* ******

** **** *** *** ****** **** nor **** ** *** **** ****** that ******. ** ****** ** *** rain ** *** **** ****** *** will **** ** ****** ** *** report.

******: **** ***********

***** * ~* **** ****** ** steady **** ** ******* *** *****, our **** *** *** ****** *** false ******** ** ******** ****** ** classification.

Detection *********

*** ******* ************ ******** ******* ********** at *** ****** ***** **** ********* in *** ***** *****. *** *******, the ***** ********* ****** ********* ****** 25' ** *** *** **' ** night, *** *** ****** ********* ****** at **** ** ***** ******.

*** ***** ***** ***** ~**' ************** of * ******* ** ******* *****:

IPVM Image

Good *** **********

******* ****** ******* ** *** **** ADC's ********* ********* ** ****** ********** due ** ****** *******, ******, *** cobwebs. *** *** ****** *** *** trigger ********** ** **** ******** ************* at *** ***** ****** *** ****.

**** ** ** ******* **** ******* insect ******** **** ***** ******* *** but *** *** ***** ********* *****:

Motion/Analytic *************

** ******, *** ** ******** *** analytics *****, * ***- ** ************* 'tripwire' ** * '****** ****' **** detects **** ******* ***** ****-******* *****:

IPVM Image

** ******** ** *** *****, ***** can ********* ****** *** **** ******* must ** ** ********* ** *** rule ****** ** ***** ** ****, or **** ** ***** ***/****** ****** or **** ** *** ********** ********. The ******** **** ****** ********* ** alarm ** ******* ***** ** *** direction ** *** *****.

*** '****** ****', *** ********* **** shape ** ******** *** *********** ****** scenes ******* *********** ********* *****:

IPVM Image

IPVM Image

Camera *********

*****.*** ****** ~* ***** ****** ******, but **** * ** **** *** supported ** ***** ** *********, *** Indoor ***** */** (***-******) ********* ******** (***-*****). ** ****** *** (******* ***'*) outdoor ****, ***** ** *** *****.

Camera/Service *******

*** *** ******* ******* * ******* subscription ** ***** ** ******* ********/***** alerts ***/** **** ***** *****. ****** pricing ** ~$* **** *** ***** added ** *** ~$*/*** **** ******* 'Pro' ***** **** ** ******** *** AI *********. *** ********* ******* **** more ******** *****, ******** *** ***** to *,*** *** * ******* ** from *,***. ****** ****** *** ~$*** for ***-*****.

Versus ******/****

****'* ********* **** **** ******** **** Alarm.com *** **** *** ********* **** installation ************. ***** ****** ********* ********* do *** ******* * ************ ** receive ******. *******, ***** ***** ******** is *********** ******* ***** ***** ************, ******* **** $*/***** *** ****** (5 ****) ******* $**/***** *** ** days. *** **** ******* ** ** available******** **** **** *** $***.

Versus ****

*******'* ****** ********* *** **** ***** alarms *** ******* **** ****** ******** distances/higher *** **** ***. **** ****** detection ********* ******* * ************ ****** *****, ******** ** $*.** *** ******, including * **** *******, *** ** to $**.** *** ** **** *******. Arlo ******* *** **** ******* ***** at $*** *** *** ************ ****** $*** *** ******** **** *

Versus ******/****

** *******, ***'* ********* *** ****** performing **** ******/****, *** ******* ***** quality *** **** ** *** **** and ***** **** ***** ** ******* WiFi.

****'* "******** ****** *********" ****** ********* does *** ******* * ************ *** motion ******, ******** * ************ ** needed ** **** ******** ***** ******** at $* *** *****.**** ********* ********** *** $*** *** *** ** purchased **** *** **** *******.

Comments (13)
JH
John Honovich
•Nov 30, 2018
IPVM

Brian, good report!

Vendors need to be more cautious and conservative about using "AI" in marketing, as mediocre performance like this is going to disappoint and undermine the confidence in "AI" generally.

On the one hand, since machine learning is generally considered part of AI and surely Alarm.com is using some form of machine learning, they could technically make the case that it is "AI".

The problem is that most people expect "AI" to be, well, intelligent or at least as intelligent as a 5-year-old. For $1 a month, this is not a bad offering but it's not good enough to be trusted for high accuracy or to be marketed as "AI".

(4)
Avatar
Brian Rhodes
•Nov 30, 2018
IPVMU Certified

One interesting aspect of the 'AI' implementation is that 'learning' is fairly coarse and only happens without user refinement.

Right now, AI video learning happens as all video is ingested into ADC's cloud via an opt-in sharing feature:

Alarm.com mentioned to us in a call they plan to add a self-reporting feature allowing users to flag incorrectly classified objects.

The platform does not currently have this or any method to manually 'teach' or 'learn' the system based on user feedback.

U
Undisclosed #1
•Nov 30, 2018
IPVMU Certified
In reply to Brian Rhodes

Right now, AI video learning happens as all video is ingested into ADC's cloud via an opt-in sharing feature...The platform does not currently have this or any method to manually 'teach' or 'learn' the system based on user feedback.

Without feedback, what/how can it learn then from the video ingestation?

Avatar
Brian Rhodes
•Nov 30, 2018
IPVMU Certified
In reply to Undisclosed #1

ADC only comments in general terms, that their 'internal process' refines results.  The company does not describe whether ongoing learning is automated, human spot checking, or some other refinement method.

Avatar
Michael Gonzalez
•Nov 30, 2018
Confidential

Excellent report Brian! Thanks for doing this, it saved me a massive amount of time.

(1)
SS
Scott Sheldrake
•Dec 01, 2018

Thanks for the detailed report.  Please update when you have gone through a good rain.  That's going to be the real test.

Do you have a rough idea of the number of clips you get in a typical day?  How well does the ridiculous 1000 clip/month recommendation from ADC hold up?  1000 clips /mo= 33 clips per day or 1.3 clips per hour.  Seems unlikely you would be able to have less than 1000 clips per month.

Avatar
Brian Rhodes
•Dec 02, 2018
IPVMU Certified
In reply to Scott Sheldrake

I'm crunching these averages now.  It looks like our single test camera averages 23 clips per day (n=8 days, 183 clips).

Extrapolating that out, 23 clips/day * 31 days = 713 clips/month.

However, our test camera was pointed at a fairly quiet location.  A busy scene, even a residential one, could exceed 1000 clips/m per camera.

 

U
Undisclosed #2
•Dec 01, 2018

highest-performing AI to date: The Answer

(1)
(1)
Avatar
Brian Rhodes
•Dec 02, 2018
IPVMU Certified
In reply to Undisclosed #2

score = 9/10

Avatar
Brian Rhodes
•Dec 02, 2018
IPVMU Certified

UPDATE: RAIN PERFORMANCE (added above to report)

After a ~3 hour period of steady rain in evening and night, our test did not record any false positive or observed errors in classification.

U
Undisclosed #1
•Dec 02, 2018
IPVMU Certified
In reply to Brian Rhodes

...our test did not record any false positive or observed errors in classification.

Cue up that possum ;)

(1)
Avatar
Brian Rhodes
•Dec 02, 2018
IPVMU Certified
In reply to Undisclosed #1

Around here, there's the possibility that sucker is already in someone's stewpot...

SS
Scott Sheldrake
•Dec 15, 2018

This about summarizes ADC video : 

(2)