Alarm.com "AI" Video Analytics Tested

By Brian Rhodes, Published Nov 30, 2018, 07:15am EST

Alarm.com has announced what it calls an "artificial intelligence (AI) architecture and video analytics service", touting that

Alarm.com's computer vision research team trained the AI engine with millions of frames of video donated from field-deployed cameras, and extensive feedback from service provider partners and beta program participants.

Indeed, this follows Alarm.com's 2017 acquisition of Object Video, the original, but much-maligned, video surveillance analytics provider.

How good is Alarm.com's "AI" Engine? We own an Alarm.com system (tested here), connected a supported camera, enabled the "AI" analytics and tested the system.

tested 2

In this report, we share our results including:

  • Analytic performance rankings
  • False alert resistance versus foliage, shadows, animals, lights, and more
  • Detection distances/PPF levels
  • Configuration/zone creation for each
  • How pricing compares, for cameras and cloud service

Key ********

*** ******* ****** **** the ****** *** ******* decent ******** ** ************ VMD *** *** ****** that ***********, ********* **** 2 **** ******:

  1. *** ****** *** ****** restrictive / *********** ************ 'recommendations', **** ******* ** - **° ********.
  2. **** **** *** ******** recommendation, ***** **** ***** false ********* (*.*., ****** being *********** **** ** people **** *******).
  3. *******, *** ********* *** restrictive ******** **************, *** false ********* **** **** more ***********.

**** *****, ******** ** conventional ***, ******* ***** alerts **** *****.

Solid ***** ***** **********

** *** *******, *****.***'* Video ****** ********* (***** Analytics) *** ********* ********* in ******** ****** ***** alerts ****** ** ******* foliage, ****** ******, *****, insects, *******, *** ****** which ********** ********* ************** ********* ** *** testing.

********** *** *********** *** customers, ***** *** ***** Analytics ****** **** *** down ** *** **** 'false *****' ******. *******, it *** ******* *** reliable ****** ** ********* all ******.

Classification ******

*** ******* ******** *********** is *** ******** ***-***-****, even **** ******* *** installed ********* ** *************. Some *******/****** **** ***** to ** **** ****** inaccurate **** ******. *** example, *** ****** ******** often ******** *** ******* 'people' ** *** **** scene ** ********:

** ******* ** ***** events, ** ****** **** present, ******* *** *****. The ******** ** ** LP-Natural **** *** *****, in *********** **** * moving *******, ***** ******** in *** ****** ********* the *** ***** ******** as * '******':

** ***** *****, *** system ****** ****** *** animals ********, ** ** this *****:

not all people animals detected

****** *** ********** ** the *** **** * person ******* ** ** classification ****.

Small ****** *********

*** **** ***** ******** the '******' ************** ** working. *** ****** *** not **** ** ****** and ****** ****** ******** accurately (** **** *********, dogs *** *******), *** the ****** *** *** classify *** ******** ** 'animal' ********* ** *** way ** ******* ****** and ********.

*** **** ***** ** an ******* ** ********, but *** **********, ****** movement:

********** *********** ***** ****** may ***** ********* ********, especially ** ***** *** rules ** ****** *** general ****** ****** **** a **** *****-***** ***********, but ******* ******* ****** to ***** ** ******* or **** **** *** be ********* ** **** that **** ****** *****.

** ***** *****, *** zone ****** ********* *** have ** ** *** as **** ** **** 'fast' ***** ***** ** 0.5 ******* ** **** an ***** ** ****** small ** ****-****** *******, but ***** ******** *** also ************* ******** ******** alarms ** *** **** time.

Steep ******** *********

****** **** *** ******** analytic ****** ** ******, Alarm.com *** *********** *********** requirements ** **-**° ********, mounted ** ***** * feet **** *** ** coverage ****** **' ** night:

steep downtilt setup recommendation

*** *********** ** ***** makes ** ***** *********** in *** ** *** be ***** *** *** far ** *** *****, even ******** ***** ** consumer *********.

Nonconformance ********* ******

******* *** ****** ** be *** ** **** shallower *********, ******** **-**° for ****** *** ** users ** ********** *** familiar **** ***'* **************.

*******, ** **** *****, we *********** ***** ******** errors ** * ******. For *******, ** **** case * ******* *** not ********** ** ** approached *** ******, *** was ********** ******** ***** the **** ** ** was *******:

**** **** ***** ******* was ******* ** ** activity ***** ** *** system, *** ********* *** clip ***** ** *********** movement ** *** ****** at ***:

***** ******** ******** ****** did *** ******* **** adjusted ** *** ******** specification *********** ** ***, but ***** ******** ***** are *** ***** ** highlighted ** ********* ********* during *****.

Rain/Snow ********* ******

** **** *** *** tested **** *** **** to *** **** ****** that ******. ** ****** to *** **** ** the **** ****** *** will **** ** ****** in *** ******.

******: **** ***********

***** * ~* **** period ** ****** **** in ******* *** *****, our **** *** *** record *** ***** ******** or ******** ****** ** classification.

Detection *********

*** ******* ************ ******** objects ********** ** *** levels ***** **** ********* in *** ***** *****. For *******, *** ***** disclaims ****** ********* ****** 25' ** *** *** 15' ** *****, *** the ****** ********* ****** at **** ** ***** ranges.

*** ***** ***** ***** ~50' ************** ** * vehicle ** ******* *****:

Good *** **********

******* ****** ******* ** VMD **** ***'* ********* overcomes ** ****** ********** due ** ****** *******, leaves, *** *******. *** ADC ****** *** *** trigger ********** ** **** nuisance ************* ** *** point ****** *** ****.

**** ** ** ******* clip ******* ****** ******** that ***** ******* *** but *** *** ***** Analytics *****:

Motion/Analytic *************

** ******, *** ** offering *** ********* *****, a ***- ** ************* 'tripwire' ** * '****** zone' **** ******* **** objects ***** ****-******* *****:

** ******** ** *** rules, ***** *** ********* assign *** **** ******* must ** ** ********* of *** **** ****** an ***** ** ****, or **** ** ***** arm/disarm ****** ** **** of *** ********** ********. The ******** **** ****** assigning ** ***** ** objects ***** ** *** direction ** *** *****.

*** '****** ****', *** detection **** ***** ** flexible *** *********** ****** scenes ******* *********** ********* areas:

Camera *********

*****.*** ****** ~* ***** camera ******, *** **** 2 ** **** *** supported ** ***** ** analytics, *** ****** ***** w/IR (***-******) ********* ******** (***-*****). ** ****** *** (Vivotek ***'*) ******* ****, shown ** *** *****.

Camera/Service *******

*** *** ******* ******* a ******* ************ ** order ** ******* ********/***** alerts ***/** **** ***** video. ****** ******* ** ~$1 **** *** ***** added ** *** ~$*/*** four ******* '***' ***** plan ** ******** *** AI *********. *** ********* package **** **** ******** clips, ******** *** ***** to *,*** *** * cameras ** **** *,***. Camera ****** *** ~$*** for ***-*****.

Versus ******/****

****'* ********* **** **** accurate **** *****.*** *** does *** ********* **** installation ************. ***** ****** detection ********* ** *** require * ************ ** receive ******. *******, ***** video ******** ** *********** without ***** ***** ************, ******* **** $*/***** per ****** (* ****) through $**/***** *** ** days. *** **** ******* IQ ** ***************** **** **** *** $349.

Versus ****

*******'* ****** ********* *** more ***** ****** *** require **** ****** ******** distances/higher *** **** ***. Arlo ****** ********* ********* require * ************ ****** *****, ******** ** $*.** per ******, ********* * days *******, *** ** to $**.** *** ** days *******. **** ******* and **** ******* ***** at $*** *** *** original**** ****** $*** *** ******** **** *

Versus ******/****

** *******, ***'* ********* are ****** ********** **** Amazon/Ring, *** ******* ***** quality *** **** ** not **** *** ***** made ***** ** ******* WiFi.

****'* "******** ****** *********" motion ********* **** *** require * ************ *** motion ******, ******** * subscription ** ****** ** view ******** ***** ******** at $* *** *****.**** ********* ********** *** $*** *** can ** ********* **** the **** *******.

Comments (13)

John Honovich

IPVM | 11/30/18 12:19pm

Brian, good report!

Vendors need to be more cautious and conservative about using "AI" in marketing, as mediocre performance like this is going to disappoint and undermine the confidence in "AI" generally.

On the one hand, since machine learning is generally considered part of AI and surely Alarm.com is using some form of machine learning, they could technically make the case that it is "AI".

The problem is that most people expect "AI" to be, well, intelligent or at least as intelligent as a 5-year-old. For $1 a month, this is not a bad offering but it's not good enough to be trusted for high accuracy or to be marketed as "AI".

Avatar

Brian Rhodes

IPVMU Certified | 11/30/18 03:45pm

One interesting aspect of the 'AI' implementation is that 'learning' is fairly coarse and only happens without user refinement.

Right now, AI video learning happens as all video is ingested into ADC's cloud via an opt-in sharing feature:

Alarm.com mentioned to us in a call they plan to add a self-reporting feature allowing users to flag incorrectly classified objects.

The platform does not currently have this or any method to manually 'teach' or 'learn' the system based on user feedback.

Undisclosed #1

IPVMU Certified | In reply to Brian Rhodes | 11/30/18 04:52pm

Right now, AI video learning happens as all video is ingested into ADC's cloud via an opt-in sharing feature...The platform does not currently have this or any method to manually 'teach' or 'learn' the system based on user feedback.

Without feedback, what/how can it learn then from the video ingestation?

Avatar

Brian Rhodes

IPVMU Certified | In reply to Undisclosed #1 | 11/30/18 05:49pm

ADC only comments in general terms, that their 'internal process' refines results.  The company does not describe whether ongoing learning is automated, human spot checking, or some other refinement method.

Avatar

Michael Gonzalez

Integrated Security Technologies, Inc.
11/30/18 06:49pm

Excellent report Brian! Thanks for doing this, it saved me a massive amount of time.

Scott Sheldrake

12/01/18 12:58am

Thanks for the detailed report.  Please update when you have gone through a good rain.  That's going to be the real test.

Do you have a rough idea of the number of clips you get in a typical day?  How well does the ridiculous 1000 clip/month recommendation from ADC hold up?  1000 clips /mo= 33 clips per day or 1.3 clips per hour.  Seems unlikely you would be able to have less than 1000 clips per month.

Avatar

Brian Rhodes

IPVMU Certified | In reply to Scott Sheldrake | 12/02/18 04:08am

I'm crunching these averages now.  It looks like our single test camera averages 23 clips per day (n=8 days, 183 clips).

Extrapolating that out, 23 clips/day * 31 days = 713 clips/month.

However, our test camera was pointed at a fairly quiet location.  A busy scene, even a residential one, could exceed 1000 clips/m per camera.

 

Undisclosed #2

12/01/18 06:19pm

highest-performing AI to date: The Answer

Avatar

Brian Rhodes

IPVMU Certified | In reply to Undisclosed #2 | 12/02/18 03:55am

score = 9/10

Avatar

Brian Rhodes

IPVMU Certified | 12/02/18 04:21am

UPDATE: RAIN PERFORMANCE (added above to report)

After a ~3 hour period of steady rain in evening and night, our test did not record any false positive or observed errors in classification.

Undisclosed #1

IPVMU Certified | In reply to Brian Rhodes | 12/02/18 07:23am

...our test did not record any false positive or observed errors in classification.

Cue up that possum ;)

Avatar

Brian Rhodes

IPVMU Certified | In reply to Undisclosed #1 | 12/02/18 04:17pm

Around here, there's the possibility that sucker is already in someone's stewpot...

Scott Sheldrake

12/15/18 09:52pm

This about summarizes ADC video : 

Read this IPVM report for free.

This article is part of IPVM's 6,803 reports, 913 tests and is only available to members. To get a one-time preview of our work, enter your work email to access the full article.

Already a member? Login here | Join now
Loading Related Reports