Video Analytics Measuring Accuracy / Accuracy Issues Guide

By IPVM Team, Published Apr 01, 2021, 09:36am EDT

Video surveillance manufacturers often tout "high accuracy" analytics as a key marketing focus, typically with 90%+ values. But how do they reach those figures?

IPVM Image

** **** ****** ** break **** *** ******* each ** *** *********:

  • ****** ***** *** ***** Surveillance
  • ********** ******** ********
  • ********* *** ******
  • **** ******** **** ** False ******** ****
  • *** ***** / **** Under *****
  • **-***** - ******** *******
  • **** ******* *********
  • *********** ******** ******
  • *** **** ***** *********
  • ***** ****** *********** ********* Challenges
  • ************ *********** ****
  • ************* **** **** ******** Rates
  • ************* *** *********
  • ******* *******

**** ** **** ** our***** ********* ************** ** *** *** of *****.

Ground *****

IPVM Image

****** ***** ** **** to ******** *********** ******** through ****** ***********, ****** than ********/*********. * ****** example ** *** **** marbles *** ** * glass ***.

** ********* ***** ******** how **** ******* **** in *** *** (*.*., by ********** *** **** of *** ******* *** the ****** ** *** jar, ***.).

*******, ****** ***** *** only ** ********** ** a ****** ******** *** marbles *** ******** *** total.

** ***** ************, ****** truth ** ******** ** a ***** ******** ********** results ** *** ********* from ******** *****. ******* to *** ******* ** a *** *******, **** is *** **** *** to ********* ** ********* are ********* **** **** should, ******* *******, ** making ********.

****** ***** ***** ********* developers ** ********* ****-***** examples ** ********, ** that ********* *** ** improved ******* ******** ***/** calibration.

Ground ***** ********** ******

*********** ****** ***** *** alerts ** **** *******, by **********, *** ****** is ********** ** *****/*** which ** ********. ***** events *** ** ********** as ******** ** **********.

*******, *********** ****** ***** for **** ******* *** not ******** ** ****, because *** ****** **** not ****** ** *****/***** to ******. **** ***** all ***** ***** **** to ** ********, ********* the ******* ** ***** analytics. ********, ** **** applications, ****** ********** *** a ******* *******, **** serious **** *****************.

*** *******, **** ***** monitoring/alarm ************ ******* ********* only ****** ******/******, *** continuously ********** ***** *** missed **********.

IPVM Image

**** ** * *********** risk *** ***** ***** of *******.

******* ******* ***** ****** truth ** ********* ** determine ** ***** ******* since ** ***** ******* taking *** ************ ** all ****** ********.

Fundamental ******** ********

*** ********* **** * outcomes **** ****** *** accurate * ****** ** performing:

  • **** ********* - **: A ****** ** ******** as * ******
  • **** ********* - **: A **** ** *** detected ** * ******
  • ***** ********* - **: A **** ** ******** as * ******
  • ***** ********* - **: A ****** *** *** detected ** * ******

IPVM Image

Simplistic ***********

* ********** ****** *** calculating ******** ******** ** finding *** ***** **** it *** *******, *** events ***** ***** ******, compared ** *** ******:

IPVM Image

*******, **** ********** ******** is ******* ** *** value ** *********** *** total *********** ** ** algorithm. *** *******, ** a *** ********* ********* processed *** ****** *** detected ** ****, *** 5 ****** *** ****, it ** **% ******** by **** ******, *** failed ** ****** **** 100% ** *** ****.

** ****, ********* ********** factor ** **** ********* considerations **** ******** ***** algorithms. *** *******, ** the ******** *** ******* any ****** **** ******** than ****** ****** ** false ****** ************** * deer ** * ******?

IPVM Image

******* ** ***** **************, there *** ********** ******* to ***** ******** ********.

Precision *** ******

******* ***** ********** ******** calculations *** *** ************, there *** * *********** metrics **** ** ********** how **** ** ********* works; ********* *** ******.

*********

********* ******** *** ***** a ********** ****** ** correct (*.*. * ****** is * ******) ** the ***** ** **** Positives ** *** *********. Precision ******* ** ***** positives ********.

****** ** *** ******* of *** ******* *** analytic ** ** ********* all ****** ******* (*.*. all ******, *** *****, all ********) ** *** ratio ** **** ********* to **** ********* **** False *********. ****** ******* as ***** ********* ********.

*******, ** ********* ***** have ********* *** ****** values ** *.*, *** in ****-***** *******, ** Precision ********* ****** ********* or **** *****.

IPVM Image

********* ********

********* ********** *** **** the ********* ***** ******* correctly *** **** *** take **** ******* ******* that ** ****** (***** Negatives), ** ********* ******* (True *********). ****** *********** access ******* ******* ******* high *********, *** ******** access ** *** ***** person, *******, ****** ************ (False *********) *** *********** and ************* ****** *****.

*** *******, ** * test ** * *** detection ********* ********* **,*** people *** ******** * guns; * ****** **** had * *** *** 3 **** *** ***, and ****** ** ******* that *** ****. *** precision ** **** *** detection ** *.**.

  • ********* ********** * - True *********
  • *** *** ***** ** - ***** ********* (**** not ****** **** *********)
  • *********** ********** * - False *********

***** ********* ** *** ratio ** **** ********* to *** *********, *** precision **** ** * / * = **%. However, **** ** **** that *** ****** ***** had * *********** ****** of ***** ********* **** are *** ******** **** precision.

****** ********

****** ********** *** **** the ********* ******* *** target ******* *** **** not **** **** ******* how **** ** ********* ignores ***-******* ** *********** identifies ** ******. ****** is **** ******** **** to ***** ********* ********, especially **** ******* ** correctly ****** **** ******** outcomes.

*** *******, *** *** detection ******* ***** *** a ****** ** ~*.** (true ********* ** * / **** ********* ** 1 + ***** ********* of **, *.*., * / **).

****** ** **** ***** as *********** ** **** Positive **** (***) *** is ******** *** *********** algorithm ***********.

Sensitivity / **** ******** **** ** ***** ******** ****

*********** ** **** ******** Rate (***) ********** *** many ********* ***** **** be, **** **** *** false.***** ******** **** (***) identifies *** ****** ** algorithm **** ***** ***** (both ******** *** ********).

******** *** *** ****** FPR ** **** ** compare ********* ****** ****** testing, ** **** *** best **********.

ROC ***** / **** ***** *****

******** *** *** *** creates * ******** ********* Characteristic (***) *****, ***** is **** ** ******* the *********** ** ******** models. ************, *** **** under *** ***** (***) can ** **** ** an ******* *********** *** accuracy.

IPVM Image

*** ****, ****** *** green ***** ***** ********* 3 ********* ****** **** curves ******** ** *** to *** **** ********** best *********** (**** **** Positives *** ******). *******, because *** *** ***** is **** ** ** many ****** ** ********* thresholds ** *** *** FPR, ********** *** * secondary ****** ***** *** area ***** *** *** curve, ** ********* ***********:

IPVM Image

*** ** *** *-*********** area ***** *** *** curve *** ** * value ******* * *** 1, **** * ***** of * ********** *** best ******** ***********.

*** ** ******** *** judging *********** *** *** main *******:

  • ** ********* *** ******** an *********/***** ************* ******* each ************** (*.*. ****** vs *******)
  • ** ******** *** ******* of *** *****'* ************** capabilities, ****** *** **** range ** *** *** curve.

*******, **** ***** *** also ***** *** ********** of *** ** ******* use *****:

  • ****** **** **** *** may ************* ****** *********** on ******** ****, ************ in ***** ***** *********** biases ** *** ******** images, *** *** *** reflect ****-***** ***********.

  • ** ***** ***** ***** are **** *********** ** the **** ** ***** negatives **. ***** *********, it *** ** ******** to ******** *** **** of ************** *****. *** example, **** ***** ****** recognition ****** *******, *** likely **** ** ********** minimizing ***** ********* (**** if **** ******* ** a *********** ******** ** false *********). *** ***'* a ****** ****** *** this **** ** ************.

F1 ***** - ******** *******

******* ****** *** ********* analytic *********** ** * model's * *****, ***** finds *** ******** ********* average ** ********* *** Recall:

IPVM Image

* **** ** ***** is ******** ***** ** academic ****** *********** ******** for ****** *********** ******* identifying *** ******* ******* and *** ******* ********/********* persons *** ********* ******* important.

Mean ******* *********

******* ********* ****** *** algorithm *********** ** ***** Mean ******* ********* ** determine *** ****** *** model ******* *** ********** each ****** **********, ******** to *** ******** ****** truth.

** ******* * ********* for *** ******* ******** intersection **** *****, ******* will ********* ******** **** may ********* * ****** FPR **** *** ****** of *** ****** ** detected:

IPVM Image

**** ******* ********* ** not ********** ** ********* Precision ****** (*** *****), but ** ********* *** overlap **** ** *** detected ****** (********* ***) compared ** *** ****** truth:

IPVM Image

**** ******* ********* ** less ** * ******* for ****** ** ******* detection, ** **** ************ users *** ****** *** concerned ***** *** **** of *** ****** *** bounding *** *******, **** that **** ******* ** alert.

*******, ** ** ******** an ***** ******* ** dynamic ******* ********, ** incomplete ** ********** ******** can **** ** ****** performance:

Fundamental ******** ******

************ ************ ****** *********** challenges ** ********* *********** that ***** ******** ******; too **** ***** ********* and ***** **********.

*** **** ***** *********

** ******* ************, *** balance ** *** ********* of *** *** *** is ******** *** *********** and ***** ********, *** analytics **** ******* * non-trivial ****** ** ***** Positives **** ****** *********** problems.

*** **** ** ***** issues ********* ** *** number ** ******** *** analyzed, *** ***** ** a ***** ************ ******* the ****** ** ******** that ****** ** ******** as **** ********* (*.*. non-suspected *********) *** **** positives (*.*. ****** *********).

IPVM Image

*** **** ****** **** is ***** ***** *******, where ***** ****** ** disable ****** **********. **** is **** ****** **** simple *** ** *********-***** analytics ***** ****** **** high *** *** ** lighting *** *******.

***** **** ******, *** potentially **** ****** ****, arresting * ****** ***** on *****-******** ****** *********** is * ***** ********* for *** ***********, *** has **** ************* ****** ********** *** **.

***** ****** *********** **********

**** * ****** *********** watchlist ******** *********, **** of *********, **** ******* ***** (**** Captis ************ ****), ****** *********** ********* performance ** ********* ** false-positive ****.

*** *******, **** * 100,000 **** ***** **** a **** ********** ********* will ****** ********* ** false ************, ***** *** algorithms **** **** ****** false ************, *** ** a ************* ***** **** or ****** ** ****.

******* ************ *** *****, within * ***** ****** watchlist, ***** ** ***** to ** * ****** that *** ******* ****** facial ******** ** **** the ********, *** **** human *********. **** ** a ****** ***** *****, as *** ****** ** often *** *****:

IPVM Image

**** ** ******** ***** celebrities, **** ***** "*************" easily ******** ** ******:

IPVM Image

*******, ********* ****** ******* is ****** ** ****** recognition, ***** ** ******* for ********* ******** ****** suspects, ***** *** ** ground-truthed **** ***** ************.

Manufacturer *********** ****

** *** ******* ** selling **** ********, ************* will *** *** ****** for ******** *** *********** that ********** ***** ******** best, ***** ** ***** misleading. **** **** ***** justify **** ** ****** a ****** *** **** with ******* *********.

IPVM Image

********, ************* **** ***** report ******** ******* ****-****** datasets (*.*.******* ***** ** *** Wild - ***), ***** *** *** challenging, *** ******** ******* in **%+ ********.

******* *** **** ******* manufacturers *** ** ********* and ********** *** *** matter, ** ******* ** controlled ************ **** ***** subjects *** ****** **** results. **** ****-***** ******** can **** ** ********** through **-******** *******.

Manufacturers **** **** ******** *****

**** ************* **** ************ **** True ******** **** *** accuracy, ***** ** *********** because ** ******* *** objects **** ** ******.

IPVM Image

*** *******, ** ***/**** manufacturer **** ***** **% accuracy *********** *** ************ characters ** * ******* plate, *** **** ** typically *** ****** **** were ****/********. **** ****** that **** **** *********** count ******* **** ******** metric, *** ****** **** were *** **** ** all, ***** ** ****** in *** ****-*****.

****** ****** **** ***** missed ****** (******* *****, high ******, *** *******, damaged ******, ******** ******) are *** ******* ******* the ********, ***** **** not ******* *** ****-***** performance ** *** ******** system.

Manufacturers *** *********

*** **** *********, ***** are ** ********** ********* ratings ** ******* (*.*. WDR, ******* ** ***** levels) **** *** ** measured *** ***** **** they ****.

********, ************* **** ********* accuracy ******, **** ***** generic ***** ********** **** accuracy:

IPVM Image

****'* ****-***** ******* ***** that ********* ***** ******** to **** "****" ** exceed **% ********, *** to ************* *** ********** challenges, ********** ** ****** lights *** ***** *******.

****** ***** ************ ** controlled ************ (*.*. ****** recognition ****** *******, *** tolling) *** ****** ** quantify ***/*** **** **** common ****** ** ******* detection.

Ranking *******

********* ********** ****** ** ranking/ordering ******* ** ******** is * ****** ****** analytic ********* ***** ** help ****** ****** ********:

IPVM Image

******* ******* ****** * human ** ****** *** top *******, ****** **** by *********** ******* ***** a ************* ********/********** *********.

*** ******* **** ******* or ********* ********** ****** is ********* ******* **** provide ********* ****** ** no ******, ****** ** difficult ** ******* *******. It ** ****** *** systems ** ***** ******* (e.g. ****, ******, *** Low) ********** *******, *** not ******* ********** ******.

****** *********** ******* *** generally ** ********* ** this, ** **** ******* provide ********** *****:

IPVM Image

**** ********, ********* ******** boxes *** *** ******* in *** **** *********, making ****** ** **** average ********* **********.

Comments (2)

***** *** ** **** for **** ***** *****, Shawn. * ***** *** easy *** **** **** to **********! * ** have * ***** ********, you **** ****** ***** setting ****** ***** *** for ****** ******* ********. What *** *** ******** on ****** ***** *** deep ******** ************, ** you ***** **** **** it ** *** ********* (out ** *** ***)? Thank ***!

Agree
Disagree
Informative
Unhelpful
Funny

****** ***** (*** ***********/******) applies ** ******* *** deep ********, ********* ****** the *********** *** ******** of *** *********.

Agree
Disagree
Informative: 2
Unhelpful
Funny
Read this IPVM report for free.

This article is part of IPVM's 6,891 reports, 921 tests and is only available to members. To get a one-time preview of our work, enter your work email to access the full article.

Already a member? Login here | Join now
Loading Related Reports