Testing Color Analytics Performance

Author: Ethan Ace, Published on Dec 16, 2013

In an industry known for overhyping its own value, color analytics have been considered pure science fiction, with few, if any, stories from the field to back up manufacturer claims.

In this report, we test Bosch's IVA color analytics to see which practical applications they performed in, and where they failed.

First, in an interior conference room with a human subject:

And an outdoor parking lot/driveway with different colored vehicles:

We varied the precision of the analytics to understand how performance varied, from least to most sensitive using their built-in scale:

We then tested a variety of colors from red, green, blue, orange, neons, etc.

**** ************* *** ********** *** *** *****, ***** ********* **** **** considered **** ******* *******, **** ***, ** ***, ******* **** the ***** ** **** ** ************ ******.

** **** ******, ** *********'* *** ***** *********** *** ***** ********* ************ **** ********* **, *** ***** they ******.

*****, ** ** ******** ********** **** **** * ***** *******:

*** ** ******* ******* ***/******** **** ********* ******* ********:

** ****** *** ********* ** *** ********* ** ********** *** performance ******, **** ***** ** **** ********* ***** ***** *****-** scale:

** **** ****** * ******* ** ****** **** ***, *****, blue, ******, *****, ***.

[***************]

Key ********

  • ** ****** *** *****, ******* *** ****** ** ******* **** the ***** ***, ****** ******** ********* ** ****** *** **** difficult. **** *******, ******* *****, **** **** ** ****** ****.
  • ******* ********* *** *** ******* ** ****** ******** ** *** desired *** ** *** ***** ***** ********** ****** (*.*., ****** and **** ********** **** **** ** *******).
  • ******************* ** * (********), **** *** **** ***** *******, ******* the ******** ** ***** ******* **** ***** ******, *** **** increases ****** ** ****, ** ****** **** ** **** ******** in ***** ** *******.
  • ******************* ** * ** * (**** ** **** ****) ******** in ******* ******* ** ****** ******* (**% *** **) ** alerts *** ********* **** ** * ***** ***** ** ****.
  • ** *** *****, ********* **** ****** ******** ******* ** * nearly ***% ******* ** ******, ** *** ****** ****** ******* colors ********, ****** *** ***** *** ***.
  • ** * ****** ******* ******* *** *****, ******** **** ************ detected **** ***** ********* *. ** ********* * *** *****, rules ****** ** ******** ** ***. 

***************

***** *** ********* ******* ******* ******** *** ***** ******, ** do *** ********* ***** ********* ** **** ** ******** ************. Alerting ** **** ******** ****** **** ****** ** **** ****** matches *** ** ****** *********** ** *****, ***, ** ******** lighting *** *******. *******, ******** ********* ******* ** ******** ****** for ****** ***** **** **** *** ** ********. ******* ** this, ***** ********* ****** **** ** **** **** ******* ** reduce *** ****** ** ****** **** ** **** ******** ************.

It ***** ** *****

*** ***** ********* **** **** ***** ** ** * **** end ****** (*** ***** ***-****) **** *** ***** **** ** *** ***** ******** **** *** *** ****** *** ***** ***********. *********** ***** ****** ** **** ***** ** ***** ******* cameras *** **** **** ****** ***** ***** ********* **** **** to **** **** ******* ******* *** ****** ****** ***********.

Configuring ******

**** ********** ******* ************* ** ***** ********* ** **** ** some ****** ****** ** *** **** ***** ********:

Selecting ******

*** ******* ******** ** *********** *****'* ***** ********* ******** ** compensating *** *** ********** ******* **** *** ****** "****" *** the ***** ******** *** *** ***** *****. *** *******, ** this *****, *** ****** ******* *** ** *** **** *******, as **** ** *** ****** ********** ** *** ***** *****, and ******** *** ********* ****:

*******, ** ***** *****, *** ****** *** ** ***, ****** mostly **** *******, ******* ** ******:

** ***** *** ******* **** **** *** *****:

  • *** ****** ** ****** *** *******'* ***** ** *** ******** color, ***** ** ****** ******.
  • ** ***** *****, ********** *** ***** ***** *** ***** **** the ****** ********** ** ** ****** ****** *** **** ******.

*******, **** ******* *** ******, ******* ********** ** *** ***** of *** ********** ****, *** ****** ***** ******** **** ****** as ****** ****, ********** ** ****** ***. **** *** **** of *** **** ***, *** ****, *****, ******, ******, ***.

****** ******

** *** *****, ********* *** *** **** ********* **** **** colors **** ** *****, ****, *** *****. ****** **** *** simply ******* *** ****** ***** *** *** ***** ** *** same ******** **** ** *** ***** *****, ****** *** ** variances ** ********** **** ******, ***.

*** *** **** ****** *** ******** ** *** ***** ***** also *** **** ******** ** ******, ** **** *****. ****** the ******** ****** ***** ***** ********* ***** ** *** ****** of *** ***** ******* ** **** ** *****. *******, ** also *********** ******* *** ****** ******* ** **** ***** **** the ****** ** *** *****, ****** ********** ********** **** **** difficult.

Color ********* **********

*** ***** ********* ******* ****** ******** **** ** ** ******** down ** **** * *** ******, ******** ** ******** ********. This ******* ****** **** * (*** ***** *******) ** * (only *** ******** *** *******), ** **** *****:

** *** *****, ** ***** **** ********* ***** * ******* reduced ******** *****. *******, ** **** *****, ****** ***** **** the ******** *** *** ******* ******, ** ****** *** ****** may ******* **** *** ** ********, *** *******. ** ********* 2 *** *****, *******, ******** *** ************, **** ***** *** rates **** **** ********* *, ** *** **** ********* ** all, ** **** ** ******** *****.

Color ********

** ***** ** * **** *** ****, ***** *** ***. Our ******* ****** ****** *** **** *** *****, ********* ** the ********* *******. ***** ***** **** ************ ********** ********** ** color, ********** ** ********* *, ***** ********* ** ***% ** attempts.

*****, ***, *** **** *** *****************, **** ***% ******** ** ********* *, ******** ** **% at ********* *. ****** ****, ** ********** * *** *, exact ****** ****** *** ******** ** ******* *** ****:

****** **** ****** **** ** *** ****** *** ***** ******** below **** ********* ** ***% **** ********* *, *** *** not ******* ** *** ** ********* * ** *****. **** was ******* *** ****** ***** *** ***** ****** ** ****** grey ** *****, ***** *** ******** ** ****** ********* ******.

Low *****

******** ***** ** ***** * ***, ******** ** **** ********* 1 ***** ************* ** ****** *** *****. ** **** ***** level, ** **** ***** ********* *** ********** ****** ** ********** for ****** ********* **** ** *** *****, *** **** *** no ****** ** *********** **********.

*****, ***** ***** ** ***** **% ******** ** ********* *, and ** ********** ** * *** *****:

 

********* ** *** ******* ****, **** ***** ***** ** ********* 1 ******** ** **** **%, **** ** ******** ** * or *****:

**** ********* ****** **** ****** *** ** *****, *********** ***** 70% ********, ****** ***** ******* ** ***** ****** ********* *:

**** ***** ********* **** *** ** *** ****** ** *** light, *** ** *** ******* **********. ** ********* *, ** remained ***% ********. ********** ** ********* *, ** ********** * 30% ***** ****, ****** **** *** ***** ***** ******:

*******, ****** *** ** ******** **** ****** **% ** ********* 1 ** *** *****, *** *** *** ******** ** * or *****: 

Vehicle ********* *********** 

** ********* *, ******** **** ************ ******** *** ** ***** larger ****. *******, *** ** *** *******'* ********, *** ***** changes ** ********* ******* *** ********* *** ******* ****, ****** the **** ****** ***** ****** ** ********* * *** ***** fail ** *******. **** *** **** ***** ******** ****** ** vehicle, ********* ****:

*** ***:

Comments (19)

Can this form of analytics be more usefull with thermal units delivering a colorized output?

Hello Birger.

No, unless you use radiometric thermal cameras giving the absolute temperature. The "standard" thermal cameras for video surveillance are differential, not radiometric: this means that they percept and show bodies with different temperatures, but you don't know if that temperature is 37° or 15°.. You just know that it is different, and the false colors (never use them with analytics; always black/white mode) are definitely relative and built frame by frame by the camera normalizing the whole image according to the different temperatures of the different bodies present in the scene in that moment.

This means that those colors are even much less reliable to search and track than when you search a color in an optical camera.

Cheers,

Simone

(TechnoAware)

Thank you for a very informative answer Simone.

/Birger

Finally!

Thank you so much John for this great report!

I will forward it to all people against whom I "fight" daily trying to convince them how much it is difficult to consider "color" as a reliable feature for analytics! And to all the writers of tenders who believes always to unlikely commercial brochures (not only from Bosh of course!..) without simply reasoning on the practical phisical issues..

Have a great time,

Simone

(TechnoAware)

Thank You Ethan & John

Great Report , Goes beyond the traditional reviews of the color dynamics of past reviews .

Great Job

Curious how it would work in a "perfect" scenario?

ie: assembly line, parking lot entrance/exit, doorway, etc. Basically if you had a setting with perfect lighting and small FOV.

Chris, parking lot entrance/exit and doorways are going to be especially challenging as varying light levels and strong backlighting are going to change the color appearance of objects over time.

I was thinking of possible applications that were made "perfect" vs "real world" applications.

Where you could control lighting and the "scene" at all times, possibly even the colors (no variations).

Yes, so if you can control things 'perfectly' it should work well. For example (I am making this up but I suspect this would work well), you have a conveyor carrying fruit and you want to automatically detect and flag fruit with 'off' colors. In such a scenario, you could carefully set it up and optimize it to accurately identify those off color fruits.

Yes, that would be a fine example.

So in a "perfect" scenario it should have a high % of accuracy otherwise don't bother...

I would assume it would also be best to pilot test the application before selling it ;-)

Chris, actually the point it's exactely this one you wrote, about making tests before.. Actually it's the main problem of this analytics market.. The problem is when the "smart" commercial always makes tests or shows demos in perfect scenarioes, with perfect stable conditions, for few minutes.... But unfortunately maybe that is not the reality.. Maybe that's just a trick, because unfortunately in the real world then the conditions are slightly never that stable for long time..

But after buying, the Customers verify in real life that it's all fake, that instead nothing works, and then none wants to see analytics anymore....... Of course Customers watch CSI in the evening and they want miracles; so "smart" commercials of course feel in need to show miracles.. But maybe it's not that "smart" and healthy to go on that way..

There are for example still many many products working by fixed background reference.. And believe me, there is no better method at all if you have 5 minutes of indoor perfectly stable and controlled environment, no shadows, no reflections, perfect stable light.. Uh, you can count the hair of a person by that method!.... But for 5 minutes... ...maybe! Nice trick..

As I always say to our Partners who tell to me "oh but what you say, I saw it by my eyes!....": yes, me too yesterday evening I was in a show of a good magician and I saw by my eyes a woman divided in 2...... But that doesn't mean the woman was really divided in 2, right?....;)

Cheers,

Simone

You got it ;-)

Guess that's why we are all here (reading IPVM), we appreciate the work they area doing in clearing the smoke screens and verifying reality...

We are using the color filter mostly for searching in the recorded video. It can help a lot to retrieve a certain person.

Thank you Bosch employee for that vague undisclosed endorsement. Care to elaborate on what people, situations, accuracy, and time frames you are doing this in?

John, perhaps you should add the additional caveat of 'Identity will be known to site admins' right after the "Post Without Disclosing Your Name" checkbox.

Some people must think they are truly anonymous, how else can you explain their audacity?

How's that? Guess we have to buy Micro SD cards then and record on the camera then run the analytics after the fact?

How does that help with alerting of an issue while it's happening vs hrs/days/weeks/etc after when it's realized, if the video is still on the card?

I would seem to think that analytics would be most powerful being run pro-actively vs re-actively, no?

Good luck finding which box the off color fruit is in, after you finally get time to run the analytics against the recorded video...

Yes, for "forensic analysis"...... Of course, because the issues of color when they come from an off-line video instead of a live one magically are solved, right?.. Wow, magic hard disk!....

Or maybe he was meaning that they open the off-line video in a Video Editor, they go to paint with bright red color the dresses of the person they want to search, then they run the analytics searching for red color and say "Oh! That's it!!..."........;)

The typical rejoinder for search is that it does not need to be as accurate as alerts. For example, if I search for a red car and get 50 matches and only one is really red, I can quickly scan through the 50 thumbnails rather than scan the full video over hours. This of course is premised on the fact that the true red car is not missed entirely.

But that's exactely the point John. As you correctly showed, the main problem is to see the color and find it.. Even in indoor as you showed, and we see then outdoor what it happens: when there are shadows, camera over/under-exposures, same camera light compensations and automatic gains, reflections of sun or any light, with artificial lights, by night, ...

I absolutely don't say it's useless, not at all. There are many examples of valuable possible applications, given a priori timely simplifying assumptions in the environment or in the application itself.. Infact also us we have this color filtering function. Only, as usual in the market there's a lot of science fiction and this kind of reports help to heal this market from confusion and false expectations..

Thanks, cheers

Simone

(TechnoAware)

Login to read this IPVM report.
Why do I need to log in?
IPVM conducts unique testing and research funded by member's payments enabling us to offer the most independent, accurate and in-depth information.

Related Reports

IFSEC 2018 Final Show Report on Jun 20, 2018
IPVM attended the IFSEC show for the first time this year. The Chinese took over the show, centered on Hikvision, flanked by Dahua, Huawei and a...
Hikvision 12MP Fisheye Camera Tested (DS-2CD63C2F-IV) on Jun 14, 2018
Hikvision's DS-2CD63C2F-IV is their flagship panoramic camera, with a 12MP imager, 15m integrated IR, smart codec, and more. We tested the 63C2 in...
ReconaSense - The AI / Access Control / Analytics / IoT / Video Company Profile on Jun 12, 2018
One company's ISC West booth stood out for displaying a light-up tower of buzzwords. The company, ReconaSense, pledged to be 'making sense of it...
Introducing Effective PPF (ePPF) - Improving Video Surveillance Designs on Jun 11, 2018
Pixel density (PPF / PPM) is the best metric the industry has to define and project video quality. It allows simple communication of estimated...
Hanwha Low-Cost 4MP Camera Tested (QNV-7010R) on Jun 11, 2018
4MP usage is increasing noticeably, as IPVM 2018 resolution statistics show. And low-cost, fixed focal cameras, are popular for budget...
Bosch IVA Video Analytics And Motion+ VMD Tested on Jun 06, 2018
Bosch's video analytics now ship on nearly every model, from indoor domes to high-end 5MP starlight cameras.  In this test, we evaluate Bosch's...
Hikvision PanoVu 20MP Flexible Camera Tested on Jun 01, 2018
Hikvision has released their first repositionable multi imager cameras with integrated IR included, atypical in competitors. We bought and tested...
Stats: Upgrading Cameras Far More Common Than Replacing Failed Cameras on May 30, 2018
The old saying "If It Ain't Broke, Don't Fix It" does not apply here. New IPVM statistics show that 60% of cameras that are replaced still work but...
Oncam 12MP Fisheye Camera Tested on May 29, 2018
Oncam has made their name since the early 2000s as a fisheye specialist, focusing only on panoramic cameras. To see how this specialist stacks up...
Hanwha Wisenet X Analytics and VMD Test on May 24, 2018
Continuing our updated testing of camera analytics, we tested Hanwha's Wisenet X analytics for over two weeks in multiple scenes, indoors and out,...

Most Recent Industry Reports

July 2018 IP Networking Course on Jun 22, 2018
  This is the only networking course designed specifically for video surveillance professionals. Register now. Lots of network training exists...
Installation Hardware for Video Surveillance - Indoor Fasteners on Jun 22, 2018
As part of our Installation for Video Surveillance series, in this note, we cover drywall anchors. A key part of installing security hardware is...
Hikvision ColorVu Integrated Visible Light Cameras Examined on Jun 22, 2018
When it comes to low light, infrared light has become the defacto standard in surveillance. But IR is limited to monochrome images, making colors...
'Secure Channel' OSDP Access Control Examined on Jun 21, 2018
Despite claiming to be better than Wiegand, OSDP's initial releases did not address the lack of encryption between reader and controller, leaving...
Most Wanted Improvements In Manufacturer Technical Support (Statistics) on Jun 21, 2018
5 key areas of improvement and 1 clear wanted support feature were voiced by 140+ integrator responses to: What improvement in manufacturer...
GDPR / ICO Complaint Filed Against IFSEC Show Facial Recognition on Jun 20, 2018
IPVM has filed a complaint against IFSEC’s parent company UBM based on our concern that the conference violates core GDPR principles on...
IFSEC 2018 Final Show Report on Jun 20, 2018
IPVM attended the IFSEC show for the first time this year. The Chinese took over the show, centered on Hikvision, flanked by Dahua, Huawei and a...
Mobotix Releases 'Move' Into 21st Century on Jun 20, 2018
For years, Mobotix stood resolutely against, well, every other manufacturer, selling it as a virtue: MOBOTIX equipment is designed with no...
Cybersecurity Startup VDOO Disclosing 10 Manufacturer Vulnerabilities Starting With Axis And Foscam on Jun 20, 2018
Cybersecurity startup VDOO has uncovered significant vulnerabilities in Axis cameras along with many others not yet disclosed. In this report, we...
Axis Guardian - Cloud VMS And Alarm Monitoring - Released on Jun 19, 2018
Axis has struggled to deliver a cloud-based managed service video platform. Video service providers have utilized AVHS for over a decade, and have...

The world's leading video surveillance information source, IPVM provides the best reporting, testing and training for 10,000+ members globally. Dedicated to independent and objective information, we uniquely refuse any and all advertisements, sponsorship and consulting from manufacturers.

About | FAQ | Contact