Testing Color Analytics Performance

By: Ethan Ace, Published on Dec 16, 2013

In an industry known for overhyping its own value, color analytics have been considered pure science fiction, with few, if any, stories from the field to back up manufacturer claims.

In this report, we test Bosch's IVA color analytics to see which practical applications they performed in, and where they failed.

First, in an interior conference room with a human subject:

And an outdoor parking lot/driveway with different colored vehicles:

We varied the precision of the analytics to understand how performance varied, from least to most sensitive using their built-in scale:

  • In nearly all cases, cameras saw colors as lighter than the human eye, making accurate selection of proper hue very difficult. Many objects, despite color, were seen as simply grey.
  • Setting precision too low results in colors adjacent to the desired hue on the color wheel triggering alerts (e.g., yellow and blue triggering when blue is desired).
  • Increasing precision to 2 (moderate), only one step above minimum, removed the majority of false matches from other colors, but also increases misses as well, as colors must be very specific in order to trigger.
  • Increasing precision to 3 or 4 (high or very high) resulted in extreme numbers of missed matches (75% and up) as alerts are triggered only on a small range of hues.
  • In low light, precision over lowest settings results in a nearly 100% failure to detect, as the camera cannot discern colors properly, though the human eye can.
  • In a bright outdoor parking lot scene, vehicles were consistently detected when using precision 1. At precision 2 and above, rules failed to activate at all. 

Recommendations

Given the tradeoffs present between accuracy and false alerts, we do not recommend color analytics be used in critical applications. Alerting on very specific colors will result in many missed matches due to slight differences in shade, hue, or changing lighting and shadows. However, lowering precision results in numerous alerts for colors other than what may be intended. Because of this, these analytics should only be used when looking to reduce the number of alerts seen in less critical applications.

Get Notified of Video Surveillance Breaking News
Get Notified of Video Surveillance Breaking News

It Could Be Worse

The color analytics used were built in to a high end camera (the Bosch NBN-733V) that did quite well in our color fidelity test and has strong low light performance. Performance would likely be even worse on lower quality cameras and used with server based color analytics that have to deal with various cameras who render colors differently.

Configuring Alerts

This screencast reviews configuration of color analytics as well as some common issues we saw with color accuracy:

Selecting Colors

The biggest obstacle to configuring Bosch's color analytics properly is compensating for the difference between what the camera "sees" and the color selected via the color wheel. For example, in this image, the camera detects red in our test subject, as seen in the object properties in the lower right, and triggers the analytics rule:

However, at other times, the camera saw no red, seeing mostly grey instead, failing to detect:

We found two reasons that this may occur:

  • The camera is seeing the subject's pants as the dominant color, which it detect as grey.
  • At other times, background and floor color may bleed into the object properties as he walked across the grey carpet.

However, even against the bright, neutral background of the walls of the conference room, the camera often detected many colors as simply grey, regardless of actual hue. This was true of not only red, but blue, green, orange, purple, etc.

Darker Colors

In our tests, analytics had the most difficult with dark colors such as black, grey, and brown. Partly this was simply because the camera never saw the color as the same darkness seen on the color wheel, mostly due to variances in highlights from lights, etc.

The way dark colors are selected on the color wheel also was very specific by nature, as seen below. Moving the darkness slider lower while selecting white in the center of the wheel results in grey or black. However, it also drastically narrows the colors matched to only those near the center of the wheel, making triggering accurately even more difficult.

Color Precision Guidelines

The color precision setting allows specific hues to be narrowed down to just a few shades, intended to increase accuracy. This setting ranges from 0 (any color allowed) to 4 (only one specific hue allowed), as seen below:

In our tests, we found that precision above 1 greatly reduced matching rates. However, at this level, colors other than the intended hue may trigger alerts, so orange and yellow may trigger when red is selected, for example. At precision 2 and above, however, matching was inconsistent, well below the rates seen with precision 1, if the rule triggered at all, as seen in examples below.

Color Examples

We began in a well lit room, about 160 lux. Our subject walked across the room ten times, reflected in the following results. Match rates were surprisingly consistent regardless of color, especially at precision 1, which triggered in 100% of attempts.

Green, red, and blue all performed simiarly, with 100% accuracy at precision 1, dropping to 50% at precision 2. Beyond this, at precisions 3 and 4, exact shades became too specific to trigger the rule:

Bright neon colors such as the orange and green examples below also performed at 100% with precision 1, but did not trigger at all at precision 2 or above. This was because the camera often saw these colors as bright grey or white, which are excluded at higher precision levels.

Low Light

Lowering light to about 2 lux, accuracy at even precision 1 drops substantially in almost all cases. At this light level, we also tried adjusting the brightness slider to compensate for colors appearing dark in dim light, but this had no effect on performance whatsoever.

First, green drops to about 50% accuracy at precision 1, and is undetected at 2 and above:

 

Detection of red suffers most, with match rates at precision 1 dropping to only 20%, with no triggers at 2 or above:

Blue performed better than either red or green, maintaining about 70% accuracy, though again failing to match beyond precision 1:

Neon green performed best out of all colors in dim light, due to its extreme brightness. At precision 1, it remained 100% accurate. Increasing to precision 2, it maintained a 30% match rate, better than any other color tested:

Finally, orange had an matching rate around 50% at precision 1 in dim light, but was not detected at 2 or above: 

Vehicle Detection Performance 

At precision 1, vehicles were consistently detected due to their larger size. However, due to the vehicle's movement, its color changes as different objects are reflected and shadows move, making the more narrow color ranges of precision 2 and above fail to trigger. This was true using multiple colors of vehicle, including blue:

And red:

1 report cite this report:

Avigilon Appearance Search Aims to Transform Video Surveillance on Jun 20, 2016
After a 'revolutionary' edge storage release this Spring, Avigilon is now aiming to 'transform video surveillance' with its upcoming appearance...
Comments (19) : Members only. Login. or Join.

Related Reports

Mobotix 7 Line Camera Tested on Mar 12, 2020
Mobotix is attempting a turn-around, struggling for years, then releasing the OEMed Move line, Mobotix is aiming for its new 7th generation line to...
Yi Home Camera 3 AI Analytics Tested on Sep 10, 2019
Yi Technology is claiming "new AI features" in its $50 Home Camera 3 "eliminates 'false positives' caused by flying insects, small pets, or light...
Avigilon H5A Analytic Cameras Tested on Oct 07, 2019
Avigilon has released its H5A analytic cameras, claiming to "detect more objects with greater accuracy even in crowded scenes." We tested the...
Bosch Budget 3000i Cameras Tested on Dec 05, 2019
Bosch has long had a hole in its lineup for, as it describes, "competitively-priced cameras". Now, Bosch has released its 3000i series cameras...
Hikvision DeepinView Camera Analytics Tested on May 08, 2019
Hikvision is expanding its 'deep learning' offerings with a new camera series called 'DeepinView' claiming false alarm reduction and improved...
Camera Analytics Shootout 2020 - Avigilon, Axis, Bosch, Dahua, Hanwha, Hikvision, Uniview, Vivotek on Jan 22, 2020
Analytics are hot again, thanks to a slew of AI-powered cameras, but whose analytics really work? And how do these new smart cameras compare to top...
Hikvision Dual Lens Face Recognition Camera Tested on Nov 19, 2019
Hikvision's Dual Lens Facial Recognition camera, claims that it "adopts advanced deep learning algorithm and powerful GPU to realize instant face...
Hikvision Acusense Analytics Tested on Sep 23, 2019
Hikvision touts "The Magic Behind It All" in their new Acusense line are 'deep learning algorithms' inside these cameras and recorders. But how...
Dahua Smart Motion Detection Camera Tested on Mar 03, 2020
Dahua has introduced Smart Motion Detection, AI-based VMD, claiming to use an advanced algorithm to differentiate human and vehicular shapes within...
Anyvision Facial Recognition Tested on Aug 21, 2019
Anyvision is aiming for $1 billion in revenue by 2022, backed by $74 million in funding. But does their performance live up to the hype they have...

Most Recent Industry Reports

The IPVM New Products Online Show April 2020 Opens With 40+ Manufacturers on Mar 31, 2020
IPVM is excited to announce the first New Products Online show, with 40+ manufacturers, to be held April 14 to the 16th, free to IPVM members,...
USA's Feevr Thermal Temperature System Examined on Mar 31, 2020
This US company has burst on to the scene, brashly naming itself 'feevr' and branding itself as a "COVID 19 - AI BASED NON CONTACT THERMAL...
JCI Coronavirus Cuts on Mar 31, 2020
JCI has made coronavirus cuts, the company told employees in an email that IPVM has reviewed. Inside this note, we examine the cuts made, the...
Add Door Operators To Fight Coronavirus on Mar 31, 2020
IPVM recommends that integrators advocate and end-users consider adding door operators to fight the spread of coronavirus. This delivers...
Video Surveillance Business 101 on Mar 30, 2020
This report explains the fundamental elements of the video surveillance business for those new to the industry. This is part of our Video...
FDA Gives Guidance on 'Coronavirus' Thermal Fever Detection Systems on Mar 30, 2020
The US FDA has given IPVM guidance on the use of thermal fever detection systems being marketed for coronavirus, as an explosion of such devices...
Worsen: Integrators Hit Even Harder By Coronavirus on Mar 30, 2020
Integrator's problems have worsened over the past 2 weeks, according to new IPVM survey results. Inside this report, we share statistics and...
Pivot3 Mass Layoffs on Mar 27, 2020
Pivot3 has conducted mass layoffs, the culmination of grand hopes, a quarter of a billion dollars in VC funding, and multiple failures to gain...
Athena CEO Criticizes 'Deplorable' 'Nitpicking', IPVM Refutes on Mar 27, 2020
UPDATE: NBC News Report Cites IPVM On Coronavirus 'Fever Detection' Cameras Athena Security's CEO Lisa Falzone has strongly objected to IPVM's...
Hikvision Admits Sanctions Harming Its Financial Performance on Mar 27, 2020
While Hikvision initially downplayed being sanctioned for human rights abuses, the company is now admitting a significant impact in a new PRC...