Testing Color Analytics Performance

In an industry known for overhyping its own value, color analytics have been considered pure science fiction, with few, if any, stories from the field to back up manufacturer claims.

In this report, we test Bosch's IVA color analytics to see which practical applications they performed in, and where they failed.

First, in an interior conference room with a human subject:

And an outdoor parking lot/driveway with different colored vehicles:

We varied the precision of the analytics to understand how performance varied, from least to most sensitive using their built-in scale:

• In nearly all cases, cameras saw colors as lighter than the human eye, making accurate selection of proper hue very difficult. Many objects, despite color, were seen as simply grey.
• Setting precision too low results in colors adjacent to the desired hue on the color wheel triggering alerts (e.g., yellow and blue triggering when blue is desired).
• Increasing precision to 2 (moderate), only one step above minimum, removed the majority of false matches from other colors, but also increases misses as well, as colors must be very specific in order to trigger.
• Increasing precision to 3 or 4 (high or very high) resulted in extreme numbers of missed matches (75% and up) as alerts are triggered only on a small range of hues.
• In low light, precision over lowest settings results in a nearly 100% failure to detect, as the camera cannot discern colors properly, though the human eye can.
• In a bright outdoor parking lot scene, vehicles were consistently detected when using precision 1. At precision 2 and above, rules failed to activate at all. 

Recommendations

Given the tradeoffs present between accuracy and false alerts, we do not recommend color analytics be used in critical applications. Alerting on very specific colors will result in many missed matches due to slight differences in shade, hue, or changing lighting and shadows. However, lowering precision results in numerous alerts for colors other than what may be intended. Because of this, these analytics should only be used when looking to reduce the number of alerts seen in less critical applications.

It Could Be Worse

The color analytics used were built in to a high end camera (the Bosch NBN-733V) that did quite well in our color fidelity test and has strong low light performance. Performance would likely be even worse on lower quality cameras and used with server based color analytics that have to deal with various cameras who render colors differently.

Configuring Alerts

This screencast reviews configuration of color analytics as well as some common issues we saw with color accuracy:

Selecting Colors

The biggest obstacle to configuring Bosch's color analytics properly is compensating for the difference between what the camera "sees" and the color selected via the color wheel. For example, in this image, the camera detects red in our test subject, as seen in the object properties in the lower right, and triggers the analytics rule:

However, at other times, the camera saw no red, seeing mostly grey instead, failing to detect:

We found two reasons that this may occur:

• The camera is seeing the subject's pants as the dominant color, which it detect as grey.
• At other times, background and floor color may bleed into the object properties as he walked across the grey carpet.

However, even against the bright, neutral background of the walls of the conference room, the camera often detected many colors as simply grey, regardless of actual hue. This was true of not only red, but blue, green, orange, purple, etc.

Darker Colors

In our tests, analytics had the most difficult with dark colors such as black, grey, and brown. Partly this was simply because the camera never saw the color as the same darkness seen on the color wheel, mostly due to variances in highlights from lights, etc.

The way dark colors are selected on the color wheel also was very specific by nature, as seen below. Moving the darkness slider lower while selecting white in the center of the wheel results in grey or black. However, it also drastically narrows the colors matched to only those near the center of the wheel, making triggering accurately even more difficult.

Color Precision Guidelines

The color precision setting allows specific hues to be narrowed down to just a few shades, intended to increase accuracy. This setting ranges from 0 (any color allowed) to 4 (only one specific hue allowed), as seen below:

In our tests, we found that precision above 1 greatly reduced matching rates. However, at this level, colors other than the intended hue may trigger alerts, so orange and yellow may trigger when red is selected, for example. At precision 2 and above, however, matching was inconsistent, well below the rates seen with precision 1, if the rule triggered at all, as seen in examples below.

Color Examples

We began in a well lit room, about 160 lux. Our subject walked across the room ten times, reflected in the following results. Match rates were surprisingly consistent regardless of color, especially at precision 1, which triggered in 100% of attempts.

Green, red, and blue all performed simiarly, with 100% accuracy at precision 1, dropping to 50% at precision 2. Beyond this, at precisions 3 and 4, exact shades became too specific to trigger the rule:

Bright neon colors such as the orange and green examples below also performed at 100% with precision 1, but did not trigger at all at precision 2 or above. This was because the camera often saw these colors as bright grey or white, which are excluded at higher precision levels.

Low Light

Lowering light to about 2 lux, accuracy at even precision 1 drops substantially in almost all cases. At this light level, we also tried adjusting the brightness slider to compensate for colors appearing dark in dim light, but this had no effect on performance whatsoever.

First, green drops to about 50% accuracy at precision 1, and is undetected at 2 and above:

 

Detection of red suffers most, with match rates at precision 1 dropping to only 20%, with no triggers at 2 or above:

Blue performed better than either red or green, maintaining about 70% accuracy, though again failing to match beyond precision 1:

Neon green performed best out of all colors in dim light, due to its extreme brightness. At precision 1, it remained 100% accurate. Increasing to precision 2, it maintained a 30% match rate, better than any other color tested:

Finally, orange had an matching rate around 50% at precision 1 in dim light, but was not detected at 2 or above: 

Vehicle Detection Performance 

At precision 1, vehicles were consistently detected due to their larger size. However, due to the vehicle's movement, its color changes as different objects are reflected and shadows move, making the more narrow color ranges of precision 2 and above fail to trigger. This was true using multiple colors of vehicle, including blue:

And red: