Testing Color Analytics Performance

By: Ethan Ace, Published on Dec 16, 2013

In an industry known for overhyping its own value, color analytics have been considered pure science fiction, with few, if any, stories from the field to back up manufacturer claims.

In this report, we test Bosch's IVA color analytics to see which practical applications they performed in, and where they failed.

First, in an interior conference room with a human subject:

And an outdoor parking lot/driveway with different colored vehicles:

We varied the precision of the analytics to understand how performance varied, from least to most sensitive using their built-in scale:

  • In nearly all cases, cameras saw colors as lighter than the human eye, making accurate selection of proper hue very difficult. Many objects, despite color, were seen as simply grey.
  • Setting precision too low results in colors adjacent to the desired hue on the color wheel triggering alerts (e.g., yellow and blue triggering when blue is desired).
  • Increasing precision to 2 (moderate), only one step above minimum, removed the majority of false matches from other colors, but also increases misses as well, as colors must be very specific in order to trigger.
  • Increasing precision to 3 or 4 (high or very high) resulted in extreme numbers of missed matches (75% and up) as alerts are triggered only on a small range of hues.
  • In low light, precision over lowest settings results in a nearly 100% failure to detect, as the camera cannot discern colors properly, though the human eye can.
  • In a bright outdoor parking lot scene, vehicles were consistently detected when using precision 1. At precision 2 and above, rules failed to activate at all. 

Recommendations

Given the tradeoffs present between accuracy and false alerts, we do not recommend color analytics be used in critical applications. Alerting on very specific colors will result in many missed matches due to slight differences in shade, hue, or changing lighting and shadows. However, lowering precision results in numerous alerts for colors other than what may be intended. Because of this, these analytics should only be used when looking to reduce the number of alerts seen in less critical applications.

Get Notified of Video Surveillance Breaking News
Get Notified of Video Surveillance Breaking News

It Could Be Worse

The color analytics used were built in to a high end camera (the Bosch NBN-733V) that did quite well in our color fidelity test and has strong low light performance. Performance would likely be even worse on lower quality cameras and used with server based color analytics that have to deal with various cameras who render colors differently.

Configuring Alerts

This screencast reviews configuration of color analytics as well as some common issues we saw with color accuracy:

Selecting Colors

The biggest obstacle to configuring Bosch's color analytics properly is compensating for the difference between what the camera "sees" and the color selected via the color wheel. For example, in this image, the camera detects red in our test subject, as seen in the object properties in the lower right, and triggers the analytics rule:

However, at other times, the camera saw no red, seeing mostly grey instead, failing to detect:

We found two reasons that this may occur:

  • The camera is seeing the subject's pants as the dominant color, which it detect as grey.
  • At other times, background and floor color may bleed into the object properties as he walked across the grey carpet.

However, even against the bright, neutral background of the walls of the conference room, the camera often detected many colors as simply grey, regardless of actual hue. This was true of not only red, but blue, green, orange, purple, etc.

Darker Colors

In our tests, analytics had the most difficult with dark colors such as black, grey, and brown. Partly this was simply because the camera never saw the color as the same darkness seen on the color wheel, mostly due to variances in highlights from lights, etc.

The way dark colors are selected on the color wheel also was very specific by nature, as seen below. Moving the darkness slider lower while selecting white in the center of the wheel results in grey or black. However, it also drastically narrows the colors matched to only those near the center of the wheel, making triggering accurately even more difficult.

Color Precision Guidelines

The color precision setting allows specific hues to be narrowed down to just a few shades, intended to increase accuracy. This setting ranges from 0 (any color allowed) to 4 (only one specific hue allowed), as seen below:

In our tests, we found that precision above 1 greatly reduced matching rates. However, at this level, colors other than the intended hue may trigger alerts, so orange and yellow may trigger when red is selected, for example. At precision 2 and above, however, matching was inconsistent, well below the rates seen with precision 1, if the rule triggered at all, as seen in examples below.

Color Examples

We began in a well lit room, about 160 lux. Our subject walked across the room ten times, reflected in the following results. Match rates were surprisingly consistent regardless of color, especially at precision 1, which triggered in 100% of attempts.

Green, red, and blue all performed simiarly, with 100% accuracy at precision 1, dropping to 50% at precision 2. Beyond this, at precisions 3 and 4, exact shades became too specific to trigger the rule:

Bright neon colors such as the orange and green examples below also performed at 100% with precision 1, but did not trigger at all at precision 2 or above. This was because the camera often saw these colors as bright grey or white, which are excluded at higher precision levels.

Low Light

Lowering light to about 2 lux, accuracy at even precision 1 drops substantially in almost all cases. At this light level, we also tried adjusting the brightness slider to compensate for colors appearing dark in dim light, but this had no effect on performance whatsoever.

First, green drops to about 50% accuracy at precision 1, and is undetected at 2 and above:

 

Detection of red suffers most, with match rates at precision 1 dropping to only 20%, with no triggers at 2 or above:

Blue performed better than either red or green, maintaining about 70% accuracy, though again failing to match beyond precision 1:

Neon green performed best out of all colors in dim light, due to its extreme brightness. At precision 1, it remained 100% accurate. Increasing to precision 2, it maintained a 30% match rate, better than any other color tested:

Finally, orange had an matching rate around 50% at precision 1 in dim light, but was not detected at 2 or above: 

Vehicle Detection Performance 

At precision 1, vehicles were consistently detected due to their larger size. However, due to the vehicle's movement, its color changes as different objects are reflected and shadows move, making the more narrow color ranges of precision 2 and above fail to trigger. This was true using multiple colors of vehicle, including blue:

And red:

1 report cite this report:

Avigilon Appearance Search Aims to Transform Video Surveillance on Jun 20, 2016
After a 'revolutionary' edge storage release this Spring, Avigilon is now aiming to 'transform video surveillance' with its upcoming appearance...
Comments (19) : Members only. Login. or Join.

Related Reports

Vivotek LPR Camera Tested on Apr 15, 2020
Vivotek has historically sold license plate capture cameras but not LPR. Now, they have released their own cameras with embedded LPR. How do they...
Camera Analytics Shootout 2020 - Avigilon, Axis, Bosch, Dahua, Hanwha, Hikvision, Uniview, Vivotek on Jan 22, 2020
Analytics are hot again, thanks to a slew of AI-powered cameras, but whose analytics really work? And how do these new smart cameras compare to top...
Hikvision Temperature Screening Tested on May 20, 2020
Hikvision has ramped up the promotion of its 'temperature screening' system, including their salespeople arguing for no blackbody needed. But how...
Directory of 164 "Fever" Camera Suppliers on Jul 09, 2020
This directory provides a list of "Fever" scanning thermal camera providers to help you see and research what options are available. There are...
Integrated IR Camera Shootout 2020 - Avigilon, Axis, Bosch, Dahua, Hanwha, Hikvision, Panasonic, Uniview, Vivotek on Jan 30, 2020
The best and worst cameras tested in this IPVM shootout showed major differences in quality and performance. While many view IR as being a...
IronYun AI Analytics Tested on Feb 17, 2020
Taiwan / US startup IronYun has raised tens of millions for its "mission to be the leading Artificial Intelligence, big data video software as a...
Mobotix Thermal Detection Camera Tested on Jun 09, 2020
For years Mobotix has struggled but now sales are surging driven by Mobotix's thermal camera sales as a response to coronavirus, with both their...
Cheap Camera Problems at Night on Feb 19, 2020
Cheap cameras generally have problems at night, despite the common perception that integrated IR makes cameras mostly the same, according to new...
Tiandy Super Starlight Cameras Tested on Apr 30, 2020
Tiandy is taking aim at China competitors Dahua and Hikvision, with a new series of "super starlight" cameras, including white light illumination,...
Verkada Falsely Claims "First Native Cloud-based Access Control and Video Security Solution" on Jun 18, 2020
Verkada's false claims continue, this time to be the first native cloud-based access and video solution. There are a number of manufacturers...

Most Recent Industry Reports

Ban Rules Released: Use Dahua or Hikvision, No Government Contracts on Jul 13, 2020
The US government has released the rules implementing the "Prohibition on Contracting with Entities Using" Dahua, Hikvision, and Huawei based...
JCI Sues Genetec For Patent Infringement on Jul 13, 2020
Surprisingly, security giant JCI has sued their partner, security software developer Genetec, for patent infringement. Inside this note,...
Remove Dahua and Hikvision Equipment "Immediately" Or Else Banned From US Government Contracts on Jul 10, 2020
The US government has directed contractors to remove covered equipment, such as Dahua, Hikvision, and Huawei Hisilicon products, "immediately." If...
ZeroEyes Presents Firearm Detection Video Analytics on Jul 09, 2020
ZeroEyes presented its Firearm detection Video Analytics system at the May 2020 IPVM Startups show. A 30-minute video from ZeroEyes...
Directory of 164 "Fever" Camera Suppliers on Jul 09, 2020
This directory provides a list of "Fever" scanning thermal camera providers to help you see and research what options are available. There are...
Clinton Public View Monitor (PVM) Mask Detection Tested on Jul 09, 2020
Face mask detection, or more specifically not wearing one, is expanding amidst the pandemic. Clinton Electronics has added this capability to their...
These Florida Real Estate Agents Are Now Selling "SafeCheck USA" Temperature Detectors on Jul 09, 2020
The "Kakon Brothers", William and Nathan, are self-described "south Florida Power Agents specializing in Luxury Real Estate" who "have closed over...
Avigilon ACC Cloud Tested on Jul 08, 2020
Avigilon merged Blue and ACC, adding VSaaS features to its on-premise VMS, offering remote video and health monitoring that was previously limited...
Hikvision's India Dominance Faces Threat on Jul 08, 2020
While Hikvision has become a dominant video surveillance provider in India, recent tension between the governments of India and the PRC is...
The US Fight Over Facial Recognition Explained on Jul 08, 2020
The controversy around facial recognition has grown significantly in 2020, with Congress members and activists speaking out against it while video...