Testing Color Analytics Performance

By: Ethan Ace, Published on Dec 16, 2013

In an industry known for overhyping its own value, color analytics have been considered pure science fiction, with few, if any, stories from the field to back up manufacturer claims.

In this report, we test Bosch's IVA color analytics to see which practical applications they performed in, and where they failed.

First, in an interior conference room with a human subject:

And an outdoor parking lot/driveway with different colored vehicles:

We varied the precision of the analytics to understand how performance varied, from least to most sensitive using their built-in scale:

  • In nearly all cases, cameras saw colors as lighter than the human eye, making accurate selection of proper hue very difficult. Many objects, despite color, were seen as simply grey.
  • Setting precision too low results in colors adjacent to the desired hue on the color wheel triggering alerts (e.g., yellow and blue triggering when blue is desired).
  • Increasing precision to 2 (moderate), only one step above minimum, removed the majority of false matches from other colors, but also increases misses as well, as colors must be very specific in order to trigger.
  • Increasing precision to 3 or 4 (high or very high) resulted in extreme numbers of missed matches (75% and up) as alerts are triggered only on a small range of hues.
  • In low light, precision over lowest settings results in a nearly 100% failure to detect, as the camera cannot discern colors properly, though the human eye can.
  • In a bright outdoor parking lot scene, vehicles were consistently detected when using precision 1. At precision 2 and above, rules failed to activate at all. 

Recommendations

Given the tradeoffs present between accuracy and false alerts, we do not recommend color analytics be used in critical applications. Alerting on very specific colors will result in many missed matches due to slight differences in shade, hue, or changing lighting and shadows. However, lowering precision results in numerous alerts for colors other than what may be intended. Because of this, these analytics should only be used when looking to reduce the number of alerts seen in less critical applications.

Get Notified of Video Surveillance Breaking News
Get Notified of Video Surveillance Breaking News

It Could Be Worse

The color analytics used were built in to a high end camera (the Bosch NBN-733V) that did quite well in our color fidelity test and has strong low light performance. Performance would likely be even worse on lower quality cameras and used with server based color analytics that have to deal with various cameras who render colors differently.

Configuring Alerts

This screencast reviews configuration of color analytics as well as some common issues we saw with color accuracy:

Selecting Colors

The biggest obstacle to configuring Bosch's color analytics properly is compensating for the difference between what the camera "sees" and the color selected via the color wheel. For example, in this image, the camera detects red in our test subject, as seen in the object properties in the lower right, and triggers the analytics rule:

However, at other times, the camera saw no red, seeing mostly grey instead, failing to detect:

We found two reasons that this may occur:

  • The camera is seeing the subject's pants as the dominant color, which it detect as grey.
  • At other times, background and floor color may bleed into the object properties as he walked across the grey carpet.

However, even against the bright, neutral background of the walls of the conference room, the camera often detected many colors as simply grey, regardless of actual hue. This was true of not only red, but blue, green, orange, purple, etc.

Darker Colors

In our tests, analytics had the most difficult with dark colors such as black, grey, and brown. Partly this was simply because the camera never saw the color as the same darkness seen on the color wheel, mostly due to variances in highlights from lights, etc.

The way dark colors are selected on the color wheel also was very specific by nature, as seen below. Moving the darkness slider lower while selecting white in the center of the wheel results in grey or black. However, it also drastically narrows the colors matched to only those near the center of the wheel, making triggering accurately even more difficult.

Color Precision Guidelines

The color precision setting allows specific hues to be narrowed down to just a few shades, intended to increase accuracy. This setting ranges from 0 (any color allowed) to 4 (only one specific hue allowed), as seen below:

In our tests, we found that precision above 1 greatly reduced matching rates. However, at this level, colors other than the intended hue may trigger alerts, so orange and yellow may trigger when red is selected, for example. At precision 2 and above, however, matching was inconsistent, well below the rates seen with precision 1, if the rule triggered at all, as seen in examples below.

Color Examples

We began in a well lit room, about 160 lux. Our subject walked across the room ten times, reflected in the following results. Match rates were surprisingly consistent regardless of color, especially at precision 1, which triggered in 100% of attempts.

Green, red, and blue all performed simiarly, with 100% accuracy at precision 1, dropping to 50% at precision 2. Beyond this, at precisions 3 and 4, exact shades became too specific to trigger the rule:

Bright neon colors such as the orange and green examples below also performed at 100% with precision 1, but did not trigger at all at precision 2 or above. This was because the camera often saw these colors as bright grey or white, which are excluded at higher precision levels.

Low Light

Lowering light to about 2 lux, accuracy at even precision 1 drops substantially in almost all cases. At this light level, we also tried adjusting the brightness slider to compensate for colors appearing dark in dim light, but this had no effect on performance whatsoever.

First, green drops to about 50% accuracy at precision 1, and is undetected at 2 and above:

 

Detection of red suffers most, with match rates at precision 1 dropping to only 20%, with no triggers at 2 or above:

Blue performed better than either red or green, maintaining about 70% accuracy, though again failing to match beyond precision 1:

Neon green performed best out of all colors in dim light, due to its extreme brightness. At precision 1, it remained 100% accurate. Increasing to precision 2, it maintained a 30% match rate, better than any other color tested:

Finally, orange had an matching rate around 50% at precision 1 in dim light, but was not detected at 2 or above: 

Vehicle Detection Performance 

At precision 1, vehicles were consistently detected due to their larger size. However, due to the vehicle's movement, its color changes as different objects are reflected and shadows move, making the more narrow color ranges of precision 2 and above fail to trigger. This was true using multiple colors of vehicle, including blue:

And red:

1 report cite this report:

Avigilon Appearance Search Aims to Transform Video Surveillance on Jun 20, 2016
After a 'revolutionary' edge storage release this Spring, Avigilon is now...
Comments (19) : Members only. Login. or Join.

Related Reports

Uniview Color Low Light Camera Tested Vs Hikvision ColorVu on Jan 07, 2020
Hikvision has been marketing ColorVu aggressively. Now, Hikvision domestic...
IronYun AI Analytics Tested on Feb 17, 2020
Taiwan / US startup IronYun has raised tens of millions for its "mission to...
Vivotek LPR Camera Tested on Apr 15, 2020
Vivotek has historically sold license plate capture cameras but not LPR. Now,...
BriefCam Video Analytics Tested on Jan 06, 2020
BriefCam, acquired by Canon in 2018, is one of the most commonly used video...
Video Analytics 101 on Mar 16, 2020
This guide teaches the fundamentals of video surveillance...
Color Low Light Mega Camera Shootout - Avigilon, Axis, Bosch, Dahua, Hanwha, Hikvision, Panasonic, Speco, Sony, Vivotek on Nov 12, 2019
This is the biggest color low light shootout ever, testing 20+ super low...
Tiandy Super Starlight Cameras Tested on Apr 30, 2020
Tiandy is taking aim at China competitors Dahua and Hikvision, with a new...
Hikvision Temperature Screening Tested on May 20, 2020
Hikvision has ramped up the promotion of its 'temperature screening' system,...
Integrated IR Camera Shootout 2020 - Avigilon, Axis, Bosch, Dahua, Hanwha, Hikvision, Panasonic, Uniview, Vivotek on Jan 30, 2020
The best and worst cameras tested in this IPVM shootout showed major...
Mobotix Thermal Detection Camera Tested on Jun 09, 2020
For years Mobotix has struggled but now sales are surging driven by Mobotix's...
Cheap Camera Problems at Night on Feb 19, 2020
Cheap cameras generally have problems at night, despite the common perception...
Actual Coronavirus Testing Options Examined on Aug 13, 2020
Fever cameras have emerged as an indirect and flawed way to test for...
Uniview Deep Learning Camera Tested on Jul 14, 2020
Uniview's intrusion analytics have performed poorly in our shootouts. Now,...
Avigilon Elevated Temperature Detection Camera Tested on Sep 17, 2020
Avigilon has entered the temperature screening market with the release of...
Directory of 206 "Fever" Camera Suppliers on Aug 04, 2020
This directory provides a list of "Fever" scanning thermal camera providers...

Recent Reports

OnTech Smart Services Partners With Google and Amazon To Compete With Integrators on Sep 25, 2020
A pain point for many homeowners to use consumer security and surveillance is...
The Future of Metalens For Video Surveillance Cameras - MIT / UMass / Immervision on Sep 25, 2020
Panoramic cameras using 'fisheye' lens have become commonplace in video...
Hikvision Sues Over Brazilian Airport Loss on Sep 24, 2020
Hikvision was excluded from a Brazilian airport project because it is owned...
China General Chamber of Commerce Calls Out US Politics on Sep 24, 2020
While US-China relations at an all-time low, optimism about relations...
Verkada Disruptive Embedded Live Help on Sep 24, 2020
Call up your integrator? Have someone come by the next day? Verkada is...
IP Networking Course Fall 2020 - Last Chance - Register Now on Sep 23, 2020
Today is the last chance to register for the only IP networking course...
Drain Wire For Access Control Reader Tutorial on Sep 23, 2020
An easy-to-miss cabling specification plays a key role in access control, yet...
Norway Council of Ethics Finds Hikvision Human Rights Abuses "Ongoing" on Sep 23, 2020
Hikvision's involvement in "serious human rights abuse" in Xinjiang is...
IPVM Camera Calculator User Manual / Guide on Sep 23, 2020
Learn how to use the IPVM Camera Calculator (updated for Version 3.1). The...
Installation Course Fall 2020 - Save $50 - Last Chance on Sep 22, 2020
This is a unique installation course in a market where little practical...
SimpliSafe Business Security Launched Examined on Sep 22, 2020
SimpliSafe has launched "SimpliSafe Business Security" that the company...
FLIR CEO: Many New Fever Entrants "Making Claims That The Science Just Won't Support" on Sep 22, 2020
FLIR's CEO joins a growing number calling out risks with fever / screening...
China Bems Temperature Measurement Terminal Tested on Sep 22, 2020
Guangzhou Bems (brand Benshi) is the manufacturer behind temperature...
Axis Exports To China Police Criticized By Amnesty International on Sep 21, 2020
Axis Communications and other EU surveillance providers are under fire from...
Milestone XProtect on AWS Tested on Sep 21, 2020
Milestone finally launched multiple cloud solutions in 2020, taking a...