Google Clips Camera Tested - Weak AI

By Rob Kilpatrick, Published on Mar 12, 2018

The Google Clips is drawing a lot of interest, especially given its use of artificial intelligence (AI) to deliver a 'smart' camera. Indeed, Google claims:

We purchased one and tested these claims. Inside, we report our findings on:

  • Camera pricing
  • App usability
  • Inconsistent suggested clips
  • Face/pet/expression recognition issues
  • Camera hardware

Most notably, we examine how the camera was not very 'smart' at all.

Summary

However, in our tests, we saw several issues contradicting this claim:

  • Clips were generated by practically any sort of motion (people, pets, etc.) or even in still scenes (empty rooms, walls, etc.).
  • Further, while it claims to suggest clips that users will like most, these suggestions had seemingly no rhyme or reason, with nearly identical clips suggested and not suggested directly after each other.
  • Finally, while Google also claims that expression recognition is used to determine which clips to capture, we saw no difference in the number of captures regardless of expression (smiling, laughing, neutral, etc.).

Despite these issues, many users may still find the camera useful, as it may be placed in a stationary location during events or when playing with children or pets, automatically taking photos without them needing to use their phones.

Note: Claims To Improve Over Time

Google claims that Clips improve over time, with the camera learning what is important to users. We saw no difference in performance during our testing, ~5 days. However, it may improve with additional time and use.

Get Notified of Video Surveillance Breaking News
Get Notified of Video Surveillance Breaking News

Pricing

The Google Clips camera can be purchased from the Google Store for $249 USD, or as a bundle with a tripod mount case for an additional $14.99.

Physical overview

The video below provides a physical overview of the google clips camera and included case:

Clips Mobile App Overview

The Clips mobile app is essentially the only UI to the camera. The main interface of the app is a list of captured clips, which users scroll through to review, swipe to save/delete, etc., and so-called "Suggested Clips" are highlighted with a star icon, though we found these suggestions poor (discussed below). The camera may also be viewed live for positioning and switched from video to gif mode, etc.

We review these functions in this video:

Failure: Clips Generated On Anything/Nothing

According to Google, clips should be created when the camera sees human faces and pets. However, during our tests, the camera created several clips which contained none of these, including an empty room, object waved in front of the camera, people without faces visible, etc.

Because this feature worked so inconsistently, the Clips camera functions more like a camera recording on motion might, recording when it perceives changes in the scene.

For example, while we were writing this report, the camera captured six images of the side of a file cabinet:

Note that there is no way to verify what the camera "saw" to trigger a clip, nor is there any clear sensitivity adjustment.

Inconsistent Suggested Clips

Google states that the camera suggests specific clips based on recognizing familiar faces or pets, which it then highlights with a star icon in the top right corner of the clip. However, in our tests, this was incredibly inconsistent. The app suggested clips, but others immediately before and after those suggested were not tagged, despite containing the same subject(s) and scene.

For example, the image below shows two clips taken back to back, with one recommended and one not, but the content nearly identical.

The same was true of clips with pets. The clip below left was recommended, but the very similar clip on the right was not.

Failed Expression Recognition

Google claims that facial expressions are one criteria which it uses to capture clips. However, in our tests, clips were generated regardless of expression. Users smiling, laughing, frowning, etc., were just as likely to trigger a clip as those with neutral expressions.

Camera Teardown

In the video below we review components of the Clips camera. The camera's AI is powered by a Movidius MA2150 "Vision Processing Unit" (VPU), visible on the front of the board (see Intel Movidius Targets Video Surveillance Market). The only other chip with visible markings is its 16GB solid-state storage, on the rear of the board.

Note that in order to tear down the camera, it must be broken, with no way to remove and replace the lens or front cover and delicate ribbon cables connecting components. The teardown was conducted after performance testing.

Test Conditions

Note that our tests were performed in areas with few subjects present, with activity ranging from sporadic to constant. If the camera were used in an area with a crowd of people, such as a party, performance may differ (known people suggested instead of unknown, fewer or more clips shown, etc.), though given how much it struggled with simple scenes, we are skeptical of how it would perform in harder ones.

Firmware Versions Used

The following firmware versions were used during testing:

  • App version: 1.3.185005366
  • Camera Version: 1.3.5.1185005431

1 report cite this report:

Worst Products Tested In Past Year on Jan 09, 2019
IPVM has done over 100 tests in the past year. But which products performed...

Comments (5)

Only IPVM Members may comment. Login or Join.

As an owner of the Google Pixel 2 XL, I can assure you that the AI will improve noticeably over development cycles (which is pretty quick).

Promise?

Cross his heart and hope to lose market share.

Interesting, lets see whether it goes towards another flagship Google product and revolutionize entire CCTV/camera industry;  ... ... or just gets killed by Google itself. 

Related Reports

Facial Recognition: Weak Sales, Anti Regulation, No Favorite, Says Security Integrators on Jul 07, 2020
While facial recognition has gained greater prominence, a new IPVM study of...
Uniview Deep Learning Camera Tested on Jul 14, 2020
Uniview's intrusion analytics have performed poorly in our shootouts. Now,...
Anixter Runs Fake Coronavirus Marketing Using Shutterstock Watermarked Images on Jul 24, 2020
Coronavirus faked marketing is regrettably commonplace right now but Anixter...
South Korea Bus Outdoor Temperature Screening Endangers Public on Aug 26, 2020
These $80,000+ South Korea bus stations have gained world-wide attention but...
Gait Recognition Examined on Sep 14, 2020
Facial recognition faces increasing ethical and political criticisms while...
Six Flags' FDA Violating Outdoor Dahua Fever Cameras on Oct 26, 2020
As Six Flags scrambled to reopen parks amid plummeting revenues caused by the...
Faulty Hikvision Cali Colombia Fever Camera Implementation on Jul 20, 2020
The mayor of one of Colombia's largest cities has promoted a faulty Hikvision...
Hanwha AI Analytics Camera Tested on Aug 11, 2020
Hanwha has released their Wisenet P AI camera, adding person and vehicle...
Beware Rigged China Fever Cameras on Sep 08, 2020
Many China fever camera manufacturers have rigged algorithms dynamically...
Ubiquiti Access Control Tested on Oct 21, 2020
Ubiquiti has become one of the most widely used wireless and switch providers...
No Blackbody Mistake, Half Million Dollar, Hikvision Fever Camera System in Georgia on Sep 16, 2020
A Georgia school district touted buying Hikvision fever screening "about...
Face Masks Increase Face Recognition Errors Says NIST on Aug 04, 2020
COVID-19 has led to widespread facemask use, which as IPVM testing has shown...
Briefcam Responsible Use Examined on Aug 24, 2020
While mega-companies Amazon, IBM, and Microsoft have been criticized for...
ADT Slides Back, Disappointing Results, Poor Commercial Performance on Aug 06, 2020
While ADT had an incredible start to the week, driven by the Google...
Top Video Surveillance Service Call Problems 2020 on Oct 23, 2020
3 primary and 4 secondary issues stood out as causing the most problems when...

Recent Reports

Recruiters Online Show LIVE Today! on Oct 29, 2020
IPVM's 7th online show resumes today with 12 recruiters presenting themselves...
Hikvision AcuSense G2 Camera Test on Oct 29, 2020
Hikvision has released their next generation of AcuSense analytic cameras...
Biggest Problems Selling Access Control 2020 on Oct 29, 2020
Access control can cause integrators big headaches. What practical issues do...
Taiwan Geovision AI Analytics and NDAA Examined on Oct 29, 2020
Taiwan manufacturer Geovision's revenue has been falling for years. However,...
Bedside Cough and Sneeze Detector (Sound Intelligence and CLB) on Oct 28, 2020
Coronavirus has increased interest in detecting symptoms such as fever and...
Fever Tablet Thermal Sensors Examined (Melexis) on Oct 28, 2020
Fever tablet suppliers heavily rely on the accuracy and specs of...
Verkada Fires 3 on Oct 28, 2020
Verkada has fired three employees over an incident where female colleagues...
Eagle Eye Networks Raises $40 Million on Oct 27, 2020
Eagle Eye has raised $40 million aiming to "reinvent video...
Hikvision Q3 2020 Global Revenue Rises, US Revenue Falls on Oct 27, 2020
While Hikvision's global revenue rises driven by domestic recovery, its US...
VICE Investigates Verkada's Harassing "RawVerkadawgz" on Oct 26, 2020
This month, IPVM investigated Verkada's sexism, discrimination, and cultural...
Six Flags' FDA Violating Outdoor Dahua Fever Cameras on Oct 26, 2020
As Six Flags scrambled to reopen parks amid plummeting revenues caused by the...
ISC Brasil Digital Experience 2020 Report on Oct 23, 2020
ISC Brasil 2020 rebranded itself to ISC Digital Experience and, like its...
Top Video Surveillance Service Call Problems 2020 on Oct 23, 2020
3 primary and 4 secondary issues stood out as causing the most problems when...
GDPR Impact On Temperature / Fever Screening Explained on Oct 22, 2020
What impact does GDPR have on temperature screening? Do you risk a GDPR fine...
Security And Safety Things (S&ST) Tested on Oct 22, 2020
S&ST, a Bosch spinout, is spending tens of millions of dollars aiming to...