Amazon Go Store Tested

Published Nov 13, 2018 14:37 PM
PUBLIC - This article does not require an IPVM subscription. Feel free to share.

[2023 Update: Amazon never got anywhere close to the planned thousands of stores and now has started to shut some, citing economic issues.]

IPVM's trip to and testing of Amazon Go's San Francisco store shows a number of significant operational and economic issues that undermine the hype.

IPVM Image

While the press has overwhelmingly fawned at Amazon Go's store and the report that Amazon will deploy 3,000 of these stores in the next few years, our review of the operation points to significant challenges in turning this into a viable enterprise.

In this note, we examine:

  • The time delay and behind the scenes human intervention involved
  • The density and cost of sensors used
  • Amazon mischarged IPVM during testing
  • Limitations in the system's capabilities
  • Video surveillance and access control used
  • Positives for Amazon and Go customers
  • Outlook on Amazon Go stores

Executive Summary

Given the massive amount of:

  • Infrastructure involved - hundreds of large sensors for a small convenience store
  • Computing and electricity consumed - undisclosed but clearly heavy amount of back-end processing for these hundreds of sensors
  • And most notably human operators reviewing transactions

IPVM is skeptical that Amazon is anywhere near close to being able to run Go stores as a profitable business. Now, given Amazon's scale and ability to absorb losses, this may not be an issue to them but it does undermine the hype that this is economically viable for real businesses.

Density And Cost of Sensors

The most notable hardware differences for the stores are their uses of turnstiles to enter (which are easy to see) and their mass of hundreds of sensors overhead that most people ignore. This image from the entrance overviews the sensors laid out:

IPVM Image

And this image shows how many sensors are needed just to cover the ~10' long chips section:

IPVM Image

The following video shows the sensors across half the back side of the store:

Sensors are square shaped, ~1' per side, with what appears to be 2 imagers:

IPVM Image

These sensors are put right above every shelf as well as throughout the aisles. Based on limited Amazon explanations, they read / detect what objects each user is picking up.

Most Important - Receipt Delay and Human Review

The most surprising and under-reported element is that receipts take 5 - 10 minutes normally to be generated. By contrast, Amazon's own marketing video shows it to be done instantaneously. And CNET who just a few weeks ago went to the same store we did, hides that as well, as the video excerpt below shows:

CNET edits it to make it look instant but the timestamps from their own video shows it was captured 25 minutes later:

IPVM Image

Indeed, long delays for receipts are normal, according to 3 different Amazon Go employees who spoke to us. They even explained that they have people reviewing transactions though they either did not know or were not authorized to explain how often they did so.

However, 5+ minutes is an eternity for computers whose timescale is the milliseconds or less to process information. Amazon is not letting on to how much humans need to be involved but such a delay points to serious concern for errors and the need for human intervention.

Mischarge / Error

Amazon charged us for a bottle of Teriyaki sauce instead of a bottle of water. In fairness to Amazon, this was done on purpose to test the system but even though they took 3 extra hours to have humans analyze this, they still made a mistake:

IPVM Image

Here is what happened:

  • Take water bottle from back shelf
  • Walk to front shelf, take Teriyaki sauce off shelf
  • Put down water bottle in the Teriyaki sauce column
  • Go back to back shelf, put Teriyaki sauce in water bottle column
  • Return to front shelf, grab water from Teriyaki sauce column, put in pocket, leave store

Our hypothesis is that the system thought we left with a teriyaki sauce bottle since that is the area where we picked it up last and the system could not differentiate between the two, even ostensibly after hours of human intervention.

We certainly do not want to overdo this as some sort of Achille's heel but it does point to limitations in the intelligence of the system and the cost of humans to review transactions.

Limitation - Cannot Detect Bags

One other peculiar limitation of the system is that it cannot detect bags, requiring users to manually add bags in the app as the display captured below shows:

IPVM Image

Oddity - Strong Glare Lighting

While people did not seem to mind (or at least call it out), the lighting in the store was harsh with significant glaring, as the video below shows:

Since the strong lightning was centered over aisles and next to sensors, we suspect this was done to help the sensors better detect objects that would be undermined by normal lightning levels.

Limited Cameras And Access Control In the Store

Outside of the literally massive array of sensors, conventional security cameras and access control were limited. Only a few unbranded surveillance cameras were visible in the public area of the store, this one below from the entrance area:

IPVM Image

Another element was access control credentials with employees swiping RF IDeas pcProx cards:

IPVM Image

Positives For Amazon Go Store And Customers

Customers were overwhelmingly happy in our trips to the store. While there was some confusion installing the Amazon Go app (separate from the Amazon app) and the delay in getting a receipt, the store had a variety of products at low cost and with generous refund terms.

Amazon Go employees cheerfully encouraged visitors to refund any product bought. Indeed, the 5+ million viewed YouTube video We Stole Tampons from the Cashier-less Amazon Go Store highlights this (spoiler: they did not steal anything, they simply pointed out Amazon's DIY refund / no returns needed policy).

Moreover, many customers remarked to the Amazon Go greeter about how low the prices were, comparing it to the CVS down the block.

Losing Money

These stores are clearly losing money. While that in and of itself is no surprise for Amazon, we believe the losses are significant and fundamental. The need for an employee anyway, the high cost of the technology installed, the ongoing processing costs (computing and electricity) and the undisclosed but ongoing human transaction monitors combine for a significantly more expensive store than conventional ones.

Outlook

While technology certain improves and Amazon has a well documented long-term outlook plus the ability to absorb losses for years, we are skeptical that cashier-less stores will be financially viable for many years to come. The need for massive numbers of sensors / cameras is unlikely to diminish as they cannot see through people and will need to be spread out all over and on top of shelves to overcome that. Add the difficulty of ensuring accurate transactions means that expert human operators (far more expensive than cashiers) will need to be involved.

It is completely sensible for Amazon to have a limited number of stores as effectively a public R&D experiment (with the bonus of massive marketing gains) but beware the viability of such stores, on real economic terms, for years to come.

Vote / Poll

Comments (35)
U
Undisclosed #1
Nov 13, 2018

A 1' x 1' sensor is pretty large by modern standards.

A Brickstream people counter is about 2.5" x 6", for example. It has a slightly different purpose, but fundamentally a very similar device. It can count people, monitor queues, store data, etc.

Even if the Amazon sensor is "only" 9" x 9", it's a pretty big box. My pure speculation is that they have something like a Jetson dev board inside, or some similar hardware that is a turn-key reference design/development platform and most of Amazon's work is in the software side.

The Teriyaki sauce/Water bottle experiment implies they are throwing most of the resources at segmentation masking, figuring out the boundaries and location of each object on the shelf, and not at actually identifying the specific object. It assumes that an object in a given area that roughly fits the segmentation mask must be whatever is in inventory at that location. If true, this also adds a lot of overhead to stocking the store, if shelf space layout changes (common in retail) they have to go into the software and redefine where everything is.

From some of the images, the shelf spacing looks very generous, lots of room between side by side objects, and very dark shelves vs. the white/beige you more commonly see in retail. This, along with the overhead grow lights, would certainly help with seg masking.

From your report, the system seems pretty fragile right now, and would not likely stand up to any kind of super high volume convenience store crowd, and is probably very easily fooled in a number of as-yet-undiscovered ways. It looks like it is mostly predicated on cooperative shoppers with no ill intent.

(1)
(6)
(1)
JH
John Honovich
Nov 13, 2018
IPVM

Good points!

would not likely stand up to any kind of super high volume convenience store crowd

Other videos mention how they (ironically) have lines to get in because they cannot handle high volume. For my trips, there was a handful of people each time but not anything out of the ordinary for a convenience store.

The Teriyaki sauce/Water bottle experiment implies they are throwing most of the resources at segmentation masking, figuring out the boundaries and location of each object on the shelf, and not at actually identifying the specific object.

Sounds reasonable to me. I simply picked 2 objects with similar sizes and got lucky there. Given how the cameras / sensors are positioned facing down, it would likely be hard to read / detect the specific product anyway.

One thing I did not do and regret is try to take sandwiches that had labels / numbers on them, e.g.:

My guess is that the cameras use OCR to read the label / number so I am curious what would happen if you simply covered over the label / number. How would they know what sandwich you picked?

U
Undisclosed #1
Nov 13, 2018

My guess is that the cameras use OCR to read the label

Yes, those are very clearly camera friendly labels. Can't tell from your image if they are placed "code up" so the cameras can read each one as they are taken.

How would they know what sandwich you picked?

They know which sandwiches are there before you approached, and which remain after you left, so basic "subtraction" to figure out which one you have.

JH
John Honovich
Nov 13, 2018
IPVM

They know which sandwiches are there before you approached, and which remain after you left, so basic "subtraction" to figure out which one you have.

If they knew that already, why waste time and effort putting numbers / labels on each sandwich? And how do they know what remains after you left? I am not seeing that.

U
Undisclosed #2
Nov 13, 2018
IPVMU Certified

If they knew that already, why waste time and effort putting numbers / labels on each sandwich? And how do they know what remains after you left? I am not seeing that.

Perhaps the price is the same for all sandwiches, and it knows when you took one (just like other items), but uses the label to provide specific replenishment information, if it can read the label.

Also possible is the label is not used at sale time, but is a vestige of some other part of the items life-cycle.

Did other items have similar, Braile-esque, marks?

JH
John Honovich
Nov 13, 2018
IPVM

Did other items have similar, Braile-esque, marks?

Generally, no. I did not inspect every item but I looked at various 'regular' goods (bottled water, snapple, drinks, health bars, etc.) and none of them had any special marks or tags or anything like that. Just the regular product.

They had the normal UPC codes on the packages but no reason to believe they were scanning them from 10' high.

U
Undisclosed #1
Nov 13, 2018

What I meant was they know which sandwiches are there VIA the numbers/labels. The sandwiches are likely inconsistent enough that they couldn't apply whatever pattern recognition they are using to reliably get accurate data or detail for each specific sandwich. The labels, specifically the dots, are a way to serial number them and do some form of OCR.

It terms of knowing what remains, my assumption is the sandwiches are setup in a way that the cameras can see all the labels of the top layer of sandwiches to count/identify each one.

Let's say you have a stack of sandwiches 3 layers deep:

(camera)

     

AAAAAAAA

BBBBBBBB

CCCCCCCC

The system can see each sandwich in the A layer, and the setup is arranged to discourage pulling one from lower layers, but where it could still likely catch the label if you did. If you block the cameras view and take a sandwich from the A layer and then move so the camera can now see the sandwiches again, it can tell which one is missing, and assign it to your shopping bag. If a B-layer sandwich is revealed, it can now see that label, and could track if someone took it.

If you walked up and turned over all the sandwiches and also put two in your bag, the system probably wouldn't be able to know which one you took. In that case, it probably relies on the army of interns in the back room, or in an Indian call center to manually review the video and try to figure out what you "bought".

The above of course is just my speculation. Could be totally different. For example, they might be call center employees in Thailand.

(3)
Avatar
Fabian Muyawa
Nov 13, 2018
LONTECH SYSTEMS • IPVMU Certified

There is more work to be done in order for Amazon Go Store to be stable and reliable. There appear to be a light and the end of the tunnel however, more effort is indeed require to winning the confidence of the customers.

(1)
Avatar
Brian Rhodes
Nov 13, 2018
IPVMU Certified

The maintenance cost for the store's sensor farm must be high, too.

Maybe not when the store is new, but how about after a few months of dirt and grime fall on them?

If stores catch a cashier being a deadbeat now, they just fire them and hire another teenager.

What is the comparative cost of replacing sensors when they malfunction?

(1)
(2)
JH
John Honovich
Nov 13, 2018
IPVM

Maintenance is an interesting point I did not consider but given how many devices they are using that, in itself, has to be significant. And if there is an issue with the computer vision system, the store is basically shut down... though, in fairness Amazon would probably just let people take the food for free.

UI
Undisclosed Integrator #3
Nov 13, 2018

though, in fairness Amazon would probably just let people take the food for free.

I couldn't agree more.  Amazon is using this a real word test.  To run real store numbers against employee costs.  Yes they may need to hire more "technical" employees vs. "another dumb teenager" but a lot less of them.  I'm sure somewhere in Skynet...Um Amazon headquarters, is a beancounter, weighing and measuring every aspect.  Not just the hourly employee, but the costs to have them as an employee.  Healthcare, Insurance, liability, etc.  That "low-cost" hourly employee becomes very expensive.  If the technology isn't stable enough, I'm sure Amazon will buy a company that is making a new sensor and use that :)

(3)
(1)
JH
John Honovich
Nov 13, 2018
IPVM

Yes they may need to hire more "technical" employees vs. "another dumb teenager" but a lot less of them.

They are still going to need a greater / someone in the store (to answer questions about products, how to use the app, ensure people don't steal, etc.). Given that, it's hard to understand how much labor savings this will bring compared to the added costs that are certain.

Avatar
Sean Nelson
Nov 13, 2018
Nelly's Security

Im sure they will eventually find a way to make this profitable, im sure its still conceptual to them at this point. If they can find a way to reduce human employees, i cant see why it wouldnt. Its a great concept and im glad they are blazing the trail

(1)
UM
Undisclosed Manufacturer #4
Nov 14, 2018

Never happen without RFID.  To even think you can track product movement, regardless of the intelligence behind the optical engine in lieu of RFID is absurd.  There are so many companies on the cusp of achieving this with "good" accuracy utilizing the latter, Amazon should look outside the box and perhaps into the logistics business, where this is tried and true (they know this, but like every company of their size, they choose who ever did the best job with the presentation to the C-Suite).  They will get it, but with a trillion valuation, what's a billion thrown against the wall when it's all said and done.

RFID/Optical has come a long way in the last year in the logistics world. I'm in the physical security world, but play in the passive RFID world. It's the real deal, but integration with the core physical security solutions is lacking,

So sad, maybe if they get their act together, I'll get my prime membership lowered.

(8)
(2)
JH
John Honovich
Nov 14, 2018
IPVM

#4, thanks. On the RFID side, any particular players that you think are especially promising?

UM
Undisclosed Manufacturer #4
Nov 14, 2018

Zebra - SMARTLENS (insert TM Symbol)

(1)
JH
John Honovich
Nov 14, 2018
IPVM

Thanks! Here is their website and below is their 1-minute marketing video:

Are they just using labels / stickers like the ones below?

Avatar
Matt Connolly
Nov 14, 2018

Alien Technology Long-term player in the RFID space.  Passive RFID in conjunction with cameras and back-end processing could do this pretty accurately.  "Smart shelves" with RFID readers are able to detect and report when an item is removed.  I worked with 915mhz RFID passive tag technology 10 years ago and the tags were approximately $0.10 at that time.  I'd guess they are as little as $0.03 now.

(1)
(2)
U
Undisclosed #2
Nov 14, 2018
IPVMU Certified

Amazon should look outside the box

Though Amazon apparently abandoned RFID after filing patents using it specifically for Go.

(1)
Avatar
Jason Thomas
Nov 19, 2018
IPVMU Certified

That is what I was going to say. Why not just use RFID? Seems way easier and cheaper to deploy.

UM
Undisclosed Manufacturer #4
Nov 14, 2018

As a note, tagging the product with a food safe RFID tag cost another 1.5 cent per item, but historically, that is not an issue with AMZN.  Given their propensity to rule the world in the retail space at all cost, this is a no brainier.  They will get it, probably buy the company, and evolve.

UM
Undisclosed Manufacturer #4
Nov 14, 2018

John,

RFID has evolved.  If your not into it, you have to play catch-up.  You can print RFID now without a "dedicated" label and it's food safe should that be the route your going.  Not getting into it tonight, but would certainly do a sidebar.

I used to think passive RFID was irreverent, but with the polling appliances available today, it's amazing what can be achieved.

This will bleed into the physical world sooner than later.

I know your early evening, but I'm out.

(1)
UM
Undisclosed Manufacturer #5
Nov 14, 2018

U4 - I used to sell RFID and the cost wasn't the issue, it was the labor / logistics to apply the tags. Tag price points were around .05 back then so good to see the drop to .015 you mentioned

The only Retailers who could make this work controlled the supply chain from point of manufacture through the finishing process (think softlines). Combo RFID / EAS  tags were either embedded into the price hang tag or sewn into garment: 

Until the CPG abandon the UPC (which costs nothing to print on packaging) you'll never see the widespread adaptation of RFID. Tests were piloted with companies like P&G with EAS/ RFID but eventually fizzled. The win for some of the product lines back then was they were not in locked cases or behind counters. But even that led to costs that neither the CPG or the retailer wanted to absorb.

Today all the RFID use cases I see are purely inventory (again think softlines) - where an employee can scan an RFID wand over a rack of sweaters and correlate what's in that rack that to what should be on hand or needs replenishment 

There's also a few technical problems that may or may not have been overcome by now, the biggest of which was high metal content (think foil blister packs for cold medicine or foil bagged chips) blocked / inhibited the RF signal. High moisture content items (like packaged meats) also suffered the same problem.

Maybe the major RFID players have solved some of these problems, but unless you're Amazon who can lose money and not care, most major Retailers are not going to be potential customers for the technology

just my .02

(2)
(4)
Avatar
Fabian Muyawa
Nov 14, 2018
LONTECH SYSTEMS • IPVMU Certified

There appears to be a lot of substance in Zebra Smartlens for retail.

U
Undisclosed #6
Nov 14, 2018

I visited the one in Seattle, and noted the high density of sensors and was wondering about the costs of these sensors, energy consumption, and infrastructure as well.  I also noticed several (6+) Amazon employees that were there answering questions and solving issues, and imagine there are many behind the scenes, so reducing headcount may or may not be a reality.  I was also surprised by the footprint of the store itself.  It was much smaller than I imagined.

My opinion is that this is helping Amazon understand the process of small retail.  Eventually they will reduce the size and cost, but I feel that the future deployments of this technology are going to be much different.  This also is WAY too costly/complex and too many issues for large box retail.

(1)
JH
John Honovich
Nov 14, 2018
IPVM

Thanks for sharing. Good point about it being smaller than imagined.

One open question is how many constraints does this technology place on what can be offered. Not being able to detect / count bags is one item. What else cannot be done? For example, I noticed there was no individual fruit being offered. Weighing things would be a clear issue but many convenience stores sell fruit by the piece (banana, apple, etc.). I'd be curious to learn what other conventional store offerings do not work / fit into the technology's capabilities.

(1)
RA
Ramy Ayad
Nov 14, 2018

I noticed some of the sensors didn't have the green light (Working) it might have been the angle of the video shot.

If in fact some of them are powered off or standby, may they are backup sensors or may be some did fail or can't run for too long!! which like you said it might mean that the technology is not there yet.

Avatar
Brian Selltiz
Nov 14, 2018
IPVMU Certified

Excellent article John, thanks!

(2)
U
Undisclosed #7
Nov 15, 2018

At the present time most assumptions are subjective. If you had a time a machine you will see all the child technologic spawns created by taking steps forward. 

I like IPVM however I am relentlessly skeptic to the depth of your observational calculations created in your own living room (IPVM.COM).

Amazon spends money to make money that does not need to be spent twice. 

Cool photos, simple write up, sorta click bait. Thanks for putting a few simple things together for a beta perspective.

(1)
(1)
UM
Undisclosed Manufacturer #8
Nov 15, 2018

Isn't this just an oversized vending machine with unreliable inventory management? Why not put everything in a backend mechanical sorter  behind a piece of touch/rfid sensitive glass for users to select their choices with their  unique chip(with their account info). Employee costs vastly reduced as desired. Shopper grabs their bag of goodies at end of "luggage handler" and goes home.

All comes down to the value of a person and human interaction to make a sale. 

 

(3)
KB
Keith Beatty
Nov 15, 2018
IPVMU Certified

Worlds collide!

LTT mentioned on IPVM.

What next, a Donald Trump impersonator takes over the comments section in one of IPVM's articl...oh wait...

JH
John Honovich
Nov 15, 2018
IPVM

Keith, lol,

For others, the LLT reference is to the video we cited in the report above, embedded below:

JH
John Honovich
Dec 04, 2018
IPVM

Update: WSJ reports "Amazon Tests Its Cashierless Technology for Bigger Stores", though no real details about how well it's going and some speculation about how it might be used for Whole Foods (which is silly, at least in terms the whole range of Whole Foods current products, which has no chance of fitting into this technology).

SD
Shannon Davis
Dec 04, 2018
IPVMU Certified

Great report John. The Walmart corporation is doing whatever it takes to keep up technologically. Either developing systems internally, buying systems and services and even buying startups to remain relevant. Walmart and Sams have their new cleaning robot to automate the floor sweeping and mopping as well as many other automation's being added to the stores. For instance their isn't a cash office person anymore. They simply put any money into an ATM like device. 

My wife has worked for Sam's Club for over 29 years and got her second displacement notice in a year and a half. The first time it didn't make sense to leave but now it definitely does. This time they have combined 5 positions into 2 to run the back end of the club. Which is great until one or the other takes vacation. The first time they eliminated the Audit Lead, you know inventory control. Inventory control isn't as important as it once was I guess. One position that she did have in the past was the Comp Shopper and that position is still going but others, probably more important aren't. Sam's especially is trying to automate every facet they can which in turn eliminates hundreds of years of experience with every downsizing they are doing lately. By the way she is going to take a nice 2 to 3 month vacation before finding new work. 

The new cashier-less Sams Club in Texas is starting soon. Will only have about 40 employees or so. Everyone using the store will have to use the Scan-and-Go App", which by the way is really cool as you don't have to wait in line to check out. One way they are trying to eliminate lines at the registers is the use of Bosch cameras and Bosch analytics to call the club manager when the lines get too long at the checkouts. Not long after the store was remodeled and the cameras were upgraded my wife was with the club manager when he got that call. She came home and said don't they have anything better to do than spy on us, then I explained what was most likely happening. I knew as soon as I saw the new Bosch cameras go in what was happening. 

Some of the automation's I completely understand as that can make a business more profitable but when a company starts getting so cheap and start laying off well experienced personnel go because they make too much money is where I start questioning a businesses method and practice. 

My take from all of this and what I have heard and observed is Walmart is really struggling to stay in business, especially the Sam's Club side. The Walmart corporation has fallen way behind the curve when it comes to technology and being a leader in the industry. Who knew a company who has put thousands of small to large size businesses  out of business may someday suffer the same fate. Even Jeff Bezos is on record saying that someday Amazon will get to big and fail. Not Walmart though.

My big prediction is that Walmart is trying whatever it takes to make Sams Club work or better yet making it fail so they can sell that division off to Amazon. Think about it, sell Sam's club to Amazon and now Amazon has small distribution centers all around the country.

Anyways just my two cents!

(1)
(1)
JH
John Honovich
Sep 22, 2019
IPVM

Update: The Information is reporting that Amazon is way behind its plan to expand these Go stores.