Amazon Go Store TestedBy John Honovich, Published on Nov 13, 2018
IPVM's trip to and testing of Amazon Go's San Francisco store shows a number of significant operational and economic issues that undermine the hype.
While the press has overwhelmingly fawned at Amazon Go's store and the report that Amazon will deploy 3,000 of these stores in the next few years, our review of the operation points to significant challenges in turning this into a viable enterprise.
In this note, we examine:
- The time delay and behind the scenes human intervention involved
- The density and cost of sensors used
- Amazon mischarged IPVM during testing
- Limitations in the system's capabilities
- Video surveillance and access control used
- Positives for Amazon and Go customers
- Outlook on Amazon Go stores
Given the massive amount of:
- Infrastructure involved - hundreds of large sensors for a small convenience store
- Computing and electricity consumed - undisclosed but clearly heavy amount of back-end processing for these hundreds of sensors
- And most notably human operators reviewing transactions
IPVM is skeptical that Amazon is anywhere near close to being able to run Go stores as a profitable business. Now, given Amazon's scale and ability to absorb losses, this may not be an issue to them but it does undermine the hype that this is economically viable for real businesses.
Density And Cost of Sensors
The most notable hardware differences for the stores are their uses of turnstiles to enter (which are easy to see) and their mass of hundreds of sensors overhead that most people ignore. This image from the entrance overviews the sensors laid out:
And this image shows how many sensors are needed just to cover the ~10' long chips section:
The following video shows the sensors across half the back side of the store:
Sensors are square shaped, ~1' per side, with what appears to be 2 imagers:
These sensors are put right above every shelf as well as throughout the aisles. Based on limited Amazon explanations, they read / detect what objects each user is picking up.
Most Important - Receipt Delay and Human Review
The most surprising and under-reported element is that receipts take 5 - 10 minutes normally to be generated. By contrast, Amazon's own marketing video shows it to be done instantaneously. And CNET who just a few weeks ago went to the same store we did, hides that as well, as the video excerpt below shows:
CNET edits it to make it look instant but the timestamps from their own video shows it was captured 25 minutes later:
Indeed, long delays for receipts are normal, according to 3 different Amazon Go employees who spoke to us. They even explained that they have people reviewing transactions though they either did not know or were not authorized to explain how often they did so.
However, 5+ minutes is an eternity for computers whose timescale is the milliseconds or less to process information. Amazon is not letting on to how much humans need to be involved but such a delay points to serious concern for errors and the need for human intervention.
Mischarge / Error
Amazon charged us for a bottle of Teriyaki sauce instead of a bottle of water. In fairness to Amazon, this was done on purpose to test the system but even though they took 3 extra hours to have humans analyze this, they still made a mistake:
Here is what happened:
- Take water bottle from back shelf
- Walk to front shelf, take Teriyaki sauce off shelf
- Put down water bottle in the Teriyaki sauce column
- Go back to back shelf, put Teriyaki sauce in water bottle column
- Return to front shelf, grab water from Teriyaki sauce column, put in pocket, leave store
Our hypothesis is that the system thought we left with a teriyaki sauce bottle since that is the area where we picked it up last and the system could not differentiate between the two, even ostensibly after hours of human intervention.
We certainly do not want to overdo this as some sort of Achille's heel but it does point to limitations in the intelligence of the system and the cost of humans to review transactions.
Limitation - Cannot Detect Bags
One other peculiar limitation of the system is that it cannot detect bags, requiring users to manually add bags in the app as the display captured below shows:
Oddity - Strong Glare Lighting
While people did not seem to mind (or at least call it out), the lighting in the store was harsh with significant glaring, as the video below shows:
Since the strong lightning was centered over aisles and next to sensors, we suspect this was done to help the sensors better detect objects that would be undermined by normal lightning levels.
Limited Cameras And Access Control In the Store
Outside of the literally massive array of sensors, conventional security cameras and access control were limited. Only a few unbranded surveillance cameras were visible in the public area of the store, this one below from the entrance area:
Another element was access control credentials with employees swiping RF IDeas pcProx cards:
Positives For Amazon Go Store And Customers
Customers were overwhelmingly happy in our trips to the store. While there was some confusion installing the Amazon Go app (separate from the Amazon app) and the delay in getting a receipt, the store had a variety of products at low cost and with generous refund terms.
Amazon Go employees cheerfully encouraged visitors to refund any product bought. Indeed, the 5+ million viewed YouTube video We Stole Tampons from the Cashier-less Amazon Go Store highlights this (spoiler: they did not steal anything, they simply pointed out Amazon's DIY refund / no returns needed policy).
Moreover, many customers remarked to the Amazon Go greeter about how low the prices were, comparing it to the CVS down the block.
These stores are clearly losing money. While that in and of itself is no surprise for Amazon, we believe the losses are significant and fundamental. The need for an employee anyway, the high cost of the technology installed, the ongoing processing costs (computing and electricity) and the undisclosed but ongoing human transaction monitors combine for a significantly more expensive store than conventional ones.
While technology certain improves and Amazon has a well documented long-term outlook plus the ability to absorb losses for years, we are skeptical that cashier-less stores will be financially viable for many years to come. The need for massive numbers of sensors / cameras is unlikely to diminish as they cannot see through people and will need to be spread out all over and on top of shelves to overcome that. Add the difficulty of ensuring accurate transactions means that expert human operators (far more expensive than cashiers) will need to be involved.
It is completely sensible for Amazon to have a limited number of stores as effectively a public R&D experiment (with the bonus of massive marketing gains) but beware the viability of such stores, on real economic terms, for years to come.
Vote / Poll