Covert Facial Recognition Using Axis and Amazon By NYTimes

By John Honovich, Published May 20, 2019, 02:44pm EDT

What if you took a 33MP Axis camera covering one of the busiest parks in the US and ran Amazon Facial Recognition against it?

That is what the NY Times recently did.

Inside this note, we examine how they did it, what lessons to learn and not from their test.

**********

** ***** **, ****, the ** ***** *** a ******, ** ***** ** ************ (but *****) ****** *********** Machine.

How ** *****

****** **** *** ** Axis **** ********* ******, ** *** ***** describes:

*** ******* ** ****** **** **** installed **** **** * decade *** ** **** people ***** *** ******* the **** *** **** for **********, *** *******, or ***** *** **** the *** ******* **** was ** *** ******. They *** *** ******** to ** * ******** device, ********* ** *** corporation **** **** *** park.

** ** ******-*** (***** ** ****** in **** **** ***** released). ** *** ****** explained ** **** ****, it ***** ********* ***** image ******* (*.*., *** sides *** *********) *** the ***** ****** ** pixels **** ** ******* notable ******* **** ** a **** ***.

**** ****** ** ******** available **************** (***** ** ****** here).

** *** ** ***** took *** ******** ********* stream *** ******** **** Amazon's *********** ****** ******* (** **** ****** *** facial ********* ****, ******* *** ********* part ** ** **** strong). *** ***** *********:

** *********** *** **** it ** ** ***** people ******* ***** *********, we ********* ****** ****** of ****** *** ****** near ****** **** (********* on ***** *********’ ********, for *** **** ****) and *** *** *** of ******* ******* ******’* commercial ****** *********** *******. 

*** ***** ** ******* to*** ** ***** ******, ***** **** *** ingest ****** *** ***** the ******* **** ******** like ****** ***********.

Results ******

*** ***** ********* *** results:

*** ****** ******** *,*** faces **** * ****-**** period (*** *********** ****** people, ***** * ****** could ** ******** ** multiple ******). ** ******** several ******** ***************, ********* one ***** ******* ** a **** **** ** Richard *******, * ********* at *** **** ******* of *********, **** ** 89 ******* ********** *****. The ***** ****: ***** $60.

**** **** ****** **** one ******** ***** - the ********* **** *** image **** *** **** camera ******** *** ********** headshot *****:

Negatives ** *** ****

*** **** ** ********* to ********* *** **** it ******. ** *** not. *** * **** that ****, *,*** ****  captures *** ***** ***, especially ***** ****** *********** will ******* *** **** face **** *** **** again ** *** *****. Moreover, ******* ********* ** people *** **** *****, they **** '******** ******* possible ***************'. *** ** Times ***** *** ****** out ** * ********.

But ********** *********** *** *********** 

** *** ***** ****, a ********* ****** **** people ****** **** ** right ****** *** ***** of **** ** **** poorly *** ** *** conducting ****** ***********. ** the ****** ************, **** people **** ******** ** a ******* ********, *********** the *******:

***, ** ***** **** the ****** ****** (*** Q3709 *** ***** *******), there **** **** ****** headshots **** **:

*******, ***** ***************, **** in *** ******, **** people **** *** ******* at *** ******, ** this ******* *****:

** *** *** ****, a ****** ********** *** selected *** ********* ***** would ** **** ******. On *** *****, **** still ***** *** ******* of ****** ** ******* people's ***** ******** **** they *** ****** ** various **********.

Doing  ******?

***, ** *** ** Times ********** ****** ***, if *** **** * massive ******** ** ********* images *** ******* *** may ** ******:

*** ****** *** *********** may **** **** ****** to * **** ******* of *******. ******* **** with * ************* ******** of ***** — **** a ******’* ******* ******** — *** **’* ******** to ***** ******** ********** an ****** ****** ** real ****.

*** ********* ***** ******** is ******* *** (*.*.,** ** **** ****** the ***). ******, *** ******* still ******* * ***** problem ** **% ** even *** ******* *** unlikely ** ********* ******* the ******* ** ********* needed. *** *** ***** does *** ******** *** challenges ** ******** ********* results, ********* ***** *******, that ***** ***** ** a ***** *********.

 

Comments (21)

Undisclosed #1

05/20/19 07:02pm

If you want mass Facial Recognition, partner up with all the cell providers who look at your face everyday each time you write a text. Perhaps your photo is already attached to your Verizon, Vodafone, Sprint, ATT account. 

Unify this database with a mass facial recognition network or create a  Multi-factor authentication app, say to log into email, amazon, your accounts and presto. Instant world facial recognition database everyone dreams of. 

Now take that massive database, and poke thru some of those Axis 33mp deployments, boom! now you know why Lucius Fox quit his job, TMI!  (you're welcome 5G!)

In the end we need this technology, take the good with the bad. Trust your government. Perhaps we can finally catch a few aliens walking amongst us. :D #TinfoilHat

Agree: 2
Disagree
Informative
Unhelpful
Funny: 6

Undisclosed Integrator #2

In reply to Undisclosed #1 | 05/20/19 09:12pm

Most government agencies can already track the Average Joe via their digital signature. The facial recognition suite you are describing would probably be overkill, but it wouldn't surprise me if it isn't on the horizon. #TinFoilHat #IsThatADron...

Agree: 4
Disagree
Informative
Unhelpful
Funny

Undisclosed #3

05/20/19 09:27pm

Now that biometric data is remotely and easily collectible and able to be parsed in real time, I am glad some are taking enough notice to begin to regulate it as San Francisco has done recently. I'd like to think America can keep from becoming Xinjiang. I'm not willing to trade freedom (privacy) for 'security' as is the usual selling point.

Agree: 2
Disagree
Informative
Unhelpful
Funny

Undisclosed #4

IPVMU Certified | 05/20/19 10:30pm

“My first reaction was, ‘Oh my god, that is unbelievable,’” Dr. Madonna said, after we reached him and explained the experiment. 

his second reaction was “Oh my god, and it only got me?”

 
Agree
Disagree
Informative
Unhelpful: 1
Funny: 5
Avatar

Anton Miller

Shaked Projects
05/21/19 03:40pm

Well, it's either NY Times have not even most basic understanding on how video surveillance in general and face recognition is particular are working or they intentionally are trying to misguide the public by downplaying how it can be done very efficiently even with most basic equipment.

Just one properly installed prehistoric b/w 320p camera connected to a time-lapse vr would achieve better results.

The reference image database is clearly key (e.g., as is used inside the PRC)

Not necessarily. If you have a city-wide CCTV system just start filming and storing headshots and after few months you'll have enough data to figure out location of residency, work and personal relations of pretty much all inhabitants of that city.

And the Times does not consider the challenges of actually reviewing results, including false matches, that would still be a major challenge

Reviewing process is very much automated these days and false matches are eliminated by using multiple cameras.

Agree
Disagree
Informative
Unhelpful
Funny

John Honovich

IPVM | In reply to Anton Miller | 05/21/19 03:42pm

Not necessarily. If you have a city-wide CCTV system just start filming and storing headshots and after few months you'll have enough data to figure out location of residency, work and personal relations of pretty much all inhabitants of that city.

Any examples of where this is being done or what companies claim to be able to do such a thing without a reference database?

Agree
Disagree
Informative
Unhelpful
Funny
Avatar

Anton Miller

Shaked Projects
In reply to John Honovich | 05/21/19 05:08pm

Axxon can work without reference database. Just pick any stored headshot and it will find you all occurrences where this person was filmed.

Adding such insight capabilities is a rather trivial big data task.

1.Assign all headshots reference numbers
2. Group all headshots by over 90% similarity (pick one, find all that are similar, store their numbers, pick next one..., repeat until run out of headshots)
3. Assign each group a reference number and apply standard big data analysis and filtering techniques.

For example, look up for two reference numbers that were registered in close proximity to each other repeatedly. Now, look up hours and locations where they were registered, apply them to a grid and you can further extrapolate data such as age, income and marital status. 

If there's no such system available by now, which is highly unlikely, I probably should look into making one lol

Agree
Disagree
Informative
Unhelpful
Funny

John Honovich

IPVM | In reply to Anton Miller | 05/21/19 05:15pm

Anton, you claimed:

you'll have enough data to figure out location of residency, work and personal relations

How are you figuring out the residency, work and personal relations?

I understand you can search for people by faces captured on many systems but you did not just claim that, you claimed to be able to figure out personal relations. Minimally, you are talking about a massive amount of human work.

And not sure where you are from, but most 'city-wide CCTV system' have few cameras that are currently positioned to give the level of details / angles needed for your 90% similarity search.

That said, don't let me discourage you. If you are confident you can deliver on that, don't waste time talking to me, go and sell it for millions to various cities.

Agree
Disagree
Informative
Unhelpful
Funny
Avatar

Anton Miller

Shaked Projects
In reply to John Honovich | 05/21/19 08:05pm

I live in Israel. Here pretty much all larger municipalities are members of so-called "Safe City" program, where cities receive subsidies in exchange for installing a city-wide surveillance system. Right now there's not enough cameras to provide the level of resolution I'm talking about but more are added every day and I clearly foresee the future where there will be one or two on each lighting pole. Just $334 apiece...

There are, however, city-wide CCTV networks in pretty much all somewhat developed cities. I'm talking about CCTV systems installed and operated by non-government entities and private citizens. 

you claimed to be able to figure out personal relations. 

Oh, yeah, absolutely. With the sufficient amount of data, such as a camera on each lighting pole recording for few months I can figure out quite a lot. 

Minimally, you are talking about a massive amount of human work.

No human work needed at all.

If #38568092898M in past was frequently registered in immediate vicinity of #58970854808F but now #58970854808F is only seen with #50932952452M and #38568092898M is often registered nearby bowling, booze store and at night time drunk (full human skeleton analysis is a thing since early 2000's, even easier than face recognition) alone, than you don't need any sort of intelligence to figure that sending him an invitation to a dating website might be fruitful.

Agree
Disagree
Informative
Unhelpful
Funny

John Honovich

IPVM | In reply to Anton Miller | 05/21/19 08:17pm

If #38568092898M ...

That assumes the data points are accurate (i.e., match correctly to a single real individual and not that a single real individual's faces are split across multiple ids or multiple individuals are mismatched).

If you could trust that all observations were accurate, I certainly agree that a machine that could deduce the relationships. But I think the accuracy issues would compound in such an analysis.

Agree
Disagree
Informative
Unhelpful
Funny
Avatar

Anton Miller

Shaked Projects
In reply to John Honovich | 05/21/19 08:48pm

I believe that accuracy issues can be negated with the sufficient number of cameras and the singular cases of lookalikes and twins won't cause too much trouble for such cases are very easy to identify: same face at two different locations at time period that is not sufficient to travel between there locations is a clear identification of a twin, a lookalike or some suspicious activity.

Besides, color is not needed for facial recognition, meaning that a more sensitive, higher resolution (just remove the RGB filter and you have 3x pixels) and cheaper sensor can (and will be) used to significantly improve recognition at long distances. 

Agree
Disagree
Informative
Unhelpful
Funny

John Honovich

IPVM | In reply to Anton Miller | 05/21/19 08:57pm

No, Anton, don't move the goal posts. You said:

If you have a city-wide CCTV system just start filming and storing headshots and after few months you'll have enough data to figure out location of residency, work and personal relations of pretty much all inhabitants of that city.

Now you want:

sufficient number of cameras... a more sensitive, higher resolution (just remove the RGB filter and you have 3x pixels) and cheaper sensor can (and will be) used to significantly improve recognition at long distances.

For sure, you can improve things with more and better cameras but that's not what 'city-wide CCTV systems are today'

As for:

singular cases of lookalikes and twins won't cause too much trouble

When you have to factor in varying lighting conditions, various angles and the fact that people's appearances change over time, you are going to have a lot more mismatches.

Agree: 1
Disagree
Informative
Unhelpful
Funny

Undisclosed #4

IPVMU Certified | In reply to Anton Miller | 05/21/19 05:16pm

i’m sure the NYT system could have created a reference database ‘on the fly’ as well.  Then it might have had hundreds of hits from the same persons, with pictures taken at the same angle, even moments apart.

would that have really impressed you more?

Agree
Disagree
Informative
Unhelpful
Funny
Avatar

Anton Miller

Shaked Projects
In reply to Undisclosed #4 | 05/21/19 08:16pm

Except, I'm not talking about data from one camera. 

However, even such database could prove to be useful. Frequent visitors can be identified and approached by local businesses. Troublemakers can be identified to summon a police unit before something happens. Identification of age and gender could help target ads.

Agree
Disagree
Informative
Unhelpful
Funny

Undisclosed #4

IPVMU Certified | In reply to Anton Miller | 05/21/19 09:40pm

Frequent visitors can be identified and approached by local businesses. Troublemakers can be identified to summon a police unit before something happens...

Cop: This is the fourth time today you’ve been seen just hanging around, looking for trouble.

Kid: Actually it’s the first time I’ve been here today.

Cop:  Maybe, but do me a favor and scram anyway, ok?  We’re over our quota for people who look like you today.

Agree: 1
Disagree
Informative
Unhelpful
Funny: 1
Avatar

Anton Miller

Shaked Projects
In reply to Undisclosed #4 | 05/25/19 03:47pm

Yep. That's exactly how it's gonna work.

Agree
Disagree
Informative
Unhelpful
Funny
Avatar

Thomas Lienhard

05/21/19 06:38pm

 At ISC West I put my face into the nose of the camera and it told me ( a 54-year-old Caucasian male) that I was a 74-year-old woman. I took off my glasses straightened out the fur in my soul patch-goatee smiled demurely and gave it another go and I am still a 74-year-old woman.  I ruffled my thinning hair and tried to look “butch” and nope. I’m a lady.  My wife disagrees with the camera by the way !  We have a long way to go.  

Agree
Disagree
Informative
Unhelpful
Funny: 2
Avatar

Jack Ives

CyberExtruder
05/22/19 12:40pm

So The NYT reported that this experiment cost $60 to capture 2750 faces over 9 hours. Doing a back of the envelope calculation -

24 hours in a day x 30 days in a month = 720 hours per month

722 hours / 9 hour segments = 80 segments

80 x $60 = $4,800 per month to operate

2,750 faces x 80 = 220,000 (non-unique) faces captured

220,000 / $4, 800 = $45.83 per face capture

Not very cost effective if you ask me-never-mind the questions of accuracy. No wonder Amazon is taking over the world...

Agree
Disagree
Informative
Unhelpful
Funny
Avatar

Cody McCormick

In reply to Jack Ives | 05/24/19 09:39pm

Your math is backwards... It is actually $.02 per face capture.  At 45.83/capture the cost would have to be 10M+

Agree: 2
Disagree
Informative
Unhelpful
Funny

Undisclosed #4

IPVMU Certified | In reply to Cody McCormick | 05/25/19 06:29pm

he should have used a napkin ;)

Agree
Disagree
Informative: 1
Unhelpful
Funny

Undisclosed #5

05/25/19 11:31pm

Funny

Chinese driver gets a ticket for scratching his face

Agree
Disagree
Informative
Unhelpful
Funny
Read this IPVM report for free.

This article is part of IPVM's 7,095 reports and 940 tests and is only available to members. To get a one-time preview of our work, enter your work email to access the full article.

Already a member? Login here | Join now
Loading Related Reports