Google’s Super Resolution Enhancement Examined

This is somewhat different than LPR tech, though a lot of the core learning frameworks are the same.

In LPR, you are trying to find an exact match for a set of characters, this makes parts of the problem easier because you are generally limited to a very small set of possibilities (A-Z, 0-9, etc.). Also, if you can figure out (or guess) the state you may be able to limit it further if the state uses particular sequence formats.

For this Google application, they are really attempting to "paint" an image that looks convincing to a human. It is less about precise accuracy, and more about filling in details that make the objects recognizable. So, for this reason, it would not be ideal for LPR, as the algorithm might be able to take a blob of pixels and turn it into an image that looks very much like what you would expect a license plate to look like, it would probably be less concerned with differentiating between an "O" and a "Q" (for example). Or, the training data could cause it make other errors- if none of the input plates had "Q"s, but lots of them had "O"s, it might take a pixel blob of a plate that was "QQQ" and paint it as "OOO". (this is my interpretation from reading the whitepaper, there might be more to it).

Matt -

I thought about the social media integration angle. Most pics on Facebook/etc. tend to be high resolution, they could make a good source library for input images. 

However, there can be challenges for an algorithm like this that is trying to "paint", not trying to "match". You could end up with it creating an output image that is the compilation of facial features of several people. For the purposes of creating a "painting", the image would be highly representative of the reality (a person with a round face, short hair and small nose, or a person with a square face, long hair and big ears), but it might not accurate in terms of identifying a specific person.

If you are using a smaller dataset, say just the employees from a building, or just the students on a campus, you will likely have outputs that better match the specific individual than if you use a dataset of "all people".

Part of these problems they do attempt to tackle, it comes down to figuring out what parameters can be "normalized" and which ones make up unique qualities. In the whitepaper they use an example of sampling cars, if you average the paint colors of all cars together you would wind up with a very drab "average" color that would likely not be applicable to any vehicle. But you could average minor differences among reds or blues or greys to produce output images that were still representative.

Taken to an extreme, this technology could be useful to create a next generation smart CODEC. If you can compress the background of a scene down to "there is a red truck, a blue roadster and a brown delivery vehicle" that could be less data to store than recording the actual pixels of those objects. When playing back video, the algorithm would create representations of those objects on the fly. Or you could store a few reference frames of the vehicles, along with movement paths, and then dynamically draw the objects in along the way, used learned data about appearances to create frames that realistically represent how the vehicles would look as they moved or turned, even if you did not have that actual data recorded.

Further, if the recreation can be done fast enough, and with an "average" processor, it could be done client-side, allowing the server to record and stream very low bitrate data, while still allowing a client to piece together a more detailed image.

I do not think we will see applications of this in the near-term for surveillance, but opening up this level of image processing may lead to some amazing capabilities.

I don't know that Google's software is really changing anything here.

Whether you use a completely automated algorithm (from Google or from anyone else), a semi-automated algorithm (in which a person clicks a button to apply an enhancement but doesn't necessarily know what the button does), or a completely manual alteration, you still have the legal need to show the original image and show how you arrived at the modified image.

The only problem would occur if Google (or anyone else) was able to create an altered image that could NOT be identified as altered.

I am not an image expert, but it appears that this will become more and more challenging.

With experience, i know that most FR algorithms face difficulty in skin tones as well as with age (difficulty with younger aged individuals datasets), it would certainly be interesting to run this Google Super Resolution algorithm against faces of African origin and analyze the results, as per Tim's comment, the results would probably not be very enouraging. Having said that, this is still a very interesting and postitive research in FR and in the right direction, I can see a lot of potential but just like everyone else has said, will take a while to get there.