Samsung Rigs Tests

This is funny in light of the recent uproar about the Samsung's salesguy's camera testing 'process'. Also, kind of stunning.

Here's what they did:

"If the [Samsung tablet] system detects that one of a hard-coded list of apps is running, it turbo-charges the graphics processing unit (GPU), yielding that 20% boost. Then when the test is over, it scales it right back, because turbo-charging uses up the battery."

So Samsung knew what was going to be on the test and then cheated to game the test when a list of apps on the test list was used.

The tester caught them:

"To prove this, the Ars team took the code from a benchmark called Geekbench and tweaked how it would present itself to the system. "Stealthbench", as they christened it, was the same benchmarking code, yet the Note 3's GPU didn't turbo-charge. No 20% boost – just the same performance as the LG."

However, Samsung denied it but could not explain the discrepancy in performance by simply changing the app's name.

Oh and Anandtech has a table of test cheaters - Samsung gets highest scores!

This is another example why accurate tests must be as independent from the manufacturer as possible. Unfortunately, surveillance users often do crazy things like let the manufacturer set their own cameras up for the test. Manufacturers will routinely try to get an extra edge, even if it is unfair or skews the test.

Good job Samsung! Thanks for proving my point.

So in essence they optimized their software/firmware for their phones and tablets to increase performance for benchmark apps. It's actually pretty clever in a Captain Kirk Kobayashi Maru sort of way. Does anybody other than the smart phone uber-geeks really pay much attention to those tests when considering a smart phone purchase? Probably not. I've had an S3, and I would spring for an S4 in a heartbeat if I needed another phone, this information would not dissuade me. I still want one of their Note tablets, I wish I would have waited to buy one of those instead of my iPad.

Back on topic...

Same thing happens with cameras to a degree which I am sure Ethan can attest. I have taken different manufacturer's cameras out of the box and set them up side by side without changing any of the default settings and some look much better than others, but then you look at the image settings, bandwidth, frame rate and each manufacturer has optimized them with different default settings. Some go for max image quality, others lower bandwidth and most take a balanced approach. Is that a bad thing?

IMO, only when a dumb sales guy says "look how much better ours is out of the box" which was alluded to in the Samsung sales guy post is it seedy and cheap. Caveat Emptor, do your homework. At least their marketing and sales guys are looking for an edge, give them credit for that, it's what they do. It's a dog eat dog world, and 2nd place gets you a set of steak knives. This is why I always insisted on them reps loaning us demo cameras to tinker with in the lab before purchasing them, and the first thing we would do is default them to avoid any hot setups. If all manufacturers were straight-up honest about everything, IPVM would have little to write about.

It would be cool to see manufacturers optimize their cameras to perform better with VMS's instead of playing parlor tricks with image quality and number of pixels.

Thanks. And I think it's a pretty good find. Pretty interesting attempt even though they were so easily caught.

I added an explicit reference to tablet at the beginning.

I don't see a problem calling out Samsung on their phone phone and tablet tests since they are part of the same company and you have to expect some similar business practices across divisions, but I think it would have been more informative to me as a reader if the title of the article, or at least early on in the article it specifies the tests in question were about their phone and tablet devices. Otherwise it gives me the impression it specifically concerns their security devices.

Here's another Anandtech article that digged into what Samsung was doing:

"Opening the [application] file in a hex editor and looking at strings inside ... pointed at what appeared to be hard coded profiles/exceptions for certain applications. The string "BenchmarkBooster" is a particularly telling one."

So Samsung evidently added a custom section to boost performance for specific benchmark applications. Nicely done.

Those citations are for phones / tablets. I don't know about cameras.

Are they cheating hardware-wise with their cameras or just their phones and tablets? Or just dumb sales guy tricks?