Galaxy S 4 on steroids: Samsung caught doping in benchmarks

Tipped off by a report from hacker in Luxembourg that accused Samsung of increasing the clock speed of its Android-based Galaxy S 4 specifically, and only, when running certain benchmarks, AnandTech investigated and confirmed that it is indeed cheating to win in performance tests, by design.

A report by Anand Lal Shimpi & Brian Klug examined versions of Samsung's Galaxy S 4 equipped with the company's new Exynos 5 Octa SoC.

Chip, set, match

Samsung announced the new Exynos 5 Octa component earlier this year as its answer to Apple's A6X; both chips are manufactured at Samsung's System LSI chip fab.

Apple uses its own custom "Swift" CPU core design, paired with GPU cores tech it licenses from Imagination Technologies, providing four PowerVR SGX 554MP4 cores to power graphics on iPad 4.

Samsung's Exynos 5 Octa was the first to license ARM's stock Cortex-A15 CPU core design, and pairs four of them with essentially the same Imagination GPU design as Apple, a shift from Samsung's earlier chips that formerly used ARM's Mali GPU design.

via Imagination.

Designed to win benchmarks

Their shared GPU design makes the two chips easier to compare, even though Samsung's are clocked to run faster (533MHz vs 300MHz).

However, in looking at Samsung's chip, AnandTech was able to verify what a poster had reported on graphics website Beyond3D: Samsung was specifically enabling full speed GPU performance only when running specific benchmarks, and then reverting to slower 480MHz speeds when being used to play games or run other apps.

The site first confirmed that, in normal use, the Galaxy S 4's Exynos 5 Octa was running at 480MHz, noting, "Samsung never publicly claimed max GPU frequencies for the Exynos 5 Octa (our information came from internal sources), so no harm no foul thus far."

However, when running tests including GLBenchmark 2.5.1, AnTuTu and Quadrant, the device "triggers a GPU clock not available elsewhere: 532MHz."

The CPU side behaved similarly when running benchmarks, but it used maximum performance modes available to other apps. "The 532MHz max GPU frequency on the other hand," the report stated, "is only available to these specific benchmarks."

Digging into files in the software that controls the chip's frequency, AnandTech discovered "what appeared to be hard coded profiles/exceptions for certain applications," including the text string "BenchmarkBooster."

Source: AnandTech

The file directly revealed custom Samsung support for changing how the Galaxy S 4 works when running specific benchmarks.

Further, the site discovered "there are strings for Fusion3 (the Snapdragon 600 + MDM9x15 combo) and Adonis (the codename for Exynos 5 Octa)," indicating that Samsung isn't just cheating on benchmarks for one of its processors; it's part of the company's culture.

Samsung: focus on user experience, not cheating at performance tests

The site noted, "you should be careful about comparing Exynos 5 Octa based Galaxy S 4s using any of the affected benchmarks to other devices and drawing conclusions based on that. This seems to be purely an optimization to produce repeatable (and high) results in CPU tests, and deliver the highest possible GPU performance benchmarks."

It further recommended, "what Samsung needs to do going forward is either open up these settings for all users/applications" or simply "remove the optimization altogether."

If Samsung keeps up software doping to run the Galaxy S 4 on steroids only when its being tested, the site warned that the industry will "end up in an arms race between all of the SoC and device makers where non-insignificant amounts of time and engineering effort is spent on gaming the benchmarks rather than improving user experience," something it noted has occured in the PC industry.