Samsung issues second, unsubstantiated denial of Galaxy S4, Note 3 benchmark cheating

In the face of a second set of investigative reports detailing exactly how Samsung leads other Android makers in exaggerating its performance on specific benchmarks, the company has issued its second denial this year, and which still fails to address any specifics.

In a statement issued to CNET UK today, Samsung stated, "The Galaxy Note 3 maximises its CPU/GPU frequencies when running features that demand substantial performance. This was not an attempt to exaggerate particular benchmarking results. We remain committed to providing our customers with the best possible user experience."

Samsung's benchmarks for the Galaxy Note 3 do indeed indicate that the device "maximises its CPU/GPU frequencies when running" benchmarks, which certainly are "features that demand substantial performance."

However, Samsung's claim that this "was not an attempt to exaggerate particular benchmarking results" is belied by the fact that the maximization only occurs when running specific benchmarks and stops happening when the name of benchmark app is changed.

Or when the benchmark maker specifically changes their benchmark to stop such cheating, as was the case this week with AnTuTu, one of the benchmark apps Samsung's flagship devices identify by name when running, expressly in order to exaggerate their results.

This all happened before

Back in July, Samsung offered the same explanation to John Paczkowski of the Wall Street Journal "All Things Digital" blog, which the site characterized as "Samsung’s Bizarre Benchmark-Boosting Explanation."

Samsung's statement then was nearly identical: "the maximum GPU [graphical processing unit] frequencies for the GALAXY S4 have been varied to provide optimal user experience for our customers, and were not intended to improve certain benchmark results."

However, the company also contradicted itself and confirmed what AnandTech had reported about specifically juicing certain benchmarks within the same statement, adding:

"A maximum GPU frequency of 533MHz is applicable for running apps that are usually used in full-screen mode," Samsung said, "such as the S Browser, Gallery, Camera, Video Player, and certain benchmarking apps, which also demand substantial performance."

Additionally, the facts showed back then that tests involving Samsung's Browser, Gallery and Video Player didn't results in a GPU boost, instead leaving the processor clocked at 266MHz. And of course, the name of the software controlling the change was "BenchmarkBooster."

So Samsung admitted boosting benchmarks back in July while also saying it didn't (in the same statement!), and said it was also boosting other apps that it actually wasn't. This time around, Samsung has only removed the explicit admission that it considers benchmarks among the apps that "demand substantial performance" and require special handling from its BenchmarkBooster code.

Why Samsung's cheating doesn't win

Computing benchmarks are similar to a road test designed to compare the speed and agility of different cars driving on a specific route. However, mobile benchmarks also aim to test not just the brute speed, but rather the overall performance typical of running within real-world efficiency parameters related to battery consumption and heat dissipation.

Mobile benchmarks are therefore like a test track that involves not just a speed element, but also additional constraints related to mileage, emissions and overheating, to show not just what a souped up modified vehicle could do, but instead how well a customer's car will actually perform in real world use. Samsung does not activate "maximized frequencies" when running real apps, because if it did, its other benchmarks would suffer, particularly battery life and product reliability.

Samsung activates various performance modes when running specific benchmarks, including activating all idle cores, increasing the clock rate and making other, unknown changes that boost its graphics performance over identical hardware.

While these changes boost scores by 20 to up to 50 percent (in AnandTech testing), they do so at the cost of overheating components and running down the battery. These impacts are so costly to the overall experience that Samsung does not activate "maximized frequencies" when running real apps, because if it did, its other benchmarks would suffer, particularly battery life and product reliability.

Samsung's efforts to shoo attention away from investigations by AnandTech and ArsTechnica weren't enough to convince Wall Street Journal or CNET, but did seem to pacify the concerns of some Android fans.

"Which is true? Will we ever know? Does it even matter?" asked Zach Epstein of BGR after reciting Samsung's statement.

"Bam! Straight from the horses mouth, they don't cheat!" one reader commented. Another, represented by an Android avatar with the caption "hope," wrote, "I agree. I will believe anything from Google/DROID/Samsung. They are honest. I will never listen to or believe anything from or about Apple."

A higher rated comment observed, "Cheating or not on benchmarks, their Exynos and Qualcomm chips they use in their Galaxy line up are getting beat on every benchmark by Apple's A7. So Samsung is playing catch up."