Daniel Eran Dilger
Futuremark has delisted the scores of Samsung and HTC products from its public benchmark listings over new allegations of performance doping.
According to a report by Tony Smith of the Register, Futuremark accused the two companies of rigging results in its 3DMark GPU benchmarking app for Android.
The company removed the scores of Samsung's Galaxy Note 10.1 tablet and Note III phablet as well as HTC's One and One Mini smartphones, dropping the models to the bottom of its listings, below devices with a score of zero (above).
Futuremark President Oliver Baltuch told the site, "People rely on Futuremark benchmarks to produce accurate and unbiased results. That's why we have clear rules for hardware manufacturers and software developers that specify how a platform can interact with our benchmark software.
Baltuch explained, "in simple terms, a device must run our benchmarks without modification as if they were any other application."
Futuremark's rules stated "the platform may not detect the launch of the benchmark executable" and that devices "must not alter, replace or override any parameters or parts of the test, nor modify the usual functioning of the platform based on the detection of the benchmark."
The leading device in Futuremark's listings remains the Samsung Galaxy Round, version of the Note III with a curved body and screen and lacking a stylus. The low volume product is offered in Korea for close to $1000, essentially as a technology demonstration.
This all happened before
Samsung was previously caught faking benchmark scores for the Exynos 5 version of the Galaxy S4, as well as its Galaxy Note 3 using a Qualcomm Snapdragon chip and its Galaxy Tab 3 10.1 powered by an Intel CloverTrail+ processor.
This kind of benchmark fudging is a practice opposed by Qualcomm and Intel, as noted by AnandTech, which called attention to the benchmark doping in its testing this summer.
Popular benchmark vendor AnTuTu announced a new version of its app designed to work around cheating by Samsung and other companies early last month.
Samsung has issued multiple, contradictory denials that it is doing anything wrong, and certain Android fan sites have defended the practice.
While the difference in numbers that such cheating produces appears to be small, it allows Samsung to claim, for example, a performance edge over Apple's iPhone 5c with its own Galaxy S 4 flagship model, when in reality the S4 is about the same or scores slightly lower (above) when it is tested without being able to cheat.
When allowed to cheat, Samsung's Galaxy S4 appears to be nearly as fast as Apple's higher end iPhone 5s, and the Galaxy Note III appears to be faster, despite actually being slower in real world tasks, even though it has more memory, a larger battery and is set to run at a faster clock speed.
Amber Neely
Andrew Orr
William Gallagher
Christine McKee
AppleInsider Staff
Chip Loder
169 Comments
Just imagine the uproar if Apple did this. We'd never hear the end of it.
We can expect no less than cheating from manufacturers when their entire Android borg is based upon theft and deceit.
So when is the government going to get involved here?
That sort of behavior is beyond silly. You can expect if not accept some tiny start ups to do all sorts of crazy guerrilla marketing ideas to try and stand out and get some attention and traction but not from multi-billon dollar corporations. That is just sleazy. I hope all these benchmarking apps manage to thwart their cheating attempts by updating their apps to avoid the over clock cheat.
At the end of the day unless all you do is run benchmarking apps they really don't mean much. I have to admit I am curious to see how my 5s stacks up to the competition but only if the tests are not skewed by over clocking. I would also be very curious to see actual day to day type of benchmark tests that measure common tasks and apps.
I'm curious to see what insane scores a doped up iPhone 5S could crank out.