Rampant cheating by Android knockoffs prompts AnTuTu to launch new benchmark

While some U.S. tech media reports have portrayed Samsung's benchmark fraud as a minor embarrassment related to numbers "nobody cares about," faking benchmarks is a big business in Asia, particularly among White Box Tablets, a market segment the U.S. tech media does consider relevant when talking about Apple's iPad market share of "shipments."

A recent report by TechCrunch described Samsung's fraudulent efforts to get its "Galaxy" product model names to the top of benchmark charts as a dubious benefit only likely catch the attention of some tech enthusiasts.

However, in Asia and particularly China, faking benchmarks is a huge business because the market for cheap, domestic "White Box" iPhone and iPad knockoffs is legitimized by reassuring benchmark scores.

Samsung has faked benchmark scores for its products regardless of their internals, having cheated for not only its own Exynos 5 chip in the Galaxy S4, but also for its Galaxy Note 3 using a Qualcomm Snapdragon chip and its Galaxy Tab 3 10.1 powered by an Intel CloverTrail+ processor. Such benchmark fudging is a practice that's opposed by Qualcomm and Intel, as noted by AnandTech.

In China, however, component makers and device assemblers are commonly united in creating fake benchmarks that portray their devices to be worth more than they are, or of similar quality to the products they are knocking off. The problem is so out of hand that earlier this week, popular Android benchmark AnTuTu, which bills itself as an "impartial and unbiased smart phones benchmark," announced a new version of its tool specifically designed to work around the rampant cheating.

How many ways a cheater can cheat

Not all "optimizations" are full blown cheating, but many are. As AnTuTu explained, "there are some smartphone makers who do system optimization especially aiming at AnTuTu Benchmark to improve the test scores. A legal system optimization operation on devices is what we encourage, but the action of tricking customers by cheating is what we are firmly against."

The group describes a variety of ways the Android hardware makers are cheating end users. The most egregious way is to modify AnTuTu's benchmark tool to report "fake hardware specs list and test scores," claiming, for example, more RAM and faster components than are even present. "This kind of cheating method always turns out on high imitation and fake devices," the group states.

A second cheat is similar to what Samsung has been doing: temporarily boosting the results when the device senses that a benchmark is running. "As a consequence, the result customers saw is differ with the real fluency feel in daily use," the AnTuTu states.

Another cheat type involves faking graphics performance. "After recognition of AnTuTu Benchmark and other Benchmark, some chips will self omit test frames, while the system shows correctly. For example, when operating 3DRating test, the frames number which the system has received is 10, while the real frames showed is only 5 per minutes, in this case, test scores will have huge improvement."

The group noted, "We [at] AnTuTu feel sorry about these behaviors and we need to alert all users to these behaviors of Fake-phone makers adopted on devices."

"New Android White Box Tablets" just like old PC netbooks

As blogger Mike Cane notes, the scope of cheating among China's "White Box" iPad clones goes far beyond faking benchmarks. Sloppy efforts to create knockoff products infect the entire manufacturing chain, "from the chipmakers to the component suppliers to the tablet makers."

"Chipmakers," Cane writes, "find a way to make their chips detect AnTuTu and then run a certain way to produce a better score. They also claim their chips will run at a certain speed but tablet makers must downclock them because of chip yield problems — which the chipmakers paper over by simply discounting their chips. Component makers get customers not on the basis of quality but by cutting deals on price."

On top, "tablet makers hype their products without full and honest specs and with abysmal warranties, return policies, and repair policies. One investigation into tablets revealed a defect rate of one in three!" he noted.

Cane referenced the White Box Tablet market that analysts like Strategy Analytics are directly comparing to the iPad, noting, "on the whole, domestic Chinese tablet makers — en toto — are held in low regard by the Chinese themselves. Yet we outside of China think we’re getting some sort of bargain!"

Source: Mike Cane

One example of this is in screen quality. Products like the "Teclast P89 Mini," "Cimi/Simi X8," "Onda V819 Mini" and "Vido Mini One" essentially claim to be lower cost, generic versions of Apple's iPad mini. However, they ship with screens that can't reproduce the same color depth, resulting in washed out images (a broad range of fakes, depicted above). Forum boards are also full of users complaining about failed chargers, dysfunctional buttons and unstable software.

The state of China's Android-based White Box Tablets is virtually identical to the low quality PC netbooks that briefly generated big shipments before collapsing at the feet of unimpressed consumers as Apple's iPad began to gain traction immediately after its launch in 2010. That hasn't stopped analysts' regular comparisons of Apple's consistent iPad sales against the global "shipments" of such White Box devices, arriving at the conclusion that Apple's "market share" is plummeting in Asia.

The extremely poor quality of such low end builds helps to explain why purported shipments of Android tablets aren't showing up in web logs and other metrics of customer use; rather than cheap iPads, these White Box Tablets are actually more like very expensive ewaste, destined to be thrown in a drawer rather than put to use.