Testing stalwart Consumer Reports declared this week that Apple's HomePod falls behind Google's Home Max -- and even the Sonos One -- in terms of audio quality. To anyone who knows the publication's controversial history with Apple products, that conclusion is utterly unsurprising.
In Friday's screed, the same day as the product was released, Consumer Reports, after less than a day of use and comparison, declared the Sonos One, and the Google Home Max the champions for sound quality. That's noteworthy, because virtually every other major test conducted since the launch of HomePod, including AppleInsider's own comparisons, found that Apple's product offers superior sound.
Don't just take our word for it (twice), though. Engadget on Tuesday agreed with the prevailing wisdom on it, as did USA Today, The Verge, What Hi-Fi, a very particular Redditor, and a pile of other venues.
Consumer Reports is, of course, entitled to its opinion -- sound quality assessments are highly subjective.
But two factors suggest that the report isn't relevant, nor of any particular value. Primarily, Consumer Reports has a long history of berating Apple products for issues that don't seem to actually manifest in any real-world environment. Second, their HomePod comparison has come to a conclusion that only they seem to hold.
Antennagate
Back in 2010, Consumer Reports used its public-facing webpage to flog issues it had with the iPhone 4 antenna. But it required a subscription to the site to view the entire comparative analysis, where it ranked the iPhone 4 above all other smartphones of the day, saying that the antenna issue was completely solved by applying piece of transparent tape over the antenna line in the casing, using a protective case, or not squeezing it with a "death grip" while making a phone call.
That fired up the internet as a whole, who saw only the public-facing post decrying the antenna -- yet the same complainants were not widely exposed to the full report stashed behind a paywall that said that the problems were incredibly minor and the iPhone 4 was at the top of the heap.
As a result of the growing controversy, Apple uncharacteristically made a public statement, and provided free cases for early adopters, in an attempt to put the issue to bed.
Gripping any mobile phone will result in some attenuation of its antenna performance, with certain places being worse than others depending on the placement of the antennas. This is a fact of life for every wireless phone. If you ever experience this on your iPhone 4, avoid gripping it in the lower left corner in a way that covers both sides of the black strip in the metal band, or simply use one of many available cases.
The laws of physics are inescapable, and every smartphone even now can be shielded by human flesh when pressed up against an antenna gap
And yet, somehow, this only applied to the iPhone 4, and not the other devices on the list according to the venue -- even though at the same time literally every other phone on their evaluation was seeing the same thing.
Consumer Reports didn't bother to say anything about it afterward.
The 2016 MacBook Pro
Apple released the MacBook Pro redesign in the fall of 2016, and there was much to be said about it. As it always does, Consumer Reports got their hands on one shortly after the machines shipped in late October and early November.
After about a month and a half of testing, in a post on their website, the publication said that while the new machines earned high marks in display quality and performance, they were found lacking in terms of battery life. Specifically, the battery performance of models tested "varied dramatically" during trials, and was as low as four hours, with the computers that were not under serious load.
However, it was discovered that, during their testing, Consumer Reports toggled a hidden developer setting, which triggered an obscure bug. While the bug's presence was Apple's responsibility, tooling around in the developer's settings for Safari isn't a "real world" test, contrary to what publication claims.
"This is not a setting used by customers and does not reflect real-world usage. Their use of this developer setting also triggered an obscure and intermittent bug reloading icons which created inconsistent results in their lab," Apple said. "After we asked Consumer Reports to run the same test using normal user settings, they told us their MacBook Pro systems consistently delivered the expected battery life."
Following the bug fix, and Consumer Reports not messing around with developer settings, the product testers reinstated their recommendation for the machine -- an improvement over how they handled the iPhone 4 debacle.
Unlike its declaration about the HomePod Monday just three days after release, Consumer Reports waited until its month-long testing was fully complete on the MacBook Pro before it published the results.
The iPhone X
The publication also disclosed its testing on the iPhone X and a bevy of other smartphones on Dec. 5 of last year, a month after the iPhone X became available. That report found the publication recommending the Galaxy S8 above the iPhone 8 Plus, with the iPhone X in a very close third spot.
The testing was dramatic, with the phones taking major screen damage in a steel tumbling device after 50 rotations. We're not sure how "real world" this is, as the vast majority of iPhone drops are from approximately three feet to concrete, and only a few times in a product's life.
The iPhone X tumbler
However, again, Consumer Reports didn't talk about early results until they had been beating on the phones in question for a month -- more than 10 times the length of time it had before it judged the HomePod.
Review challenges
Consumer Reportshas every legal right to share its opinions, the same as AppleInsider or any other venue. Examining a product in a review is, by definition, a subjective process, with audio product reviews producing the widest gap of opinion.
If you're looking for an "objective review," or similar phrasing, you aren't looking for a review at all. Instead, find a news item dispassionately discussing the technical specifications of a product. By definition, a review isn't that, and is chock-full of opinions about a product.
Perhaps Consumer Reports felt like they had collected all the information they needed after just a few days. That's possible, and with a product evaluated in a vacuum, and not compared to much else, a review can be done in less than that timeframe. However, this is contrary to every other consumer electronics comparison that the organization has ever done.
Damning with faint praise
It would have been better had Consumer Reports waited until their comparative testing was done to scream loudly that one smart speaker sounds better than the next. But, instead, this nearly century-old publication decided to say very little about how they reached their conclusions. Instead, they sensationally disclosed their findings, some of which flew contrary to everyone else's. And they did so just days after a product release, and yet they still give themselves an out if their exhaustive testing proved otherwise.
Why they took this atypical path to publication, we don't know, and can only guess, since they have yet to respond to questions about the release.
Right now, we don't know anything about the testing methodology. We don't know why Consumer Reports decided what it did, despite their press release vividly declaring that Google's offering is better.
I am not here to defend the HomePod, and Apple neither requires nor desires my assistance -- and that's how it should be. I don't have a HomePod, nor do I have a Google Home Max, so I can't speak to the accuracy of the Consumer Reports declaration.
But Monday's report, how it was disclosed, and a lack of a supporting narrative for the opinion, is not a good look for Consumer Reports, given their history with Apple products. The release is contrary to their own stated mission of "rigorously testing products" before advising the public.
Worse yet, the early announcement stands in violation of what I think is their most profound mission -- they have failed to be a publication capable of "fearlessly investigating where markets have failed," and look to have become part of the problem instead.
Update: Consumer Reports published a rebuttal on Tuesday evening. They noted in the rebuttal that they published their first look at the HomePod on Friday, and not Monday, and differed with some of our interpretations of the examination -- but we stand by our stance on the matter.
AppleInsider has asked some follow-up questions, specifically about how Consumer Reports came to their conclusion about the HomePod and their interpretation of the MacBook Pro testing regimen versus what Apple had to say about the dialog, and will update accordingly.