Stanford study finds Apple Watch top-notch heart rate monitor, mediocre calorie counter

Published in the Journal of Personalized Medicine on Wednesday, the study seeks to determine the validity of readings from commonly worn fitness trackers. The growing number of consumers buying and wearing devices with biometric capabilities presents a unique opportunity for preventative cardiovascular medicine, but error rates of these commercial products are largely unknown, the study says.

"People are basing life decisions on the data provided by these devices," Euan Ashley, DPhil, FRCP, professor of cardiovascular medicine, of genetics and of biomedical data science at Stanford said in a statement. He went on to say that consumer devices are not bound by the same regulations as medical-grade equipment, making it difficult for doctors to quantify or otherwise apply generated data to diagnoses.

To better understand the limitations of popular fitness trackers, the study pit the Apple Watch, Basis Peak, Fitbit Surge, Microsoft Band, MIO Alpha 2, PulseOn and Samsung Gear S2 against FDA approved equipment.

A total of 60 volunteers (31 women and 29 men) donned up to four consumer devices and participated in 80 physical tests ranging from cycling to running. Test subjects were simultaneously monitored by a 12-lead electrocardiogram and continuous clinical grade indirect calorimetry, the latter measuring for expired gas.

Researchers set an acceptable error rate at 5 percent.

Apple Watch achieved the highest heart rate accuracy across measured modes of activity with an error rate of 2 percent, followed by the Basis Peak and Fitbit Surge. Samsung's Gear S2 exhibited the highest HR error rate at 6.8 percent, outside of the study's acceptable limits.

All devices fared poorly in energy expenditure, or calorie counting, tests. The most accurate device, Fitbit's Surge, managed an error rate of 27.4 percent, while the least accurate product, the PulseOn, put in a dismal performance of 92.6 percent. Interestingly, the devices logged the lowest error rates during activities like walking and running, while low impact tasks like sitting tracked measurably worse with an average error rate of 52.4 percent.

"The heart rate measurements performed far better than we expected, but the energy expenditure measures were way off the mark," Ashley said. "The magnitude of just how bad they were surprised me."

Researchers were unsure as to why energy expenditure rates were so far off from gold standard equipment, but the study notes each device uses its own proprietary algorithm for calculating calorie burn. These calculations are in large part based on individual user metrics like height, weight, BMI, fitness level, age and more. Whereas heart rate is measured directly from a user's wrist, calorie burn is an estimate derived through complex algorithms.

Ashley and his team are working on an extension to the study that takes testing beyond the laboratory and out into the real world.