Study: Wearables’ accuracy varies widely for caloric burn

A new paper published yesterday in JAMA Internal Medicine shows that, when it comes to measuring energy expenditure, many leading fitness trackers have a margin of error of about 200 calories per day in either direction.

The study was conducted by Japanese researchers led by Haruka Murakami from the National Institute of Health and Nutrition in Tokyo. Nineteen participants, aged 21 to 50, wore 12 fitness trackers a piece, a mix of popular consumer devices (Fitbit Flex, Jawbone UP24, Misfit Shine, Epson Pulsense PS-100, Garmin Vivofit, Tanita AM-160, Omron CalorieScan HJA-403C, and Withings Pulse O2) and devices more often used in healthcare (Panasonic Actimarker EW 4800, Suzuken Lifecorder EX, Omron Active style Pro HJA-350IT, and ActiGraph GT3X).

The readings from these devices were compared to two more robust ways of tracking calories. In the first experiment, the participants spent 24 hours in a metabolic chamber where scientists could track oxygen consumption and carbon dioxide production, and assess calorie expenditure from that. In the second, researchers used urine samples to calculate participants’ calorie expenditures in their normal lives.

In the metabolic chamber test, the Withings and Jawbone devices underestimated expenditure by about 270 calories while both Omron devices overestimated calories by about 200 calories. The other devices all fell in between that range, with the Panasonic and Epson devices the closest to the standard.

In the urine test, on the other hand, all the devices underestimated calorie output, by margins ranging from 69 calories short (the Omron Active Style Pro) to 590 calories short (the Jawbone UP24).

Researchers noted a few factors that might have contributed to the disconnect, including the fact that participants couldn’t wear the trackers all the time. They also noted that they tested only a healthy weight population, and that future studies should include overweight participants as well.

In an accompanying commentary, Dr. Adam Schoenfeld, an internist at UCSF, wrote about how this study demonstrates the need for FDA regulation of mobile apps that are going to be used as medical devices.

“Use of health-related apps may pose little risk in healthy persons if the information collected is not used for medical decision-making, but these apps should still undergo basic testing for accuracy,” he wrote. “For mHealth apps that are designed to play a role in medical care, documenting the validity and accuracy of measurements is crucial.”

Plenty of recent studies, of varying degrees of robustness, have called into question the accuracy of fitness trackers. Particularly relevant to this study is one from February by the American Council on Exercise, which also found that commercial fitness trackers had accuracy problems in the realm of calorie tracking.

At the time we wondered how important clinical accuracy really is to activity trackers, provided they successfully provide motivation and behavior change. There’s still something to that response, but at a time when more and more consumer activity trackers are showing up in clinical studies, it also makes sense to determine how accurate these devices really are.