You’d be hard pressed nowadays NOT to see someone with some type of fitness tracker strapped to their wrist. (I mean, this isn’t even the first article we’ve written about this topic – Check Out: The Social Network: Peer Pressure and Fitness) Whether it be an Apple Watch, FitBit, Nike+, or Garmin – we rely on our fitness trackers to let us know where we’re at for the day in terms of heart rate, calories burned, and steps walked. But the question still remains: Just how accurate is their count??? Well, the answer may or may not surprise you…. Not Very (But good enough). Did I just confuse you there? Let’s check out the data:
Researchers at Japan’s National Institute of Health and Nutrition ran a study that compared 12 different wearable devices, including the Fitbit Flex, Jawbone UP24, Misfit Shine, Garmin Vivofit and Withings Pulse O2, to two widely accepted methods of estimating calorie burn. The results were published in an edition of JAMA Internal Medicine. In the study, 19 volunteers wore all 12 wearables AT THE SAME TIME. (Yes – you read that right). SIX wrist bands, FOUR waist-mounted devices, and TWO pocket devices were all worn on the same individual AT THE SAME TIME.
In the first part of the experiment, the individuals spent 24 hours of the experiment in a metabolic chamber while following a standardized protocol that included “3 meals, desk work, watching TV, housework, treadmill walking, and sleeping” while their caloric expenditure was measured by carefully monitoring the chamber’s temperature, gas composition, etc. On average, the subjects burned 2,093 calories during this 24-hour period. You can see how much the wearable fitness trackers deviated from that value below:
Figure 1: Deviation of calories tracked versus Metabolic Chamber Measurements.
Consistent differences in the devices showed up across the board. For example, the Jawbone and Garmin devices underestimated calorie burn by a couple hundred calories on average, while the Fitbit and Misfit both overestimated.
In the second part of the experiment, the subjects were taken out of the lab and into the field. The subjects drank water that contained rare isotopes of both hydrogen and water. Their calorie burn could be estimated by tracking how long it took for those isotopes to appear in their urine. The subjects then spent the next 15 days living their normal lives – while wearing all 12 devices AT THE SAME TIME – and collecting their urine for later analysis. They were only allowed to take the trackers off when bathing, charging the battery, or in special cases when wearing the tracker would be difficult for them. Over those 15 days, the subjects burned an average of 2,314 calories per day. Again, you can see how much the wearable fitness trackers deviated from that value below:
Figure 2: Left – Deviation of calories tracked versus Metabolic Chamber Measurements (as seen in the previous Figure 1). Right – Deviation of calories tracked versus Isotope Measurements.
And again, consistent differences in the devices showed up across the board – but this time ALL devices underestimated calorie burn but the overall trend stayed the same – . the Jawbone and Garmin tracked lower values while the Fitbit and Misfit tracked higher values. The underestimation isn’t really surprising, after all, over a 15-day period, there would be more instances where the device would be taken off – i.e.bathing, charging the battery, or in special cases when wearing the tracker would be difficult for user.
The conclusion of the study was that the findings presented suggested that most wearable devices did NOT produce a valid measure of total energy expenditure. AND I totally get that – from a research perspective, where an actual concrete number of calories being burned needs to be tracked – that most definitely is the case. HOWEVER, when it comes to a real-world scenario, the results were actually NOT THAT BAD. When we track our calories day-to-day what we’re really looking for is getting consistent relative results so you, the wearer, can tell whether you’re burning more or less calories overall as compared to another day in the week. AND these trackers are pretty damn good at telling you that. Statistical analysis showed that for the most part the devices correctly ranked the individual participants against each other (i.e. if the isotope water showed that you were the 3rd highest calorie burner – so did most of the wearables). And for self-tracking, that’s probably good enough.