Capriciousness of little health benefits

May 15, 2018

(This article was originally published at Big Data, Plainly Spoken (aka Numbers Rule Your World), and syndicated at StatsBlogs.)


Typing "health benefits" into the search box on Google News gives me a long list of activities and foods that are supposedly good for me. Scanning the headlines, I see fasting, running, garlic, honey, tumeric, amchur, drinking hydrogen peroxide, gardening, low-carb diets, green tea, hummus, etc.

I am a skeptic when it comes to these claims. Here's how I think about them.

Hypothetically, someone – hopefully a nutrition scientist – is making the claim that a substance X will lead to an improvement in health metric Y. The effect of X on Y has a direction and a magnitude. The direction of the effect can be positive or negative while its magnitude is either large or small. So, any effect X on Y is captured by one of the four squares as shown.


Most reported effects X on Y tend to be small and positive. As such:

1) We are talking about a small average effect. This is wrongly interpreted as meaning everyone who takes substance X will accrue a small benefit on Y. That's usually not how we obtain a small average effect. A better interpretation is that a small proportion of people who take substance X will accrue a benefit. Most people who take X won't.

You might still argue with me in this way: I understand I am just buying a lottery ticket; the chance of winning is low but what's the harm? This takes me to:

2) When the effect size is small, it's possible that the direction of the effect is wrongly measured. The difference between a small positive effect and a small negative effect is not much! Instead of a benefit Y, you might end up with a harm Z.

The kicker: if the small magnitude of the benefit is enough for you to take seriously, then you should also worry about an effect of equal but opposite magnitude.

Or, you just decide that you don't care about these little effects, which is why I don't pay much attention to those news stories.

Please comment on the article here: Big Data, Plainly Spoken (aka Numbers Rule Your World)

Tags: , , , , , , , ,