Alzheimer’s Mouse research on the Orient Express

January 9, 2018

(This article was originally published at Statistical Modeling, Causal Inference, and Social Science, and syndicated at StatsBlogs.)

Paul Alper sends along an article from Joy Victory at Health News Review, shooting down a bunch of newspaper headlines (“Extra virgin olive oil staves off Alzheimer’s, preserves memory, new study shows” from USA Today, the only marginally better “Can extra-virgin olive oil preserve memory and prevent Alzheimer’s?” from the Atlanta Journal-Constitution, and the better but still misleading “Temple finds olive oil is good for the brain — in mice” from the Philadelphia Inquirer) which were based on a university’s misleading press release. That’s a story we’ve heard before.

The clickbait also made its way into traditionally respected outlets Newsweek and Voice of America. And NPR, kinda.

Here’s Joy Victory:

It’s pretty great clickbait—a common, devastating disease cured by something many of us already have in our pantries! . . . To deconstruct how this went off the rails, let’s start with the university news release sent to journalists: “Temple study: Extra-virgin olive oil preserves memory & protects brain against Alzheimer’s.”

That’s a headline that surely got journalists’ attention. It’s not until after two very long opening paragraphs extolling the virtues of the nearly magical powers of extra virgin olive oil that we find out who, exactly this was tested on.

Mice. . . .

It’s never mentioned in the news release, but this hypothesis was tested on only 22 mice, just 10 of which got the olive oil rich diet, and 12 of which got a standard diet.

As in: The sample size here is so small that we can’t be very sure what the results are telling us.

The release does at least make it pretty clear that these were also genetically modified mice, who had their DNA fracked with so that they developed “three key characteristics of the disease: memory impairment, amyloid plaques, and neurofibrillary tangles.”

Victory continues:

Stating the obvious, here, but genetically modified mice are a far, far cry from people. . . .

In fact, even drugs that apparently do a great job of getting rid of amyloid in thousands of actual humans don’t seem to have much effect on the symptoms of Alzheimer’s disease. . . .

There are other limitations of the study that should have been in the news release, but we’ll stop here.

I looked briefly at the published research article and am concerned about forking paths, type M errors, and type S errors. Put briefly, I doubt such strong results would show up in a replication of this study.

“No actual interviewing took place.”

Victory continues her article with a discussion of the news coverage:

With a news release like this, journalists were primed to do a poor job writing about the study.

To its credit, the Inquirer was upfront about this being a mouse study. . . . Yet, the story included no independent experts to offer some perspective—something we saw across the board in the stories we read.

USA Today and the Atlanta Journal Constitution didn’t even bother to disclose that all the quotes in their stories come directly from the news release. As in: No actual interviewing took place.


Victory concludes:

This is all great news for Temple’s PR team—their messaging made it out to the public with very little editing. But this isn’t great news for the public.

Countless people across the country who have loved ones with dementia likely read these stories with a whole lot of hope.

What they get instead is a whole lot of hype.

It’s ok to publish raw data and speculation—just present them as such.

I have no objection to this research being published—indeed, a key reason for having a scientific publication process in the first place is so that these sort of preliminary, exploratory results can be shared, and others can replicate.

The key is to present the results as data rather than as strong conclusions, and to avoid this sort of strong conclusion:

Taken together, our findings support a beneficial effect of EVOO consumption on all major features of the AD phenotype (behavioral deficits, synaptic pathology, Aβ and tau neuropathology), and demonstrate that autophagy activation is the mechanism underlying these biological actions.

That’s from the published paper. Can they really conclude all that from 22 genetically modified mice and a pile of bar graphs and p-values? I don’t think so.

The ethics question

Is it unethical for a team of researchers to overstate their results? Is it unethical for a university public relations office to hype a statistical study about 22 genetically modified mice with the following inaccurate headline: “Extra-virgin olive oil preserves memory & protects brain against Alzheimer’s”?

I feel like everyone’s passing the buck here:
– Newspapers don’t have the resources to spend on science reporting;
– Science writers are busy and feel under pressure to present good news, maybe even attach themselves to the kind of “breakthrough” they can write about in more detail;
– Lots of pressure to get “clicks”;
– Public relations offices are judged on media exposure, not accuracy;
– Academic researchers know that it’s hard to publish in top journals without making big claims.

Somewhere along the line, though, something unethical seems to be happening. And it seems worth discussing in a context such as this: legitimate (I assume) research on an important topic that is just being majorly hyped. This is not the Brian Wansink show, it’s not power pose or himmicanes or ovulation and clothing or beauty and sex ratio or all the other ridiculous noise-mining exercises that we’ve used to sharpen our intuition about understanding and communicating variation. It’s potentially real research, being unethically hyped. But the hyping is done in stages. It’s a Murder on the Orient Express situation in which . . . . SPOILER ALERT! . . . all the players are guilty.

And all this is happening in a media environment that has less journalism and more public relations than in the past. Fewer eyes on the street, as Jane Jacobs would say.

The post <del>Alzheimer’s</del> Mouse research on the Orient Express appeared first on Statistical Modeling, Causal Inference, and Social Science.

Please comment on the article here: Statistical Modeling, Causal Inference, and Social Science

Tags: , , ,