Jonathan Falk writes:
Here’s an interesting story right in your sweet spot:
Large effects from something whose possible effects couldn’t be that large? Check.
Finding something in a sample of 1024 people that requires 34,000 to gain adequate power? Check.
Misuse of p values? Check
Science journalist hype? Check
Searching for the cause of an effect that isn’t real enough to matter? Check.
Searching for random associatedness? (Nostalgia-proneness?) Check.
Multiple negative studies ignored? Check.And some great observations from Scott Alexander [author of the linked post]:
First, what bothers me isn’t just that people said 5-HTTLPR mattered and it didn’t. It’s that we built whole imaginary edifices, whole castles in the air on top of this idea of 5-HTTLPR mattering. We “figured out” how 5-HTTLPR exerted its effects, what parts of the brain it was active in, what sorts of things it interacted with, how its effects were enhanced or suppressed by the effects of other imaginary depression genes. This isn’t just an explorer coming back from the Orient and claiming there are unicorns there. It’s the explorer describing the life cycle of unicorns, what unicorns eat, all the different subspecies of unicorn, which cuts of unicorn meat are tastiest, and a blow-by-blow account of a wrestling match between unicorns and Bigfoot.
Alexander links to this letter by
Nina Rieckmann, Michael Rapp, and Jacqueline Müller-Nordhorn published in 2009 in the Journal of the American Medical Association:
Dr Risch and colleagues concluded that the results of a study showing that the serotonin transporter gene (5-HTTLPR) genotype moderates the effect of stressful life events on the risk of depression could not be replicated in a meta-analysis of 14 studies. The authors pointed out the importance of replication studies before new findings are translated into clinical and health practices. We believe that it is also important to note that editorial practices of scientific journals may contribute to the lack of attention received by studies that fail to replicate original findings.
The original study was published in 2003 in Science, a prominent journal with a very high impact factor that year. In the year following its publication, it was cited 110 times in sources indexed in the Web of Science citation report. In 2005, the first study that failed to replicate the original finding in a sample of 1091 participants was published in Psychological Medicine, a specialized journal with a relatively low impact factor. That study was cited 24 times in the following year.
We believe that unless editors actively encourage the submission of null findings, replication studies, and contradictory results alike, the premature uncritical adoption of new findings will continue to influence the way resources are allocated in research and clinical practice settings. Studies that do not replicate an important finding and that meet high methodological standards should be considered for publication by influential journals at the same level as the respective original reports. This will encourage researchers to conduct replication studies and to make primary data easily accessible.
I can’t really comment on the substance or the statistics here, since I haven’t read any of these articles, nor do I have any understanding of the arguments from genetics.
Setting all that aside, it’s striking (a) that Rieckmann et al. made this impassioned speech back in 2009, well before the general awareness of replication issues in science, and (b) this was all 10 years ago but it seems that this is still a live issue, as Alexander is writing about it now. Rieckmann et al. must find all this very frustrating.
Jonathan pointed me to this story in May, and I informed him that this post would appear in Oct. Given that the problem was pointed out 10 years ago (in JAMA, no less!), I guess there’s no rush.
P.S. More on this from savvy science writer Ed Yong.