Some people asked me what I thought about this story. A reporter wrote to me about it last week, asking if it looked like fraud. Here’s my reply:
Based on the description, there does not seem to be the implication of fraud. The editor’s report mentioned “protocol deviations, including the enrollment of participants who were not randomized.” I suppose this could be fraud, or it could be simply incompetence, or it could be something in between, for example maybe the researchers realized there were problems with the randomization but they didn’t bother to mention it in the paper because they thought it was no big deal, all studies are imperfect, etc. I have no idea. In any case, it’s good that the NEJM looked into this.
Looking at the new article, it sees that it was only a small subset of patients that were not randomized as designed: out of a total of 7447 patients, 425 shared a household with a previously enrolled participant, and 467 participants were at the site where patients were not randomized. There may be some overlap, but, even if not, this is only 892 people, that’s 12% of the people in the study. So the easiest thing is to just analyze the 88% of the people who were in the trial as designed. Sure, that’s throwing away some data, but that will be the cleanest way to go.
I’m surprised this story is getting so much play, as it appears that the conclusions weren’t really changed by the removal of the questionable 12% of the data. (According to the above-linked report, the data problems were “affecting 10 percent of respondents,” and according to this news article, it’s 14%. So now we have three numbers: 10% from one report, 14% from another, and 12% from my crude calculation. A minor mystery, I guess.)
It says here that the researchers “spent a year working on the re-analysis.” If it was me, I think I would’ve taken the easy way out just analyzed the 88% (or 86%, or 90%) of the data that were clean, but I can’t fault them for trying to squeeze out as much information as they could. I’m similarly confused by the quotes from skeptical statisticians. Is the skepticism on specific statistical grounds, or is it just that, if 12% of the study was botched, there’s concern about other, undiscovered biases lurking in the data? As usual when the study involves real-life decisions, I end up with more questions than answers.
P.S. Just read Hilda Bastian’s discussion. Lots of interesting details.