(This article was originally published at Statistical Modeling, Causal Inference, and Social Science, and syndicated at StatsBlogs.)
We recently discussed an article by Isabel Scott and Nicholas Pound entitled, “Menstrual Cycle Phase Does Not Predict Political Conservatism,” in which Scott and Pound definitively shot down some research that was so ridiculous it never even deserved the dignity of being shot down. The trouble is, the original article, “The Fluctuating Female Vote: Politics, Religion, and the Ovulatory Cycle,” had been published in Psychological Science, a journal which is psychology’s version of Science and Nature and PPNAS, a “tabloid” that goes for short sloppy papers with headline-worthy claims. So, Scott and Pound went to the trouble of trying and failing to replicate; as they reported:
We found no evidence of a relationship between estimated cyclical fertility changes and conservatism, and no evidence of an interaction between relationship status and cyclical fertility in determining political attitudes.
The thing that bugged me when Scott and Pound wrote:
Our results are therefore difficult to reconcile with those of Durante et al, particularly since we attempted the analyses using a range of approaches and exclusion criteria, including tests similar to those used by Durante et al, and our results were similar under all of them.
As I wrote earlier: Huh? Why “difficult to reconcile”? The reconciliation seems obvious to me: There’s no evidence of anything going on here. Durante et al. had a small noisy dataset and went all garden-of-forking-paths on it. And they found a statistically significant comparison in one of their interactions. No news here.
Scott and Pound continued:
Lack of statistical power does not seem a likely explanation for the discrepancy between our results and those reported in Durante et al, since even after the most restrictive exclusion criteria were applied, we retained a sample large enough to detect a moderate effect . . .
Again, I feel like they’re missing the elephant in the room: “Lack of statistical power” is exactly what was going on with the original study by Durante et al.: They were estimating tiny effects in a very noisy way. It was a kangaroo situation.
Anyway, I suspect (but am not sure) that Scott and Pound agree with me that Durante et al. were chasing noise and that there’s no problem at all what had been found in that earlier study (bits of statistical significance in the garden of forking paths) with what was found in the replication (no pattern).
And I suspect (but am not sure) that Scott and Pound wrote the way they did in order to make a sort of minimalist argument. Instead of saying that the earlier study is consistent with pure noise, and their replication is also consistent with that, they make a weaker, innocent-sounding statement that the two studies are “difficult reconcile,” leaving the rest of us to read between the lines.
And so here’s my question: When is it appropriate to make a minimalist argument, and when is it appropriate to say what you really think?
My earlier post had a comment from Mb, who wrote:
By not being able to explain the discrepancy with problems of the replication study or “theoretically interesting” measurement differences, they are showing that the non-replication is likely due to low power etc of the original study. It is a rhetorical device to convince those skeptical of the replication.
I replied that this makes sense. I just wonder when such rhetorical devices are a good idea. The question is whether it makes sense to say what you really think, or whether it’s better to understate to make a more bulletproof argument. This has come up occasionally in blog comments: I’ll say XYZ and a comment will say I should’ve just said XY or even just X because that would make my case stronger. My usual reply is that I’m not trying to make a case, I’m just trying to share my understanding of the problem. But I know that’s not the only game.
P.S. Just to clarify: I think strategic communication and honesty are both valid approaches. I’m not saying my no-filter approach is best, nor do I think it’s correct to say that savviness is always the right way to go either. It’s probably good that there are people like me who speak our minds, and people like Scott and Pound who are more careful. (Or of course maybe Scott and Pound don’t agree with me on this at all; I’m just imputing to them my attitude on the scientific questions here.)
The post Being polite vs. saying what we really think appeared first on Statistical Modeling, Causal Inference, and Social Science.
Please comment on the article here: Statistical Modeling, Causal Inference, and Social Science
