(This article was originally published at Statistical Modeling, Causal Inference, and Social Science, and syndicated at StatsBlogs.)
People point me to things on the internet that they’re sure I’ll hate. I read one of these awhile ago—unfortunately I can’t remember who wrote it or where it appeared, but it raised a criticism, not specifically of me, I believe, but more generally of skeptics such as Uri Simonsohn and myself who keep bringing up p-hacking and the garden of forking paths.
The criticism that I read is wrong, I think, but it has a superficial appeal, so I thought it would be worth addressing it here.
The criticism went like this: People slam classical null-hypothesis-significance-testing (NHST) reasoning (the stuff I hate, all those “p less than .05” papers in Psychological Science, PPNAS, etc., on ESP, himmicanes, power pose, air rage, . . .) on two grounds: first, that NHST makes no sense, and second, that published p-values are wrong because of selection, p-hacking, forking paths, etc. But, the criticism continues, this anti-NHST attitude is itself self-contradictory: if you don’t like p-values anyway, why care that they’re being done wrong?
The author of this post (that I now can’t find) characterized anti-NHST critics (like me!) as being like the diner in that famous Borscht Belt routine, who complains about the food being so bad. And such small portions!
And, indeed, if we think NHST is such a bad idea, why do we then turn around and say that p-values are being computed wrong? Are we in the position of the atheist who goes into church in order to criticize the minister on his theology?
No, and here’s why. Suppose I come across some piece of published junk science, like the ovulation-and-clothing study. I can (and do) criticize it on a number of grounds. Suppose I lay off the p-values, saying that I wouldn’t compute a p-value here in any case so who cares. Then a defender of the study could easily stand up and say, Hey, who cares about these criticisms? The study has p less than .05, this will only happen 5% of the time if the null hypothesis is true, thus this is good evidence that the null hypothesis is false and there’s something going on! Or, a paper reports 9 different studies, each of which is statistically significant at the 5% level. Under the null hypothesis, the probability of this happening is (1/20)^9, thus the null hypothesis can be clearly rejected. In these cases, I think it’s very helpful to be able to go back and say, No, because of p-hacking and forking paths, the probability you’ll find something statistically significant in each of these experiments is quite high.
The reason I’m pointing out the problems with published p-values is not because I think researchers should be doing p-values “right,” whatever that means. I’m doing it in order to reduce the cognitive dissonance. So, when a paper such as power pose fails to replicate, my reaction is not, What a surprise: this statistically significant finding did not replicate!, but rather, No surprise: this noisy result looks different in a replication.
It’s similar to my attitude toward preregistration. It’s not that I think that studies should be preregistered; it’s that when a study is not preregistered, it’s hard to take p-values, and the reasoning that usually flows from them, at face value.
The post Why the garden-of-forking-paths criticism of p-values is not like a famous Borscht Belt comedy bit appeared first on Statistical Modeling, Causal Inference, and Social Science.
Please comment on the article here: Statistical Modeling, Causal Inference, and Social Science
