(This article was originally published at Statistical Modeling, Causal Inference, and Social Science, and syndicated at StatsBlogs.)
Juan Carlos Lopez writes:
Here’s a news article: . . .
Here’s the paper: . . .
[Details removed to avoid embarrassing the authors of the article in question.]
I [Lopez] am especially bothered by the abstract of this paper, which makes bold claims in the context of a small and noisy study which measurements are not closely tied to the underlying constructs of interest—at best, they are quantifying a very context-dependent, special case.
Anyhow, I think you can get the gist of the article (and its problems) by only reading the abstract, Table 1, and Figure 1.
My reply:
Yes, there’s no need to take the paper seriously: it’s an exercise in noise mining, and if anyone would ever go to the trouble of replicating it—which I doubt will ever happen—I expect they’d see some other set of interactions pop up as statistically significant. In the news article, one of the authors describes the results in the paper as “surprising”—without realizing that it’s no surprise at all that if you shuffle around a bunch of random numbers, out will pop some random statistically significant comparisons.
The whole thing is a disaster, from data collection to analysis to writeup to publication to publicity—for the general reasons discussed here, and I think I’d be doing the authors a favor, at some level, to tell them that—but for the usual reasons of avoiding conflict I won’t bother doing this. It really makes me sad, not angry. This particular paper that you sent me is not on a particularly important or exciting topic (it’s just quirky enough to get into the news), it’s just routine cargo-cult science that we see every day. For lots of people, it’s their career and they just don’t know better.
Lopez followed up with another question:
In the setting of, say, a research seminar presentation, how do you answer the question “Why are you not including p-values in your Results section”?
Some context for my question: I’m a Ph.D. candidate at a university where most people are still using p-values in the usual ways which you criticize in McShane et al. (2017). I have trouble answering the question above in a way that doesn’t derail the entire discussion. Recently, I’ve discovered that the most effective way to avoid a long—and sometimes counterproductive—discussion on the topic is to appeal to authority by saying I’m following the ASA guidelines. This has become my go-to, 30-second answer.
My response: I don’t object to people including p-values—they do tell you something! My objection is when p-values are used to select a subset of results. I say: give all the results, not just a subset.
The post The cargo cult continues appeared first on Statistical Modeling, Causal Inference, and Social Science.
Please comment on the article here: Statistical Modeling, Causal Inference, and Social Science
