Fair Warning

March 13, 2017

(This article was originally published at Statistical Modeling, Causal Inference, and Social Science, and syndicated at StatsBlogs.)

A few months ago we shared Rolf Zwaan’s satirical advice on how to conduct a research project in social psychology, write it up, and publicize it, under the principle of minimal effort in the research, maximum claims in the writeup, and maximal publicity in the aftermath. I called it, “From zero to Ted talk in 18 simple steps.”

I thought this was all a joke, but at least one psychology research team seems to have taken it more seriously, following Zwaan’s 18 steps almost to the letter in this paper, “Caught Red-Minded: Evidence-Induced Denial of Mental Transgressions.” I’ll leave it to the reader to count researcher degrees of freedom in this paper.

At this point, you might say: Sure, fine, so the work is speculative. Even so, interesting speculation is important. A new speculative idea is a lot more valuable than a carping blog post.

And I’d agree with you. But I’d add three things:

1. It took me 15 minutes to write this post, and it took the authors of this paper and the participants of this experiment . . . ummm, I dunno, 1000 person-hours in total? to produce that paper. I can accept that their paper is more valuable than this post, but I don’t think it’s 4000 times more valuable.

2. Here are the first and last sentences of the abstract of that paper: “We suggest that when confronted with evidence of their socially inappropriate thoughts and feelings, people are sometimes less likely—and not more likely—to acknowledge them because evidence can elicit psychological responses that inhibit candid self-reflection. . . . These results suggest that under some circumstances, confronting people with public evidence of their private shortcomings can be counterproductive.” Is this actually interesting? I don’t know but let’s just say it’s not a slam-dunk case.

3. To the extent that the idea in that paper is interesting—and I’m willing to give the authors the benefit of the doubt on this one—it’d be just as legitimately interesting if presented as theory + speculation + qualitative observation, without the random numbers that are the quantitative results, the meaningless p-values and all the rest. As I often say about this sort of work: if you think it’s important, go for it! Publish it! Promote it! Just recognize it for the qualitative work that it is, and don’t fool yourself with numbers, or you might end up making claims that don’t replicate, and making lots of other people look like fools too.

Remember, despite what some people might say, it’s not actually the case that “the replication rate in psychology is quite high—indeed, it is statistically indistinguishable from 100%.”

P.S. Hey, these Westlake titles really work!

The post Fair Warning appeared first on Statistical Modeling, Causal Inference, and Social Science.

Please comment on the article here: Statistical Modeling, Causal Inference, and Social Science