Someone pointed me with suspicion to a newspaper article that reported a cool-looking social science result, and asked me for my thoughts.
I replied, Yes, not only am I suspicious of the claims in that article, I’m also suspicious of all the individual claims from these links. And I pointed to a bunch of links in the news article.
I continued by saying that I’m started to get exhausted fighting these things. It takes some work to figure out the flaws in all these papers. I’m starting to see the appeal of preregistered replications, just in reducing the number of spurious findings that we have to waste time talking about.
My correspondent replied:
Yeah, it was those sentences that triggered my “alert response”. Then the rest of the article looked bogus too.
I agree, it’s like “whack-a-mole”.
The frustrating thing is that correspondent and I don’t feel like spending the time going through each of these papers and figuring out exactly what made me suspicious, and figuring out alternative explanations for the published findings. These particular studies are on a topic I’ve never written about (at least, not that I remember).
The news article in question is here, and I was also suspicious of all the individual claims from the links in these two sentences:
For example, in France and the United States, a study showed that judges give shorter sentences when defendants show up to court on their birthdays. Judges are also less likely to grant asylum to a refugee if they have done so for the previous asylum applicant — or when it’s hot outside, or their alma mater has unexpectedly lost a football game.
Again, I don’t have the energy to read these papers carefully enough to figure out what went wrong—or, for that matter, to be convinced that the claims in question are actually backed by solid research—so let me be clear that I’m not here offering any specific arguments as to the flaws of these research claims; I’m just skeptical for the usual reasons that it’s all too easy to look at data and find dramatic patterns that don’t replicate. From a statistical standpoint, the general concerns are confounding (these are not randomly assigned treatments) and forking paths (lots of ways to find patterns).
But, again, don’t take this as a criticism of these particular claims; it’s just a general warning.