Maybe this paper is a parody, maybe it’s a semibluff

September 18, 2017

(This article was originally published at Statistical Modeling, Causal Inference, and Social Science, and syndicated at StatsBlogs.)

Peter DeScioli writes:

I was wondering if you saw this paper about people reading Harry Potter and then disliking Trump, attached. It seems to fit the shark attack genre.

In this case, the issue seems to be judging causation from multiple regression with observational data, assuming that control variables are enough to narrow down to causality (or that it’s up to a critic to find the confounds). It speaks to a bigger issue about how researchers interpret multiple regression in causal terms.

Any thoughts on this, or obvious/good references critiquing causal interpretations of multiple regression? (like to assign to my PhD students)

My reply: Hi, yes, I saw this paper months ago. I suspected it was a parody but someone told me that it was actually supposed to be serious. I still think it’s a kind of half-parody, it’s what social scientists might call a “fun” result, and it’s published in a non-serious journal, so I doubt the author takes it completely seriously. Kinda like this: you find an interesting pattern in data, it’s probably no big deal, but who knows, so get it out there and people can make of it what they will.

Twenty years ago, social scientists could do this and it would be no problem; nowadays with all this stuff on shark attacks, college football, power pose, contagion of obesity, etc., it seems that people have more difficult putting such speculations into perspective: any damn data pattern they see, they want to insist it’s a big deal, from data analysis to publication to Ted talk and NPR. In some sense this Harry Potter paper is a throwback and it would probably be best to interpret it the way we’d have taken it twenty or thirty years ago.

It’s impossible for me to tell whether the author, Diana Mutz, is writing this paper as a parody. Intonation is notoriously difficult to convey in typed speech. It’s a funny thing: if the paper’s not a parody and I say it is, then I’m kinda being insulting. But if the paper is a parody and I take it seriously, then I’m not getting the joke. So there’s no safe interpretation here! (I could ask Mutz directly but that’s not much of a general solution; I’d rather think of a published article and its implications as standing on their own and not requiring typically unobtainable “meta-data” on authorial intentions.)

DeScioli responded:

It does have some whimsical passages so maybe it is half-parody.

And I continued:

Yeah, there’s this genre of research which is not entirely serious but not entirely a joke, kinda what in poker we’d call a semibluff. Back in the good old days before Gladwell, PPNAS, NPR hype, etc., it was reasonable for researchers to try out some of these ideas, they were long shots but had some appeal as part of the mix of science. For awhile, though, it was seeming like this sort of open-ended-speculation-backed-by-statistically-significant-p-values had become most of social science, and this has reduced all of our patience for this sort of thing. Which is kinda too bad. Another example is that observation that several recent presidents were left-handed. It seems like it should be possible to point to such data patterns, and even run some statistics on them, without making large claims.

DeScioli followed up:

Seems it could still be as fun and interesting to look for these types of correlations without claiming causality. I was just surprised to see the paper double down on the causal interpretation with the argument that the analysis controlled for everything they could think of. (My assumption is that observational data has countless confounding correlations that no one could think of.) I don’t think this paper is worse in over-interpreting than many others I’ve seen. It’s just easier to notice because of the whimsical topic.

What’s the lesson for avoiding this for a more serious-sounding theory? I typically restrict causal judgments to experimental manipulations. But maybe that is too restrictive? The only other thing I can think of is if a researcher knew so much about their subject that they could boil down the possible causes to a handful. Then maybe multiple regression with controls could help sort between them. If so, the issue with Harry Potter is that it’s one of millions of similar cultural influences that are all hopelessly tangled and so can’t be untangled with observational methods.

The post Maybe this paper is a parody, maybe it’s a semibluff appeared first on Statistical Modeling, Causal Inference, and Social Science.

Please comment on the article here: Statistical Modeling, Causal Inference, and Social Science

Tags: , ,