(This article was originally published at Statistical Modeling, Causal Inference, and Social Science, and syndicated at StatsBlogs.)
In our discussion of the Bargh, Chen, and Burrows priming-with-elderly-related-words-makes-people-walk-slowly-paper (the study which famously failed in a preregistered replication), commenter Lois wrote:
Curious as to what people think of this comment on the Bargh et al. (1996) paper from Pubpeer: (see below).
In Experiment 3, the experimenter rated participants on irritability, hostility, anger, and uncooperativeness on 10-point scales. Independently, two blind coders rated the same participants’ facial expressions (from video-tapes) on a 5-point scale of hostility.
Figure 3 in the paper displays the mean hostility ratings from the experimenter and the blind coders for participants in the two conditions (Caucasian prime and African-American prime). The means from the experimenter and from the blind coders appear almost identical (they are indistinguishable to my eye).
How likely is it that the experimenter and the blind coders provided identical ratings, using different scales, in both conditions, for ratings of something as subjective as expressed hostility?
Are the bars shown in Figure 3 an error?
Did Bargh et al. cheat? Did their research assistant cheat? Did they make up data? Did they make a Excel error? Did they do everything correctly and we have nothing here but a suspicious PubPeer commenter?
I don’t know. But from a scientific perspective, it doesn’t really matter.
In our discussion of the Carney, Cuddy, and Yap paper on power pose (another one with a failed replication), commenters here and here noticed some irregularities in this and an earlier paper of Cuddy: test statistics were miscalculated, often in ways that moved a result from p greater than .05 to p less than .05.
Did Cuddy et al. cheat? Did they enter the wrong values in their calculator? Were the calculations performed by an unscrupulous research assistant who was paid by the star?
I don’t know. But from a scientific perspective, it doesn’t really matter.
Why doesn’t it matter? Because these studies are, from a quantitative perspective, nothing but the manipulation of noise. Any effects are too variable to be detected by these experiments. Indeed, this variability is implicitly recognized in the literature, where every new study reports the discovery of a new interaction.
So, whether these p-values less than .05 come from cheating, sloppiness, the garden of forking paths, or some combination of the above (for example, opportunistic rounding might fall somewhere in the borderline between cheating and sloppiness, and Daryl Bem’s thicket of forking paths is so tangled that in my opinion it counts as sloppiness at the very least), it just doesn’t matter. These quantitative results are useless anyway.
Enter Lacour
In contrast, consider the fraud conducted by Michael Lacour in his paper with Don Green on voter persuasion. This one, if his numbers had been true, really would’ve represented strong evidence in favor of his hypothesis. He had a clean study, a large sample, and a large and persistent effect. Indeed, when I first saw that paper, my first thought was forking paths but upon a quick read it was clear that there was something real there (conditional on the data being true, of course). No way these results could’ve been manufactured simply by excluding some awkward data points or playing around with predictors in the regression.
So in the Lacour case, the revelations of fraud were necessary. The forensics were a key part of the story. For Cuddy, etc., sure, maybe they engaged in unethical research practices, or maybe they were just sloppy, but it’s not really worth our while to try to find out. Cos even if they did everything they did with pure heart, their numbers are not offering real support for their claims.
One might say that Lacour was too clever by half. He could’ve got that paper in Science without any faking of data at all! Just do the survey, gather the data, then use the garden of forking paths to get a few statistically significant p-values. Here are some of the steps he could’ve taken:
1. If he gathers real data, lots of the data won’t fit his story: there will be people who got the treatment who remained hostile to gay rights and people who got the control who still end up supporting gay marriage. Start out by cleaning your data, removing as many of these aberrant cases as you can get away with. It’s not so hard: just go through, case by case, and see what you can find to disqualify these people. Maybe some of their responses to other survey questions were inconsistent, maybe the interview was conducted outside the official window for the study (which you can of course alter at will, and it looks so innocuous in the paper: “We conducted X interviews between May 1 and June 30, 2011” or whatever). If the aggregate results for any canvasser look particularly bad, you could figure out a way to discard that whole batch of data—for example, maybe this was the one canvasser who was over 40 years old, or the one canvasser who did not live in the L.A. area, or whatever. If there’s a particular precinct where the results did not turn out as they should, find a way to exclude it: perhaps the vote in the previous election there was more than 2 standard deviations away from the mean. Whatever.
2. Now comes the analysis stage. Subsets, comparisons, interactions. Run regressions including whatever you want. If you find no main effect, do a path analysis—that was enough to convince a renowned political scientist that a subliminally flashing smiley face could cause a large change in people’s attitudes toward immigration. The smiley-face conclusion was a stretch (to say the least); by comparison, Lacour and Green’s claims, while large, were somewhat plausible.
3. The writeup. Obviously you report any statistically significant thing you happen to find. But you can do more. You can report any non-significant comparisons as evidence against alternative explanations of your data. And if you’re really ballsy you can report p-values such as .08 as evidence for weak effects. What you’re doing is spinning a story using all these p-values as raw material. This is important: don’t think of statistical significance as the culmination of your study: feel confidence that using all your researcher degrees of freedom, you can get as many statistically significant p-values as you want, and look ahead to the next step of telling your tale. You already have statistical significance; no question about that. To get the win—the publication in Nature, Science, or PPNAS—you need that pow! that connects your empirical findings to your theory.
4. Finally, once you’ve created your statistical significance, you can use that as backing to make as many graphs as you want. So Lacour still gets to use that skill of his.
Isn’t it funny? He could’ve got all the success he wanted—NPR coverage, Ted Talk, tenure-track job, a series of papers in top journals—without faking the data at all!
For decades the statistical community completely missed the point. We obsessed about stopping rules and publication bias, pretty much ignoring the real action is in data exclusion, subsetting, and degrees of freedom in analysis.
P.S. I wrote this post months ago; it just happened to come up today, in the midst of all the discussion of the replication crisis.
The post Michael Lacour vs John Bargh and Amy Cuddy appeared first on Statistical Modeling, Causal Inference, and Social Science.
Please comment on the article here: Statistical Modeling, Causal Inference, and Social Science
