(This article was originally published at Statistical Modeling, Causal Inference, and Social Science, and syndicated at StatsBlogs.)
Tim van der Zee, Jordan Anaya, and Nicholas Brown posted this very detailed criticism of four papers published by food researcher and business school professor Brian Wansink. The papers are all in obscure journals and became notorious only after Wansink blogged about them in the context of some advice he was giving to graduate students. Last month someone pointed out Wansink’s blog, and I took a look at the papers and criticized them here. The short story is that all four papers come from a single experiment that Wansink himself had characterized as “flawed.” And, although the four papers were all based on the same data, they differed in all sorts of detail, which suggested that the authors opportunistically used data exclusion, data coding, and data analysis choices to obtain publishable (that is, p less than .05) results.
I also disagreed with Wansink’s claim that “doing the analyses for a paper” was more valuable than “Facebook, Twitter, Game of Thrones, Starbucks, spinning class.” I think that watching a season’s worth of episodes of Game of Thrones is more valuable than writing a paper such as “Eating Heavily: Men Eat More in the Company of Women.” Views on this may differ, however.
Zee, Anaya, and Brown did a more thorough job than I did. They went through all four papers in excruciating detail, and they find literally dozens of errors. Some of these errors are small. For example, the following table has inconsistencies marked in red:
I thought I’d check:
> (43.67 - 44.55)/sqrt(18.5^2/43 + 14.3^2/52) [1] -0.2551872 > (68.65 - 66.51)/sqrt(3.67^2/43 + 9.44^2/52) [1] 1.503114 > (184.83 - 178.38)/sqrt(63.7^2/43 + 45.71^2/52) [1] 0.556064
OK, the first one isn’t so bad. The t statistic is 0.26 (to two decimal places) but they mark it as 0.25. The next two aren’t so far off either—1.38 and 0.52 instead of 1.50 and 0.56—still, it’s kind of impressive , in some sort of perverse way, that they got something wrong in every line of their table. The next two tables, too, have errors in every line. How can you manage that? Really this is kind of amazing. This Wansink guy is giving Richard Tol a run for a his money.
I guess Tol is worse because he has a political agenda, whereas I assume that Wansink just wants to churn out publications.
But wait . . . there’s more!
And there are other inconsistencies even beyond what’s in that PeerJ article. See here:
1. Wansink flat-out admits to selection bias. He writes that “Plan A [of his data analysis] had failed” and then he shares four papers from this study—these must be the results of Plans B, C, D, and E, or something like that—but we never hear about failed Plan A. This is never published, and in none of his four papers (or in his subsequent blog) does he ever say what Plan A was. I don’t think I’ve ever heard of such a blatant and acknowledged example of selection bias. I can only assume that Wansink doesn’t realize how serious this is, or else he wouldn’t’ve so blithely mentioned it. Fake it till you make it, indeed.
2. There’s a fundamental inconsistency between two of Wansink’s statements: (a) He describes his experiment as a “failed study which had null results” and (b) he seems to think the data are so great that they’re worth 4 papers. If he can get 4 papers out of a failed study, it kinda makes you wonder what he can get out of a successful study. 8 papers, maybe? 16?
3. See the P.P.P.S. at the above-linked blog post. A note to one of the papers contains this, in the note on Authors’ contributions: “OS collected and analyzed data, and helped draft the manuscript.” But in his post, he clearly states that the study had been designed and the data collected before Ozge Sigirci (“OS”) arrived. So this is a flat-out contradiction.
Haters gonna hate?
Finally, I would not be surprised if Wansink’s response to all this is to shrug it off: Don’t feed the trolls, Haters gonna hate, Who cares about picky people, etc. (He seems too mellow to go the “replication terrorists” route.) So I’d pre-empt this by saying: Nobody forced Wansink to write and publish a series of false statements. He could’ve written the truth at any point. It is kind of exhausting to sift through published work and find error after error, but I’d say the responsibility for these errors falls on the authors.
The post “Statistical heartburn: An attempt to digest four pizza publications from the Cornell Food and Brand Lab” appeared first on Statistical Modeling, Causal Inference, and Social Science.
Please comment on the article here: Statistical Modeling, Causal Inference, and Social Science
