Recently we discussed law professor and policy intellectual Cass Sunstein’s statement that people who ask for social science findings to be replicated are like the former East German secret police.
In that discussion I alluded to a few issues:
1. The replication movement is fueled in large part by high-profile work, lauded by Sunstein and other Ivy League luminaries, that did not replicate.
2. Until outsiders loudly criticized the unreplicated work, those unreplicated claims were essentially uncriticized in the popular and academic press. And the criticism had to be loud, Loud, LOUD. Recall the Javert paradox.
3. That work wasn’t just Gladwell and NPR-bait, it also had real-world implications.
For example, check this out from the Nudge blog, several years ago:
As noted above, Sunstein had no affiliation with that blog. My point is that his brand was, unwittingly, promoting bad research.
And this brings me to my main point for today. Sunstein likens research critics to the former East German secret police, echoing something that a psychology professor wrote a few years ago regarding “methodological terrorists.” But . . . without these hateful people who are some cross between the Stasi and Al Qaeda, those destructive little second-stringers etc. . . . without them, Sunstein would I assume still be promoting claims based on garbage research. (And, yes, sure, Wansink’s claims could still be true, research flaws notwithstanding: It’s possible that the guy just had a great intuition about behavior and was right every time—but then it’s still a mistake to present those intuitions as being evidence-based.)
For example, see this recent post:
The link states that “A field study and a laboratory study with American participants found that calorie counts to the left (vs. right) decreased calories ordered by 16.31%.” 16.31%, huh? OK, I’ll believe it when it’s replicated for real, not before. The point is that, without the research critics—including aggressive research critics, the Javerts who annoy Sunstein and his friends so much—junk science would expand until it entirely filled up the world of policy analysis. Gresham, baby, Gresham.
So, again, Sunstein should be thanking, not smearing, people who ask for replications.
The bearer of bad tidings is your friend, not your enemy.
P.S. Probably not a good idea to believe anything Brian Wansink has ever written, at least not until you see clearly documented replication. This overview by Elizabeth Nolan Brown gives some background on the problems with Wansink’s work, along with discussions of some political concerns:
For the better half of a decade, American public schools have been part of a grand experiment in “choice architecture” dressed up as simple, practical steps to spur healthy eating. But new research reveals the “Smarter Lunchrooms” program is based largely on junk science.
Smarter Lunchrooms, launched in 2010 with funding from the U.S. Department of Agriculture (USDA) . . . is full of “common sense,” TED-talk-ready, Malcolm Gladwell-esque insights into how school cafeterias can encourage students to select and eat more nutritious foods. . . . This “light touch” is the foundation upon which Wansink, a former executive director of the USDA’s Center for Nutrition Policy and Promotion and a drafter of U.S. Dietary Guidelines, has earned ample speaking and consulting gigs and media coverage. . . .
The first serious study testing the program’s effectiveness was published just this year. At the end of nine weeks, students in Smarter Lunchroom cafeterias consumed an average of 0.10 more fruit units per day—the equivalent of about one or two bites of an apple. Wansink and company called it a “significant” increase in fruit consumption.
But “whether this increase is meaningful and has real world benefit is questionable,” Robinson* writes.
Nonetheless, the USDA claims that the “strategies that the Smarter Lunchrooms Movement endorses have been studied and proven effective in a variety of schools across the nation.” More than 29,000 U.S. public schools now employ Smarter Lunchrooms strategies, and the number of school food service directors trained on these tactics increased threefold in 2015 over the year before.
Also this:
One study touted by the USDA even notes that since food service directors who belong to professional membership associations were more likely to know about the Smarter Lunchrooms program, policy makers and school districts “consider allocating funds to encourage [directors] to engage more fully in professional association meetings and activities.”
But now that Wansink’s work has been discredited, the government will back off and stop wasting all this time and money, right?
Ummm . . .
A spokesman for the USDA told The Washington Post that while they had some concerns about the research coming out of Cornell, “it’s important to remember that Smarter Lunchrooms strategies are based upon widely researched principles of behavioral economics, as well as a strong body of practice that supports their ongoing use.”
Brown summarizes:
We might disagree on whether federal authorities should micromanage lunchroom menus or if local school districts should have more control, and what dietary principles they should follow; whether the emphasis of school cafeterias should be fundraising or nutrition; or whether school meals need more funding. But confronting these challenges head-on is a hell of a lot better than a tepid consensus for feel-good fairytales about banana placement.
Or celebrating the “coolest behavioral finding of 2019.”