Awhile ago we came up with the time-reversal heuristic, which was a reaction to the common situation that there’s a noisy study, followed by an unsuccessful replication, but all sorts of people want to take the original claim as the baseline and construct high walls to make it difficult to move away from that claim. The time-reversal heuristic is to imagine the two studies in reverse order: First a large and careful study that finds nothing of interest, then a small noisy replication whose authors fish around in the data and find an unexpected statistically significant result. The idea is to remove the “research incumbency effect” and to consider each study on its own merits.
Recently we discussed something similar, a status-reversal heuristic:
Sometimes I think the world would be a better place if, every time an economist or a journalist saw a published claim by an economist, they were to be told that the research had been performed by a sociologist, or an anthropologist. This would induce in them an appropriate level of skepticism.
Similarly with medical research: Suppose that every time a doctor or a journalist saw a published claim by a M.D., they were told that the research had been done by a nurse, or a social worker. Again, then maybe they’d be appropriately skeptical.
Oooh, I like this game. Here’s another one: Every time you see a paper by a Harvard professor, mentally change the affiliation to State U. Then you’ll be appropriately skeptical.
And, every time you see a paper endorsed by a member of the National Academy of Sciences . . . ummm, I guess that one’s ok, we already know not to believe it!
Arguments against the status-reversal heuristic, and responses to those arguments
I can think of two arguments against. (Maybe you can think of more—that’s what the comments section is for, to explain to me how wrong I am!)
1. It’s kind of weird to see me advocating this status-reversal heuristic, as I have all sorts of status: the Ph.D., the Ivy League faculty position, the access to national media, etc. I’ve even published in the Proceedings of the National Academy of Sciences!
In response, all I can say is: Sure, the status-reversal argument can apply to me too. Feel free to evaluate this post, and other things I write, as if I had no pre-existing status. I think this post, and what we have on this blog more generally, holds up fine under the status-reversal argument.
2. The status-reversal heuristic is anti-Bayesian. Status provides relevant prior information. Sure, some Ivy League professors are blowhards at best and frauds at worst, but lots of us have done good work—indeed, that’s what can get us to the Ivies in the first place. Similarly, if economists have more default credibility than sociologists, maybe there’s a reason for that.
I have two responses here. First, sometimes I don’t think status adds any information at all. For example, if the topic is medical policy, I don’t see why we should expect an M.D. to have a more informed opinion than a nurse or a social worker. Indeed, the M.D. could well be less informed, if he or she has been trained to ignore the opinions of non-M.D.’s. Second, even in areas where status is correlated with expertise, I’m guessing that people have overweighted the prior information associated with that status. So removing that weighting can be a step forward. Third, status can be abused. Here I’m thinking of journals such as New England Journal of Medicine and Lancet that will publish bad research that pushes a political agenda, or PNAS, which publishes, bad research that pushes a particularly set of scientific theories.
My main argument here is the second one I just gave: Even in areas where status is correlated with expertise, I’m guessing that people have overweighted the prior information associated with that status. So removing that weighting can be a step forward. Perhaps this could be studied empirically in some way. I’m not quite sure how, but maybe there’s a way using prediction markets? We could ask Anna Dreber. She doesn’t teach at Harvard, but she’s an economist and she’s coauthored with some famous people, so there’s that.