So, back on 4/20 we linked to the post by Sam Harper and Adam Palayew shooting down a silly article, published in JAMA and publicized around the world, that claimed excess road deaths on 4/20 (“cannabis day”).
I googled the authors of that silly JAMA paper and found that one of them, Dr. Donald Redelmeier, was featured in a New York Times article from 2010, where he’s referred to as “perhaps the leading debunker of preconceived notions in the medical world.”
Kind of ironic that he’s called a “debunker,” considering that his work needed to be debunked.
Anyway, here’s how that NYT article begins:
Presidential elections can be fatal.
Win an Academy Award and you’re likely to live longer than had you been a runner-up.
Interview for medical school on a rainy day, and your chances of being selected could fall.
Such are some of the surprising findings of Dr. Donald A. Redelmeier, a physician-researcher and perhaps the leading debunker of preconceived notions in the medical world.
In his 20 years as a researcher, first at Stanford University, now at the University of Toronto, Dr. Redelmeier, 50, has applied scientific rigor to topics that in lesser hands might have been dismissed as quirky and iconoclastic. In doing so, his work has shattered myths and revealed some deep truths about the predictors of longevity, the organization of health care and the workings of the medical mind.
OK, let me be clear. Wacky ideas are fine. Some of the ideas might be wrong, and that’s ok. Science needs its careful researchers and also its eccentrics. I have no problem with that, nor do I have problems with journals publishing speculations.
Here are the three problems I have with the above news article:
1. “Scientific rigor.”
2. “Shattered myths.”
3. “Deep truths.”
Let me go through these things one at a time.
1. This work is not scientifically rigorous. Check out Harper and Palayew for more on this. Yes, it’s possible that Redelmeier’s work was rigorous in the past and now he’s switched to non-rigorous work, but I doubt it.
2. The news article does not mention any myths that were shattered. I have no idea what they’re talking about.
3. What are the “deep truths”? That medical school interviews are different on rainy days??
First of all, it’s an open question which of these claims are “truths” in the sense of being at all applicable to the wider world beyond what particular datasets this guy was studying. Second, the only way you can possibly characterize these claims, if true, as “deep,” is if they reveal some general insight about human nature. But in what sense does a grab-bag of data patterns represent depth?
“He’ll go totally against intuition, and come up with a beautiful finding,” said Eldar Shafir, a professor of psychology and public affairs at Princeton University who has worked with Dr. Redelmeier on research into medical decision-making.
I’m not sure how Redelmeier’s claims go “totally against intuition”—all of them have convenient stories attached—but, setting that aside, my point here is the problem with the traditional heroic mode of science reporting, the kind of thing we’re familiar with from Ted talks and the Edge foundation.
The scientist-as-hero narrative seems to me to be a disaster, and I think we need something new. Indeed, lots has changed since 2010, and I’d like to think that a modern version of this science reporting would show a bit more skepticism.
Why do I write about this?
The reason I write about this is not to pick on some doctor in Toronto who has a sideline in writing amusing noise-mining epidemiology papers. As I’ve said many times, I have no problem with journals publishing papers on 4/20 Day, or ESP, air rage, himmicanes, etc. (I have more problem with that paper on homicides and NRA conventions because it had political content so it wasn’t just a harmless feature story.) But there is a problem in that similar statistical fallacies arise in more serious medical research. So it’s interesting to break down this particular news story and see how it presses all the “scientist as hero” buttons. We can do better now, and that’s great.
And I can’t blame Redelmeier for the fact that he made statistics errors and got hyped in the newspaper. We all make statistics errors, and we don’t control what is written about us. I’am concerned that “scientist as hero” treatment can encourage a lack of self-questioning, and that’s too bad.
P.S. To her credit, Katie Hafner, author of that news article, does present a dissenting view:
Professor Tibshirani, for instance, has reservations about some of Dr. Redelmeier’s choices, and declined to collaborate on the Academy Awards study.
“I honestly thought it was frivolous, and we’ve argued about it,” Professor Tibshirani said. He also questioned the Election Day research. “Of course there’s more traffic, so it seemed self-evident,“ he said.
The article continues:
That perspective amuses rather than offends Dr. Redelmeier.
Dude could be a little less amused, a little more willing to realize he could be on the wrong track with a lot of his research.