“and, indeed, that my study is consistent with X having a negative effect on Y.”

March 9, 2018
By

(This article was originally published at Statistical Modeling, Causal Inference, and Social Science, and syndicated at StatsBlogs.)

David Allison shares this article:

Pediatrics: letter to the editor – Metformin for Obesity in Prepubertal and Pubertal Children A Randomized Controlled Trial

and the authors’ reply:

RE: Clarification of statistical interpretation in metformin trial paper

The authors of the original paper were polite in their response, but they didn’t seem to get the point of the criticism they were purportedly responding to.

Let’s step back a moment

Forget about the details of this paper, Allison’s criticism, and the authors’ reply.

Instead let’s ask a more basic question: How does one respond to scientific criticism?

It’s my impression that, something like 99% of the time, authors response to criticism is predicated on the assumption that they were completely correct all along: the idea is that criticism is something to be managed. Tactical issues arise—Should the authors sidestep the criticism or face it head on? Should they be angry, hurt, dismissive, deferential, or equanimous?—but the starting point is the expectation of zero changes in the original claims.

That’s a problem. We all make mistakes. The way we move forward is by learning from our mistakes. Not from denying them.

Here was my response to Allison: you think that’s bad; check out this journal-editor horror story. These people are actively lying.

Admitting and learning from our errors

Allison responded:

We (meaning the scientific community in its broadest form) definitely have a long way to go in learning how to adhere scrupulously to truthfulness, to give and respond to criticism constructively and civilly, and how to admit mistakes and correct them.

I like this line from Eric Church: “And when you’re wrong, you should just say so; I learned that from a three year old.”

I wish more people would be willing to say:

You’re right. I made a mistake. My study does not show that X causes Y. I may still believe that X causes Y, but I acknowledge that my study does not show it.

We do occasionally get folks to write that in response to our comments, but it is all too rare.

Anyway, right now I have been looking at papers that make unjustified causal inferences because of neglecting (or not realizing) the phenomenon of regression to the mean. Regression to the mean really seems to confuse people.

And I replied: You write:

I wish more people would be willing to say:

You’re right. I made a mistake. My study does not show that X causes Y. I may still believe that X causes Y, but I acknowledge that my study does not show it.

I’d continue with, “and, indeed, that my study is consistent with X having a negative effect on Y. Or, more generally, having an effect that varies by context and is sometimes positive and sometimes negative.

Also, I think that the causal discussion can mislead, in that almost all these issues arise with purely correlational studies. For example, the silly claim that beautiful parents are more likely to have daughters. Forget about causality; the real point is that there’s no evidence supporting the idea that there is such a correlation in the population. There’s a tendency of people to jump from the “stylized fact” to the purported causal explanation, without recognizing that there’s no good evidence for the stylized fact.

The post “and, indeed, that my study is consistent with X having a negative effect on Y.” appeared first on Statistical Modeling, Causal Inference, and Social Science.



Please comment on the article here: Statistical Modeling, Causal Inference, and Social Science

Tags: , ,


Subscribe

Email:

  Subscribe