Trying to make some sense of it all, but I can see it makes no sense at all . . . stuck in the middle with you

“Mediation analysis” is this thing where you have a treatment and an outcome and you’re trying to model how the treatment works: how much does it directly affect the outcome, and how much is the effect “mediated” through intermediate variables. Fabrizia Mealli was discussing this with me the other day, and she pointed out that the “direct effect” is defined only relative to a model: the direct effect of a treatment can be thought of as a residual effect, after considering all the other pathways being considered. This is not the same as simply fitting a multiple regression on the outcome and looking at the coefficient of the treatment, controlling for all the intermediate variables: that won’t work at all, but there are these methods such as path analysis or mediation analysis that can fit these models, under some assumptions.

In the real world, it’s my impression that almost all the mediation analyses that people actually fit in the social and medical sciences are misguided: lots of examples where the assumptions aren’t clear and where, in any case, coefficient estimates are hopelessly noisy and where confused people will over-interpret statistical significance (see here, for example).

So it’s natural to take what would seem to be a conservative position and forget about mediation analysis, taking the fallback position of intent-to-treat analysis or, more generally, just trying to estimate treatment effects without untangling causal pathways. And indeed that’s pretty much what I’ve done, as you can see in the causal inference chapters in our books. In specific applications I’ve worked with particular causal mechanisms, but I’ve not tried to use general techniques for mediation analysis.

But . . . more and more I’ve been coming to the conclusion that the standard causal inference paradigm is broken. I’m talking about the paradigm under which a researcher dreams up an idea for a treatment and then designs a study, collects data, and estimates “the treatment effect.” It ain’t working: nowadays, treatment effects are small and variable, not large and stable (the low-hanging fruit have already been plucked). Our little experiments don’t have enough data to allow us to estimate real-world treatment effects and their variation; at the same time, we’re not efficiently using data to come up with our treatments. On both grounds, I think the way forward has to involve intermediate outcomes and modeling/estimation of causal paths.

So how to do it? I don’t think traditional path analysis or other multivariate methods of the throw-all-the-data-in-the-blender-and-let-God-sort-em-out variety will do the job. Instead we need some structure and some prior information.

A good starting point might be the literature on causal inference for time series (longitudinal data, as they call it in biostatistics), as such models have built-in structure. Also examples such as compliance where there are some natural constraints that can be assumed on the casual relationships.

Fabrizia Mealli pointed me to some recent papers:

Bayesian inference for causal mechanisms with application to a randomized study for postoperative pain control

Identification and Estimation of Causal Mechanisms in Clustered Encouragement Designs: Disentangling Bed Nets Using Bayesian Principal Stratification

A Bayesian Semiparametric Approach to Intermediate Variables in Causal Inference

Augmented designs to assess principal strata direct effects

The post Trying to make some sense of it all, but I can see it makes no sense at all . . . stuck in the middle with you appeared first on Statistical Modeling, Causal Inference, and Social Science.