(This article was originally published at Statistical Modeling, Causal Inference, and Social Science, and syndicated at StatsBlogs.)
David Shor writes:
Suppose you’re conducting an experiment on the effectiveness of a pain medication, but in the post survey, measure a large number of indicators of well being (Sleep quality, self reported pain, ability to get tasks done, anxiety levels, etc).
After the experiment, the results are insignificant (or the posterior effect size isn’t large or whatever) on 9th of the 10 measures, but is significant on the tenth. You then declare the experiment a success and publish.
I think most people would find that scenario unseemly, but what is the “right” thing to do in that situation? Traditional techniques to avoid “p-hacking” (Regularization, multiple comparison corrections, etc) don’t really apply because the observations across measures aren’t independent of each other.
I can think of some approaches: Trying to regularize in a multi-task learning environment or maybe doing some sort of factor analysis and only doing inference on the first factor. But I’m curious if you’ve thought of the problem before.
The post My quick answer is that I would analyze all 10 outcomes using a multilevel model. appeared first on Statistical Modeling, Causal Inference, and Social Science.
Please comment on the article here: Statistical Modeling, Causal Inference, and Social Science
