(This article was originally published at Statistical Modeling, Causal Inference, and Social Science, and syndicated at StatsBlogs.)
Shira Mitchell and Mariel Finucane, two statisticians at Mathematica Policy Research (that’s the policy-analysis organization, not the Wolfram software company) write:
We here at Mathematica have questions about priors for a health policy evaluation. Here’s the setting:
In our dataset, healthcare (per person per month) expenditures are highly variable (sd = $2500), but from prior studies, we don’t expect the policy being studied to have average impact of much greater than $20. The sample sizes are large enough to have hope to estimate small effects (medical claims records for millions of beneficiaries), but prior knowledge is still useful.
As described here, we consider these levels of priors for the average impact:
1. (essentially) flat prior
2. generic weakly-informative prior: normal(0,1) scaled by sd of the data, so normal with mean 0 and sd $2500 [that’s normal(0, $2500) using the mean, sd parameterization of the distribution]
3. specific informative prior: “One principle: write down what you think the prior should be, then spread it out.”
3a. If we think about a large collection of prior studies, we’d believe normal(0, $20), so we could use something like Normal(0, $50) as a weakly informative prior
3b. Or a specific prior study that was related: we could center the prior at the point estimate from that evaluation, with a standard deviation twice the SE of that estimate?Questions:
Thoughts on #2 vs #3 (a vs b)? For #2, which sd to use? Raw sd of expenditures, or a residual sd? Expenditures are highly non-Normal, does this affect the usefulness of the sd as a scale?
My reply:
First, I’m concerned about the challenges of estimating an average treatment effect of 20 or so, in the context of variation that is 100 times as large. I’m not saying it’s impossible—you just need a sample size in the zillions, of the sort that you can get from administrative records—just that your inference will be highly sensitive to small biases.
[Shira and Mariel elaborate: Say sd = 1 and we want to estimate a true effect = 1 with n = 100, so SE = 0.1. If bias is 0.01, this is only 1% of the true effect, not a big deal. Now say we want to estimate a true effect = 0.01 with n = 1000000, so SE = 0.001. If bias is again 0.01, this is 100% of the true effect, which is a big deal. Larger sample sizes won’t reduce bias, and we expect bias proportional to the sd of the raw data, and not the true effect.]
A randomized trial gets rid of one sort of bias, but there’s still dropout, nonresponse, and plain old variation in the treatment effect. So it would be best if you could somehow decompose the outcome and pull out the part that’s responsive to the treatment and then focus on that part, on the hope that the sd will be a lot less than 2500. [Shira and Mariel say: for example, if we imagine the program decreases hospital expenditures specifically (instead of total expenditures)? or decreases expenditures primarily for high-risk patients?]
Second, to get to the problem at hand, I have no problem with something like a conservative normal(0,20^2) prior or a weakly informative normal(0,200^2) prior on the treatment effect. To me, the “conservative” choice is not the flat prior but rather the prior that pulls strongly toward 0; for more on this point, see this paper with Aleks Jakulin.
Shira and Mariel follow up:
We share your notion of conservative. How should this be balanced with prior information we have from a previous (very related) study? Do we center at 0 or the previous study’s estimate? Somewhere in-between? If centered at a non-zero value, then a strong prior is no longer your version of conservative, correct?
My response:
In this context, yes, I feel that centering the prior at a positive value rather than 0 would not be conservative. But I suppose it depends on the context. For example, suppose you label the treatment effect in a previous study as theta_1 and the treatment effect in the current study as theta_2 (these represent the underlying parameters, not the point estimates), and you think of theta_1, theta_2, etc., as draws from some distribution of treatment effects with center mu and scale sigma. Then you could have a conservative prior on mu with mean 0. This will be different from a conservative prior on theta_2 with mean 0. I guess the lesson is that a conservative prior is defined with respect to the problem being studied and the data being modeled.
Shira and Mariel point out that in the sort of hierarchical model I’m suggesting, you’ll want an informative prior on the variation in the treatment effect. That’s right. In statistical practice we’re not used to setting up these priors, but I think full Bayes is the way to go here, and that it’s not so horrible to have to justify your particular choice of prior distribution as part of the analysis, in the same way that you’re already expected to justify your choice of predictor variables, transformations, and so forth.
The post Bayes for estimating a small effect in the context of large variation appeared first on Statistical Modeling, Causal Inference, and Social Science.
Please comment on the article here: Statistical Modeling, Causal Inference, and Social Science
