(This article was originally published at Statistical Modeling, Causal Inference, and Social Science, and syndicated at StatsBlogs.)
Baruch Eitam writes:
My colleague and I have gotten into a slight dispute about prior selection. Below are our 3 different opinions, the first is the uniform (will get to that in a sec) and the other two are the priors of dispute. The parameter we are trying to estimate is people’s reporting ability under conditions in which a stimulus which they have just seen is “task irrelevant” — as they has 5 options to pick from, chance level is .2.
My preferred prior is the higher one as it reflects our more conservative estimation of the effect (higher modal theta -> reflects less errors), My colleague on the other hand opted for averaging all our previous experiments which ended up giving larger proportion of errors (i.e. a lower prior modal theta).
Initially, i thought that i should go with a more conservative estimation of the effect size as the purpose of the current study was to test candidate mechanism underlying this phenomenon (which we named “irrelevance based blindness”) by seeing whether it exists under conditions that, theoretically, preclude the operation of (some) mechanisms and by doing so constraining the number of possible ones. hope this makes some sense. Anyway, with this goal in mind i thought that also including what i thought might be extreme results we may be losing sensitivity here. why so? because our conclusion will be based on comparing the posterior CI and seeing whether we have evidence for ‘blindness’. on a practical level this ends up not making much difference because, surprisingly, participants ended up showing much stronger “blindness” then we have ever seen under similar conditions but still i think me and my colleagues’ debate touches upon interesting issues for experimentalists trying to use Bayesian estimation. for example — should such “sensitivity” consideration factor in when choosing a prior or is this merely a relic of hypothesis testing practice?
I don’t have much to say about this particular example, but in general one way to understand these choices is, for each model, to simulate replicated datasets from the prior predictive distribution, and see if these make sense. This can be viewed as a way of engaging with the prior information that is already in your head. Also remember that this prior predictive distribution represents different datasets that might be observed in different experiments studying different treatments, and you can also feel free to play with the sample sizes of these hypothetical replications.
The post Setting up a prior distribution in an experimental analysis appeared first on Statistical Modeling, Causal Inference, and Social Science.
Please comment on the article here: Statistical Modeling, Causal Inference, and Social Science
