Dirk Nachbar writes:
We were having a debate about how much of a problem collinearity is in Bayesian models. I was arguing that it is not much of a problem. Imagine we have this model
Y ~ N(a + bX1 + cX2, sigma)
where X1 and X2 have some positive correlation (r > .5), they also have similar distributions. I would argue that if we assume 0 centered priors for b and c, then multi chain MCMC should find some balance between the estimates.
In frequentist/OLS models it is a problem and both estimates of b and c will be biased.
With synthetic data, some people have shown that Bayesian estimates are pretty close to biased frequentist estimates.
What do you think? How does it change if we have more parameters than we have data points (low DF)?
Yes, with an informative prior distribution on the coefficients you should be fine. Near-collinearity of predictors implies that the data can’t tell you so much about the individual coefficients—you can learn about the linear combination but not as much about the separate parameters—hence it makes sense to include prior information to do better.