I keep being drawn to thinking there is a away to explain statistical reasoning to others that will actually do more good than harm. Now, I also keep thinking I should know better – but can’t stop.
My recent attempt starts with a shadow metaphor, then a review of analytical chemistry and moves to the concept of abstract fake universes (AFUs). AFU’s allow you to be the god of a universe, though not a real one ;-). However, ones you can conveniently define using probability models where it is easy to discern what would repeated happen – given an exactly set truth.
The shadow metaphor emphasizes that though you see shadows, you are really interested in what is casting the shadows. The analytical chemistry metaphor emphasizes the advantage of making exactly known truths by spiking a set amount of a chemical in test tubes and repeatedly measuring the test tube contents with the inherently noisy assays. For many empirical questions such spiking is not possible (e.g. underlying cancer incidence) so we have no choice but the think abstractly. Now, abstractly a probability model is an abstract shadow generating machine: with set parameters values it can generate shadows. Well actually samples. Then it seems advantageous to think of probability models as an ideal means to make AFUs with exactly set truths where it is easy to discern what would repeated happen.
Now, my enthusiasm is buoyed by the realization that one of the best routes for doing that is the prior predictive. The prior predictive generates a large set of hopefully appropriate fake universes, where you know the truth (prior parameters drawn) and can discern what the posterior would be (given the joint model proposed for the analysis and the fake data generated). That is, in each fake universe from the prior predictive, you have the true parameter value(s), the fake sample that the data generating model generated from these and can discern what the posterior would have been calculated to be. Immediately (given computation time) one obtains a large sample of what would repeatedly happen using the proposed joint model in varied fake universes. Various measures of goodness can then be assessed and various averages then calculated.
Good for what? And in which appropriate collective of AFUs (aka Bayesian reference set)?
An earlier attempt of mine to do this in a lecture was recorded and Bob Carpenter has kindly uploaded it as Lectures: Principled Bayesian Workflow—Practicing Safe Bayes (YouTube) Keith O’Rourke (2019). I you decide to watch it, I would suggest setting the play back speed at 1.25. For those who don’t like videos slides and code are here.
The rest of the post below provides some background material for those who may lack background in prior predictive simulation and two stage sampling to obtain a sample from the posterior.