(This article was originally published at Statistical Modeling, Causal Inference, and Social Science, and syndicated at StatsBlogs.)
It is not that unusual in statistics to get the same statistical output (uncertainty interval, estimate, tail probability,etc.) for every sample, or some samples or the same distribution of outputs or the same expectations of outputs or just close enough expectations of outputs. Then, I would argue one has a variation on a DuckRabbit. In the DuckRabbit, the same sign represents different objects with different interpretations (what to make of it) whereas here we have differing signs (models) representing the same object (an inference of interest) with different interpretations (what to make of them). I will imaginatively call this a RabbitDuck ;-)
Does one always choose a Rabbit or a Duck, or sometimes one or another or always both? I would argue the higher road is both – that is to use differing models to collect and consider the different interpretations. Multiple perspectives can always be more informative (if properly processed), increasing our hopes to find out how things actually are by increasing the chances and rate of getting less wrong. Though this getting less wrong is in expectation only – it really is an uncertain world.
Of course, in statistics a good guess for the Rabbit interpretation would be Bayesian and for the Duck, Frequentest (Canadian spelling). Though, as one of Andrew’s colleagues once argued it is really modellers versus non modellers rather than Bayesians versus Frequentests and that makes a lot of sense to me. Representists are Rabbits “conjecturing, assessing, and adopting idealized representations of reality, predominantly using probability generating models for both parameters and data” while Propertyists are Ducks “primarily being about discerning procedures with good properties that are uniform over a wide range of possible underlying realities and restricting use, especially in science, to just those procedures” from here. Given that “idealized representations of reality” can only be indirectly checked (i.e. always remain possibly wrong) and “good properties” always beg the question “good for what?” (as well as only hold over a range of possible but largely unrepresented realities) – it should be a no brainer? that would it be more profitable than not to thoroughly think through both perspectives (and more actually).
An alternative view might be Leo Breiman’s “two cultures” paper.
This issue of multiple perspectives also came up in Bob’s recent post where the possibility arose that some might think it taboo to mix Bayes and Frequentist perspectives.
Some case studies would be:
In this blog post, Andrew implements and contrasts both the Rabbit route and the Duck route to get uncertainty intervals (using simulation for ease of wide understanding). It turns out that the intervals will not be different under a flat prior, while increasingly different under increasingly informative priors. Now the Duck route guarantees a property that is considered to be important and good by many – “uniform confidence coverage” and by some, even mandatory (e.g. see here). The Rabbit route with a flat prior will also happens to have this property (as it gives the same intervals). Perhaps to inform the good for what property, Andrew evaluates another property of making “claims with confidence” (type S and M error rates) and additionally evaluates that property.
With respect to this property “claims with confidence”, the Duck route (and the Rabbit route with flat prior) does not do so well – horribly actually. Now, informed with these two perspectives, it seems almost obvious that if a prior centred at zero and not too wide (implying large and very large effects are unlikely) is a reasonable “idealized representations of reality” for the area one is working in, the Rabbit route’s will have good properties while the Duck route’s guaranteed “good property” ain’t so good for you. On the other hand if effects of any size are all just as likely (which would be a strange universe to live in, perhaps not even possible) and you always keep in mind all the intervals you encounter, the Duck route will be fine.
Case study 2: The Bayesian Bootstrap
In the paper, Rubin outlines a Bayesian bootstrap that provides close enough expectations of outputs to the simple or vanilla bootstrap and argues that the implicit prior involved is _silly_ for some or many empirical research applications and hence shows the vanilla bootstrap is not an “analytic panacea that allows us to pull ourselves up by the bootstraps”. The bootstrap simply cannot avoid sensitivity to model assumptions. And in this post I am emphasising that _any_ model assumptions that give rise to a procedure with similar enough properties whether considered, used or even believed relevant? should be thought through. Not sure where this “case study” sits today – at one point Brad Efron was advancing ideas based on the bootstrap “as an automatic device for constructing Welch and Peers’ (1963) “probability matching priors” .
An aside, I find interesting in this paper of Rubin is the italicized phrase “with replacement”. It might be common knowledge today that the vanilla bootstrap simply samples from all possible sample paths of length n with replacement, but certainly in 1981 few seemed to realise that. I know because when Peter McCullagh presented work that was later published in Re-sampling and exchangeable arrays 2000 at the University of Toronto, I pointed this out to him and his response indicated he was not aware of this.
Case study 3: Bayarri et al Rejection Odds and Rejection Ratios .
This is a suggested Bayesian/Frequentest compromise for replacing the dreaded p_value/NHST. It is not being put forward as the best method for a replacement but rather one that can be easily adopted widely – Bayes with training wheels or a Frequentest approach with better balanced errors. Essentially a Bayesian inference that matches a frequentest expectation with the argument that “Any curve that has the right frequentist expectation is a valid frequentist report.”
I am not expected most readers will read even one of these case studies, but rather readers who do or have already read them, might share their views in comments.
Please comment on the article here: Statistical Modeling, Causal Inference, and Social Science