(This article was originally published at Statistical Modeling, Causal Inference, and Social Science, and syndicated at StatsBlogs.)
Mark Girolami sends along a new paper by Francois-Xavier Briol, Chris Oates, Michael Osborne, Dino Sejdinovic, and himself. The idea is to consider numerical integration as a statistical problem, to say that the integral being estimated is an unknown parameter and then to perform inference about it. This is related to ideas of Xiao-Li Meng, also in some general way related to the calculus of variations, I think.
In reading it I was also reminded of a remark of Don Rubin’s from many years ago, when he said that he considers whatever approximation he is currently using to be his posterior distribution. Rubin’s remark made sense to me, but I’ve always worried about the overfitting problem. Suppose, for example, we fit a big Bayesian model but using a crude, mode-based approximation. The posterior mode could well overfit the data, which then would not show up as a problem in a posterior predictive check. The usual Bayesian answer to the overfitting problem is that if you overfit you’ll get big standard errors, but that won’t happen if your approximation or simulation algorithm doesn’t fully explore the posterior distribution.
Anyway, that’s a bit of a side point, it’s not a criticism of the linked paper by Briol et al., of which I think this is the key passage:
Bayesian Quadrature (BQ; O’Hagan, 1991) is a probabilistic numerics method that performs integration from a statistical perspective. Specifically, BQ assigns a Gaussian process prior measure over the integrand f and then, based on data D = {xi,fi}ni=1 with xi ∈ X and fi = f(xi), outputs a Gaussian process posterior measure f|D according to Bayes’ rule. In turn this implies a Gaussian posterior distribution over Π[f], since Π is a linear functional in f, representing a probabilistic model for uncertainty over the true value of Π[f]. The approach applies equally to a pre-determined set of states {xi}ni=1 or to states that are realisations from a random process, such as samples from a probability distribution which is often, but not necessarily, Π. In the latter, randomised case the method is known as Bayesian Monte Carlo (BMC; Rasmussen and Ghahramani, 2002).
Compared to non-probabilistic integrators, BQ has lacked rigorous theoretical foundations; this is addressed by the present paper.
So I guess that’s the point—they have some convergence proofs for this approach in which integrals are computed by fitting an underlying Gaussian process.
They also write:
The performance of probabilistic integrators is guaranteed to be no worse than non-probabilistic integrators and is, in many cases, asymptotically superior.
I assume when they refer to “performance,” here, it is scaled by number of iterations or number of simulation draws, not by wall time. I’d think that fitting a Gaussian process could be time consuming.
In any case, I think this is a valuable perspective. I agree with their basic idea that computational problems should be framed statistically.
The post Probabilistic Integration appeared first on Statistical Modeling, Causal Inference, and Social Science.
Please comment on the article here: Statistical Modeling, Causal Inference, and Social Science
