(This article was originally published at Statistical Modeling, Causal Inference, and Social Science, and syndicated at StatsBlogs.)
Apparently there’s an idea out there that Bayesian inference with Gaussian processes automatically avoids overfitting. But no, you can still overfit.
To be precise, Bayesian inference by design avoids overfitting—if the evaluation is performed by averaging over the prior distribution. But to the extent this is not the case—to the extent that the frequency-evaluation distribution and the prior distribution are different things—we do need to worry about overfitting (or, I guess, underfitting).
But what do I know about Gaussian processes? Let’s hand the mike over to the experts.
I searched in the book and found exactly four mentions of “overfitting”:
The post Rasmussen and Williams never said that Gaussian processes resolve the problem of overfitting appeared first on Statistical Modeling, Causal Inference, and Social Science.
Please comment on the article here: Statistical Modeling, Causal Inference, and Social Science
