Daniel Lakeland writes:
Ok, so it’s really deep into the comments and I’m guessing there’s a chance you will miss it so I wanted to point at my comments here and here.
In particular, the second one, which suggests something that it might be useful to recommend for Bayesian workflows: calculating a likelihood ratio for data compared to peak likelihood.
I imagine in Stan using generated quantities to calculate say the .001, .01, .1, .25, .5 quantiles of log(L(Data)/Lmax) or something like that and using this as a measure of model misfit on a routine basis. I think it would be useful to know for example that for a given posterior draw from the parameters, the least likely data points are no less than say 10^-4 times as likely as the data value at the mode of the likelihood, and you’d definitely like to know if some percentage of the data is 10^-37 times less likely ;-) that would flag some serious model mis-fitting.
Since the idea deserves elaboration, but the comments on this blog post are sort of played out, what do you think I should do with this idea? Is there a productive place to discuss it or maybe set up some kind of example vignette or something?
I don’t have the energy to think this through so I thought I’d just post it here. My only quick thought is that the whole peak likelihood thing can’t work in general because (a) the likelihood can blow up, and (b) wherever is the peak likelihood can be super-noisy at times. So I’d replace “peak likelihood” with something using an informative prior. But maybe there’s something to the rest of this?