[R-sig-ME] MCMC model selection reference

[Ryan King]

I'd be interested if you find something good. Lately I've been
interested in a new criterion, WAIC. It's supposed to be an improved
estimate of out-of-sample / cross-validation prediction error (which
DIC is supposed to be also).

Discussion here:
http://andrewgelman.com/2011/06/plummer_on_dic/
http://andrewgelman.com/2011/06/deviance_dic_ai/

>From plummer's 2008 paper discussion:

"The question of what constitutes a noteworthy difference in DIC
between 2 models has not yet received a satisfactory answer. Whereas
calibration scales have been proposed for Bayes factors (Kass and
Raftery, 1995), no credible scale has been proposed for the difference
in DIC between 2 models. Indeed, such a scale is unlikely be useful.
Ripley (1996) shows that the sampling error of the difference in AIC
between 2 models is Op(1) when the models are nested, and the smaller
model is true, but the error may be Op(sqrt(n)) for nonnested models.
No absolute scale for interpretation of AIC could be valid in both
situations. DIC inherits this behavior since it includes AIC as a
special case."

The related question of how much cross-validation error is a lot is
unknown; in fact it is known that there does not exist an unbiased
estimate of the variance of CV error.
Bengio, Y. & Grandvalet, Y. No unbiased estimator of the variance of
k-fold cross-validation. The Journal of Machine Learning Research 5,
1089–1105 (2004).

There are concentration
inequalities which are generally too loose to be useful (or so I'm
told). There are
some promising recent theory papers that I haven't been able to fully
read. Plummer goes on to suggest parametric simulation to calibrate
his DIC variant. That or non-parametric simulation (bootstrap,
permutation) is all that I've seen applied papers do. I've tried
simulating calibration scales and found that the sampling variance of
 CV depends on how complex the model is / is allowed to be.

The WAIC paper is nice in that is says explicitly what it is trying
the calculate: the cross-validation loss of the log-evidence and shows
its relationship to DIC and BIC.

Ryan King
University of Chicago
Dept Health Studies