[R-sig-ME] WAIC calculation in MCMCglmm

[Jarrod Hadfield]

Hi Jed,

The problem with all of these information criterion is that the deviance 
is 'focussed' at the highest level in MCMCglmm. For most scientific 
inference the focus should probably be at the lowest level. The problem 
is that the deviance cannot be calculated at the lowest level for GLMM 
(except in the Gaussian case) - that is why MCMC is being used. If the 
response is Gaussian you could refocus the deviance after the chain is 
ran. If not, brute force cross validation is probably the way to go, but 
of course in an MCMC context this can be costly in terms of computing time.

Cheers,

Jarrod


On 28/11/2017 01:06, Jed Macdonald wrote:
> Dear list,
>
> I’ve fitted a series of univariate mixed models of varying complexity in
> the 'MCMCglmm' package, and would like to compute WAIC for model selection
> purposes, for comparison with DIC, and with AICc returned for equivalent
> models fitted in 'lme4'. As I understand it, a first step in the WAIC
> calculation is to compute the log pointwise predictive density (i.e.
> pointwise log-likelihood), which is evaluated using draws from the retained
> posterior simulations (after burn-in). For the number of data points *N*
> and number of retained draws *S*, we can then get a *N* x *S*
> log-likelihood matrix, which can be used to estimate pointwise
> out-of-sample prediction accuracy (e.g. using WAIC or LOO cross-validation
> in the ‘loo’ package) (see Gelman et al. 2014, Vehtari et al. 2016 for an
> overview).
>
> MCMCglmm doesn’t return the pointwise log-likelihood directly, so my
> thinking was to use the deviance (D), given by D = −2log-likelihood in
> MCMCglmm, which is returned for each chain iteration. My question(s) is, do
> these values reflect the mean deviance across all *N* data points for a
> given iteration? And if so, is there a way to decompose this to pointwise
> deviance (and hence pointwise log-likelihood) values in an MCMCglmm model?
>
> Any advice would much appreciated!
>
> Best regards,
> Jed
>
> Gelman, A., Hwand, J. and Vehtari, A. (2014) Understanding predictive
> information criteria for Bayesian models. Stat Comput 24, 997-1016.
> Vehtari, A., Gelman, A. and Gabry, J. (2016) Practical Bayesian model
> evaluation using leave-one-out cross-validation and WAIC. arXiv:1507.04544.
>
>
>


-- 
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.