[R-sig-ME] F test vs. mcmcpvalue
spencer.graves at pdf.com
Mon Jul 7 00:16:08 CEST 2008
I haven't used 'mcmcpvalue', but I would naively expect that MCMC
should in most cases give better answers in violations of the standard
assumptions. This is NOT necessarily the case, however, because it is
known that MCMC does not always converge to a unique answer. For
example, with multimodal posteriors, sufficiently distinct modes may not
be adequately explored, leading possibly to inappropriate estimates of p
values. In such cases, if there are no gross violations of the standard
assumptions, I would trust the F test more.
Have you done normal probability plots and other plots of the
response variable, residuals, and random coefficients? This might help
you identify gross violations of the standard assumptions. It's
possible that removing an outlier might substantially reduce the
difference between your F tests and 'mcmcpvalue'.
Hope this helps.
Hank Stevens wrote:
> Hi folks,
> Are there general situations in which we might expect very different
> answers from F tests vs. mcmcpvalue with orthogonal contrasts (Helmert)?
> I helping someone with a normal linear model with a moderate sized,
> noisy data set, and I am getting very different probabilities between
> F-tests and mcmcpvalue for some interactions.
> I get similar F-test results whether I use lm (and ignore the random
> effect of subject), lme, and lmer with an DDF approximation.
> When I use mcmcpvalue, I get huge changes in P-value of a main effect
> (0.6 to 0.01) when I remove its interactions. In contrast, the F-test
> (using trace of the hat matrix DF's) are much more consistent when I
> change the fixed effect structure.
> I think mcmcpvalue is much more sensitive to overfitting the model. In
> some cases, removing the interactions results in a lower AIC (with ML
> In the full model, we have 28 fixed coefs (22 continuous variables or
> slope interactions) and about 500 obs.
> The data are VERY unbalanced.
> R version 2.7.1 (2008-06-23)
> attached base packages:
>  stats graphics grDevices utils datasets methods base
> other attached packages:
>  foreign_0.8-26 Hmisc_3.4-3 lme4_0.999375-20
>  lattice_0.17-8
> loaded via a namespace (and not attached):
>  cluster_1.11.11 grid_2.7.1 tools_2.7.1
> R-sig-mixed-models at r-project.org mailing list
More information about the R-sig-mixed-models