[R-sig-ME] F test vs. mcmcpvalue

[Spencer Graves]

      I haven't used 'mcmcpvalue', but I would naively expect that MCMC 
should in most cases give better answers in violations of the standard 
assumptions.  This is NOT necessarily the case, however, because it is 
known that MCMC does not always converge to a unique answer.  For 
example, with multimodal posteriors, sufficiently distinct modes may not 
be adequately explored, leading possibly to inappropriate estimates of p 
values.  In such cases, if there are no gross violations of the standard 
assumptions, I would trust the F test more. 

      Have you done normal probability plots and other plots of the 
response variable, residuals, and random coefficients?  This might help 
you identify gross violations of the standard assumptions.  It's 
possible that removing an outlier might substantially reduce the 
difference between your F tests and 'mcmcpvalue'. 
    
      Hope this helps. 
      Spencer

Hank Stevens wrote:
> Hi folks,
>
> Are there general situations in which we might expect very different 
> answers from F tests vs. mcmcpvalue with orthogonal contrasts (Helmert)?
>
> I helping someone with a normal linear model with a moderate sized, 
> noisy data set, and I am getting very different probabilities between 
> F-tests and mcmcpvalue for some interactions.
>
> I get similar F-test results whether I use lm (and ignore the random 
> effect of subject), lme, and lmer with an DDF approximation.
>
> When I use mcmcpvalue, I get huge changes in P-value of a main effect 
> (0.6 to 0.01) when I remove its interactions. In contrast, the F-test 
> (using trace of the hat matrix DF's) are much more consistent when I 
> change the fixed effect structure.
>
> I think mcmcpvalue is much more sensitive to overfitting the model. In 
> some cases, removing the interactions results in a lower AIC (with ML 
> fits).
>
>
>
> In the full model, we have 28 fixed coefs (22 continuous variables or 
> slope interactions) and about 500 obs.
>
> The data are VERY unbalanced.
>
> sessionInfo()
> R version 2.7.1 (2008-06-23)
> i386-apple-darwin8.10.1
>
> locale:
> C
>
> attached base packages:
> [1] stats     graphics  grDevices utils     datasets  methods   base
>
> other attached packages:
> [1] foreign_0.8-26     Hmisc_3.4-3        lme4_0.999375-20   
> Matrix_0.999375-10
> [5] lattice_0.17-8
>
> loaded via a namespace (and not attached):
> [1] cluster_1.11.11 grid_2.7.1      tools_2.7.1
> >
>
>
> Hank
>
> _______________________________________________
> R-sig-mixed-models at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models