[R-sig-ME] F test vs. mcmcpvalue

[Hank Stevens]

Hi folks,

Are there general situations in which we might expect very different  
answers from F tests vs. mcmcpvalue with orthogonal contrasts (Helmert)?

I helping someone with a normal linear model with a moderate sized,  
noisy data set, and I am getting very different probabilities between  
F-tests and mcmcpvalue for some interactions.

I get similar F-test results whether I use lm (and ignore the random  
effect of subject), lme, and lmer with an DDF approximation.

When I use mcmcpvalue, I get huge changes in P-value of a main effect  
(0.6 to 0.01) when I remove its interactions. In contrast, the F-test  
(using trace of the hat matrix DF's) are much more consistent when I  
change the fixed effect structure.

I think mcmcpvalue is much more sensitive to overfitting the model. In  
some cases, removing the interactions results in a lower AIC (with ML  
fits).



In the full model, we have 28 fixed coefs (22 continuous variables or  
slope interactions) and about 500 obs.

The data are VERY unbalanced.

sessionInfo()
R version 2.7.1 (2008-06-23)
i386-apple-darwin8.10.1

locale:
C

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

other attached packages:
[1] foreign_0.8-26     Hmisc_3.4-3        lme4_0.999375-20    
Matrix_0.999375-10
[5] lattice_0.17-8

loaded via a namespace (and not attached):
[1] cluster_1.11.11 grid_2.7.1      tools_2.7.1
 >


Hank