[R-sig-ME] GLMM, LRTs and post-hocs

[John Maindonald]

1) The LRT is a more reliable guide than what I assume are the modified
Wald statistics that you are using for the post-hoc comparisons.  The
Wald statistics involve approximations that can play havoc with the
distributional assumptions that are commonly use.
2) Even if both tests are pretty much beyond reproach, as in some 
normal theory contexts, different testing procedures may give different 
answers. The LRT is an overall test, automatically taking a/c of multiplicity.  
If you believe in adjustments for multiplicity, or (a better reason, I consider) 
think that such an adjustment accords with the aims of the project, you 
might go with the LRT.

-----------------------------------------------------------------------------------------------

Now for what may seem something of a hobby-horse.  I wish we could 
move away from this heavy reliance on p-values.  Often, it will do much 
better justice to the data to say:

a) These (. . . .) are the effects stand out clearly, pretty much irrespective 
of the twists and turns of the way that the results might be interpreted . . .

b) Next note effects that are borderline . . .  Remember that there can be
model uncertainty as well as the statistical uncertainty that a particular
form of analysis identifies.  With GLMMs that have a fixed scale factor, 
there is the issue of whether observation level random effects are
required to adequately account for the variation.  If there are such
effects, then unless they are very small and/or there are many 
observations, think carefully before taking too much notice of p-values 
that may be given for the Wald statistics.

c) Whichever way one shakes it, remaining effect estimates are 
consistent with statistical variation.

This way of presenting results would, often, give a clearer and
more honest account of the data and of model uncertainty.

John Maindonald             email: john.maindonald at anu.edu.au
phone : +61 2 (6125)3473    fax  : +61 2(6125)5549
Centre for Mathematics & Its Applications, Room 1194,
John Dedman Mathematical Sciences Building (Building 27)
Australian National University, Canberra ACT 0200.
http://www.maths.anu.edu.au/~johnm

On 26/08/2010, at 2:53 AM, Kay Cecil Cichini wrote:

> hello everyone,
> 
> i did a GLMM with two fixed crossed (2 levels each) and one random factor. modelselection by LRTs gives no significance for the one fixed factor i'm mainly interested in, neither for the interaction with the second factor. however when i do post-hoc tests on the first factor within each level of the second i get significant effects at one of the two levels of the second factor, saying factor one is significant but only within level X of the factor two.
> 
> now i'm clueless how to proceed with this result, because LRTs tell me to dismiss the factor of interest but by pos-hocs i know in fact the one factor has an effect, however restricted on one level of the second factor...
> 
> any comments on this would be greatly appreciated,
> kay
> 
> _______________________________________________
> R-sig-mixed-models at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models