[R] lme X lmer results

[Peter Dalgaard]

Dave Atkins <datkins at u.washington.edu> writes:

> Message: 18
> Date: Fri, 30 Dec 2005 12:51:59 -0600
> From: Douglas Bates <dmbates at gmail.com>
> Subject: Re: [R] lme X lmer results
> To: John Maindonald <john.maindonald at anu.edu.au>
> Cc: r-help at stat.math.ethz.ch
> Message-ID:
> 	<40e66e0b0512301051i2dc0f257r745c70e749c250f0 at mail.gmail.com>
> Content-Type: text/plain; charset=ISO-8859-1
> 
> On 12/29/05, John Maindonald <john.maindonald at anu.edu.au> wrote:
> 
>  >> Surely there is a correct denominator degrees of freedom if the design
>  >> is balanced, as Ronaldo's design seems to be. Assuming that he has
>  >> specified the design correctly to lme() and that lme() is getting the df
>  >> right, the difference is between 2 df and 878 df.  If the t-statistic
>  >> for the
>  >> second level of Xvar had been 3.0 rather than 1.1, the difference
>  >> would be between a t-statistic equal to 0.095 and 1e-6.  In a design
>  >> where there are 10 observations on each experimental unit, and all
>  >> comparisons are at the level of experimental units or above, df for
>  >> all comparisons will be inflated by a factor of at least 9.
> 
> Doug Bates commented:
> 
> I don't want to be obtuse and argumentative but I still am not
> convinced that there is a correct denominator degrees of freedom for
> _this_ F statistic.  I may be wrong about this but I think you are
> referring to an F statistic based on a denominator from a different
> error stratum, which is not what is being quoted.  (Those are not
> given because they don't generalize to unbalanced designs.)
> 
> This is why I would like to see someone undertake a simulation study
> to compare various approaches to inference for the fixed effects terms
> in a mixed model, using realistic (i.e. unbalanced) examples.
> 
> Doug--
> 
> Here is a paper that focused on the various alternatives to denominator degrees 
> of freedom in SAS and does report some simulation results:
> 
> http://www2.sas.com/proceedings/sugi26/p262-26.pdf
> 
> Not sure whether it argues convincingly one way or the other in the present 
> discussion.
> 
> cheers, Dave

It's certainly informative. I was not quite aware that what SAS calls
"Satterthwaite" could perform so poorly relative to "KenwardRoger"
(which to my mind is much closer in structure to a generalized
Satterthwaite approximation).

In either case I think it might be worth emphasizing that the issue is
not really whether you are testing at alpha=.050 or at alpha=.093 --
the corrections are likely to be so highly dependent on higher order
moments of the normal distribution to be wildly off in practice. The
important thing is that you get a low denominator DF value when you
are far from "Asymptopia" and any good statistician should know better
than to trust such tests. 

The nasty problem with the "containment" type methods is that they can
get things completely wrong and, e.g., give DFs on the order of the
number of observations where in truth (insofar as it exists) it should
be closer to the number of subjects in the study.

I don't think anyone, including Doug, would be opposed to including a
Kenward-Roger style DF calculation in lmer. It "just" has to be worked
out how to convert the calculations to work with the sparse-matrix,
penalized least squares techniques that it uses, and Doug himself has
his mind elsewhere. 

-- 
   O__  ---- Peter Dalgaard             Øster Farimagsgade 5, Entr.B
  c/ /'_ --- Dept. of Biostatistics     PO Box 2099, 1014 Cph. K
 (*) \(*) -- University of Copenhagen   Denmark          Ph:  (+45) 35327918
~~~~~~~~~~ - (p.dalgaard at biostat.ku.dk)                  FAX: (+45) 35327907