[R-sig-ME] degrees of freedom in mixed model

Greg Snow 538280 at gmail.com
Sat Jan 25 00:04:55 CET 2014


This post:  https://stat.ethz.ch/pipermail/r-sig-mixed-models/2009q1/001819.html
from a few years ago simulates data (based on a real dataset) and
computes a p-value, then looks at the distribution of the p-value.  In
some cases the p-value was distributed uniformly (when the null was
true) and so the approximation was reasonble, in another case the
p-values were very non-uniform (when the null was true), so for that
case the approximation failed.  It also shows that instead of worrying
about the exact distribution, you can just use the distribution
simulated from the null.

Similar simulations could be done to compare the above suggestions for
the structure of the data and questions of interest.

On Fri, Jan 24, 2014 at 11:19 AM, Ben Bolker <bbolker at gmail.com> wrote:
> On 14-01-24 01:07 PM, Jake Westfall wrote:
>> Meh... My feeling is that the amount of controversy on this point is
>> rather more limited than S Ellison lets on. Of course Bates is
>> (famously, at this point) deeply skeptical about the approximate
>> degrees of freedom approaches, but I get the impression that few of
>> the rest of us who have spent some time thinking about the matter
>> have any sort of strong feelings about it. The Satterthwaite method
>> (implemented in lmerTest) is widely used, well understood, and
>> basically seems to work quite well for controlling error rates in
>> most cases, based on simulations. I think if you are wanting to
>> scrutinize your model and the tests of the coefficients therein,
>> there are far bigger fish to fry than worrying about the issue of
>> approximate DFs vs. bootstrapping vs. MCMC vs. ... Jake
>
>     I more or less agree.  The issue with F distributions, degrees of
> freedom, etc etc., is mostly a problem with complex designs that don't
> fit into the classical method-of-moments/ANOVA paradigm (R-side effects
> [which lme4 doesn't do yet], crossed and partially crossed random
> effects, etc.).  In simple cases (as in the example here), the results
> of (restricted) ML analyses should more or less line up with the
> classical results.  In addition to lmerTest, as pointed out by Søren
> Hojsgaard in the original thread on r-help, the Kenward-Roger
> approximation is available in the PBKRtest package ...
>
>   If you asked me about 'denominator df' calculations for GLMMs I would
> be considerably more pessimistic ...
>
> Schaalje, G., J. McBride, and G. Fellingham. 2002. “Adequacy of
> Approximations to Distributions of Test Statistics in Complex Mixed
> Linear Models.” Journal of Agricultural, Biological & Environmental
> Statistics 7 (14): 512–24.
> http://www.ingentaconnect.com/content/asa/jabes/2002/00000007/00000004/art00004.
>
>
>>> From: jbaldwin at fs.fed.us To: S.Ellison at LGCGroup.com;
>>> iaingallagher at btopenworld.com; r-sig-mixed-models at r-project.org
>>> Date: Fri, 24 Jan 2014 15:40:32 +0000 Subject: Re: [R-sig-ME]
>>> degrees of freedom in mixed model
>>>
>>> S Ellison:  Despite you being a chemist, I think you're at least
>>> mostly correct.  But from the construction of my statement, it's
>>> obvious that I am a statistician and I'm allowed, by law, to be
>>> wrong 5% of the time.  And if I claim to be a Frequentist, I don't
>>> even have to identify which of my particular statements are
>>> incorrect.
>>>
>>> Jim
>>>
>>> Jim Baldwin Station Statistician Pacific Southwest Research
>>> Station USDA Forest Service
>>>
>>> -----Original Message----- From:
>>> r-sig-mixed-models-bounces at r-project.org
>>> [mailto:r-sig-mixed-models-bounces at r-project.org] On Behalf Of S
>>> Ellison Sent: Friday, January 24, 2014 5:40 AM To: Iain Gallagher;
>>> r-sig-mixed-models at r-project.org Subject: Re: [R-sig-ME] degrees of
>>> freedom in mixed model
>>>
>>>
>>>> library(lme4) model1 <- lmer(value~group + (1|animal), data=bip)
>>>> summary(model1)
>>>>
>>> .......
>>>> so I'd then have:
>>>>
>>>> qf(0.95,3,5) or qf(0.95,3,4)
>>>>
>>>> for my critical F value?
>>>>
>>>> Any advice (incuding whether the appraoch is right) would be
>>>> useful.
>>>
>>> It's the wrong approach.
>>>
>>> You are using lmer, which uses maximum likelihood estimation, not
>>> classical sums of squares. The degrees of freedom don't mean the
>>> same thing, and the distribution of REML estimates of variance
>>> isn't necessarily chi-squared. So F is interpretable in the same
>>> way as it would be in classical anova.
>>>
>>> If you want p-values from an lmer model, you could get hold of the
>>> lmerTest package. Other recommended approaches include variants on
>>> MCMC. There is a great deal of controversy on this point, though;
>>> try Googling "p-values from lmer" with particular attention to
>>> anything by Douglas Bates (the package author). You _should_ find
>>> enough to make you worry that the method used by lmerTest (which as
>>> I understand it implements a method used by SAS) comes with quite
>>> strong theoretical objections. I am quite sure the lmerTest authors
>>> know that perfectly well and offer lmerTest as a package for those
>>> who want to find out or for those whose management insist on a
>>> SAS-compatible answer. But if I read correctly, that doesn't make
>>> it the right thing to do
>>>
>>> [Caveat - I'm a chemist. I could be wrong about this]
>>>
>>>>
>>>> Best
>>>>
>>>> iain
>>>>
>>>> [[alternative HTML version deleted]]
>>>
>>>
>>>
>>> *******************************************************************
>>>
>>>
> This email and any attachments are confidential. Any u...{{dropped:9}}
>>
>> _______________________________________________
>> R-sig-mixed-models at r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>>
>
> _______________________________________________
> R-sig-mixed-models at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models



-- 
Gregory (Greg) L. Snow Ph.D.
538280 at gmail.com



More information about the R-sig-mixed-models mailing list