[R-sig-ME] Penalty = shrinkage = ?

[Prew, Paul]

Douglas,  thank you for the explanation and the slides.  I understand the mixed modeling approach better now, or think I do.  Shrinkage seems analogous to weighted least squares (a method that's commonly taught and rarely if ever used, from what I've seen).  Regards, Paul


Paul Prew  |  Statistician
651-795-5942   |   fax 651-204-7504 
Ecolab Research Center  | Mail Stop ESC-F4412-A 
655 Lone Oak Drive  |  Eagan, MN 55121-1560 


-----Original Message-----
From: dmbates at gmail.com [mailto:dmbates at gmail.com] On Behalf Of Douglas Bates
Sent: Thursday, November 19, 2009 12:10 PM
To: Prew, Paul
Cc: r-sig-mixed-models at r-project.org
Subject: Re: [R-sig-ME] Penalty = shrinkage = ?

On Thu, Nov 19, 2009 at 11:56 AM, Prew, Paul <Paul.Prew at ecolab.com> wrote:
> I found the comment below interesting in one of yesterday's threads, 
> as I am currently analyzing a data set with a random effect fully 
> nested within a fixed factor.  Could anyone elaborate on what is meant 
> by "penalty on the random effect"? Is this also what is deemed 
> "shrinkage"?  How does it work?  Thanks, Paul

Look at slide 25 in
http://lme4.r-forge.r-project.org/slides/2009-07-21-Seewiesen/5LongitudinalD.pdf

In this slide the parameter estimates that you would have gotten by fitting each subject's data separately are compared with the estimates from a mixed-effects model with random effects for slope and intercept.  The effective slope and intercept for each subject is shrunk toward the population-wide estimate compared to the within-subject estimate.  John Tukey referred to this as "borrowing strength" from the population.

The extent of the shrinkage is controlled by the variance-covariance matrix of the random effects.  A large variance results in parameter estimates that are close to the within-subject estimates.  In terms of the discussion on fidelity to the data versus model complexity in another thread, such a model has high complexity and high fidelity.
The opposite case, very low variance for the random effects provides a low complexity model but with correspondingly low fidelity to the data.

Slide 26 in that presentation shows that the subjects whose data is rather noisy, and hence whose within-subject parameter estimates are poorly determined (330 or 331), have their coefficients "shrunk" more than those whose data determines the within-subject estimates very well (309 or 349).

> "I understand that when a random effect is fully nested within a fixed 
> effect, the penalty on the random effect resolves the singularity and 
> allows estimation of both. (That is, if appropriate, you could model 
> depfemr as a fixed effect?)"
>
>
>
> Paul Prew   ▪  Statistician
> 651-795-5942   ▪   fax 651-204-7504
> Ecolab Research Center   ▪  Mail Stop ESC-F4412-A
> 655 Lone Oak Drive   ▪   Eagan, MN 55121-1560
>
>
>
> CONFIDENTIALITY NOTICE:
> This e-mail communication and any attachments may contain proprietary and privileged information for the use of the designated recipients named above.
> Any unauthorized review, use, disclosure or distribution is prohibited.
> If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.
>
>
>        [[alternative HTML version deleted]]
>
>
> _______________________________________________
> R-sig-mixed-models at r-project.org mailing list 
> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>
>

CONFIDENTIALITY NOTICE: 
This e-mail communication and any attachments may contain proprietary and privileged information for the use of the designated recipients named above. 
Any unauthorized review, use, disclosure or distribution is prohibited. 
If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.