[R] Fwd: Variance Components in R

Spencer Graves spencer.graves at pdf.com
Thu Aug 17 19:46:45 CEST 2006


      Burt Gunter just reminded me that the completion time could also 
be affected by the numbers of levels of each of the factors, especially 
random effects:  With N records, any variance components / mixed model 
software using MLE or REML will have to invert repeatedly an N x N 
matrix for the covariance structure of the random effects and noise.  If 
the software recognizes your design as having some simple structure, 
this can be quite fast;  otherwise, it could be a Herculean task.  In 
your case with N = 9500 records, just one copy of this covariance matrix 
could consume a substantial portion of 1GB RAM.  I compute 
8*9500*(9500-1)/2 = 361Mbytes. 

      However, any software that recognizes special structure in your 
design may be able to do the required computations without ever 
constructing a matrix this large.  I would say that it's still worth a 
try in R on your laptop or on the machine with 1GB RAM:  'lmer' might 
recognize special structure that neither of the other two do (and vice 
versa). 

      Hope this helps. 
      Spencer Graves    

Iuri Gavronski wrote:
> We have tried on many machines, from my laptop to a dual core Intel 
> processor with 1GB of RAM.
>
> On 8/17/06, *Spencer Graves* < spencer.graves at pdf.com 
> <mailto:spencer.graves at pdf.com>> wrote:
>
>     Hi, Iuri:
>
>           How much RAM and how fast a microprocessor (and what version of
>     Windows) do you have?  You might still try it in R under Windows.  The
>     results might be comparable or dramatically better in R than in
>     SPSS or
>     SAS.
>
>           hope this helps.
>           Spencer Graves
>
>     Iuri Gavronski wrote:
>     > 9500 records. It didn`t run in SPSS or SAS on Windows machines,
>     so I am
>     > trying to convert the SPSS script to R to run in a RISC station
>     at the
>     > university.
>     >
>     > On 8/17/06, Doran, Harold < HDoran at air.org
>     <mailto:HDoran at air.org>> wrote:
>     >
>     >> Iuri:
>     >>
>     >> The lmer function is optimal for large data with crossed random
>     effects.
>     >> How large are your data?
>     >>
>     >>
>     >>> -----Original Message-----
>     >>> From: r-help-bounces at stat.math.ethz.ch
>     <mailto:r-help-bounces at stat.math.ethz.ch>
>     >>> [mailto: r-help-bounces at stat.math.ethz.ch
>     <mailto:r-help-bounces at stat.math.ethz.ch>] On Behalf Of Iuri Gavronski
>     >>> Sent: Thursday, August 17, 2006 11:08 AM
>     >>> To: Spencer Graves
>     >>> Cc: r-help at stat.math.ethz.ch <mailto:r-help at stat.math.ethz.ch>
>     >>> Subject: Re: [R] Variance Components in R
>     >>>
>     >>> Thank you for your reply.
>     >>> VARCOMP is available at SPSS advanced models, I'm not sure
>     >>> for how long it exists... I only work with SPSS for the last
>     >>> 4 years...
>     >>> My model only has crossed random effects, what perhaps would
>     >>> drive me to lmer().
>     >>> However, as I have unbalanced data (why it is normally called
>     >>> 'unbalanced design'? the data was not intended to be
>     >>> unbalanced, only I could not get responses for all cells...),
>     >>> I'm afraid that REML would take too much CPU, memory and time
>     >>> to execute, and MINQUE would be faster and provide similar
>     >>> variance estimates (please, correct me if I'm wrong on that
>     point).
>     >>> I only found MINQUE on the maanova package, but as my study
>     >>> is very far from genetics, I'm not sure I can use this package.
>     >>> Any comment would be appreciated.
>     >>> Iuri
>     >>>
>     >>> On 8/16/06, Spencer Graves < spencer.graves at pdf.com
>     <mailto:spencer.graves at pdf.com>> wrote:
>     >>>
>     >>>>       I used SPSS over 25 years ago, but I don't recall
>     >>>>
>     >>> ever fitting a
>     >>>
>     >>>> variance components model with it.  Are all your random
>     >>>>
>     >>> effects nested?
>     >>>
>     >>>> If they were, I would recommend you use 'lme' in the 'nlme'
>     package.
>     >>>> However, if you have crossed random effects, I suggest you
>     >>>>
>     >>> try 'lmer'
>     >>>
>     >>>> associated with the 'lme4' package.
>     >>>>
>     >>>>       For 'lmer', documentation is available in Douglas
>     >>>>
>     >>> Bates. Fitting
>     >>>
>     >>>> linear mixed models in R. /R News/, 5(1):27-30, May 2005
>     >>>> (www.r-project.org <http://www.r-project.org> ->
>     newsletter).  I also recommend you try the
>     >>>> vignette available with the 'mlmRev' package (see, e.g.,
>     >>>> http://finzi.psych.upenn.edu/R/Rhelp02a/archive/81375.html
>     <http://finzi.psych.upenn.edu/R/Rhelp02a/archive/81375.html>).
>     >>>>
>     >>>>        Excellent documentation for both 'lme' (and indirectly for
>     >>>> 'lmer') is available in Pinheiro and Bates (2000)
>     >>>>
>     >>> Mixed-Effects Models
>     >>>
>     >>>> in S and S-Plus (Springer).  I have personally recommended
>     >>>>
>     >>> this book
>     >>>
>     >>>> so many times on this listserve that I just now got 234 hits for
>     >>>> RSiteSearch("graves pinheiro").  Please don't hesitate to
>     pass this
>     >>>> recommendation to your university library.  This book is
>     >>>>
>     >>> the primary
>     >>>
>     >>>> documentation for the 'nlme' package, which is part of the
>     >>>>
>     >>> standard R
>     >>>
>     >>>> distribution.  A subdirectory "~library\nlme\scripts" of your R
>     >>>> installation includes files named "ch01.R", "ch02.R", ...,
>     >>>>
>     >>> "ch06.R",
>     >>>
>     >>>> "ch08.R", containing the R scripts described in the
>     book.  These R
>     >>>> script files make it much easier and more enjoyable to study that
>     >>>> book, because they make it much easier to try the commands
>     >>>>
>     >>> described
>     >>>
>     >>>> in the book, one line at a time, testing modifications to
>     check you
>     >>>> comprehension, etc.  In addition to avoiding problems with
>     >>>> typographical errors, it also automatically overcomes a few
>     >>>>
>     >>> minor but
>     >>>
>     >>>> substantive changes in the notation between S-Plus and R.
>     >>>>
>     >>>>       Also, the "MINQUE" method has been obsolete for over
>     >>>>
>     >>> 25 years.
>     >>>
>     >>>> I recommend you use method = "REML" except for when you want to
>     >>>> compare two nested models with different fixed effects;  in
>     >>>>
>     >>> that case,
>     >>>
>     >>>> you should use method = "ML", as explained in Pinheiro and
>     >>>>
>     >>> Bates (2000).
>     >>>
>     >>>>       Hope this helps.
>     >>>>       Spencer Graves
>     >>>>
>     >>>> Iuri Gavronski wrote:
>     >>>>
>     >>>>> Hi,
>     >>>>>
>     >>>>> I'm trying to fit a model using variance components in R,
>     but if
>     >>>>> very new on it, so I'm asking for your help.
>     >>>>>
>     >>>>> I have imported the SPSS database onto R, but I don't know
>     how to
>     >>>>> convert the commands... the SPSS commands I'm trying to
>     >>>>>
>     >>> convert are:
>     >>>
>     >>>>> VARCOMP
>     >>>>>    RATING BY CHAIN SECTOR RESP ASPECT ITEM
>     >>>>>    /RANDOM = CHAIN SECTOR RESP ASPECT ITEM
>     >>>>>    /METHOD = MINQUE (1)
>     >>>>>    /DESIGN = CHAIN SECTOR RESP ASPECT ITEM
>     >>>>>                SECTOR*RESP SECTOR*ASPECT SECTOR*ITEM CHAIN*RESP
>     >>>>> CHAIN*ASPECT CHAIN*ITEM RESP*ASPECT RESP*ITEM
>     >>>>>                SECTOR*RESP*ASPECT SECTOR*RESP*ITEM
>     >>>>>
>     >>> CHAIN*RESP*ASPECT
>     >>>
>     >>>>>    /INTERCEPT = INCLUDE.
>     >>>>>
>     >>>>> VARCOMP
>     >>>>>    RATING BY CHAIN SECTOR RESP ASPECT ITEM
>     >>>>>    /RANDOM = CHAIN SECTOR RESP ASPECT ITEM
>     >>>>>    /METHOD = REML
>     >>>>>    /DESIGN = CHAIN SECTOR RESP ASPECT ITEM
>     >>>>>                SECTOR*RESP SECTOR*ASPECT SECTOR*ITEM CHAIN*RESP
>     >>>>> CHAIN*ASPECT CHAIN*ITEM RESP*ASPECT RESP*ITEM
>     >>>>>                SECTOR*RESP*ASPECT SECTOR*RESP*ITEM
>     >>>>>
>     >>> CHAIN*RESP*ASPECT
>     >>>
>     >>>>>    /INTERCEPT = INCLUDE.
>     >>>>>
>     >>>>> Thank you for your help.
>     >>>>>
>     >>>>> Best regards,
>     >>>>>
>     >>>>> Iuri.
>     >>>>>
>     >>>>> _______________________________________
>     >>>>> Iuri Gavronski - iuri at ufrgs.br <mailto:iuri at ufrgs.br>
>     >>>>> doutorando
>     >>>>> UFRGS/PPGA/NITEC - www.ppga.ufrgs.br
>     <http://www.ppga.ufrgs.br> Brazil
>     >>>>>
>     >>>>> ______________________________________________
>     >>>>> R-help at stat.math.ethz.ch <mailto:R-help at stat.math.ethz.ch>
>     mailing list
>     >>>>> https://stat.ethz.ch/mailman/listinfo/r-help
>     <https://stat.ethz.ch/mailman/listinfo/r-help>
>     >>>>> PLEASE do read the posting guide
>     >>>>>
>     >>>> http://www.R-project.org/posting-guide.html
>     >>>>
>     >>>>> and provide commented, minimal, self-contained, reproducible
>     code.
>     >>>>>
>     >>>>>
>     >>>       [[alternative HTML version deleted]]
>     >>>
>     >>> ______________________________________________
>     >>> R-help at stat.math.ethz.ch <mailto:R-help at stat.math.ethz.ch>
>     mailing list
>     >>> https://stat.ethz.ch/mailman/listinfo/r-help
>     >>> PLEASE do read the posting guide
>     >>> http://www.R-project.org/posting-guide.html
>     >>> and provide commented, minimal, self-contained, reproducible
>     code.
>     >>>
>     >>>
>     >
>     >       [[alternative HTML version deleted]]
>     >
>     > ______________________________________________
>     > R-help at stat.math.ethz.ch <mailto:R-help at stat.math.ethz.ch>
>     mailing list
>     > https://stat.ethz.ch/mailman/listinfo/r-help
>     > PLEASE do read the posting guide
>     http://www.R-project.org/posting-guide.html
>     > and provide commented, minimal, self-contained, reproducible code.
>     >
>
>



More information about the R-help mailing list