[R-sig-ME] large dataset - lmer2 vector size specified is too large

[Gregor Gorjanc]

Hi,

Douglas Bates wrote:
> On 3/2/07, Gregor Gorjanc <gregor.gorjanc at bfro.uni-lj.si> wrote:
>> Douglas Bates <bates at ...> writes:
>> ...
>> > but even that formula means that you are estimating 448 fixed effects
>> > for which the model matrix is of size 448 * 43680 * 8 bytes (about 150
>> > MB).  In addition you are attempting to estimate
>>
>> Not directly related. Is it really necessary to build model matrix
>> i.e. X.
> 
> Necessary? For linear mixed models no - for generalized linear mixed
> models yes (think of how the IRLS algorithm works).

hmm, I will need to learn IRLS first :(

>> Building X'X directly would be more efficient i.e. only 448 * 448 * 8
>> bytes.
...
> Generating A as a sparse matrix directly from the data or directly
> from a model.frame object would result in considerable savings in
> memory usage but doing so is far from trivial.  Most useRs are not
> aware (nor need they be aware) of the complexity of the operations
...

You are right! I am far from being aware how complex this things are and
I am very grateful to you for providing us with such a great software
for nothing. However, I know that in animal breeding (AB) settings A can
really be huge, mainly due to hugeness of Z matrix. And I know that
some packages used in AB area setup MME directly from the data.
Unfortunately, I am also not up to the task to tell the details. I am
providing list of packages that are used in AB and whose authors might
be as generous as you to show the light. Additionally, these packages
can also deal with many response variables i.e. multi-trait/multivariate
models with different effects so their implementations are surely
general. Though, they are mainly for linear mixed models, but some also
cope with generalized mixed model.

Thank you again!

ASReml
http://www.vsni.co.uk/products/asreml/

BGF90
http://nce.ads.uga.edu/%7Eignacy/numpub/

DMU
http://www.dmu.agrsci.dk/
http://www.wcgalp8.org.br/wcgalp8/articles/paper/27_302-413.pdf

PEST/VCE
http://w3.tzv.fal.de/%7Eeg/
http://vce.tzv.fal.de/index.pl

MATVEC
http://statistics.unl.edu/faculty/steve/software/matvec/

WOMBAT
http://agbu.une.edu.au/~kmeyer/wombat.html

-- 
Lep pozdrav / With regards,
    Gregor Gorjanc
----------------------------------------------------------------------
University of Ljubljana     PhD student
Biotechnical Faculty        www: http://www.bfro.uni-lj.si/MR/ggorjan
Zootechnical Department     blog: http://ggorjan.blogspot.com
Groblje 3                   mail: gregor.gorjanc <at> bfro.uni-lj.si
SI-1230 Domzale             fax: +386 (0)1 72 17 888
Slovenia, Europe            tel: +386 (0)1 72 17 861