[R-sig-ME] A three-level GLMM with binomial link in R

Hedyeh Ahmadi hedyeh@h @end|ng |rom u@c@edu
Mon Mar 8 17:29:56 CET 2021 

Thank you for the toy example. This same example in my PC gives the following memory use output:

> mem
                     Function_Call                            Elapsed_Time_sec         Total_RAM_Used_MiB    Peak_RAM_Used_MiB
  m<-lmer(form,data=dd,REML=FALSE)             2.64                                     10.3                                       130.1

When I run peakRAM() for my actual data, it take one hour plus then my PC slows down, then I have to stop R to be able to use my PC.

Best,

Hedyeh Ahmadi, Ph.D.
Statistician
Keck School of Medicine
Department of Preventive Medicine
University of Southern California

Postdoctoral Scholar
Institute for Interdisciplinary Salivary Bioscience Research (IISBR)
University of California, Irvine

LinkedIn
www.linkedin.com/in/hedyeh-ahmadi<http://www.linkedin.com/in/hedyeh-ahmadi>
<http://www.linkedin.com/in/hedyeh-ahmadi><http://www.linkedin.com/in/hedyeh-ahmadi>




________________________________
From: R-sig-mixed-models <r-sig-mixed-models-bounces using r-project.org> on behalf of Ben Bolker <bbolker using gmail.com>
Sent: Friday, March 5, 2021 5:44 PM
To: r-sig-mixed-models using r-project.org <r-sig-mixed-models using r-project.org>
Subject: Re: [R-sig-ME] A three-level GLMM with binomial link in R

   Here's an example that conforms approximately to the structure of
your data: on my machine the peak RAM usage is 121 Mb, far short of your
2.3 Gb ...  so I still suspect there is something going on that we don't
know about/haven't guessed yet.

set.seed(101)
N <- 22945
ns <- 100
nf <- 100
nid <- 100
dd <- data.frame(X1=rnorm(N),
                  X2=rnorm(N),
                  X3=rnorm(N),
                  site=sample(ns, replace=TRUE, size=N),
                  family=sample(nf, replace=TRUE, size=N),
                  id=sample(nid, replace=TRUE, size=N))
form <- Y ~ 1 + X1 + X2 + X3 + (1|site/family/id)
dd$Y <- simulate(form[-2], newdata=dd, family=gaussian,
                  newparams=list(beta=rep(1,4),theta=rep(1,3),sigma=1))[[1]]
library(peakRAM)
mem <- peakRAM(
     m <- lmer(form, data=dd, REML=FALSE)
)

mem
                      Function_Call Elapsed_Time_sec Total_RAM_Used_MiB
1 m<-lmer(form,data=dd,REML=FALSE)            1.754               10.3
   Peak_RAM_Used_MiB
1             120.7


On 3/5/21 4:52 PM, Robert Long wrote:
> Perhaps because of the different ways they store objects internally while
> fitting the models.
>
> On Fri, 5 Mar 2021, 21:47 Hedyeh Ahmadi, <hedyehah using usc.edu> wrote:
>
>> Thank you for the reply Robert - If I am running out of memory in lmer(),
>> do you know why lme() runs just fine?
>>
>> Best,
>>
>> Hedyeh Ahmadi, Ph.D.
>> Statistician
>> Keck School of Medicine
>> Department of Preventive Medicine
>> University of Southern California
>>
>> Postdoctoral Scholar
>> Institute for Interdisciplinary Salivary Bioscience Research (IISBR)
>> University of California, Irvine
>>
>> LinkedIn
>> https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!7bcqr_mYJEwUBwSQ-hoyGUZPrR-Wz2RGDxVbFlFtbA0oveGsE7tg-2k70OXW4fU$
>> <https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!7bcqr_mYJEwUBwSQ-hoyGUZPrR-Wz2RGDxVbFlFtbA0oveGsE7tg-2k70OXW4fU$ >
>> <https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!7bcqr_mYJEwUBwSQ-hoyGUZPrR-Wz2RGDxVbFlFtbA0oveGsE7tg-2k70OXW4fU$ >
>>
>>
>>
>>
>> ------------------------------
>> *From:* Robert Long <longrob604 using gmail.com>
>> *Sent:* Friday, March 5, 2021 1:43 PM
>> *To:* Hedyeh Ahmadi <hedyehah using usc.edu>
>> *Cc:* Dexter Locke <dexter.locke using gmail.com>; R-mixed models mailing list <
>> r-sig-mixed-models using r-project.org>; Megan M. Herting <herting using usc.edu>;
>> Elisabeth Burnor <burnor using usc.edu>
>> *Subject:* Re: [R-sig-ME] A three-level GLMM with binomial link in R
>>
>> You've run out of memory. Try running it on a machine with much larger RAM.
>>
>> On Fri, 5 Mar 2021, 21:18 Hedyeh Ahmadi, <hedyehah using usc.edu> wrote:
>>
>> Hi - Thank you for the informative replies.
>>
>> I will try those other packages. If MASS::glmmPQL uses lme() then this
>> should work for me.
>>
>> My data structure is complex so it's hard to give reproducible example but
>> for example for two of my models I get the following errors in lmer() while
>> lme() runs smoothly.
>>
>> 1- Error: cannot allocate vector of size 2.3 Gb.
>>
>> 2- Error: couldn't evaluate grouping factor id:(family:site) within model
>> frame: try adding grouping factor to data frame explicitly if possible.
>> (note that the id variable works for simpler lmer() models so the variable
>> itself is not an issue)
>>
>> My model looks like the following nested structure:
>> lmer(Y ~ 1 + X1 + X2 + X3 + (1|site/family/id), data=dd, REML = FALSE)
>>
>> Best,
>>
>> Hedyeh Ahmadi, Ph.D.
>> Statistician
>> Keck School of Medicine
>> Department of Preventive Medicine
>> University of Southern California
>>
>> Postdoctoral Scholar
>> Institute for Interdisciplinary Salivary Bioscience Research (IISBR)
>> University of California, Irvine
>>
>> LinkedIn
>> https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!7bcqr_mYJEwUBwSQ-hoyGUZPrR-Wz2RGDxVbFlFtbA0oveGsE7tg-2k70OXW4fU$
>> <https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!5uCe7IsPHtgAoEp8Qsbn8bXOUBLGi7pVfXIjHwMUHVDbmnOSRCu7OHKGGuIvhag$>
>> <https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!7bcqr_mYJEwUBwSQ-hoyGUZPrR-Wz2RGDxVbFlFtbA0oveGsE7tg-2k70OXW4fU$
>> <https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!5uCe7IsPHtgAoEp8Qsbn8bXOUBLGi7pVfXIjHwMUHVDbmnOSRCu7OHKGGuIvhag$>
>>>
>> <https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!7bcqr_mYJEwUBwSQ-hoyGUZPrR-Wz2RGDxVbFlFtbA0oveGsE7tg-2k70OXW4fU$
>> <https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!5uCe7IsPHtgAoEp8Qsbn8bXOUBLGi7pVfXIjHwMUHVDbmnOSRCu7OHKGGuIvhag$>
>>> <https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!7bcqr_mYJEwUBwSQ-hoyGUZPrR-Wz2RGDxVbFlFtbA0oveGsE7tg-2k70OXW4fU$
>> <https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!5uCe7IsPHtgAoEp8Qsbn8bXOUBLGi7pVfXIjHwMUHVDbmnOSRCu7OHKGGuIvhag$>
>>>
>>
>>
>>
>>
>> ________________________________
>> From: Dexter Locke <dexter.locke using gmail.com>
>> Sent: Friday, March 5, 2021 1:01 PM
>> To: Hedyeh Ahmadi <hedyehah using usc.edu>
>> Cc: r-sig-mixed-models using r-project.org <r-sig-mixed-models using r-project.org>;
>> Megan M. Herting <herting using usc.edu>; Elisabeth Burnor <burnor using usc.edu>
>> Subject: Re: [R-sig-ME] A three-level GLMM with binomial link in R
>>
>> Hi Hedyeh,
>>
>> What is the problem you are having? Specifically, what is the estimation
>> issue with lmer and lme?
>>
>> Can you provide a reproducible example? Can you provide the complete
>> errors you are seeing?
>>
>> The current questions are vague, so it will be challenging for list
>> members to provide much guidance.
>>
>> -Dexter
>>
>>
>> On Fri, Mar 5, 2021 at 3:27 PM Hedyeh Ahmadi <hedyehah using usc.edu<mailto:
>> hedyehah using usc.edu>> wrote:
>> Hi All,
>> I was wondering what would be a powerful package in R to run GLMM with
>> logit link that can handle a data set with N=22945 and 3 nested random
>> intercepts. So far, I have tried glmer() from lme4 and it's giving me a lot
>> of estimation issues. Any other package I should try?
>>
>> I am asking for another package as I am having the same issue with lmer()
>> for similar LMM with continuous outcome, while lme() from nlme package runs
>> the models with no problem.
>>
>> Thank you in advance.
>>
>> Best,
>>
>> Hedyeh Ahmadi, Ph.D.
>> Statistician
>> Keck School of Medicine
>> Department of Preventive Medicine
>> University of Southern California
>>
>> Postdoctoral Scholar
>> Institute for Interdisciplinary Salivary Bioscience Research (IISBR)
>> University of California, Irvine
>>
>> LinkedIn
>> https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!7bcqr_mYJEwUBwSQ-hoyGUZPrR-Wz2RGDxVbFlFtbA0oveGsE7tg-2k70OXW4fU$
>> <https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!5uCe7IsPHtgAoEp8Qsbn8bXOUBLGi7pVfXIjHwMUHVDbmnOSRCu7OHKGGuIvhag$>
>> <
>> https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!8UfGozITdR3cNo929aSVEsbVNgbIul9f91q4hA3PAf0ywits0jIKVDAQdYkAXFg$
>>> <https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!7bcqr_mYJEwUBwSQ-hoyGUZPrR-Wz2RGDxVbFlFtbA0oveGsE7tg-2k70OXW4fU$
>> <https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!5uCe7IsPHtgAoEp8Qsbn8bXOUBLGi7pVfXIjHwMUHVDbmnOSRCu7OHKGGuIvhag$>
>> <
>> https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!8UfGozITdR3cNo929aSVEsbVNgbIul9f91q4hA3PAf0ywits0jIKVDAQdYkAXFg$
>>>>
>> <https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!7bcqr_mYJEwUBwSQ-hoyGUZPrR-Wz2RGDxVbFlFtbA0oveGsE7tg-2k70OXW4fU$
>> <https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!5uCe7IsPHtgAoEp8Qsbn8bXOUBLGi7pVfXIjHwMUHVDbmnOSRCu7OHKGGuIvhag$>
>> <
>> https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!8UfGozITdR3cNo929aSVEsbVNgbIul9f91q4hA3PAf0ywits0jIKVDAQdYkAXFg$
>>>> <https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!7bcqr_mYJEwUBwSQ-hoyGUZPrR-Wz2RGDxVbFlFtbA0oveGsE7tg-2k70OXW4fU$
>> <https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!5uCe7IsPHtgAoEp8Qsbn8bXOUBLGi7pVfXIjHwMUHVDbmnOSRCu7OHKGGuIvhag$>
>> <
>> https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!8UfGozITdR3cNo929aSVEsbVNgbIul9f91q4hA3PAf0ywits0jIKVDAQdYkAXFg$
>>>>
>>
>>
>>
>>
>>
>>          [[alternative HTML version deleted]]
>>
>> _______________________________________________
>> R-sig-mixed-models using r-project.org<mailto:R-sig-mixed-models using r-project.org>
>> mailing list
>> https://urldefense.com/v3/__https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models__;!!LIr3w8kk_Xxm!7bcqr_mYJEwUBwSQ-hoyGUZPrR-Wz2RGDxVbFlFtbA0oveGsE7tg-2k7pcSE0ic$
>> <https://urldefense.com/v3/__https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models__;!!LIr3w8kk_Xxm!5uCe7IsPHtgAoEp8Qsbn8bXOUBLGi7pVfXIjHwMUHVDbmnOSRCu7OHKG8rJcsmY$>
>> <
>> https://urldefense.com/v3/__https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models__;!!LIr3w8kk_Xxm!8UfGozITdR3cNo929aSVEsbVNgbIul9f91q4hA3PAf0ywits0jIKVDAQM6wKtNs$
>>>
>>
>>          [[alternative HTML version deleted]]
>>
>> _______________________________________________
>> R-sig-mixed-models using r-project.org mailing list
>> https://urldefense.com/v3/__https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models__;!!LIr3w8kk_Xxm!7bcqr_mYJEwUBwSQ-hoyGUZPrR-Wz2RGDxVbFlFtbA0oveGsE7tg-2k7pcSE0ic$
>> <https://urldefense.com/v3/__https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models__;!!LIr3w8kk_Xxm!5uCe7IsPHtgAoEp8Qsbn8bXOUBLGi7pVfXIjHwMUHVDbmnOSRCu7OHKG8rJcsmY$>
>>
>>
>
>        [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-mixed-models using r-project.org mailing list
> https://urldefense.com/v3/__https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models__;!!LIr3w8kk_Xxm!7bcqr_mYJEwUBwSQ-hoyGUZPrR-Wz2RGDxVbFlFtbA0oveGsE7tg-2k7pcSE0ic$
>

_______________________________________________
R-sig-mixed-models using r-project.org mailing list
https://urldefense.com/v3/__https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models__;!!LIr3w8kk_Xxm!7bcqr_mYJEwUBwSQ-hoyGUZPrR-Wz2RGDxVbFlFtbA0oveGsE7tg-2k7pcSE0ic$

	[[alternative HTML version deleted]]

More information about the R-sig-mixed-models mailing list