[R-sig-ME] A three-level GLMM with binomial link in R

Robert Long |ongrob604 @end|ng |rom gm@||@com
Mon Mar 8 20:00:57 CET 2021 

How many sites, families and Id's do you have?

Please also give us the output of str(dd)

On Mon, 8 Mar 2021, 16:30 Hedyeh Ahmadi, <hedyehah using usc.edu> wrote:

> Thank you for the toy example. This same example in my PC gives the
> following memory use output:
>
> > mem
>                      Function_Call
> Elapsed_Time_sec         Total_RAM_Used_MiB    Peak_RAM_Used_MiB
>   m<-lmer(form,data=dd,REML=FALSE)             2.64
>              10.3                                       130.1
>
> When I run peakRAM() for my actual data, it take one hour plus then my PC
> slows down, then I have to stop R to be able to use my PC.
>
> Best,
>
> Hedyeh Ahmadi, Ph.D.
> Statistician
> Keck School of Medicine
> Department of Preventive Medicine
> University of Southern California
>
> Postdoctoral Scholar
> Institute for Interdisciplinary Salivary Bioscience Research (IISBR)
> University of California, Irvine
>
> LinkedIn
> www.linkedin.com/in/hedyeh-ahmadi<http://www.linkedin.com/in/hedyeh-ahmadi
> >
> <http://www.linkedin.com/in/hedyeh-ahmadi><
> http://www.linkedin.com/in/hedyeh-ahmadi>
>
>
>
>
> ________________________________
> From: R-sig-mixed-models <r-sig-mixed-models-bounces using r-project.org> on
> behalf of Ben Bolker <bbolker using gmail.com>
> Sent: Friday, March 5, 2021 5:44 PM
> To: r-sig-mixed-models using r-project.org <r-sig-mixed-models using r-project.org>
> Subject: Re: [R-sig-ME] A three-level GLMM with binomial link in R
>
>    Here's an example that conforms approximately to the structure of
> your data: on my machine the peak RAM usage is 121 Mb, far short of your
> 2.3 Gb ...  so I still suspect there is something going on that we don't
> know about/haven't guessed yet.
>
> set.seed(101)
> N <- 22945
> ns <- 100
> nf <- 100
> nid <- 100
> dd <- data.frame(X1=rnorm(N),
>                   X2=rnorm(N),
>                   X3=rnorm(N),
>                   site=sample(ns, replace=TRUE, size=N),
>                   family=sample(nf, replace=TRUE, size=N),
>                   id=sample(nid, replace=TRUE, size=N))
> form <- Y ~ 1 + X1 + X2 + X3 + (1|site/family/id)
> dd$Y <- simulate(form[-2], newdata=dd, family=gaussian,
>
> newparams=list(beta=rep(1,4),theta=rep(1,3),sigma=1))[[1]]
> library(peakRAM)
> mem <- peakRAM(
>      m <- lmer(form, data=dd, REML=FALSE)
> )
>
> mem
>                       Function_Call Elapsed_Time_sec Total_RAM_Used_MiB
> 1 m<-lmer(form,data=dd,REML=FALSE)            1.754               10.3
>    Peak_RAM_Used_MiB
> 1             120.7
>
>
> On 3/5/21 4:52 PM, Robert Long wrote:
> > Perhaps because of the different ways they store objects internally while
> > fitting the models.
> >
> > On Fri, 5 Mar 2021, 21:47 Hedyeh Ahmadi, <hedyehah using usc.edu> wrote:
> >
> >> Thank you for the reply Robert - If I am running out of memory in
> lmer(),
> >> do you know why lme() runs just fine?
> >>
> >> Best,
> >>
> >> Hedyeh Ahmadi, Ph.D.
> >> Statistician
> >> Keck School of Medicine
> >> Department of Preventive Medicine
> >> University of Southern California
> >>
> >> Postdoctoral Scholar
> >> Institute for Interdisciplinary Salivary Bioscience Research (IISBR)
> >> University of California, Irvine
> >>
> >> LinkedIn
> >>
> https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!7bcqr_mYJEwUBwSQ-hoyGUZPrR-Wz2RGDxVbFlFtbA0oveGsE7tg-2k70OXW4fU$
> >> <
> https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!7bcqr_mYJEwUBwSQ-hoyGUZPrR-Wz2RGDxVbFlFtbA0oveGsE7tg-2k70OXW4fU$
> >
> >> <
> https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!7bcqr_mYJEwUBwSQ-hoyGUZPrR-Wz2RGDxVbFlFtbA0oveGsE7tg-2k70OXW4fU$
> >
> >>
> >>
> >>
> >>
> >> ------------------------------
> >> *From:* Robert Long <longrob604 using gmail.com>
> >> *Sent:* Friday, March 5, 2021 1:43 PM
> >> *To:* Hedyeh Ahmadi <hedyehah using usc.edu>
> >> *Cc:* Dexter Locke <dexter.locke using gmail.com>; R-mixed models mailing
> list <
> >> r-sig-mixed-models using r-project.org>; Megan M. Herting <herting using usc.edu>;
> >> Elisabeth Burnor <burnor using usc.edu>
> >> *Subject:* Re: [R-sig-ME] A three-level GLMM with binomial link in R
> >>
> >> You've run out of memory. Try running it on a machine with much larger
> RAM.
> >>
> >> On Fri, 5 Mar 2021, 21:18 Hedyeh Ahmadi, <hedyehah using usc.edu> wrote:
> >>
> >> Hi - Thank you for the informative replies.
> >>
> >> I will try those other packages. If MASS::glmmPQL uses lme() then this
> >> should work for me.
> >>
> >> My data structure is complex so it's hard to give reproducible example
> but
> >> for example for two of my models I get the following errors in lmer()
> while
> >> lme() runs smoothly.
> >>
> >> 1- Error: cannot allocate vector of size 2.3 Gb.
> >>
> >> 2- Error: couldn't evaluate grouping factor id:(family:site) within
> model
> >> frame: try adding grouping factor to data frame explicitly if possible.
> >> (note that the id variable works for simpler lmer() models so the
> variable
> >> itself is not an issue)
> >>
> >> My model looks like the following nested structure:
> >> lmer(Y ~ 1 + X1 + X2 + X3 + (1|site/family/id), data=dd, REML = FALSE)
> >>
> >> Best,
> >>
> >> Hedyeh Ahmadi, Ph.D.
> >> Statistician
> >> Keck School of Medicine
> >> Department of Preventive Medicine
> >> University of Southern California
> >>
> >> Postdoctoral Scholar
> >> Institute for Interdisciplinary Salivary Bioscience Research (IISBR)
> >> University of California, Irvine
> >>
> >> LinkedIn
> >>
> https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!7bcqr_mYJEwUBwSQ-hoyGUZPrR-Wz2RGDxVbFlFtbA0oveGsE7tg-2k70OXW4fU$
> >> <
> https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!5uCe7IsPHtgAoEp8Qsbn8bXOUBLGi7pVfXIjHwMUHVDbmnOSRCu7OHKGGuIvhag$
> >
> >> <
> https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!7bcqr_mYJEwUBwSQ-hoyGUZPrR-Wz2RGDxVbFlFtbA0oveGsE7tg-2k70OXW4fU$
> >> <
> https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!5uCe7IsPHtgAoEp8Qsbn8bXOUBLGi7pVfXIjHwMUHVDbmnOSRCu7OHKGGuIvhag$
> >
> >>>
> >> <
> https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!7bcqr_mYJEwUBwSQ-hoyGUZPrR-Wz2RGDxVbFlFtbA0oveGsE7tg-2k70OXW4fU$
> >> <
> https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!5uCe7IsPHtgAoEp8Qsbn8bXOUBLGi7pVfXIjHwMUHVDbmnOSRCu7OHKGGuIvhag$
> >
> >>> <
> https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!7bcqr_mYJEwUBwSQ-hoyGUZPrR-Wz2RGDxVbFlFtbA0oveGsE7tg-2k70OXW4fU$
> >> <
> https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!5uCe7IsPHtgAoEp8Qsbn8bXOUBLGi7pVfXIjHwMUHVDbmnOSRCu7OHKGGuIvhag$
> >
> >>>
> >>
> >>
> >>
> >>
> >> ________________________________
> >> From: Dexter Locke <dexter.locke using gmail.com>
> >> Sent: Friday, March 5, 2021 1:01 PM
> >> To: Hedyeh Ahmadi <hedyehah using usc.edu>
> >> Cc: r-sig-mixed-models using r-project.org <r-sig-mixed-models using r-project.org
> >;
> >> Megan M. Herting <herting using usc.edu>; Elisabeth Burnor <burnor using usc.edu>
> >> Subject: Re: [R-sig-ME] A three-level GLMM with binomial link in R
> >>
> >> Hi Hedyeh,
> >>
> >> What is the problem you are having? Specifically, what is the estimation
> >> issue with lmer and lme?
> >>
> >> Can you provide a reproducible example? Can you provide the complete
> >> errors you are seeing?
> >>
> >> The current questions are vague, so it will be challenging for list
> >> members to provide much guidance.
> >>
> >> -Dexter
> >>
> >>
> >> On Fri, Mar 5, 2021 at 3:27 PM Hedyeh Ahmadi <hedyehah using usc.edu<mailto:
> >> hedyehah using usc.edu>> wrote:
> >> Hi All,
> >> I was wondering what would be a powerful package in R to run GLMM with
> >> logit link that can handle a data set with N=22945 and 3 nested random
> >> intercepts. So far, I have tried glmer() from lme4 and it's giving me a
> lot
> >> of estimation issues. Any other package I should try?
> >>
> >> I am asking for another package as I am having the same issue with
> lmer()
> >> for similar LMM with continuous outcome, while lme() from nlme package
> runs
> >> the models with no problem.
> >>
> >> Thank you in advance.
> >>
> >> Best,
> >>
> >> Hedyeh Ahmadi, Ph.D.
> >> Statistician
> >> Keck School of Medicine
> >> Department of Preventive Medicine
> >> University of Southern California
> >>
> >> Postdoctoral Scholar
> >> Institute for Interdisciplinary Salivary Bioscience Research (IISBR)
> >> University of California, Irvine
> >>
> >> LinkedIn
> >>
> https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!7bcqr_mYJEwUBwSQ-hoyGUZPrR-Wz2RGDxVbFlFtbA0oveGsE7tg-2k70OXW4fU$
> >> <
> https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!5uCe7IsPHtgAoEp8Qsbn8bXOUBLGi7pVfXIjHwMUHVDbmnOSRCu7OHKGGuIvhag$
> >
> >> <
> >>
> https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!8UfGozITdR3cNo929aSVEsbVNgbIul9f91q4hA3PAf0ywits0jIKVDAQdYkAXFg$
> >>> <
> https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!7bcqr_mYJEwUBwSQ-hoyGUZPrR-Wz2RGDxVbFlFtbA0oveGsE7tg-2k70OXW4fU$
> >> <
> https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!5uCe7IsPHtgAoEp8Qsbn8bXOUBLGi7pVfXIjHwMUHVDbmnOSRCu7OHKGGuIvhag$
> >
> >> <
> >>
> https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!8UfGozITdR3cNo929aSVEsbVNgbIul9f91q4hA3PAf0ywits0jIKVDAQdYkAXFg$
> >>>>
> >> <
> https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!7bcqr_mYJEwUBwSQ-hoyGUZPrR-Wz2RGDxVbFlFtbA0oveGsE7tg-2k70OXW4fU$
> >> <
> https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!5uCe7IsPHtgAoEp8Qsbn8bXOUBLGi7pVfXIjHwMUHVDbmnOSRCu7OHKGGuIvhag$
> >
> >> <
> >>
> https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!8UfGozITdR3cNo929aSVEsbVNgbIul9f91q4hA3PAf0ywits0jIKVDAQdYkAXFg$
> >>>> <
> https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!7bcqr_mYJEwUBwSQ-hoyGUZPrR-Wz2RGDxVbFlFtbA0oveGsE7tg-2k70OXW4fU$
> >> <
> https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!5uCe7IsPHtgAoEp8Qsbn8bXOUBLGi7pVfXIjHwMUHVDbmnOSRCu7OHKGGuIvhag$
> >
> >> <
> >>
> https://urldefense.com/v3/__http://www.linkedin.com/in/hedyeh-ahmadi__;!!LIr3w8kk_Xxm!8UfGozITdR3cNo929aSVEsbVNgbIul9f91q4hA3PAf0ywits0jIKVDAQdYkAXFg$
> >>>>
> >>
> >>
> >>
> >>
> >>
> >>          [[alternative HTML version deleted]]
> >>
> >> _______________________________________________
> >> R-sig-mixed-models using r-project.org<mailto:
> R-sig-mixed-models using r-project.org>
> >> mailing list
> >>
> https://urldefense.com/v3/__https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models__;!!LIr3w8kk_Xxm!7bcqr_mYJEwUBwSQ-hoyGUZPrR-Wz2RGDxVbFlFtbA0oveGsE7tg-2k7pcSE0ic$
> >> <
> https://urldefense.com/v3/__https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models__;!!LIr3w8kk_Xxm!5uCe7IsPHtgAoEp8Qsbn8bXOUBLGi7pVfXIjHwMUHVDbmnOSRCu7OHKG8rJcsmY$
> >
> >> <
> >>
> https://urldefense.com/v3/__https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models__;!!LIr3w8kk_Xxm!8UfGozITdR3cNo929aSVEsbVNgbIul9f91q4hA3PAf0ywits0jIKVDAQM6wKtNs$
> >>>
> >>
> >>          [[alternative HTML version deleted]]
> >>
> >> _______________________________________________
> >> R-sig-mixed-models using r-project.org mailing list
> >>
> https://urldefense.com/v3/__https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models__;!!LIr3w8kk_Xxm!7bcqr_mYJEwUBwSQ-hoyGUZPrR-Wz2RGDxVbFlFtbA0oveGsE7tg-2k7pcSE0ic$
> >> <
> https://urldefense.com/v3/__https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models__;!!LIr3w8kk_Xxm!5uCe7IsPHtgAoEp8Qsbn8bXOUBLGi7pVfXIjHwMUHVDbmnOSRCu7OHKG8rJcsmY$
> >
> >>
> >>
> >
> >        [[alternative HTML version deleted]]
> >
> > _______________________________________________
> > R-sig-mixed-models using r-project.org mailing list
> >
> https://urldefense.com/v3/__https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models__;!!LIr3w8kk_Xxm!7bcqr_mYJEwUBwSQ-hoyGUZPrR-Wz2RGDxVbFlFtbA0oveGsE7tg-2k7pcSE0ic$
> >
>
> _______________________________________________
> R-sig-mixed-models using r-project.org mailing list
>
> https://urldefense.com/v3/__https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models__;!!LIr3w8kk_Xxm!7bcqr_mYJEwUBwSQ-hoyGUZPrR-Wz2RGDxVbFlFtbA0oveGsE7tg-2k7pcSE0ic$
>
>         [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-mixed-models using r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>

	[[alternative HTML version deleted]]

More information about the R-sig-mixed-models mailing list