[R-sig-ME] How to use all the cores while running glmer on a piecewise exponential survival with

[Manuel Ramon]

Not sure if this can be useful:
bigglm: faster-generalised-linear-models-in-largeish-data
<https://notstatschat.rbind.io/2018/03/05/faster-generalised-linear-models-in-largeish-data/>

Manuel


On Fri, Aug 24, 2018 at 12:47 AM Adam Mills-Campisi <
adammillscampisi using gmail.com> wrote:

> Thanks! We are looking into our options. The MixedModels package in Julia
> benchmarks at about 2 orders of magnitude faster than R on a small dataset;
> however, I would think a lot of that is just overhead from R. On a model of
> this size, the computational time should converge because everyone is using
> the same BLAS libraries. It might be worth further investigation if timing
> remains an issue.
>
> On Thu, Aug 23, 2018 at 2:27 PM D. Rizopoulos <d.rizopoulos using erasmusmc.nl>
> wrote:
>
> > As suggested, an approach could be to split the original big sample in
> > manageable pieces, do the analysis in each, and then combine the results.
> >
> > Geert Molenberghs, Geert Verbeke and colleagues have worked on this; a
> > relevant recent papers seems to be:
> > https://lirias2repo.kuleuven.be/bitstream/id/470902/
> >
> > I hope it helps.
> >
> > Best,
> > Dimitris
> >
> >
> > From: Ben Bolker <bbolker using gmail.com<mailto:bbolker using gmail.com>>
> > Date: Thursday, 23 Aug 2018, 10:29 PM
> > To: r-sig-mixed-models using r-project.org <r-sig-mixed-models using r-project.org
> > <mailto:r-sig-mixed-models using r-project.org>>
> > Subject: Re: [R-sig-ME] How to use all the cores while running glmer on a
> > piecewise exponential survival with
> >
> >
> >   Are the frequentist methods *not* faster?  I'd be pretty surprised,
> > unless some you're hitting some terrible memory bottleneck or something.
> >
> >
> > On 2018-08-23 03:30 PM, Adam Mills-Campisi wrote:
> > > We originally tried to use stan to estimate the model, we were getting
> > > performance issues. I assumed that the frequentist approaches would be
> > > faster.
> > >
> > > On Thu, Aug 23, 2018 at 12:28 PM Doran, Harold <HDoran using air.org> wrote:
> > >
> > >> No. You can change to an improved BLAS or I have found the Microsoft R
> > has
> > >> some built in multithreading that is fast for matrix algebra and it
> > passes
> > >> that benefit to lmer. From some experience, you can improve
> > computational
> > >> time of an lmer model with Microsoft R
> > >>
> > >> -----Original Message-----
> > >> From: R-sig-mixed-models <r-sig-mixed-models-bounces using r-project.org>
> On
> > >> Behalf Of Adam Mills-Campisi
> > >> Sent: Thursday, August 23, 2018 3:18 PM
> > >> To: r-sig-mixed-models using r-project.org
> > >> Subject: [R-sig-ME] How to use all the cores while running glmer on a
> > >> piecewise exponential survival with
> > >>
> > >> I am estimating a piecewise exponential, mixed-effects, survival model
> > >> with recurrent events. Each individual in the dataset gets an
> individual
> > >> interpret (where using a PWP approach). Our full dataset has 10
> million
> > >> individuals, with 180 million events. I am not sure that there is any
> > >> framework which can accommodate data at that size, so we are going to
> > >> sample. Our final sample size largely depends on how quickly we can
> > >> estimate the model, which brings me to my question: Is there a way to
> > >> mutli-thread/core the model? I tried to find some kind of instruction
> on
> > >> the web and the best lead I could find was a reference to this list
> > serve.
> > >> Any help would be greatly appreciated.
> > >>
> > >>         [[alternative HTML version deleted]]
> > >>
> > >> _______________________________________________
> > >> R-sig-mixed-models using r-project.org mailing list
> > >> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
> > >>
> > >>
> > >
> > >        [[alternative HTML version deleted]]
> > >
> > > _______________________________________________
> > > R-sig-mixed-models using r-project.org mailing list
> > > https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
> > >
> >
> > _______________________________________________
> > R-sig-mixed-models using r-project.org mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
> >
> >         [[alternative HTML version deleted]]
> >
> > _______________________________________________
> > R-sig-mixed-models using r-project.org mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
> >
>
>         [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-mixed-models using r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>

	[[alternative HTML version deleted]]