[R-sig-ME] How to use all the cores while running glmer on a piecewise exponential survival with

[Jonathan Judge]

Have you given INLA a try? That’s a lot of individuals, but as long as they are members of one “group” it might only cost you one hyperparameter, which seems to be the primary limitation for that approach. 

Jonathan

Sent from my iPhone

> On Aug 23, 2018, at 5:47 PM, Adam Mills-Campisi <adammillscampisi using gmail.com> wrote:
> 
> Thanks! We are looking into our options. The MixedModels package in Julia
> benchmarks at about 2 orders of magnitude faster than R on a small dataset;
> however, I would think a lot of that is just overhead from R. On a model of
> this size, the computational time should converge because everyone is using
> the same BLAS libraries. It might be worth further investigation if timing
> remains an issue.
> 
> On Thu, Aug 23, 2018 at 2:27 PM D. Rizopoulos <d.rizopoulos using erasmusmc.nl>
> wrote:
> 
>> As suggested, an approach could be to split the original big sample in
>> manageable pieces, do the analysis in each, and then combine the results.
>> 
>> Geert Molenberghs, Geert Verbeke and colleagues have worked on this; a
>> relevant recent papers seems to be:
>> https://lirias2repo.kuleuven.be/bitstream/id/470902/
>> 
>> I hope it helps.
>> 
>> Best,
>> Dimitris
>> 
>> 
>> From: Ben Bolker <bbolker using gmail.com<mailto:bbolker using gmail.com>>
>> Date: Thursday, 23 Aug 2018, 10:29 PM
>> To: r-sig-mixed-models using r-project.org <r-sig-mixed-models using r-project.org
>> <mailto:r-sig-mixed-models using r-project.org>>
>> Subject: Re: [R-sig-ME] How to use all the cores while running glmer on a
>> piecewise exponential survival with
>> 
>> 
>>  Are the frequentist methods *not* faster?  I'd be pretty surprised,
>> unless some you're hitting some terrible memory bottleneck or something.
>> 
>> 
>>> On 2018-08-23 03:30 PM, Adam Mills-Campisi wrote:
>>> We originally tried to use stan to estimate the model, we were getting
>>> performance issues. I assumed that the frequentist approaches would be
>>> faster.
>>> 
>>>> On Thu, Aug 23, 2018 at 12:28 PM Doran, Harold <HDoran using air.org> wrote:
>>>> 
>>>> No. You can change to an improved BLAS or I have found the Microsoft R
>> has
>>>> some built in multithreading that is fast for matrix algebra and it
>> passes
>>>> that benefit to lmer. From some experience, you can improve
>> computational
>>>> time of an lmer model with Microsoft R
>>>> 
>>>> -----Original Message-----
>>>> From: R-sig-mixed-models <r-sig-mixed-models-bounces using r-project.org> On
>>>> Behalf Of Adam Mills-Campisi
>>>> Sent: Thursday, August 23, 2018 3:18 PM
>>>> To: r-sig-mixed-models using r-project.org
>>>> Subject: [R-sig-ME] How to use all the cores while running glmer on a
>>>> piecewise exponential survival with
>>>> 
>>>> I am estimating a piecewise exponential, mixed-effects, survival model
>>>> with recurrent events. Each individual in the dataset gets an individual
>>>> interpret (where using a PWP approach). Our full dataset has 10 million
>>>> individuals, with 180 million events. I am not sure that there is any
>>>> framework which can accommodate data at that size, so we are going to
>>>> sample. Our final sample size largely depends on how quickly we can
>>>> estimate the model, which brings me to my question: Is there a way to
>>>> mutli-thread/core the model? I tried to find some kind of instruction on
>>>> the web and the best lead I could find was a reference to this list
>> serve.
>>>> Any help would be greatly appreciated.
>>>> 
>>>>        [[alternative HTML version deleted]]
>>>> 
>>>> _______________________________________________
>>>> R-sig-mixed-models using r-project.org mailing list
>>>> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>>>> 
>>>> 
>>> 
>>>       [[alternative HTML version deleted]]
>>> 
>>> _______________________________________________
>>> R-sig-mixed-models using r-project.org mailing list
>>> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>>> 
>> 
>> _______________________________________________
>> R-sig-mixed-models using r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>> 
>>        [[alternative HTML version deleted]]
>> 
>> _______________________________________________
>> R-sig-mixed-models using r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>> 
> 
>    [[alternative HTML version deleted]]
> 
> _______________________________________________
> R-sig-mixed-models using r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models