[R-sig-ME] How to use all the cores while running glmer on a piecewise exponential survival with

[Adam Mills-Campisi]

We're going to look into INLA and biglm. As long as we are considering
alternatives, does anyone know of a really know of a really fast
implementation of other survival models? If this is off topic, there is no
need to reply. Our only constraint is that we need to control for the
repeated events (we have an uneven number of responses per individual).
We've looked at coxme, but we are getting the same performance bottlenecks
(we can only run single core and there are exponential time costs when we
increase the number of observations).

On Fri, Aug 24, 2018 at 7:36 AM D. Rizopoulos <d.rizopoulos using erasmusmc.nl>
wrote:

> I think the idea of how to efficiently implement Laplace and adaptive
> Gauss-Hermite integration in nested random effects designs for GLMMS is
> described in https://dx.doi.org/10.1198/106186006X96962
>
>
>
> -----Original Message-----
> From: R-sig-mixed-models <r-sig-mixed-models-bounces using r-project.org> On
> Behalf Of Doran, Harold
> Sent: Friday, August 24, 2018 4:21 PM
> To: Ben Bolker <bbolker using gmail.com>; r-sig-mixed-models using r-project.org
> Subject: Re: [R-sig-ME] How to use all the cores while running glmer on a
> piecewise exponential survival with
>
> @ben, I like that idea. I've done that with some recent work that reduces
> some dimensionality in the integration and makes the problem easier to
> compute. I just don't know the current problem well enough to know if that
> is feasible here.
>
> But, it's certainly an idea to explore.
>
>
>
> -----Original Message-----
> From: R-sig-mixed-models <r-sig-mixed-models-bounces using r-project.org> On
> Behalf Of Ben Bolker
> Sent: Thursday, August 23, 2018 4:09 PM
> To: r-sig-mixed-models using r-project.org
> Subject: Re: [R-sig-ME] How to use all the cores while running glmer on a
> piecewise exponential survival with
>
>
>   Harold, what do you think of my suggestion (partition problem into
> multiple conditionally independent subsets, evaluate separate deviances on
> workers, run top-level optimization on a central 'master' processor)?
>  Am I missing something (except that some problems can't easily be
> partitioned that way?)
>
>   FWIW I think Doug Bates has pointed out in the past that for simple
> (e.g. nested, not crossed) designs, the whole problem can be reformulated
> in a more efficient way (of course I can't dig up that e-mail ...).  lme4's
> strength is that it can handle the complex cases, and so far no-one has had
> the time/energy/interest/capability of implementing any of Doug's "special
> case" strategies, at least in lme4
> -- may be done elsewhere in R, or in Doug's MixedModels.jl ...
>
>   cheers
>    Ben Bolker
>
>
> On 2018-08-23 03:32 PM, Doran, Harold wrote:
> > Running the model on multiple cores won’t work because lmer isn’t
> written that way. One idea I’ve toyed with is start with a small-ish sample
> and get results. Plug those in as starting values to your next run which
> uses larger sample, but takes fewer steps because you’re closer to the max.
> Repeat until the difference in the param estimates from prior run is less
> than some tolerance.
> >
> >
> > From: Adam Mills-Campisi <adammillscampisi using gmail.com>
> > Sent: Thursday, August 23, 2018 3:25 PM
> > To: Doran, Harold <HDoran using air.org>
> > Cc: r-sig-mixed-models using r-project.org
> > Subject: Re: [R-sig-ME] How to use all the cores while running glmer
> > on a piecewise exponential survival with
> >
> > That's the plan, the real question is how big should the samples be. The
> faster we can estimate the model, the bigger the sample can be. If I can
> run the model on multiple cores that would significantly increase the
> sample size.
> >
> > On Thu, Aug 23, 2018 at 12:23 PM Doran, Harold <HDoran using air.org<mailto:
> HDoran using air.org>> wrote:
> > One idea, though, is you can take samples from your very large data set
> and estimate models on the samples very quickly.
> >
> > -----Original Message-----
> > From: R-sig-mixed-models
> > <r-sig-mixed-models-bounces using r-project.org<mailto:r-sig-mixed-models-bo
> > unces using r-project.org>> On Behalf Of Adam Mills-Campisi
> > Sent: Thursday, August 23, 2018 3:18 PM
> > To:
> > r-sig-mixed-models using r-project.org<mailto:r-sig-mixed-models using r-project.o
> > rg>
> > Subject: [R-sig-ME] How to use all the cores while running glmer on a
> > piecewise exponential survival with
> >
> > I am estimating a piecewise exponential, mixed-effects, survival model
> with recurrent events. Each individual in the dataset gets an individual
> interpret (where using a PWP approach). Our full dataset has 10 million
> individuals, with 180 million events. I am not sure that there is any
> framework which can accommodate data at that size, so we are going to
> sample. Our final sample size largely depends on how quickly we can
> estimate the model, which brings me to my question: Is there a way to
> mutli-thread/core the model? I tried to find some kind of instruction on
> the web and the best lead I could find was a reference to this list serve.
> > Any help would be greatly appreciated.
> >
> >         [[alternative HTML version deleted]]
> >
> > _______________________________________________
> > R-sig-mixed-models using r-project.org<mailto:R-sig-mixed-models using r-project.o
> > rg> mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
> >
> >       [[alternative HTML version deleted]]
> >
> > _______________________________________________
> > R-sig-mixed-models using r-project.org mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
> >
>
> _______________________________________________
> R-sig-mixed-models using r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>
> _______________________________________________
> R-sig-mixed-models using r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
> _______________________________________________
> R-sig-mixed-models using r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>

	[[alternative HTML version deleted]]