[R-sig-ME] spaMM::fitme() - a glmm for longitudinal data that accounts for spatial autocorrelation

Thierry Onkelinx th|erry@onke||nx @end|ng |rom |nbo@be
Fri Jul 17 15:20:34 CEST 2020


Dear François,

Point taken about the mesh size. Halving the cutoff to 0.5 increased the
number of points on the mesh to 31k nodes. This increases the runtime to 65
min on my laptop.
https://drive.google.com/drive/folders/1Ocq88Yq9u_lM-loayRQlMyBS2HLy_Tio

I'm curious what you mean by "Matern" if it isn't similar to the spde model
of INLA.

Best regards,

ir. Thierry Onkelinx
Statisticus / Statistician

Vlaamse Overheid / Government of Flanders
INSTITUUT VOOR NATUUR- EN BOSONDERZOEK / RESEARCH INSTITUTE FOR NATURE AND
FOREST
Team Biometrie & Kwaliteitszorg / Team Biometrics & Quality Assurance
thierry.onkelinx using inbo.be
Havenlaan 88 bus 73, 1000 Brussel
www.inbo.be

///////////////////////////////////////////////////////////////////////////////////////////
To call in the statistician after the experiment is done may be no more
than asking him to perform a post-mortem examination: he may be able to say
what the experiment died of. ~ Sir Ronald Aylmer Fisher
The plural of anecdote is not data. ~ Roger Brinner
The combination of some data and an aching desire for an answer does not
ensure that a reasonable answer can be extracted from a given body of data.
~ John Tukey
///////////////////////////////////////////////////////////////////////////////////////////

<https://www.inbo.be>


Op wo 15 jul. 2020 om 16:01 schreef Francois Rousset <
francois.rousset using umontpellier.fr>:

> Dear Thierry,
>
> thanks. So (expectedly) this is a different issue. spaMM can fit some
> correlation models described by objects produced by
> INLA::inla.spde2.matern() and then, in my past experiments, the computation
> times were close to those of INLA, and the memory requirements were much
> smaller than what I wrote previously where this is not what I meant by
> "Matern".
>
> Beyond general features that contribute to these computational differences
> (the use of sparse matrix methods, and to a lesser extent the constraint on
> the smoothness parameter of the approximated Matern model), the 'cutoff'
> argument in your call to inla.mesh.2d() appears important to reduce the
> number  of locations actually considered, in the most costly computations,
> below the number of locations in the data (to 8804 rather than 30K, if I
> get it right), and this would also allow a faster fit by spaMM when called
> on the resulting inla.spde2 object.
>
> Best,
>
> F.
> Le 15/07/2020 à 12:50, Thierry Onkelinx a écrit :
>
> Dear François,
>
> Here you go:
> https://drive.google.com/drive/folders/1Ocq88Yq9u_lM-loayRQlMyBS2HLy_Tio
> Almost 30K locations. Fit in little over 7 min on my laptop with 16 GB RAM.
>
> Best regards,
>
> ir. Thierry Onkelinx
> Statisticus / Statistician
>
> Vlaamse Overheid / Government of Flanders
> INSTITUUT VOOR NATUUR- EN BOSONDERZOEK / RESEARCH INSTITUTE FOR NATURE AND
> FOREST
> Team Biometrie & Kwaliteitszorg / Team Biometrics & Quality Assurance
> thierry.onkelinx using inbo.be
> Havenlaan 88 bus 73, 1000 Brussel
> www.inbo.be
>
>
> ///////////////////////////////////////////////////////////////////////////////////////////
> To call in the statistician after the experiment is done may be no more
> than asking him to perform a post-mortem examination: he may be able to say
> what the experiment died of. ~ Sir Ronald Aylmer Fisher
> The plural of anecdote is not data. ~ Roger Brinner
> The combination of some data and an aching desire for an answer does not
> ensure that a reasonable answer can be extracted from a given body of data.
> ~ John Tukey
>
> ///////////////////////////////////////////////////////////////////////////////////////////
>
> <https://www.inbo.be>
>
>
> Op wo 15 jul. 2020 om 00:10 schreef Francois Rousset <
> francois.rousset using umontpellier.fr>:
>
>> Dear Thierry,
>>
>> please provide a reproducible example so that we know what you have
>> actually done.
>>
>> Best,
>>
>> F.
>> Le 14/07/2020 à 20:00, Thierry Onkelinx a écrit :
>>
>> Dear François and Sarah,
>>
>> INLA seems more efficient. I ran a model with Mattern correlation
>> structure on 13K locations (1 observation per location) in under 10 minutes
>> on a laptop with 16GB RAM.
>>
>> Best regards,
>>
>> ir. Thierry Onkelinx
>> Statisticus / Statistician
>>
>> Vlaamse Overheid / Government of Flanders
>> INSTITUUT VOOR NATUUR- EN BOSONDERZOEK / RESEARCH INSTITUTE FOR NATURE
>> AND FOREST
>> Team Biometrie & Kwaliteitszorg / Team Biometrics & Quality Assurance
>> thierry.onkelinx using inbo.be
>> Havenlaan 88 bus 73, 1000 Brussel
>> www.inbo.be
>>
>>
>> ///////////////////////////////////////////////////////////////////////////////////////////
>> To call in the statistician after the experiment is done may be no more
>> than asking him to perform a post-mortem examination: he may be able to say
>> what the experiment died of. ~ Sir Ronald Aylmer Fisher
>> The plural of anecdote is not data. ~ Roger Brinner
>> The combination of some data and an aching desire for an answer does not
>> ensure that a reasonable answer can be extracted from a given body of data.
>> ~ John Tukey
>>
>> ///////////////////////////////////////////////////////////////////////////////////////////
>>
>> <https://www.inbo.be>
>>
>>
>> Op di 14 jul. 2020 om 18:22 schreef Francois Rousset <
>> francois.rousset using umontpellier.fr>:
>>
>>> Dear Sarah,
>>>
>>> Le 14/07/2020 à 16:55, Sarah Chisholm a écrit :
>>> > Hi Mollie, thank you for your suggestion. glmmTMB seems like a good
>>> > option for my needs as well. In your sample code above, can you
>>> > explain what the term 'group' does in matern(pos+0|group)? Does this
>>> > allow the spatial correlation structure to be applied to specific
>>> > groupings in the data (in my case, for example, by 'continent')?
>>> >
>>> > Francois, thank you for this very clear answer. This is a very
>>> > convenient feature of the function! May I ask you a couple of other
>>> > questions about some issues that I've had with spaMM::fitme()?
>>> >
>>> > In particular, when I try fitting this model to a large data set (~14
>>> > 000 rows x 7 columns, ~2 MB), the model will run for an extended
>>> > period of time, to the point where I've had to terminate the
>>> > computation. I've tried applying the suggestions that are mentioned in
>>> > the user guide, i.e. setting init=list(lambda=0.1)
>>> > and init=list(lambda=NaN). Implementing init=list(lambda=0.1) returned
>>> > an error suggesting that there was a lack of memory, while running the
>>> > model with init=list(lambda=NaN) also ran for an extended period of
>>> > time without completing. Is there something else I can do to speed up
>>> > the fit of these models?
>>> >
>>> > I've had a similar problem with an even larger data set (~185 000 rows
>>> > x 8 columns, ~21 MB), where, when I try running the model, this error
>>> > is returned immediately:
>>> >
>>> > ErrorinZA %*%xmatrix :Cholmoderror 'problem too large'at file
>>> > ../Core/cholmod_dense.c,line 105
>>> >
>>> > I've tried running this model on two devices, both with a 64-bit OS
>>> > with Windows 10, one with 32 GB of RAM and the other with 64 GB. I've
>>> > gotten the same error from both devices. Is there a way that fitme()
>>> > can accommodate these large data sets?
>>>
>>> spaMM can handle large data sets, but the first issue to consider here
>>> is the number of distinct locations for the spatial random effect. The
>>> large correlation matrices of geostatistical models will always be a
>>> problem, both in terms of memory requirements and of potentially huge
>>> computation times. My guess from past experiments is that one should
>>> still be able to fit models with ~ 10K locations within a few days on a
>>> computer with <60 Gb of RAM (given perhaps some tinkering of the
>>> arguments), so at least the data set of 14 000 rows should be feasible,
>>> particularly if the number of locations is smaller.
>>>
>>> Anyone planning to analyze large spatial data sets should anticipate
>>> these problems and check by themselves whether there is any practical
>>> alternative suitable for their particular problem. The discussion in
>>> section 6.2 of the "gentle introduction" to spaMM may then be useful.
>>>
>>> Best,
>>>
>>> F.
>>>
>>> >
>>> > Thank you,
>>> >
>>> > Sarah
>>>
>>>         [[alternative HTML version deleted]]
>>>
>>> _______________________________________________
>>> R-sig-mixed-models using r-project.org mailing list
>>> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>>>
>>

	[[alternative HTML version deleted]]



More information about the R-sig-mixed-models mailing list