[R-sig-ME] segfaults from lmer on a 64-bit system

[Douglas Bates]

Thanks for sending the data and the formula, Ben.

I have good news and bad news.  The good news is that I was able to
get consistent behavior on this example from different systems.  The
bad news is that I did so by having R segfault on a Mac.

I'm not sure exactly where the segfault occurs but the underlying
problem is your random effects specification.  You are specifying a
huge number of random effects (27 terms, some of which may correspond
to more than one random effect) for each level of the grouping factor,
by.var, which has only 4 levels.  You are trying to estimate something
like 380 variances or covariances from 4 groups.

Specifying the random effects is not simply a matter of incorporating
every fixed effect as a random effects and allowing a general
variance-covariance structure.  In this case about the best that you
can hope to include with only 4 levels of the grouping factor is
(1|by.var).


On Thu, Aug 14, 2008 at 5:21 AM, Ben Hansen <bbh at umich.edu> wrote:
> Thanks for these suggestions.
>
> I updated to the latest versions of the packages, and the segfault persists.
>
> I have isolated an example for reproduction purposes. The data are given in
>    http://www.stat.lsa.umich.edu/~bbh/segfault-bh.rda
>
> I get segfaults when I do:
>>  library(lme4) ; load("segfault-bh.rda") ; mylme <-lmer(myfmla, data=mydata)
>
> The segfault occurs on a couple of setups:
> 1.  64-bit Red Hat linux; R version 2.7.1:
>> sessionInfo()
> R version 2.7.1 (2008-06-23)
> x86_64-unknown-linux-gnu
>
> locale:
> LC_CTYPE=en_US.UTF-8;LC_NUMERIC=C;LC_TIME=en_US.UTF-8;LC_COLLATE=en_US.UTF-8
> ;LC_MONETARY=C;LC_MESSAGES=en_US.UTF-8;LC_PAPER=en_US.UTF-8;LC_NAME=C;LC_ADD
> RESS=C;LC_TELEPHONE=C;LC_MEASUREMENT=en_US.UTF-8;LC_IDENTIFICATION=C
>
> attached base packages:
> [1] stats     graphics  grDevices utils     datasets  methods   base
>
> other attached packages:
> [1] lme4_0.999375-24   Matrix_0.999375-11 lattice_0.17-8
>
> loaded via a namespace (and not attached):
> [1] grid_2.7.1  tools_2.7.1
>
> 2. Mac OS; R version 2.7.1:
>> sessionInfo()
> R version 2.7.1 (2008-06-23)
> powerpc-apple-darwin8.10.1
>
> locale:
> C
>
> attached base packages:
> [1] stats     graphics  grDevices utils     datasets  methods   base
>
> other attached packages:
> [1] lme4_0.999375-22   Matrix_0.999375-10 lattice_0.17-8
>
> loaded via a namespace (and not attached):
> [1] grid_2.7.1
>
> 3.  I also got the segfault on the 64-bit linux machine using R version
> 2.6.1, lme4 version 0.99875-9 (I think), and Matrix version 0.999375-9 (I
> think).
>
> 4. *On the other hand,* it sort of works (without segfaulting at least)
> using older R, lme4 and Matrix on my Mac:
>> sessionInfo()
> R version 2.6.2 (2008-02-08)
> powerpc-apple-darwin8.10.1
>
> locale:
> C
>
> attached base packages:
> [1] stats     graphics  grDevices utils     datasets  methods   base
>
> other attached packages:
> [1] lme4_0.99875-9    Matrix_0.999375-9 lattice_0.17-6
>
> loaded via a namespace (and not attached):
> [1] grid_2.6.2
>
>> mylme <- lmer(myfmla, data=mydata)
>
> Warning messages:
> 1: In .local(x, ..., value) :
>  Estimated variance-covariance for factor 'by.var' is singular
>
> 2: In .local(x, ..., value) :
>  nlminb returned message false convergence (8)
>
> --Perhaps the warnings contain a clue as to what the problem is?
>
> Thanks for your consideration of this.
>
> Ben
> On 8/13/08 2:55 AM, "Douglas Bates" <bates at stat.wisc.edu> wrote:
>
>> I would first try later versions of both the Matrix and lme4 packages.
>>  Versions currently on CRAN are Matrix_0.999375-11 and
>> lme4_0.999375-24
>>
>> If you continue to get segfaults then having a reproducible example,
>> along with the data - anonymized if you wish, will be important.
>>
>> You can determine the versions of the packages that you are using with
>>
>> sessionInfo()
>>
>>
>> On Wed, Aug 13, 2008 at 12:06 AM, Hansen, Ben <bbh at umich.edu> wrote:
>>> Hello all,
>>>
>>>
>>>
>>> I am encountering trouble with lme4 on a 64-bit linux system, and I'm
>>> wondering if any of you might have advice on how to circumvent it. Using
>>> recent versions of lme4 and R, I was able to compile and install the
>>> package, and use it with success on relatively small problems.  However,
>>> when I tried to scale up to work on a big problem, applying lmer() with
>>> thousands of observations and a hundred or so random effects, my R fails
>>> reporting a segmentation fault.  I don't see the same problem on a
>>> different 32-bit Mac OSX system, which runs the same code without
>>> evident difficulty.
>>>
>>>
>>>
>>> Some details of the setup where the problem occurs.  We've tried a few
>>> combinations of things, so I write what we tried first, with variants
>>> noted in parentheses:
>>>
>>>
>>>
>>> Red Hat Enterprise Linux 4 (version 5)
>>>
>>> R version 2.7.1 (2.6.1)
>>>
>>> gcc/g++ compilers for C and C++, Portland Group Fortran compilers
>>> (gcc/g++/gfortran, ie full GNU compiler suite)
>>>
>>> Matrix package version 0.999375-10  (installed with and without
>>> --with-package-versions flag)
>>>
>>> lme4 version 0.999375-16 (installed with and without
>>> --with-package-versions flag)
>>>
>>>
>>>
>>> Any suggestions?  I'd be very appreciative.
>>>
>>>
>>>
>>> Ben
>>>
>>>
>>>
>>>
>>>
>>>
>>>        [[alternative HTML version deleted]]
>>>
>>> _______________________________________________
>>> R-sig-mixed-models at r-project.org mailing list
>>> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>>>
>
>