[R-sig-ME] segfaults from lmer on a 64-bit system

[Hansen, Ben]

Thanks much for looking into this, Doug. 

I recognize that what I'm doing here may seem odd.  My hope was to adapt
the package to a non-standard purpose of mine.  For this purpose, the
variances of the random effects are not of central interest, and it's OK
if they (those variances) are not particularly well estimated.  I found
encouragement in Gelman and Hill's regression book (\S 12.9, pp.275 ff),
which says it's a mistake to think that multilevel models require a
minimum number of groups.  As they put it, the issue is just that when
the number of groups is small then the random-effects variances will
tend to be overestimated.  For my purposes, that would have been OK. 

You suggest only modeling the intercept as random, given that I've only
got four groups.  For my particular purposes, that isn't an option.
Could lmer be expected to better handle my (admittedly ornate) random
effects specfication if there were more groups than 4 -- say, 6 or 8?

Thanks again for the attention.

Ben

Ben B. Hansen
Assistant Professor
Statistics Department and Institute for Social Research
www.stat.lsa.umich.edu/~bbh/
University of Michigan
Ann Arbor, MI 48109-1107


-----Original Message-----
From: dmbates at gmail.com [mailto:dmbates at gmail.com] On Behalf Of Douglas
Bates
Sent: Thursday, August 14, 2008 5:08 AM
To: Hansen, Ben
Cc: r-sig-mixed-models at r-project.org; Caird, Andrew J
Subject: Re: [R-sig-ME] segfaults from lmer on a 64-bit system

Thanks for sending the data and the formula, Ben.

I have good news and bad news.  The good news is that I was able to
get consistent behavior on this example from different systems.  The
bad news is that I did so by having R segfault on a Mac.

I'm not sure exactly where the segfault occurs but the underlying
problem is your random effects specification.  You are specifying a
huge number of random effects (27 terms, some of which may correspond
to more than one random effect) for each level of the grouping factor,
by.var, which has only 4 levels.  You are trying to estimate something
like 380 variances or covariances from 4 groups.

Specifying the random effects is not simply a matter of incorporating
every fixed effect as a random effects and allowing a general
variance-covariance structure.  In this case about the best that you
can hope to include with only 4 levels of the grouping factor is
(1|by.var).


On Thu, Aug 14, 2008 at 5:21 AM, Ben Hansen <bbh at umich.edu> wrote:
> Thanks for these suggestions.
>
> I updated to the latest versions of the packages, and the segfault
persists.
>
> I have isolated an example for reproduction purposes. The data are
given in
>    http://www.stat.lsa.umich.edu/~bbh/segfault-bh.rda
>
> I get segfaults when I do:
>>  library(lme4) ; load("segfault-bh.rda") ; mylme <-lmer(myfmla,
data=mydata)
>
> The segfault occurs on a couple of setups:
> 1.  64-bit Red Hat linux; R version 2.7.1:
>> sessionInfo()
> R version 2.7.1 (2008-06-23)
> x86_64-unknown-linux-gnu
>
> locale:
>
LC_CTYPE=en_US.UTF-8;LC_NUMERIC=C;LC_TIME=en_US.UTF-8;LC_COLLATE=en_US.U
TF-8
>
;LC_MONETARY=C;LC_MESSAGES=en_US.UTF-8;LC_PAPER=en_US.UTF-8;LC_NAME=C;LC
_ADD
> RESS=C;LC_TELEPHONE=C;LC_MEASUREMENT=en_US.UTF-8;LC_IDENTIFICATION=C
>
> attached base packages:
> [1] stats     graphics  grDevices utils     datasets  methods   base
>
> other attached packages:
> [1] lme4_0.999375-24   Matrix_0.999375-11 lattice_0.17-8
>
> loaded via a namespace (and not attached):
> [1] grid_2.7.1  tools_2.7.1
>
> 2. Mac OS; R version 2.7.1:
>> sessionInfo()
> R version 2.7.1 (2008-06-23)
> powerpc-apple-darwin8.10.1
>
> locale:
> C
>
> attached base packages:
> [1] stats     graphics  grDevices utils     datasets  methods   base
>
> other attached packages:
> [1] lme4_0.999375-22   Matrix_0.999375-10 lattice_0.17-8
>
> loaded via a namespace (and not attached):
> [1] grid_2.7.1
>
> 3.  I also got the segfault on the 64-bit linux machine using R
version
> 2.6.1, lme4 version 0.99875-9 (I think), and Matrix version 0.999375-9
(I
> think).
>
> 4. *On the other hand,* it sort of works (without segfaulting at
least)
> using older R, lme4 and Matrix on my Mac:
>> sessionInfo()
> R version 2.6.2 (2008-02-08)
> powerpc-apple-darwin8.10.1
>
> locale:
> C
>
> attached base packages:
> [1] stats     graphics  grDevices utils     datasets  methods   base
>
> other attached packages:
> [1] lme4_0.99875-9    Matrix_0.999375-9 lattice_0.17-6
>
> loaded via a namespace (and not attached):
> [1] grid_2.6.2
>
>> mylme <- lmer(myfmla, data=mydata)
>
> Warning messages:
> 1: In .local(x, ..., value) :
>  Estimated variance-covariance for factor 'by.var' is singular
>
> 2: In .local(x, ..., value) :
>  nlminb returned message false convergence (8)
>
> --Perhaps the warnings contain a clue as to what the problem is?
>
> Thanks for your consideration of this.
>
> Ben
> On 8/13/08 2:55 AM, "Douglas Bates" <bates at stat.wisc.edu> wrote:
>
>> I would first try later versions of both the Matrix and lme4
packages.
>>  Versions currently on CRAN are Matrix_0.999375-11 and
>> lme4_0.999375-24
>>
>> If you continue to get segfaults then having a reproducible example,
>> along with the data - anonymized if you wish, will be important.
>>
>> You can determine the versions of the packages that you are using
with
>>
>> sessionInfo()
>>
>>
>> On Wed, Aug 13, 2008 at 12:06 AM, Hansen, Ben <bbh at umich.edu> wrote:
>>> Hello all,
>>>
>>>
>>>
>>> I am encountering trouble with lme4 on a 64-bit linux system, and
I'm
>>> wondering if any of you might have advice on how to circumvent it.
Using
>>> recent versions of lme4 and R, I was able to compile and install the
>>> package, and use it with success on relatively small problems.
However,
>>> when I tried to scale up to work on a big problem, applying lmer()
with
>>> thousands of observations and a hundred or so random effects, my R
fails
>>> reporting a segmentation fault.  I don't see the same problem on a
>>> different 32-bit Mac OSX system, which runs the same code without
>>> evident difficulty.
>>>
>>>
>>>
>>> Some details of the setup where the problem occurs.  We've tried a
few
>>> combinations of things, so I write what we tried first, with
variants
>>> noted in parentheses:
>>>
>>>
>>>
>>> Red Hat Enterprise Linux 4 (version 5)
>>>
>>> R version 2.7.1 (2.6.1)
>>>
>>> gcc/g++ compilers for C and C++, Portland Group Fortran compilers
>>> (gcc/g++/gfortran, ie full GNU compiler suite)
>>>
>>> Matrix package version 0.999375-10  (installed with and without
>>> --with-package-versions flag)
>>>
>>> lme4 version 0.999375-16 (installed with and without
>>> --with-package-versions flag)
>>>
>>>
>>>
>>> Any suggestions?  I'd be very appreciative.
>>>
>>>
>>>
>>> Ben
>>>
>>>
>>>
>>>
>>>
>>>
>>>        [[alternative HTML version deleted]]
>>>
>>> _______________________________________________
>>> R-sig-mixed-models at r-project.org mailing list
>>> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>>>
>
>