[R-sig-ME] Bug report for lme

[Dominick Samperi]

On Mon, Mar 14, 2011 at 12:26 PM, Ben Bolker <bbolker at gmail.com> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 11-03-14 10:53 AM, ONKELINX, Thierry wrote:
>> Michal,
>>
>> SPL has 199 levels. Thus (SPL|Plate:Aliq) requires a 199 x 199
>> variance-covariance matrix, which is huge. Have a look at
>> table(b$SPL, factor(b$Plate:b$Aliq)). Notice that you have many
>> combinations without data. Furthermore you have only 6 levels of
>> Plate:Aliq, which is rather low to get good variance estimates.
>>
>> Bottomline: your model is to complex for the data you have.
>>
>> Best regards,
>>
>> Thierry
>
>  So in the short term the answer comes from the old joke: "Doctor, it
> hurts when I do this." "Well, then, don't do that."
>
>  In the medium/long term, the standard in R is that it should be
> *impossible* to actually crash the R session, no matter how silly the
> model one tries to fit.  More ambiguously than that, it is very nice
> (though not always possible) to have the code warn you, with as
> informative an error message as possible, when it detects that you are
> trying to do something silly ...

A common problem is that memory is used up yet there is no
mechanism to catch this and abort gracefully. This is not specific
to R. Any application run under Windows or Linux (in my experience)
that grabs huge chunks of the memory available can cripple the OS
and lead to strange error messages from R (perhaps because it cannot
allocate the memory for a meaningful error message!)

I guess there is a working assumption in many programs that the OS
will not run out of memory, and there is no early warning system that
signals when memory resources are running low.

This kind of early warning system should not be difficult to add to
R...

Dominick

>> ----------------------------------------------------------------------------
>>
>>
> ir. Thierry Onkelinx
>> Instituut voor natuur- en bosonderzoek team Biometrie &
>> Kwaliteitszorg Gaverstraat 4 9500 Geraardsbergen Belgium
>>
>> Research Institute for Nature and Forest team Biometrics & Quality
>> Assurance Gaverstraat 4 9500 Geraardsbergen Belgium
>>
>> tel. + 32 54/436 185 Thierry.Onkelinx at inbo.be www.inbo.be
>>
>> To call in the statistician after the experiment is done may be no
>> more than asking him to perform a post-mortem examination: he may be
>> able to say what the experiment died of. ~ Sir Ronald Aylmer Fisher
>>
>> The plural of anecdote is not data. ~ Roger Brinner
>>
>> The combination of some data and an aching desire for an answer does
>> not ensure that a reasonable answer can be extracted from a given
>> body of data. ~ John Tukey
>>
>>
>>> -----Oorspronkelijk bericht----- Van:
>>> r-sig-mixed-models-bounces at r-project.org
>>> [mailto:r-sig-mixed-models-bounces at r-project.org] Namens Michal
>>> Figurski Verzonden: maandag 14 maart 2011 14:34 Aan: Douglas Bates
>>> CC: R-SIG-Mixed-Models at r-project.org Onderwerp: Re: [R-sig-ME] Bug
>>> report for lme
>>>
>>> Douglas,
>>>
>>> It seems that I didn't make myself entirely clear, so here are a
>>> few clarifications. I also attach some data and a reproducible
>>> example that crashes R -  at least on my machine.
>>>
>>> Clarifications: - "hundreds of lines" - this was actually just one
>>> line in R output. This was part of the error message. I selected it
>>> and pasted into the email body, and it got converted to multiple
>>> lines... I agree this is unlikely to be helpful, though I was not
>>> sure, so I pasted the entire error message. - formula with
>>> 'paste()' is not a problem to me - I use it daily and it always
>>> works. Though for your convenience, in the attached code I used
>>> explicit formula - the outcome is unfortunately the same. - my
>>> .RData file is not corrupt and R starts normally. It only crashes
>>> after I run the problem code.
>>>
>>> The code to reproduce the error:
>>>
>>> # **************************************************************
>>> ***** # # Working example to reproduce crash library(lme4)
>>>
>>> # Read the data in - replace path with your path b      <-
>>> read.csv("crash-example.csv") b$Aliq <- factor(b$Aliq) b$SPL  <-
>>> factor(b$SPL)
>>>
>>> # 1st example, to show that it works - on a selected set of data #
>>> It takes a few moments, but it works - at least it doesn't crash
>>> R: bb  <- b[b$SPL %in% unique(b$SPL[b$Plate=="TRT1"]),] b.1 <-
>>> lmer(Conc~SPL + (SPL|Plate:Aliq)-1, data=bb)
>>>
>>> # 2nd example - it takes only a few seconds and R crashes: #
>>> R-version 2.12.2 on Win7 64bit # Package lme4 version 0.999375-39 #
>>> Previously, when used in R 2.11.1 with lme4 version 0.999375-35, #
>>> I got the error that I pasted in the message before, but the R #
>>> did not crash. b.2 <- lmer(Conc~SPL + (SPL|Plate:Aliq)-1, data=b) #
>>>
>>> *******************************************************************
>>> #
>>>
>>> Kind regards,
>>>
>>> -- Michal J. Figurski, PhD HUP, Pathology & Laboratory Medicine
>>> Biomarker Research Laboratory 3400 Spruce St. 7 Maloney S
>>> Philadelphia, PA 19104 tel. (215) 662-3413
>>>
>>>
>>> On 3/11/2011 4:00 PM, Douglas Bates wrote:
>>>> On Fri, Mar 11, 2011 at 2:32 PM, Michal Figurski
>>>> <figurski at mail.med.upenn.edu>  wrote:
>>>>> Dear Prof. Bates,
>>>>
>>>>> I'm working with a lmer model using your lme4 library version
>>>>> 0.999375-35 with R 2.11.1 on Win7 64bit. Shortly after
>>> starting the
>>>>> function, the R window clears and the following is being
>>>>> displayed:
>>>>>
>>>>> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
>>> 0, 0, 0, 0,
>>>>> 0, 0,
>>>>
>>>> ... Hundreds of lines deleted
>>>>
>>>>> REML = TRUE,  verbose = 0L) 4: do.call(lmer_finalize, ans) 5:
>>>>> lmer(formula(paste(i1, "~SPL + (SPL|Plate:Aliq)-1")),
>>> data = a)
>>>>> Error in mer_finalize(ans) : caught access violation -
>>> continue with
>>>>> care
>>>>>>
>>>>
>>>>> OK, so I updated R and lme4 to the latest versions, and now R
>>>>> just plainly crashes without any message. Let me know if you
>>>>> need additional information to troubleshoot.
>>>>
>>>> Well, hundreds, perhaps thousands, of lines of (part of)  a
>>> dump of an
>>>> object are unlikely to be helpful.
>>>>
>>>> To get R working again you probably need to delete the file
>>> .RData in
>>>> the default directory in which you start R.  It is most likely
>>>> that you have a corrupt object in the saved worksheet.
>>>>
>>>> Secondly, using paste to create a formula is not a great idea.
>>>> The formula may have ended up being unevaluated.  If you really
>>> do need to
>>>> use a character string to name the response variable the
>>>> preferred approach is
>>>>
>>>> form<- substitute(foo ~ SPL + (SPL | Plate:Aliq) - 1, list(foo =
>>>>  as.name(i1))) lmer(form, data=a)
>>>>
>>>> If those suggestions don't help then please provide a
>>>> reproducible example and preferrably on less than a gigantic data
>>>> set.
>>>>
>>>> I am taking the liberty of cc:ing the
>>> R-SIG-Mixed-Models at R-project.org
>>>> mailing list on this reply.  Many of those who read the list may
>>>> be able to help you and it is unlikely I will be able to give
>>>> much assistance soon because I am travelling for the next 10
>>>> days.
>>>
>> _______________________________________________
>> R-sig-mixed-models at r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.10 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAk1+QbMACgkQc5UpGjwzenPiDgCfVHRzFijDxcQMOfYSLzIm6FqP
> S0MAn0ETslK1SbpiE9GJEG7f4MZt6LYi
> =KeMo
> -----END PGP SIGNATURE-----
>
> _______________________________________________
> R-sig-mixed-models at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>