[R-sig-ME] current r-forge version fails R CMD check ... ?
Martin Maechler
maechler at stat.math.ethz.ch
Mon Aug 3 12:06:11 CEST 2009
BB> Does this suggest a memory/pointer reference problem
BB> (the only way I can think of getting non-deterministic
BB> behavior of this type) ... ? ugh, ugh, ugh.
BB> Tried running with valgrind, but nothing pops up.
You could try
gctorture()
as well. But beware, that needs a lot of patience.
BB> After running the example below to create D, I can get
BB> two different results from the *same* lmer call ...
>> table(replicate(40,ranef(lmer(y~(x1+x2)|ff,data=D))$ff[1,1]))
BB> 8.35055995996088 8.48042553563304 17 23
aha. This at least seems "logical" to me, given the identical()
problems mentioned earlier.
Since, I've said all the time, the two ways really must give
identical results, and using all.equal() instead is really the
wrong test for the purpose ...
BB> I am also worried (without much justification) that
BB> the problem might lie in Matrix, which is even Deeper
BB> Magic to me than lme4 ...
:-)
If I only found a way to reproduce your findings on one of my
Linux platforms,...
then I could start digging...
Martin
BB> Reinhold Kliegl wrote:
>> When I run Martin's example several times, using
>> "set.seed(1)" before each run, I get all possible
>> outcomes: (a) Error for m0 vs. m1, (b) Error for m2 vs
>> m3, and (c) no error.
>>
>> Reinhold
>>
>> other attached packages: [1] lme4_0.999375-32
>> Matrix_0.999375-30 lattice_0.17-25
>>
>>> ##------------------------------------------
>> # Maechler 01-08-09
>> set.seed(1)
>> ##----------------------------------------------------
>> D <- data.frame(y= rnorm(20,10), ff = gl(4,5),
> + x1=rnorm(20,3), x2=rnorm(20,7),
> + x3=rnorm(20,1))
>> m0 <- lmer(y ~ (x1 + x2)|ff, data = D)
>> m1 <- lmer(y ~ x1 + x2|ff , data = D)
>> m2 <- lmer(y ~ x1 + (x2|ff), data = D)
>> m3 <- lmer(y ~ (x2|ff) + x1, data = D)
>> stopifnot(identical(ranef(m0), ranef(m1)),
> + identical(ranef(m2), ranef(m3)))
> Fehler: identical(ranef(m2), ranef(m3)) is not TRUE
> + cat("Ok\n")
> Ok
>> ##------------------------------------------
>> # Maechler 01-08-09
>> set.seed(1)
>> ##----------------------------------------------------
>> D <- data.frame(y= rnorm(20,10), ff = gl(4,5),
> + x1=rnorm(20,3), x2=rnorm(20,7),
> + x3=rnorm(20,1))
>> m0 <- lmer(y ~ (x1 + x2)|ff, data = D)
>> m1 <- lmer(y ~ x1 + x2|ff , data = D)
>> m2 <- lmer(y ~ x1 + (x2|ff), data = D)
>> m3 <- lmer(y ~ (x2|ff) + x1, data = D)
>> stopifnot(identical(ranef(m0), ranef(m1)),
> + identical(ranef(m2), ranef(m3)))
> Fehler: identical(ranef(m0), ranef(m1)) is not TRUE
> + cat("Ok\n")
> Ok
>> ##------------------------------------------
>> # Maechler 01-08-09
>> set.seed(1)
>> ##----------------------------------------------------
>> D <- data.frame(y= rnorm(20,10), ff = gl(4,5),
> + x1=rnorm(20,3), x2=rnorm(20,7),
> + x3=rnorm(20,1))
>> m0 <- lmer(y ~ (x1 + x2)|ff, data = D)
>> m1 <- lmer(y ~ x1 + x2|ff , data = D)
>> m2 <- lmer(y ~ x1 + (x2|ff), data = D)
>> m3 <- lmer(y ~ (x2|ff) + x1, data = D)
>> stopifnot(identical(ranef(m0), ranef(m1)),
> + identical(ranef(m2), ranef(m3)))
>> cat("Ok\n")
> Ok
>> ##------------------------------------------
>
>
> On Sat, Aug 1, 2009 at 7:53 PM, Ben Bolker<bolker at ufl.edu> wrote:
>> Thanks, Reinhold, I'm glad I'm not completely nuts. With Doug Bates
>> (quite reasonably) occupied with other things, it strikes me it might be
>> a little hard to dig deep enough into the guts to see what's going on
>> ... I will see how far I can get, but this is the kind of problem where
>> **if** we understood what was going on and it looked hard to fix, it
>> would seem reasonable to replace the "must be identical" criterion with
>> "abs(difference)<1e-7" or some such in the tests ...
>>
>> Ben
>>
>> Reinhold Kliegl wrote:
>>> Just updated to Matrix_0.999375-30. The previous problem persists and
>>> now it also reports:
>>> Fehler: identical(ranef(om2), ranef(om3)) is not TRUE
>>>
>>> Reinhold
>>>
>>>> stopifnot(identical(ranef(om2), ranef(om3)),
>>> + identical(deviance(om2), deviance(om3)))
>>> Fehler: identical(ranef(om2), ranef(om3)) is not TRUE
>>> + if (identical(TRUE, all.equal(fixef(m2), fixef(om2))))
>>> + stop("offset does not change the fixed effects")
>>>> cat('Time elapsed: ', proc.time(),'\n') # for ``statistical reasons''
>>> Time elapsed: 13.588 0.399 14.297 0 0
>>>> sessionInfo()
>>> R version 2.9.1 (2009-06-26)
>>> i386-apple-darwin8.11.1
>>>
>>> locale:
>>> de_DE.UTF-8/en_US.UTF-8/C/C/de_DE.UTF-8/de_DE.UTF-8
>>>
>>> attached base packages:
>>> [1] stats graphics grDevices utils datasets methods
>>> [7] base
>>>
>>> other attached packages:
>>> [1] lme4_0.999375-31 Matrix_0.999375-30 lattice_0.17-25
>>>
>>>
>>> On Sat, Aug 1, 2009 at 7:37 PM, Reinhold
>>> Kliegl<reinhold.kliegl at gmail.com> wrote:
>>>> Ben's problem shows up with my implementation, too. Info below.
>>>>
>>>> Reinhold
>>>>
>>>>> stopifnot(identical(ranef(m0), ranef(m1)),
>>>> + identical(ranef(m2), ranef(m3)),
>>>> + inherits(tryCatch(lmer(y ~ x2|ff + x1, data = D), error =
>>>> function(e)e),"error"))
>>>> CHOLMOD error: xG˝ LÛR
>>>> Fehler: identical(ranef(m0), ranef(m1)) is not TRUE
>>>> Zusätzlich: Warnmeldung:
>>>> In Ops.factor(ff, x1) : + nicht sinnvoll für Faktoren
>>>> +
>>>>> ## Check the use of offset
>>>>> om2 <- lmer(y ~ x1 + (x2|ff), data = D, offset = x3)
>>>>> om3 <- lmer(y ~ x1 + (x2|ff) + offset(x3), data = D)
>>>>>
>>>>> stopifnot(identical(ranef(om2), ranef(om3)),
>>>> + identical(deviance(om2), deviance(om3)))
>>>>> if (identical(TRUE, all.equal(fixef(m2), fixef(om2))))
>>>> + stop("offset does not change the fixed effects")
>>>>> cat('Time elapsed: ', proc.time(),'\n') # for ``statistical reasons''
>>>> Time elapsed: 11.608 0.369 12.353 0 0
>>>>> sessionInfo()
>>>> R version 2.9.1 (2009-06-26)
>>>> i386-apple-darwin8.11.1
>>>>
>>>> locale:
>>>> de_DE.UTF-8/en_US.UTF-8/C/C/de_DE.UTF-8/de_DE.UTF-8
>>>>
>>>> attached base packages:
>>>> [1] stats graphics grDevices utils datasets methods
>>>> [7] base
>>>>
>>>> other attached packages:
>>>> [1] lme4_0.999375-31 Matrix_0.999375-29 lattice_0.17-25
>>>>
>>>> loaded via a namespace (and not attached):
>>>> [1] grid_2.9.1
>>>>
>>>> On Sat, Aug 1, 2009 at 6:31 PM, Ben Bolker<bolker at ufl.edu> wrote:
>>>>> I don't mind it being public.
>>>>>
>>>>> I got similar results with the CRAN lme4 (0.999375-31),
>>>>> with Matrix ...-30. BATCH fails on m2 != m3 (consistently);
>>>>> source() fails on m0 != m1.
>>>>>
>>>>> I'm probably doing something really really dumb, would appreciate
>>>>> anyone else who can try this on their systems ...
>>>>>
>>>>> If you don't feel like downloading or running all of lmer-1.R, the
>>>>> following code chunk should demonstrate the problem ...
>>>>>
>>>>> =================
>>>>> library(lme4)
>>>>>
>>>>> set.seed(1)
>>>>> ## Wrong formula gave a seg.fault at times:
>>>>> D <- data.frame(y= rnorm(20,10), ff = gl(4,5),
>>>>> x1=rnorm(20,3), x2=rnorm(20,7),
>>>>> x3=rnorm(20,1))
>>>>> m0 <- lmer(y ~ (x1 + x2)|ff, data = D)
>>>>> m1 <- lmer(y ~ x1 + x2|ff , data = D)
>>>>> m2 <- lmer(y ~ x1 + (x2|ff), data = D)
>>>>> m3 <- lmer(y ~ (x2|ff) + x1, data = D)
>>>>> stopifnot(identical(ranef(m0), ranef(m1)),
>>>>> identical(ranef(m2), ranef(m3)),
>>>>> inherits(tryCatch(lmer(y ~ x2|ff + x1, data = D), error =
>>>>> function(e)e),
>>>>> "error"))
>>>>>
>>>>> ## Check the use of offset
>>>>> om2 <- lmer(y ~ x1 + (x2|ff), data = D, offset = x3)
>>>>> om3 <- lmer(y ~ x1 + (x2|ff) + offset(x3), data = D)
>>>>>
>>>>> stopifnot(identical(ranef(om2), ranef(om3)),
>>>>> identical(deviance(om2), deviance(om3)))
>>>>> if (identical(TRUE, all.equal(fixef(m2), fixef(om2))))
>>>>> stop("offset does not change the fixed effects")
>>>>>
>>>>> cat('Time elapsed: ', proc.time(),'\n') # for ``statistical reasons''
>>>>>
>>>>>
>>>>> Martin Maechler wrote:
>>>>>> Hi Ben,
>>>>>> as you took this "private", I'd like at least Doug Bates
>>>>>> to be in the CC ..
>>>>>> Personally I would prefer to have this continue in the R-SIG-ME list
>>>>>> rather than privately... I'll be pretty offline from now till Monday
>>>>>> in any case
>>>>>>
>>>>>> On Fri, Jul 31, 2009 at 20:17, Ben Bolker<bolker at ufl.edu> wrote:
>>>>>>> Martin Maechler wrote:
>>>>>>>>>>>>> "BB" == Ben Bolker <bolker at ufl.edu>
>>>>>>>>>>>>> on Thu, 30 Jul 2009 17:30:17 -0400 writes:
>>>>>>>> BB> When I use the latest r-forge version of lme4
>>>>>>>> BB> ( 0.999375-32 ) it seems to fail R CMD check on a tiny
>>>>>>>> BB> numerical mismatch of two objects that are supposed
>>>>>>>> BB> (??) to be identical (I also
>>>>>>>> BB> get a mangled CHOLMOD error message, but I suspect that
>>>>>>>> BB> comes from somewhere within Matrix ...)
>>>>>>>>
>>>>>>>> yes, and those should be gone with the version of Matrix
>>>>>>>> (0.999375-30) of two days ago.
>>>>>>>>
>>>>>>>> BB> can anyone confirm?
>>>>>>>>
>>>>>>>> No. To the contrary.
>>>>>>>> I have had a slightly updated version of tests/lmer-1.Rout.save
>>>>>>>> ready to be committed for a while, but that's only trivial
>>>>>>>> changes.
>>>>>>>>
>>>>>>>> and below, from your sessionInfo(), it looks like you are using
>>>>>>>> a current version of R and packages ...
>>>>>>>> hmm ...
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Martin
>>>>>>>>
>>>>>>>>
>>>>>>>> BB> can anyone confirm? any ideas for a fix?
>>>>>>>>
>>>>>>>>
>>>>>>>> BB> The offending mismatch between ranef(m2) and ranef(m3)
>>>>>>>> BB> is very small ...
>>>>>>>>
>>>>>>>> well; it's interesting that the offending mismatch in the error
>>>>>>>> message below is between m0 and m1, ...
>>>>>>> hmmm indeed. Maybe I was already hacking things. I have
>>>>>>> (1) updated Matrix, (2) installed lme4 directly from r-forge.
>>>>>>> sessionInfo() says
>>>>>>>
>>>>>>> lme4_0.999375-32 Matrix_0.999375-30
>>>>>>>
>>>>>>> in ../tests, I do
>>>>>>>
>>>>>>> R --vanilla
>>>>>>> library(lme4)
>>>>>>> source("lmer-1.R",echo=TRUE)
>>>>>>>
>>>>>>> or
>>>>>>>
>>>>>>> R CMD BATCH --vanilla lmer-1.R
>>>>>>>
>>>>>>> oddly, the second (BATCH) always fails on m0/m1; the
>>>>>>> first (source) fails at different comparisons (sometimes m0/m1;
>>>>>>> sometimes m2/m3; sometimes om2/om3 in the next section ... ???
>>>>>> I just can't understand how that *can* happen.
>>>>>> It would mean that the algorithms used were slightly "random", or
>>>>>> e.g. using slightly different precision depending on memory
>>>>>> allocation, or ??,
>>>>>> ???
>>>>>>
>>>>>> As I said i the first e-mail: The slightly different formula should
>>>>>> produce absolutely identical matrices and vectors which define the
>>>>>> loglikelihood (or RE-LogLik.) and then the minimization really should
>>>>>> be 100% reproducible on a given R+Platform+Installed-Packages setup.
>>>>>>
>>>>>> I assume you have tried the same with the CRAN-version of lme4 ...
>>>>>> which has exactly the same tests/lmer-1.R ?
>>>>>> ....
>>>>>> the phenomenon looks so illogical, I even start to wonder if it's a
>>>>>> bug in your computer (hardware-low-level software combination)?
>>>>>> Maybe you could ask again on R-SIg-ME if others could reproduce?
>>>>>>
>>>>>>>> BTW: Have you noticed that we (Doug Bates and I, when at the
>>>>>>>> useR/DSC meetings) have moved the former 'allcoef' branch into a
>>>>>>>> ``regular R-forge package'' called 'lme4a'
>>>>>>> yes.
>>>>>>>> But yes, that definitely does not pass 'CMD check at the moment'.
>>>>>>>>
>>>>>>>> >> getwd()
>>>>>>>> BB> [1] "/home/ben/lib/R/pkgs/lme4/pkg/lme4/tests"
>>>>>>>>
>>>>>>>> >> source("lmer-1.R",echo=TRUE)
>>>>>>>>
>>>>>>>> BB> ...
>>>>>>>> >> D <- data.frame(y= rnorm(20,10), ff = gl(4,5),
>>>>>>>> BB> x1=rnorm(20,3), x2=rnorm(20,7),
>>>>>>>> BB> x3=rnorm(20,1))
>>>>>>>> >> m0 <- lmer(y ~ (x1 + x2)|ff, data = D)
>>>>>>>> >> m1 <- lmer(y ~ x1 + x2|ff , data = D)
>>>>>>>>
>>>>>>>> We had added these checks exactly *because* we wanted to be sure
>>>>>>>> that a slightly different use of formulas would lead to the
>>>>>>>> identical 'X', 'Z', .... matrices, and L(theta)
>>>>>>>> parametrizations,
>>>>>>>> so I wonder how your version of lme4 could give different
>>>>>>>> results here....
>>>>>>>>
>>>>>>>> >> m2 <- lmer(y ~ x1 + (x2|ff), data = D)
>>>>>>>> >> m3 <- lmer(y ~ (x2|ff) + x1, data = D)
>>>>>>>> >> stopifnot(identical(ranef(m0), ranef(m1)),
>>>>>>>> BB> + identical(ranef(m2), ranef(m3)),
>>>>>>>> BB> + inherits(tryCatch(lmer(y ~ x2|ff + x1, data = D) ....
>>>>>>>> BB> [TRUNCATED]
>>>>>>>> BB> CHOLMOD error: =*ᶈ1ñ¿@ÀTôoá¶
>>>>>>>> BB> Error: identical(ranef(m0), ranef(m1)) is not TRUE
>>>>>>>> BB> In addition: Warning message:
>>>>>>>> BB> In Ops.factor(ff, x1) : + not meaningful for factors
>>>>>>>>
>>>>>>>> Note that the cholmod error and warning is from the
>>>>>>>> lmer(y ~ x2|ff + x1, data = D)
>>>>>>>> part {which is wrapped in tryCatch(...)}.
>>>>>>>>
>>>>>>>> Also, if I execute
>>>>>>>>
>>>>>>>> ##----------------------------------------------------
>>>>>>>> D <- data.frame(y= rnorm(20,10), ff = gl(4,5),
>>>>>>>> x1=rnorm(20,3), x2=rnorm(20,7),
>>>>>>>> x3=rnorm(20,1))
>>>>>>>> m0 <- lmer(y ~ (x1 + x2)|ff, data = D)
>>>>>>>> m1 <- lmer(y ~ x1 + x2|ff , data = D)
>>>>>>>> m2 <- lmer(y ~ x1 + (x2|ff), data = D)
>>>>>>>> m3 <- lmer(y ~ (x2|ff) + x1, data = D)
>>>>>>>> stopifnot(identical(ranef(m0), ranef(m1)),
>>>>>>>> identical(ranef(m2), ranef(m3)))
>>>>>>>> cat("Ok\n")
>>>>>>>> ##----------------------------------------------------
>>>>>>>>
>>>>>>>> many times, I never see a problem.
>>>>>>>>
>>>>>>>> Are you sure you are not using your already-hacked version of
>>>>>>>> lme4 ???
>>>>>>>>
>>>>>>>> Martin Maechler, ETH Zurich
>>>>>>>>
>>>>>>> I'm not 100.0000% sure, but I don't see how I could be ...
>>>>>>>
>>>>>>> Ben
>>>>>>>
>>>>>>>
>>>>> --
>>>>> Ben Bolker
>>>>> Associate professor, Biology Dep't, Univ. of Florida
>>>>> bolker at ufl.edu / www.zoology.ufl.edu/bolker
>>>>> GPG key: www.zoology.ufl.edu/bolker/benbolker-publickey.asc
>>>>>
>>>>> _______________________________________________
>>>>> R-sig-mixed-models at r-project.org mailing list
>>>>> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>>>>>
>>
>> --
>> Ben Bolker
>> Associate professor, Biology Dep't, Univ. of Florida
>> bolker at ufl.edu / www.zoology.ufl.edu/bolker
>> GPG key: www.zoology.ufl.edu/bolker/benbolker-publickey.asc
>>
--
Ben Bolker
Associate professor, Biology Dep't, Univ. of Florida
bolker at ufl.edu / www.zoology.ufl.edu/bolker
GPG key: www.zoology.ufl.edu/bolker/benbolker-publickey.asc
_______________________________________________
R-sig-mixed-models at r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
More information about the R-sig-mixed-models
mailing list