[Rd] parse_Rd and/or lazyload problem

[Duncan Murdoch]

On 01/11/2009 3:12 PM, Mark.Bravington at csiro.au wrote:
>> Okay, then we both agree we should drop it.
>> Duncan Murdoch
> 
> 
> No we don't. I can't provide a functioning mvbutils, or debug, until this is resolved.
> 
> I am trying to be a good citizen and prepare reproducible bug reports-- e.g. the 3 line example. It would be quicker for me to write some ugly hack that modifies base R and gets round the problem *for me*, but that doesn't seem the best outcome for R. A culture which discourages careful bug reporting is unhealthy culture.

Sorry.  What I thought you said was that you had spent several hours on 
it and didn't want to spend more time on it.  I've told you I don't want 
to work on it either.

If there is no way to trigger this bug without using internals, then it 
has not been demonstrated to be a bug in R.  It might be one, or it 
might be a bug in your code.  Often I'll work on things that are 
demonstrated bugs, but I won't commit several hours to debugging your code.

Duncan Murdoch

> Mark Bravington
> 
> 
> ________________________________________
> From: Duncan Murdoch [murdoch at stats.uwo.ca]
> Sent: 02 November 2009 01:08
> To: Bravington, Mark (CMIS, Hobart)
> Cc: r-devel at r-project.org
> Subject: Re: [Rd] parse_Rd and/or lazyload problem
> 
> On 31/10/2009 10:18 PM, Mark.Bravington at csiro.au wrote:
>>> Does this happen in R-patched?  I've seen similar errors in 2.10.0, but
>>> not in a current build.
>> Yes, still there in R-patched.
>>
>> (Still haven't got to your code, this was in
>>> mine.  I'm reluctant to spend time on code that is messing with
>>> internals, because you might be using things in a way not intended by
>>> the author.  Now, if you can show me some code that demonstrates the
>>> problem without using internals directly, I'll follow up.)
>> I did try, but it's not completely possible, because 'makeLazyLoadDB' is internal and there is no public alternative (a pity-- it's useful). However, the problem(s) can be demonstrated without directly calling 'parse_Rd', and with 'lazyLoad' (public) instead of 'fetchRdDB' (private), as per "pointer 1" below. If you have a look at 'tools:::.install_package_Rd_objects', you'll see that my use of 'makeLazyLoadDB' is quite standard.
>>
>> The problem is not easy to reproduce. It took 4-5 hours work to get the 3-line reproducible example that I posted, plus another couple since, so I'm also reluctant to spend more time...
> 
> Okay, then we both agree we should drop it.
> Duncan Murdoch
> 
> 
>> The examples in my previous post still apply-- the first one involves just 3 statements-- but here are some more pointers I've unearthed since:
>>
>>
>> 1. Sometimes 'fetchRdDB' or 'lazyLoad' called directly from the prompt doesn't work, but public 'Rd_db' (which directly calls 'fetchRdDB') does. I've experimented with copying the installed 'tools' package into a new library "d:/temp/fakelib", then stuff like this:
>>
>> test> e <- new.env()
>> test> lazyLoad( 'd:/temp/fakelib/tools/help/tools', e) # original files tools.rdx, tools.rdb
>> test> e <- as.list( e) # force evaluation
>> test> tools:::makeLazyLoadDB( e, 'd:/temp/fakelib/tools/help/tools') # modify tools.rd*
>> test> e1 <- new.env()
>> test> lazyLoad( 'd:/temp/fakelib/tools/help/tools', e1)
>> test> as.list( e1) # try to force evaluation...
>> Error in as.list.environment(e1) :
>>   cannot allocate memory block of size 2.7 Gb
>> test>
>> test> Rd_db( 'tools', 'd:/temp/fakelib') # no probs !?
>>
>>
>> 2. Sometimes 'fetchRdDB' or 'lazyLoad' will fail in one R session, but will work in a fresh session on exactly the same files. For example, after restarting R, the previous commands involving 'e1' work fine.
>>
>> Mark
>>
>>> Duncan Murdoch
>>>
>>>>> I'm encountering problems when making lazy-loadable databases of the output from 'parse_Rd'. The lazy-
>>>>> load database is of seemingly limitless size when I try to reload it... Admittedly I am using functions
>>>>> that I'm not really supposed to use, which is why this isn't a bug report, but there does seem to be
>>>>> something strange going on; my code is very similar to code that lives inside
>>>>> 'tools:::.install_package_Rd_objects'. The problems occur with just-released R2.10.0 on Windows.
>>>> object.size() has problems when working on Rd objects, because it counts
>>>> every environment separately, even though they may all be references to
>>>> the same one. I haven't looked at your code, but that could be a problem.
>>>>
>>>>
>>> Mark
>>>
>>> ________________________________________
>>> From: Duncan Murdoch [murdoch at stats.uwo.ca]
>>> Sent: 31 October 2009 08:59
>>> To: Bravington, Mark (CMIS, Hobart)
>>> Cc: r-devel at r-project.org
>>> Subject: Re: [Rd] parse_Rd and/or lazyload problem
>>>
>>> On 30/10/2009 12:50 AM, Mark.Bravington at csiro.au wrote:
>>>> I'm encountering problems when making lazy-loadable databases of the output from 'parse_Rd'. The lazy-load database is of seemingly limitless size when I try to reload it... Admittedly I am using functions that I'm not really supposed to use, which is why this isn't a bug report, but there does seem to be something strange going on; my code is very similar to code that lives inside 'tools:::.install_package_Rd_objects'. The problems occur with just-released R2.10.0 on Windows.
>>> object.size() has problems when working on Rd objects, because it counts
>>> every environment separately, even though they may all be references to
>>> the same one. I haven't looked at your code, but that could be a problem.
>>>
>>> Duncan Murdoch
>>>
>>>> The examples below use files which can be found at ftp://ftp.csiro.au/MarkBravington, but you'll obviously need to modify the paths. The file "scrunge.Rd" is just "Rdiff.Rd" from the 'tools' package. The file "fakepack.7z" should unzip to create a fake package with a DESCRIPTION file and a "man" directory that contains two Rd files.
>>>>
>>>> Transcript of first example:
>>>> eglist <- list( scrunge=tools:::prepare_Rd(  'd:/temp/scrunge.Rd',
>>>>     defines=.Platform$OS.type, stages='install', warningCalls=FALSE))
>>>> tools:::makeLazyLoadDB( eglist, 'd:/temp/ll', compress=FALSE)
>>>> tools:::fetchRdDB( 'd:/temp/ll')
>>>> # Error: cannot allocate vector of size 1.4 Gb
>>>>
>>>> The second example triggers an error with 'tools:::.install_package_Rd_objects' itself. It doesn't happen the first time, but frequently does after the Rd files have been changed. Transcript:
>>>>
>>>> # Make sure d:/temp/help/ is empty, then
>>>> test> tools:::.install_package_Rd_objects( 'd:/temp/fakepack', 'd:/temp')
>>>> test> tools:::fetchRdDB( 'd:/temp/help/temp')
>>>> # All good. Next, I *removed* one of the two Rd files in "d:/temp/fakepack/man", ...
>>>> # ...deleted "d:/temp/help/temp*", and tried again
>>>> test> tools:::.install_package_Rd_objects( 'd:/temp/fakepack', 'd:/temp')
>>>> test> tools:::fetchRdDB( 'd:/temp/help/temp')
>>>> Warning: Reached total allocation of 1535Mb: see help(memory.size)
>>>> Warning: Reached total allocation of 1535Mb: see help(memory.size)
>>>> Warning: Reached total allocation of 1535Mb: see help(memory.size)
>>>> Warning: Reached total allocation of 1535Mb: see help(memory.size)
>>>> Error: cannot allocate vector of size 1.9 Gb
>>>> # Or on other occasions I get
>>>> Error: internal error -3 in R_decompress1
>>>>
>>>> Mark Bravington
>>>> CSIRO CMIS
>>>> Hobart
>>>> Australia
>>>>
>>>> --please do not edit the information below--
>>>> Version:
>>>>  platform = i386-pc-mingw32
>>>>  arch = i386
>>>>  os = mingw32
>>>>  system = i386, mingw32
>>>>  status =
>>>>  major = 2
>>>>  minor = 10.0
>>>>  year = 2009
>>>>  month = 10
>>>>  day = 26
>>>>  svn rev = 50208
>>>>  language = R
>>>>  version.string = R version 2.10.0 (2009-10-26)
>>>> Windows XP (build 2600) Service Pack 2
>>>> Locale:
>>>> LC_COLLATE=English_Australia.1252;LC_CTYPE=English_Australia.1252;LC_MONETARY=English_Australia.1252;LC_NUMERIC=C;LC_TIME=English_Australia.1252
>>>> Search Path:
>>>>  .GlobalEnv, ROOT, package:grDevices, package:ad, package:chstuff, package:handy2, package:tweedie, package:statmod, package:handy, package:debug, package:mvbutils, mvb.session.info, package:tools, package:tcltk, package:stats, package:graphics, package:utils, package:methods, Autoloads, package:base
>>>> ______________________________________________
>>>> R-devel at r-project.org mailing list
>>>> https://stat.ethz.ch/mailman/listinfo/r-devel