[Rd] parse_Rd and/or lazyload problem

[Duncan Murdoch]

On 30/10/2009 10:58 PM, Mark.Bravington at csiro.au wrote:
> The errors are generated inside lazyLoadDBfetch; I don't call object.size, so unfortunately that's not the issue (unless o.s. is called somewhere inside the .Primitive for lazyLoadDBfetch).

Does this happen in R-patched?  I've seen similar errors in 2.10.0, but 
not in a current build.  (Still haven't got to your code, this was in 
mine.  I'm reluctant to spend time on code that is messing with 
internals, because you might be using things in a way not intended by 
the author.  Now, if you can show me some code that demonstrates the 
problem without using internals directly, I'll follow up.)

Duncan Murdoch

> 
>>> I'm encountering problems when making lazy-loadable databases of the output from 'parse_Rd'. The lazy-
>>> load database is of seemingly limitless size when I try to reload it... Admittedly I am using functions
>>> that I'm not really supposed to use, which is why this isn't a bug report, but there does seem to be
>>> something strange going on; my code is very similar to code that lives inside
>>> 'tools:::.install_package_Rd_objects'. The problems occur with just-released R2.10.0 on Windows.
>> object.size() has problems when working on Rd objects, because it counts
>> every environment separately, even though they may all be references to
>> the same one. I haven't looked at your code, but that could be a problem.
>>
>>
> 
> Mark
> 
> ________________________________________
> From: Duncan Murdoch [murdoch at stats.uwo.ca]
> Sent: 31 October 2009 08:59
> To: Bravington, Mark (CMIS, Hobart)
> Cc: r-devel at r-project.org
> Subject: Re: [Rd] parse_Rd and/or lazyload problem
> 
> On 30/10/2009 12:50 AM, Mark.Bravington at csiro.au wrote:
>> I'm encountering problems when making lazy-loadable databases of the output from 'parse_Rd'. The lazy-load database is of seemingly limitless size when I try to reload it... Admittedly I am using functions that I'm not really supposed to use, which is why this isn't a bug report, but there does seem to be something strange going on; my code is very similar to code that lives inside 'tools:::.install_package_Rd_objects'. The problems occur with just-released R2.10.0 on Windows.
> 
> object.size() has problems when working on Rd objects, because it counts
> every environment separately, even though they may all be references to
> the same one. I haven't looked at your code, but that could be a problem.
> 
> Duncan Murdoch
> 
>> The examples below use files which can be found at ftp://ftp.csiro.au/MarkBravington, but you'll obviously need to modify the paths. The file "scrunge.Rd" is just "Rdiff.Rd" from the 'tools' package. The file "fakepack.7z" should unzip to create a fake package with a DESCRIPTION file and a "man" directory that contains two Rd files.
>>
>> Transcript of first example:
>> eglist <- list( scrunge=tools:::prepare_Rd(  'd:/temp/scrunge.Rd',
>>     defines=.Platform$OS.type, stages='install', warningCalls=FALSE))
>> tools:::makeLazyLoadDB( eglist, 'd:/temp/ll', compress=FALSE)
>> tools:::fetchRdDB( 'd:/temp/ll')
>> # Error: cannot allocate vector of size 1.4 Gb
>>
>> The second example triggers an error with 'tools:::.install_package_Rd_objects' itself. It doesn't happen the first time, but frequently does after the Rd files have been changed. Transcript:
>>
>> # Make sure d:/temp/help/ is empty, then
>> test> tools:::.install_package_Rd_objects( 'd:/temp/fakepack', 'd:/temp')
>> test> tools:::fetchRdDB( 'd:/temp/help/temp')
>> # All good. Next, I *removed* one of the two Rd files in "d:/temp/fakepack/man", ...
>> # ...deleted "d:/temp/help/temp*", and tried again
>> test> tools:::.install_package_Rd_objects( 'd:/temp/fakepack', 'd:/temp')
>> test> tools:::fetchRdDB( 'd:/temp/help/temp')
>> Warning: Reached total allocation of 1535Mb: see help(memory.size)
>> Warning: Reached total allocation of 1535Mb: see help(memory.size)
>> Warning: Reached total allocation of 1535Mb: see help(memory.size)
>> Warning: Reached total allocation of 1535Mb: see help(memory.size)
>> Error: cannot allocate vector of size 1.9 Gb
>> # Or on other occasions I get
>> Error: internal error -3 in R_decompress1
>>
>> Mark Bravington
>> CSIRO CMIS
>> Hobart
>> Australia
>>
>> --please do not edit the information below--
>> Version:
>>  platform = i386-pc-mingw32
>>  arch = i386
>>  os = mingw32
>>  system = i386, mingw32
>>  status =
>>  major = 2
>>  minor = 10.0
>>  year = 2009
>>  month = 10
>>  day = 26
>>  svn rev = 50208
>>  language = R
>>  version.string = R version 2.10.0 (2009-10-26)
>> Windows XP (build 2600) Service Pack 2
>> Locale:
>> LC_COLLATE=English_Australia.1252;LC_CTYPE=English_Australia.1252;LC_MONETARY=English_Australia.1252;LC_NUMERIC=C;LC_TIME=English_Australia.1252
>> Search Path:
>>  .GlobalEnv, ROOT, package:grDevices, package:ad, package:chstuff, package:handy2, package:tweedie, package:statmod, package:handy, package:debug, package:mvbutils, mvb.session.info, package:tools, package:tcltk, package:stats, package:graphics, package:utils, package:methods, Autoloads, package:base
>> ______________________________________________
>> R-devel at r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-devel