[Bioc-devel] lazyData

[Kasper Daniel Hansen]

With LazyData true you indeed don't load the data until it is available.
My guess, from skimming the code extremely fast, is that the extreme
requirements (memory and time) during installation is because the data
objects needs to get loaded and somehow modified for this to happen.

Re. the global environment: if my package has an object TEST, and LazyData
is TRUE, when I do (say)
  data(TEST)
or use TEST somehow, TEST doesn't exists in the Global environment.  But if
LazyData is FALSE and I do data(TEST), TEST gets copied into the Global
environment, which is kind of irritating when it is annotation data because
it seems fragile to me (perhaps it is not).

Best,
Kasper

On Fri, Jul 29, 2016 at 3:38 PM, Martin Morgan <
martin.morgan at roswellpark.org> wrote:

> On 07/18/2016 10:52 AM, Kasper Daniel Hansen wrote:
>
>> This is a report on my testing with lazyData turned on and off wrt.
>> installation time and memory requirements.  It turns out that using
>> lazyData dramatically increases memory consumption and time for a
>> (admittedly large) annotation package.  Perhaps this is something we
>> should
>> think about wrt. annotation and data packages.
>>
>> Test example is
>>   IlluminaHumanMethylationEPICanno.ilm10b2.hg19
>> an annotation package for minfi.  The .tar.gz for the this package is 113
>> so its not small.
>>
>> I have explored using
>>   LazyData: yes/no in DESCRIPTION
>>   adding a single line data/datalist file containing the objects in the
>> package
>>
>> What follows are timings and memory consumption of R CMD build + INSTALL
>> on
>> my Mac laptop using an SSD drive.
>>
>>
>>   LazyData: yes
>>   datalist: no
>>   285 seconds
>>   3.22 GB (values as high as 3.8GB seen)
>>
>>   LazyData: no
>>   datalist: no
>>   81s
>>   1.64 GB
>>
>>   LazyData: no
>>   datalist: yes
>>   19s
>>   0.38 GB
>>
>
> Hi Kasper -- I have to admit my ignorance on the miracle of lazy data. Can
> you clarify what one gains from LazyData? I kind of though that with
> LazyData: true the data was only loaded when needed, but that doesn't seem
> consistent with the picture you paint above? Also, what's the discussion
> about global variables?
>
> Martin
>
>
>> (following combination is not mentioned by R-exts, and while it still uses
>> tons of memory, it seems to be 1 minute faster; redid measuring once to
>> confirm this)
>>   LazyData: yes
>>   datalist: yes
>>   226 s
>>   3.26 GB (values as high as 3.9GB seen)
>>
>> Make the data LazyLoaded is pretty nice; one thing is it avoids polluting
>> the global environment.
>>
>> But it seems that it would be worthwhile to consider if some of this could
>> be done prior to the package build time.  Perhaps not, but for sure we are
>> spending resources on the building and installing of this by the build
>> system.
>>
>> I started going down this route because my Travis build starting being
>> killed due to 3+GB being used. I really don't like turning off LazyLoad
>> because of the global environment issue, but the number are kind of
>> extreme
>> here.
>>
>> Best,
>> Kasper
>>
>>         [[alternative HTML version deleted]]
>>
>> _______________________________________________
>> Bioc-devel at r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/bioc-devel
>>
>>
>
> This email message may contain legally privileged and/or confidential
> information.  If you are not the intended recipient(s), or the employee or
> agent responsible for the delivery of this message to the intended
> recipient(s), you are hereby notified that any disclosure, copying,
> distribution, or use of this email message is prohibited.  If you have
> received this message in error, please notify the sender immediately by
> e-mail and delete this email message from your computer. Thank you.
>

	[[alternative HTML version deleted]]