[Rd] [External] memory consumption of nested (un)serialize of sys.frames()

Fri Apr 9 02:43:51 CEST 2021

Looks like this is an unavoidable interaction between the way source
references and lazy loading are implemented. The link back to the
crash_dumps environment comes though source references on an
unevaluated argument promise. Creating a fresh environment is
.onLoad() avoids this and is probably your best bet.

Having an option to serialize without source references might be nice
but would probably not be high enough on anyone's priority list to get
done anytime soon.

Best,

luke

On Thu, 8 Apr 2021, luke-tierney using uiowa.edu wrote:

> I see that now also. Not sure yet what is going on.
>
> One work-around that may work for you is to create a fresh crash dump
> in a .onLoad function; somehting like
>
> crash_dumps <- NULL
> .onLoad <- function(...) crash_dumps <<- new.env()
>
> Best,
>
> luke
>
> On Wed, 7 Apr 2021, Andreas Kersting wrote:
>
>> Hi Dirk, hi Luke,
>> 
>> Thanks for checking!
>> 
>> I could narrow it down further. I have the issue only if I install 
>> --with-keep.source, i.e.
>> 
>> R CMD INSTALL --with-keep.source dumpTest
>> 
>> Since this is the default in RStudio when clicking "Install and Restart", I 
>> was always having the issue - also from base R. If I install using e.g. 
>> devtools::install_github() directly it is also fine for me.
>> 
>> Could you please confirm? Thanks!
>> 
>> Regards,
>> Andreas
>> 
>> 2021-04-07 16:20 GMT+02:00 "Dirk Eddelbuettel" <edd using debian.org>:
>>> 
>>> On 7 April 2021 at 16:06, Andreas Kersting wrote:
>>> | Hi Luke,
>>> |
>>> | Please see https://github.com/akersting/dumpTest for the package.
>>> |
>>> | Here a session showing my issue:
>>> |
>>> | > library(dumpTest)
>>> | > sessionInfo()
>>> | R version 4.0.5 (2021-03-31)
>>> | Platform: x86_64-pc-linux-gnu (64-bit)
>>> | Running under: Debian GNU/Linux 10 (buster)
>>> |
>>> | Matrix products: default
>>> | BLAS:   /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.8.0
>>> | LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.8.0
>>> |
>>> | locale:
>>> |  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C
>>> |  [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8
>>> |  [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8
>>> |  [7] LC_PAPER=en_US.UTF-8       LC_NAME=C
>>> |  [9] LC_ADDRESS=C               LC_TELEPHONE=C
>>> | [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
>>> |
>>> | attached base packages:
>>> | [1] stats     graphics  grDevices utils     datasets  methods   base
>>> |
>>> | other attached packages:
>>> | [1] dumpTest_0.1.0
>>> |
>>> | loaded via a namespace (and not attached):
>>> | [1] compiler_4.0.5
>>> | > for (i in 1:100) {
>>> | +   print(i)
>>> | +   print(system.time(f()))
>>> | + }
>>> | [1] 1
>>> |    user  system elapsed
>>> |   0.028   0.004   0.034
>>> | [1] 2
>>> |    user  system elapsed
>>> |   0.067   0.008   0.075
>>> | [1] 3
>>> |    user  system elapsed
>>> |   0.176   0.000   0.176
>>> | [1] 4
>>> |    user  system elapsed
>>> |   0.335   0.012   0.349
>>> | [1] 5
>>> |    user  system elapsed
>>> |   0.745   0.023   0.770
>>> | [1] 6
>>> |    user  system elapsed
>>> |   1.495   0.060   1.572
>>> | [1] 7
>>> |    user  system elapsed
>>> |   2.902   0.136   3.040
>>> | [1] 8
>>> |    user  system elapsed
>>> |   5.753   0.272   6.034
>>> | [1] 9
>>> |    user  system elapsed
>>> |  11.807   0.708  12.597
>>> | [1] 10
>>> | ^C
>>> | Timing stopped at: 6.638 0.549 7.214
>>> |
>>> | I had to interrupt in iteration 10 because I was running low on RAM.
>>> 
>>> No issue here.  Ubuntu 20.10, R 4.0.5 'from CRAN' i.e. Michael's PPA build
>>> off my Debian package, hence instrumentation as in the Debian package.
>>> 
>>> edd using rob:~$ installGithub.r akersting/dumpTest
>>> Using github PAT from envvar GITHUB_PAT
>>> Downloading GitHub repo akersting/dumpTest using HEAD
>>> ✔  checking for file 
>>> ‘/tmp/remotes3f9af733166ccd/akersting-dumpTest-3bed8e2/DESCRIPTION’ ...
>>> ─  preparing ‘dumpTest’:
>>> ✔  checking DESCRIPTION meta-information ...
>>> ─  checking for LF line-endings in source and make files and shell scripts
>>> ─  checking for empty or unneeded directories
>>> ─  building ‘dumpTest_0.1.0.tar.gz’
>>> 
>>> Installing package into ‘/usr/local/lib/R/site-library’
>>> (as ‘lib’ is unspecified)
>>> * installing *source* package ‘dumpTest’ ...
>>> ** using staged installation
>>> ** R
>>> ** byte-compile and prepare package for lazy loading
>>> ** help
>>> No man pages found in package  ‘dumpTest’
>>> *** installing help indices
>>> ** building package indices
>>> ** testing if installed package can be loaded from temporary location
>>> ** testing if installed package can be loaded from final location
>>> ** testing if installed package keeps a record of temporary installation 
>>> path
>>> * DONE (dumpTest)
>>> edd using rob:~$ Rscript -e 'system.time({for (i in 1:100) dumpTest::f()})'
>>>    user  system elapsed
>>>   0.481   0.019   0.500
>>> edd using rob:~$
>>> 
>>> (I also ran the variant you showed with the dual print statements, it just
>>> consumes more screen real estate and ends on
>>> 
>>> [...]
>>> [1] 97
>>>    user  system elapsed
>>>   0.004   0.000   0.005
>>> [1] 98
>>>    user  system elapsed
>>>   0.004   0.000   0.005
>>> [1] 99
>>>    user  system elapsed
>>>   0.004   0.000   0.004
>>> [1] 100
>>>    user  system elapsed
>>>   0.005   0.000   0.005
>>> edd using rob:~$ )
>>> 
>>> Dirk
>>> 
>>> --
>>> https://dirk.eddelbuettel.com | @eddelbuettel | edd using debian.org
>>> 
>
>

-- 
Luke Tierney
Ralph E. Wareham Professor of Mathematical Sciences
University of Iowa                  Phone:             319-335-3386
Department of Statistics and        Fax:               319-335-3017
    Actuarial Science
241 Schaeffer Hall                  email:   luke-tierney using uiowa.edu
Iowa City, IA 52242                 WWW:  http://www.stat.uiowa.edu