[R-pkg-devel] memory exhausted (limit reached?) and R3.6.0 memory leak in data.table

[luke-tier@ey m@ili@g off uiow@@edu]

The issue with data.table in R 3.5 and R 3.6 is a race condition in
data.table's use of multiple threads via OpenMP. Running with the
environment variable OMP_NUM_THREADS=1 makes the memory problem go
away.

Calling DATAPTR for ALTREP objects, like compact arithmetic sequences,
temporarily disables the GC. Calling DATAPTR from multiple threads
without synchronization creates a race condition that can leave the GC
disabled, so memory use grows. It's not a leak: signaling a user
interrupt, or anything that triggers a jump to top level, re-enables
the GC.

The issue exists in R 3.5 as well but is a little less apparent since,
due to an oversight, byte compiled code often does not produce compact
arithmetic sequences. This will hopefully be addressed in R 3.5.1.

Best,

luke

On Mon, 14 May 2018, Bailey, Paul wrote:

> A package I'm submitting depends on data.table, which currently has a memory leak on Windows and so I'm getting rejected because of "memory exhausted (limit reached?)". I'm unsure how to proceed. It seems the issue is with R3.6.0 and data.table, so I don't think users would have an immediate issue if my package were updated. I could limit the tests that use data.table to keep under a memory size, but how much memory does CRAN allocate on Windows, and how do I enforce that?
>
> You can see the memory leak issue on GitHub at https://github.com/Rdatatable/data.table/issues/2866
>
> Best,
> Paul
>
> ______________________________________________
> R-package-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-package-devel
>

-- 
Luke Tierney
Ralph E. Wareham Professor of Mathematical Sciences
University of Iowa                  Phone:             319-335-3386
Department of Statistics and        Fax:               319-335-3017
    Actuarial Science
241 Schaeffer Hall                  email:   luke-tierney at uiowa.edu
Iowa City, IA 52242                 WWW:  http://www.stat.uiowa.edu