[R] Why does a 2 GB RData file exceed my 16GB memory limit when reading it in?

[David Jones]

Thank you Uwe, John, and Bert - this is very helpful context.

If it helps inform the discussion, to address John and Bert's
questions - I actually had less memory free when I originally ran the
analyses and saved the workspace, than when I read in the data back in
later on (I rebooted in an attempt to free all possible memory before
rereading the workspace back in).



On Wed, Sep 2, 2020 at 1:27 PM John via R-help <r-help using
r-project.org> wrote:

>> On Wed, 2 Sep 2020 13:36:43 +0200
>> Uwe Ligges <ligges using statistik.tu-dortmund.de> wrote:
>>
>> > On 02.09.2020 04:44, David Jones wrote:
>> > > I ran a number of analyses in R and saved the workspace, which
>> > > resulted in a 2GB .RData file. When I try to read the file back
>> > > into R
>> >
>> > Compressed in RData but uncompressed in main memory....
>> >
>> >
>> > > later, it won't read into R and provides the error: "Error: cannot
>> > > allocate vector of size 37 Kb"
>> > >
>> > > This error comes after 1 minute of trying to read things in - I
>> > > presume a single vector sends it over the memory limit. But,
>> > > memory.limit() shows that I have access to a full 16gb of ram on my
>> > > machine (12 GB are free when I try to load the RData file).
>> >
>> > But the data may need more....
>> >
>> >
>> > > gc() shows the following after I receive this error:
>> > >
>> > > used (Mb) gc trigger (Mb) max used (Mb)
>> > > Ncells 623130 33.3 4134347 220.8 5715387 305.3
>> > > Vcells 1535682 11.8 883084810 6737.5 2100594002 16026.3
>> >
>> > So 16GB were used when R gave up.
>> >
>> > Best,
>> > Uwe Ligges
>>
>> For my own part, looking at the OP's question, it does seem curious
>> that R could write that .RData file, but on the same system not be able
>> to reload something it created.  How would that work.  Wouldn't the
>> memory limit have been exceeded BEFORE the the .RData file was written
>> the FIRST time?
>>
>> JDougherty


>R experts may give you a detailed explanation, but it is certainly possible
>that the memory available to R when it wrote the file was different than
>when it tried to read it, is it not?

>Bert Gunter

>"The trouble with having an open mind is that people keep coming along and
>sticking things into it."
>-- Opus (aka Berkeley Breathed in his "Bloom County" comic strip )