[R] Error: cannot allocate vector of size xxx Mb

[Ralf B]

Thank you for such a careful and thorough analysis of the problem and
your comparison with your configuration. I very much appreciate.
For completeness and (perhaps) further comparison, I have executed
'version' and sessionInfo() as well:


> version
               _
platform       i386-pc-mingw32
arch           i386
os             mingw32
system         i386, mingw32
status         RC
major          2
minor          10.0
year           2009
month          10
day            25
svn rev        50206
language       R
version.string R version 2.10.0 RC (2009-10-25 r50206)
> sessionInfo()
R version 2.10.0 RC (2009-10-25 r50206)
i386-pc-mingw32

locale:
[1] LC_COLLATE=English_United States.1252
[2] LC_CTYPE=English_United States.1252
[3] LC_MONETARY=English_United States.1252
[4] LC_NUMERIC=C
[5] LC_TIME=English_United States.1252

attached base packages:
 [1] splines   stats4    grid      stats     graphics  grDevices utils
 [8] datasets  methods   base

other attached packages:
 [1] flexmix_2.2-7     multcomp_1.1-7    survival_2.35-8   mvtnorm_0.9-9
 [5] modeltools_0.2-16 lattice_0.18-3    car_1.2-16        psych_1.0-88
 [9] nortest_1.0       gplots_2.8.0      caTools_1.10      bitops_1.0-4.1
[13] gdata_2.8.0       gtools_2.6.2      ggplot2_0.8.7     digest_0.4.2
[17] reshape_0.8.3     plyr_0.1.9        proto_0.3-8       RJDBC_0.1-5
[21] rJava_0.8-2       DBI_0.2-5

loaded via a namespace (and not attached):
[1] tools_2.10.0

> memory.limit()
[1] 2047



Also, the example i presented was a simplified reproduction of the
real data structure. My real data structure does not have reused
vectors. I merely wanted to show the error occurring when processing
large vectors into data frames and then binding these data frames
together. I hope this additional information helps. I might add that I
am running this in StatET under Eclipse with 512 MB of allocated RAM
in the environment.

Besides adding more memory, can you spot simple ways of how memory use
can be improved? I know that I am running quite a bit of baggage.
Unfortunately my script is rather comprehensive and my example is
really just a simplified part that I created to reproduce the problem.

Thanks,
Ralf






On Thu, Aug 5, 2010 at 4:44 AM, Petr PIKAL <petr.pikal at precheza.cz> wrote:
> Hi
>
> r-help-bounces at r-project.org napsal dne 05.08.2010 09:53:21:
>
>> I am dealing with very large data frames, artificially created with
>> the following code, that are combined using rbind.
>>
>>
>> a <- rnorm(5000000)
>> b <- rnorm(5000000)
>> c <- rnorm(5000000)
>> d <- rnorm(5000000)
>> first <- data.frame(one=a, two=b, three=c, four=d)
>> second <- data.frame(one=d, two=c, three=b, four=a)
>
> Up to this point there is no error on my system
>
>> version
>               _
> platform       i386-pc-mingw32
> arch           i386
> os             mingw32
> system         i386, mingw32
> status         Under development (unstable)
> major          2
> minor          12.0
> year           2010
> month          05
> day            31
> svn rev        52164
> language       R
> version.string R version 2.12.0 Under development (unstable) (2010-05-31
> r52164)
>
>> sessionInfo()
> R version 2.12.0 Under development (unstable) (2010-05-31 r52164)
> Platform: i386-pc-mingw32/i386 (32-bit)
>
> attached base packages:
> [1] stats     grDevices datasets  utils     graphics  methods   base
>
> other attached packages:
> [1] lattice_0.18-8 fun_1.0
>
> loaded via a namespace (and not attached):
> [1] grid_2.12.0  tools_2.12.0
>
>> rbind(first, second)
>
> Although size of first and second is only roughly 160 MB their
> concatenation probably consumes all remaining memory space as you already
> have a-d first and second in memory.
>
> Regards
> Petr
>
>>
>> which results in the following error for each of the statements:
>>
>> > a <- rnorm(5000000)
>> Error: cannot allocate vector of size 38.1 Mb
>> > b <- rnorm(5000000)
>> Error: cannot allocate vector of size 38.1 Mb
>> > c <- rnorm(5000000)
>> Error: cannot allocate vector of size 38.1 Mb
>> > d <- rnorm(5000000)
>> Error: cannot allocate vector of size 38.1 Mb
>> > first <- data.frame(one=a, two=b, three=c, four=d)
>> Error: cannot allocate vector of size 38.1 Mb
>> > second <- data.frame(one=d, two=c, three=b, four=a)
>> Error: cannot allocate vector of size 38.1 Mb
>> > rbind(first, second)
>>
>> When running memory.limit() I am getting this:
>>
>> memory.limit()
>> [1] 2047
>>
>> Which shows me that I have 2 GB of memory available. What is wrong?
>> Shouldn't 38 MB be very feasible?
>>
>> Best,
>> Ralf
>>
>> ______________________________________________
>> R-help at r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
>> and provide commented, minimal, self-contained, reproducible code.
>
>