[R] memory allocation problem
Lorenzo Cattarino
l.cattarino at uq.edu.au
Wed Nov 3 06:13:42 CET 2010
Following on my memory allocation problem...
I tried to run my code on our university HPU facility, requesting 61 GB
of memory, and it still can not allocate a vector of 5 MB of size.
> load('/home/uqlcatta/test_scripts/.RData')
>
> myfun <- function(Range, H1, H2, p, coeff)
+ {
+
-(coeff[1]+coeff[2]*H1+coeff[3]*H2+coeff[4]*p)*exp(-(coeff[5]+coeff[6]*H
1+coeff[7]*H2+coeff[8]*p)*Range)+coeff[9]+coeff[10]*H1+coeff[11]*H2+coef
f[12]*p
+ }
>
> SS <- function(coeff,steps,Range,H1,H2,p)
+ {
+ sum((steps - myfun(Range,H1,H2,p,coeff))^2)
+ }
>
> coeff <- c(1,1,1,1,1,1,1,1,1,1,1,1)
>
> est_coeff <- optim(coeff,SS, steps=org_results$no.steps,
Range=org_results$Range, H1=org_results$H1, H2=org_results$H2,
p=org_results$p)
Error: cannot allocate vector of size 5.0 Mb
Execution halted
May it be a proble of the function?
Any input is very much appreciated
Lorenzo
-----Original Message-----
From: Lorenzo Cattarino
Sent: Wednesday, 3 November 2010 2:22 PM
To: 'David Winsemius'; 'Peter Langfelder'
Cc: r-help at r-project.org
Subject: RE: [R] memory allocation problem
Thanks for all your suggestions,
This is what I get after removing all the other (not useful) objects and
run my code:
> getsizes()
[,1]
org_results 47240832
myfun 11672
getsizes 4176
SS 3248
coeff 168
<NA> NA
<NA> NA
<NA> NA
<NA> NA
<NA> NA
> est_coeff <- optim(coeff,SS, steps=org_results$no.steps,
Range=org_results$Range, H1=org_results$H1, H2=org_results$H2,
p=org_results$p)
Error: cannot allocate vector of size 5.0 Mb
In addition: Warning messages:
1: In optim(coeff, SS, steps = org_results$no.steps, Range =
org_results$Range, :
Reached total allocation of 4055Mb: see help(memory.size)
2: In optim(coeff, SS, steps = org_results$no.steps, Range =
org_results$Range, :
Reached total allocation of 4055Mb: see help(memory.size)
3: In optim(coeff, SS, steps = org_results$no.steps, Range =
org_results$Range, :
Reached total allocation of 4055Mb: see help(memory.size)
4: In optim(coeff, SS, steps = org_results$no.steps, Range =
org_results$Range, :
Reached total allocation of 4055Mb: see help(memory.size)
>
It seems that R is using all the default availabe memory (4 GB, which is
the RAM of my processor).
> memory.limit()
[1] 4055
> memory.size()
[1] 4049.07
>
My dataframe has a size of 47240832 bytes, or about 45 Mb. So it should
not be a problem in terms of memory usage?
I do not understand what is going on.
Thanks for your help anyway
Lorenzo
-----Original Message-----
From: David Winsemius [mailto:dwinsemius at comcast.net]
Sent: Wednesday, 3 November 2010 12:48 PM
To: Lorenzo Cattarino
Cc: r-help at r-project.org
Subject: Re: [R] memory allocation problem
Restart your computer. (Yeah, I know that what the help-desk always
says.)
Start R before doing anything else.
Then run your code in a clean session. Check ls() oafter starte up to
make sure you don't have a bunch f useless stuff in your .Rdata
file. Don't load anything that is not germane to this problem. Use
this function to see what sort of space issues you might have after
loading objects:
getsizes <- function() {z <- sapply(ls(envir=globalenv()),
function(x) object.size(get(x)))
(tmp <- as.matrix(rev(sort(z))[1:10]))}
Then run your code.
--
David.
On Nov 2, 2010, at 10:13 PM, Lorenzo Cattarino wrote:
> I would also like to include details on my R version
>
>
>
>> version _
>
> platform x86_64-pc-mingw32
> arch x86_64
>
> os mingw32
> system x86_64, mingw32
> status
> major 2
> minor 11.1
> year 2010
> month 05
> day 31
> svn rev 52157
> language R
> version.string R version 2.11.1 (2010-05-31)
>
> from FAQ 2.9
>
(http://cran.r-project.org/bin/windows/base/rw-FAQ.html#There-seems-to-b
> e-a-limit-on-the-memory-it-uses_0021
>
<http://cran.r-project.org/bin/windows/base/rw-FAQ.html#There-seems-to-b
> e-a-limit-on-the-memory-it-uses_0021> ) it says that:
> "For a 64-bit build, the default is the amount of RAM"
>
> So in my case the amount of RAM would be 4 GB. R should be able to
> allocate a vector of size 5 Mb without me typing any command (either
> as
> memory.limit() or appended string in the target path), is that right?
>
>
>
> From: Lorenzo Cattarino
> Sent: Wednesday, 3 November 2010 10:55 AM
> To: 'r-help at r-project.org'
> Subject: memory allocation problem
>
>
>
> I forgot to mention that I am using windows 7 (64-bit) and the R
> version
> 2.11.1 (64-bit)
>
>
>
> From: Lorenzo Cattarino
>
> I am trying to run a non linear parameter optimization using the
> function optim() and I have problems regarding memory allocation.
>
> My data are in a dataframe with 9 columns. There are 656100 rows.
>
>> head(org_results)
>
> comb.id p H1 H2 Range Rep no.steps dist aver.hab.amount
>
> 1 1 0.1 0 0 1 100 0
> 0.2528321 0.1393901
>
> 2 1 0.1 0 0 1 100 0
> 0.4605934 0.1011841
>
> 3 1 0.1 0 0 1 100 4
> 3.4273670 0.1052789
>
> 4 1 0.1 0 0 1 100 4
> 2.8766364 0.1022138
>
> 5 1 0.1 0 0 1 100 0
> 0.3496872 0.1041056
>
> 6 1 0.1 0 0 1 100 0
> 0.1050840 0.3572036
>
>> est_coeff <- optim(coeff,SS, steps=org_results$no.steps,
> Range=org_results$Range, H1=org_results$H1, H2=org_results$H2,
> p=org_results$p)
>
> Error: cannot allocate vector of size 5.0 Mb
>
> In addition: Warning messages:
>
> 1: In optim(coeff, SS, steps = org_results$no.steps, Range =
> org_results$Range, : Reached total allocation of 10000Mb: see
> help(memory.size)
>
> 2: In optim(coeff, SS, steps = org_results$no.steps, Range =
> org_results$Range, : Reached total allocation of 10000Mb: see
> help(memory.size)
>
> 3: In optim(coeff, SS, steps = org_results$no.steps, Range =
> org_results$Range, : Reached total allocation of 10000Mb: see
> help(memory.size)
>
> 4: In optim(coeff, SS, steps = org_results$no.steps, Range =
> org_results$Range, : Reached total allocation of 10000Mb: see
> help(memory.size)
>
>> memory.size()
>
> [1] 9978.19
>
>> memory.limit()
>
> [1] 10000
>
>>
>
> I know that I am not sending reproducible codes but I was hoping that
> you could help me understand what is going on. I set a maximum limit
> of
> 10000 mega byte (by writing this string --max-mem-size=10000M after
> the
> target path, right click on R icon, shortcut tab). And R is telling me
> that it cannot allocate a vector of size 5 Mb???
>
>
David Winsemius, MD
West Hartford, CT
More information about the R-help
mailing list