The optim function is very resource hungry. I have had similar problems in 
the past when dealing with extremely large datasets.

What is perhaps happening is that each 'step' of the optimization 
algorithm stores some info so that it can compare to the next 'step', and 
while the original vector may only be a few Mb of data, over many 
iterations a huge amount memory is allocated to the optimization steps.

Maybe look at the control options under ?optim, particularly stuff like 
trace, fnscale, ndeps, etc. that may cut down on the amount of data being 
stored each step as well as the number of steps needed.

Good luck!
--------------------------------------
Jonathan P. Daily
Technician - USGS Leetown Science Center
11649 Leetown Road
Kearneysville WV, 25430
(304) 724-4480
"Is the room still a room when its empty? Does the room,
 the thing itself have purpose? Or do we, what's the word... imbue it."
     - Jubal Early, Firefly



From:
Lorenzo Cattarino <l.cattarino@uq.edu.au>
To:
David Winsemius <dwinsemius@comcast.net>, Peter Langfelder 
<peter.langfelder@gmail.com>
Cc:
r-help@r-project.org
Date:
11/03/2010 03:26 AM
Subject:
Re: [R] memory allocation problem
Sent by:
r-help-bounces@r-project.org



Thanks for all your suggestions,

This is what I get after removing all the other (not useful) objects and
run my code:

> getsizes()
                [,1]
org_results 47240832
myfun          11672
getsizes        4176
SS              3248
coeff            168
<NA>              NA
<NA>              NA
<NA>              NA
<NA>              NA
<NA>              NA

> est_coeff <- optim(coeff,SS, steps=org_results$no.steps,
Range=org_results$Range, H1=org_results$H1, H2=org_results$H2,
p=org_results$p)
Error: cannot allocate vector of size 5.0 Mb
In addition: Warning messages:
1: In optim(coeff, SS, steps = org_results$no.steps, Range =
org_results$Range,  :
  Reached total allocation of 4055Mb: see help(memory.size)
2: In optim(coeff, SS, steps = org_results$no.steps, Range =
org_results$Range,  :
  Reached total allocation of 4055Mb: see help(memory.size)
3: In optim(coeff, SS, steps = org_results$no.steps, Range =
org_results$Range,  :
  Reached total allocation of 4055Mb: see help(memory.size)
4: In optim(coeff, SS, steps = org_results$no.steps, Range =
org_results$Range,  :
  Reached total allocation of 4055Mb: see help(memory.size)
>

It seems that R is using all the default availabe memory (4 GB, which is
the RAM of my processor).

> memory.limit()
[1] 4055
> memory.size()
[1] 4049.07
>

My dataframe has a size of 47240832 bytes, or about 45 Mb. So it should
not be a problem in terms of memory usage?

I do not understand what is going on.

Thanks for your help anyway

Lorenzo

-----Original Message-----
From: David Winsemius [mailto:dwinsemius@comcast.net] 
Sent: Wednesday, 3 November 2010 12:48 PM
To: Lorenzo Cattarino
Cc: r-help@r-project.org
Subject: Re: [R] memory allocation problem

Restart your computer. (Yeah, I know that what the help-desk always 
says.)
Start R before doing anything else.

Then run your code in a clean session. Check ls() oafter starte up to 
make sure you don't have a bunch f useless stuff in your .Rdata 
file.   Don't load anything that is not germane to this problem.  Use 
this function to see what sort of space issues you might have after 
loading objects:

  getsizes <- function() {z <- sapply(ls(envir=globalenv()),
                                 function(x) object.size(get(x)))
                (tmp <- as.matrix(rev(sort(z))[1:10]))}

Then run your code.

-- 
David.

On Nov 2, 2010, at 10:13 PM, Lorenzo Cattarino wrote:

> I would also like to include details on my R version
>
>
>
>> version              _
>
> platform       x86_64-pc-mingw32
> arch           x86_64
>
> os             mingw32
> system         x86_64, mingw32
> status
> major          2
> minor          11.1
> year           2010
> month          05
> day            31
> svn rev        52157
> language       R
> version.string R version 2.11.1 (2010-05-31)
>
> from FAQ 2.9
>
(http://cran.r-project.org/bin/windows/base/rw-FAQ.html#There-seems-to-b
> e-a-limit-on-the-memory-it-uses_0021
>
<http://cran.r-project.org/bin/windows/base/rw-FAQ.html#There-seems-to-b
> e-a-limit-on-the-memory-it-uses_0021> ) it says that:
> "For a 64-bit build, the default is the amount of RAM"
>
> So in my case the amount of RAM would be 4 GB. R should be able to
> allocate a vector of size 5 Mb without me typing any command (either 
> as
> memory.limit() or appended string in the target path), is that right?
>
>
>
> From: Lorenzo Cattarino
> Sent: Wednesday, 3 November 2010 10:55 AM
> To: 'r-help@r-project.org'
> Subject: memory allocation problem
>
>
>
> I forgot to mention that I am using windows 7 (64-bit) and the R 
> version
> 2.11.1 (64-bit)
>
>
>
> From: Lorenzo Cattarino
>
> I am trying to run a non linear parameter optimization using the
> function optim() and I have problems regarding memory allocation.
>
> My data are in a dataframe with 9 columns. There are 656100 rows.
>
>> head(org_results)
>
> comb.id   p H1 H2 Range Rep no.steps      dist         aver.hab.amount
>
> 1       1       0.1  0  0     1            100        0
> 0.2528321              0.1393901
>
> 2       1       0.1  0  0     1            100        0
> 0.4605934              0.1011841
>
> 3       1       0.1  0  0     1            100        4
> 3.4273670              0.1052789
>
> 4       1       0.1  0  0     1            100        4
> 2.8766364              0.1022138
>
> 5       1       0.1  0  0     1            100        0
> 0.3496872              0.1041056
>
> 6       1       0.1  0  0     1            100        0
> 0.1050840              0.3572036
>
>> est_coeff <- optim(coeff,SS, steps=org_results$no.steps,
> Range=org_results$Range, H1=org_results$H1, H2=org_results$H2,
> p=org_results$p)
>
> Error: cannot allocate vector of size 5.0 Mb
>
> In addition: Warning messages:
>
> 1: In optim(coeff, SS, steps = org_results$no.steps, Range =
> org_results$Range,  : Reached total allocation of 10000Mb: see
> help(memory.size)
>
> 2: In optim(coeff, SS, steps = org_results$no.steps, Range =
> org_results$Range,  : Reached total allocation of 10000Mb: see
> help(memory.size)
>
> 3: In optim(coeff, SS, steps = org_results$no.steps, Range =
> org_results$Range,  : Reached total allocation of 10000Mb: see
> help(memory.size)
>
> 4: In optim(coeff, SS, steps = org_results$no.steps, Range =
> org_results$Range,  : Reached total allocation of 10000Mb: see
> help(memory.size)
>
>> memory.size()
>
> [1] 9978.19
>
>> memory.limit()
>
> [1] 10000
>
>>
>
> I know that I am not sending reproducible codes but I was hoping that
> you could help me understand what is going on. I set a maximum limit 
> of
> 10000 mega byte (by writing this string --max-mem-size=10000M after 
> the
> target path, right click on R icon, shortcut tab). And R is telling me
> that it cannot allocate a vector of size 5 Mb???
>
>

David Winsemius, MD
West Hartford, CT

______________________________________________
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide 
http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.



	[[alternative HTML version deleted]]

