[R-sig-Geo] memory Usage setting

[Didier Leibovici]

These days in GIS on may have to manipulate big datasets or arrays.

What is the best way of tuning the R memory ?
or how can we not worry about the memory usage
i.e. could it that if it is needed it will swap on the disk (the system 
I suppose)
That means declaring even more RAM than existing ...

Here I am on WINDOWS I have a 4Gb
my aim was to have an array of dim 298249 12 10 22 but that's 2.9Gb

Here is some tries to find the size I could have :

First of all I did:
 > memory.limit(size=3500)
NULL
 > memory.limit(size=NA)/2^20
[1] 3500

 > memory.size(max=TRUE)/2^20
[1] 341.125


 >ZZ=array(rep(0,round(298249/16)*12*10*22),c(round(298249/16),12,10,22))
 > gc()
            used  (Mb) gc trigger   (Mb)  max used   (Mb)
Ncells   835490  22.4    1368491   36.6    954139   25.5
Vcells 90663806 691.8  154550742 1179.2 139876061 1067.2
 >
 >  298249 *12*10*22
[1] 787377360
 >  dd=as.vector(rep(0,298249 *12*10*22))
Error: cannot allocate vector of size 2.9 Gb
In addition: Warning messages:
1: Reached total allocation of 3500Mb: see help(memory.size)

 >  dd=as.vector(rep(0,round(298249/4) *12*10*22))
Error: cannot allocate vector of size 750.9 Mb
 >  dd=as.vector(rep(0,round(298249/2) *12*10*22))
Error: cannot allocate vector of size 1.5 Gb
 >  dd=as.vector(rep(0,round(298249/9) *12*10*22))
Error: cannot allocate vector of size 333.7 Mb
 >  dd=as.vector(rep(0,round(298249/10) *12*10*22))
Error: cannot allocate vector of size 300.4 Mb
 > object.size(ZZ)
[1] 393698040
 > object.size(ZZ)/2^20
[1] 375.4597
 > gc()
            used  (Mb) gc trigger   (Mb)  max used   (Mb)
Ncells   835535  22.4    1368491   36.6    954139   25.5
Vcells 90663830 691.8  162358279 1238.7 139876061 1067.2
 > rm(ZZ)
 > gc()
            used  (Mb) gc trigger  (Mb)  max used   (Mb)
Ncells   835622  22.4    1368491  36.6    954139   25.5
Vcells 41451603 316.3  129886623 991.0 139876061 1067.2

 >  dd=as.vector(rep(0,round(298249/10) *12*10*22))
Error: cannot allocate vector of size 600.7 Mb
 >
It is also strange that once a dd needed 300.4Mb and then 600.7Mb (?) as 
also I made some room in removing ZZ?

I had already modiied the starting of windows by putting

[operating systems]
multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Windows Server 2003, 
Standard" /noexecute=optout /fastdetect /redirect /5GB
in the boot.ini
which I don't really know if it took into account as the limit is 
greater than the physical RAM of 4GB. ...?

would it be easier using Linux ?

Thanks

-- 
Dr Didier Leibovici
	http://www.nottingham.ac.uk/cgs/leibovici.shtml
Centre for Geospatial Science
Sir Clive Granger Building
University of Nottingham,
University Park
Nottingham NG7 2RD, UK
	Tel: +44 - (0)115 84 66058   Fax: +44 (0)115 95 15249

This message has been checked for viruses but the contents of an attachment
may still contain software viruses, which could damage your computer system:
you are advised to perform your own checks. Email communications with the
University of Nottingham may be monitored as permitted by UK legislation.