[R-sig-Geo] memory issues w/ raster processing

[Robert J. Hijmans]

Marcia,

If the problem persists you can try to force it to write to disk if it does not.

setOptions(todisk=TRUE)

If it still fails:

setOptions(chunksize=100000)


I would normally not recommend to use these options.

The below worked for me (also with your number of rows and columns,
but then it takes a while!)

library(raster)

s <- list()
r <- raster(ncol=26, nrow=29)
for (i in 1:10) {
	s[i] <- setValues(r, round(runif(ncell(r)+0.5) * 3))
}
s <- stack(s)

f.232 = function(x){
   # adapted from code by Robert Hijmans
   # changes 232  to 222, but does not touch 233 or 223
   a <- which(x == 3)
   a <- subset(a, a > 1 & a < length(x))
   b <- which(x[a-1] == 2 & x[a+1] == 2)
   x[a[b]] <- 2
   return(x)
}

x <- calc(s, f.232)


On Wed, Dec 29, 2010 at 10:00 AM, Marcia Macedo
<marcia.n.macedo at gmail.com> wrote:
> Dear List,
>
> I have been trying to run a simple function on a raster stack (dimensions
> are roughly 75 MB with 2600 samples x 2900 lines x 10 layers). The function
> (adapted from code that Robert posted last week) is as follows:
>
> f.232 = function(x){
>    # adapted from code by Robert Hijmans
>    # changes 232  to 222, but does not touch 233 or 223
>
>    a <- which(x == 3)
>    a <- subset(a, a > 1 & a < length(x))
>    b <- which(x[a-1] == 2 & x[a+1] == 2)
>    x[a[b]] <- 2
>    return(x)
> }
>
> The problem is that the function takes a very long time to run and often
> crashes or gives me memory warnings like this one:
>
> Warning messages:
> 1: In dimnames(data) <- dimnames :
>  Reached total allocation of 7854Mb: see help(memory.size)
> 2: In dimnames(data) <- dimnames :
>  Reached total allocation of 7854Mb: see help(memory.size)
>
> I am confused because this data cube is not that large and I think my system
> should be able to handle it (see specs. below).
>
> *System specs:*
>
> R version 2.12.1 (2010-12-16)
> Platform: x86_64-pc-mingw32/x64 (64-bit)
> Installed memory (RAM): 8 GB
> Processor: Intel Core i3, 2.13 GHz
>
> I must admit that I don't really understand how the raster package manages
> memory, but I was imagining that it would be more efficient to write the
> results to file iteratively (in blocks) while running the calculations
> (rather than writing all results to memory and then writing to
> file)...However, my attempts to adapt the writeValues code in the package
> documentation have not worked (i.e. the output file gets created but never
> written to and the process crashes as described above).
>
> Any thoughts about what might be going on with the memory? I am very new to
> raster processing in R (was previously working with ENVI+IDL) so forgive me
> if this is a naive question!
>
> Thanks,
> Marcia
>
>        [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-Geo mailing list
> R-sig-Geo at r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>