[R-sig-Geo] R crashes when looping with function writeRaster

[Robert J. Hijmans]

Romolo,

For your questions about memory, please read vignette('raster').

> r[,]=ifelse(r[,]<=(-2000), NA, r[,]/10000) # is this the culprit?

probably yes, because you are loading all the values into RAM if you do that.
Consider doing something like this:

fileDisp=list.files(getwd())
for (f in 1:length(fileDisp))  :
      r <- raster(fileDisp[f])
   # never overwrite your input files!
   # preferably also send the output to another directory
     outf <- paste0(extension(fileDisp, ''), "_out.tif"
     x <- calc(r, function(x){ x[x <= -2000] <- NA;  x/10000 }, filename=outf)
}


# or perhaps

fileDisp=list.files(getwd())
for (f in 1:length(fileDisp))  :
      r <- raster(fileDisp[f])
     NAvalue(r) <- -2000
     outf <- paste0(extension(fileDisp, ''), "_out.tif"
     x <- calc(r, function(x)  x/10000, filename=outf)
}


Robert

On Thu, Feb 14, 2013 at 12:35 AM, Romolo Salini <salinir at tiscali.it> wrote:
>  Jonathan, thanks for your reply. I adjusted the code (fileAnno wasn't
> correct)...
>
> fileDisp=list.files(getwd(), full=TRUE)
> for (f in 1:length(fileDisp))
> {
> r=raster(fileDisp[f])
> r[,]=ifelse(r[,]<=(-2000), NA, r[,]/10000) # is this the culprit?
> writeRaster(r,filename=fileDisp[f],format="GTiff",overwrite=TRUE)
> }
>
> Also I don't understand the behaviour (in memory managment),
> for the following examples:
>
> LayerTot = stack(fileDisp)
> #ex1
> MeanL=calc(LayerTot, fun=mean) #gives a memory allocation error, while...
> #ex2
> MeanL=mean(LayerTot) #doesn't
>
> Furthermore, when I ran:
> #ex3
> ss <- calc(LayerTot,fun=myfunction)
> which gives a rasterbrick, a .grd file was written into my local temporary
> directory.
> is it possible to force this behaviour each time I call the function "calc"
> (for ex1, this could avoid the error)?
>
> Romolo
>
> 2013/2/13 Jonathan Greenberg <jgrn at illinois.edu>
>
>> Romolo:
>>
>> I feel like:
>>   #do some calculation on r and write it with same name
>> Might be the culprit.  What calculation were you doing?  Can you paste in
>> the code?  Creating a raster and writing a raster take almost no memory
>> AFAIK.
>>
>> Also, your code shows the looping through the length of fileAnno, but I
>> don't see that defined anywhere.
>>
>> --j
>>
>>
>> On Thu, Feb 7, 2013 at 2:22 AM, Romolo Salini <salinir at tiscali.it> wrote:
>>
>>> Dear R-sig-geo,
>>> I'm looping the function "writeRaster", as in:
>>>
>>> fileDisp=list.files(getwd(),full=TRUE)
>>> for (f in 1:length(fileAnno))
>>>   {
>>>   r=raster(fileDisp[f])
>>>   #do some calculation on r and write it with same name
>>>   writeRaster(r,filename=fileDisp[f],format="GTiff",overwrite=TRUE)
>>>   }
>>>
>>> After about 10/15 iterations, R quits.
>>> As it seemed to be a memory allocation problem, I tried adding
>>> (uselessly):
>>>
>>> rm(r)
>>> Sys.sleep(5)
>>>
>>> Does anybody had the same problem?
>>>
>>> > sessionInfo()
>>> R version 2.15.1 (2012-06-22)
>>> Platform: i386-pc-mingw32/i386 (32-bit)
>>>
>>> locale:
>>> [1] LC_COLLATE=Italian_Italy.1252  LC_CTYPE=Italian_Italy.1252
>>> [3] LC_MONETARY=Italian_Italy.1252 LC_NUMERIC=C
>>> [5] LC_TIME=Italian_Italy.1252
>>>
>>> attached base packages:
>>> [1] stats     graphics  grDevices utils     datasets  methods   base
>>>
>>> Sys.info()
>>>                      sysname                      release
>>>                    "Windows"                         "XP"
>>>
>>>         [[alternative HTML version deleted]]
>>>
>>> _______________________________________________
>>> R-sig-Geo mailing list
>>> R-sig-Geo at r-project.org
>>> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>>>
>>
>>
>>
>> --
>> Jonathan A. Greenberg, PhD
>> Assistant Professor
>> Global Environmental Analysis and Remote Sensing (GEARS) Laboratory
>> Department of Geography and Geographic Information Science
>> University of Illinois at Urbana-Champaign
>> 607 South Mathews Avenue, MC 150
>> Urbana, IL 61801
>> Phone: 217-300-1924
>> http://www.geog.illinois.edu/~jgrn/
>> AIM: jgrn307, MSN: jgrn307 at hotmail.com, Gchat: jgrn307, Skype: jgrn3007
>>
>
>         [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-Geo mailing list
> R-sig-Geo at r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-geo