[R-sig-Geo] Raster math on a stack of large rasters

[Benjamin Leutner]

You could stick to the native raster format (grd), in which case calc() 
writes to your final file directly.
Of course that doesn't change the file size issue, but saves you the 
translation step to geotiff.

For the file size you could consider restricting the datatype to integer 
(see ?dataType).
If I have reflectance data [0,1] for example; I scale them by a factor 
10000 and then save as "INT4U" or "INT2U" (depending on your maximally 
expected value range), or "INT4S" or "INT2S" if you have negative 
values. That can bring down the filesize quite a bit, while retaining 
most of the relevant precission (beware: remaining decimal places will 
be cut-off,)

e.g. calc(..., function(x) { yourcalculations(x) * 10000 }, datatype = 
"INT4S")

pro tip: the argument in calc() (or more precisely in writeRaster()) is 
called datatype, not to be confused with the stand-alone function 
dataType() with a capital T. That one has bitten me many times, because 
due to the "..." argument there will be no warning if you mistype it ;-)



On 06.04.2017 19:27, Gregovich, Dave P (DFG) wrote:
> Hi,
> I am performing a math operation on a stack of large rasters. The code below uses smaller files for illustration and reproducibility.
> Any alternative way of performing this task that does not create huge temporary files, and perhaps cuts down on processing time, would be greatly appreciated. The 'calc'  process creates a couple of temporary raster files that are huge. The first one is 142 GB, and I don't have hard drive space for that one and the second one that begins writing during the process.
> Thanks kindly for any advice!
> Dave.
>
> #create raster stack and coefficients...
> library(raster)
> mod.coefs <- rnorm(10)
> s <- stack()
> r <- raster(nrow = 100, ncol = 100)
> #actual rasters I am working with are about 40000, pixels square, with each GeoTiff raster in the stack taking about 2.5 GB on disk
>
> for(i in 1:10){
>    r[] <- rnorm(10000)
>    s <- addLayer(s, r)
> }
>
> #attempt to perform raster math...
> out.file <- 'C:/ out.rast.tif'
> out.rast <- calc(s, function(x){exp(sum(mod.coefs * x))}, filename = out.file)
>
> #at this point, the temporary files in the \AppData\Local\Temp\RtmpqQYzfS\raster folder eventually become quite large,
> #with one .GRI file reaching 142 GB, and another now growing to 8 GB before I ended the process
> #the file 'out.file' has not been created at that point.
> #_____________end____________________________________________________________________
>
> 	[[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-Geo mailing list
> R-sig-Geo at r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>

-- 
Benjamin Leutner M.Sc.
PhD candidate

Department of Remote Sensing
University of Wuerzburg
Campus Hubland Nord 86
97074 Wuerzburg, Germany

Tel: +49-(0)931-31 89594
Email: benjamin.leutner at uni-wuerzburg.de
Web: http://www.fernerkundung.uni-wuerzburg.de