[R-sig-Geo] Raster math on a stack of large rasters

[Melanie Bacou]

Also take a look at using raster:: clusterR() in combination with calc() 
to use multiple cores. This works well if your your calculation function 
does not require neighboring cells.

--Mel.


On 04/07/2017 09:50 AM, Benjamin Leutner wrote:
> You could stick to the native raster format (grd), in which case 
> calc() writes to your final file directly.
> Of course that doesn't change the file size issue, but saves you the 
> translation step to geotiff.
>
> For the file size you could consider restricting the datatype to 
> integer (see ?dataType).
> If I have reflectance data [0,1] for example; I scale them by a factor 
> 10000 and then save as "INT4U" or "INT2U" (depending on your maximally 
> expected value range), or "INT4S" or "INT2S" if you have negative 
> values. That can bring down the filesize quite a bit, while retaining 
> most of the relevant precission (beware: remaining decimal places will 
> be cut-off,)
>
> e.g. calc(..., function(x) { yourcalculations(x) * 10000 }, datatype = 
> "INT4S")
>
> pro tip: the argument in calc() (or more precisely in writeRaster()) 
> is called datatype, not to be confused with the stand-alone function 
> dataType() with a capital T. That one has bitten me many times, 
> because due to the "..." argument there will be no warning if you 
> mistype it ;-)
>
>
>
> On 06.04.2017 19:27, Gregovich, Dave P (DFG) wrote:
>> Hi,
>> I am performing a math operation on a stack of large rasters. The 
>> code below uses smaller files for illustration and reproducibility.
>> Any alternative way of performing this task that does not create huge 
>> temporary files, and perhaps cuts down on processing time, would be 
>> greatly appreciated. The 'calc'  process creates a couple of 
>> temporary raster files that are huge. The first one is 142 GB, and I 
>> don't have hard drive space for that one and the second one that 
>> begins writing during the process.
>> Thanks kindly for any advice!
>> Dave.
>>
>> #create raster stack and coefficients...
>> library(raster)
>> mod.coefs <- rnorm(10)
>> s <- stack()
>> r <- raster(nrow = 100, ncol = 100)
>> #actual rasters I am working with are about 40000, pixels square, 
>> with each GeoTiff raster in the stack taking about 2.5 GB on disk
>>
>> for(i in 1:10){
>>    r[] <- rnorm(10000)
>>    s <- addLayer(s, r)
>> }
>>
>> #attempt to perform raster math...
>> out.file <- 'C:/ out.rast.tif'
>> out.rast <- calc(s, function(x){exp(sum(mod.coefs * x))}, filename = 
>> out.file)
>>
>> #at this point, the temporary files in the 
>> \AppData\Local\Temp\RtmpqQYzfS\raster folder eventually become quite 
>> large,
>> #with one .GRI file reaching 142 GB, and another now growing to 8 GB 
>> before I ended the process
>> #the file 'out.file' has not been created at that point.
>> #_____________end____________________________________________________________________ 
>>
>>
>>     [[alternative HTML version deleted]]
>>
>> _______________________________________________
>> R-sig-Geo mailing list
>> R-sig-Geo at r-project.org
>> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>>
>