[R-sig-Geo] Faster way to get raster average?

[Rainer M Krug]

"Thiago V. dos Santos" <thi_veloso at yahoo.com.br> writes:

> Hi all,
> I am working with some terabytes of CMIP5 climate files. Each file is
> a netcdf with multiple layers (timesteps) representing monthly data. 
> For each file, I need to extract the average value of the raster and
> put all values in a data frame. This is my current
> approach:---------------
> library(raster)
> # make up some datacmip <- brick(nc=150, nr=114, nl=1872)cmip <- setValues(cmip, matrix(rep(1:17100, 1872), nc=1872))
> # get mean values (area average) as data framescmip.mean <- as.data.frame(cellStats(cmip, mean, na.rm=T))---------------
> which works pretty fast in this example:
>> system.time(as.data.frame(cellStats(cmip, mean, na.rm=T)))
>    user  system elapsed 
>   0.069   0.012   0.081 
> However, the calculation with my actual data is substantially slower:
>> system.time(as.data.frame(cellStats(cmip, mean, na.rm=T)))
>    user  system elapsed 
>   4.600   1.105   5.704 
> Since I will have to deal with thousands of files, here comes my question: is there a faster way to get a the average value of a raster ? 

I am not an expert on netcdf files, but I would probably look outside R
to do these calculations. GRASS, and possibly even gdal, come to
mind. Depending on how often you have to do the calculations, putting
them into a spatial database and do the calculations there might be an
option?

Concerning paralelization: if, as Robert points out, the disk access is
the bottleneck (memory allocation in R will also take some time,
depending on the size of the individual map), paralelization will not
speed up (possibly even slow it down). In this case, your option is an
SSD, or a raid, and, as you are using a Mac, an internal or Thunderbolt
raid (all faster then even USB 3) - and definitely not an USB 2.0 HDD.

Otherwise, just using mcapply() over all maps would help.

Cheers,

Rainer


> Many thanks,
> --
> Thiago V. dos Santos
> PhD student
> Land and Atmospheric Science
> University of Minnesota
> http://www.laas.umn.edu/CurrentStudents/MeettheStudents/ThiagodosSantos/index.htm
> Phone: (612) 323 9898
> 	[[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-Geo mailing list
> R-sig-Geo at r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-geo

-- 
Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation Biology, UCT), Dipl. Phys. (Germany)

Centre of Excellence for Invasion Biology
Stellenbosch University
South Africa

Tel :       +33 - (0)9 53 10 27 44
Cell:       +33 - (0)6 85 62 59 98
Fax :       +33 - (0)9 58 10 27 44

Fax (D):    +49 - (0)3 21 21 25 22 44

email:      Rainer at krugs.de

Skype:      RMkrug

PGP: 0x0F52F982
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 494 bytes
Desc: not available
URL: <https://stat.ethz.ch/pipermail/r-sig-geo/attachments/20150506/2f4d3235/attachment.bin>