[R-sig-Geo] Faster way to get raster average?

[Thiago V. dos Santos]

Thank you all for the responses. All of them helped me to learn new approaches.
Since I wanted to stick to R (even though I call CDO from R), I ended up adopting Michael's suggestion to reshape data as a matrix before taking the averages.
While relatively hard to read and understand, the time saved in the operation will probably make a huge difference in the final script looping through thousands of files. Greetings,
--
Thiago V. dos Santos
PhD student
Land and Atmospheric Science
University of Minnesota
http://www.laas.umn.edu/CurrentStudents/MeettheStudents/ThiagodosSantos/index.htm
Phone: (612) 323 9898 


     On Wednesday, May 6, 2015 7:25 AM, Byman HIkanyona <bymanh at gmail.com> wrote:
   

 Hi, 

My approach, in brief, is similar to what Micheal  Summer suggested, i.e using the netcdf tools (CDO, NCO, etc.). Whether in Linux, Mac,or windows. these tools run more efficiently - memory use and speed. I sometimes use R to automate the run since most are run on (DOS prompt like environment, batch). You can also 'pipe' the operation for example in CDO to avoid saving extra files. For example (CDO - windows) 'R CMD BATCH cdo ensmean ifile[1-100] ofile'  computes the ensemble mean of 100 files listed and saves the result in output file, which you can read using Raster packages. I don't know if this helps.

Cheers, Byman

On Wed, May 6, 2015 at 10:02 AM, Rainer M Krug <Rainer at krugs.de> wrote:

"Thiago V. dos Santos" <thi_veloso at yahoo.com.br> writes:

> Oh, I forgot to mention that I work on a quad-core Mac with 12GB.

> So, if multi-core can potentially accelerate cellStats, or whatever
> function is faster to get a raster average, I would be glad to see
> some examples on how to implement it. Greetings,

The key question is: how often do you have to do the calculations and if
it would be feasible to just let them run for a few days and be done
with it - optimization takes time!

Cheers,

Rainer

> --
> Thiago V. dos Santos
> PhD student
> Land and Atmospheric Science
> University of Minnesota
> http://www.laas.umn.edu/CurrentStudents/MeettheStudents/ThiagodosSantos/index.htm
> Phone: (612) 323 9898
>
>
>      On Tuesday, May 5, 2015 8:48 PM, Thiago V. dos Santos <thi_veloso at yahoo.com.br> wrote:
>
>
>  Hi all,
> I am working with some terabytes of CMIP5 climate files. Each file is a netcdf with multiple layers (timesteps) representing monthly data. 
> For each file, I need to extract the average value of the raster and put all values in a data frame. This is my current approach:---------------
> library(raster)
> # make up some datacmip <- brick(nc=150, nr=114, nl=1872)cmip <- setValues(cmip, matrix(rep(1:17100, 1872), nc=1872))
> # get mean values (area average) as data framescmip.mean <- as.data.frame(cellStats(cmip, mean, na.rm=T))---------------
> which works pretty fast in this example:
>> system.time(as.data.frame(cellStats(cmip, mean, na.rm=T)))
>   user  system elapsed
>   0.069  0.012  0.081
> However, the calculation with my actual data is substantially slower:
>> system.time(as.data.frame(cellStats(cmip, mean, na.rm=T)))
>   user  system elapsed
>   4.600  1.105  5.704
> Since I will have to deal with thousands of files, here comes my
> question: is there a faster way to get a the average value of
> a raster ? 
> Many thanks,
> --
> Thiago V. dos Santos
> PhD student
> Land and Atmospheric Science
> University of Minnesota
> http://www.laas.umn.edu/CurrentStudents/MeettheStudents/ThiagodosSantos/index.htm
> Phone: (612) 323 9898
>     [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-Geo mailing list
> R-sig-Geo at r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>
>
>
>       [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-Geo mailing list
> R-sig-Geo at r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-geo

--
Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation Biology, UCT), Dipl. Phys. (Germany)

Centre of Excellence for Invasion Biology
Stellenbosch University
South Africa

Tel :       +33 - (0)9 53 10 27 44
Cell:       +33 - (0)6 85 62 59 98
Fax :       +33 - (0)9 58 10 27 44

Fax (D):    +49 - (0)3 21 21 25 22 44

email:      Rainer at krugs.de

Skype:      RMkrug

PGP: 0x0F52F982

_______________________________________________
R-sig-Geo mailing list
R-sig-Geo at r-project.org
https://stat.ethz.ch/mailman/listinfo/r-sig-geo





-- 
"The highest reward for a person's toil is not what they get for it, but what they become by it."
--John Ruskin,
British art critic

Byman Hamududu



  
	[[alternative HTML version deleted]]