[R-sig-Geo] Faster way to get raster average?

[Byman HIkanyona]

Hi,

My approach, in brief, is similar to what Micheal  Summer suggested, i.e
using the netcdf tools (CDO, NCO, etc.). Whether in Linux, Mac,or windows.
these tools run more efficiently - memory use and speed. I sometimes use R
to automate the run since most are run on (DOS prompt like environment,
batch). You can also 'pipe' the operation for example in CDO to avoid
saving extra files. For example (CDO - windows) '*R* CMD BATCH cdo ensmean
ifile[1-100] ofile'  computes the ensemble mean of 100 files listed and
saves the result in output file, which you can read using Raster packages. I
don't know if this helps.

Cheers, Byman

On Wed, May 6, 2015 at 10:02 AM, Rainer M Krug <Rainer at krugs.de> wrote:

> "Thiago V. dos Santos" <thi_veloso at yahoo.com.br> writes:
>
> > Oh, I forgot to mention that I work on a quad-core Mac with 12GB.
>
> > So, if multi-core can potentially accelerate cellStats, or whatever
> > function is faster to get a raster average, I would be glad to see
> > some examples on how to implement it. Greetings,
>
> The key question is: how often do you have to do the calculations and if
> it would be feasible to just let them run for a few days and be done
> with it - optimization takes time!
>
> Cheers,
>
> Rainer
>
> > --
> > Thiago V. dos Santos
> > PhD student
> > Land and Atmospheric Science
> > University of Minnesota
> >
> http://www.laas.umn.edu/CurrentStudents/MeettheStudents/ThiagodosSantos/index.htm
> > Phone: (612) 323 9898
> >
> >
> >      On Tuesday, May 5, 2015 8:48 PM, Thiago V. dos Santos <
> thi_veloso at yahoo.com.br> wrote:
> >
> >
> >  Hi all,
> > I am working with some terabytes of CMIP5 climate files. Each file is a
> netcdf with multiple layers (timesteps) representing monthly data.
> > For each file, I need to extract the average value of the raster and put
> all values in a data frame. This is my current approach:---------------
> > library(raster)
> > # make up some datacmip <- brick(nc=150, nr=114, nl=1872)cmip <-
> setValues(cmip, matrix(rep(1:17100, 1872), nc=1872))
> > # get mean values (area average) as data framescmip.mean <-
> as.data.frame(cellStats(cmip, mean, na.rm=T))---------------
> > which works pretty fast in this example:
> >> system.time(as.data.frame(cellStats(cmip, mean, na.rm=T)))
> >   user  system elapsed
> >   0.069  0.012  0.081
> > However, the calculation with my actual data is substantially slower:
> >> system.time(as.data.frame(cellStats(cmip, mean, na.rm=T)))
> >   user  system elapsed
> >   4.600  1.105  5.704
> > Since I will have to deal with thousands of files, here comes my
> > question: is there a faster way to get a the average value of
> > a raster ?
> > Many thanks,
> > --
> > Thiago V. dos Santos
> > PhD student
> > Land and Atmospheric Science
> > University of Minnesota
> >
> http://www.laas.umn.edu/CurrentStudents/MeettheStudents/ThiagodosSantos/index.htm
> > Phone: (612) 323 9898
> >     [[alternative HTML version deleted]]
> >
> > _______________________________________________
> > R-sig-Geo mailing list
> > R-sig-Geo at r-project.org
> > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> >
> >
> >
> >       [[alternative HTML version deleted]]
> >
> > _______________________________________________
> > R-sig-Geo mailing list
> > R-sig-Geo at r-project.org
> > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>
> --
> Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation
> Biology, UCT), Dipl. Phys. (Germany)
>
> Centre of Excellence for Invasion Biology
> Stellenbosch University
> South Africa
>
> Tel :       +33 - (0)9 53 10 27 44
> Cell:       +33 - (0)6 85 62 59 98
> Fax :       +33 - (0)9 58 10 27 44
>
> Fax (D):    +49 - (0)3 21 21 25 22 44
>
> email:      Rainer at krugs.de
>
> Skype:      RMkrug
>
> PGP: 0x0F52F982
>
> _______________________________________________
> R-sig-Geo mailing list
> R-sig-Geo at r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>
>


-- 
*"The highest reward for a person's toil is not what they get for it, but
what they become by it."*
*--John Ruskin,*
*British art critic*

Byman Hamududu

	[[alternative HTML version deleted]]