[R-sig-Geo] loops in rasterEngine

[Alex Zvoleff]

On Wed, Mar 12, 2014 at 11:29 PM, Boulanger, Yan
<Yan.Boulanger at rncan-nrcan.gc.ca> wrote:
> Actually, I have several rasters of more than 440 000 000 pixels (MODIS covering all Canada) and I have a 32-cores machine so I would like to take advantage of it! ;-)
>
> Time is money (really?!!)

As mentioned earlier, I would be careful about using rasterEngine for
this kind of task. It may actually slow you down. I would recommend
testing on smaller subsets to determine your gains (or losses) from
doing this type of calculation in parallel versus sequentially. While
I have seen great speed increases for CPU intensive calculations from
using rasterEngine, it sounds like your processing is heavily IO
intensive. I am not sure 32 cores will help you unless you have a very
fast disk or RAID array.

Alex

>
> Thanks again!
> yan
>
> Yan Boulanger, Chercheur scientifique / Research scientist
> Ressources Naturelles Canada, Canadian Forest Service
> Centre de Foresterie des Laurentides
> 1055, rue du P.E.P.S.
> C.P. 10380, succ. Sainte-Foy
> Québec (Québec) Canada
> G1V 4C7
> Tel. : +1 418 649-6859
>
> From: Forrest Stevens [mailto:forrest at ufl.edu]
> Sent: 12 mars 2014 22:25
> To: Boulanger, Yan
> Cc: r-sig-geo at r-project.org
> Subject: Re: [R-sig-Geo] loops in rasterEngine
>
> Hi Yan, I guess I would be surprised for such a simple process if rasterEngine() would be worth the overhead? Though, admittedly, Jonathan Greenberg might have more information on the topic.  To do such an operation this is the approach I would take without using rasterEngine():
>
>
> for (i in 1:5) {
>   assign(paste("Safranyik_zones_1961_1990b_",i, sep=""), Safranyik_zones_1961_1990b == i)
> }
>
>
> To do it using rasterEngine() this is the function definition that I would use. This of course requires that you've already created a cluster using one of the various supported parallel backends otherwise you'll gain nothing from the parallel processing.
>
>
> require("spatial.tools")
>
> ## Begin a parallel cluster and register it with foreach:
> ## The number of nodes/cores to use in the cluster
> cpus = 2
> cl <- makeCluster(spec = cpus, type = "PSOCK", methods = FALSE)
> ## Register the cluster with foreach:
> registerDoParallel(cl)
>
> ##       Or use the following, quick and dirty way:
> #sfQuickInit(cpus=2)
>
> fun_zone <- function( zones, i, ...) {
>   return(zones == i)
> }
>
> for (j in 1:5){
>   assign(paste("Safranyik_zones_1961_1990b_",j, sep=""), rasterEngine( zones=Safranyik_zones_1961_1990b, args=list("i"=j), fun=fun_zone) )
> }
>
> stopCluster(cl)
> #sfQuickStop()
>
>
> Hope this helps,
> Forrest
>
> --
> Forrest R. Stevens
> Ph.D. Candidate, QSE3 IGERT Fellow
> Department of Geography
> Land Use and Environmental Change Institute
> University of Florida
> www.clas.ufl.edu/users/forrest<http://www.clas.ufl.edu/users/forrest>
>
> On Wed, Mar 12, 2014 at 8:51 PM, Boulanger, Yan <Yan.Boulanger at rncan-nrcan.gc.ca<mailto:Yan.Boulanger at rncan-nrcan.gc.ca>> wrote:
> Hi folks,
>
> I guess I have a lot to learn to write functions but I'm stuck when using rasterEngine. It seems that it should be very easy to do but I'm missing something, apparently... I have a raster, Safranyik_zones_1961_1990, with values (integer) from 1 to 5. I would like to create five rasters for which value will be 1 when the raster Safranyik_zones_1961_1990 is equal to "i", and NA otherwise. I would like to run everything in a loop . Here's what I thought would be ok.
>
> fun_zone <- function(Safranyik_zones,i,...) {
> Safranyik_zonesb <- Safranyik_zones
> Safranyik_zonesb[] <- NA
> Safranyik_zonesb[Safranyik_zones == i] <- 1
> return(Safranyik_zonesb)
> }
>
> for (i in 1:5){
> Safranyik_zones_1961_1990b <- rasterEngine(Safranyik_zones=Safranyik_zones_1961_1990,i=i, fun=fun_zone)
> assign(paste("Safranyik_zones_1961_1990b_",i, sep=""),Safranyik_zones_1961_1990b[[1]])
> }
>
> Of course, it says that « i » is missing...:
>
>>Erreur dans Safranyik_zones == i : 'i' est manquant
>
> Any help?
>
> Thanks in advance,
>
> Yan
>
>
> Yan Boulanger, Chercheur scientifique / Research scientist
> Ressources Naturelles Canada, Canadian Forest Service
> Centre de Foresterie des Laurentides
> 1055, rue du P.E.P.S.
> C.P. 10380, succ. Sainte-Foy
> Québec (Québec) Canada
> G1V 4C7
> Tel. : +1 418 649-6859
>
>
>
>
>         [[alternative HTML version deleted]]
>
>
> _______________________________________________
> R-sig-Geo mailing list
> R-sig-Geo at r-project.org<mailto:R-sig-Geo at r-project.org>
> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>
>
>         [[alternative HTML version deleted]]
>
>
> _______________________________________________
> R-sig-Geo mailing list
> R-sig-Geo at r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>



-- 
Alex Zvoleff
Postdoctoral Associate
Tropical Ecology Assessment and Monitoring (TEAM) Network
Conservation International
2011 Crystal Dr. Suite 500, Arlington, Virginia 22202, USA
Tel: +1-703-341-2749, Fax: +1-703-979-0953, Skype: azvoleff
http://www.teamnetwork.org | http://www.conservation.org