[R-sig-Geo] Using 'raster' package functions with rasterEngine

Thu Aug 21 19:22:36 CEST 2014

Ah, ok, I see what is going on.  The reason you are having this issue is
because clump is a focal window analysis, not a pixel-based analysis.  On
top of that, this is also a more complicated algorithm then you might think
since it deals with connectivity.  In this case, rasterEngine is almost
certainly not going to work the way you want it to without a lot of
tweaking.  You might want to open up the code of clump and see how Robert
does chunking (you also might see if Robert already built in parallel
processing to clump) -- you may be able to use this function inside of
rasterEngine.

--j

On Tue, Aug 19, 2014 at 12:21 PM, Dave Marvin <dmarvin at carnegiescience.edu>
wrote:

> Thank you for the comments Jonathan. I coerced the output from my
> function to an array, and that got it working. As you said the
> dimensions need to be ordered as (col, row). However, as you will see,
> there is an issue with the output in parallel mode:
>
> test_input=raster(system.file("external/tahoe_lidar_highesthit.tif",
> package="spatial.tools"))
> test_input=test_input>2535 #create distinct tree canopy objects.
> plot(test_input)
>
> clump_fun=function(raster_layer,...){
>      clump_raster=clump(raster_layer,directions=8)
>
>  clump_array=array(getValues(clump_raster),dim=c(dim(raster_layer)[c(2:1)],1))
>      return(clump_array)
> }
>
> cpus=3
> cl=makeCluster(spec = cpus, type = "PSOCK", methods = FALSE)
> registerDoParallel(cl)
>
>
> test_raster_out_a=rasterEngine(raster_layer=test_input,fun=clump_fun,chunk_format="raster",debugmode=FALSE)
>
> stopCluster(cl)
>
> **While this gives me an output, it is quite different from running
> the same function in either sequential mode or outside rasterEngine
> entirely:
>
> registerDoSEQ()
>
> test_raster_out_b=rasterEngine(raster_layer=test_input,fun=clump_fun,chunk_format="raster",debugmode=FALSE)
> test_raster_out_c=clump(test_input,directions=8)
>
> dim(freq(test_raster_out_a)$layer) #parallel output
> dim(freq(test_raster_out_b)$layer) #sequential mode output
> dim(freq(test_raster_out_c)) #non-rasterEngine output (same as "b")
>
> In fact, the number of clumps varies depending on the number of cores
> I use in parallel mode. Any idea what is happening here?
>
> Thank you,
> -Dave
>
> ----
> Dave Marvin | davidcmarvin.org
> Postdoctoral Research Fellow
> Department of Global Ecology
> Carnegie Institution for Science
>
>
> On Mon, Aug 11, 2014 at 8:55 AM, Jonathan Greenberg <jgrn at illinois.edu>
> wrote:
> > Hi Dave:
> >
> > If you can provide some code examples that would be easier, but I can
> give
> > you a couple of hints.  First, use debugmode=TRUE when you are testing
> your
> > function -- this will save you a lot of headaches down the line.  R will
> > enter its debugmode inside of the rasterEngine function so you can track
> > what is happening.
> >
> > The first issue is that clump only works on raster objects.
> rasterEngine,
> > in its default mode, sends "chunks" of the image to your function
> formatted
> > as *arrays*, not raster objects.  When you did chunk_format to raster, it
> > did, indeed, send the chunk as a small, cropped raster so clump worked,
> BUT:
> > all *outputs* from rasterEngine (at present) MUST be coerced to an array
> > that has the same number of columns and rows as the input (in that
> order).
> > I might mod this down the road to allow raster outputs, but for now
> you'll
> > need to make sure the output is of type array.  I suspect you are trying
> to
> > return the chunk from your function as a raster, which is why it isn't
> > working.
> >
> > Again: debugmode=TRUE, and track your function down to just before your
> > return the output -- check the type of the to-be-returned object and
> confirm
> > it is an array with dim()[1:2] equal to the input raster.
> >
> > Good luck!
> >
> > --j
> >
> >
> > On Thu, Aug 7, 2014 at 12:02 PM, Dave Marvin <
> dmarvin at carnegiescience.edu>
> > wrote:
> >>
> >> Hi,
> >> I am able to set up functions and apply them to a raster dataset in
> >> parallel using the rasterEngine function of the spatial.tools package,
> but
> >> only if the function performs just a numerical operation on the raster.
> >> When I try to incorporate a function from the raster package, such as
> >> clump(), the rasterEngine fails and I get an error:
> >>
> >> > Error in (function (classes, fdef, mtable): unable to find an
> inherited
> >> method for function ‘clump’ for signature ‘"array"’
> >>
> >> If I change "chunk_format" to "raster" in the rasterEngine function,
> then
> >> i
> >> get an error:
> >>
> >> > chunk processing units require array vector outputs.  Please check
> your
> >> function.
> >>
> >> 1) Am I doing doing something wrong, or does this particular function
> just
> >> not work with rasterEngine?
> >>
> >> 2) If it is the latter, which of the raster package functions are
> >> available
> >> for use in parallel processing with rasterEngine?
> >>
> >> Thank you.
> >> -Dave
> >>
> >>         [[alternative HTML version deleted]]
> >>
> >>
> >> _______________________________________________
> >> R-sig-Geo mailing list
> >> R-sig-Geo at r-project.org
> >> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> >>
> >
> >
> >
> > --
> > Jonathan A. Greenberg, PhD
> > Assistant Professor
> > Global Environmental Analysis and Remote Sensing (GEARS) Laboratory
> > Department of Geography and Geographic Information Science
> > University of Illinois at Urbana-Champaign
> > 259 Computing Applications Building, MC-150
> > 605 East Springfield Avenue
> > Champaign, IL  61820-6371
> > Phone: 217-300-1924
> > http://www.geog.illinois.edu/~jgrn/
> > AIM: jgrn307, MSN: jgrn307 at hotmail.com, Gchat: jgrn307, Skype: jgrn3007
>

-- 
Jonathan A. Greenberg, PhD
Assistant Professor
Global Environmental Analysis and Remote Sensing (GEARS) Laboratory
Department of Geography and Geographic Information Science
University of Illinois at Urbana-Champaign
259 Computing Applications Building, MC-150
605 East Springfield Avenue
Champaign, IL  61820-6371
Phone: 217-300-1924
http://www.geog.illinois.edu/~jgrn/
AIM: jgrn307, MSN: jgrn307 at hotmail.com, Gchat: jgrn307, Skype: jgrn3007

	[[alternative HTML version deleted]]