[R-sig-Geo] Performance SpatialGridDataFrame to rasterStack

Mon Mar 5 12:14:44 CET 2012

Hi,

-------- Original-Nachricht --------
> Datum: Fri, 2 Mar 2012 12:51:00 -0800
> Von: "Robert J. Hijmans" <r.hijmans at gmail.com>
> An: Johannes Radinger <JRadinger at gmx.at>
> CC: r-sig-geo at r-project.org
> Betreff: Re: [R-sig-Geo] Performance SpatialGridDataFrame to rasterStack

> Johannes,
> 
> please provide your  sessionInfo()  which is particularly relevant for
> this
> type (performance related) questions. I am guessing you are using a mac,
> as
> there have been some similar reports with macs; apparently performance can
> get a tremendous hit when RAM is exhausted and gets swapped to disk.

Yes, as you already guessed, I am working on a Mac. Here my sessionInfo():
>  sessionInfo()
R version 2.14.2 (2012-02-29)
Platform: x86_64-apple-darwin9.8.0/x86_64 (64-bit)

locale:
[1] C

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] rgdal_0.7-8    raster_1.9-70  spgrass6_0.7-9 XML_3.9-4      sp_0.9-95     
[6] rj_1.0.0-3    

loaded via a namespace (and not attached):
[1] grid_2.14.2    lattice_0.20-0 rj.gd_1.0.0-1  tools_2.14.2

Anyway... I followed your suggestion to save the single rasters to my harddisk as a e.g. tiff and then load them with raster("file") directly from the file. As my rasters in GRASS are already created by a script it is an easy step to save them as GeoTIFF directly within the script-run. So I don't need to import them to R with spgrass6, I can even skip the spgrass6 package then completlely.
There are just two things I have in mind:
1) the export settings of GRASS GIS (GeoTIFF), what about Null-Values, color tables, etc. Does anyone have experience with that already?
2) How to create a loop that loads all rasters from a special folder on my computer (basically all files of one folder).

Best regards,
Johannes

> 
> You have 60 rasters in memory, and then copy them into a RasterStack. To
> avoid that, I think that Johannes Signer solution might indeed work, as
> you
> could have a loop in which you use readRast to read a  raster, save it to
> disk (e.g. 'grd' or 'tif') and keep the filenames in a vector or list.
> Then
> make a stack using the filenames, and there should be no memory issue.
> 
> Robert
> 
> On Fri, Mar 2, 2012 at 4:37 AM, Johannes Radinger <JRadinger at gmx.at>
> wrote:
> 
> > Hi,
> >
> > I am not sure what you mean exactly with building a raster stack
> directly?
> > My original raster files are GRASS GIS raster files in a GRASS GIS
> mapset.
> > AFAIK these files can only be loaded with readRAST6 (with the result of
> a
> > SpatialGridDataFrame).
> >
> > So I am not quite sure if you mean I have to use a differnet process to
> > built the stack as my original files are no tifs (like the example).
> >
> > best regards,
> > Johannes
> >
> > -------- Original-Nachricht --------
> > > Datum: Fri, 2 Mar 2012 13:25:29 +0100
> > > Von: Johannes Signer <j.m.signer at gmail.com>
> > > An: Johannes Radinger <JRadinger at gmx.at>
> > > CC: r-sig-geo at r-project.org
> > > Betreff: Re: [R-sig-Geo] Performance SpatialGridDataFrame to
> rasterStack
> >
> > > Hello,
> > >
> > > not sure if it works for you, but why not directly build a raster
> > > stack (example below).
> > >
> > > Hope this helps,
> > > Johannes
> > >
> > > #
> > >
> >
> ----------------------------------------------------------------------------
> > #
> > > library(raster)
> > >
> > > # Create dummy data
> > > r1 <- raster(nrows=1000, ncols=800, xmn=0, xmx=100)
> > > r1 <- setValues(r1, rnorm(800000))
> > >
> > > # write dummy rasters
> > > sapply(1:60, function(x) writeRaster(r1, paste("r", x, ".tif",
> > > sep=""), overwrite=T))
> > >
> > > # Read read raster into a stack
> > > system.time(r <- stack <- do.call("stack",
> > > lapply(list.files(pattern="*.tif"), raster)))
> > >
> > > # on a 3 GB netbook this took just a bit more than 11 seconds
> > >
> > > On Fri, Mar 2, 2012 at 12:28 PM, Johannes Radinger <JRadinger at gmx.at>
> > > wrote:
> > > > Hi,
> > > >
> > > > I want to use raster maps for Species Distribution Modelling (SDM)
> with
> > > > the package 'dismo'. Therefore I load several rastermaps (around 60)
> > > with the readRAST6 command. For further processing I want to transform
> > them
> > > into a raster stack. Each of the raster has the same spatial extent
> and
> > the
> > > same resolution (per rastermap: around 765000 cells, thereof 760000
> NA's,
> > > 5000 non null cells).
> > > >
> > > > The readRAST6 command to load all rasters in one step into a
> > > SpatialGridDataFrame works really good (1-2 min). The problem now is
> the
> > tranformation
> > > into a stack. This takes very long time (I waited for 45 min), the CPU
> is
> > > mostly 100%, the pyhsical memory usage 3 GB (only a few MB left) and
> the
> > > virtual memory usage for R 3 GB, the HDD activity 20-30 MB/Sec data
> > written.
> > > So my Computer seem to be at its maximum, and I don't have access to a
> > > faster/better one. I am doint all my R processes within an Eclipse
> > workspace.
> > > >
> > > > What I do:
> > > > rast <- readRAST6(rast.list, cat=(FALSE,length(rast.list)),
> > > ignore.stderr=TRUE, plugin=NULL)
> > > >
> > > > rast.stack <- stack(rast)
> > > >
> > > >
> > > > Is there any way how I can improve the performance? As most of the
> > > raster cells are NULL cells maybe there is a special way consider
> that?
> > Or
> > > splitting the SpatialGridDataFrame up and recombining the stacks in
> the
> > end?
> > > > Or saving substeps to the harddisk?
> > > > Any help/suggestion is mostly welcome!
> > > >
> > > > Best regards,
> > > > Johannes
> > > >
> > > > --
> > > >
> > > > Jetzt informieren:
> http://mobile.1und1.de/?ac=OM.PW.PW003K20328T7073a
> > > >
> > > > _______________________________________________
> > > > R-sig-Geo mailing list
> > > > R-sig-Geo at r-project.org
> > > > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> >
> > --
> >
> > Jetzt informieren: http://mobile.1und1.de/?ac=OM.PW.PW003K20328T7073a
> >
> > _______________________________________________
> > R-sig-Geo mailing list
> > R-sig-Geo at r-project.org
> > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> >

-- 

Jetzt informieren: http://mobile.1und1.de/?ac=OM.PW.PW003K20328T7073a