[R-sig-Geo] Making web-repository of gridded maps: NetCDF or WKT Raster?

[Tomislav Hengl]

Dear Edzer, Barry, Servet, and Miguel, 

Thank you for your tips and for sharing your experiences with us (R-sig-geo keeps on surprising me
considering the amount of support I get :).

I will dig into the links you have forwarded. Most probably, we will try to implement two/three
paths and then compare them to see which is 'better' (the main issue is that we want to do majority
of processing on the server):

(A) The rasdaman approach (http://kahlua.eecs.jacobs-university.de/trac/rasdaman/wiki/Rasdaman_8.0)
1. Install PostgreSQL support on server and download and install rasdaman File Set;
2. Create a database and put the grids in PostgreSQL database (raslib applications);
3. Test some examples with overlaying, subsetting, resampling, upscaling and visualizing data (WMS,
KML); 

(B) The OPeNDAP/NetCDF approach (http://www.unidata.ucar.edu/software/netcdf/software.html#GDAL)
1. Install OPeNDAP server and UMN Mapserver to serve data;
2. Put all grids into a single (CF)NetCDF file and upload (using WebDAV) to the server;
3. Test some examples (R + GDAL) with overlaying, subsetting, resampling, upscaling and visualizing
the data (WMS, KML);

I will keep you informed once we produce some live examples. We might first start testing a WCPS
with the worldmaps repository (http://spatial-analyst.net/worldmaps/); then develop some examples (R
code) and test flexibility/speed of both approaches. 
cheers,

T. Hengl
http://home.medewerker.uva.nl/t.hengl/


> -----Original Message-----
> From: r-sig-geo-bounces at stat.math.ethz.ch [mailto:r-sig-geo-bounces at stat.math.ethz.ch] On Behalf
> Of Edzer J. Pebesma
> Sent: Sunday, December 13, 2009 9:05 PM
> To: Stefano Casalegno
> Cc: r-sig-geo
> Subject: Re: [R-sig-Geo] Making web-repository of gridded maps: NetCDF or WKT Raster?
> 
> Dear Tom,
> 
> as your grids are spatial grids, you may want to look into the web
> service standards of the OGC (www.opengeospatial.org) for them, WCS (for
> reading) and WCS-T (for writing) in particular. We're experimenting with
> this currently, using the recently open sourced rasdaman software
> (www.rasdaman.org), which uses a data base in the back end. I don't
> think OPeNDAP has special infrastructure for spatial grids; CFNetCDF
> does, but by choosing this you may limit integration with other software
> that does assume spatial data to be served over OGC standards. CFNetCDF
> is one of the possible formats to be served over WCS (and GeoTIFF etc)
> -- if I'm correctly informed.
> 
> Bests,
> --
> Edzer
> 
> Stefano Casalegno wrote:
> > Dear Tom,
> > my best acknowledgment for the Ecogrid challenging initiative.
> > I guess the main limitation will be "time" and this is related to the
> > size of grids.
> > What is the extent of the grids ?
> > What tool you plan to use for point 1. Overlay some point data and
> > get the values of grids ?
> >
> > On a more theoretical basis,
> > "distribution models should rely upon data that covers the whole
> > extent of the distribution of the biota under examination"
> >
> > Please consider forest data from
> > http://forest.jrc.ec.europa.eu/climate-change
> > as supplementary predictors variables within an European biodiversity
> > and nature conservation application.
> >
> > Cheers
> > Stefano
> >
> >
> > ________________________________
> >
> > Stefano Casalegno, Ph.D.
> >
> > via Greppi 16, 21021 Angera - Italy
> > email: stefano at casalegno.net
> > web: http://www.casalegno.net/stefano
> > ________________________________
> >
> >
> > On Dec 13, 2009, at 6:29 PM, Tomislav Hengl wrote:
> >
> >
> >> Dear R-sig-geo,
> >>
> >> As a part of our project (EcoGRID.nl) we have prepared some 60
> >> thematic grids that we use as auxiliary predictors for species
> >> distribution modeling. At this stage, we would like to put the
> >> gridded maps (50/100 m base resolution) into some efficient
> >> sharable geo-database.
> >>
> >> We will most probably put the data into the NetCDF format (http://
> >> www.unidata.ucar.edu/software/netcdf/) because it can handle any-
> >> dimensional array data, and because it has been in continuous
> >> development and widespread use for many years. NetCDF grids can be
> >> read relatively easy into R using the RNetCDF package (e.g. http://
> >> spatial-analyst.net/DATA/readNCDF.zip). Another alternative is to
> >> use PostGIS WKT Raster format (http://trac.osgeo.org/postgis/wiki/
> >> WKTRaster), but this seems to be still rather experimental (?).
> >>
> >> Once we put the grids into NetCDF format. We plan to install
> >> OPeNDAP (http://www.opendap.org) server on top to make the files
> >> accessible through the web; then Geoserver (http://
> >> www.geoserver.org) or UMN Mapserver (http://mapserver.org) to feed
> >> a WMS from NetCDF files (raster data). Finally, we plan to add a
> >> simple OpenLayers interface on top of that (Geoserver has it built
> >> in) to allow direct browsing of the data and metadata (e.g. such as
> >> this one: http://africamap.harvard.edu/)
> >>
> >> Just to be clear, we want to put the data on a server because we
> >> would like to run a number of operations directly on the server
> >> (via rgdal?):
> >>
> >> 1. Overlay some point data and get the values of grids (without a
> >> need to download the grids locally);
> >> 2. Subset/mask and resample gridded data of interest (for a given
> >> bounding box and proj4 string; again without a need to download the
> >> data locally);
> >> 3. Upscale the grids from 100 m to 250, 500 m and 1 km resolution
> >> (then download the upscaled grids).
> >> 4. Write/upload new grids to the database (e.g. using WebDAV).
> >> 5. Browse the grids (and metadata) via the OpenLayers.
> >>
> >> These are only our wishes of course. We do not know if all this is
> >> really possible with the current software.
> >>
> >> Any examples or comments/suggestions/experiences are welcome
> >> (before we start installing and testing the functionality).
> >>
> >> Thanx!
> >>
> >> Tomislav Hengl and Lourens Veen
> >>
> >> _______________________________________________
> >> R-sig-Geo mailing list
> >> R-sig-Geo at stat.math.ethz.ch
> >> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> >>
> >>
> >
> >
> >
> >
> >
> >
> > 	[[alternative HTML version deleted]]
> >
> > _______________________________________________
> > R-sig-Geo mailing list
> > R-sig-Geo at stat.math.ethz.ch
> > https://stat.ethz.ch/mailman/listinfo/r-sig-geo
> >
> 
> _______________________________________________
> R-sig-Geo mailing list
> R-sig-Geo at stat.math.ethz.ch
> https://stat.ethz.ch/mailman/listinfo/r-sig-geo