[R-sig-Geo] parallelize projectRaster()

[Jonathan Greenberg]

Can you make your test file available via google drive or dropbox or
something?  I'm really surprised GDAL is taking more time than the R
version.  It is hard to replicate your example without the test file.

As an FYI, to enable parallel processing of a single gdalwarp, use
(from: http://lists.osgeo.org/pipermail/gdal-dev/2012-June/033084.html):

gdalwarp -multi src.tif dst.tif -wo NUM_THREADS=ALL_CPUS

You can use these options from gdalUtils if you'd like.

--j

On Wed, Jun 18, 2014 at 2:32 PM, Yerguner <yasebaytok at gmail.com> wrote:
> Hi Jonathan,
>
> Believe me I tried that and all other different combinations but starting
> with the gdal_translate step only adds extra hrs to finish the whole
> process. Actually, when I run the below gdalwarp commanline outside R, it
> completes the same small netcdf file in ~ 2hrs but I just do the
> reprojection and then do the resampling step in R in ~25 min. So, this was
> the my first approach alternative to projectRaster() approach and 2,5 hrs is
> better than 8 hrs. But again, unless there is a solution to speed up the
> process (at least the reprojection part), both approaches do not work my
> case. That's why I'm looking for way to do this all process in R to be able
> to parallelize the functions as for the first approach. And I have also many
> large files (each about the same size 5.5 GB) to repeat this process. So,
> both the file sizes are large and many, in addition to doParallel(),
> foreach() package might also be a solution?
>
> gdalwarp and resample() approach:
>
> gdalwarp -t_srs "+proj=laea +lat_0=45 +lon_0=-100 +x_0=0 +y_0=0 +no_defs
> +a=6370997 +b=6370997 +to_meter=1.0" NETCDF:"colds1.nc":tmin colds1.tif
> ## ~2 hrs
>
> library(raster)
> library(rgdal)
> rasterOptions(tmpdir="some_tmp_path")
> source_file1 <- "mod2000.tif"
> source_file2 <- "colds1.tif"
> mod00 <- raster(source_file1)
> cold00s1 <- brick(source_file2)
> submod00 <- crop(mod00,cold00s1)
> rsmp1 <- resample(cold00s1,submod00,method="ngb")     ## 25 min
> stackSelect
> (rsmp1,submod00,filename="modcold00.tif",options="INTERLEAVE=BAND")
>
> Thank you so much and looking forward to hearing from you,
> Yasemin
>
>
>
>
> --
> View this message in context: http://r-sig-geo.2731867.n2.nabble.com/parallelize-projectRaster-tp7586598p7586608.html
> Sent from the R-sig-geo mailing list archive at Nabble.com.
>
> _______________________________________________
> R-sig-Geo mailing list
> R-sig-Geo at r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-geo



-- 
Jonathan A. Greenberg, PhD
Assistant Professor
Global Environmental Analysis and Remote Sensing (GEARS) Laboratory
Department of Geography and Geographic Information Science
University of Illinois at Urbana-Champaign
259 Computing Applications Building, MC-150
605 East Springfield Avenue
Champaign, IL  61820-6371
Phone: 217-300-1924
http://www.geog.illinois.edu/~jgrn/
AIM: jgrn307, MSN: jgrn307 at hotmail.com, Gchat: jgrn307, Skype: jgrn3007