[R-sig-Geo] Converting large RasterStack to CSVs fast

[Éder Comunello]

Hi, Mohammad!

I'm sorry, I expected a better result! I tried an alternative, but I got a
worst result! I added it at the end of the previous script.

I'll try to think of something better, if I succeed I´ll post here.

### <code r>
require(rgdal); require(raster)

getwd()
### download some data to test
getData("worldclim", var = "tmin", res = 10) ### tmin
fn <- dir("wc10", patt=".bil$", full=T)
fn <- fn[order(nchar(fn), fn)]; fn
#  [1] "wc10/tmin1.bil"  "wc10/tmin2.bil"  "wc10/tmin3.bil"  ...

### read images
s <- stack(fn) ### dimensions  : 900, 2160, 1944000, 12  (nrow, ncol,
ncell, nlayers)
fromDisk(s)

### extents of subsets
bor <- extent(s); bor
res <- 45 ### subsets resolution
X   <- unique(c(seq(bor at xmin, bor at xmax, by=res), bor at xmax)); X
Y   <- unique(c(seq(bor at ymin, bor at ymax, by=res), bor at ymax)); Y
ext <- cbind(expand.grid(Xmin=X[-length(X)], Ymin=Y[-length(Y)]),
             expand.grid(Xmax=X[-1], Ymax=Y[-1]))[,c(1,3,2,4)]
head(ext); nrow(ext)

plot(s, 1)
system.time(
sapply(1:nrow(ext), function(i) {
     mask <- ext[i,]
     subset <- with(mask, extent(c(Xmin, Xmax, Ymin, Ymax)))
     plot(subset, add=T)
     text(rowMeans(mask[,1:2]), rowMeans(mask[,3:4]), lab=i)
     c <- crop(s, subset)
     write.table(as.data.frame(rasterToPoints(c)), paste0("p",i,".txt"), )
}))
#    user  system elapsed
#   80.94    2.60   86.11

txt <- dir(patt="^p[0-9]+.txt$")
txt <- txt[order(nchar(txt), txt)]; txt
#  [1] "p1.txt"  "p2.txt"  "p3.txt"  "p4.txt"  ...

##########################################################
### Added code ############################################
###########################################################
plot(s, 1)
system.time(
sapply(1:nrow(ext), function(i) {
     mask <- ext[i,]
     subset <- with(mask, extent(c(Xmin, Xmax, Ymin, Ymax)))
     plot(subset, add=T)
     text(rowMeans(mask[,1:2]), rowMeans(mask[,3:4]), lab=i)
     cells <- cellsFromExtent(s, subset)
     c <- data.frame(xyFromCell(s, cells), extract(s, cells))
     write.table(c, paste0("pp",i,".txt"))
})
)
#    user  system elapsed
#  318.86   16.63  338.15

### </code>


​
================================================
Éder Comunello
PhD Student in Agricultural Systems Engineering (USP/Esalq)
Brazilian Agricultural Research Corporation (Embrapa)
Dourados, MS, Brazil [22 16.5'S, 54 49.0'W]




2015-05-22 7:51 GMT-04:00 Mohammad Abdel-Razek <kimofos at yahoo.com>:

> Hello Comunello!
>
> Thanks for the reply. Your vectorized code is faster than the one I have,
> by a margin of 5%, which is an improvement. I think the limiting step is
> clipping. Is there any paralleled  version of clip?
>
> Mohammad
>
>

	[[alternative HTML version deleted]]