[R-sig-Geo] Spatial.tools rasterEngine subscript out of bounds; setting blocksize works but why?

[Javi.Navarro at csiro.au]

Hi all,

I've had an issue using rasterEngine (spatial.tools) to speed up a moving window analysis on some .tiff rasters.

I managed to solve the issue but I would like to gain an understanding as to why my code didn't work in the first place.

The short version is: When I call rasterEngine to process a comparably large raster (6GB as opposed to 1GB), rasterEngine exits nearly at the end and sends out a subscript out of bounds error (see below). I noticed the difference with other rasters was that in this case rasterEngine divided the task in an odd number of iterations (319) whereas in succesfully-processed rasters the iterations were always an even number. When I specified a blocksize to force rasterEngine to process the large raster in an even number of iterations, it worked well.

Why would this be? We want to publish an R package (and a paper) for ecological modelling and we're improving it by using rasterEngine, so understanding the root of this issue would be handy.

The long version follows:

I ran our code for 3 regions, 2 of them worked well but the third and largest didn't (details below):

Region 1 (OK): 12620 cols x 9766 rows
	       10m x 10m
	       Uncompressed size 940MB

Region 2 (OK): 15158 cols x 11389 rows
	       10m x 10m
	       Uncompressed size 1.29GB

Region 3 (FAIL): 28348 cols x 27995 rows
		 10m x 10m
		 Uncompressed size 5.91GB

Region 3 TEST 1 (OK): 14174 cols x 13998 rows
		      20m x 20m
		      Uncompressed size 1.48GB

Region 3 TEST 2 (OK): 18899 cols x 18663 rows
		      15m x 15m
		      Uncompressed size 2.63GB

In the failed attempt rasterEngine exits with this error:

Iteration:  318  of  319 
Error in X[, texture_tr$row[j]:texture_tr$row2[j], , drop = FALSE] : 
  subscript out of bounds
Calls: calc.moving.window ... focal_hpc_focal_processing -> mapply -> <Anonymous> -> sapply -> lapply -> FUN
Timing stopped at: 708.364 131.9 5757.259 
Execution halted

And the code invoking rasterEngine is this:

calc.moving.window <- function(r.input, func, r.output.name, circular_filter, radius, res, val, block_size){
  system.time(
    r.output <- rasterEngine(
      inraster=r.input,
      fun=func,
      args = list(circ_filter = circular_filter, dim = 1+(2*radius/res), centre = 1+(radius/res), value = val),
      window_dims=c(1+(2*radius/res),1+(2*radius/res)), # ncol = 1+(2*radius/res), nrow = 1+(2*radius/res) in the neighborhood
      filename = r.output.name,
      compileFunction=FALSE,
      debugmode=FALSE,
      setMinMax=FALSE,
      verbose=TRUE,
      additional_header = NULL,
      outformat = "GTiff",
      blocksize = block_size # Note before I found a fix for the error this line didn't exist
    )
  )
  
  writeRaster(r.output, r.output.name, format="GTiff")
}

I specified a blocksize of 140 rows, therefore forcing rasterEngine to perform 200 iterations. Then the code ran well.

I tried this in different machines, one in Windows 2012 R2 64bit (16 cores, 64GB RAM) runnng R_3.1.3 and latest version of spatial.tools and another Linux machine (requesting 16 cores, 6GB RAM) running R_3.2.5 and latest version of spatial.tools. Same results in both, same workaround.

Many thanks,

Javi

Javier Navarro
Experimental Scientist 
Health and Biosecurity 
CSIRO
E javi.navarro at csiro.au T +61 7 3833 5723 M +61 403 664 056
EcoSciences Precinct
Site Address: 41 Boggo Road, Dutton Park QLD 4102, Australia
Postal Address: GPO Box 2583, Brisbane QLD 4001, Australia
www.csiro.au | http://people.csiro.au/N/J/Javi-Navarro.aspx


PLEASE NOTE
The information contained in this email may be confidential or privileged. Any unauthorised use or disclosure is prohibited. If you have received this email in error, please delete it immediately and notify the sender by return email. Thank you. To the extent permitted by law, CSIRO does not represent, warrant and/or guarantee that the integrity of this communication has been maintained or that the communication is free of errors, virus, interception or interference. 
Please consider the environment before printing this email.