[R-sig-Geo] getValues: error extracting values of large raster

Jonathan Greenberg jgrn at illinois.edu
Thu Jul 17 07:43:29 CEST 2014 

Herry:

If you are just doing reclassification, consider using rasterEngine in
the spatial.tools package which can be used for HPC computing of
rasters.

--j

On Wed, Jul 16, 2014 at 10:43 PM,  <Alexander.Herr at csiro.au> wrote:
> Thanks Johathan,
>
> I am working on a HPC, so could get the memory, but it is obviously the vector size limit... I was attempting to load the raster values to reclassify and thought it might be faster working on vectors in memory.
>
> I have since moved to getValuesBlock() and am working it in junks.
>
>
> Cheers
> Herry
>
>
> -----Original Message-----
> From: jgrn307 at gmail.com [mailto:jgrn307 at gmail.com] On Behalf Of Jonathan Greenberg
> Sent: Thursday, 17 July 2014 12:54 PM
> To: Herr, Alexander Herr - Herry (L&W, Black Mountain)
> Cc: r-sig-geo at r-project.org
> Subject: Re: [R-sig-Geo] getValues: error extracting values of large raster
>
> Herry:
>
> This is going to be an issue with your R install, not something specific to raster -- you are attempting to load an entire raster into an in-memory vector/matrix, and some versions/installs of R only allow vectors of length 2^31-1 = 2,147,483,647 values.  Your raster has
> 6,154,548,000 values, approximately 3x this size.  If your data is being read in as a 16 bit integer, this means you need 6,154,548,000 times 16 bits equals approximately 12 gb of RAM.  If it is being read in as a double precision floating point, this means the vector needs to have at least 48gb of RAM.
>
> You might want to search for that error "long vectors not supported yet" and see what solutions people have.
>
> As a general rule, you should try to avoid loading an entire raster into memory if you can avoid it -- why are you trying to do a full read-into-memory?  Can you do chunk-based processing?  Subsampling?
>
> --j
>
>
> On Tue, Jul 15, 2014 at 1:07 AM,  <Alexander.Herr at csiro.au> wrote:
>> Hello,
>> I am getting the following:
>>
>>
>> While extracting all values with:
>>> rs<-getValues(rster)
>>
>> Error in getRasterData(con, offset = offs, region.dim = reg, band =
>> object at data@band): long vectors not supported yet: memory.c:3308
>>
>>> rster
>> class       : RasterLayer
>> dimensions  : 76740, 80200, 6154548000  (nrow, ncol, ncell) resolution
>> : 50, 50  (x, y)
>> extent      : -1888000, 2122000, -4847000, -1010000  (xmin, xmax, ymin, ymax)
>> coord. ref. : +proj=aea +lat_1=-18 +lat_2=-36 +lat_0=0 +lon_0=132
>> +x_0=0 +y_0=0 +ellps=GRS80 +towgs84=0,0,0,0,0,0,0 +units=m +no_defs data source : /data/her134/BA/GDA94albers/clum50m0314
>> names       : clum50m0314
>> values      : 100, 663  (min, max)
>> attributes  :
>>         ID  COUNT LU_CODEV7 LU_CODEV7N                                  TERTIARY_V7
>>  from: 100 540931     1.0.0        100  1.0.0 Conservation and natural environments
>>  to  : 663   1755     6.6.3        663 6.6.3 Estuary/coastal waters - intensive use
>>                               SECONDARY_V7                              PRIMARY_V7 CLASSES_18
>>  1.0 Conservation and natural environments 1 Conservation and natural environments          1
>>                 6.6 Estuary/coastal waters                                 6 Water         17
>>            C18_DESCRIPTION
>>  Nature conservation (1.1)
>>                Water (6.0)
>>
>>
>>
>> Using:
>> platform       x86_64-unknown-linux-gnu
>> arch           x86_64
>> os             linux-gnu
>> system         x86_64, linux-gnu
>> status
>> major          3
>> minor          1.0
>> year           2014
>> month          04
>> day            10
>> svn rev        65387
>> language       R
>> version.string R version 3.1.0 (2014-04-10)
>> nickname       Spring Dance
>>
>> raster version 2.2.31
>>
>> Linux:
>> LSB Version:    core-2.0-noarch:core-3.2-noarch:core-4.0-noarch:core-2.0-x86_64:core-3.2-x86_64:core-4.0-x86_64:desktop-4.0-amd64:desktop-4.0-noarch:graphics-2.0-amd64:graphics-2.0-noarch:graphics-3.2-amd64:graphics-3.2-noarch:graphics-4.0-amd64:graphics-4.0-noarch
>> Distributor ID: SUSE LINUX
>> Description:    SUSE Linux Enterprise Server 11 (x86_64)
>> Release:        11
>> Codename:       n/a
>>
>>
>> I used to be able to handle this sized raster in earlier versions. Any ideas of a workaround?
>>
>> Cheers and thanks
>> Herry
>>
>> _______________________________________________
>> R-sig-Geo mailing list
>> R-sig-Geo at r-project.org
>> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>
>
>
> --
> Jonathan A. Greenberg, PhD
> Assistant Professor
> Global Environmental Analysis and Remote Sensing (GEARS) Laboratory Department of Geography and Geographic Information Science University of Illinois at Urbana-Champaign
> 259 Computing Applications Building, MC-150
> 605 East Springfield Avenue
> Champaign, IL  61820-6371
> Phone: 217-300-1924
> http://www.geog.illinois.edu/~jgrn/
> AIM: jgrn307, MSN: jgrn307 at hotmail.com, Gchat: jgrn307, Skype: jgrn3007



-- 
Jonathan A. Greenberg, PhD
Assistant Professor
Global Environmental Analysis and Remote Sensing (GEARS) Laboratory
Department of Geography and Geographic Information Science
University of Illinois at Urbana-Champaign
259 Computing Applications Building, MC-150
605 East Springfield Avenue
Champaign, IL  61820-6371
Phone: 217-300-1924
http://www.geog.illinois.edu/~jgrn/
AIM: jgrn307, MSN: jgrn307 at hotmail.com, Gchat: jgrn307, Skype: jgrn3007

More information about the R-sig-Geo mailing list