[R-sig-Geo] getValues: error extracting values of large raster

[Jonathan Greenberg]

Herry:

This is going to be an issue with your R install, not something
specific to raster -- you are attempting to load an entire raster into
an in-memory vector/matrix, and some versions/installs of R only allow
vectors of length 2^31-1 = 2,147,483,647 values.  Your raster has
6,154,548,000 values, approximately 3x this size.  If your data is
being read in as a 16 bit integer, this means you need 6,154,548,000
times 16 bits equals approximately 12 gb of RAM.  If it is being read
in as a double precision floating point, this means the vector needs
to have at least 48gb of RAM.

You might want to search for that error "long vectors not supported
yet" and see what solutions people have.

As a general rule, you should try to avoid loading an entire raster
into memory if you can avoid it -- why are you trying to do a full
read-into-memory?  Can you do chunk-based processing?  Subsampling?

--j


On Tue, Jul 15, 2014 at 1:07 AM,  <Alexander.Herr at csiro.au> wrote:
> Hello,
> I am getting the following:
>
>
> While extracting all values with:
>> rs<-getValues(rster)
>
> Error in getRasterData(con, offset = offs, region.dim = reg, band = object at data@band): long vectors not supported yet: memory.c:3308
>
>> rster
> class       : RasterLayer
> dimensions  : 76740, 80200, 6154548000  (nrow, ncol, ncell)
> resolution  : 50, 50  (x, y)
> extent      : -1888000, 2122000, -4847000, -1010000  (xmin, xmax, ymin, ymax)
> coord. ref. : +proj=aea +lat_1=-18 +lat_2=-36 +lat_0=0 +lon_0=132 +x_0=0 +y_0=0 +ellps=GRS80 +towgs84=0,0,0,0,0,0,0 +units=m +no_defs
> data source : /data/her134/BA/GDA94albers/clum50m0314
> names       : clum50m0314
> values      : 100, 663  (min, max)
> attributes  :
>         ID  COUNT LU_CODEV7 LU_CODEV7N                                  TERTIARY_V7
>  from: 100 540931     1.0.0        100  1.0.0 Conservation and natural environments
>  to  : 663   1755     6.6.3        663 6.6.3 Estuary/coastal waters - intensive use
>                               SECONDARY_V7                              PRIMARY_V7 CLASSES_18
>  1.0 Conservation and natural environments 1 Conservation and natural environments          1
>                 6.6 Estuary/coastal waters                                 6 Water         17
>            C18_DESCRIPTION
>  Nature conservation (1.1)
>                Water (6.0)
>
>
>
> Using:
> platform       x86_64-unknown-linux-gnu
> arch           x86_64
> os             linux-gnu
> system         x86_64, linux-gnu
> status
> major          3
> minor          1.0
> year           2014
> month          04
> day            10
> svn rev        65387
> language       R
> version.string R version 3.1.0 (2014-04-10)
> nickname       Spring Dance
>
> raster version 2.2.31
>
> Linux:
> LSB Version:    core-2.0-noarch:core-3.2-noarch:core-4.0-noarch:core-2.0-x86_64:core-3.2-x86_64:core-4.0-x86_64:desktop-4.0-amd64:desktop-4.0-noarch:graphics-2.0-amd64:graphics-2.0-noarch:graphics-3.2-amd64:graphics-3.2-noarch:graphics-4.0-amd64:graphics-4.0-noarch
> Distributor ID: SUSE LINUX
> Description:    SUSE Linux Enterprise Server 11 (x86_64)
> Release:        11
> Codename:       n/a
>
>
> I used to be able to handle this sized raster in earlier versions. Any ideas of a workaround?
>
> Cheers and thanks
> Herry
>
> _______________________________________________
> R-sig-Geo mailing list
> R-sig-Geo at r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-geo



-- 
Jonathan A. Greenberg, PhD
Assistant Professor
Global Environmental Analysis and Remote Sensing (GEARS) Laboratory
Department of Geography and Geographic Information Science
University of Illinois at Urbana-Champaign
259 Computing Applications Building, MC-150
605 East Springfield Avenue
Champaign, IL  61820-6371
Phone: 217-300-1924
http://www.geog.illinois.edu/~jgrn/
AIM: jgrn307, MSN: jgrn307 at hotmail.com, Gchat: jgrn307, Skype: jgrn3007