[R-sig-Geo] Error:can not allocate vector of size ..., using ff or bigmemory?

[Roger Bivand]

On Sun, 2 Dec 2012, Raffaele Morelli wrote:

> 2012/12/2 ADAM PETER CARDILINI <apcar at deakin.edu.au>
>
>> G'day Everyone,
>>
>> I have recently been trying to run some analysis on very large raster
>> files (~200mb, file found here<
>> https://www.ga.gov.au/products/servlet/controller?event=FILE_SELECTION&catno=71071>
>> the first option "Tagged Image File Format (TIFF), Contains the final 34
>> DLC classes, 25.77MB") for Australian land cover. I can successfully read
>> this file into R and isolate the information I am interested in, eg.,
>>
>> # to get a tree cover map, where tree cover is 1 and everything else is 0
>> library(raster)
>>                                 landcover<-raster("landcover_file.tiff") #
>> I can only open the large TIFF file as a raster?
>> isBecomes<
>> -cbind(c(1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34),c(NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA,1,1,NA,NA))
>> treecover <- reclass(landcover, rcl=isBecomes)
>>
>> I would then like to get some distance data from this treecover map using
>> gdistance. I try to create a transition matrix from the treecover raster
>> (which will be incredibly huge) and get the Error: can not allocate vector
>> of size ... .
>>
>>                                 library(gdistance)
>> tr1 <- transition(r, transitionFunction=mean, directions=8)
>>
>> Error: cannot allocate vector of size 1.1 Gb
>> In addition: Warning messages:
>> 1: In `slot<-`(.Object, slotName, check = FALSE, slotVal) :
>>   Reached total allocation of 3892Mb: see help(memory.size)
>> 2: In `slot<-`(.Object, slotName, check = FALSE, slotVal) :
>>   Reached total allocation of 3892Mb: see help(memory.size)
>> 3: In `slot<-`(.Object, slotName, check = FALSE, slotVal) :
>>   Reached total allocation of 3892Mb: see help(memory.size)
>> 4: In `slot<-`(.Object, slotName, check = FALSE, slotVal) :
>>   Reached total allocation of 3892Mb: see help(memory.size)
>>
>> I am wondering what I could do to get around this? Is there a better way
>> to work with such large files? I have been reading up on ff and bigmemory
>> but am finding it hard to find a simple introduction that explains how to
>> work with raster files in these frameworks. In other words, I sort of get
>> what these packages are doing I just don't understand how to implement
>> them. Does anyone have example code for importing a raster using ff and
>> then working on it with some functions?
>>
>> I am running 64bit R on a laptop with only 4GB RAM. The only other thing I
>> can think of doing is using a machine with more RAM, unfortunately that's
>> not really an option. Any suggestions or help would be great and really
>> appreciated. Thanks in advance for your time and suggestions.
>>
>>
>> Kind regards,
>>
>> Adam Cardilini
>>
>
> More RAM won't help you, I've had the same problem on a 9GB RAM Dell server
> with 8 processors and it really seems that R can't handle such big
> rasters...

I believe that you mean that you "can't handle such big rasters..."? The 
underlying question is related to your workflow, and to whether your 
distances could better be represented as sparse matrices, dropping 
measuements beyond a given threshold. Probably you don't need all the 
distances, so if you think through the logic of what you want to do, I 
don't doubt that you can get there. If you ask any system to do something 
rash (like measure and store lots of distances you will subsequently 
discard), you shouldn't be surprised when it runs out of resources.

Roger

>
> BTW I didn't know about "ff" and "bigmemory", I'll have a look
>
> -raffaele
>
> 	[[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-Geo mailing list
> R-sig-Geo at r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>

-- 
Roger Bivand
Department of Economics, NHH Norwegian School of Economics,
Helleveien 30, N-5045 Bergen, Norway.
voice: +47 55 95 93 55; fax +47 55 95 95 43
e-mail: Roger.Bivand at nhh.no