[R-sig-Geo] stack many files without loading into memory

[Michael Sumner]

On Thu Feb 05 2015 at 7:41:33 AM Dominik Schneider <
Dominik.Schneider at colorado.edu> wrote:

> I think you are correct.
> s=stack(fn,quick=T)
> writeRaster(s,'localpath/2012data')
>
>

Ugh, sorry yes that's me reading too fast. I should have suggested the next
step to writeRaster, I'm not sure why you don't include the file extension
here though? Why not

writeRaster(s,'localpath/2012data.grd')




would get the data local. I guess the trade off is that the file size is an
> order of magnitude bigger than if I saved them in an .RData file but much
> quicker to access.
>

You might achieve similar compression if you choose GeoTIFF, with the right
options (and you need rgdal). Try a test with a single layer, e.g.

s=stack(fn,quick = TRUE)
require(rgdal)
writeRaster(s[[1]],'localpath/2012data_temp01.tif', options =
c("COMPRESS=LZW", "TILED=YES")

Does the file size of "2012data_temp01.tif" look promising?

The native "rasterfile" format does not support compression as far as I
know. Tiling may be of help or hindrance, depending on the dimensions and
the extra margin added by the tiles if they need to extend beyond the
margins - you can control tile size with BLOCKX/YSIZE  if needed:
http://www.gdal.org/frmt_gtiff.html

(NetCDF4 - with ncdf4 package - can also compress and tile natively, but I
haven't tried that via raster myself).

Cheers, Mike.


> ds
>
> On Wed, Feb 4, 2015 at 12:51 PM, Dominik Schneider <
> dominik.schneider at colorado.edu> wrote:
>
>> Wouldn't that keep the link to the server on which they are stored now?
>>
> On Wed, Feb 4, 2015 at 12:50 PM, Michael Sumner <mdsumner at gmail.com>
>> wrote:
>>
>>> Why not stack(fn)
>>>
>>> ?
>>>
>>> On Thu, 5 Feb 2015 06:41 Dominik Schneider <
>>> Dominik.Schneider at colorado.edu> wrote:
>>>
>>>> Hi -
>>>> I have some data on a server but would like to bring them local in a
>>>> somewhat compressed format that is still easy to access.
>>>>
>>>> /Volumes/hD/2012 -> 100 geotiffs
>>>> ~/project/data/ -> store those geotiffs here without needing server
>>>> access.
>>>>
>>>> untested, I think I could do something like:
>>>> s=stack()
>>>> writeRaster(s,'2012stack')
>>>> fn=list.files('/Volumes/hD/2012',pattern='*.tif',full.names=T)
>>>> lapply(fn,function(f){
>>>> s=stack('2012stack')
>>>> r=raster(f)
>>>> names(r)=gsub(pattern='.tif',replacement='',basename(f))
>>>> s=addLayer(s,r)
>>>> writeRaster(s,'2012stack')
>>>> })
>>>> Or is it better to save to a .RData?
>>>> Is there a better way that doesn't require me to loop through each
>>>> geotiff
>>>> since I can't load it all into memory.
>>>> Thanks
>>>> Dominik
>>>>
>>>>         [[alternative HTML version deleted]]
>>>>
>>>> _______________________________________________
>>>> R-sig-Geo mailing list
>>>> R-sig-Geo at r-project.org
>>>> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>>>>
>>>
>>

	[[alternative HTML version deleted]]