[R-sig-Geo] how to handle large datafile in R?

[Robert Hijmans]

Anwesha,

Assuming you want the data timestep by timestep (but you can adjust it
for all timesteps, pixel by pixel) you can do something like the
below. I copied it straight from  ?get.var.ncdf


library(ncdf)
ncdata<-open.ncdf("air.2m.1950-1999.nc")
lon<-get.var.ncdf(ncdata,"lon")
lat<-get.var.ncdf(ncdata,"lat")
time<-get.var.ncdf(ncdata,"time")

v3      <- ncdata$var[["air"]]
varsize <- v3$varsize
ndims   <- v3$ndims

nt      <- varsize[ndims]  # Remember timelike dim is always the LAST dimension!
for( i in 1:nt ) {
        # Initialize start and count to read one timestep of the variable.
        start <- rep(1,ndims)   # begin with start=(1,1,1,...,1)
        start[ndims] <- i       # change to start=(1,1,1,...,i) to
read timestep i
        count <- varsize        # begin w/count=(nx,ny,nz,...,nt),
reads entire var
        count[ndims] <- 1       # change to count=(nx,ny,nz,...,1) to
read 1 tstep
        data3 <- get.var.ncdf( ncdata, v3, start=start, count=count )

        # Now read in the value of the timelike dimension
        timeval <- get.var.ncdf( ncdata, v3$dim[[ndims]]$name,
start=i, count=1 )

        print(paste("Data for variable",v3$name,"at timestep",i,
                " (time value=",timeval,v3$dim[[ndims]]$units,"):"))
#       print(data3)
        }

close.ncdf(nc)



On Tue, Apr 7, 2009 at 3:31 PM, Anwesha Chakrabarti <c.anwesha at gmail.com> wrote:
> i'm really sorry for that.
> let me state the problem once again with the codes.
>
> graphics.off()
>> library(ncdf)
>> ncdata<-open.ncdf("air.2m.1950-1999.nc")
>> lon<-get.var.ncdf(ncdata,"lon")
>> lat<-get.var.ncdf(ncdata,"lat")
>> time<-get.var.ncdf(ncdata,"time")
>> dat<-get.var.ncdf(ncdata,"air")
> Error: cannot allocate vector of size 1.2 Gb
> now my question is how do i load the subsets of the data?
>
> regards
>
> On Tue, Apr 7, 2009 at 12:42 PM, Roger Bivand <Roger.Bivand at nhh.no> wrote:
>
>>  On Tue, 7 Apr 2009, Anwesha Chakrabarti wrote:
>>
>> hi all,
>>> i want to read a .netcdf file in R. the file contains one variable that
>>> has
>>> three dimensions lon, lat and time.
>>> and the data consists of a 192x94 matrix of the variable. whenever i'm
>>> trying to open the file i'm getting the following error msg
>>> "can not allocate vector of size 1.2gb"!
>>> so i guess i have to open the data in short slices.
>>> could anyone tell me how can i do this?
>>>
>>
>> They cannot. In order to get any help, it is essential that you paste a
>> copy of your code into your message - including the error message(s),
>> followed by the output of traceback() and sessionInfo() in that order. We
>> don't even know which function in which package you are using, and whether
>> you are using it correctly. As far as I know, no-one on the list can see
>> over your shoulder ...
>>
>> Roger
>>
>>
>>> regards
>>> anwesha
>>>
>>>        [[alternative HTML version deleted]]
>>>
>>> _______________________________________________
>>> R-sig-Geo mailing list
>>> R-sig-Geo at stat.math.ethz.ch
>>> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>>>
>>>
>> --
>> Roger Bivand
>> Economic Geography Section, Department of Economics, Norwegian School of
>> Economics and Business Administration, Helleveien 30, N-5045 Bergen,
>> Norway. voice: +47 55 95 93 55; fax +47 55 95 95 43
>> e-mail: Roger.Bivand at nhh.no
>>
>>
>
>        [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-Geo mailing list
> R-sig-Geo at stat.math.ethz.ch
> https://stat.ethz.ch/mailman/listinfo/r-sig-geo
>