[R] Error in vector("double", length) : vector size specified is too large....VLDs

[Tom Colson]

Yes, using geoR. 

I can interpolate the DEM quite easily in Grass (v.surf.rst, kriging) and
block kriging in ArcInfo. What we need, though, is to be able to "estimate"
or even nail down the variogram for these data sets. Where am I going with
this? I'm guessing that variables such as slope, ruggedness, etc.. are going
to produce different sill, range, and nugget values, which I can then use to
fine tune the interpolation process, rather than using the same spline or
kriging parameters on say, a whole state boundary worth of Lidar data.  And
yes, I can estimate the variogram in ArcInfo (limited to 10000 points) and
can also import the DEM from grass into R using spgrass....but the point is
to analyze the point data BEFORE I make the DEM. 

So I'm guessing the geoR isn't ever going to handle this size data, and I
need to be using gstat? (As I write this, gstat(variogram) is plugging away
for last 10 minute with no errors.....)

Thanks for quick replies


Thomas Colson
North Carolina State University
Department of Forestry and Environmental Resources
(919) 673 8023
tom_colson at ncsu.edu

Calendar:
www4.ncsu.edu/~tpcolson
 

-----Original Message-----
From: Roger Bivand [mailto:Roger.Bivand at nhh.no] 
Sent: Thursday, September 15, 2005 3:28 PM
To: Peter Dalgaard
Cc: Tom Colson; r-help at stat.math.ethz.ch
Subject: Re: [R] Error in vector("double", length) : vector size specified
is too large....VLDs

On 15 Sep 2005, Peter Dalgaard wrote:

> "Tom Colson" <tom_colson at ncsu.edu> writes:
> 
> >  
> > > rm(data1)
> > > variog.1.b <- variog(raw.data)
> > variog: computing omnidirectional variogram Error in 
> > vector("double", length) : vector size specified is too large
> > 
> > Turns out I was wrong re: # of rows...it's 304,000
> > 
> > 
> > Same problem. Version is 2.1.1, hardware is Dual Xeon 3.6 4 GB RAM, 
> > XP Pro
> > 64 Bit. Can reproduce the problem with 64Bit R 2.1.1 running on 
> > Fedora 4, same hardware.
> > 
> 
> Variograms involve the differences between all pairs of points which 
> can become a rather large number of values. 304000*303999/2 in your 
> case, about 344GB by my reckoning. And the distances between them 
> makes for a similar quantity.
> 
> Now, some algorithms may be smarter than to keep all values in memory, 
> but you haven't even told us where you got the variog() from. It 
> doesn't seem to be in the standard packages, although we do have
> variogram() and Variogram() in spatial and nlme.

Right, this is from geoR, which uses full matrices. I think both fields and
gstat can work with larger data sets. Whether model-based geostatistics is
what you need for interpolating a digital elevation model is another
question.

> 
> 

-- 
Roger Bivand
Economic Geography Section, Department of Economics, Norwegian School of
Economics and Business Administration, Helleveien 30, N-5045 Bergen,
Norway. voice: +47 55 95 93 55; fax +47 55 95 95 43
e-mail: Roger.Bivand at nhh.no