[R] Error in vector("double", length) : vector size specified is too large....VLDs

[Roger Bivand]

On 15 Sep 2005, Peter Dalgaard wrote:

> "Tom Colson" <tom_colson at ncsu.edu> writes:
> 
> >  
> > > rm(data1)
> > > variog.1.b <- variog(raw.data)
> > variog: computing omnidirectional variogram
> > Error in vector("double", length) : vector size specified is too large
> > 
> > Turns out I was wrong re: # of rows...it's 304,000
> > 
> > 
> > Same problem. Version is 2.1.1, hardware is Dual Xeon 3.6 4 GB RAM, XP Pro
> > 64 Bit. Can reproduce the problem with 64Bit R 2.1.1 running on Fedora 4,
> > same hardware. 
> > 
> 
> Variograms involve the differences between all pairs of points which
> can become a rather large number of values. 304000*303999/2 in your
> case, about 344GB by my reckoning. And the distances between them
> makes for a similar quantity.
> 
> Now, some algorithms may be smarter than to keep all values in memory,
> but you haven't even told us where you got the variog() from. It
> doesn't seem to be in the standard packages, although we do have
> variogram() and Variogram() in spatial and nlme.

Right, this is from geoR, which uses full matrices. I think both fields 
and gstat can work with larger data sets. Whether model-based 
geostatistics is what you need for interpolating a digital elevation model 
is another question.

> 
> 

-- 
Roger Bivand
Economic Geography Section, Department of Economics, Norwegian School of
Economics and Business Administration, Helleveien 30, N-5045 Bergen,
Norway. voice: +47 55 95 93 55; fax +47 55 95 95 43
e-mail: Roger.Bivand at nhh.no