[R] help about the dist() error message

[Douglas Bates]

Thomas Lumley <tlumley at u.washington.edu> writes:

> On Wed, 16 Apr 2003, Meng Xue wrote:
> 
> > hello,
> >
> > I have a problem when call dist() of package mva on a large data
> > set. The following error message are obtained when I use dist()
> > on m, a two dimensional data set with the size 100,000. It seems
> > that the length of vector out of bound, since when I compute
> > dist on small data set(10,000), it works well. but for large
> > data set, it   exit with the message "negative length vector are
> > not allowed" .
> >
> 
> With 100,000 points there are 50 billion pairwise distances, which would
> take at least 400 Gb of memory.  It probably wouldn't even fit on your
> disk, let alone in memory.

You exaggerate.  It's a mere 40 GB, I believe.

> N = 100000
> 8*(N * (N - 1))/2
[1] 39999600000
> (8*(N * (N - 1))/2)/2^30
[1] 37.25253