[R] Computing very large distance matrix

[Arnav Sheth]

Hello,

Thank you all for your replies.

I have just discovered that now, for some reason, R does not let me increase
its memory limit beyond 4095 Mb.

Also, I get a different error message when computing the matrix using
daisy() as opposed to dist():

Error: Cannot allocate vector of size 135168 Kb

Does this mean that there is no way in which I can compute this distance
matrix on the machine I described below?

Is there any possible solution to this problem?

Thanks again to all,
Arnav.



----- Original Message -----
From: "Prof Brian Ripley" <ripley at stats.ox.ac.uk>
To: "Arnav Sheth" <sheth at economics.rutgers.edu>
Cc: "R-Help" <R-help at stat.math.ethz.ch>
Sent: Wednesday, February 25, 2004 2:12 PM
Subject: Re: [R] Computing very large distance matrix


> On Wed, 25 Feb 2004, Arnav Sheth wrote:
>
> >
> > Hello All,
> >
> > I have a 131072x132 matrix for which I need to compute a regular
euclidean distance matrix, which I then need to transform and run agnes() on
this transformed matrix. I am having trouble computing the distance matrix
as it is fairly large and I am sure I have gone over the max.
> >
> > The specific error I am getting is:
> > Error in vector("double", length) : negative length vectors are not
allowed
>
> We've improved that message for the next release.
>
> > I have increased the memory limit to the maximum capacity of my hard
> > drive (which is around 20gb), with no success.
>
> You'd need to increase the 32-bit limit of Windows, too ....
>
> > I am running the RGUI on Windows XP with 512 mb of RAM.
> >
> > Would anyone have any suggestions as to how I can overcome this problem?
> > I would be most grateful for any help.
>
> No. The distance matrix is 8Gb all by itself.
>
> --
> Brian D. Ripley,                  ripley at stats.ox.ac.uk
> Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
> University of Oxford,             Tel:  +44 1865 272861 (self)
> 1 South Parks Road,                     +44 1865 272866 (PA)
> Oxford OX1 3TG, UK                Fax:  +44 1865 272595
>
>