[R] cluster/distance large matrix

[Christian Hennig]

Dear Bart,

a strange thing in your question is that the term "Ward's method" 
usually refers to a method based on the k-means criterion, which, in its 
standard form, is not based on dissimilarities, but on 
"objects*variables-data".
So I wonder how and why you want to use Ward's method on a dissimilarity 
matrix in the first place (I know that the "k-means" criterion 
can in principle be translated to dissimilarity data - this is probably 
what hclust's method="ward" does if fed with a dissimilarity matrix, but 
I'm not sure -, but then it loses its justification).

One thing you could think about is using the function pam in library 
cluster. Chances are that this won't work on 38,000 cases either, but you 
may cluster a subsample of, say, 2,000 cases and assign all further 
objects to the most similar cluster medoid.

It is well know that hierarchical methods are problematic with too large 
dissimilarity matrices; even if you resolve the memory problem, the number 
of operations required is enormous.

Hope this helps,
Christian


On Thu, 11 Feb 2010, Bart Thijs wrote:

>
> Hi all,
>
> I've stumbled upon some memory limitations for the analysis that I want to
> run.
>
> I've a matrix of distances between 38000 objects. These distances were
> calculated outside of R.
> I want to cluster these objects.
>
> For smaller sets (egn=100) this is how I proceed:
> A<-matrix(scan(file, n=100*100),100,100, byrow=TRUE)
> ad<-as.dist(A)
> ahc<-hclust(ad,method="ward",members=NULL)
> ....
>
> However if I try this with the real dataset I end up with memory problems.
> I've the 64bit version of R installed on a machine with 40Gb RAM (Windows
> 2003 64bit version).
>
> I'm thinking about using only the lower triangle of the matrix but I can't
> create a distance object for the clustering from the lower.tri
>
> Can someone help me with a suggestion for which way to go?
>
> Best Regards
> Bart Thijs
> --
> View this message in context: http://n4.nabble.com/cluster-distance-large-matrix-tp1477237p1477237.html
> Sent from the R help mailing list archive at Nabble.com.
>
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>

*** --- ***
Christian Hennig
University College London, Department of Statistical Science
Gower St., London WC1E 6BT, phone +44 207 679 1698
chrish at stats.ucl.ac.uk, www.homepages.ucl.ac.uk/~ucakche