Thierry, 

 Search of CRAN with "sparse clustering" yielded cluster.dist {cba}, 
defined as "Clustering a Sparse Symmetric Distance Matrix".  There were 
also sparse PCA packages and sparse matrix classes.  I have no experience 
with these procedures. 

As additional background, you might like to say what kind of clustering 
you want to do and whether some particular similarity/distance will be 
involved. 
Does your cluster analysis program take a data frame as input? 

However, it sounds like you are having problems with preliminary data 
processing, and may not yet know whether some cluster analysis procedure 
or other would choke on your matrix, once it is computed. 

It does seem surprising that you are having problems with a problem of 
this size.  I assume you have checked that you have a couple G or so free, 
at least. 

Farrar 




r-sig-ecology-bounces@r-project.org wrote on 10/07/2008 08:35:39 AM:

> "ONKELINX, Thierry" <Thierry.ONKELINX@inbo.be>
> writes:
> 
> > Dear all,
> >
> > We have a problem with a large dataset that we want to cluster. The
> > dataset is in a long format: 1154024 rows with presence data. Each row
> > has the name of the species and the location. We have 1381 species and
> > 6354 locations.
> > The main problem is that we need the data in wide format (one row for
> > each location, one column for each species) for the clustering
> > algorithms. But the 6354 x 1381 dataframe is too big to fit into the
> > memory. At least when we use cast from the reshape package to convert
> > the dataframe from a long to a wide format.
> >
> > Are there any clustering tools available that can work with the data 
in
> > a long format or with sparse matrices (only 13% of the matrix is
> > non-zero)? If the work with sparse matrices: how to convert our 
dataset
> > to a sparse matrix? Other suggestions are welcome.
> >
> 
> 6354 x 1381 should be well within your memory limit, so I assume it's
> the intermediate steps that are fouling you up. Maybe you can do it in
> pieces: 
> 
> 1. subset the original two-column matrix to include only the first 100 
sites
> 2. convert this subset to wide form
> 3. repeat 63 times for different subsets
> 4. rbind the resulting matrices
> 
> Good luck,
> 
> Tyler
> 
> -- 
> Watching a recorded television broadcast more than once will be illegal
> under Bill C-61. 
> 
> http://www.michaelgeist.ca/content/view/3046/125/
> 
> _______________________________________________
> R-sig-ecology mailing list
> R-sig-ecology@r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-ecology

	[[alternative HTML version deleted]]

