AW: [R] estimating number of clusters ("Null or more")

[Christian Hennig]

Dear Valery,

On Thu, 24 Apr 2003, Khamenia, Valery wrote:

>  Meanwhile I'd like to note that probably it 
>   is a good idea to put 2-3 lines of R-code demonstrating such a 
>   simple needs somnewhere in docs of `cluster' package. E.g.
> 
>   x<-rnorm(500)
>   ... # output means we have rather 1 claster
> 
>   x<-c(rnorm(500), rnorm(500)+5)
>   ... # output means we have rather 2 or more claster
> 
>   It would be nice not only for me.

I agree totally.

> > EMclust of library mclust decides about an optimal number of mixture
> > components using the BIC.
> 
> It is not clear for me whether one could use BIC without a
> statement about the familiy of distribution. Indeed BIC is based 
> on likelihood, and what the likelihood should be if the only 
> adequate statement about the destribution is the ECDF itself?..

The problem is that you have to formalize what a cluster is, and this is
not a well defined notion. It has different meanings in different
applications. My interpretation of the normal mixture/BIC approach is that
it should work well if *your* concept of a cluster is that it looks
normal-shaped (and the clusters do not need to be separated too strongly).
Normal mixtures (sometimes with lots of components) are reasonable
approximations to a wide class of distributions, so the validity of the
approach is rather a question of your cluster concept than of the
distribution of the data. (However, if your concept of "homogeneity" does
not look normal, BIC may often decide for more than one component for
*in your sense* homogeneous data.)

Some material about my own point of view is given in "What clusters are
generated by Normal mixtures?" on
http://www.math.uni-hamburg.de/home/hennig/ -> Papers/publications
with associated R-software (fixed point clusters) on the same website. 

> > it's non-trivial and at least you have to design the 
> > simulation so that rejection/acceptance is not a 
> > consequence of different scaling of data and null model. 
> 
> not clear here :-)

This means: Do not use N(0,1) as null distribution for homogeneous data if your
data has variance 5 and the test statistics is not scale equivariant (as
k-nearest neighbors and others). A bit more general you have to think about
which features of your data should enter into your homogeneous null model
(which makes the procedure a parametric bootstrap with non-guaranteed
validity of p-values). 

Best,
Christian

-- 
***********************************************************************
Christian Hennig
Seminar fuer Statistik, ETH-Zentrum (LEO), CH-8092 Zuerich (currently)
and Fachbereich Mathematik-SPST/ZMS, Universitaet Hamburg
hennig at stat.math.ethz.ch, http://stat.ethz.ch/~hennig/
hennig at math.uni-hamburg.de, http://www.math.uni-hamburg.de/home/hennig/
#######################################################################
ich empfehle www.boag.de