[R] agnes clustering and NAs

Dario Strbenac D.Strbenac at garvan.org.au
Sun Jan 30 04:00:05 CET 2011


Hello,

Thankyou for the clarification about the NAs. For your interest, thankfully my end goal was not to plot a dendrogram with 23371 elements, but just to use the output of the clustering to re-order the rows of a matrix before plotting it with image(). Since clara() and pam() are partitioning based approaches, I suppose I could instead stay with hclust() after removing the offending rows, so that I have the ordering position of each gene, not its cluster membership. I have 12 GB RAM on my 64-bit system, so the time it takes to run should be my only problem.

- Dario.

---- Original message ----
>Date: Fri, 28 Jan 2011 12:34:26 +0100
>From: Martin Maechler <maechler at stat.math.ethz.ch>  
>Subject: Re: [R] agnes clustering and NAs  
>To: gavin.simpson at ucl.ac.uk
>Cc: D.Strbenac at garvan.org.au, r-help at r-project.org, Uwe Ligges <ligges at statistik.tu-dortmund.de>
>
>>>>>> Gavin Simpson <gavin.simpson at ucl.ac.uk>
>>>>>>     on Fri, 28 Jan 2011 09:23:05 +0000 writes:
>
>    > On Fri, 2011-01-28 at 10:00 +1100, Dario Strbenac wrote:
>    >> Hello,
>    >> 
>    >> Yes, that's right, it is a values matrix. Not a dissimilarity matrix.
>    >> 
>    >> i.e.
>    >> 
>    >> > str(iMatrix)
>    >> num [1:23371, 1:56] -0.407 0.198 NA -0.133 NA ...
>    >> - attr(*, "dimnames")=List of 2
>    >> ..$ : NULL
>    >> ..$ : chr [1:56] "-8100" "-7900" "-7700" "-7500" ...
>
>Ok, so in the end you want to draw a dendrogram for  23'371
>observational units, really ?
>
>I think I would not use a hierarchical clustering method for so
>many units, but rather  clara() or maybe pam() or then model
>based or other methods, rather than fully hierarchical ones....
>...
>but yes, that's not the issue here, and see further down ...
>
>BTW:  The object 'iMatrix' you provided for download has only 50
>      columns, not 56...
>    >> 
>    >> For the snippet of checking for NAs, I get all TRUEs, so I have at least one NA in each column.
>
>    GS> Sorry, my bad. Try this:
>
>    GS> apply(iMatrix, 1, function(x) all(is.na(x)))
>
>    GS> will check that you have no fully `NA` rows.
>
>    GS> Also look at str(iMatrix) for potential problems.
>
>    GS> Finally, try:
>
>    GS> out <- dist(iMatrix) any(is.na(out))
>
>    GS> should repeat what agnes is doing to compute the
>    GS> dissimilarity matrix.  If that returns TRUE, go and find
>    GS> which samples are giving NA dissimilarity and why.
>
>    GS> The issue is not NA in the input data, but that your
>    GS> input data is leading to NA in the computed
>    GS> dissimilarities. This might be due to NA's in your input
>    GS> data, where a pair of samples has no common set of data
>    GS> for example.
>
>Yes, that's right on spot, thank you Gavin.
>
>This is indeed to true:  
>It *does* allow for NA's (in the data matrix), but if the
>pattern of NA's is such that the dissimilarity between two
>observations becomes undefined, namely e.g. if they have no
>common non-missings, then ``that's too much''.
>
>In general, I'd recommend to use 
>  dm <- daisy(....,...) 
>trying methods, that are better with NAs, e.g. Gower's metric,
>until dm() has {nearly} no NAs,
>and then figure out some imputation to replace all NA's in   dm
>by "reasonable values",
>then do clustering with the resulting dissimilarity "matrix" dm.
>
>HOWEVER, in your case, dm would correspond to 
> 23371 x 23371 dissimilarity matrix,
>stored as a double precision matrix (on a 64-bit platform)
>that's an object of size 4.4 GBytes, not very convenient to work
>with.
>as dissimilarity object it will only be about half of that size,
>but that's still ``a bit large''..
>As I said above, for such data, I would never do fully
>hierarchical clustering,
>but rather something else.
>
>Martin Maechler, ETH Zurich
>
>
>    GS> HTH
>    GS> G
>
>    >> The part of the agnes documentation I was referring to is :
>    >> 
>    >> "In case of a matrix or data frame, each row corresponds to an observation, and each column corresponds to a variable. All variables must be numeric.  Missing values (NAs) are allowed."
>    >> 
>    >> So, I'm under the impression it handles NAs on its own ?
>    >> 
>    >> - Dario.
>    >> 
>    >> ---- Original message ----
>    >> >Date: Thu, 27 Jan 2011 12:53:27 +0000
>    >> >From: Gavin Simpson <gavin.simpson at ucl.ac.uk>  
>    >> >Subject: Re: [R] agnes clustering and NAs  
>    >> >To: Uwe Ligges <ligges at statistik.tu-dortmund.de>
>    >> >Cc: D.Strbenac at garvan.org.au, r-help at r-project.org
>    >> >
>    >> >On Thu, 2011-01-27 at 10:45 +0100, Uwe Ligges wrote:
>    >> >> 
>    >> >> On 27.01.2011 05:00, Dario Strbenac wrote:
>    >> >> > Hello,
>    >> >> >
>    >> >> > In the documentation for agnes in the package 'cluster', it says that NAs are allowed, and sure enough it works for a small example like :
>    >> >> >
>    >> >> >> m<- matrix(c(
>    >> >> > 1, 1, 1, 2,
>    >> >> > 1, NA, 1, 1,
>    >> >> > 1, 2, 2, 2), nrow = 3, byrow = TRUE)
>    >> >> >> agnes(m)
>    >> >> > Call:    agnes(x = m)
>    >> >> > Agglomerative coefficient:  0.1614168
>    >> >> > Order of objects:
>    >> >> > [1] 1 2 3
>    >> >> > Height (summary):
>    >> >> >     Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
>    >> >> >    1.155   1.247   1.339   1.339   1.431   1.524
>    >> >> >
>    >> >> > Available components:
>    >> >> > [1] "order"  "height" "ac"     "merge"  "diss"   "call"   "method" "data"
>    >> >> >
>    >> >> > But I have a large matrix (23371 rows, 50 columns) with some NAs in it and it runs for about a minute, then gives an error :
>    >> >> >
>    >> >> >> agnes(iMatrix)
>    >> >> > Error in agnes(iMatrix) :
>    >> >> >    No clustering performed, NA-values in the dissimilarity matrix.
>    >> >> >
>    >> >> > I've also tried getting rid of rows with all NAs in them, and it still gave me the same error. Is this a bug in agnes() ? It doesn't seem to fulfil the claim made by its documentation.
>    >> >> 
>    >> >> 
>    >> >> I haven't looked in the file, but you need to get rid of all NA, or in 
>    >> >> other words, all rows that contain *any* NA values.
>    >> >
>    >> >If one believes the documentation, then that only applies to the case
>    >> >where `x` is a dissimilarity matrix. `NA`s are allowed if x is the raw
>    >> >data matrix or data frame.
>    >> >
>    >> >The only way the OP could have gotten that error with the call shown is
>    >> >if iMatrix were not a dissimilarity matrix inheriting from class "dist",
>    >> >so `NA`s should be allowed.
>    >> >
>    >> >My guess would be that the OP didn't get rid of all the `NA`s.
>    >> >
>    >> >Dario: what does:
>    >> >
>    >> >sapply(iMatrix, function(x) any(is.na(x)))
>    >> >
>    >> >or if iMatrix is a matrix:
>    >> >
>    >> >apply(iMatrix, 2, function(x) any(is.na(x)))
>    >> >
>    >> >say?
>    >> >
>    >> >G
>    >> >
>    >> >> Uwe Ligges
>    >> >> 
>    >> >> 
>    >> >> 
>    >> >> > The matrix I'm using can be obtained here :
>    >> >> > http://129.94.136.7/file_dump/dario/iMatrix.obj
>    >> >> >
>    >> >> > --------------------------------------
>    >> >> > Dario Strbenac
>    >> >> > Research Assistant
>    >> >> > Cancer Epigenetics
>    >> >> > Garvan Institute of Medical Research
>    >> >> > Darlinghurst NSW 2010
>    >> >> > Australia
>    >> >> >
>
>    >> >-- 
>    >> >%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%
>    >> > Dr. Gavin Simpson             [t] +44 (0)20 7679 0522
>    >> > ECRC, UCL Geography,          [f] +44 (0)20 7679 0565
>    >> > Pearson Building,             [e] gavin.simpsonATNOSPAMucl.ac.uk
>    >> > Gower Street, London          [w] http://www.ucl.ac.uk/~ucfagls/
>    >> > UK. WC1E 6BT.                 [w] http://www.freshwaters.org.uk
>    >> >%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%


--------------------------------------
Dario Strbenac
Research Assistant
Cancer Epigenetics
Garvan Institute of Medical Research
Darlinghurst NSW 2010
Australia



More information about the R-help mailing list