Hi
   
  On another email list (classification society discussion of clustering and classification) I described a simulation experiment experiment.  In brief, generate a 20 by 1000 matrix of random numbers, assign the first ten rows to class 1 and the second 10 rows to class 2.  With 1,000 columns of random numbers perfect discrimination between the two classes is assured (try it yourself).
   
  Doug Carroll, a famous statistician in the cluster/classification literature made th following insightful comment:
   
  "It's well known that the two group discriminant analysis problem Bill defines is equivalent to multiple linear regression predicting one dependent variable with (in this case) 900 independent variables.  The resulting R^2 (R-squared) will have an expected value virtually equal to 1.0 (900/999= .9009), which would translate into a (nominally) near perfect discriminant analysis.  What needs to be done is to correct the R^2 for attenuation-- in which case, under the circumstances described, the expected ADJUSTED R^2 would be zero (0.0).  There are ways to do the discriminant analysis (whether two group or multigroup) correcting for number of parameters (independent variables), and are no doubt ways to do so in the tree software problem you're concerned with as well."

   
  The lesson to me is to be very cautious when working in the large P small N problem.  Strategies such as bootstrapping, CV etc will appear good but that is because the high dimensional space results in spurious separation of groups.
   
  I approach all high throughput molecular data analysis as exploratory (think Tukey) with the results of a microarray/cgh/proteomics/metabolimics... study being useful only for feedback to the lab for further experiments.  These studies should not be conclusive and building classifiers should be avoided.
   
   
  Thank you.
   
  Bill Shannon, PhD
BioRankings, LLC
Founder and President
314-704-8725 
http://BioRankings.com
   
  "Helping you gain understanding of statistical data to protect your investment."
   
  Associate Professor of Biostatistics in Medicine
Washington University School of Medicine
http://ilya.wustl.edu/~shannon

  


martin.schumacher@novartis.com wrote:
  Dear Roger,

You may want to consult the paper cited below.

Best regards,
Martin





************************************************************

Bioinformatics. 2004 Feb 12;20(3):374-80 



Is cross-validation valid for small-sample microarray classification?
Braga-Neto UM, Dougherty ER.
Section of Clinical Cancer Genetics, University of Texas MD Anderson 
Cancer Center, Houston, TX, USA.
MOTIVATION: Microarray classification typically possesses two striking 
attributes: (1) classifier design and error estimation are based on 
remarkably small samples and (2) cross-validation error estimation is 
employed in the majority of the papers. Thus, it is necessary to have a 
quantifiable understanding of the behavior of cross-validation in the 
context of very small samples. RESULTS: An extensive simulation study has 
been performed comparing cross-validation, resubstitution and bootstrap 
estimation for three popular classification rules-linear discriminant 
analysis, 3-nearest-neighbor and decision trees (CART)-using both 
synthetic and real breast-cancer patient data. Comparison is via the 
distribution of differences between the estimated and true errors. Various 
statistics for the deviation distribution have been computed: mean (for 
estimator bias), variance (for estimator precision), root-mean square 
error (for composition of bias and variance) and quartile ranges, 
including outlier behavior. In general, while cross-validation error 
estimation is much less biased than resubstitution, it displays excessive 
variance, which makes individual estimates unreliable for small samples. 
Bootstrap methods provide improved performance relative to variance, but 
at a high computational cost and often with increased bias (albeit, much 
less than with resubstitution).






"Roger Liu" 
Sent by: bioconductor-bounces@stat.math.ethz.ch
31.08.2007 17:18

To
bioconductor 
cc

Subject
[BioC] small sample size,microarray, cross-validation






Dear Friends,

I am trying to use some supervised method to identify the genes which 
permit
the separation of two patients conditions.

Due to the extreme difficulty, I can only have the test set (about 17
patients), could I just use these 17 data sets as training set as well as
test, such as using leave one out cross validation?
I know its result may be optimistic, but I am just wondering:

1. if you have any good idea about this small sample size issue in
classification analysis.
2. is the cross-validation a legitimate method to support that I have
identified the genes which can separate the groups as different 
conditions?
we try to get this result published, will it be an OK as using this
cross-validation?
3. what packages, methods, will you recommend for this issue?

Thanks.


Regards,
-Roger

[[alternative HTML version deleted]]

_______________________________________________
Bioconductor mailing list
Bioconductor@stat.math.ethz.ch
https://stat.ethz.ch/mailman/listinfo/bioconductor
Search the archives: 
http://news.gmane.org/gmane.science.biology.informatics.conductor


[[alternative HTML version deleted]]

_______________________________________________
Bioconductor mailing list
Bioconductor@stat.math.ethz.ch
https://stat.ethz.ch/mailman/listinfo/bioconductor
Search the archives: http://news.gmane.org/gmane.science.biology.informatics.conductor


	[[alternative HTML version deleted]]

