Thanks a lot for your email. That's helpful, below is my response

Joern Toedling <toedling@ebi.ac.uk> wrote: Hello,

I am a bit puzzled about what you actually want to ask.

James Anderson wrote:
> For binary classification problem in microarray, if you do some random subsampling classification (every time split data into 80% training and 20% test with stratification (perserving the ratio in each class), repeat many times). When you get some results, one thing you would normally look at is how significantly different is your results from what you are going to get by chance, that's why people do label permutation test. My question is that: Are the final results of label permutation test for accuracy equal to the proportion of the large class (say there are 80 normal vs. 20 disease, is the mean accuracy of label permutation test equal to 80/(80+20) as long as you repeat enough times? Is this classifier independent? 
>   

The mean accuracy of your classifier after label permutation, in a
cross-validation setting presumably, depends very much on your
classifier. 
 
It makes perfect sense that accuracy after label permutation test should depend on classifier, does it have anything to do with the sample unbalance? Intuitively it should, for example, if one class has 1000 samples, the other has 50 samples, it seems that the accuracy for label permutation test should be very high. 


What you should contrast it to is the accuracy of the naive
classifier "assign every sample to the larger class", 80% in your case.
A good reason for label permutation in your case is that you want to
assess the classifier's generalizability, because one can always
construct a classifier that has an accuracy of 100% on the training
data, but performs badly on independent test data. That is one reason
why people do label permutation with classification because the
classifier's mean accuracy in a cross-validation setting gives a better
estimate of the classifier's accuracy on test data. (You have to make
sure that you do not use any aspect of the set-aside training data for
training the classifier, though.) An even better estimate for your
classifier's performance, however, would be its accuracy on a completely
independent test data set. Cross-validation on your training data could
then be used to select parameters of your classifier, if needed.

Hope this helps.
Regards,
Joern

> Thanks a lot!
>
> James
>
>        
> ---------------------------------
> Building a website is a piece of cake. 
>   

well, classification sometimes isn't.


       
---------------------------------

	[[alternative HTML version deleted]]

