[BioC] Best options for cross validation machine learning

[Kasper Daniel Hansen]

On Jan 19, 2010, at 11:11 AM, Daniel Brewer wrote:

> 1) Pick a group of genes that best predict which group a sample belongs to.
> 2) Determine how stable these prediction sets are through some sort of
> cross-validation (I would prefer not to divide my set into a training
> and test set for stage one)

If you don't do this (the statement in ()) you will most likely get crap.  Note the astounding amount of papers in the literature that have attempted to do this.  And note that these papers never gets replicated, most likely because the statistical analysis is overly optimistic.

The track record for being able to do this is extremely bad, despite the number of papers claiming that their signature method is like 99% accurate.

Kasper