[R] LDA with previous PCA for dimensionality reduction

Bjørn-Helge Mevik bhx2 at mevik.net
Thu Nov 25 15:05:08 CET 2004

Torsten Hothorn writes:

> as long as one does not use the information in the response (the class
> variable, in this case) I don't think that one ends up with an
> optimistically biased estimate of the error

I would be a little careful, though.  The left-out sample in the
LDA-cross-validation, will still have influenced the PCA used to build
the LDA on the rest of the samples.  The sample will have a tendency
to lie closer to the centre of the "complete" PCA than of a PCA on the
remaining samples.  Also, if the sample has a high leverage on the
PCA, the directions of the two PCAs can be quite different.  Thus, the
LDA is built on data that "fits" better to the left-out sample than if
the sample was a completely new sample.

I have no proofs or numerical studies showing that this gives
over-optimistic error rates, but I would not recommend placing the PCA
"outside" the cross-validation.  (The same for any resampling-based

Bjørn-Helge Mevik

More information about the R-help mailing list