[R] cross validation and parameter determination

[Liaw, Andy]

In all likelihood, you'll get an overly optimistic estimate of performance
that way.

Andy

> From: array chip
> 
> Hi all,
> 
> In Tibshirani's PNAS paper about nearest shrunken
> centroid analysis of microarrays (PNAS vol 99:6567),
> they used cross validation to choose the amount of
> shrinkage used in the model, and then test the
> performance of the model with the cross-validated
> shrinkage in separate independent testing set. If I
> don't have the luxury of having independent testing
> set, can I just use the cross validation performance
> as the performance estimate? In other words, can I use
> the same single cross-validation to both choose the
> value of the parameter (amount of shrinkage in this
> case) and estimate the performance which was based on
> the value of the parameter chosen by the same
> cross-validation? I kind of feel awkward by getting
> both on a single cross validation, because it seems
> like I used the dataset in training set manner. Am I
> wrong/right?
> 
> Thanks!
> 
> ______________________________________________
> R-help at stat.math.ethz.ch mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide! 
> http://www.R-project.org/posting-guide.html
> 
> 
>