[R] Leave one out Cross validation (LOO)

[Frank E Harrell Jr]

Alex Roy wrote:
> Dear Frank,
>                    Thanks for your comments. But in my situation, I do 
> not have any future data and I want to calculate Mean Square Error for 
> prediction on future data. So, is it not it a good idea to go for LOO?
>  
> thanks
>  
> Alex

With resampling you should be able to estimate any parameter including 
sigma.  The Design package's validate.ols function can estimate sigma 
using the bootstrap or c-v, penalizing for backward stepdown variable 
selection, although I have found some counter-intuitive estimates of 
sigma using Efron's optimism bootstrap.

Frank

> 
> On Tue, Feb 24, 2009 at 7:15 PM, Frank E Harrell Jr 
> <f.harrell at vanderbilt.edu <mailto:f.harrell at vanderbilt.edu>> wrote:
> 
>     Alex Roy wrote:
> 
>         Dear R user,
>                               I am working with LOO. Can any one who is
>         working
>         with leave one out cross validation (LOO) could send me the code?
> 
>         Thanks in advance
> 
>         Alex
> 
> 
>     I don't think that LOO adequately penalizes for model uncertainty.
>      I recommend the bootstrap or 50 repeats of 10-fold
>     cross-validation.  See for example the validate and calibrate
>     functions in the R Design package.
> 
>     Frank
>     --