[R] Getting error message, "LOOCV is not compatible with `resamples()` since only one resampling estimate is available. "

[Bert Gunter]

2 1/2 suggestions:

1. Provide a small reproducible example with **minimal code** . It can be
difficult to sort through dozens of lines of code, and I, anyway, would be
unwilling to spend time trying to debug/isolate the problem when you have
apparently not made much of an effort to do so yourself. Others may well be
both more knowledgeable and more tolerant, of course.

2. If, **after a suitable wait ** you have not received useful answers,
contact the package maintainer of the package you used **which you have
again failed to identify** (the caret package?) . Also check to see whether
the package has its own user support structure. Some do, and this should be
the first point of contact anyway if so.

2 1/2 . Post in **plain text** not html, though I don't think it mattered
here.


Bert Gunter

"The trouble with having an open mind is that people keep coming along and
sticking things into it."
-- Opus (aka Berkeley Breathed in his "Bloom County" comic strip )


On Tue, Mar 3, 2020 at 3:30 AM javed khan <javedbtk111 using gmail.com> wrote:

> Hi, I am using different validation methods for random search and grid
> search. The validation methods are 10 fold CV, bootstrap and LOOCV but for
> LOOCV, I get the error message when I draw boxplots for all the results.
>
> Error is , LOOCV is not compatible with `resamples()` since only one
> resampling estimate is available.
>
> The code is below.
>
> d=readARFF("china.arff")
> index <- createDataPartition(d$Effort, p = .70,list = FALSE)
> tr <- d[index, ]
> ts <- d[-index, ]
> index_2 <- createFolds(tr$Effort, returnTrain = TRUE, list = TRUE)
>
>
>
>
> ct_rand <- trainControl(method = "repeatedcv", number=10, repeats=10,index
> = index_2, search="random")
> ct_grid <- trainControl(method = "repeatedcv", number=10, repeats=10,index
> = index_2, search="grid")
>
>
> ct_boot1 <- trainControl(method = "boot", number=100,  index = index_2,
> search="random")
> ct_boot2 <- trainControl(method = "boot", number=100,  index = index_2,
> search="grid")
>
> ct_locv <- trainControl(method = "LOOCV",  search="random")
> ct_locv2 <- trainControl(method = "LOOCV",   search="grid")
>
> set.seed(30218)
> ran_CV <- train(Effort ~ ., data = tr,
>                     method = "pls",
>                     tuneLength = 15,
>                     metric = "MAE",
>                     preProc = c("center", "scale", "zv"),
>                     trControl = ct_rand)
> getTrainPerf(ran_CV)
> rn <- predict(ran_CV, newdata = ts)
>
> ## ## ## ## ##grid search CV
>
> set.seed(30218)
> grid_CV <- train(Effort ~ ., data = tr,
>                      method = "pls",
>                      metric = "MAE",
>                      preProc = c("center", "scale", "zv"),
>                      trControl = ct_grid)
>
> getTrainPerf(grid_CV)
>
> set.seed(30218)
> ran_boot <- train(Effort ~ ., data = tr,
>                 method = "pls",
>                 tuneLength = 15,
>                 metric = "MAE",
>                 preProc = c("center", "scale", "zv"),
>                 trControl = ct_boot1)
> getTrainPerf(ran_boot)
> rn <- predict(ran_search, newdata = ts)
> ##MAE(rn, ts$Effort)
>
>
> ## ## ## ## ##grid search boot
>
> set.seed(30218)
> grid_boot <- train(Effort ~ ., data = tr,
>                  method = "pls",
>                  metric = "MAE",
>                  preProc = c("center", "scale", "zv"),
>                  trControl = ct_boot2)
>
> getTrainPerf(grid_boot)
>
>
> set.seed(30218)
> ran_locv <- train(Effort ~ ., data = tr,
>                 method = "pls",
>                 tuneLength = 15,
>                 metric = "MAE",
>                 preProc = c("center", "scale", "zv"),
>                 trControl = ct_locv)
> getTrainPerf(ran_locv)
> rn <- predict(ran_search, newdata = ts)
> ##MAE(rn, ts$Effort)
>
>
> ## ## ## ## ##grid search CV
>
> set.seed(30218)
> grid_locv <- train(Effort ~ ., data = tr,
>                  method = "pls",
>                  metric = "MAE",
>                  preProc = c("center", "scale", "zv"),
>                  trControl = ct_locv2)
>
> getTrainPerf(grid_locv)
>
>
> rValues <- resamples(list(Random_Search_CV=ran_CV, Grid_Search_CV=grid_CV,
> Random_Search_Boot=ran_boot, Grid_Search_Boot=grid_boot ,
>                           Random_Search_LOOCV=ran_locv,
> Grid_Search_LOOCV=grid_locv))
>
> bwplot(rValues,metric="MAE", scales=list(cex=1), col="Green")
>
>         [[alternative HTML version deleted]]
>
> ______________________________________________
> R-help using r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>

	[[alternative HTML version deleted]]