I was wondering if this is the correct way to get adaptive sampling in caret working with a pre-specified validation set using index. I can get this to work using the 'cv' method in caret like so

index_train = sample(nrow(iris), floor(nrow(iris) * .8))
index_list <- rep(list(index_train), 10)

iris <- select(iris, -Species)

ctrl <- trainControl(method = 'cv',
                    number = 1,
                    index = list(index_train)

gbm_fit <- caret::train(Sepal.Length ~ ., data = iris,
                        method = 'gbm',
                        verbose = TRUE, distribution = 'gaussian',
                        trControl = ctrl)

However, when I switch to adaptive_cv, this throws an error

> ctrl <- trainControl(method = 'adaptive_cv',
+                      number = 1,
+                      index = list(index_train)
+ )
Error in trainControl(method = "adaptive_cv", number = 1, index = index_list) : 
adaptive$min should be less than 1

Playing around with adaptive_min and number, it seems like adaptive_min has to be less than number but greater than 1. This means that I have to create artificial cross-folds even though I'm using the same train and validation holdout set. I do this by just repping my index vector 10x into a list (R will throw a subscript out of bounds error if you increase number and keep your index list the same).

index_list <- rep(list(index_train), 10)

ctrl <- trainControl(method = 'adaptive_cv',
                    number = 10,
                    adaptive = list(min = 5, alpha = 0.05, method = "gls", complete = TRUE),
                    index = index_list

This works but feels super forced, like I'm forcing caret to run 10 cv folds even though the folds are all the same train and validate set. Would this create 10x the amount of work for the computer? I really just want to run adaptive sampling on the hyperparameters of the model using my own train and validation set.

And just as a matter of curiosity, are there any other packages that would do bayesian optimization or adaptive resampling of hyperparameters better? Thanks!


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.