I want to determine if XGBoost is better than random forest or logistic regression for building a binary classification model. The model will be a composite model, with a feature selection model to select the relevant features, and a classification model which classified data based on the selected features.

I have 74 samples in my experiment, from which I have extracted about 450 features. I decided to use nested cross validation as I'm not confident that a single split of the samples into a training/ testing (ratio 80:20) set would yield a robust result. My attempts at fine tuning the XGBoost and random forest using LOOCV to split the training set into a training/validation subset yielded models that overfit on the test set.

SO as I understand, a nested cross validation method would go as follows:

  1. Outer loop CV to split the data into a training/testing set (normalize data after split to prevent data leakage)
  2. Inner loop CV to tune a feature selection model and select features (for XGBoost and random forest, I decided to tune the hyperparameters in two steps because of the large number of hyperparameters to explore)
  3. After the features are selected in the inner loop, a classification model is trained on all the samples in the training set, and its performance is validated in the testing set. The hyperparameters of the classification model are the same as those found during the tuning in step 2.
  4. Re-split the data with a different fold used for testing.

From my understanding, the nested CV approach should tell me which classification model performs the best by looking at their average performance across all folds in the outer loop. Looking at the features selected by a particular feature selection model across all folds will also give me an idea of which features are considered robust by a given feature selection model (eg. if all folds made by feature selection model A contain feature a, but only some contain feature b, feature a is considered 'selected' and not feature b). So using nested CV, I can get both the robust features, and a robust measure of performance of the classification model. However, I presume the hyperparameters chosen during tuning across the outer loop folds could be slightly different. So how can I determine the optimal hyperparameter configuration of a particular model that generalizes across all the folds?

I wonder if getting the optimal hyperparameter configuration that works for all folds could be obtained from just using LOOCV on the entire dataset (74 folds), but I'm not sure.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.