1
$\begingroup$

I have a small dataset of 150 records with 25 features (too small to do train/test). I'm using nested cv for both hyperparameter tuning and feature selection. 10cv in the outer loop, 5 cv in the inner loop. Eventually i'm getting 10 sets of hyperparameters and 10 sets of selected features. If i'm publishing my results, how eventually would i know which features should be selected for a model to be tried on an external data. (Currently i don't have another external data to test the model)

thank you

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

Assuming the feature selection method is always the same, on an external data (or final training set), you would simply apply the exact same method. The actual set of selected features does not matter.

If there was any difference in the selection method, for instance if you are selecting different number of features, you would do like with any other hyper-parameter: select the best model according to the nested CV process, then apply the same hyper-parameters (including for instance number of features) when training the final model.

Improve this answer
$\endgroup$
2
  • $\begingroup$ thank you, using the best hyperparamters and best features - should i train it on the same entire training data (used for nested CV) without CV prior to applying on external data? $\endgroup$
    – XPeriment
    Commented Dec 1, 2020 at 0:56
  • $\begingroup$ @XPeriment yes, that's what I would do. $\endgroup$
    – Erwan
    Commented Dec 1, 2020 at 1:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.