I am trying to solve binary classification problem using deep neural networks. I want to compare different approaches (model architectures) and I have no hyperparameters which I want to tune. So my question is can I simply use K-fold cross validation here without splitting data to train and test in advance? I mean, I have a dataset and I don't split it to train and test, just take it as it is, do 10-fold splits, for each validation split I compute metrics (let's say accuracy). Then after models have been trained, I aggregate metrics over all splits and compare them. Is this approach valid?
3 Answers
$\begingroup$
$\endgroup$
This is a reasonable approach, it's basically the traditional use of cross-validation in order to better leverage the entire dataset for both training and testing rather than relying on a single train-test split. The distribution of the performance metrics across the test folds is useful itself, but is often summarized as the mean value. You may be best off explicitly stratifying the folds in order to make sure the models are comparable and learning from similar populations.
$\begingroup$
$\endgroup$
I would say this depends on the problem you are solving and the objective. If your intent is prediction, I would recommend a train test split to prevent any sort of information leakage and concomitant bias. However, if your intent is to ID important features or any other auxiliary requirements, you may proceed without a train test split.
An additional aspect to consider in certain cases is the total number of observations. In cases where the number of observations is low and your intent is not to predict, it is better to not create a train-test split to provide as much data as possible for the model to learn from.
Hope this helps.
$\begingroup$
$\endgroup$
1
Hyperparameters are not a requirement for using cross validation. In fact for Keras you can get a validation error during training that you can use to correctly compare architectures.
You can get a lot of information from the scrikit-learn doc for cross-validation: scikit-learn cross validation doc.
-
$\begingroup$ Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center. $\endgroup$– Community BotMay 4 at 4:48