0
$\begingroup$

I'm new to data science and I have a problem understanding what dataset to use when using cross validation for model evaluation.

Let's say I have two models: LogisticRegression and RandomForestClassifier. After I train them on training set I want to have an idea which one is better. If I want to do cross validation, do I do it on the training set, or the whole dataset (training and testing)? I have seen some contradicting methods.

$\endgroup$

3 Answers 3

5
$\begingroup$

It's generally recommended to use the training set for cross-validation to avoid data leakage. The primary role of the test set is to provide an unbiased evaluation of a model's performance. If you include the test set in your cross-validation, you risk leaking information from the test set into the training process. Data leakage can lead to overfitting, meaning that your model will make good predictions for the data you used, but will perform poorly on unseen data.

$\endgroup$
2
$\begingroup$

The test and train set should be different. The Cross Validate set comes in training set. The test data should be totally unseen by the model.

While there is no hard and fast rule for making test train split I generally use 20% of the total data as Test set and 80% as Train set. From this 80% data I use 80%(64% of total data) of train as actual training and 20%(16% of total data) as Cross Validate data.

$\endgroup$
-2
$\begingroup$

for comparison you should check the same metrics for both models (F-metric, accuracy, ROC and etc). Cross validation is a technique used in machine learning to evaluate the performance of a model on unseen data. It involves dividing the available data into multiple folds or subsets, using one of these folds as a validation set, and training the model on the remaining folds. This process is repeated multiple times, each time using a different fold as the validation set. When you train your machine using scikit-lear(example) you can tune cross-validation.

$\endgroup$
1
  • $\begingroup$ As it’s currently written, your answer is unclear. Please edit to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers in the help center. $\endgroup$
    – Community Bot
    Aug 15, 2023 at 10:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.