Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I'm new to data science and I have a problem understanding what dataset to use when using cross validation for model evaluation.
Let's say I have two models: LogisticRegression and RandomForestClassifier. After I train them on training set I want to have an idea which one is better. If I want to do cross validation, do I do it on the training set, or the whole dataset (training and testing)? I have seen some contradicting methods.
It's generally recommended to use the training set for cross-validation to avoid data leakage. The primary role of the test set is to provide an unbiased evaluation of a model's performance. If you include the test set in your cross-validation, you risk leaking information from the test set into the training process. Data leakage can lead to overfitting, meaning that your model will make good predictions for the data you used, but will perform poorly on unseen data.
The test and train set should be different. The Cross Validate set comes in training set. The test data should be totally unseen by the model.
While there is no hard and fast rule for making test train split I generally use 20% of the total data as Test set and 80% as Train set. From this 80% data I use 80%(64% of total data) of train as actual training and 20%(16% of total data) as Cross Validate data.
for comparison you should check the same metrics for both models (F-metric, accuracy, ROC and etc). Cross validation is a technique used in machine learning to evaluate the performance of a model on unseen data. It involves dividing the available data into multiple folds or subsets, using one of these folds as a validation set, and training the model on the remaining folds. This process is repeated multiple times, each time using a different fold as the validation set. When you train your machine using scikit-lear(example) you can tune cross-validation.
