Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I am working with a dataset that I downloaded from Kaggle. The data set is already divided into two CSVs for Train and Test.
I built a model using the training set because I imported the train CSV into a Jupyter Notebook. I predicted using the Train CSV itself. I would like to perform cross validation. Should I perform cross validation on the train CSV and split it into two parts again Train and Test? Or, should I import a new CSV file Test and combine both CSVs into ONE?
You shouldn't touch test data until you finish. To do cross validation you have to split the training dataset into train and validation sets or you can do k-fold cross validation (or any other method).
Here is some information: https://en.wikipedia.org/wiki/Cross-validation_(statistics)
But, never use test data to train or adjust your model, because if you do, your model will be trained for your test data and then your results won't be valid.
Chapter 5 of "Introduction to Statistical Learning" covers CV and bootstrap. I strongly recommend to read this chapter, since sampling methods are extremely relevant in practice.
Cross validation (CV) usually means that you split some training dataset in k pieces in order to generate different train/validation sets. By doing so you can see how well a model learns (and is able to make predictions) on different samples of a training dataset.
During training and model tuning, your model should not see the test data! The idea is that you reserve a truly "exogenous" dataset (never used during training) to test how well your model does in the end. If you use your test set during training, information may leak from your test set to the model, and you cannot demonstrate exogenous validity of your model anymore (becasue information from the test set leaked to your model).
In the image below, you would make four different splits on your training set, where blue would represent training cases and orange the validation cases. You run the same regression/classification on each of the four blue parts, and obtain predictions on each of the four orange parts.
There are different strategies to deal with data characteristics. In order to balance groups etc, you can use a stratified sampling strategy. See an sklearn implementation here.
Since sampling strategies can be relevant, it is advisable to let some tool like sklearn do the CV splits. Most tools/methods come with some CV options if relevant.
There is a great book written by Andrew Ng, called "Machine Learning Yearning". You can download it for free here https://www.deeplearning.ai/machine-learning-yearning/
In the Kaggle's dataset now. The datasets you have are the training and the test. Even if you wanted to use the test dataset, you cannot. If you open it, you will see that only the features are there. The target column is missing, because this is the file that you need to use the model and predict the values to upload it on Kaggle.
Reading the chapters 5, 6 and 7 of the linked book, you will get a lot of great advises on how to handle the training data. In short.
You should split the training dataset in 3 parts.
- Training set — Which you run your learning algorithm on.
- Dev (development) set — Which you use to tune parameters, select features, and make other decisions regarding the learning algorithm. Sometimes also called the hold-out cross validation set.
- Test set — which you use to evaluate the performance of the algorithm, but not to make any decisions regarding what learning algorithm or parameters to use.
Quoted from chapter 5, page 15.
Keep in mind that the above Test set has nothing to do with the test file that you got from Kaggle.
