When should train and test data be merged?

Question

I am very new to data science and machine learning. I am following courses on datacamp, and then trying to solve problems on kaggle/drivendata.

Very often, I try to use the sklearn.model_selection train_test_split()-method, but because my training (X) and test (y) data is not same shape, I get the error:

ValueError: Found input variables with inconsistent numbers of samples: [913000, 45000]

When I look at other people's solution, it looks like they very often combine training and test data (datasets in this case: train & test), like this:

all_data = train.append(test, sort = False)

Then later, they split this up again into variables X and y, which they can use on the train_test_split method. (You can probably hear I am very new and don't quite understand what's happening yet).

Can anyone explain me, why I need to combine my training and test data, when I already have them in separate .csv files? Maybe share some resources that explain it? Thanks a lot

Answer 1

The answer to this is simple.

When we perform the cleaning of the dataset we'll need to do the whole cleaning process for training data first then we'll do the same data cleaning process for the test dataset too. So to avoid doing the same data cleaning process twice, we merge the training and testing data then we perform the data cleaning process and after that we separate both dataset.