I am very new to data science and machine learning. I am following courses on datacamp, and then trying to solve problems on kaggle/drivendata.

Very often, I try to use the sklearn.model_selection train_test_split()-method, but because my training (X) and test (y) data is not same shape, I get the error:

ValueError: Found input variables with inconsistent numbers of samples: [913000, 45000]

When I look at other people's solution, it looks like they very often combine training and test data (datasets in this case: train & test), like this:

all_data = train.append(test, sort = False)

Then later, they split this up again into variables X and y, which they can use on the train_test_split method. (You can probably hear I am very new and don't quite understand what's happening yet).

Can anyone explain me, why I need to combine my training and test data, when I already have them in separate .csv files? Maybe share some resources that explain it? Thanks a lot


1 Answer 1


The answer to this is simple.

When we perform the cleaning of the dataset we'll need to do the whole cleaning process for training data first then we'll do the same data cleaning process for the test dataset too. So to avoid doing the same data cleaning process twice, we merge the training and testing data then we perform the data cleaning process and after that we separate both dataset.

  • $\begingroup$ Thank you. Do you happen to know any resources on this, that you could refer me to? $\endgroup$
    – HTTP 418
    Mar 17, 2020 at 9:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.