I have predefined train and test sets. On generating some statistics like value_counts and checking the unique values, I feel that there is a 'lot' of difference between the distributions of the variables. What should be done with this?
Suppose if I want to delete a column from the train_set for any reason like near-zero variance, should I repeat the same for the test_set (even if there is no such problem in the test_set's frequency tables?
I ran the following code
for dataset in both_datasets: # both_datasets contains train_set and test_set print(dataset.nunique()) print('\n')
Then in order to see how many unique values my columns contain, I ran the following code
for dataset in both_datasets: print('-'*120) for col in dataset.columns: unique = len(dataset[col].unique()) percentage = float(unique)/ dataset.shape*100 print(percentage, col)
So there are clear differences between the ratios of the percentage presence of unique values.
The question is should I
Avoid taking any insights from test_data. Do changes WRT the insights taken from the train_set only. However, every change I make should be replicated in test_set as well
Use test_data too for insights and do preprocess accordingly.
Do something to change the test_data to make it more balanced and representative.
Somethings else :S