test data is not a good representation of train data

Question

I have predefined train and test sets. On generating some statistics like value_counts and checking the unique values, I feel that there is a 'lot' of difference between the distributions of the variables. What should be done with this?

Suppose if I want to delete a column from the train_set for any reason like near-zero variance, should I repeat the same for the test_set (even if there is no such problem in the test_set's frequency tables?

I ran the following code

for dataset in both_datasets:       # both_datasets contains train_set and test_set 
print(dataset.nunique())
print('\n')

And this is the output (I compiled it for a better view and highlighted some extreme cases) You might observe that for the column specific_code_lesion, the test_set misses an entire category!

Then in order to see how many unique values my columns contain, I ran the following code

for dataset in both_datasets:
print('-'*120)
for col in dataset.columns:
    unique = len(dataset[col].unique())
    percentage = float(unique)/ dataset.shape[0]*100
    print(percentage, col)
    

And here is the output:

So there are clear differences between the ratios of the percentage presence of unique values.

The question is should I

1. Avoid taking any insights from test_data. Do changes WRT the insights taken from the train_set only. However, every change I make should be replicated in test_set as well

2. Use test_data too for insights and do preprocess accordingly.

3. Do something to change the test_data to make it more balanced and representative.

4. Somethings else :S

Answer 1

Here's is my attempt to answer these questions:

1. Avoid taking any insights from test_data. Do changes WRT the insights taken from the train_set only. However, every change I make should be replicated in test_set as well:
   The second part is mostly true. Only in some rare cases where you may be applying class balancing techniques to the train set which should not be applied to the test/evaluation set. Test set should represent the real world data as much possible, while, train set should do the same, some tweaks to this data set may be applied independently to make it easy/possible to train with the chosen ML algorithm. However, any scaling/normalization/transformation parameters must be replicated exactly to the test set as well.
   

2. Use test_data too for insights and do preprocess accordingly.
   There should be no harm in conducting an EDA on both train and test datasets and using the combined knowledge for taking feature engineering decisions.

3. Do something to change the test_data to make it more balanced and representative.
   NO. As noted in the #1 above, the test data should represent the real world data as much as possible. How else would you rely on the test set's evaluation metrics for the unseen real world data? You can read some more about it here Should I balance the test set when i have highly unbalanced data?

4. Somethings else :S
   You should not be worried if a category/class of a certain feature is missing from the test set. However, the reverse is not true: there must not be any new category/class in the test or real world data set which was absent from the train set - the preprocessing routine and model will not know how to handle it.
   Also, if there are a very low number of unique values or near zero variance in a feature, you should consider dropping it (from both train and test sets). In the end, this should reflect in low feature importance and highlighted by any of the feature selection procedures like this Feature selection