Why does downsampling leads classification to only predict one class?

Question

I have a multi-class classification problem. It performs quite well but on the least represented classes it doesn't. Indeed, here is the distribution :

And here are the classification results of my former classification (I took the numbers off the labels):

.

Therefore I tried to improve classificationby downsampling the marjority classes

import os
from sklearn.utils import resample

# rebalance data
#df = resample_data(df)

if True:

    count_class_A, count_class_B,count_class_C, count_class_D,count_class_E, count_class_F, count_class_G = df.grade.value_counts()
    count_df = df.shape[0] 
    class_dict = {"A": count_class_A,"B" :count_class_B,"C": count_class_C,"D": count_class_D,"E": count_class_E, "F": count_class_F, "G": count_class_G}
    counts = [count_class_A, count_class_B,count_class_C, count_class_D,count_class_E, count_class_F, count_class_G]
    median = statistics.median(counts)


    for key in class_dict:
        if class_dict[key]>median:
            print(key)
            df[df.grade == key] = df[df.grade == key].sample(int(count_df/7), replace = False) 
                                             #replace=False,    # sample without replacement
                                             #n_samples=int(count_df/7),     # to match minority class
                                             #random_state=123)
# Divide the data set into training and test sets
x_train, x_test, y_train, y_test = split_data(df, APPLICANT_NUMERIC + CREDIT_NUMERIC,
                  APPLICANT_CATEGORICAL,
                  TARGET,
                  test_size = 0.2,
                  #row_limit = os.environ.get("sample"))
                  row_limit = 552160)

However it the results where catastrophic. The model accuracy and the model loss looked to have some issues :

And everything was classified in "A" on the test set.

Answer 1

For the looks of it the sampling method you used isnt doing the trick. The way you use pandas slicing and assignment is a bit off.

Something like this might work:

dfs = []
for key in class_dict:

    if class_dict[key] > median:
        factor = int(count_df/7)
        key_df = df[df.grade == key].sample(factor, replace=False)
    else:
        key_df = df[df.grade == key]

    dfs.append(key_df)
new_train_df = pd.concat(dfs, sort=False)

While this might work, there are more straight forward ways down sampling the data(like setting the exact number of samples expected from each class).
This question has nice answers too.

Answer 2

You are implementing downsampling from scratch. There may be bugs in your implementation.

Another option is using an established package for down sampling, such as imblearn's under_sampling.