1
$\begingroup$

I have a multi-class classification problem. It performs quite well but on the least represented classes it doesn't. Indeed, here is the distribution :

enter image description here

And here are the classification results (I took the numbers off the labels):

enter image description here.

Therefore how to improve classifcation when some are less represented ?

I thought of duplicating a few rows of the classes it doesn't predict well in the train sample. But maybe this assumption is entirely false, maybe it is not because they are less represented that are badly classified. Maybe I should have a look on the feature selection I did by hand and rather do a PCA ?

Update

class weight with inverted frequency

I passed the class_weight parameter in model.fit() which is a list of the inverted frequency of the classes on the dataset:

>>> lossWeights = df['grade'].value_counts(normalize=True)
>>> lossWeights = lossWeights.sort_index().tolist()
>>> print(lossWeights)
[0.204064039408867, 0.2954361054766734, 0.29536185163720663, 0.13638619240799768, 0.04878839466821211, 0.014684149521877717, 0.0052792668791654595]
weights = {0: 1 / 0.204064,
       1: 1 / 0.295436, 
       2: 1 / 0.295362,
       3: 1 / 0.136386, 
       4: 1 / 0.048788,
       5: 1 / 0.014684,
       6: 1 / 0.005279}

history = model.fit(x_train.as_matrix(),
                y_train.as_matrix(),
                validation_split=0.2,
                epochs=epochs,  
                batch_size=batch_sz, # Can I tweak the batch here to get evenly distributed data ?
                verbose=2,
                class_weight = weights,
                callbacks=[checkpoint])

It diminished on the test Set Accuracy: 86.57% (it was 88.54% before) but better balanced the results on the confusion matrix :

enter image description here

class weight with inverted frequency + focal loss

Focal loss is designed to address class imbalance by down-weighting inliers (easy examples) such that their contribution to the total loss is small even if their number is large. It focuses on training a sparse set of hard examples.

def focal_loss(gamma=2., alpha=4.):

    gamma = float(gamma)
    alpha = float(alpha)

    def focal_loss_fixed(y_true, y_pred):
        """Focal loss for multi-classification
        FL(p_t)=-alpha(1-p_t)^{gamma}ln(p_t)
        Notice: y_pred is probability after softmax
        gradient is d(Fl)/d(p_t) not d(Fl)/d(x) as described in paper
        d(Fl)/d(p_t) * [p_t(1-p_t)] = d(Fl)/d(x)
        Focal Loss for Dense Object Detection
        https://arxiv.org/abs/1708.02002

        Arguments:
            y_true {tensor} -- ground truth labels, shape of [batch_size, num_cls]
            y_pred {tensor} -- model's output, shape of [batch_size, num_cls]

        Keyword Arguments:
            gamma {float} -- (default: {2.0})
            alpha {float} -- (default: {4.0})

        Returns:
            [tensor] -- loss.
        """
        epsilon = 1.e-9
        y_true = tf.convert_to_tensor(y_true, tf.float32)
        y_pred = tf.convert_to_tensor(y_pred, tf.float32)

        model_out = tf.add(y_pred, epsilon)
        ce = tf.multiply(y_true, -tf.log(model_out))
        weight = tf.multiply(y_true, tf.pow(tf.subtract(1., model_out), gamma))
        fl = tf.multiply(alpha, tf.multiply(weight, ce))
        reduced_fl = tf.reduce_max(fl, axis=1)
        return tf.reduce_mean(reduced_fl)
    return focal_loss_fixed

model.compile(loss=focal_loss(alpha=1),
              optimizer='nadam',
              metrics=['accuracy'])
model.fit(X_train, y_train, epochs=3, batch_size=1000)

I just had to add it to my model:

def create_model(input_dim, output_dim):
    print(output_dim)
    # create model
    model = Sequential()
    # input layer
    model.add(Dense(100, input_dim=input_dim, activation='relu', kernel_constraint=maxnorm(3)))
    model.add(Dropout(0.2))

    # hidden layer
    model.add(Dense(60, activation='relu', kernel_constraint=maxnorm(3)))
    model.add(Dropout(0.2))

    # output layer
    model.add(Dense(output_dim, activation='softmax'))

    # Compile model
    # model.compile(loss='categorical_crossentropy', loss_weights=None, optimizer='adam', metrics=['accuracy'])
    model.compile(loss=focal_loss(alpha=1), loss_weights=None, optimizer='adam', metrics=['accuracy'])

    return model

It gave me an overall accuracy of 88%. However it gave me back a very bad classification on the least represented class:

enter image description here

focal loss

It has a decent test Set Accuracy: 88.27% and the classification is better balanced :

enter image description here

Now I have to questions. I'm still not satisfied. How to improve this classification ? Which model should I use between the first and the last updates ?

Resample

I tried to down sample the majority classes

import os
from sklearn.utils import resample

# rebalance data
#df = resample_data(df)

if True:

    count_class_A, count_class_B,count_class_C, count_class_D,count_class_E, count_class_F, count_class_G = df.grade.value_counts()
    count_df = df.shape[0] 
    class_dict = {"A": count_class_A,"B" :count_class_B,"C": count_class_C,"D": count_class_D,"E": count_class_E, "F": count_class_F, "G": count_class_G}
    counts = [count_class_A, count_class_B,count_class_C, count_class_D,count_class_E, count_class_F, count_class_G]
    median = statistics.median(counts)


    for key in class_dict:
        if class_dict[key]>median:
            print(key)
            df[df.grade == key] = df[df.grade == key].sample(int(count_df/7), replace = False) 
                                             #replace=False,    # sample without replacement
                                             #n_samples=int(count_df/7),     # to match minority class
                                             #random_state=123)
# Divide the data set into training and test sets
x_train, x_test, y_train, y_test = split_data(df, APPLICANT_NUMERIC + CREDIT_NUMERIC,
                  APPLICANT_CATEGORICAL,
                  TARGET,
                  test_size = 0.2,
                  #row_limit = os.environ.get("sample"))
                  row_limit = 552160)

# Inspect our training data
print("x_train contains {} rows and {} features".format(x_train.shape[0], x_train.shape[1]))
print("y_train contains {} rows and {} features".format(y_train.shape[0], y_train.shape[1]))

print("x_test contains {} rows and {} features".format(x_test.shape[0], x_test.shape[1]))
print("y_test contains {} rows and {} features".format(y_test.shape[0], y_test.shape[1]))

# Loan grade has been one-hot encoded
print("Sample one-hot encoded 'y' value: \n{}".format(y_train.sample()))

However it the results where catastrophic. The model accuracy and the model loss looked to have some issues :

enter image description here

And everything was classified in "A" on the test set.

$\endgroup$
  • 2
    $\begingroup$ This is called class imbalance, and it is a huge subtopic in itself; I suggest you start googling ruthlessly. $\endgroup$ – desertnaut Sep 14 at 15:15
  • $\begingroup$ What models are you employing? $\endgroup$ – Leevo Sep 14 at 15:53
3
$\begingroup$

Class imbalance is a common problem, there are several ways in which people tackle this problem, below are few.

  1. If possible try augmenting the class for which the data is less. (Some might call it oversampling).
  2. You can use class weighting which penalizes less (during training) the class whose data is not sufficient.
  3. Using Focal Loss, an example of it can be found at - Using focal loss for Fraud detection

There are other ways as well possible, try the one which is best suitable and convenient in your scenario.

$\endgroup$
  • $\begingroup$ Thanks for your help ! Unfortunately I can only artificially augment the class for which the data is less by "cloning it" or reducing the other data. What do you think about this ? I used class weighting and it better balanced the classification but reduced my overall accuracy. I'm going to look after Focal Loss. What do you think about PCA ? Unfortunately I'm super bad at doing it in Python :((((( $\endgroup$ – IggyPass Sep 17 at 20:07
  • $\begingroup$ I tried using focal loss for Fraud detection with weighting and it gave back almost the same overall accuracy but a super worst one on the last class. I'm going to add teh code and the screenshot. $\endgroup$ – IggyPass Sep 17 at 22:14
1
$\begingroup$

Maybe I should have a look on the feature selection I did by hand and rather do a PCA ?

That's an option, and it might work.

Alternatively, you can try:

  1. Train a Neural Network with mini-batch gradient descent, and artificially build each batch of data in a way to make all the classes evenly distributed.

  2. If you can, break the problem into subproblems. Classifiers' performance goes down as the number of classes increases. If some classes represent a separate problem, you can train models on subsets of your data, so that each model can learn to classify on fewer classes. This very often leads to better predictions.

$\endgroup$
  • $\begingroup$ Thanks for uour insight on PCA. However 1. how can I know how many components do I need ? I ask this question because I found on this webpage that the author provided the number of features pca = PCA(n_components=3). 2. I have understood how to do a mini-batch gradient descent with this article, but I don't know how to artificially build each batch of data in a way to make all the classes evenly distributed $\endgroup$ – IggyPass Sep 17 at 14:44
  • $\begingroup$ The number of PCA components is an hyperparameter you can tweak, there is no right/wrong solution. To build batches: 1. You can use eager execution in TF 2.0 and hard code some fetch_batch() function (a difficult implementation but gives you more freedom). 2. In Keras, into model.compile() there are arguments you can use to change observation weights. Take a look at the official doc page, especially loss_weights and sample_weight_mode args. I never used them, but I think you can feed a vector of weights inversely proportional to class frequency $\endgroup$ – Leevo Sep 17 at 14:57
  • $\begingroup$ Okay, I have 105 features, I'm going to go 10 to 10 and plot the accuracy. I have understood 2: loss_weights. It weights the loss contributions of different model outputs. I only have to provide a list, which is expected to have a 1:1 mapping to the model's outputs. However how do I pick these values ? Do you mind if I share with you my repository or jupyter notebook if you have any more insights ? $\endgroup$ – IggyPass Sep 17 at 15:43
  • $\begingroup$ You probably want to choose weights that are inverse of their class frequency, so that classes that are less frequent will have comparatively greater weight. $\endgroup$ – Leevo Sep 17 at 15:53
  • $\begingroup$ Ok, share your Notebooks $\endgroup$ – Leevo Sep 17 at 15:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.