is there a way to customize my loss function to increase recall in one class only?

Question

First, thank you.

I'm using Tensorflow 1.13 with Python 3. I'm trying to create a CNN classifier on data with 4 classes, one-hot encoded. However, the insights I need it to produce are really around spotting either classes 0 or 3. If it tells me a data row is of class 1 or 2, that's of little use to my goals. Output from sklearn classification_report is below and I want to increase recall on classes 0 and 3 during model training. Precision would be nice, too.

               precision   recall   f1-score  support

          0       0.13      0.42      0.20      1484
          1       0.15      0.74      0.24       696
          2       0.81      0.34      0.48     12325
          3       0.13      0.23      0.17      1495

avg / total       0.65      0.35      0.41     16000  

I'm relatively new to both python and TF, having gone through several tutorials, and am just now starting to build my own models. I've not really been able to stretch myself out past capabilities discussed in any of the learning courses I've watched. I know that Keras, for example, has custom loss function capability, but I'm not sure I understand how to integrate it into what I've already done. Relevant bits from my existing project are below.

# my current loss function, what I want to 
# replace because just plain accuracy isn't 
# terribly helpful
cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels=y_true,logits=y_pred))

# setting up the trainer
optimizer = tf.train.AdamOptimizer(learning_rate=0.0001)
train = optimizer.minimize(cross_entropy)

# run it
init = tf.global_variables_initializer()

with tf.Session() as sess:
    sess.run(init)
    for i in range(steps):
        batch_x, batch_y = dataHelper.next_batch(100)

        sess.run(train,feed_dict={x:batch_x,y_true:batch_y,hold_prob:0.5})
        if (i + 1) % 100 == 0:
            print('on step {}'.format(i))
            preds_decoded = tf.argmax(y_pred,1)
            trues_decoded = tf.argmax(y_true,1)
            final_predictions, true_values = sess.run([preds_decoded, trues_decoded],
                                    feed_dict={x:price_data_only_test,y_true:test_labels,hold_prob:1.0})
            print(classification_report(y_true=true_values,y_pred=final_predictions))
```

Answer 1

What you want to do to boost the performance of one of your classes is to add class weights to your loss function. This will essentially tell your network to pay more attention to some classes (note that this might deteriorate performance on the rest of the classes).

How to compute the weights.

To see what weight you'll give to each class you can either put it yourself or you can let scikit-learn help you. The problem is that your data is imbalanced. This means that you have much more samples in class 2 than you have in classes 0, 1 and 3. Through the use of class weights you can make it so that your model pays more attention to the under-represented classes, so that they appear balanced. Scikit-learn has a function that calculates what weights you should give each sample.

from sklearn.utils.class_weight import compute_class_weight
weight_array = compute_class_weight('balanced', np.unique(y) y)  # y is the array containing your labels 

weight_dict = dict(zip(np.unique(y), weight_array))  # dictionary containing your weights

Keras

In keras you can also add class weights in the fit method (which trains the model).

This is easier as you just need to pass a dictionary with the weight of each class (i.e. what we calculated previously)

model.fit(X_train, y_train, class_weight=weight_dict, ...)

TensorFlow

In TensorFlow you can do this simply by using this softmax_cross_entropy loss instead of the one you are currently using. This supports class weights (weights attribute).

This requires you to pass a Tensor with a shape of (batch_size,) where its elements are the weight you want to pass to each sample. What you need to do is to see what samples batch_y has and give them the respective weight!

batch_weights = [weight_dict(x) for x in batch_y]  # assuming that your batch_y is a numpy array that is not one-hot encoded

# if it is one-hot encoded:
batch_weights = [weight_dict(x) for x in np.argmax(batch_y, axis=1)]  # assuming batch y is a numpy array

# if batch_y is not a numpy array you need to get it through
batch_y_arr = sess.run(batch_y)

Then pass the batch_weights to the weights attribute of the loss (through a placeholder using the feed_dict - like you do with batch_x and batch_y)