Tensorflow Adjusting Cost Function for Imbalanced Data

Question

I have a classification problem with highly imbalanced data. I have read that over and undersampling as well as changing the cost for underrepresented categorical outputs will lead to better fitting. Before this was done tensorflow would categorize each input as the majority group (and gain over 90% accuracy, as meaningless as that is).

I have noticed that the log of the inverse percentage of each group has made the best multiplier that I have tried. Is there a more standard manipulation for the cost function? Is this implemented correctly?

from collections import Counter
counts = Counter(category_train)
weightsArray =[]
for i in range(n_classes):
    weightsArray.append(math.log(category_train.shape[0]/max(counts[i],1))+1)

class_weight = tf.constant(weightsArray)
weighted_logits = tf.mul(pred, class_weight)
cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(weighted_logits, y))
optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)

Answer 1

This seems like a good solution for the loss function. I've had success with a similar approach recently, but I think you'd want to reorder where you multiply in the class_weight.

Thinking about it logically, the class_weight will be a constant w.r.t. the output, so it will be carried along and applied to the gradient in the same way it's being applied to the cost function. There is one problem though.

The way you have it, the class_weight would affect the prediction value. But you want it to affect the scale of the gradient. If I'm not wrong I think you'd want to reverse the order of operations:

# Take the cost like normal
error = tf.nn.softmax_cross_entropy_with_logits(pred, y)

# Scale the cost by the class weights
scaled_error = tf.mul(error, class_weight)

# Reduce
cost = tf.reduce_mean(scaled_error)

I'd be very interested to know how this performs in comparison to simply oversampling the underrepresented class, which is more typical. So if you gain some insight there post about it! :)

Interestingly I successfully used a very similar technique in a different problem domain just recently (which brought me to this post):

Multi-task learning, finding a loss function that "ignores" certain samples

Answer 2

Checkout tf.nn.weighted_cross_entropy_with_logits():

Computes a weighted cross entropy.

This is like sigmoid_cross_entropy_with_logits() except that pos_weight, allows one to trade off recall and precision by up- or down-weighting the cost of a positive error relative to a negative error.

This should let you do what you want.

Answer 3

I have 2 different implementations:

1. with 'regular' softmax with logits : tf.nn.softmax_cross_entropy_with_logits

Where the class_weight is a placeholder I fill in on everey batch iteration.

self.class_weight  = tf.placeholder(tf.float32, shape=self.batch_size,self._num_classes], name='class_weight')    
self._final_output = tf.matmul(self._states,self._weights["linear_layer"]) + self._biases["linear_layer"] 
self.scaled_logits = tf.multiply(self._final_output, self.class_weight)
self.softmax = tf.nn.softmax_cross_entropy_with_logits(logits=self.scaled_logits,labels= self._labels)

2. with tf.nn.softmax_cross_entropy_with_logits

Where I use the implemented tensorflow function but I need to calculate the weights for the batch. The docs are a bit confusing about it. There are 2 ways to do it with tf.gather or like this:

self.scaled_class_weights=tf.reduce_sum(tf.multiply(self._labels,self.class_weight),1)
self.softmax = tf.losses.softmax_cross_entropy(logits=self._final_output,
                                                   onehot_labels=self._labels,weights=self.scaled_class_weights)

here there is a nice discussion about it

And finally as I did not want to marry to any of the implemetnations permenantly I added a little tf.case and I pass in on training time the strategy I want to use.

self.sensitive_learning_strategy = tf.placeholder(tf.int32 , name='sensitive_learning_strategy')
self.softmax =tf.case([
            (tf.equal(self.sensitive_learning_strategy, 0), lambda: self.softmax_0),
            (tf.equal(self.sensitive_learning_strategy, 1), lambda: self.softmax_1),
            (tf.equal(self.sensitive_learning_strategy, 2), lambda: self.softmax_2)