234
$\begingroup$

I know that there is a possibility in Keras with the class_weights parameter dictionary at fitting, but I couldn't find any example. Would somebody so kind to provide one?

By the way, in this case the appropriate praxis is simply to weight up the minority class proportionally to its underrepresentation?

$\endgroup$
0

11 Answers 11

196
$\begingroup$

If you are talking about the regular case, where your network produces only one output, then your assumption is correct. In order to force your algorithm to treat every instance of class 1 as 50 instances of class 0 you have to:

  1. Define a dictionary with your labels and their associated weights

    class_weight = {0: 1.,
                    1: 50.,
                    2: 2.}
    
  2. Feed the dictionary as a parameter:

    model.fit(X_train, Y_train, nb_epoch=5, batch_size=32, class_weight=class_weight)
    

EDIT: "treat every instance of class 1 as 50 instances of class 0" means that in your loss function you assign higher value to these instances. Hence, the loss becomes a weighted average, where the weight of each sample is specified by class_weight and its corresponding class.

From Keras docs:

class_weight: Optional dictionary mapping class indices (integers) to a weight (float) value, used for weighting the loss function (during training only).

$\endgroup$
8
  • 3
    $\begingroup$ Also have a look at github.com/fchollet/keras/issues/3653 if you're working with 3D data. $\endgroup$
    – herve
    Apr 26 '17 at 9:12
  • $\begingroup$ For me it gives a error dic don't has shape attribute. $\endgroup$ May 23 '17 at 0:11
  • 2
    $\begingroup$ Does this work for one-hot-encoded labels? $\endgroup$ Jan 8 '18 at 19:49
  • 9
    $\begingroup$ @layser Does this work only for 'category_crossentropy' loss? How do you give class_weight to keras for 'sigmoid' and 'binary_crossentropy' loss? $\endgroup$
    – Naman
    Apr 15 '18 at 19:26
  • 2
    $\begingroup$ @layser Can you explain ` to treat every instance of class 1 as 50 instances of class 0 ` ? Is it that the in training set, row corresponding to class 1 is duplicated 50 times in order to make it balanced or some other process follows? $\endgroup$ Jun 12 '18 at 5:12
211
$\begingroup$

You could simply implement the class_weight from sklearn:

  1. Let's import the module first

    from sklearn.utils import class_weight
    
  2. In order to calculate the class weight do the following

    class_weights = class_weight.compute_class_weight('balanced',
                                                     np.unique(y_train),
                                                     y_train)
    
  3. Thirdly and lastly add it to the model fitting

    model.fit(X_train, y_train, class_weight=class_weights)
    

Attention: I edited this post and changed the variable name from class_weight to class_weights in order to not to overwrite the imported module. Adjust accordingly when copying code from the comments.

$\endgroup$
4
  • 39
    $\begingroup$ For me, class_weight.compute_class_weight produces an array, I need to change it to a dict in order to work with Keras. More specifically, after step 2, use class_weight_dict = dict(enumerate(class_weight)) $\endgroup$
    – C.Lee
    Oct 13 '17 at 4:33
  • 10
    $\begingroup$ This doesn't work for me. For a three class problem in keras y_train is (300096, 3) numpy array. So the class_weight= line gives me TypeError: unhashable type: 'numpy.ndarray' $\endgroup$
    – Giraffe
    Dec 14 '17 at 10:25
  • 6
    $\begingroup$ @Lembik I had a similar problem, where each row of y is a one-hot encoded vector of the class index. I fixed it by converting the one-hot representation to an int like this: y_ints = [y.argmax() for y in y_train]. $\endgroup$
    – tkocmathla
    Apr 12 '18 at 14:19
  • 6
    $\begingroup$ What if I'm doing multiclass labeling so that my y_true vectors have multiple 1s in them: [1 0 0 0 1 0 0] for instance, where some x has labels 0 and 4. Even then, the total # of each of my labels is not balanced. How would I use class weights with that? $\endgroup$
    – axolotl
    Nov 25 '18 at 18:28
37
$\begingroup$

I use this kind of rule for class_weight :

import numpy as np
import math

# labels_dict : {ind_label: count_label}
# mu : parameter to tune 

def create_class_weight(labels_dict,mu=0.15):
    total = np.sum(list(labels_dict.values()))
    keys = labels_dict.keys()
    class_weight = dict()
    
    for key in keys:
        score = math.log(mu*total/float(labels_dict[key]))
        class_weight[key] = score if score > 1.0 else 1.0
    
    return class_weight

# random labels_dict
labels_dict = {0: 2813, 1: 78, 2: 2814, 3: 78, 4: 7914, 5: 248, 6: 7914, 7: 248}

create_class_weight(labels_dict)

math.log smooths the weights for very imbalanced classes ! This returns :

{0: 1.0,
 1: 3.749820767859636,
 2: 1.0,
 3: 3.749820767859636,
 4: 1.0,
 5: 2.5931008483842453,
 6: 1.0,
 7: 2.5931008483842453}
$\endgroup$
6
  • 4
    $\begingroup$ Why use log instead of just dividing the count of samples for a class by the total number of samples? I am assume there is something I don't understand goes into the param class_weight on model.fit_generator(...) $\endgroup$ May 4 '17 at 3:11
  • 1
    $\begingroup$ @startoftext That's how I did it, but I think you have it inverted. I used n_total_samples / n_class_samples for each class. $\endgroup$
    – colllin
    Oct 19 '17 at 17:34
  • 3
    $\begingroup$ In your example class 0 (has 2813 examples) and class 6 (has 7914 examples) have weight exactly 1.0. Why is that? The class 6 is few times bigger! You would want class 0 be upscaled and class 6 downscaled to bring them to the same level. $\endgroup$ Jan 16 '18 at 20:55
  • 2
    $\begingroup$ @VladislavsDovgalecs cuz he is doing it to smooth the weights. When imbalance in classes is measured by orders of magnitude, it's not very helpful to assign weights like 100. It's gonna harm bigger class: FPs on that scarce class with high weight $\endgroup$
    – apatsekin
    Mar 3 '20 at 18:14
  • 1
    $\begingroup$ If i have a imbalance as {0: 1300000, 1: 40, 2: 2000} . Is there any intuition as to how I can set the mu parameter here ? to handle the smoothing $\endgroup$
    – lamo_738
    Sep 14 '20 at 19:17
11
$\begingroup$

class_weight is fine but as @Aalok said this won't work if you are one-hot encoding multilabeled classes. In this case, use sample_weight:

sample_weight: optional array of the same length as x, containing weights to apply to the model's loss for each sample. In the case of temporal data, you can pass a 2D array with shape (samples, sequence_length), to apply a different weight to every timestep of every sample. In this case you should make sure to specify sample_weight_mode="temporal" in compile().

sample_weights is used to provide a weight for each training sample. That means that you should pass a 1D array with the same number of elements as your training samples (indicating the weight for each of those samples).

class_weights is used to provide a weight or bias for each output class. This means you should pass a weight for each class that you are trying to classify.

sample_weight must be given a numpy array, since its shape will be evaluated.

See also this answer.

$\endgroup$
9
$\begingroup$

Adding to the solution at https://github.com/keras-team/keras/issues/2115. If you need more than class weighting where you want different costs for false positives and false negatives. With the new keras version now you can just override the respective loss function as given below. Note that weights is a square matrix.

from tensorflow.python import keras
from itertools import product
import numpy as np
from tensorflow.python.keras.utils import losses_utils

class WeightedCategoricalCrossentropy(keras.losses.CategoricalCrossentropy):

    def __init__(
        self,
        weights,
        from_logits=False,
        label_smoothing=0,
        reduction=losses_utils.ReductionV2.SUM_OVER_BATCH_SIZE,
        name='categorical_crossentropy',
    ):
        super().__init__(
            from_logits, label_smoothing, reduction, name=f"weighted_{name}"
        )
        self.weights = weights

    def call(self, y_true, y_pred):
        weights = self.weights
        nb_cl = len(weights)
        final_mask = keras.backend.zeros_like(y_pred[:, 0])
        y_pred_max = keras.backend.max(y_pred, axis=1)
        y_pred_max = keras.backend.reshape(
            y_pred_max, (keras.backend.shape(y_pred)[0], 1))
        y_pred_max_mat = keras.backend.cast(
            keras.backend.equal(y_pred, y_pred_max), keras.backend.floatx())
        for c_p, c_t in product(range(nb_cl), range(nb_cl)):
            final_mask += (
                weights[c_t, c_p] * y_pred_max_mat[:, c_p] * y_true[:, c_t])
        return super().call(y_true, y_pred) * final_mask
$\endgroup$
7
$\begingroup$

Here's a one-liner using scikit-learn:

from sklearn.utils import class_weight
class_weights = dict(zip(np.unique(y_train), class_weight.compute_class_weight('balanced', np.unique(y_train), 
                y_train))) 
$\endgroup$
4
$\begingroup$

If tf dataset is used you cannot use the class_weights parameter. Insted return the weight from a parse_function in your pipeline

weight_arr = [1.5, 0.5] #define your custom weights
    
#create a lookup table
key_tensor = tf.constant(list(range(0, len(weight_arr))), dtype=tf.int64)
val_tensor = tf.constant(weight_arr)
init = tf.lookup.KeyValueTensorInitializer(key_tensor, val_tensor)
weight_table = tf.lookup.StaticHashTable(init,default_value=-1)

def parse_function(element):
    features = element{'image'}
    label_integer = element{'label'}

    weight = weight_table.lookup(label_integer) #find the weight based on label

    return features, label_integer, weight

ds = ds.map(parse_function)
model.fit(ds)...

First you create a lookup table, which maps the given label integer to class weight. Then you fetch the weight based on your label in the pipeline.

$\endgroup$
3
  • $\begingroup$ may I ask why I can't use the class weight when tf data is used? $\endgroup$
    – AveryLiu
    Aug 13 at 8:25
  • $\begingroup$ If a remember it correctly I read is somewhere in the documentation. It may not do any harm, thought it won't work since keras do not know how to add weights to your custom tf data $\endgroup$ Aug 13 at 8:31
  • 1
    $\begingroup$ I tried using class weights with tf data and it gave me weird errors. The shape of my label is [batch_size, seq_len], where each label is between [0, 3], and the shape of the y_pred is [batch_size, seq_len, 4]. Keras complains an invalid arugment error saying index 18 is not in the range of [0, 3]. I don't even know where the 18 comes from. Labels are double-checked to make sure they are all in the range of [0, 3]. I guess I may have to flatten the y_pred to 2-D array and label to 1-D array to use the class weight feature. $\endgroup$
    – AveryLiu
    Aug 13 at 8:43
0
$\begingroup$
from collections import Counter
itemCt = Counter(trainGen.classes)
maxCt = float(max(itemCt.values()))
cw = {clsID : maxCt/numImg for clsID, numImg in itemCt.items()}

This works with a generator or standard. Your largest class will have a weight of 1 while the others will have values greater than 1 depending on how infrequent they are relative to the largest class.

Class weights accepts a dictionary type input.

$\endgroup$
0
$\begingroup$

I found the following example of coding up class weights in the loss function using the minist dataset. See link here.

def w_categorical_crossentropy(y_true, y_pred, weights):
    nb_cl = len(weights)
    final_mask = K.zeros_like(y_pred[:, 0])
    y_pred_max = K.max(y_pred, axis=1)
    y_pred_max = K.reshape(y_pred_max, (K.shape(y_pred)[0], 1))
    y_pred_max_mat = K.equal(y_pred, y_pred_max)
    for c_p, c_t in product(range(nb_cl), range(nb_cl)):
        final_mask += (weights[c_t, c_p] * y_pred_max_mat[:, c_p] * y_true[:, c_t])
    return K.categorical_crossentropy(y_pred, y_true) * final_mask
$\endgroup$
0
$\begingroup$

You can use this function to get class weights and use them in model.fit():

    def get_class_weights(labels, one_hot=False):
        if one_hot is False:
            n_classes = max(labels) + 1
        else:
            n_classes = len(labels[0])
        class_counts = [0 for _ in range(int(n_classes))]
        if one_hot is False:
            for label in labels:
                class_counts[label] += 1
        else:
            for label in labels:
                class_counts[label.index(1)] += 1
        return {i : (1. / class_counts[i]) * float(len(labels)) / float(n_classes) for i in range(int(n_classes))}


    y1 = [0,0,0,1,1,2]
    y2 = [
        [1,0,0],
        [1,0,0],
        [1,0,0],
        [0,1,0],
        [0,1,0],
        [0,0,1],
    ]
    print(get_class_weights(y1))
    print(get_class_weights(y2, one_hot=True))
    '{0: 0.6666666666666666, 1: 1.0, 2: 2.0}'
    '{0: 0.6666666666666666, 1: 1.0, 2: 2.0}'
$\endgroup$
0
$\begingroup$

Dataset flower_photos.tgz

label count
daisy 633
dandelion 898
roses 641
sunflowers 699
tulips 799

Count the number of images in different categories

import numpy as np
import tensorflow as tf

directory = 'flower_photos'
datagen = tf.keras.preprocessing.image.ImageDataGenerator()
data = datagen.flow_from_directory(directory)
unique = np.unique(data.classes, return_counts=True)
labels_dict = dict(zip(unique[0], unique[1]))
print(labels_dict)
# Found 3670 images belonging to 5 classes.
# {0: 633, 1: 898, 2: 641, 3: 699, 4: 799}

Calculate the weights of different categories

import math


def get_class_weight(labels_dict):
    """Calculate the weights of different categories

    >>> get_class_weight({0: 633, 1: 898, 2: 641, 3: 699, 4: 799})
    {0: 1.0, 1: 1.0, 2: 1.0, 3: 1.0, 4: 1.0}
    >>> get_class_weight({0: 5, 1: 78, 2: 2814, 3: 7914})
    {0: 7.366950709511269, 1: 4.619679795255778, 2: 1.034026384271035, 3: 1.0}
    """
    total = sum(labels_dict.values())
    max_num = max(labels_dict.values())
    mu = 1.0 / (total / max_num)
    class_weight = dict()
    for key, value in labels_dict.items():
        score = math.log(mu * total / float(value))
        class_weight[key] = score if score > 1.0 else 1.0
    return class_weight


labels_dict = {0: 633, 1: 898, 2: 641, 3: 699, 4: 799}
print(get_class_weight(labels_dict))
# {0: 1.0, 1: 1.0, 2: 1.0, 3: 1.0, 4: 1.0}
labels_dict = {0: 5, 1: 78, 2: 2814, 3: 7914}
print(get_class_weight(labels_dict))
# {0: 1.0, 1: 3.749820767859636, 2: 1.0, 3: 3.749820767859636, 4: 1.0, 5: 2.5931008483842453, 6: 1.0, 7: 2.5931008483842453}
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.