How does exactly class_weight in Keras work?

Question

I'm working on a multi-label problem in Keras, using binary-cross-entropy loss function with sigmoid activation. Let's say I have 4 classes, so a response might look like this:

[1, 0, 0, 1]

Though a vast majority of the responses will actually be zero's all around

[0, 0, 0, 0]

The data is also very imbalanced between the different classes, and if i did a sum on y, the total number of positives in each class would look something like this

[2000, 500, 1000, 250]

So to give equal weight to each class I would think that I should construct a dictionary-like so

weights = {0: 2.5, 1: 10., 2: 5., 3: 20}

And feed that to the class_weight parameter in Keras.

Here is my question though:

If my response variable was binary (only first class for example), I would need to feed a dictionary that defines both a factor for the 0 an 1

weights = {0: 1, 1: 2.5}

Which I take to mean that positive samples are weighted higher than negatives. But, how do weighted samples work with the multi-label (or multi-class for that matter)? since in that case, I've only given weights to the classes in general, not specifically positives and negatives

This should be the loss function I'm working with

−(ylog(p)+(1−y)log(1−p))

And with binary-cross-entropy and sigmoid, this is applied to each class and the total loss is then the sum of all class losses for all samples.

But where exactly is the class weight applied here? Is it multiplied on the entire function? If that's the case, then it's not really fixing the class imbalance as I see it, since it also weighs the negatives higher, not just the positives? I would imagine that is desirable is to only multiply positive responses?

If anyone can clear up exactly how this all works so I can understand a bit more what's going on under the hood I would be grateful.

Answer 1

A weighted classification layer computes the weighted cross entropy loss for classification problems. Weighted cross entropy is an error measure between two continuous random variables. For prediction scores Y and training targets T, the weighted cross entropy loss between Y and T is given by

$$ F = -\frac{1}{N}\sum_{i=1}^{N}\sum_{k=1}^{M} w_i \cdot T_{ni} \cdot log(p_{ij}))$$

where $N$ is the number of instances, K is the number of classes, and w is a vector of weights for each class.

The cost function per label is measuring how good our classifier is doing per label, but it is affected by the number of instances $N$ that don't contain this label.