I'm trying to train a keras model that takes in samples, let's say $x_i$ for sample $i$, and predicts multiple independent labels, $\hat{y}_{ij}$, such that $\hat{y}_{ij} = 1$ if the model predicts sample $x_i$ to have label $j$, and $0$ otherwise.
The problem I'm running into repeatedly is that the number of samples per label is not balanced across labels. Particularly, here's a count of how many samples each label has (there's double counting, of course, since we're dealing with multi-label classification). My model has 23 labels.
total: 6790
Counter({22: 4702, 0: 1749, 12: 130, 8: 43, 16: 39,
15: 30, 17: 24, 20: 17, 4: 13, 5: 13, 19: 9,
6: 7, 7: 6, 2: 4, 10: 4})
I'm not sure how to weight them by size, however. I saw this question, but it uses one-hot encodings of labels, i.e., assumes there's only one true label per sample. It is essentially a sparse categorical classification problem. I also ran into this question, but I believe that question is concerned with per-sample weighting of classes, somehow. The accepted answer does not state how to compute weights across training samples.
What I tried so far:
I computed arbitrary weights of each label relative to its occurrences vs the total samples.
class_weight = dict((c,round((1/v)*total,1)) for c,v in class_occ.items()), where class_occ is the counter shown above. Here's what I get:
class_weights: {0: 3.9, 2: 1697.5, 4: 522.3, 5: 522.3, 6: 970.0, 7: 1131.7, 8: 157.9, 10: 1697.5, 12: 52.2, 15: 226.3, 16: 174.1, 17: 282.9, 19: 754.4, 20: 399.4, 22: 1.4}
Then, while training the model, I supply it as a kwarg:
model.fit(x=...., class_weight=class_weight)
However, the model doesn't really learn thereafter, and accuracy remains at a low of 0.1 or so. To give some context, here's how I compiled the model:
out = Dense(numclasses+1, activation='sigmoid', name='out')(dense_top)
model = Model(inputs=[input], outputs=[out])
model.compile(optimizer="adam", loss="binary_crossentropy",
metrics=["categorical_accuracy"])
And here's how the training goes (blue is the loss):
I'm not sure what's going wrong- am I even using the class weights correctly? Without the class weights, the model would just predict the dominant class(es) all the time and the accuracy would a baseline of the proportion of samples that had the dominant class(es). But looks like it does poorer than random now.
Just in case it helps, here's my model.summary() output:
Layer (type) Output Shape Param #
=================================================================
input_1 (InputLayer) (None, None, 128) 0
_________________________________________________________________
lstm_1 (LSTM) (None, 192) 246528
_________________________________________________________________
dense_1 (Dense) (None, 160) 30880
_________________________________________________________________
dense_2 (Dense) (None, 128) 20608
_________________________________________________________________
dense_3 (Dense) (None, 128) 16512
_________________________________________________________________
dense_4 (Dense) (None, 64) 8256
_________________________________________________________________
dense_top_1 (Dense) (None, 64) 4160
_________________________________________________________________
dropout_42 (Dropout) (None, 64) 0
_________________________________________________________________
dense_top_2 (Dense) (None, 64) 4160
_________________________________________________________________
dropout_43 (Dropout) (None, 64) 0
_________________________________________________________________
dense_top_3 (Dense) (None, 64) 4160
_________________________________________________________________
dropout_44 (Dropout) (None, 64) 0
_________________________________________________________________
dense_top_4 (Dense) (None, 64) 4160
_________________________________________________________________
dropout_45 (Dropout) (None, 64) 0
_________________________________________________________________
dense_top_5 (Dense) (None, 32) 2080
_________________________________________________________________
out (Dense) (None, 23) 759
=================================================================
Total params: 342,263
Trainable params: 19,479
Non-trainable params: 322,784
The reason for the non-trainable params is that I'm using transfer learning: the first bunch of layers are from another model that did binary classification on the same data, for sets of related labels ('does a sample $x$ have any label from label-set A vs. label-set B?').
Just in case that's the problem, I also set trainable to True in a separate instance, but it did not help with training and getting the model to work.
