2
$\begingroup$

I'm trying to train a keras model that takes in samples, let's say $x_i$ for sample $i$, and predicts multiple independent labels, $\hat{y}_{ij}$, such that $\hat{y}_{ij} = 1$ if the model predicts sample $x_i$ to have label $j$, and $0$ otherwise.

The problem I'm running into repeatedly is that the number of samples per label is not balanced across labels. Particularly, here's a count of how many samples each label has (there's double counting, of course, since we're dealing with multi-label classification). My model has 23 labels.

total: 6790
Counter({22: 4702, 0: 1749, 12: 130, 8: 43, 16: 39,
         15: 30, 17: 24, 20: 17, 4: 13, 5: 13, 19: 9,
         6: 7, 7: 6, 2: 4, 10: 4})

I'm not sure how to weight them by size, however. I saw this question, but it uses one-hot encodings of labels, i.e., assumes there's only one true label per sample. It is essentially a sparse categorical classification problem. I also ran into this question, but I believe that question is concerned with per-sample weighting of classes, somehow. The accepted answer does not state how to compute weights across training samples.

What I tried so far: I computed arbitrary weights of each label relative to its occurrences vs the total samples. class_weight = dict((c,round((1/v)*total,1)) for c,v in class_occ.items()), where class_occ is the counter shown above. Here's what I get:

class_weights: {0: 3.9, 2: 1697.5, 4: 522.3, 5: 522.3, 6: 970.0, 7: 1131.7, 8: 157.9, 10: 1697.5, 12: 52.2, 15: 226.3, 16: 174.1, 17: 282.9, 19: 754.4, 20: 399.4, 22: 1.4}

Then, while training the model, I supply it as a kwarg:

model.fit(x=...., class_weight=class_weight)

However, the model doesn't really learn thereafter, and accuracy remains at a low of 0.1 or so. To give some context, here's how I compiled the model:

out = Dense(numclasses+1, activation='sigmoid', name='out')(dense_top)

model = Model(inputs=[input], outputs=[out])
model.compile(optimizer="adam", loss="binary_crossentropy",
              metrics=["categorical_accuracy"])

And here's how the training goes (blue is the loss): model-fit-history

I'm not sure what's going wrong- am I even using the class weights correctly? Without the class weights, the model would just predict the dominant class(es) all the time and the accuracy would a baseline of the proportion of samples that had the dominant class(es). But looks like it does poorer than random now.

Just in case it helps, here's my model.summary() output:

Layer (type)                 Output Shape              Param #   
=================================================================
input_1 (InputLayer)         (None, None, 128)         0         
_________________________________________________________________
lstm_1 (LSTM)                (None, 192)               246528    
_________________________________________________________________
dense_1 (Dense)              (None, 160)               30880     
_________________________________________________________________
dense_2 (Dense)              (None, 128)               20608     
_________________________________________________________________
dense_3 (Dense)              (None, 128)               16512     
_________________________________________________________________
dense_4 (Dense)              (None, 64)                8256      
_________________________________________________________________
dense_top_1 (Dense)          (None, 64)                4160      
_________________________________________________________________
dropout_42 (Dropout)         (None, 64)                0         
_________________________________________________________________
dense_top_2 (Dense)          (None, 64)                4160      
_________________________________________________________________
dropout_43 (Dropout)         (None, 64)                0         
_________________________________________________________________
dense_top_3 (Dense)          (None, 64)                4160      
_________________________________________________________________
dropout_44 (Dropout)         (None, 64)                0         
_________________________________________________________________
dense_top_4 (Dense)          (None, 64)                4160      
_________________________________________________________________
dropout_45 (Dropout)         (None, 64)                0         
_________________________________________________________________
dense_top_5 (Dense)          (None, 32)                2080      
_________________________________________________________________
out (Dense)                  (None, 23)                759       
=================================================================
Total params: 342,263
Trainable params: 19,479
Non-trainable params: 322,784

The reason for the non-trainable params is that I'm using transfer learning: the first bunch of layers are from another model that did binary classification on the same data, for sets of related labels ('does a sample $x$ have any label from label-set A vs. label-set B?'). Just in case that's the problem, I also set trainable to True in a separate instance, but it did not help with training and getting the model to work.

$\endgroup$
2
  • $\begingroup$ I encountered a very similar problem. Have you found a solution? Thanks, Jenny $\endgroup$ Feb 18, 2019 at 9:11
  • $\begingroup$ Unfortunately not; I ended up picking more data and resampling it so each sample got a nearly equal number of examples $\endgroup$
    – axolotl
    Feb 19, 2019 at 22:17

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.