Class weight degrades Multi Label Classification Performance

Question

I noticed something strange while I was conducting a multiple label classification problem via keras neural network. My data set consist of imbalance data with 12 features and 25 possible labels. When I instantiate my model with no class weight I get a precision of 97%, recall of 13%, subset accuracy of 14%, f1-score of 23% using the micro average. When I apply class weight these scores are significantly reduced to the below.

('Accuracy', 0.1757093081134893)
('Precision:', 0.19632925472747498)
('Recall', 0.1637291280148423)
F1 -score 0.178553363682

Also I calculate the weights with below code that I copied and modify from a previous post:

def class_out(s):
      y_classes = s#.idxmax(1, skipna=False)

      # Instantiate the label encoder
      le = LabelEncoder()

      # Fit the label encoder to our label series
      le.fit(list(y_classes))

      # Create integer based labels Series
      y_integers = le.transform(list(y_classes))

      #print y_integers
      # Create dict of labels : integer representation
      labels_and_integers = dict(zip(y_classes, y_integers))

      print labels_and_integers
      class_weights = compute_class_weight('balanced', np.unique(y_integers), y_integers)
      sample_weights = compute_sample_weight('balanced', y_integers)

      class_weights_dict = dict(zip(le.transform(list(le.classes_)), class_weights))
      class_sweights_dict = dict(zip(le.transform(list(le.classes_)), sample_weights))


      print class_weights_dict

      return class_weights_dict

Also see a sample of the model:

batch_size = 100

weights = class_out(df_all['tag'])

model = Sequential()
model.add(Dense(10, activation="relu", input_shape=(12,)))

#model.add(Dense(10, activation='relu'))

#model.add(Dense(8, activation='relu'))
#model.add(Dropout(0.50))
model.add(Dense(25, activation="sigmoid"))
model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy',precision,mcor,recall, f1])

model.fit(X_train, Y_train, batch_size=batch_size, epochs=15,class_weight=weights,
      verbose=1,validation_data=(test, target_test))

Is there a reason to believe that the model performance is best without class weights ?

Answer 1

Adding class weight but not changing the way you measure performance will usually degrade overall performance as it is designed to allow increased loss on lower-weighted classes.

I would recommend also weighting your accuracy measures. This is a bit tricky with accuracy/precision etc. so maybe calculated the weighted logloss and compare it to the unweighted logloss of the unweighted model.

Basically it comes down to the question: are you happy with a model that performs worse overall but better on your heavily weighted classes?

Answer 2

It can also depend on how imbalanced the data is. If one class has 97% of the instances, then the model will always want to predicts that class. Have a close look at the prediction from your model:

pred_class = model.predict_classes(X_test)
pred = model.predict(X_test)

So, if the prediction is always the same class you have a problem.

Also, I notice some odd choices in your above code. When compiling your model change loss='binary_crossentropy' to loss='categorical_crossentropy' for multiclass classifications. I would also use softmax and not sigmoid for the exact same reason.