3
$\begingroup$

I noticed something strange while I was conducting a multiple label classification problem via keras neural network. My data set consist of imbalance data with 12 features and 25 possible labels. When I instantiate my model with no class weight I get a precision of 97%, recall of 13%, subset accuracy of 14%, f1-score of 23% using the micro average. When I apply class weight these scores are significantly reduced to the below.

('Accuracy', 0.1757093081134893)
('Precision:', 0.19632925472747498)
('Recall', 0.1637291280148423)
F1 -score 0.178553363682

Also I calculate the weights with below code that I copied and modify from a previous post:

def class_out(s):
      y_classes = s#.idxmax(1, skipna=False)

      # Instantiate the label encoder
      le = LabelEncoder()

      # Fit the label encoder to our label series
      le.fit(list(y_classes))

      # Create integer based labels Series
      y_integers = le.transform(list(y_classes))

      #print y_integers
      # Create dict of labels : integer representation
      labels_and_integers = dict(zip(y_classes, y_integers))

      print labels_and_integers
      class_weights = compute_class_weight('balanced', np.unique(y_integers), y_integers)
      sample_weights = compute_sample_weight('balanced', y_integers)

      class_weights_dict = dict(zip(le.transform(list(le.classes_)), class_weights))
      class_sweights_dict = dict(zip(le.transform(list(le.classes_)), sample_weights))


      print class_weights_dict

      return class_weights_dict

Also see a sample of the model:

batch_size = 100

weights = class_out(df_all['tag'])

model = Sequential()
model.add(Dense(10, activation="relu", input_shape=(12,)))

#model.add(Dense(10, activation='relu'))

#model.add(Dense(8, activation='relu'))
#model.add(Dropout(0.50))
model.add(Dense(25, activation="sigmoid"))
model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy',precision,mcor,recall, f1])

model.fit(X_train, Y_train, batch_size=batch_size, epochs=15,class_weight=weights,
      verbose=1,validation_data=(test, target_test))

Is there a reason to believe that the model performance is best without class weights ?

$\endgroup$
5
$\begingroup$

Adding class weight but not changing the way you measure performance will usually degrade overall performance as it is designed to allow increased loss on lower-weighted classes.

I would recommend also weighting your accuracy measures. This is a bit tricky with accuracy/precision etc. so maybe calculated the weighted logloss and compare it to the unweighted logloss of the unweighted model.

Basically it comes down to the question: are you happy with a model that performs worse overall but better on your heavily weighted classes?

|improve this answer|||||
$\endgroup$
  • $\begingroup$ I prefer a model that performs satisfactory on precision in terms of information retrieval. The goal of the model is to tag data columns (numeric data) in a table with the appropriate tags. E.g I can have a column that represents the water levels measurements for a river bank. Therefore, I can tag that column with 'river levels' and 'water levels'. Is it ok then to use the precision_score(target_test, pred, average='weighted'), where average= weighted for the metric function to adjust its performance ? $\endgroup$ – DPascal Feb 20 '18 at 18:22
  • $\begingroup$ Looks like the sklearn precision score weights each class by its number of positive values, but I'm not sure what weighting you're using? If it's the same as your weighting then yeh it's fine. $\endgroup$ – jshep Feb 20 '18 at 18:45
  • $\begingroup$ I'm using class_weights = compute_class_weight('balanced', np.unique(y_integers), y_integers) function from the above code. $\endgroup$ – DPascal Feb 20 '18 at 18:57
1
$\begingroup$

It can also depend on how imbalanced the data is. If one class has 97% of the instances, then the model will always want to predicts that class. Have a close look at the prediction from your model:

pred_class = model.predict_classes(X_test)
pred = model.predict(X_test)

So, if the prediction is always the same class you have a problem.

Also, I notice some odd choices in your above code. When compiling your model change loss='binary_crossentropy' to loss='categorical_crossentropy' for multiclass classifications. I would also use softmax and not sigmoid for the exact same reason.

|improve this answer|||||
$\endgroup$
  • $\begingroup$ he states hes doing a multi-label model, hence his choice of loss and activation function is correct. $\endgroup$ – AaronDT Aug 9 '18 at 10:41
  • $\begingroup$ why would you choose 'binary_crossentropy' in the case of a multi-label model? reddit.com/r/learnmachinelearning/comments/88g8zf/… $\endgroup$ – cJc Aug 9 '18 at 11:55
  • $\begingroup$ There is a slight difference between a multi-label and multi-class szenario. In a multi-class scenario, you want to predict one and only one class from many. Hence you would be using softmax and categorical-crossentropy since this would optimize the model such that the probability of one class is increased while other classes are penelized. In a multi-label case there are many labels that could appear - hence you dont want to penalize other classes in favor of just one being 1. See here for more info pyimagesearch.com/2018/05/07/… $\endgroup$ – AaronDT Aug 9 '18 at 14:26
  • $\begingroup$ tnx for the clarification! $\endgroup$ – cJc Aug 9 '18 at 15:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.