1
$\begingroup$

I have a multiclass classification data where the target has 11 classes. I am trying to build a Neural Net using Keras. I am using softmax as activation function and categorical_crossentropy as the loss function. I have one hot encoded the target before passing it into the net. The issue I am facing is which Keras metric should I use for this purpose? The official documentation does not mention which metric is suitable for multiclass classification.

This link mentions to use categorical_accuracy as the metric for multiclass classification but other than that, all other question on this site are about multilabel classification metrics like this and this link.

Is there any implementation of lets say f1_score in Keras using the custom metric function, since f1_score is the go to metric for multiclass classification I guess?

EDIT 1:

Would something like this work using the custom metric functionality in Keras?

from sklearn.metrics import f1_score

def my_metric_fn(y_true, y_pred):
    f1 = f1_score(y_true, y_pred)
    return f1

model = Sequential()

model.add(Dense(input_dim = 12, units = 128, activation = 'relu', kernel_initializer = 'he_uniform'))
model.add(Dense(units = 64, activation = 'relu', kernel_initializer = 'he_uniform'))
model.add(Dense(units = 32, activation = 'relu', kernel_initializer = 'he_uniform'))
model.add(Dense(units = 11,  activation = 'softmax', kernel_initializer = 'glorot_uniform'))
model.compile(optimizer = 'adam', loss = 'categorical_crossentropy', metrics = [my_metric_fn])
history = model.fit(train_x5, train_encoded, validation_split = 0.2, epochs = 5, batch_size = 1000)

EDIT 2: This is giving me an error in the last line as follows: OperatorNotAllowedInGraphError: using a tf.Tensor as a Python bool is not allowed: AutoGraph did convert this function. This might indicate you are trying to use an unsupported feature.

$\endgroup$
6
  • $\begingroup$ For multiclass classification you can simply use a categorical cross entropy loss function. Depending on whether or not the values are one-hot encoded you would have to use either the sparse categorical cross entropy loss or the normal categorical cross entropy loss. $\endgroup$
    – Oxbowerce
    Dec 14, 2021 at 12:58
  • $\begingroup$ I am asking about the metric to use not the loss. $\endgroup$
    – spectre
    Dec 14, 2021 at 13:43
  • $\begingroup$ It would help if you said what you would like the metric to evaluate and why it should be different from the loss function. $\endgroup$
    – Dave
    Dec 14, 2021 at 14:53
  • $\begingroup$ The metric needs to be any metric that is used in multiclass classification like f1_score or kappa. But Keras has not yet implemented them yet unlike sklearn. $\endgroup$
    – spectre
    Dec 14, 2021 at 15:37
  • $\begingroup$ it should be different from the loss function. I find this statement interesting as it implies that it is not necessary to use metrics to evaluate the model. Does this mean we can just use the loss function for the evaluation and not the metrics? $\endgroup$
    – spectre
    Dec 14, 2021 at 15:38

1 Answer 1

3
$\begingroup$

One option is to implement F1 score in Keras:

from tensorflow.keras import backend as K

def f1(y_true, y_pred):    
    def recall_m(y_true, y_pred):
        TP = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
        Positives = K.sum(K.round(K.clip(y_true, 0, 1)))
        
        recall = TP / (Positives+K.epsilon())    
        return recall 
    
    
    def precision_m(y_true, y_pred):
        TP = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
        Pred_Positives = K.sum(K.round(K.clip(y_pred, 0, 1)))
    
        precision = TP / (Pred_Positives+K.epsilon())
        return precision 
    
    precision, recall = precision_m(y_true, y_pred), recall_m(y_true, y_pred)
    
    return 2*((precision*recall)/(precision+recall+K.epsilon()))
$\endgroup$
3
  • 1
    $\begingroup$ I would need to call the function in the model.compile where I would need y_pred which I can only get after I fit and evaluate the model. But fitting and evaluating comes after compiling so how can I call the function in model.compile? Even if I pass the function as a list (as I have done in EDIT 1), I will still get the same error as in EDIT 2! $\endgroup$
    – spectre
    Dec 17, 2021 at 12:11
  • 2
    $\begingroup$ Ok I implemented it in my model and its works! NVM my previous comment!XD $\endgroup$
    – spectre
    Dec 23, 2021 at 12:52
  • 1
    $\begingroup$ You could also use the built in method here tensorflow.org/api_docs/python/tf/keras/metrics/F1Score $\endgroup$ Nov 30, 2023 at 21:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.